Useful links for responsive design / mobile web development

I’ve just been in the process of modifying an existing website to work a bit better on mobiles / tablets. The site in question already had a responsive design, being based on the WordPress 2012 theme, but I wasn’t quite happy with exactly how it was displaying.

There were a few websites that proved quite helpful in helping me getting things working as I wanted. I’d like to share them here so I have a reference for similar problems in the future, and if you’re reading this, you may find them helpful too.

Continue reading

PHP script to fix server log file with records not in order

Okay, I doubt this post will be useful to anyone else, but I thought I might as well post it since I haven’t updated this blog in a while.

On ‘my’ web server I have some sites that are served by a shared Apache process. This stores daily logs, for one week, rotating the log numbering each day. I have a cron job that runs once a week and concatenates the past week’s logs into a single file, which is then stored elsewhere.

Today I was trying to run awstats to generate stats from the last months worth of logs (I check my sites stats once every month). However, I realised that my cron job was concatenating the daily logs into the weekly log file in the wrong order. This resulted in awstats only picking up the first day from each weekly log, as the rest of the log file had requests with earlier timestamps. It seems that awstats works through the log file chronologically.

So, to get my stats I had to re-order the log files so that the entries were listed chronologically as they should be. To do this I decided to use PHP, mostly just because that is the language I am most familiar with.

The script simply loops through the log file, adding each line to an array, using the request timestamp as the key. Then the array can be sorted by the key, and output back to the file.

I ran the script using the PHP CLI, and found I had to set a large memory limit to avoid out of memory errors. The exact amount of memory needed will depend on how large your logs are.

<?php

//Usage: ~/path/to/php/bin/php -d memory_limit=128M ./whatever-you-name-this-file.php

$sites=array('xoogu', 'xoogu-static1',
			 'another-domain', 'another-domain-static1',
			 'domain3', 'domain3-blog');
$dates=array('20131215', '20131222', '20131229', '20140105' );

//loop through all the sites
foreach($sites as $site){
	//loop through all the dates
	foreach($dates as $date){
		//array to hold each line from the log file
		$records=array();
		//log file location
		$logFile="/path/to/logs-archive/$site/$site-access-$date.log";
		//open the logfile for reading
		$handle=fopen($logFile, 'r');
		//initialise a counter used to ensure we don't loose log entries that occurred at the same time
		$i=0;
		//read the log one line at a time
		while($str = fgets($handle)){
			//parse the date from the log entry
			$recordDate = new DateTime(substr($str, strpos($str, '[')+1, 26));
			//store the log entry by the date with our counter concatenated on the end to avoid overwriting an existing record for a different request that occured at the same time
			$records[$recordDate->format(DateTime::ATOM).$i]=$str;
			//increment the counter
			$i++;
		}
		//close the file
		fclose($handle);
		
		//sort our records to be in date order
		ksort($records);
		//rename the unsorted log file so we still have a copy if anything goes wrong
		rename($logFile, "$logFile-unsorted");
		//the original file has been renamed, so now create a new version of the file and open it
		$handle=fopen($logFile, 'w');
		//write the records in order to the file
		foreach($records as $record){
			fwrite($handle, $record);
		}
		//close the file
		fclose($handle);
		//output progress to screen
		echo "$site-access-$date.log sorted\n";
	}
}

A couple of points to make:

My weekly logs are stored in the format $site-access-$date.log, e.g. for this site the log would be xoogu-access-2014-01-07.log.

The format of the logs is like this:

208.115.113.87 - - [28/Dec/2013:03:15:45 +0000] "GET /robots.txt HTTP/1.0" 200 125 "-" "Mozilla/5.0 (compatible; Ezooms/1.0; help@moz.com)"

So if your log uses a different format (possible), or you use a different structure for storing your logs (very likely), then you’d need to modify the script appropriately. However, I’d be pretty surprised if anyone other than me would have a need for this script. (If you do, leave a comment below). And I shouldn’t even need it any more now I’ve corrected my weekly log archiving cron job.

How to automatically watermark or batch watermark photos using ImageMagick

In this article I’ll share a couple of ways to use ImageMagick to add a watermark to a photo. ImageMagick is a command line program, which is perfect for this job as that means it can scripted.

How to automatically watermark or batch watermark photos using ImageMagick

I use it on my photography website so I can upload original size photos, then the images can automatically be resized and have a watermark added. I won’t go into detail on the image resizing in this article, as I have a feeling the article will already be quite long. But maybe I will cover that aspect in a future article.

Continue reading

Justified Horizontal Menu – CSS & Javascript Solutions

Recently I wanted to use a horizontal navigation menu on a website. I wanted the items in the menu to be evenly distributed across the width of the page, but also butting up against each other, with no space between each item. The other requirement was that the spacing between the text of each item should be equal.

Fluid width menu with justified items, each one with the same padding applied
Fluid width menu with justified items, each one with the same padding applied

Same menu on a smaller width screen, demonstrating the fluid sizing
Same menu on a smaller width screen, demonstrating the fluid sizing

Continue reading

Optimising MySQL queries – JOIN vs IN (subquery)

When searching for some advice relating to an SQL query I was writing recently, I read some advice that you should try to use JOINs rather than IN with a subquery. I modified the particular query I was working on to use a JOIN instead of a subquery, and lo and behold, it was much faster.

So I decided to try and rewrite some other queries I’d written using IN with a subquery. Re-written to use JOINs instead, they should be much faster. But rather than just blindly rewriting the queries, I decided to run a few tests.

Continue reading

Space separated rel values – HTML vs. ATOM

Just a quick note regarding an issue I came across today. HTML allows space separated rel values to indicate link types. I was using this to indicate a URL was both self and canonical, i.e.

<link rel="self canonical" href="http://www.xoogu.com/2013/space-separated-rel-values-html-vs-atom/" />

I was using the same code in both the HTML and ATOM feed versions of my site, but it turns out that ATOM does not allow space separated rel values. So for ATOM you would require two links, like:

<link rel="self" href="http://www.xoogu.com/2013/space-separated-rel-values-html-vs-atom/" />
<link rel="canonical" href="http://www.xoogu.com/2013/space-separated-rel-values-html-vs-atom/" />

Or don’t include a canonical link in your ATOM feed.

Differences between windows batch and linux bash shell script syntax

Carrying on from my previous post about using scripts created in Windows on Linux, here are some comparisons to show the syntax differences between Windows batch scripts and bash scripts.

Comments

Windows batch script

REM some comment

or

:: some comment

Linux shell script

# some comment

For loop through files in directory

Windows batch script

FOR %%i IN (%1\*) DO (
REM do stuff here
)

Linux shell script

for i in "$1"/*
do
# do stuff here
done

Setting a variable to a number and then printing the variable

Windows batch script

SET /A varInteger=0
echo The value of varInteger is %varInteger%

Linux shell script

varInteger=0
echo "The value of varInteger is $varInteger"

Execute a command and store the output in a variable

Windows batch script

FOR %%x IN ('command to be executed') DO SET result=%%x

Linux shell script

result=`command to be executed`

If statement

Windows batch script

IF %somevar% GEQ %someothervar% (
REM do something
)

Linux shell script

if [ $somevar -ge $someothervar ]
then
# do something
fi

(GEQ and -ge is the greater than comparison operator, both bash and windows batch scripts will let you use a range of comparison operators.)

While loop

Windows batch script

:loop
IF !somevar! GEQ %someothervar% GOTO endofloop
REM do something that increases the value of !somevar!
GOTO loop
:endofloop

Linux shell script

while [ ! $somevar -ge $someothervar ]
do
# do something that increases the value of $somevar
done

(If you’re wondering why I used !somevar! instead of %somevar% in the windows script, this is because in a while loop you will be wanting to update the variable. See this article for more info: Windows batch scripting: EnableDelayedExpansion.)

Check last command executed OK, if not print error message and exit

Windows batch script

somecommand
IF %ERRORLEVEL% NEQ 0 (
	echo "somecommand failed, exiting"
	GOTO :EOF
)
REM the EOF label should be located at the end of the file
:EOF

Linux shell script

somecommand
if [ $? -ne 0 ]
then
	echo "somecommand failed, exiting"
	exit 1
fi

Zeropad a number to 4 digits

Windows batch script

SET zeropadded=000%somenumber%
REM Trim zeropadded to only four digits, from the end
echo "4 digit padded = %zeropadded:~-4%"

Linux shell script

zeropadded=`printf "%04d" $somenumber`
echo "4 digit padded = $zeropadded"

Using scripts created in Windows on Linux

A few weeks ago I tried converting a Windows batch script to a Linux shell (bash) script. The syntax between windows and linux shell scripts is quite different, but I also had a problem other than syntax differences.

After modifying the script to use linux syntax (e.g. proper loops instead of labels and GOTOs), I ran the script but it returned the following message:

/bin/bash^M: bad interpreter: No such file or directory

A quick google revealed the problem was the file using windows line endings (\r\n), while linux uses \n for line endings. Luckily there is a handy utility found in most linux distros that you can use to convert files containing windows line endings to linux line endings.

In Ubuntu I used fromdos, though other distros might use dos2unix. In terminal just type

fromdos myscriptfromwindows.sh

Easy, eh?

If you want to convert a file from linux to windows line endings instead, you can just use todos or unix2dos.

Selecting distinct merged results from multiple tables in MySQL

I had an issue recently where I needed to get a list of keywords for one of my websites. The website uses two databases, one for the blog, and a different one for the main part of the site. So I wanted to select the unique (distinct) keywords from the keyword tables on both databases.

This is the query I ended up with, which I will explain in a moment:

SELECT `name` COLLATE utf8_unicode_ci FROM `db_1`.`tags_table` UNION SELECT `Subject` COLLATE utf8_unicode_ci FROM `db_2`.`keywords_table` INTO OUTFILE '/tmp/tags.txt';

So, to start with, we need to get the results from both tables merged together. I did this by using two queries joined by a UNION.

SELECT `name` FROM `tags_table` UNION SELECT `Subject` FROM `keywords_table`

Because the tables are in different databases, we need to prefix the table name followed by a dot (.) when specifying the table to retrieve the records from. E.g. db_1.tags_table selects from the table tags_table in the database db_1. You will also need to be signed into mysql as a user that can access both databases (e.g. the root user) if you want to query across multiple databases.

SELECT `name` FROM `db_1`.`tags_table` UNION SELECT `Subject` FROM `db_2`.`keywords_table`

The two tables I needed to select from had different collations. So I also needed to specify the collation in the query to avoid an Illegal mix of collations error.

SELECT `name` COLLATE utf8_unicode_ci FROM `db_1`.`tags_table` UNION SELECT `Subject` COLLATE utf8_unicode_ci FROM `db_2`.`keywords_table`

The default behaviour of UNION is that duplicate rows are removed from the result. So there is no need to use UNION DISTINCT (though you can if you want). If you do want duplicate rows, use UNION ALL.

All that’s left to do is to get the result into a text file. To do this use INTO OUTFILE at the end of the query, followed by the filename the results should be written to. The file must not already exist, and the mysql user must have write permission for the directory where you want to save the folder. The easiest thing to do is to write the file to the /tmp directory, and then you can move the file from there to wherever you want manually.

SELECT `name` COLLATE utf8_unicode_ci FROM `db_1`.`tags_table` UNION SELECT `Subject` COLLATE utf8_unicode_ci FROM `db_2`.`keywords_table` INTO OUTFILE '/tmp/tags.txt';