For some reason, it seems people are wary of server log analysis. I don’t know why. Whether it’s because sever log analysis isn’t that well known, or because of those rows in EXCEL can be quite daunting. Maybe it’s just because they don’t want to speak with their web developer (I understand the last point, but if you’re nice to your dev team they can make your life a lot easier).
I am not going to cover how to access server logs, you can read more about this in some of my other articles including what to ask a dev and how to convert your files into excel.
In this article I am going to explain the top five things you should be looking out for.
Is Google crawling your site?
This is the most important element if you want to rank in Google. Google needs to be able to crawl your website to add it to their index and for potential customers to find your site.
How often is Google crawling your site?
This is almost as important as whether it is crawling your site at all. If Google is only crawling your website once a month then any changes you make in that period are not going to affect your rankings. If you do make some serious changes to your site, use the tools in Google Search Console to request a visit from the Google bot.
How deep into your site does Google go?
Your server logs are the only place you can accurate discover how the Google bot is behaving when it visits your site. There is a fundamental truth here. If you want a page to rank in Google it needs to be crawled.
If sections of your site aren’t being crawled, you can quickly identify these to try and understand why this may be. Has a developer accidently put nofollow noindex tags on any of your pages? Are there no internal links to these pages? Has someone added nofollow into the robots file? Without this knowledge it’s almost impossible to identify the problem let alone fix it.
Response codes
302’s
It’s very rare that you actually want to use a 302. Usually, these should be 301’s and have been implemented wrongly. Identifying a full list of errors means you can hand them all to your developer in one easy list to be fixed.
You can also look to see where the links are on the site and, instead of using a redirect link, just change the original link to the correct link. This saves on server resources and also will help the page to rank better as link juice is lost through redirects.
404s
You can crawl your site and find 404 pages, but if it’s an old page with a link from a third party site, the crawlers won’t pick up on this. However, you can spot this in your logs. This is why it’s good practice to combine crawl reports and server log analysis. Whilst both return similar results, combining the two together provides great analysis.
Page sizes
Yes, there are other ways to get this information, from crawl reports for example. However, getting the data from your server log analysis shows you the size Google is downloading. This knowledge is especially important as mobile page speed is going to become the third biggest ranking factor for mobile serps. Remember, the larger a page the slower it is to load.
As an example, a clients junior designer decided to create a beautiful piece of work and add it to a page on the website. The page itself didn’t drive a lot of sales and wasn’t deemed essential to the business, but the way the designer created the artwork meant that particular page became the largest on the site.
As a result Google was wasting its resources trying to download a huge page which the company had decided wasn’t a huge priority. The issue was resolved by a senior designer he converting the image into a PNG. A user couldn’t tell the difference, but the size of the page dropped massively.
The issue came to light because I was able to analyse the data and identify the problem simply by sorting the data in EXCEL. A and quick sort on the column ‘PageSize’, and I was quickly able to see the largest pages and fix the issues
If you have the information in this article useful, why not download our PDF on the three key things you should be looking for to take you to the next stage of server log analysis?
I am the Managing Director of Coreter Media and have been in Digital Marketing since 2009. Initially in-house working for some of the UK’s biggest brands, but now I run my own agency helping small businesses grow.