Hi, my name is Andrew Halliday. In this short video, I’ll be showing you how to add a robots.txt file using cPanel. This is especially useful for anyone with a WordPress site, but it applies to any type of site you’re managing. If you have 2 minutes and access to cPanel, you can add a robots file.
It’s that simple.
In the text below, I’ll outline the importance of a robots file and why you should create one.
What does a robots.txt file do?
The robots.txt file tells search engine bots what pages they can and can’t crawl on your site. This is a great way of making them crawl the pages you want them to and keeping them out of areas you don’t want them to crawl.
All the good bots will listen to this page, but the bad bots (it’s in their name) ignore these requests. They are, after all, only requests and bots don’t have to follow the rules.
If you look in your server logs, you will see that most bots request pages several times a day (depending how frequently they crawl your site). So, if you make some changes to your file, it might take a few hours or even days for the bots to take notice.
The most simple robots.txt file is:
User-agent: *
Disallow: /search
This is saying that all bots are ok to crawl your site, but not to crawl search pages.
You can be more specific and give different bots different requests.
For example, I recently told Googlebot not to crawl the forum of a site I manage, as there are quite a few issues and I don’t want Google wasting it resources here. However, I still want to the server log data for the forum to help fix any issues, so I let the other bots still crawl it.
The reason why the page is so important is that without it, you’re allowing the bots crawl your entire site without giving them any requests, when there may be certain sections of your site you don’t want search engines crawling.
How to add robots.txt file using cPanel
Firstly, you’ll need to log into your cPanel.
Once you’re in, scroll down to the file section and click on File Manager.
Make sure the ‘web root’ folder is highlighted and click GO.
A new tab will open, showing all the files on your site.
In the left hand nav, make sure you’re editing the public html and click on ‘new file’.
Name the file – you need to call it “robots.txt”, as this is what the bots are looking for.
You can now see that a file called ‘robots.txt’ has been added to your server.
Select this new file and click “edit” in the top nav bar – a pop up will appear.
Click edit
Here, you are editing the actual file the bots will see. For this example, I’m just going to tell all the search engine bots they’re ok to crawl the entire site, except a few places that I’ll disallow. In the text below, I will link to a more in-depth article, explaining what you can include in the robots.txt file.
The reason I’ve excluded the two sections of the site is because after looking in the server log data, I can see that bots are trying to crawl these pages and I don’t want them crawling here. If you want more information about server log analysis check out the links below.
Once you’ve created your file, click “save” and you’re done.
Head over to your site – type in your domain plus robots.txt to see it working
Thanks for watching. You now have a robots.txt file live on one of the most important pages on your site, that gives clear instructions to crawler bots what they can and cannot crawl on your website.
I hope you found this video useful. Please leave a comment below if you have any questions or issues and I’ll try to help.
Please subscribe to the YouTube channel for more technical SEO videos.
I am the Managing Director of Coreter Media and have been in Digital Marketing since 2009. Initially in-house working for some of the UK’s biggest brands, but now I run my own agency helping small businesses grow.