Robots.txt – is an important text file that determines how a search engine accesses your website. Within the file, there are directives, which specify which website pages are allowed or disallowed for a search engine to access.
Setting the file up correctly is crucial in order to get good SEO results in the longer. Below, we talk about robots.txt more in-depth.
Robots.txt in terms of SEO
You may think of search engine robots as automated systems that crawl and document (aka index) the websites that they visit.
If you’re somewhat familiar with some aspects of SEO, you may have heard robots being called crawlers, spiders or simply, bots. While they all mean the same thing, each search engine bot is a bit different from each other.
In order to “communicate” with them and to point them to the content you want to get out there, you need to create a tidy robots.txt file, which will let the bots know which pages you want or don’t want to be crawled.
Using the robots.txt file
In order to know how to use the robots.txt file, we first need to familiarize ourselves with the syntax in which we will create it.
1.User agent definition
As mentioned, each search engine bot has a unique name to it, or more specifically – a user agent. You will need to state the name of the robot (be it Yahoo, or Google) that you will be addressing.
2. Disallowing access
The “disallow” command allows you define what part of the website you want block from engine bot crawling. Typically, the URLs are stated here.
3. Allowing access
The “allow” command is typically used to unblock a certain part of a website within a blocked parent directory. Here, a subdirectory path is entered in the document.
In other words, with robots.txt, you tell the bots which parts you want to index, and which you do not.
The importance of robots.txt
You may look at it, and think to yourself: “Why would I want to block access to pages on my website? After all, isn’t it important to have a well-indexed website?”.
And you would be right, however, there are some instances where disabling access is crucial in order to get everything right.
1. Sensitive information
A good example of sensitive website information would be directories. You probably would want to block the following on your site:
- /wp-admin/
- /scripts/
- /cgi-bin/
And so on.
2. Pages that have low-quality content
Google has stated on a number of occasions that the quality of a website is very important in order to get higher rankings. Pages that don’t bring anything to the table will only hinder that performance.
3. Duplicates
Preventing indexed duplicates is another crucial thing to keep in mind. Let’s say you allow people to view a “print” version of your webpage. You probably don’t want Google to scan the same thing twice, if you expect to rank high.
These are just a few examples of the pages you want to disallow when creating your robots.txt file.
Allow and Disallow formats for robots.txt
The principle on how to set up the file correctly is fairly simple, all you need to do is specify which pages you want to allow and disallow for indexing.
You’ll only need to use the disallow function once, to list all of the webpages you don’t want Google or others to index. Meanwhile, the allow command is only needed when you want a specific page to be indexed, but its parent page is disallowed.
The robots.txt file is an essential tool for website owners and webmasters as it plays a crucial role in controlling how search engine bots crawl and index their website’s content.
By including instructions in the robots.txt file, website owners can specify which pages and directories they want search engines to crawl and which ones they want to exclude. This helps to prevent search engine bots from indexing pages that are not meant to be visible to the public, such as private pages or duplicate content.
In addition, the robots.txt file can be used to control the rate at which search engine bots crawl a website, which can help to prevent server overload and ensure that website performance is not impacted by excessive crawling.
The robots.txt file is an important tool for website owners to manage their website’s search engine optimization (SEO) and ensure that their website is properly indexed by search engines. Failing to properly configure the robots.txt file can result in unintended consequences, such as pages being excluded from search engine results or website performance being impacted by excessive crawling.
Setting up robots.txt on your site
First, you will need to create the text file itself. Here, you will enter all of the directives for the search bots to follow. Afterwards, you will need to upload the file to your top-level domain directory using Cpanel.
The file should always be accessible right after your top-level domain name in the URL. For example “yourwebsite.eu/robots.txt”.
Otherwise, the bots won’t even bother looking for your specified directives.
It’s also important to know what to do if you have subdomains. For each one, you need to create a separate robots.txt file for the bots to follow.
Testing robots.txt
To test a robots.txt file, you can follow these steps:
- Open a web browser and go to Google’s robots.txt testing tool: https://www.google.com/webmasters/tools/robots-testing-tool
- Enter the URL of the website whose robots.txt file you want to test.
- Click on the “Test” button to see the results.
- Review the results to ensure that the robots.txt file is correctly formatted and allows or disallows the appropriate pages and directories.
- Make any necessary changes to the robots.txt file and retest until you are satisfied with the results.
How to unblock robots.txt in WordPress
To unblock robots.txt in WordPress, you can follow these steps:
- Log in to your WordPress dashboard and go to the “Settings” menu.
- Click on “Reading” and scroll down to the “Search Engine Visibility” section.
- Ensure that the checkbox next to “Discourage search engines from indexing this site” is unchecked. If it’s checked, uncheck it and save changes.
- Check that there is no plugin installed on your website that blocks robots.txt. If there is, you can either deactivate it or modify its settings to allow access to robots.txt.
- Ensure that the robots.txt file exists and is located in the root directory of your WordPress installation. If it doesn’t exist, you can create a new file and upload it to the root directory.
- Check that the robots.txt file is correctly formatted and allows or disallows the appropriate pages and directories. You can use the steps outlined in the previous answer to test your robots.txt file.
- Test your website to confirm that the robots.txt file is no longer blocked by accessing it through a web browser or using a robots.txt tester tool.
Once you have completed these steps, search engines will be able to access and crawl your website according to the rules specified in your robots.txt file.
If this does not help, you can request urgent WordPress help from EASYSEO developers team.
All in all
We’ve talked about robots.txt and it’s importance in website setup and SEO. Not only is it a great way to specify what to index, you may continue to use it later on as well, as you make changes and your website grows.
Loading...