The main purpose of robots.txt is to communicate search engines what to crawl or not crawl. By configuring this crucial file, site owners can control what section of their site can be accessed by crawlers. They can define certain paths that will let bots know where they should go.
If the path is undefined search engines will crawl each page and article of a site. Therefore robots.txt enables the option to include accessible content and what should be skipped.
Syntax of Robot .txt
A typical robots.txt file is a HTML text file that displays following syntax in its crude form.
User-Agent: *
Disallow: /directory/
Allow: /directory/sub-directory
Otherwise manner to write the above directives is to separate them.
User-Agent: *
Disallow: /directory/
User-Agent: *
Allow: /directory/sub-directory
User-Agent
This line specifies the search engine crawler to which the rules apply. The asterisk (*) is a wildcard that means “all bots.” So, in this example, the rules apply to all crawlers.
Disallow
This directive tells the search engine which parts of the site should not be crawled. In
this case, any content under the “/private/” directory is off-limits to search engines.
Allow
Conversely, the “Allow” directive permits crawlers to access specific content. In this example
, the “/public/page.html” page is explicitly allowed for crawling.
Robots.txt is case-sensitive, and each directive should be on a separate line. Also, you can include comments by using the “#” symbol.
How to View & Create robot.txt file
First off add /robot.txt to the end of your domain URL and run it in the browser.
testsite/com/robot.txt
The result is likely to display some information as following:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
This is the minimalist form of configured robot txt file. If you witness anything else, such as
User-agent: *
Disallow: ?
Then the file requires editing.
How to Find robot.txt file on Host
Robot.txt is easy to find on the root directory of your site’s host (wherever your site is hosted). Go to the root directory (It looks like this /home/domain) . Once you are in the root directory of your domain, click on the root name and look on the right panel you will see a robot.txt file among several other vital files. Exercise caution as you traverse.
If you cant see it there, chances are you will find it in the public_html folder in the root. No worries, you can move the file under direct root.
When you right click on the robot.txt there is a “view” option. Both the browser view and C-panel view will display the same content, but here in the panel you can edit the file.
Further Reading
Want to highlight content with Schema Markup
How to reinforce internal linking of your site
How to Edit a robot.txt file
Go through the same path as you viewed the file, right click on robot.txt file, this time go to edit option, it will redirect you to a new page where you can make required changes. Save the configuration after running a complete check.
To check the correct functionality of robots.txt, again open a new browser tab and run
testsite.com/robot.txt and you should be able to see the changes.
Editing robot.txt through Yoast plugin
Yoast is a potent SEO plugin, along with it also offers all necessary technical tools that may enhance the site performance, remove errors and safeguard your content.
If you have no idea of or reluctant about making changes in robot.txt file in c-panel or create a new one, then this standard plugin has got you covered.
Simply follow the route WordPress dashboard> Yoast SEO> Tools> File Editor (probably second from top)
In case your robot.txt is missing. It will let you create one. Click on Create a robot.txt file.
If you already have a file, then you could make changes and save it.
Should I add Sitemap to robot.txt
Yes, one should always mention a site map in robot.txt file. It will let Googlebot know the exact number of URLs on your site and whether they should consider these URLs while crawling. By constantly updating the sitemap one can make sure there web pages are up-to-date and under the knowledge of search robots.
However, sitemap is only a way to keep a systematic structure of a site, in no way it can affect the crawling process.
The format of mentioning sitemap in robot.txt file is
Sitemap: https://testsite.com/sitemap.xml
It is advisable to put sitemap either in the beginning or at the end of the file.
Steps to Submit robot.txt file
Once you have created a new robot.txt file you need to submit it to Google. Google search console is the platform where one can easily submit robot file. In the left-menu Go to Legacy tools and reports and click Learn more option. On the right panel a small window will open. Out of many try find robot.txt report option. Once click on that it will provide option to upload robot.txt file.
Best Practices for robot.txt
Clear User-Agent Directives
Clearly specify directives for user-agents in your robots.txt file. Use the “” wildcard for all bots or specify individual user-agents to tailor instructions for specific search engine crawlers.
Use a new line to declare a new user agent, this will allow for an easy and clear readability.
Also, the editor can identify each declaration, and modify or delete as and when needed. () is Google specific identifier for user agent.
User-Agent: *
Disallow: /post/
Use Disallow and Allow effectively
Utilize the “Disallow” directive to restrict access to specific areas of your site and the “Allow” directive to permit access. This helps in fine-tuning how search engines crawl and index your content.
Disallow: /x/
Allow: /z/
Case Sensitivity
Keep in mind that robots.txt is case-sensitive. Ensure consistency in your file, and be aware that “/Private/” and “/private/” may be treated differently.
Disallow: /Private/
Use Wildcards wisely
Wildcards can be powerful but use them judiciously.
For example,
Disallow: /images/.jpg
will block all JPEG images in the “/images/” directory. Disallow: /images/.jpg
Regularly Update
If your website uses parameterized URLs, address them in the robots.txt to prevent duplication issues.
For instance,
Disallow: /?
can prevent crawling of URLs with parameters. Disallow: /?
Robot.txt Advantages
The robots.txt file serves as a valuable tool for site owners to control and communicate with web crawlers, including search engine bots.
Crawling Behavior: Search engines can use the robots.txt file to know what web pages to analyze and what to ignore.
By disallowing the crawling of certain files or directories, webmasters can conserve bandwidth. This is particularly useful for preventing Googlebots from accessing resource-intensive or irrelevant parts of the site.
Robots.txt is a powerful implement to restrict indexing of sensitive sections or web pages such as- admin area, personal data.
Proper configuration of robots.txt file can significantly improve search engine optimization. It allows crawling bots to focus on the most important and relevant content, potentially improving the site’s search rankings.
By customizing robot.txt a person can prevent the possibility of plagiarism issues and duplicity of the content within the site.
The robots.txt file supports the “Crawl-delay” directive, allowing search engines to specify the time delay between successive requests from crawlers. This can help reduce the server load during peak times and prevent any negative impact on website performance.
Controlling behavior ensures that search engines index the most relevant and high-quality content. This can lead to a better user experience for visitors who are more likely to find the information they are looking for.
During updates or maintenance activities, webmaster can use the robots.txt file to temporarily disallow crawling of certain sections. This prevents Googlebot from indexing incomplete or outdated content.
