Skip to main content
SEOWebsites

What is a robots.txt file

By December 10th, 2020No Comments
robots.txt file

Within a website, a robots.txt file tells search engines how to crawl pages within the site.

This is part of the REP (Robots Exclusion Protocol) that comprise a series of web standards that govern how robots crawl websites.

The robots.txt file is a simple text file that is hosted on your web server.

Web crawler bots (i.e. from Google) will look for this file and follow its suggestions – which should include a prompt to visit your sitemap.

It is a good idea to have a robots.txt file within your website.

You should be able to find out if your website has one by adding:

/robots.txt

…to the end of your domain name in a website browser.

Here is an image of my robots.txt file:

Notice that my sitemap URL is included.

Nigel Temple

Nigel Temple is a marketing consultant, speaker, trainer and writer. 3 million words of copy | 30 years experience | 3,000+ clients Topics include: Marketing plans, Digital marketing, SEO, Copywriting, LinkedIn, Mailchimp. Nigel is available to hire for a consultancy meeting, training workshop, speaking event or marketing project. Email: nigel@nigeltemple.com Or call Nigel on: 01628 773128