Skip to main content

What is a robots.txt file

By December 10th, 2020No Comments

Within a website, a robots.txt file tells search engines how to crawl pages within the site.

This is part of the REP (Robots Exclusion Protocol) that comprise a series of web standards that govern how robots crawl websites.

The robots.txt file is a simple text file that is hosted on your web server.

Web crawler bots (i.e. from Google) will look for this file and follow its suggestions – which should include a prompt to visit your sitemap.

It is a good idea to have a robots.txt file within your website.

You should be able to find out if your website has one by adding:


…to the end of your domain name in a website browser.

Here is an image of my robots.txt file:

Notice that my sitemap URL is included.

Nigel Temple

Author Nigel Temple

Nigel Temple is a marketing consultant, speaker, trainer and writer. Topics include: AI / ChatGPT, Marketing strategy, Digital marketing, SEO, Copywriting, Mailchimp. Nigel is available to hire for a consultancy meeting, training workshop, speaking event or marketing project. Email: Or call Nigel on: 01628 773128

More posts by Nigel Temple