Robots.txt is a text file placed at the root of a website that contains instructions for bots/spiders/robots crawling your site instructing them what they should or should not include in the Search Engine Index.
You can tell Search Engine Bots not to index specific folders, files or files with specific extensions.
.
« Back to Glossary Index
