Sometimes you might want to allow or disallow pages from being indexed in search engines. You can control how search engines index your site by using a file named “robots.txt”. The robots.txt file is a file located in the root level of your website that is scanned by search engines and controls which folders or files search engines should scan or not scan.
Here’s some links to more information about the robots.txt and how to use it as well as links to information about removing a page from a search engine index.
