What is Robots.txt File
What is a robots.txt file?A robots.txt file tells search engine crawlers which pages or files the crawler can or can’t requestform your site. This is used mainly to avoid overloading your site with requests; it is not amechanism for keeping a web page out of Google . To keep a web page out of Google, …