Robots.txt can be a file that exists on the foundation directory of each Web site and may be used to instruct search engines on which directories/documents of the web site they're able to crawl and involve within their index. Engines like google have an individual purpose only. They goal to https://news.rafeeg.ae