Define Robotxt file and tell me the use?

Re: Define Robotxt file and tell me the use?

it's a small text file that tells search engine spiders/bots where to go on your site and where not to go. for example you wouldn't want a spider crawling on your checkout page if you have an e-commerce site

Re: Define Robotxt file and tell me the use?

Web Robots are also known as Web Wanderers, Crawlers are programs that traverse the Web automatically. Web site owners most oftenly use the /robots.txt file to give instructions about their site to web robots. This file tells search engine where to go on your site and where not.

Re: Define Robotxt file and tell me the use?

Robots.txt file used by webmasters to instruct the search engine bots for which page they have to crawl and which are not.

Re: Define Robotxt file and tell me the use?

It tells the search engine bots which pages of your website is important or need to be indexed.

Re: Define Robotxt file and tell me the use?

Robots.txt is a text based file for spider to allow which page of this website is indexing/ crawling or which one ignored.

Re: Define Robotxt file and tell me the use?

robot.txt is the file that is uploaded in the root directory. Which i used to stop the google to crawl the page that you dont want to google reach.