what is robotxt.file ?

Re: what is robotxt.file ?

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.

Re: what is robotxt.file ?

I appreciate this that the robbers can enter but not the good people by using robots.txt file. I find using robots.txt file to do no good.

Re: what is robotxt.file ?

its a robots.txt. You can send spider commands here which page/folder of your website should it crawl or which should not. If you want to disallow something than you can add it else its ok not to have it

Re: what is robotxt.file ?

Robots.txt is simple text file used by webmaster to indicate search engine spiders/bots which web page, directory, image or folder they have to follow or which are not. This text file needs to upload on your site control panel in your root folder.

Re: what is robotxt.file ?

Robots.txt is a text file you put on your site to tell search robots which page you don't desire them to visit. By essential a few rules in this text file, you can initiate robots to not crawl certain files, directories within your site, or at all.

Re: what is robotxt.file ?

In easy terms robot.txt is a written text which (not html) file you put on your website to tell look for robots which webpages you would like them not to check out.

Re: what is robotxt.file ?

Robots.txt is a text file which is placed in your site to tell search robots which pages of your site should not be visited by them. These pages are not crawled by crawler.

Re: what is robotxt.file ?

Robots.txt is text based file allow to search engine bots which page to crawl or not.

Re: what is robotxt.file ?

Hello pintro,
Really nice post i am not aware of Robottxt.file but when i read post then understand about it.

Re: what is robotxt.file ?

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. The Robots.txt file of a website will work when it is used as a request to specific robots to ignore directories or files specified within the Robots.txt file.

Re: what is robotxt.file ?

Hi,
Robot.txt is a text file which contain the list of urls which has not to be crawled by the search engine crawlers. The url put in this file are not checked by the search engine spiders

Re: what is robotxt.file ?

If you want to read google your whole website then you can use dofollow meta in your website, if you want to not read any thing or any page google in your website and your content then you can use nofollow where you want not crawl your website.

Re: what is robotxt.file ?

Robot.txt file is a text file which is uploaded on the website server. This text file include the list of those url in the website which we do not want the search spiders to visit. The links listed in this file are not checked or crawled by the spiders.

Re: what is robotxt.file ?

Robots.txt file is a text file that resides on your server, and controls a whole lot of features on your website. Any files or folders listed in this document will not be crawled and indexed by the search engine spiders.