Select Page
Notifications
Clear all

Robots.txt file?

Minerva2011
(@Minerva2011)
Eminent Member

Hi All,
Tell me how you might use the Robots.txt file?

Thanks in advance ....

website design | Polo Shirt | Australia free classifieds Ads

Quote
Topic starter Posted : 07/11/2011 2:23 pm
johnBMitchell
(@johnBMitchell)
Estimable Member

Re: Robots.txt file?

If you want to prevent any folder or url for indexing then use "robots.txt".

ReplyQuote
Posted : 08/11/2011 9:19 am
laylamichelle1097
(@laylamichelle1097)
Active Member

Re: Robots.txt file?

Robot.txt is a text file through which we can instruct the search engine which have to be cache or not.

ReplyQuote
Posted : 28/03/2013 1:11 pm
peterthomas
(@peterthomas)
Eminent Member

Re: Robots.txt file?

Robot.txt is a simple notepad file through which crawler crawl your website.

Coffee Machine Supplies | Lease Commercial Coffee Machines | Buy Coffee Beans

ReplyQuote
Posted : 29/03/2013 7:02 am
hopshopdrop
(@hopshopdrop)
New Member

Re: Robots.txt file?

hide the unwanted or private data from the web crawler

ReplyQuote
Posted : 24/04/2013 11:21 am
Lalasa
(@Lalasa)
Active Member

Re: Robots.txt file?

Robots.txt is text file with this you can hide unwanted file from the search engine without crawling

ReplyQuote
Posted : 08/02/2014 7:06 am
immigrationO
(@immigrationO)
Eminent Member

Re: Robots.txt file?

"robots.txt" file is text file. which you can allow for crawling by search engine as per your command.

Australia Immigration | Immigration Overseas | Canada Immigration | New Zealand Immigration

ReplyQuote
Posted : 22/02/2014 10:06 am
Lipipaliwal
(@Lipipaliwal)
Trusted Member

Re: Robots.txt file?

Well, According to my view, Robots.txt file is generally used for hide your URL or folder which you don't want to give permission to search engine to crawl your backlinks or URL. By this file, you can prevent your data or files.

Build online store

ReplyQuote
Posted : 17/02/2015 1:35 am
SaanviRao
(@SaanviRao)
Active Member

Re: Robots.txt file?

Robot.txt is the file that gives the permission to Google regarding what page google will crawl and which one is not.

build online store

ReplyQuote
Posted : 31/03/2015 6:31 am
tanishajain
(@tanishajain)
Active Member

Robots.txt is the first file which used to crawl by the search spiders. And here we can allow the crawlers which file to crawl of which one is to not.

Noida Housing Projects | Gurgaon Housing Projects

ReplyQuote
Posted : 13/04/2015 4:15 am
rachana
(@rachana)
Active Member

Robots.txt is a text file created by webmasters that instruct web robots how to crawl pages in their website. This file also indicates whether the pages of their website should be crawled or not. The crawl instructions are

ReplyQuote
Posted : 23/04/2018 3:57 am
techguru
(@techguru)
Active Member

robots.txt is a text fil use to give instruction search bots to allow or not in your pages of files.

ReplyQuote
Posted : 07/06/2018 8:43 am
Share: