Home » Tips N Tricks

Find what websites are hiding by searching for Robots.txt

It was surprising to know that even Robots.txt files are being indexed by google. On an interesting post on How to find secret articles and sites Roberts shares how you can find secret things which people tend to hide.

Just do a search on “robots.txt” “disallow:” filetype:txt in google and you will be surprised with what you see.

Robots TXt used to Hide things which is not to be indexed

Robots TXt used to Hide things which is not to be indexed

With this not you can directly go to any of those files and directly see the pages or images which are hidden from search engines and even you.

Now my point of concern is

  • Why are search engines indexing them ?
  • Do they really don’t go and look into the pages or its just a rule which is not followed.
  • If you say you don’t they why you are indexing this either. Does it make sense ?
  • Is there a way we can stop getting it indexed?

What do you think on this ?  Share with us in comments and do correct me if I am wrong.

Tagged with: | Need more help ? Ask your Questions at our Support Center | Follow us on Twitter @TSNW or Facebook
Facebook Share Print This Post

Posted on 23rd May 2008 by Ashish Mohta , A Professional Tech blogger, Editor and Writer who talks about solving day to day problems of people who use computer. He also writes on How to use the applications like Office, PC tips, Online tools,Browsers and more. All posts by Ashish Mohta | Connect with me @ Twitter | Linkedin | Facebook | Stumble

Leave your response!

Be nice. Keep it clean. Stay on topic. No spam.