It was surpirsing to know that even Robots.txt files are being indexed by google. On an interesting post on How to find secret articles and sites Roberts shares how you can find secret things which people tend to hide.

Just do a search on “robots.txt” “disallow:” filetype:txt in google and you will be surprised with what you see.

With this not you can directly go to any of those files and directly see the pages or images which are hidden from search engines and even you.

Now my point of concern is

• Why are search engines indexing them ?
• Do they really don’t go and look into the pages or its just a rule which is not followed.
• If you say you don’t they why you are indexing this either. Does it make sense ?
• Is there a way we can stop getting it indexed?

What do you think on this ?  Share with us in comments and do correct me if I am wrong.

About The Author of this article: Ashish is one of the co-author of this blog and writes on various interesting softwares, PC tips and more. You can read more of his articles here.

Enjoyed this article? Download our Toolbar ( for Free ) and read us more quickly or Free Subscribe to the Full RSS Feed or Get Post like this in your Inbox Click to get via Email ( You will have to confirm by checking you Inbox)

Readers Play Ground

• Add your Comments or Discuss further in our forums

• Enjoyed this article? Subscribe to the Full RSS Feed

• (No Ratings Yet)
  
   Loading ...
• Email This Post to your friends or somebody who can use it
• Tags: robots file, robots indexing, site indexing
  
• Get Post like this in your Inbox Click to get via Email ( You will have to confirm by checking you Inbox)
• Read More Stories from Debates