Find what websites are hiding by searching for Robots.txt
It was surprising to know that even Robots.txt files are being indexed by google. On an interesting post on How to find secret articles and sites Roberts shares how you can find secret things which people tend to hide.
Just do a search on “robots.txt” “disallow:” filetype:txt in google and you will be surprised with what you see.
Robots TXt used to Hide things which is not to be indexed
With this not you can directly go to any of those files and directly see the pages or images which are hidden from search engines and even you.
Now my point of concern is
- Why are search engines indexing them ?
- Do they really don’t go and look into the pages or its just a rule which is not followed.
- If you say you don’t they why you are indexing this either. Does it make sense ?
- Is there a way we can stop getting it indexed?
What do you think on this ? Share with us in comments and do correct me if I am wrong.
 |  |  |  |  |  |  |
 |  |  |  |  |  |  |
 |  |  |  |  |  |  |
Posted on 23rd May 2008 by Ashish Mohta , A tech blogger who writes about solving day to day problems of people who use computer. He also writes on How to use the applications like Office, PC tips, Online tools,Browsers and more. All posts by Ashish Mohta | Connect with me @
Twitter |
Linkedin |
Facebook |
Stumble
| Need more help? Ask your Questions at our Support Center
Free Email Subscription
thanks Ashish. this library are applied without...
Leave your response!