SEOClerks

How would you prevent this?



Enter a reason for deleting this comment

How would you prevent this?

Is there a way to prevent a spider from attaching itself to a URL that you do not want on the search engines?

Comments

Please login or sign up to leave a comment

Join
lucasbytegenius

Yes there is. Create a new file called robots.txt, and put this in it:
User-agent: *
Disallow: /

Where * is the useragent of the search engine or spider and / is the url to the page.
You can read more about robots.txt here:
http://www.robotstxt.org/robotstxt.html



Are you sure you want to delete this post?

Corzhens

Thank you for the information. I am amazed by the venom of spiders and I know that many website owners don’t even know that a spider is working on their link. Maybe I have to read the article in the link that you had indicated. Truly this can be of great help to me because I am in the stage of gathering data about this topic.



Are you sure you want to delete this post?

robertman11

Use your robots.txt file or use .htaccess to actually blog the spider (you can block by IP or by user-client).



Are you sure you want to delete this post?

Light

Thanks for the information guys. How would you prevent this?



Are you sure you want to delete this post?