SEOClerks

What type of Robots.txt file allow all bot to visit my site, my blog and make index it?



Write the reason you're deleting this FAQ

What type of Robots.txt file allow all bot to visit my site, my blog and make index it?

Hello everyone...
I want to know that what extension helps to allow all bot to index our sites and crawl them.
thanks

Comments

Please login or sign up to leave a comment

Join
rannt

Put this into your robot.txt file:

User-agent: *
Disallow:

Thank you
Gašper




Are you sure you want to delete this post?

Corzhens

From what I understand with the robot crawlers of search engines, they take on regular sites with no problem. Regular site means you have not robot.tx in your site because that file is an instruction to the robot crawlers on what to do. If there is no robot.txt file then that means there is no instruction so the robot crawlers will do the default which is to crawl on your web pages.




Are you sure you want to delete this post?