The largest SEO marketplace in the world
Find all the software to suit your computing needs
Can't find what you're looking for? Create it
Trade skills with others
Write the reason you're deleting this FAQ
I want to know that what extension helps to allow all bot to index our sites and crawl them.
Put this into your robot.txt file:
Are you sure you want to delete this post?
From what I understand with the robot crawlers of search engines, they take on regular sites with no problem. Regular site means you have not robot.tx in your site because that file is an instruction to the robot crawlers on what to do. If there is no robot.txt file then that means there is no instruction so the robot crawlers will do the default which is to crawl on your web pages.