Clear
0
Sign In Join Login/Signup

Why use a robots.txt file?

  • spykaushik Level 1
    • 7

    Write the reason you're deleting this FAQ

    What’s the significance of Robots.txt file in a website?
    # SEO
    Share post:

    Comments

    Please login or sign up to leave a comment

    Join
    Layla

    Layla

    A robots txt file helps the search engine bot crawl your website to find all links. Sometimes, these links will be buried in obscure places on your site.
    13 years ago
    1969-12-31T19:00:00-0500

    Are you sure you want to delete this post?

    CountD

    CountD

    The Robots.txt file is a text document that informs the search engine spiders which parts of your website you would like to be crawled and which parts you do not want the search engine crawling .
    13 years ago
    1969-12-31T19:00:00-0500

    Are you sure you want to delete this post?

    sunil0021

    sunil0021

    It inform search engine spiders to what to crawl and what not Why use a robots.txt file?

    - Sunil Bishnoi
    13 years ago
    1969-12-31T19:00:00-0500

    Are you sure you want to delete this post?

    spyindiavimlesh

    spyindiavimlesh

    Robots.txt allow or disallow to search engines for crawling.
    13 years ago
    1969-12-31T19:00:00-0500

    Are you sure you want to delete this post?