{"id":33698,"date":"2020-09-08T09:56:51","date_gmt":"2020-09-08T08:56:51","guid":{"rendered":"https:\/\/www.clickintelligence.co\/?p=33698"},"modified":"2023-08-03T08:06:11","modified_gmt":"2023-08-03T07:06:11","slug":"what-is-robots-txt-and-how-to-implement-it","status":"publish","type":"post","link":"https:\/\/staging-uk.clickintelligence.com\/us\/what-is-robots-txt-and-how-to-implement-it\/","title":{"rendered":"What Is Robots Txt And How To Implement It?"},"content":{"rendered":"

SEO is more important than ever, and there are multiple ways to boost your search engine rankings<\/a> so that you gain more online visibility. However, one of the most overlooked SEO<\/a> steps is being ignored by even digital marketing experts. Robot text (or robot txt) is either a mystery to those experts, or they overlook just how important it is when it comes to getting yourself recognized by search engines. If you want to improve your performance on search engine results pages, then you need to learn just how to implement robots txt. Here is our brief guide to the robots txt format and how to use robots txt for SEO.<\/p>\n

What is a Robots txt file?<\/h2>\n

The robots.txt file is a very small part of your website (every website has the file in its code), but many people are unaware that it exists or what it does. Designed to work with search engines, robots txt format allows you to boost your SEO performance dramatically. Even without any kind of technical skill, it’s a simple matter of looking at your website’s source code and finding the robot text.<\/p>\n

Put simply; a robots txt file lets search engines know the pages that you want them to see and crawl (and what pages they can ignore). <\/em><\/p>\n

Every time a new webpage is published, bots’ crawl’ through the page to determine what it’s about and how much value it has in regards to answering a search query. The bots that crawl websites will always check the robot txt file first. Robot text is your way of letting your web pages communicate directly with search engines, and that can only be good news for your SEO.<\/p>\n

Allow and Disallow<\/h3>\n

Google uses Googlebot to crawl through websites, while other search engines use a variety of different bots (sometimes called spiders). As different types of bots crawl your site (your website code calls them User Agents), it’s important to know which search engines can access or not access your pages. A standard robots txt example would look like this:<\/p>\n

User-agent: *<\/p>\n

Disallow: \/<\/p>\n

You could also have:<\/p>\n

User-agent: Googlebot<\/p>\n

Disallow: \/nogooglebot\/<\/p>\n

The asterisk in the first example means that you are going to address access to all bots from all search engines. It might seem counterintuitive to disallow a webpage, but there are many reasons to do so that can affect your SEO. If you have print-only pages that are a duplicate of another page, then you don’t want search engines to consider that page as a duplicate content. Doing so will reduce your value and negatively affect your SERP ranking.<\/p>\n

Bots like Googlebot will have what is called a ‘crawl budget’, and the faster that these bots can crawl through your webpages the better for them and for you. <\/em><\/p>\n

By making sure that bots can only crawl the most important pages on your website, you can dramatically speed up processing time and ensure that your other SEO changes have the maximum effect.<\/p>\n

\"\"How do I create a robots txt file?<\/h3>\n

To start this process, remember that you should only use a plain text editor. Check your website’s robots txt file, and if it’s there then delete the text (but not the file itself). Next, determine if you are going to use an asterisk (which will affect every search engine bot) or you’ll name specific crawler bots like Googlebot. It’s best to use an asterisk if you’re not sure. Then, you need to choose if you want your site to be crawlable. If you do, then make sure that the code reads ‘allow’. If you don’t want your site to be crawlable, then change the code text to ‘disallow’. It is that simple. However, this will affect every page of your website at once.<\/p>\n

How to use robots txt for SEO<\/h3>\n

Your next step is to detect the pages that you don’t want to be crawled by search engine bots. There are many reasons why you might want this. Admin pages do not need to be crawled, and neither do log-in pages. You want to disallow:<\/p>\n

    \n
  • Pages that contain sensitive or private information<\/li>\n
  • Content pages that are low quality<\/li>\n
  • Any duplicate content<\/li>\n<\/ul>\n

    That will mean going to your robots txt file and making it look like this:<\/p>\n

    User-agent: *<\/p>\n

    Disallow: \/admin\/<\/p>\n

    Disallow: \/log-in\/<\/p>\n

    Do this for every page on your site, making sure that your highest quality pages have the word Allow in the code.<\/p>\n

    Ensuring that search engine bots can crawl through the right pages and ignore what they don’t need to waste time on means that your site will get a much more dramatic SEO boost than you might have thought. Don’t overlook the importance of robot txt, or your performance on SERPs will be extremely limited.<\/p>\n","protected":false},"excerpt":{"rendered":"

    SEO is more important than ever, and there are multiple ways to boost your search engine rankings so that you gain more online visibility. However, one of the most overlooked SEO steps is being ignored by even digital marketing experts. Robot text (or robot txt) is either a mystery to those experts, or they overlook…<\/p>\n","protected":false},"author":4,"featured_media":44647,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[640],"tags":[],"acf":[],"cp_meta_data":{"_wpml_word_count":["{\"total\":859,\"to_translate\":{\"en\":859}}"],"_wpml_media_featured":["1"],"_wpml_media_duplicate":["1"],"_edit_last":["4"],"_thumbnail_id":["44647"],"slide_template":[""],"_yoast_wpseo_metadesc":["You need to learn just how to implement robots txt. Here is our brief guide to the robots txt format and how to use robots txt for SEO."],"newsletter_box":["1"],"_newsletter_box":["field_5c8b860bd5840"],"sidebar_banner":[""],"_sidebar_banner":["field_5c8b8574d583e"],"under_blog":[""],"_under_blog":["field_5c8b85dad583f"],"select_download":[""],"_select_download":["field_5c924b52af68f"],"usa":[""],"_usa":["field_5da716b6eaa2d"],"_yoast_wpseo_primary_category":["578"],"_encloseme":["1"],"_wpml_location_migration_done":["1"],"_edit_lock":["1677687187:15"],"_pingme":["1"],"custom_permalink":[""],"_wpb_vc_js_status":["false"],"_easy_image_gallery_link_images":["off"],"_yoast_wpseo_focuskeywords":["[]"],"_yoast_wpseo_keywordsynonyms":["[\"\"]"],"_yoast_wpseo_estimated-reading-time-minutes":["4"],"_yoast_indexnow_last_ping":["1691046373"]},"yoast_head":"\nWhat Is Robots Txt And How To Implement It? - Click Intelligence Staging<\/title>\n<meta name=\"description\" content=\"You need to learn just how to implement robots txt. Here is our brief guide to the robots txt format and how to use robots txt for SEO.\" \/>\n<meta name=\"robots\" content=\"noindex, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"What Is Robots Txt And How To Implement It?\" \/>\n<meta property=\"og:description\" content=\"You need to learn just how to implement robots txt. Here is our brief guide to the robots txt format and how to use robots txt for SEO.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/staging-uk.clickintelligence.com\/us\/what-is-robots-txt-and-how-to-implement-it\/\" \/>\n<meta property=\"og:site_name\" content=\"Click Intelligence Staging\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/Click-Intelligence-Ltd-Link-Building-Services-1167966826553082\/\" \/>\n<meta property=\"article:published_time\" content=\"2020-09-08T08:56:51+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-08-03T07:06:11+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/staging-uk.clickintelligence.com\/wp-content\/uploads\/2020\/09\/Photo-by-Aryan-Dhiman-on-Unsplash.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1400\" \/>\n\t<meta property=\"og:image:height\" content=\"600\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"James Owen\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@click_intel\" \/>\n<meta name=\"twitter:site\" content=\"@click_intel\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/staging-uk.clickintelligence.com\/us\/what-is-robots-txt-and-how-to-implement-it\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/staging-uk.clickintelligence.com\/us\/what-is-robots-txt-and-how-to-implement-it\/\"},\"author\":{\"name\":\"James Owen\",\"@id\":\"https:\/\/staging-uk.clickintelligence.com\/us\/#\/schema\/person\/5bac207e251b37516c72f18cf8ca2464\"},\"headline\":\"What Is Robots Txt And How To Implement It?\",\"datePublished\":\"2020-09-08T08:56:51+00:00\",\"dateModified\":\"2023-08-03T07:06:11+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/staging-uk.clickintelligence.com\/us\/what-is-robots-txt-and-how-to-implement-it\/\"},\"wordCount\":814,\"publisher\":{\"@id\":\"https:\/\/staging-uk.clickintelligence.com\/us\/#organization\"},\"articleSection\":[\"SEO\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/staging-uk.clickintelligence.com\/us\/what-is-robots-txt-and-how-to-implement-it\/\",\"url\":\"https:\/\/staging-uk.clickintelligence.com\/us\/what-is-robots-txt-and-how-to-implement-it\/\",\"name\":\"What Is Robots Txt And How To Implement It? - Click Intelligence Staging\",\"isPartOf\":{\"@id\":\"https:\/\/staging-uk.clickintelligence.com\/us\/#website\"},\"datePublished\":\"2020-09-08T08:56:51+00:00\",\"dateModified\":\"2023-08-03T07:06:11+00:00\",\"description\":\"You need to learn just how to implement robots txt. Here is our brief guide to the robots txt format and how to use robots txt for SEO.\",\"breadcrumb\":{\"@id\":\"https:\/\/staging-uk.clickintelligence.com\/us\/what-is-robots-txt-and-how-to-implement-it\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/staging-uk.clickintelligence.com\/us\/what-is-robots-txt-and-how-to-implement-it\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/staging-uk.clickintelligence.com\/us\/what-is-robots-txt-and-how-to-implement-it\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/staging-uk.clickintelligence.com\/us\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"What Is Robots Txt And How To Implement It?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/staging-uk.clickintelligence.com\/us\/#website\",\"url\":\"https:\/\/staging-uk.clickintelligence.com\/us\/\",\"name\":\"Click Intelligence Staging\",\"description\":\"Search Marketing Agency\",\"publisher\":{\"@id\":\"https:\/\/staging-uk.clickintelligence.com\/us\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/staging-uk.clickintelligence.com\/us\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/staging-uk.clickintelligence.com\/us\/#organization\",\"name\":\"Click Intelligence\",\"url\":\"https:\/\/staging-uk.clickintelligence.com\/us\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/staging-uk.clickintelligence.com\/us\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.staging-uk.clickintelligence.com\/wp-content\/uploads\/2020\/08\/ci_logo_pos_inline.svg\",\"contentUrl\":\"https:\/\/www.staging-uk.clickintelligence.com\/wp-content\/uploads\/2020\/08\/ci_logo_pos_inline.svg\",\"width\":406,\"height\":51,\"caption\":\"Click Intelligence\"},\"image\":{\"@id\":\"https:\/\/staging-uk.clickintelligence.com\/us\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/Click-Intelligence-Ltd-Link-Building-Services-1167966826553082\/\",\"https:\/\/twitter.com\/click_intel\",\"https:\/\/www.instagram.com\/clickintelligence\/\",\"https:\/\/www.linkedin.com\/company\/click-intelligence\",\"http:\/\/www.youtube.com\/c\/ClickintelligenceCoUk\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/staging-uk.clickintelligence.com\/us\/#\/schema\/person\/5bac207e251b37516c72f18cf8ca2464\",\"name\":\"James Owen\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/staging-uk.clickintelligence.com\/us\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/1549c46386ea257a645cd3160b648329?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/1549c46386ea257a645cd3160b648329?s=96&d=mm&r=g\",\"caption\":\"James Owen\"},\"description\":\"James has been involved in SEO and digital marketing projects since 2007. James has led many SEO projects for well-known brands in Travel, Gaming and Retail such as Jackpotjoy, Marriott, Intercontinental Hotels, Hotels.com, Expedia, Betway, Gumtree, 888, Ax Paris, Ebyuer, Ebay, Hotels combined, Smyths toys, love honey and Pearson to name a few. James has also been a speaker at SEO and digital marketing conferences and events such as Brighton SEO.\",\"url\":\"https:\/\/staging-uk.clickintelligence.com\/james-owen\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"What Is Robots Txt And How To Implement It? - Click Intelligence Staging","description":"You need to learn just how to implement robots txt. Here is our brief guide to the robots txt format and how to use robots txt for SEO.","robots":{"index":"noindex","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"og_locale":"en_US","og_type":"article","og_title":"What Is Robots Txt And How To Implement It?","og_description":"You need to learn just how to implement robots txt. Here is our brief guide to the robots txt format and how to use robots txt for SEO.","og_url":"https:\/\/staging-uk.clickintelligence.com\/us\/what-is-robots-txt-and-how-to-implement-it\/","og_site_name":"Click Intelligence Staging","article_publisher":"https:\/\/www.facebook.com\/Click-Intelligence-Ltd-Link-Building-Services-1167966826553082\/","article_published_time":"2020-09-08T08:56:51+00:00","article_modified_time":"2023-08-03T07:06:11+00:00","og_image":[{"width":1400,"height":600,"url":"https:\/\/staging-uk.clickintelligence.com\/wp-content\/uploads\/2020\/09\/Photo-by-Aryan-Dhiman-on-Unsplash.png","type":"image\/png"}],"author":"James Owen","twitter_card":"summary_large_image","twitter_creator":"@click_intel","twitter_site":"@click_intel","schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/staging-uk.clickintelligence.com\/us\/what-is-robots-txt-and-how-to-implement-it\/#article","isPartOf":{"@id":"https:\/\/staging-uk.clickintelligence.com\/us\/what-is-robots-txt-and-how-to-implement-it\/"},"author":{"name":"James Owen","@id":"https:\/\/staging-uk.clickintelligence.com\/us\/#\/schema\/person\/5bac207e251b37516c72f18cf8ca2464"},"headline":"What Is Robots Txt And How To Implement It?","datePublished":"2020-09-08T08:56:51+00:00","dateModified":"2023-08-03T07:06:11+00:00","mainEntityOfPage":{"@id":"https:\/\/staging-uk.clickintelligence.com\/us\/what-is-robots-txt-and-how-to-implement-it\/"},"wordCount":814,"publisher":{"@id":"https:\/\/staging-uk.clickintelligence.com\/us\/#organization"},"articleSection":["SEO"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/staging-uk.clickintelligence.com\/us\/what-is-robots-txt-and-how-to-implement-it\/","url":"https:\/\/staging-uk.clickintelligence.com\/us\/what-is-robots-txt-and-how-to-implement-it\/","name":"What Is Robots Txt And How To Implement It? - Click Intelligence Staging","isPartOf":{"@id":"https:\/\/staging-uk.clickintelligence.com\/us\/#website"},"datePublished":"2020-09-08T08:56:51+00:00","dateModified":"2023-08-03T07:06:11+00:00","description":"You need to learn just how to implement robots txt. Here is our brief guide to the robots txt format and how to use robots txt for SEO.","breadcrumb":{"@id":"https:\/\/staging-uk.clickintelligence.com\/us\/what-is-robots-txt-and-how-to-implement-it\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/staging-uk.clickintelligence.com\/us\/what-is-robots-txt-and-how-to-implement-it\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/staging-uk.clickintelligence.com\/us\/what-is-robots-txt-and-how-to-implement-it\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/staging-uk.clickintelligence.com\/us\/"},{"@type":"ListItem","position":2,"name":"What Is Robots Txt And How To Implement It?"}]},{"@type":"WebSite","@id":"https:\/\/staging-uk.clickintelligence.com\/us\/#website","url":"https:\/\/staging-uk.clickintelligence.com\/us\/","name":"Click Intelligence Staging","description":"Search Marketing Agency","publisher":{"@id":"https:\/\/staging-uk.clickintelligence.com\/us\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/staging-uk.clickintelligence.com\/us\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/staging-uk.clickintelligence.com\/us\/#organization","name":"Click Intelligence","url":"https:\/\/staging-uk.clickintelligence.com\/us\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/staging-uk.clickintelligence.com\/us\/#\/schema\/logo\/image\/","url":"https:\/\/www.staging-uk.clickintelligence.com\/wp-content\/uploads\/2020\/08\/ci_logo_pos_inline.svg","contentUrl":"https:\/\/www.staging-uk.clickintelligence.com\/wp-content\/uploads\/2020\/08\/ci_logo_pos_inline.svg","width":406,"height":51,"caption":"Click Intelligence"},"image":{"@id":"https:\/\/staging-uk.clickintelligence.com\/us\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/Click-Intelligence-Ltd-Link-Building-Services-1167966826553082\/","https:\/\/twitter.com\/click_intel","https:\/\/www.instagram.com\/clickintelligence\/","https:\/\/www.linkedin.com\/company\/click-intelligence","http:\/\/www.youtube.com\/c\/ClickintelligenceCoUk"]},{"@type":"Person","@id":"https:\/\/staging-uk.clickintelligence.com\/us\/#\/schema\/person\/5bac207e251b37516c72f18cf8ca2464","name":"James Owen","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/staging-uk.clickintelligence.com\/us\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/1549c46386ea257a645cd3160b648329?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/1549c46386ea257a645cd3160b648329?s=96&d=mm&r=g","caption":"James Owen"},"description":"James has been involved in SEO and digital marketing projects since 2007. James has led many SEO projects for well-known brands in Travel, Gaming and Retail such as Jackpotjoy, Marriott, Intercontinental Hotels, Hotels.com, Expedia, Betway, Gumtree, 888, Ax Paris, Ebyuer, Ebay, Hotels combined, Smyths toys, love honey and Pearson to name a few. James has also been a speaker at SEO and digital marketing conferences and events such as Brighton SEO.","url":"https:\/\/staging-uk.clickintelligence.com\/james-owen\/"}]}},"_links":{"self":[{"href":"https:\/\/staging-uk.clickintelligence.com\/us\/wp-json\/wp\/v2\/posts\/33698"}],"collection":[{"href":"https:\/\/staging-uk.clickintelligence.com\/us\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/staging-uk.clickintelligence.com\/us\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/staging-uk.clickintelligence.com\/us\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/staging-uk.clickintelligence.com\/us\/wp-json\/wp\/v2\/comments?post=33698"}],"version-history":[{"count":2,"href":"https:\/\/staging-uk.clickintelligence.com\/us\/wp-json\/wp\/v2\/posts\/33698\/revisions"}],"predecessor-version":[{"id":45837,"href":"https:\/\/staging-uk.clickintelligence.com\/us\/wp-json\/wp\/v2\/posts\/33698\/revisions\/45837"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/staging-uk.clickintelligence.com\/us\/wp-json\/wp\/v2\/media\/44647"}],"wp:attachment":[{"href":"https:\/\/staging-uk.clickintelligence.com\/us\/wp-json\/wp\/v2\/media?parent=33698"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/staging-uk.clickintelligence.com\/us\/wp-json\/wp\/v2\/categories?post=33698"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/staging-uk.clickintelligence.com\/us\/wp-json\/wp\/v2\/tags?post=33698"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}