{"id":33710,"date":"2020-09-08T10:51:30","date_gmt":"2020-09-08T09:51:30","guid":{"rendered":"https:\/\/www.clickintelligence.co\/?p=33710"},"modified":"2023-08-03T08:06:10","modified_gmt":"2023-08-03T07:06:10","slug":"how-to-help-google-to-crawl-your-site","status":"publish","type":"post","link":"https:\/\/staging-uk.clickintelligence.com\/us\/how-to-help-google-to-crawl-your-site\/","title":{"rendered":"How To Help Google To Crawl Your Site"},"content":{"rendered":"

If you want more consumers to find and visit your website, then you need Google to know who you are. While there are other search engines, Google remains the most popular search engine in the world, with up to one billion active users every single month. That means if you want to be found, then you need to get the maximum use from Google. If you have a current website that is not performing well on SERPs (or not appearing at all), or you are launching a fresh new site, then you need to make sure that your pages are on the Google index.<\/p>\n

Crawling and Indexing<\/h2>\n

The good news is that there are no mysteries to solve when it comes to making sure that Google knows who you are. They want to know that your website exists so that they can continue to provide the best value for their users. If you’re unsure how to get google to crawl your site, it is surprisingly straightforward. Google uses bots (often called spiders) that crawl the internet looking for new pages so that it can index them. Googlebot (the name of the bot that Google uses) follows hyperlinks, finds new pages and sites, and then stores them in the Google index. Once your site has been indexed, then you are in the race for a presence on search engine results pages.<\/p>\n

Are you Indexed?<\/h2>\n

It’s very easy to check whether you are on the Google index. Simply head to the search engine and type in:<\/p>\n

site: yourwebsitename.com<\/p>\n

\"\"<\/p>\n

If you are in the Google index, then you should then see a list of all of the pages that Google has indexed within that URL. If none of your webpages show up, then you haven’t been indexed by Google. It’s a great idea at this point to use Google Search Console and test your website URL in the Coverage Report section. This will break down the facts into an easy to read a report that will show you the valid number of pages you have. If that number is anything other than zero, then you have issues to resolve. Remember that you can request Google crawl your site at any time using Google Search Console.<\/p>\n

 <\/p>\n

Getting Indexed<\/h2>\n

If your search has shown that there are issues with your online visibility, then you need to take the following steps to rectify that issue.<\/p>\n

Remove Crawl Blocks<\/h3>\n

This is a common issue, but it’s an easy fix. There may be a possibility that you are unintentionally blocking Google from crawling your website and recording your site on the Google index!<\/p>\n

    \n
  1. Type in: yourwebsitename.com\/robots.txt<\/li>\n
  2. Look for one of the following results:<\/li>\n<\/ol>\n

    User-agent: Googlebot<\/p>\n

    Disallow: \/<\/p>\n

    Or<\/em><\/p>\n

     <\/p>\n

    User-agent: *<\/p>\n

    Disallow: \/<\/p>\n

    These lines of code mean that Google can’t see what’s on the pages, and that means they can’t rank you on SERPs. Either remove the code entirely or change the coding so that it says ‘Allow’ rather than ‘Disallow’.<\/p>\n

    \"\"<\/h3>\n

    Get Rid of Rogue Noindex Tags<\/h3>\n

    There are valid reasons why you might not want Google to crawl specific pages on your website. You can do this in a few different ways. The robots.txt is one way, but you can also put a no-index command into your metatags. Check using the Ahrefs Site Audit tool and head to the Indexability report. You will see warnings if any pages have a no-index metatag, which you can then remove if you want those pages to be incorporated within the Google index.<\/p>\n

    Using Your Sitemap<\/h3>\n

    Although Google is perfectly able to find your pages without a sitemap, it will find it much easier if those pages are included. Your sitemap is, essentially, your way of showing Google exactly what pages you think are important (and which pages can be ignored). Again, using Google Search Console is the key to working out if this is the issue. There, you will find the URL Inspection Tool, and you simply type in your URL. If you see the response’ Sitemap: N\/A’ or ‘URL is not on Google’ then you need to add those pages to your sitemap and submit site to Google for indexing using:<\/p>\n

    http:\/\/www.google.com\/ping?sitemap=http:\/\/yourwebsite.com\/sitemap_url.xml<\/p>\n

    This will make Google crawl your site much faster, indexing as it goes.<\/p>\n

    \"xmlOrphaned Pages<\/h3>\n

    If there are no links to a new page, then Google simply won’t be able to find it. Make sure that you have internal links to any new pages and use Ahrefs Site Audit for a comprehensive list of any orphan pages. Once identified, you can either remove the page entirely (if it is not important) or include it within your internal link structure.<\/p>\n

    Additional Link Issues<\/h3>\n

    It’s not just orphaned pages that are a problem. These link-related issues will also need to be addressed:<\/p>\n