Almost anybody who works at home with his own website or blog is relying this way or the other on Google AdSense. One of the biggest advantages of the AdSense platform is its ability to provide the most relevant ads which increases its click-through rate. However, there are few common crawling mistakes webmasters can make that can lessen the ad relevancy.

In this post, i would like discuss about two common mistakes webmasters are making that affecting the crawling of the site’s pages by the AdSense bot and consequently reducing significantly its ability to generate the most relevant ads: Mistakenly block AdSense bot with the robots.txt file and how to grant it access to login pages.

Mistakenly Blocking AdSense Bot With Robots.txt

As Google stated, robots.txt is the “gatekeeper” of the website that granting or denying access from bots/crawlers to the site’s pages. The AdSense bot needs to have this access to the site, in order to identify and to show the most relevant ads to match the content on the page. Therefore, not giving it permission to crawl the page, will damage its ability to provide matching ads.

To check if your robots.txt file is defined correctly, open it and verify you aren’t blocking the AdSense bot- There shouldn’t be any problem if you grant access to all bots and crawlers as you see these lines in the robots.txt file:

User-agent: *
Allow: /

It can also appear like this:

User-agent: *
Disallow:

If you do choose to block certain bots or crawlers, make sure you add the following lines:

User-agent: Mediapartners-Google
Disallow:

You can check for AdSense crawling errors in your account under the “Account Settings” tab from the “Access and authorization” section and there you can find what crawling errors you have, like in the following screenshot:

Granting Access For AdSense To Login Pages

Another common AdSense crawling issue is how to grant it access to login-protected pages. For that, Google presents a solution- The publisher can create easily a login especially for the AdSense bot, so it will be able to enter and to crawl pages behind login pages.

It is pretty simple to configure a login for AdSense: In your AdSense account enter the “Account Settings” browse down to the “Access and authorization” section, there click on “edit logins” (like in the screenshot below) and type your login details. You can also test the login, to verify it is working.