Tuesday, November 05, 2013

AdSense Friendly Custom Robots.txt for Blogger


Custom Robots.txt for Blogger


Bloggers who have SEO Knowledge, should know about robots.txt and its use. But it is quite unfamiliar to the new bloggers. But some of them might be interested in Blogger's search preferences and they must ask about what is Custom robots.txt? Well this is not so easy to explain. To the new bloggers, the explanation might be confusing. In order to explain the term robots.txt, I've to tell about search bots or search crawlers. But I will try to explain the term clearly. 


Suppose, you've opened a blog in blogger. continuously you're publishing new posts. If everything is okay, your post will appear in Google Search within few hours. If your post is matched with someone's search topic, your site will appear in the search results. How it's possible? 


Blogger's Default robots.txt:

Search engines scan your site for new posts continuously. This is known as crawling. Search bots crawl your site regularly and index your posts in their directory. By default, blogger comes with a robots.txt for your site that helps search engines crawl your site.

But as we know, the default or general settings never guarantee the accurate result. So we need to customize the settings as our requirement. Same rule is applicable for blogger's robots.txt.


Custom Robots.txt for Blogger Blog:

Today you will learn how to customize the robots.txt of blogger. There is a rumor that the default robots.txt can't index the old posts properly. If you have a proper custom robots.txt, your posts will be indexed properly. In my last post, I discussed about creating XML sitemap for blogger blog. There I gave example of XML sitemap. You can use that sitemap as your custom robots.txt content.



Custom Robots.txt Settings for Blogger


Look at the above image. This is how you can customize your robots.txt element. Or use the instruction below:

Sign in to Blogger Dashboard > Settings > Search Preferences > Crawler and Indexing > Custom robots.txt > Enable? > Yes > Now add the code below > Save Changes.


# Blogger Sitemap generated on 2013.11.06
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://www.markspcsolution.com/atom.xml?redirect=false&start-index=1&max-results=500


Here you just need to change the site URL. Replace the word munnamark with yours. And for sure, this will help search engines for better crawling of your site.



AdSense Friendly Custom Robots.txt for Blogger:

This custom robots.txt might be less useful for AdSense enabled blog. If you use this type of custom robots.txt in your blog, AdSense will not be able to show relevant ads on your site. Consequently your ads impression you will affected seriously. And ultimately, your revenue will fall.

But a little change in that could save you all the day. If your site is AdSense enabled, then use the following code:


User-agent: Mediapartners-Google
Disallow:
# Blogger Sitemap generated on 2013.11.04
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://
www.markspcsolution.com/atom.xml?redirect=false&start-index=1&max-results=500


Look, only the blue part has been added with the previous code. If you're working with AdSense then consider using this code as your robots.txt element. Actually the blue part is used by Google AdSense only. 

If you use this code (User-agent: Mediapartners-Google Disallow:), AdSense will be able to crawl your posts properly. Then it will show relevant ads on your site. And the other part of the code will be used by the search engines for indexing your contents. 



Stay with Marks PC Solution to get more interesting IT topics!



22 comments:

  1. Rahaman sir I am Bis followers of your blog. Sir I am suffering from 55 Duplicate meta description problem in my blog http://livetechnologybd.blogspot.com/. Sir I am using free downloaded template in my blog. Sir if you have any effective solution please help me. Sir my humble request please help me.

    ReplyDelete
    Replies
    1. :)
      I will never visit your site if you don't change your template - I promise. Cause I use only 1 GB monthly pack.

      Delete
    2. sir i have change my blog template, please now tell me how to fix duplicate meta description.

      Delete
    3. Actually you didn't get me. You were supposed to make your site faster. Use a simple template. Or simply use one of the default blogger templates.

      By the way - where do you get that you have duplicate meta? Have you changed your template manually? Maybe your current template has some problems. Use simple blogger template.

      Forget about styles, social gadgets. Visitors hate these. People hardly like or plus one your site. So remove them if possible. Speed is necessary.

      Delete
    4. Sir now I am using Simple Template of blogger. I get duplicate meta description in webmaster tool.

      Delete
  2. thank you very helpful information from me to increase adsense earnings on this blog. belajarkreatif [dot] net

    ReplyDelete
  3. I've used your given code in my blog but what about the :generated on 2013.11.04?? in code.Is it important to edit the date?

    ReplyDelete
    Replies
    1. Maybe not. But you can use recent date if you wish.
      You can use 2014.01.06 . . .

      Delete
  4. For Anurag My answer is Jani means Lovable as we Speak in India

    My question about Custom Robots arise when I was using Blogspot domain I was Getting around 30000 daily views and I moved to www.piratedhacker.com so now my traffic was good for some days but a sudden fall took place and it got reduced to 5000 daily. So any Solution you may be having

    ReplyDelete
    Replies
    1. and also how to use Custom Robot I mean changing the sitemap date will work or not. a

      Delete
    2. Just replace the URL in sitemap. Maybe date is not much inportant. But you can use the current date.

      And to reduce the fall in pageviews, you have to add the new domain in Google Webmaster. And only then you can change your address. I mean .blogspot to .com.

      Delete
    3. It is already added automatically. You are using the same that you mentioned here..?? then I am applying them.
      Can I get your template.. I know its simlpe template from blogger but as you said before being simple is good so I want this. Rate mine template my site - www.piratedhacker.com

      Delete
    4. :)
      You know this is simple template. But why do you want this one? I've manually edited it. And everywhere you will get my site's link. You have to change them again.

      Better you can choose the simple template (your site is too heavy). Then I can help you to customize it. Even I've tips to customize the simple template as you want. You can chat me: munnamark at Google Talk or Gmail.

      Delete
  5. This comment has been removed by the author.

    ReplyDelete
    Replies
    1. How old your site is? It may take some times to be indexed in Google. And I've checked your site. There is no content.

      Try to write something. Trying to index in the search engines without writing is not a good idea at all !!!

      Delete
  6. JazakAllah Brother...
    you are doing eternal blessed work..... just about learning..... very useful.... and I also get some out of above..... Stay Blessed!

    https://scoutshr.blogspot.com

    ReplyDelete
  7. Hello sir, can you please tell me how can i get my blog posts in google search some of my stuffs are unique some are indexed by google and some are not my blog link is www.allinone1000.blogspot.com

    ReplyDelete
    Replies
    1. Keep Writing. Some words are highly competitive. In those keywords your site will perform poor in search resutls.

      But as soon as you rank higher, your posts will start appearing in Goolge.

      Delete
  8. Asalam alaekum,

    i got 86euro since 5month ago and everything stopped after then, i hardly see 2cent per day now.
    i serious need help! my blog is http://betonmarketsecret.blogspot.com/
    let me highlight my problem more, so that You could help to the last.
    1st : i need to know how get the Right Keyword to that blog.
    2nd : i need to change it to Dot.com
    3rd : i need to know how to get Pure traffic to my blog.

    Thanks
    Abdul Akeem

    ReplyDelete
    Replies
    1. Contact me via Google Talk. munnamark@gmail.com

      Delete