Bloggers who have SEO Knowledge, should know about robots.txt and its use. But it is quite unfamiliar to the new bloggers. But some of them might be interested in Blogger's search preferences and they must ask about what is Custom robots.txt? Well this is not so easy to explain. To the new bloggers, the explanation might be confusing. In order to explain the term robots.txt, I've to tell about search bots or search crawlers. But I will try to explain the term clearly.
Suppose, you've opened a blog in blogger. continuously you're publishing new posts. If everything is okay, your post will appear in Google Search within few hours. If your post is matched with someone's search topic, your site will appear in the search results. How it's possible?
Blogger's Default robots.txt:
Search engines scan your site for new posts continuously. This is known as crawling. Search bots crawl your site regularly and index your posts in their directory. By default, blogger comes with a robots.txt for your site that helps search engines crawl your site.
But as we know, the default or general settings never guarantee the accurate result. So we need to customize the settings as our requirement. Same rule is applicable for blogger's robots.txt.
Custom Robots.txt for Blogger Blog:
Today you will learn how to customize the robots.txt of blogger. There is a rumor that the default robots.txt can't index the old posts properly. If you have a proper custom robots.txt, your posts will be indexed properly. In my last post, I discussed about creating XML sitemap for blogger blog. There I gave example of XML sitemap. You can use that sitemap as your custom robots.txt content.
Look at the above image. This is how you can customize your robots.txt element. Or use the instruction below:
Sign in to Blogger Dashboard > Settings > Search Preferences > Crawler and Indexing > Custom robots.txt > Enable? > Yes > Now add the code below > Save Changes.
# Blogger Sitemap generated on 2013.11.06
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://www.markspcsolution.com/atom.xml?redirect=false&start-index=1&max-results=500
Here you just need to change the site URL. Replace the word munnamark with yours. And for sure, this will help search engines for better crawling of your site.
AdSense Friendly Custom Robots.txt for Blogger:
This custom robots.txt might be less useful for AdSense enabled blog. If you use this type of custom robots.txt in your blog, AdSense will not be able to show relevant ads on your site. Consequently your ads impression you will affected seriously. And ultimately, your revenue will fall.
But a little change in that could save you all the day. If your site is AdSense enabled, then use the following code:
User-agent: Mediapartners-Google
Disallow:# Blogger Sitemap generated on 2013.11.04
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://www.markspcsolution.com/atom.xml?redirect=false&start-index=1&max-results=500
Look, only the blue part has been added with the previous code. If you're working with AdSense then consider using this code as your robots.txt element. Actually the blue part is used by Google AdSense only.
If you use this code (User-agent: Mediapartners-Google Disallow:), AdSense will be able to crawl your posts properly. Then it will show relevant ads on your site. And the other part of the code will be used by the search engines for indexing your contents.
Rahaman sir I am Bis followers of your blog. Sir I am suffering from 55 Duplicate meta description problem in my blog http://livetechnologybd.blogspot.com/. Sir I am using free downloaded template in my blog. Sir if you have any effective solution please help me. Sir my humble request please help me.
ReplyDelete:)
DeleteI will never visit your site if you don't change your template - I promise. Cause I use only 1 GB monthly pack.
sir i have change my blog template, please now tell me how to fix duplicate meta description.
DeleteActually you didn't get me. You were supposed to make your site faster. Use a simple template. Or simply use one of the default blogger templates.
DeleteBy the way - where do you get that you have duplicate meta? Have you changed your template manually? Maybe your current template has some problems. Use simple blogger template.
Forget about styles, social gadgets. Visitors hate these. People hardly like or plus one your site. So remove them if possible. Speed is necessary.
Sir now I am using Simple Template of blogger. I get duplicate meta description in webmaster tool.
Deletethank you very helpful information from me to increase adsense earnings on this blog. belajarkreatif [dot] net
ReplyDeleteI've used your given code in my blog but what about the :generated on 2013.11.04?? in code.Is it important to edit the date?
ReplyDeleteMaybe not. But you can use recent date if you wish.
DeleteYou can use 2014.01.06 . . .
Jani....Are you using it..??
ReplyDeleteDefinitely :)
DeleteBut what do you mean by Jani?
For Anurag My answer is Jani means Lovable as we Speak in India
ReplyDeleteMy question about Custom Robots arise when I was using Blogspot domain I was Getting around 30000 daily views and I moved to www.piratedhacker.com so now my traffic was good for some days but a sudden fall took place and it got reduced to 5000 daily. So any Solution you may be having
and also how to use Custom Robot I mean changing the sitemap date will work or not. a
DeleteJust replace the URL in sitemap. Maybe date is not much inportant. But you can use the current date.
DeleteAnd to reduce the fall in pageviews, you have to add the new domain in Google Webmaster. And only then you can change your address. I mean .blogspot to .com.
It is already added automatically. You are using the same that you mentioned here..?? then I am applying them.
DeleteCan I get your template.. I know its simlpe template from blogger but as you said before being simple is good so I want this. Rate mine template my site - www.piratedhacker.com
:)
DeleteYou know this is simple template. But why do you want this one? I've manually edited it. And everywhere you will get my site's link. You have to change them again.
Better you can choose the simple template (your site is too heavy). Then I can help you to customize it. Even I've tips to customize the simple template as you want. You can chat me: munnamark at Google Talk or Gmail.
This comment has been removed by the author.
ReplyDeleteHow old your site is? It may take some times to be indexed in Google. And I've checked your site. There is no content.
DeleteTry to write something. Trying to index in the search engines without writing is not a good idea at all !!!
JazakAllah Brother...
ReplyDeleteyou are doing eternal blessed work..... just about learning..... very useful.... and I also get some out of above..... Stay Blessed!
https://scoutshr.blogspot.com
Hello sir, can you please tell me how can i get my blog posts in google search some of my stuffs are unique some are indexed by google and some are not my blog link is www.allinone1000.blogspot.com
ReplyDeleteKeep Writing. Some words are highly competitive. In those keywords your site will perform poor in search resutls.
DeleteBut as soon as you rank higher, your posts will start appearing in Goolge.
Asalam alaekum,
ReplyDeletei got 86euro since 5month ago and everything stopped after then, i hardly see 2cent per day now.
i serious need help! my blog is http://betonmarketsecret.blogspot.com/
let me highlight my problem more, so that You could help to the last.
1st : i need to know how get the Right Keyword to that blog.
2nd : i need to change it to Dot.com
3rd : i need to know how to get Pure traffic to my blog.
Thanks
Abdul Akeem
Contact me via Google Talk. munnamark@gmail.com
Delete