Google Digital Marketing | SMO & SEO Tips | Search Engine Optimization & SEM: About Robots.txt file

Monday, June 23, 2014

About Robots.txt file

The standard of Robots.txt was proposed by Martijn Koster, when working for Nexor 1994 on a mailing list, the main communication channel for related activities at the time. Koster suggest robots.txt, after he wrote a badly-behaved web spider that caused an inadvertent denial of service attack on Koster's server.

Robots.txt uses to control the crawling of websites. If a site owner wants to give some instructions to search engine robots about the crawling, they must place a text file in website root directory folder called robots.txt (e.g.www.example.com/robots.txt). You need robots.txt file only if your site have some content that you don't want search engines to index.

If a robot wants to vists a Website, Before it does so it firsts checks for robots.txt. And check about the allow and disallow section in the website.

User-agent: *
Disallow: /

The User-agent: * means this section applies to all robots. The Disallow: / tells the robot that it should not visit any pages on the site.

To block the entire site,
User-agent: Googlebot
Disallow: /

To block a page,
User-agent: Googlebot
Disallow: /private-file.html

To remove a directory and everything in it from Google search,
User-agent: Googlebot
Disallow: /junk-directory/

To block a specific image from Google Images,
User-agent: Googlebot-Image
Disallow: /images/dogs.jpg

To block all images on your site from Google Images:
User-agent: Googlebot-Image
Disallow: /

To block files of a specific file type (for example, .jpg,.gif),
User-agent: Googlebot
Disallow: /*.gif$

Robots.txt play an important role to fix crawling errors of websites.

Malware robots can ignore your /robots.txt. That scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.

64 comments:

Aanaxa said...

Thank you for posting such a useful, impressive and informative content

SEO and SMO Company In Delhi

Anonymous said...

Great, Thanks for sharing this article.Really looking forward to read more. Awesome.E-commerce Training Courses In Coimbatore

Anonymous said...

Thank you for sharing the details. I found the information incredibly helpful.
SEO services Melbourne | email marketing services Melbourne

Unknown said...

Thanks to sharing about robots.txt file. I get many more information from this blog.

Bart said...

Great post..Thanks for this information.

emarks said...


Hi, nice post it is!!! want to involved in database business?

The clients can showcase their products and services by using such lists to reach out the Target Businesses or Consumers.

Business & Consumer Database

Unknown said...


Want to master click perfect Institute(SEO) in 2016? Join SEO

Certification Course. Join FREE Demo!

Unknown said...
This comment has been removed by the author.
Unknown said...

Nice blog! thanks for sharing the details.. keep sharing..click perfect

WHTL said...

This is an awesome article. Very informative and detailed. Thanks for spending so much effort in sharing this valuable information.

With Regards,
SEO services in Bangalore | Digital Marketing Company in Bangalore

keerthi said...

I’ve been browsing on-line greater than three hours today, but I never discovered any attention-grabbing article like yours. It is beautiful worth sufficient for me. Personally, if all webmasters and bloggers made good content material as you did, the net will be a lot more helpful than ever before.

Digital marketing company in Chennai

priyakeerthi said...

Great post!! This can be one particular of the most useful blogs We’ve ever arrive across on this subject. Basically Wonderful. I am also a specialist in this topic so I can understand your hard work.

SEO Company in Chennai | Digital Marketing Company in Chennai

Unknown said...

It brings an great informative that brings the recent updates on websites crawl which gives an best explanation for my future reference.

SEO Training in Chennai

jeffie said...

Being new to the blogging world I feel like there is still so much to learn. Your tips helped to clarify a few things for me as well as giving..
SEO Company in Chennai

SEO Bangalore said...

Very great blog. Nice to see your blog. Thanks for sharing SEO Bangalore | Internet Marketing Company Bangalore

Hannah Baker said...

pay per click campaign is very effective for promotion using online ads.

Anonymous said...

Nice blog! It's an a awesome blog you have been written, Thanks for sharing it click perfect

Daarshik said...

I do believe all of the concepts you’ve introduced in your post. They’re very convincing and will definitely work. Nonetheless, the posts are too short for novices. May you please extend them a bit from subsequent time? Thank you for the post.
SEO Company in Chennai

Unknown said...

Hey Very nice Blog,Thanks For Sharing..

smo company in lucknow
graphics designing company in lucknow
best web development company in lucknow

Navya Kumar said...

Very nice post here and thanks for it .I always like and such a super contents of these post.Excellent and very cool idea and great content of different kinds of the valuable information's.
Web Development Company in India

Unknown said...

This is an awesome post.Really very informative and creative contents. These concept is a good way to enhance the knowledge.I like it and help me to development very well.Thank you for this brief explanation and very nice information.Well, got a good knowledge.

AWS Training in Chennai

Unknown said...

Hey Gyss Check out This..

Creative Kalakar is best SEO company in Lucknow. We are Best Digital Marketing Agency specializing in helping businesses make a profitable income from the Internet.
Our services span into every facet of Digital Marketing,
including Search Engine Optimization (SEO), Pay Per Click Management (PPC),
Web Design, Graphics Design and Online Reputation Management (ORM).


Unknown said...

Hey Very nice Blog,Thanks For Sharing..

Best Seo training in lucknow

Best Seo courses in lucknow

Digital marketing institute in lucknow

Seo training institute in lucknow

Unknown said...


Hey Very nice Blog,Thanks For Sharing..

Best seo company

Best Seo Services Company in Lucknow | #1 Digital Marketing Company

Best ORM Company

Best SMM Company

Unknown said...

Hey very nice blog!!..

Top Digital Marketing Company India

Digital Marketing Agency in Delhi

top SEO services company, Delhi.

Unknown said...

Hey Very nice Blog,Thanks For Sharing..

Best Seo training in lucknow

Google Adwords classes in lucknow

PPC courses in lucknow

Digital marketing courses

Best Digital Marketing institute in lucknow

Unknown said...

Hey Very nice Blog,Thanks For Sharing..

Best Seo training in lucknow

Google Adwords classes in lucknow

PPC courses in lucknow

Digital marketing courses

Best Digital Marketing institute in lucknow

Unknown said...

Hey Gyss Check out This..

Creative Kalakar is best SEO company in Lucknow. We are Best Digital Marketing Company in lucknow specializing in helping businesses make a profitable income from the Internet.
Our services span into every facet of Digital Marketing,
including Search Engine Optimization (SEO), Pay Per Click Management (PPC),
Web Design, Graphics Design and Online Reputation Management (ORM).
For More info.......Please Click here

Jessica said...

As we know we are moving with the Digital Marketing. But when we are moving with the certain the way of exposure we need to follow with the desired group like service providers. Definitely it will be useful for people those who are searching with the digital marketing.

SEO Company in Chennai

Tessa said...

It's interesting that many of the bloggers your tips helped to clarify a few things for me as well as giving.. very specific nice content. And tell people specific ways to live their lives.Sometimes you just have to yell at people and give them a good shake to get your point across.

SEO Company in Chennai

Ancy merina said...
This comment has been removed by the author.
Shivam said...

Hii there
Nice Blog
Guys you can visit here to know more
SEO Plan India

Coepd said...

We are glad to announce that in COEPD we have introduced Digital Marketing Internship Programs (Self sponsored) for professionals who want to have hands on experience. In affiliation with IT companies we are providing this program. Presently, this program is available in COEPD Hyderabad premises. We deem in real time practical Internship program. We guide participants through real-time project examples and assignments, giving credits for Real-Time Internship. Our digital marketing certified mentors tutor our learning people through modules of Digital Marketing in an exhaustive manner. This internship is intelligently dedicated to our avid and passionate participants predominantly acknowledging and appreciating the fact that they are on the path of making a career in Digital Marketing discipline. We upskill and master the nitty-gritty of the Digital Marketing profession. More than a training institute, COEPD today stands differentiated as a mission to help you "Build your dream career" - COEPD way.

http://www.coepd.com/DMInternship.aspx

Unknown said...

WOW NICE SHARe

digital marketing course

Unknown said...

Thanks for your article and sharing your expertise, it's really appreciated.It helped me a lot for what i was searching for

Social media Marketing company India

Suruchi Pandey said...

I found the article very interesting and worth reading. I read the whole article and got much to gain. I really appreciate you for employing your efforts to impart this considerable detail to us,
Website Design Company in Lucknow | Website Redesign Services

senthilkumar said...

Nice blog with excellent information. Thank you, keep sharing.

ac service ambattur
ac installation ambattur

Unknown said...

Thanks for sharing such a wonderful post
4 star hotel in Lucknow

rgerht said...

healrun is a health news blog we provide the latest news about health, Drugs and latest Diseases and conditions. We update our users with health tips and health products reviews. If you want to know any information about health or health product (Side Effects & Benefits) Feel Free To ask HealRun Support Team.

Anonymous said...

Hey Nice Blog!!! Thank you for sharing information. Wonderful blog & good post.Its really helpful for me, waiting for a more new post. Keep Blogging!!!

http://www.websofy.com/
http://www.websofy.com/domain-hosting.html
http://www.websofy.com/graphic-logo.html

sangeethalimra said...

Great article good information ,

Top Digital Marketing Companies In Bangalore | Best Digital Marketing Agency In Bangalore | Web Development Company Bangalore | Website Design Company Bangalore

Anonymous said...

I am not sure where you are getting your info, but great topic. I needs to spend some time learning more or understanding more. Thanks for excellent info I was looking for this information for my mission.
buy facebook page likes

Rina Kumari said...

Hello,
This article is very great. Thank you sharing such a great article. Visit; Cariva Technologies

Jyoti Singh said...

Nice Blog!! Thanks for sharing. For more information visit here:-
Digital Marketing Course

Sonya Armstrong said...

You have a good sense of writing, I must say. Your post has those facts that aren't accessible from anyplace else. It's my humble request to you, and please keep writing such outstanding articles google ranking. cross-check my web site as well- خبير سيو

AzzieNegron said...

Hey there just wanted to give you a quick heads up. The text in your post seem to be running off the screen in Ie. I'm not sure if this is a format issue or something to do with web browser compatibility but I thought I'd post to let you know. The style and design look great though! Hope you get the problem solved soon. Kudos
buy youtube shares

AmelCS19 said...

Great post, thanks for sharing this information related SEO......Keep on sharing.
Professional SEO Services
Buy SEO Services
Buy SEO
SEO Services in Chandigarh
Best SEO Services in Chandigarh

Thaj Shapcott said...

Hello, i think that i saw you visited my website so i came to “return the favor”.I'm trying to find things to improve my site!I suppose its ok to use some of your ideas!!
buy youtube shares

Anonymous said...

I loved as much as you will receive carried out right here. The sketch is attractive, your authored subject matter stylish. nonetheless, you command get got an edginess over that you wish be delivering the following. unwell unquestionably come further formerly again as exactly the same nearly a lot often inside case you shield this increase.
SEO companies sydney

Euridice Technologies said...

Hey Nice Blog!!! Thank you for sharing the information. Wonderful blog & good post.It's really helpful for me, waiting for a more new post. Keep Blogging!!!

digital marketing company in noida
digital marketing company in delhi
digital marketing agency in delhi

BruceDavis said...

I truly wanted to write down a small comment to thank you for those great ideas you are sharing at this site. My particularly long internet research has now been compensated with brilliant knowledge to go over with my contacts. I 'd assume that most of us readers are extremely lucky to be in a superb network with many marvellous individuals with beneficial concepts. I feel somewhat privileged to have discovered the web page and look forward to tons of more excellent times reading here. Thank you once again for a lot of things.
buy youtube shares

Anonymous said...

Thanks for sharing the useful blog. Are you looking for the best Google Adwords Services in Chennai? We are Branding Pilot's top SEM Service in Chennai. Stop waiting further and join us! Start Increase your Sale.
Best SEM service in Chennai

Nisha Verma said...

Courses which guarantee a job

Course with job guarantee

Courses that guarantee jobs

it training with job guarantee

100 job guarantee

job guarantee courses

job guarantee courses

Homework Help said...

Are you facing problems in writing your assignments? If so, connect with me and get my Assignment Help services by paying a decent amount. I have completed my graduation and post-graduation from Oman University and working as a professor. I am also providing my knowledge to students through writing their assignments and willing to solve their queries.
Online Assignment Help
Assignment Writing Help

confoundingsolutions said...


Your blog was good and informative.

Website Designing in Lucknow
Digital Marketing Company in Lucknow
Web Design Company in Lucknow
digital marketing company
best seo company in lucknow
seo company in lucknow
digital marketing agency
digital marketing services

confoundingsolutions said...


Digital Marketing Company in Lucknow
Web Design Company in Lucknow
digital marketing services
website designing in lucknow
shopify development company india
ppc company in lucknow
website development in lucknow
best seo company in lucknow
app development company in lucknow


confoundingsolutions said...

Website Designing in Lucknow
Digital Marketing Company in Lucknow
digital marketing company
best seo company in lucknow
seo company in lucknow
digital marketing agency
app development company in india

confoundingsolutions said...



digital marketing
online marketing
google digital marketing
digital marketing is
marketing agency
digital agency

itikalasuneetha said...

Nice Blog.Information provided is very useful.Thanks for sharing Information.Are you looking for best Google Adwords Services in Chennai. We are Branding Pilot top PPC Agency in Chennai. Stop waiting further and join with us! Start Increase your Sale.
SEM Services in Chennai

Neeta Borse said...
This comment has been removed by the author.
thesharebrokers said...

Great Post. icici lombard claim form

Nutra Health Pro Diet said...



proper use can also purpose weight reduction to be not unusual. How will which one is the right weight loss for you? I assume

I highlighted that right sufficient for you so the bubble-wrapped bundle stunned them. you have to praise yourself. it's far

without a doubt crucial which you know weight reduction backward and forwards.

https://www.nutrahealthpro.com

https://www.nutrahealthpro.com/bombketo-pro/

https://www.instagram.com/p/CJv_owqBgAn/

https://www.facebook.com/nutrahealthpro/posts/197034612136340

https://twitter.com/nutrahealthpro/status/1347210899476529281

https://in.pinterest.com/pin/596867756862978909/

Rahul Saxena said...

Best Private School in Bulandshahr
Best CBSE School in Bulandshahr

Highfly Seo said...

Highflyseo is the best SEO Company in Lucknow and our company will completely help to rank your website in the top of google and we do not make fake promises.