Robot.txt File


Prev Tutorial Next Tutorial

What is Robot.txt File

Robots.txt is a text file. it gives instruction to search engine crawlers about indexing and caching of a webpage, file of a website or directory, domain. Using this file you can also block search engine for crawing.
Use the robotz.txt file to Restrict Search Engine crawlers from indexing selected areas of your websites

robots.txt file

In simple word using robots.txt file you can allow or disallow search engine crawlers to visit your website (allow or block search engine on your website).

Robots.txt file mainly uses two keywords, User-agent and Disallow. Other one keyword is Allow.

  • User-agents are search engine robots (or web crawler software).
  • Disallow is a command for the user-agent that tells it not to access a particular URL.

Syntax

User-agent: [the name of the robot the following rule applies to]

Disallow: [the URL path that you want to block]

Allow: [the URL path that you want to unblock]

Disallow all robots

User-agent: *
Disallow: /

The User-agent: * means this section applies to all robots. The Disallow: / tells the robot that it should not visit any pages on the website.

Allow all robots

User-agent: *
Disallow: 

The User-agent: * means this section applies to all robots (* repersent all). The Allow: / tells the robot visit all pages on the website.

Disallow some url

User-agent: *
Disallow: /cgi-bin/
Disallow: / private_file.html

Note: robots.txt file store inside www or public_html directory on your website.

correct location where put robots.txt file on your website is www.tutorial4us.com/robots.txt
you can see our robots.txt file Click Here

Disallow particular directory

In below syntax we disallow all search engine to visit or craw java and seo directory or folder, it means no any url is indxed form these two directory or folder.

Disallow particulat directory

User-agent: *
Disallow: /java/
Disallow: /seo/
Block search engineSyntax
Block Entire website
User-agent: *
Disallow: /
Block directory and its contents
User-agent: *
Disallow: /sample-directory/
Block prticular url
User-agent: *
Disallow: /private_file.html
Block specific image
User-agent: *
Disallow: /images/cats.jpg
Block specific image from Google Images
User-agent: Googlebot-Image
Disallow: /images/dogs.jpg
Block all images on your site from Google Images
User-agent: Googlebot-Image
Disallow: /
Block files of a specific file type (.pdf)
User-agent: *
Disallow: /*.pdf$

Prev Tutorial Next Tutorial

Google Advertisements
Download Projects

Google Adsense Advertisements

Buy 3 Year Old Domain, Tutorialday.com, SeosmoTools.com @$200

Buy This Ad Space @$50 per Month, Ad Size 300X250 Contact on: hitesh.xc@gmail.com WhatsApp no. 8882538826

Yahoo Advertisements