Bandwidth and pagerank both are very important for any website. Bandwidth might not matter much if you are hosting with unlimited bandwidth but it’s a major issue when you have limited bandwidth and your website is becoming hit day by day :) Google pagerank is always an important issue for any website.

Your site is also accessed by bots/crawlers of yahoo, google, msn, etc. They will also be using your site’s bandwidth. You can considerably reduce your site’s bandwidth by reducing their (bots) access to your pages. Bots will be crawling your wp admin pages as well which is just futile because nobody wants to show their admin pages in search results.

You can restrict the bots to crawl these pages. You can do this by adding meta robots tags or creating a robots.txt file on your site’s root. But for a worpress site, you need not have to worry because there is an excellent plugin called Robots Meta which adds meta robots tags to WordPress pages. Just install it and configure.

Robots Meta provides a lot of options. Some of them are as under:

- Prevent indexing of your search result pages, while still allowing the search engines to follow the links on them, by adding noindex,follow robots meta tags.
- Disallow indexing of subpages to your homepage, category pages, author pages and tag pages, to prevent duplicate content.
- Prevent indexing of your login, register and admin pages by adding noindex robots meta tags.
- Add noodp an noydir meta robots tags, allowing you to opt out of DMOZ and Yahoo! Directory descriptions.
- Prevent Yahoo! and Google from indexing your feeds by adding a meta tag to their head-section.
- Prevent indexing of just your comment feeds.
- Disable author and date-based archives.
- Prevent attachment pages from ranking in the search results over your articles.
- Edit your .htaccess and your robots.txt from within WordPress.
- Assign robots meta tags to individual posts & pages.

By Default, every robots at first searches for robots.txt file. It then follows the file for indexing the site content.
Any robots.txt file must contain two fields User-agent and Disallow.

User-agent: *
Disallow:

The User-agent field is for specifying robot name for which the access policy follows in the Disallow field. Disallow field specifies URLs which the specified robots have no access to.

User-agent: *
Disallow: /

Here “*” means all robots and “/” means all URLs. This is read as, “No access for any search engine to any URL”. Since all URLs are preceded by “/ ” so it bans access to all URLs when nothing follows after “/”. If partial access has to be given, only the banned URL is specified in the Disallow field.

Lets consider this example:
# access for MSNbot.
User-agent: MSNbot
Disallow:
User-agent: *
Disallow: /category/php/

Here we see that both the fields have been repeated. Multiple commands can be given for different user agents in different lines. The above commands mean that all user agents are banned access to /category/php/ except MSNbot which has full access. Characters following # are ignored up to the line termination as they are considered to be comments.

Make sure that the robots.txt file is always named in all lowercase (e.g. Robots.txt or robots.Txt is incorrect)