Bandwidth and pagerank both are very important for any website. Bandwidth might not matter much if you are hosting with unlimited bandwidth but it’s a major issue when you have limited bandwidth and your website is becoming hit day by day :) Google pagerank is always an important issue for any website.
Your site is also accessed by bots/crawlers of yahoo, google, msn, etc. They will also be using your site’s bandwidth. You can considerably reduce your site’s bandwidth by reducing their (bots) access to your pages. Bots will be crawling your wp admin pages as well which is just futile because nobody wants to show their admin pages in search results.
You can restrict the bots to crawl these pages. You can do this by adding meta robots tags or creating a robots.txt file on your site’s root. But for a worpress site, you need not have to worry because there is an excellent plugin called Robots Meta which adds meta robots tags to WordPress pages. Just install it and configure.
Robots Meta provides a lot of options. Some of them are as under:
- Prevent indexing of your search result pages, while still allowing the search engines to follow the links on them, by adding noindex,follow robots meta tags.
- Disallow indexing of subpages to your homepage, category pages, author pages and tag pages, to prevent duplicate content.
- Prevent indexing of your login, register and admin pages by adding noindex robots meta tags.
- Add noodp an noydir meta robots tags, allowing you to opt out of DMOZ and Yahoo! Directory descriptions.
- Prevent Yahoo! and Google from indexing your feeds by adding a meta tag to their head-section.
- Prevent indexing of just your comment feeds.
- Disable author and date-based archives.
- Prevent attachment pages from ranking in the search results over your articles.
- Edit your .htaccess and your robots.txt from within WordPress.
- Assign robots meta tags to individual posts & pages.