SEO and Robot.txt File for WordPress

Sample-wp-robots.txt-file

Do robots like your source code?

I was searching the other day for a robots.txt fileĀ  example and it took me a few minutes to find some fragmented samples in forums. So here is a quick post for you wordpress bloggers who are managing your own Search Engine Optimization..SEO.

Joomla fortunately, automatically sets up a robot.txt file when you do your install through cpanel.

With wordpress for some reason, you need to do it manually. I find this a bit bizarre especially since WP espouses to be very search engine friendly right out of the gate. Anyways here is a a basic robot.txt sample.

Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /feed
Disallow: /comments
Disallow: /category/*/*
Disallow: */trackback
Disallow: */feed
Disallow: */comments
Disallow: /*?*
Disallow: /*?
Allow: /wp-content/uploads

# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*

# Google AdSense
User-agent: Mediapartners-Google*
Disallow:
Allow: /*

# digg mirror
User-agent: duggmirror
Disallow: /

Sitemap: http://www.example.com/sitemap.xml

Copy and paste this into a file named robots.txt;

You can use notepad, word or any text program will work. Then, upload the this to the root level of your domain. Another way is to just add a plugin to your wordpress blog. There are several robot.txt plugins to choose from, some will add a meta robots tag to your wordpress pages.

Here is another more comprehensive wordpress robots.txt example.

For more tips on robot.txt and search engine optimization you can post a question in the comments section.

Note, if you are a building blog, wordpress has the best user interface and is great for fast preformatted content development. However if you are building a website with several different types of content that need function, I would checkĀ  out Joomla 1.5 and it’s joomla seo extensions.

What CMS platform do you prefer for building the best search engine optimization into your website, Joomla, Drupal or WordPress?

Facebooktwitterlinkedinmail

Leave a Comment

Your email address will not be published.