1. Amazon.com: set your user agent to Googlebot and you won’t see a search box anymore. Crawlers don’t do forms and Amazon saves some bandwidth.
2. Yahoo.com: almost every link is rewritten to go through a click tracking redirect but not so for crawlers. Yahoo doesn’t count the clicks from crawlers which is good for their metrics and the sites they link to get non redirected inbound links.
3. Salon.com: crawlers get the real home page without having to see a full page ad first. I think I’ll start surfing Salon as Googlebot from now on
4. CNN.com, MSN, Myspace, and EBay.com: crawlers don’t get served ad serving code saving bandwidth.

In all of the above cases these sites are serving adjusted content targeted to search engine user agents and generally it’s for the good. In the end the decision to cloak or not all comes down to good judgment and intent. If it helps crawlers avoid duplicate content or saves bandwidth then we shouldn’t be so quick to write off cloaking just because it has such a negative connotation. That said, if there’s any doubt in your mind be very weary and seek expert opinions because getting delisted for bad cloaking can result in a lot of lost revenue!

• acts_as_sluggable
• acts_as_urlnameable
• acts_as_slugable
• acts_as_friendly_param
• permalink_fu

Obie’s recent post on search engine friendly URL’s in Ruby on Rails 1.2 and greater couldn’t have been more timely. I was about to tackle search engine friendly URL’s on my little local san francisco bay area boating classifieds site and after reading his blog post, 15 minutes later it was done. Here’s the old URL structure:

http://gearandboats.com/forums/1/topics/51

and the new (longer for humans but much better for SEO):

http://gearandboats.com/forums/1-boats/topics/51-fantasia-35-mark-ii-cruiser

The most important thing is that the URL’s are backwards compatible because everything after the ID is ignored which is key for pages that are already in the search engines!

To achieve this I combined Obie’s approach with Ricks Permalink_fu plugin (for converting text to permalinks):

script/plugin install http://svn.techno-weenie.net/projects/plugins/permalink_fu

Then in my topic.rb model object I do:

and I double checked to make sure that the views always call link_to passing the model object rather than the ID (so that the to_param method has access to the id and the title). Fortunately beast.caboo.se (which is what I based my site on) make heavy use of named routes:

After that I was pretty much done but I had some really long URL’s so I truncated the title to the first 5 words by changing

to

.

CSS offers us a lot of power to style and decorate our pages with images, however, replacing text with an image is one gap not well addressed by CSS such that you often see sites doing:

<a href=”/”><img alt=”Home” src=”home.gif”/></a>

which is obviously bad for 2 reasons:

1. It violates the separation of content and design by including a design element in your HTML
2. Your SEO will suffer because image alt tags don’t carry as much weight as real text

As a result a number of people have come up with clever image replacement techniques in CSS. It began in 2003 with the Fahrner Image Replacement technique (or Classic FIR), however, the technique I currently favor is the Mike Rundle’s Phark revised method which:

1. Works well cross browser (but be sure to test in IE 5 for possible issues if you’re still supporting it)
2. Works with a screen reader such as JAWS for accessibility
3. Works well for SEO (many techniques style the text with display:none which likely won’t work with the next generation of crawlers such as Google’s new Mozilla based crawler)

Taking the example above, the correct approach would be as follows:

<a id=”home” href=”/”>Home</a>

Then apply the Phark revised method via CSS:

#home {
  display: block;
  text-indent: -9000px;
  background: url(/images/home.gif) no-repeat;
  height: 25px;
  width: 100px;
}

Here’s the result:

About Us

Now we’ve replaced the text with an image yet still kept the separation of content and design clean. I stated that it’s bad design to use image tags in your HTML but I’d like to qualify that, it’s fine if your “content” is an image such as a product photograph in a web catalog, a graph, or a headshot in your user profile. Also I think it’s fine to bail on image replacement if you start wasting a lot of time in CSS layout land, it’s just good to start with the clean text based approach first and use it until it becomes impractical.

I’ve tested extensively in modern browsers across platforms and haven’t found a single browser that doesn’t translate the entity references properly. I suspect most of these are actually email harvesting crawlers just taking the value of the HREF and throwing it into a GET request without properly translating entity references.

I think we may have found one more way to identify crawlers masquerading behind real looking user agents!

When we started moving GreatSchools from Perl to Java + SpringMVC + Hibernate one of the first things we had to figure out was how to disable jsessionid‘s from getting appended to URL’s when using c:url in a JSP page. Jsessionid is terrible for search engine optimization because crawlers that don’t have cookies enabled will get URL’s from your pages with a jsessionid parameter appended. This makes it virtually impossible for a crawler to match the URL from an inbound link to your site with a page it has already crawled. What’s worse is that you risk having your page rank hurt with Google because it thinks you’re serving duplicate content, ouch!

Take one look on Google and you’ll see millions of results that contain jsessionid in the URL, those pages will have a tougher time ranking well against competitors that have consistent URL’s for users and crawlers.

The intention is good: allow sessions to work for users that don’t have cookies enabled. In reality it never worked that well though because most sites use c:url sporadically to properly encode URL’s and most sites require cookies as a site policy anyhow. The PHP session handling code has the same issue which many folks have battled with, fortunately for PHP users, it’s an option you can disable in your php.ini by setting session.use_only_cookies.

At GreatSchools we wrote our own version of c:url since we couldn’t find any way to disable c:url from writing out jsessionid’s. I think the time is right to make session handling with cookies only the default because it’s the more common scenario and that particular “feature” has caused more problems than the good it was initially intended to do.

What I’ve been seeing at GreatSchools is a lot more dramatic! For example our traffic from Google now accounts for 69% of search traffic (up from 49% a year ago) and Yahoo now accounts for 23% (down from 40% a year ago). That’s not just a slight market change, it’s a drastic market shift!

Jeremy wrote an interesting article on referrer traffic last October and how in his case it varies so dramatically from the publicly stated market share numbers. To be honest, I’ve never come across a site that gets as much traffic from Yahoo, AOL, or MSN as the published market share numbers indicate one should and many of the websites I’ve worked on are not technology focused at all. I suspect in reality Google has more market share than the publicly available numbers really indicate!

For the most part SEO is a simple equation of inbound links, keyword density, proper HTML, and URL structure. Do these SEO practices well and ethically (what they call white hat) long enough with a great web product that users like and you’ll rank well, period.

Where it gets tricky is keeping current with subtle changes such as moving URLs via 301 redirect (which you should probably hold off on until Big Daddy settles in), the importance of contextual relevance in inbound links, using a long lived domain name over a new domain, etc…

A while back I discovered the SEO Rockstars podcast and as a result now I spend much less time each month having to piece together blogs, webmasterworld, searchenginewatch, etc… However, if you’ve ever had any doubt as to whether SEO belongs in the sales/marketing department of a company, the intro theme song says it all… “jammin and spammin”… uh… the 80′s called and they want their Def Leppard tune back. Seriously though, Greg “Webguerrilla” Boser and Todd “Oilman” Friesen really know their stuff and if you too have to stay on top of SEO then I’d recommend subscribing to their podcast!

A coworker recently sent me a link to the free SEO Analyzer tool which looks like it might be helpful in the process. I don’t think it will ever replace our SEO consultant but it’s good to have tools like this in the arsenal. The other tools we use regularly are the Google Adwords Selector tool and the Overture Keyword Selector tool for helping us get the most effective wording to convey the content of the page.

If you’re interested in learning more about SEO or are curious which blogs you should be reading I’d recommend starting with this post on Matt Cutts excellent blog as he lists some great resources!