Sometimes you inherit a non Ruby based web app written in PHP, Perl, Java, C#, or Python and you want to create an automated functional/acceptance test suite to minimize the amount of manual QA you need to do. Selenium is a great tool for doing just that and at Common Sense Media I created the Selenium Continuous Integration Runner to make it easy to run Selenese based tests from your Continuous Integration Server such as Hudson, Cruise Control, or Teamcity. That said, Selenese based tests leave something to be desired in terms of readability and maintainability.

The Ruby camp is often at the forefront of great testing frameworks with Aslak Hellesøy often leading the charge with innovations like RSpec and Cucumber. I’d read and played around with Cucumber on Rails based apps and was loving the concept of english language specs that are actually executable as well. But when my friend and former colleague Chris Kimm mentioned that the testing group at his company was using it to test their Java application, I started to get excited as that was the problem I was about to try and tackle at VolunteerMatch.

However, getting the project structure in place required a little help from the Cucumber mailing list and in the hopes of saving others the time, I’ve published the Standalone Cucumber Test Suite to Github. It’s simply a Cucumber project designed to test a non Ruby web application with a sample search.feature that uses Google search as the basis for the test. It uses Capybara with the Selenium Webdriver on the backend but you can also switch it to use Culerity or Webrat if you don’t want it actually firing up a real browser to run the tests. Fork it or download it to start testing your existing website with Cucumber!

I’ve been doing some iPhone and iPod Touch development and if like me you’re used to web development in languages like Ruby, Java, and Python, the learning curve to build a native iPhone app in Objective C is quite steep. Since my applications are online (in that you need to be connected for them to be useful) I was pysched to find a much easier way using two great open-source tools:

iUI consists of Javascript/CSS/images that allow you to build a mobile version of your app that looks and feels just like a native iPhone app

PhoneGap lets you create an iPhone application (that can be submitted to the app store) that displays a framed mobile version of your site

To build your iPhone app:

1. Make an iPhone/iPod Touch friendly version of your site with iUI (which your users can access in mobile safari)

2. Bundle it as an iPhone application (so the user has an app store version they can install) using PhoneGap and submit it to the app store. What’s even better is that with PhoneGap you can also target Blackberry and Android devices.

As a web developer I’m not sure how they could make it any easier for me, sweet! Plus you get the double exposure of the app store or when users visit your site with their iPhones they get the mobile version of your site.

Having worked with a lot of front-end developers I can confirm that almost all of them would benefit from this book since most still

• use floats for layout as opposed to position absolute (which is easily misunderstood because it’s positioning relative to the containing block)
• few seem to regularly run their XHTML through a validator (a pet peeve of mine)
• and even fewer seem to know to employ the concept of a CSS reset

It’s hard to convey all of the great stuff in this book but I can almost guarantee that if you work with CSS this book will open your eyes to a better way of doing things.

We currently front our site with 3 caching proxy servers to offload static content from the web servers and we haven’t found the right solution yet. We’ve tried a number of solutions including Oops and Apache with mod_cache (which we’re currently running) but they are either missing a critical feature, have buggy caching behavior, or crash under heavy load. The main features I’m looking for are:

1. Caches static content and pages and respects the expires header
2. Can gzip content on the fly (this saves us a bundle in bandwidth costs)
3. Can easily clear the cache when we move a new release live

I’d prefer an open-source product but if we end up needing to go with a commercial solution as a last resort I’m open to that as well, thanks for your suggestions or help!

State landing page (now called Research and Compare)

The school profile overview page:

The city page:

1. Amazon.com: set your user agent to Googlebot and you won’t see a search box anymore. Crawlers don’t do forms and Amazon saves some bandwidth.
2. Yahoo.com: almost every link is rewritten to go through a click tracking redirect but not so for crawlers. Yahoo doesn’t count the clicks from crawlers which is good for their metrics and the sites they link to get non redirected inbound links.
3. Salon.com: crawlers get the real home page without having to see a full page ad first. I think I’ll start surfing Salon as Googlebot from now on
4. CNN.com, MSN, Myspace, and EBay.com: crawlers don’t get served ad serving code saving bandwidth.

In all of the above cases these sites are serving adjusted content targeted to search engine user agents and generally it’s for the good. In the end the decision to cloak or not all comes down to good judgment and intent. If it helps crawlers avoid duplicate content or saves bandwidth then we shouldn’t be so quick to write off cloaking just because it has such a negative connotation. That said, if there’s any doubt in your mind be very weary and seek expert opinions because getting delisted for bad cloaking can result in a lot of lost revenue!

My first attempt to keep my Ruby on Rails sites running at Dreamhost involved modifying dispatch.fcgi to implement the frao_handler approach as described on the wiki. This left me with a 3% 500 error rate. Ughh!

My second attempt involved gracefully restarting dispatch.fcgi’s every hour or two to avoid the wrath of the Dreamhost process monitor. This reduced the 500 error rate down to 0.2% which was still too high!

I’ve finally gotten my 500 error rate down to 0 by making some minor changes to the signal handling code of dispatch.fcgi. If the dispatch.fcgi process is in the midst of handling a request I defer letting it be killed until the request is complete. I did this by installing a custom TERM signal handler that protects the dispatch.fcgi process while a request is being processed. This solution works for my Rails 1.1 and Rails 1.2 (previously Edge) based sites. Here’s my complete dispatch.fcgi:

#!/usr/bin/ruby1.8

require File.dirname(__FILE__) + "/../config/environment"
require 'fcgi_handler'

class RailsFCGIHandler
 private
   def busy_exit_handler(signal)
     dispatcher_log :info, "busy: asked to terminate during request signal #{signal}, deferring!"
     @when_ready = :exit
   end

   # Dreamhost sends the term signal and if we're handling a request defer it
   def term_process_request(cgi)
     install_signal_handler('TERM',method(:busy_exit_handler).to_proc)
     Dispatcher.dispatch(cgi)
   rescue Exception => e  # errors from CGI dispatch
     raise if SignalException === e
     dispatcher_error(e)
   ensure
     install_signal_handler('TERM', method(:exit_now_handler).to_proc)
   end
   alias_method :process_request, :term_process_request
end

RailsFCGIHandler.process!

Update 1/24/07: I’ve finally solved my Dreamhost 500 error problems with a different solution described in Keeping Rails running at Dreamhost part 2

This weekend I moved two of my newer Ruby on Rails based sites (http://windandtides.com and http://gearandboats.com) over to Dreamhost. You simply can’t beat their prices for hosting a small site (if you prepay for 2 years and use coupon code GABRITO to save $50 the total is $140.80) and I’ve found their customer service to be pretty responsive too.

I followed their Wiki based Ruby on Rails instructions and had the sites up and running with Capistrano in no time. However, I soon discovered that their process monitor likes to kill off long running processes (effectively taking my sites offline without warning) and the only indication of what’s wrong is this error in the apache error.log:

FastCGI: comm with (dynamic) server "/path/to/dispatch.fcgi" aborted: (first read)
  idle timeout (60 sec), referer: http://gearandboats.com
FastCGI: incomplete headers (0 bytes) received from server "/path/to/dispatch.fcgi",
  referer: http://gearandboats.com

To fix this I wrote a short ruby script fcgimaint.rb that I run once per hour via cron to restart my dispatch.fcgi processes (first with a USR1 and then a kill -9 if the USR1 didn’t work). This has the added benefit of cleaning up accumulated FastCGI processes and starting with a fresh memory footprint in case of a memory leak:

Crontab entry: 33 * * * * /home/myuser/bin/fcgimaint.rb

#!/usr/bin/ruby
def dispatchers()
  ps = `ps ux`.grep(/dispatch\.fcgi/)
  ps.map {|p| {'pid' => p.split[1], 'start' => p.split[8]}}
end
dispatchers.each { |d| `kill -USR1 #{d['pid']}` }
sleep 30
# Kill dispatchers that didn't restart nicely
dispatchers.each do |d|
  if (Time.now.hour != d['start'][0..1])
    `kill -KILL #{d['pid']}`
  end
end

One other tool I found useful (especially during this process when the sites were going down periodically) was using Geoffrey Grosenbach’s modifications to dwatch to monitor my sites and email my cell phone if there was a problem.

Caveat: if your site does more than 10,000 page views per day then a shared hosting plan is not the answer, you should be looking at dedicated or VPS hosting.