As some users have discovered the hard way, if your Gmail account gets compromised and your email is deleted, you’ll likely never get it back unless you have an off-site (outside of Google) backup.

One great service that does this for you is Backupify and their free plan will backup mailboxes up to 1GB. I personally love cloud data but prefer local backups so I use Getmail which is a little Python script to download my email. Here’s a more detailed post on how to setup Getmail by Matt Cutts. That said, I have my script backup my entire mailbox (instead of just my inbox) and I use IMAP instead of POP so here’s my getmail.gmail config file:

Here’s how we went about using a private S3 bucket to store our Git repository of Chef Recipes. Thanks to this post on using JGit to publish to S3 which got us started, the key difference is we wanted to use a private S3 bucket and it took us some experimenting to figure out how to update an existing Git repo (via fetch and merge) from S3.

Download jgit.sh, rename it to jgit and put it in your path (for example $HOME/bin).

Setup the .jgit config file and add the following (substituting your AWS keys):

vim ~/.jgit

Note, by not specifying acl: public in the .jgit file, the git files on S3 will be private (which is what we wanted). Next create an S3 bucket to store your repository in, let’s call it git-repos, and then create a git repository to upload:

In the above I’m using the s3cmd command line tool to create the bucket but you can do it via the Amazon web interface as well. Now let’s push it up to S3 (notice how we use jgit whenever we interact with S3, and standard git otherwise):

Now go somewhere else (e.g. cd /tmp) and try cloning it:

When it comes time to update it (because jgit doesn’t support merge or pull) you do it in 2 steps:

To that end one of the tools I want to give a shout out to is the MySQL Performance Tuning Primer Script. You download and run it against a production system (that has preferably been running under normal load for a day or two so that it’s gathered stats). It’s a read-only script so you don’t need to worry about it changing anything but it makes some great recommendations about which tuning parameters may need adjustment. Here’s a snippet from from a production server:

KEY BUFFER
Current MyISAM index space = 173 M
Current key_buffer_size = 1 G
Key cache miss rate is 1 : 1003
Key buffer fill ratio = 7.00 %
Your key_buffer_size seems to be too high.
Perhaps you can use these resources elsewhere
…
…
…

It’s no replacement for a DBA but if you want to get a somewhat sane my.cnf going for your particular application it’s a great place to start!

We currently front our site with 3 caching proxy servers to offload static content from the web servers and we haven’t found the right solution yet. We’ve tried a number of solutions including Oops and Apache with mod_cache (which we’re currently running) but they are either missing a critical feature, have buggy caching behavior, or crash under heavy load. The main features I’m looking for are:

1. Caches static content and pages and respects the expires header
2. Can gzip content on the fly (this saves us a bundle in bandwidth costs)
3. Can easily clear the cache when we move a new release live

I’d prefer an open-source product but if we end up needing to go with a commercial solution as a last resort I’m open to that as well, thanks for your suggestions or help!

Per Thomas Aylott’s suggestion on my previous post Keeping Rails Running at Dreamhost Part 2 I’ve decided to release my Dreamhost signal handling code as a Rails plugin. I’m using it on a Rails 1.2.1 site and a Rails 1.1.6 site.

Here’s the deal:

1. Get your Rails site running on Dreamhost (the wiki should you get you there).
2. Once you’ve got it working, if you experience occasional 500 errors try my signal handling plugin. I still had 500 errors with the frao_handler approach described on the wiki, but if my plugin doesn’t work for you it can’t hurt to try it.

Here’s how to install the plugin:

script/plugin install http://gabrito.com/svn/plugins/dreamhost

With the plugin installed, the dispatch.fcgi will defer exiting when Dreamhost sends your dispatch.fcgi process a TERM signal if it is in the middle of handling a request, otherwise it behaves normally. This has reduced the 500 error rate on my two rails based sites to zero, your mileage may vary though. If you still have problems please don’t comment here and say it doesn’t work, instead figure out the problem and email a patch to thuss [at] gabrito [o] com!

The subversion repo for the plugin is at http://gabrito.com/svn/plugins/dreamhost. If you have other Dreamhost related fixes that make sense for this plugin send them to me and I’ll incorporate them.

My first attempt to keep my Ruby on Rails sites running at Dreamhost involved modifying dispatch.fcgi to implement the frao_handler approach as described on the wiki. This left me with a 3% 500 error rate. Ughh!

My second attempt involved gracefully restarting dispatch.fcgi’s every hour or two to avoid the wrath of the Dreamhost process monitor. This reduced the 500 error rate down to 0.2% which was still too high!

I’ve finally gotten my 500 error rate down to 0 by making some minor changes to the signal handling code of dispatch.fcgi. If the dispatch.fcgi process is in the midst of handling a request I defer letting it be killed until the request is complete. I did this by installing a custom TERM signal handler that protects the dispatch.fcgi process while a request is being processed. This solution works for my Rails 1.1 and Rails 1.2 (previously Edge) based sites. Here’s my complete dispatch.fcgi:

#!/usr/bin/ruby1.8

require File.dirname(__FILE__) + "/../config/environment"
require 'fcgi_handler'

class RailsFCGIHandler
 private
   def busy_exit_handler(signal)
     dispatcher_log :info, "busy: asked to terminate during request signal #{signal}, deferring!"
     @when_ready = :exit
   end

   # Dreamhost sends the term signal and if we're handling a request defer it
   def term_process_request(cgi)
     install_signal_handler('TERM',method(:busy_exit_handler).to_proc)
     Dispatcher.dispatch(cgi)
   rescue Exception => e  # errors from CGI dispatch
     raise if SignalException === e
     dispatcher_error(e)
   ensure
     install_signal_handler('TERM', method(:exit_now_handler).to_proc)
   end
   alias_method :process_request, :term_process_request
end

RailsFCGIHandler.process!

Update 1/24/07: I’ve finally solved my Dreamhost 500 error problems with a different solution described in Keeping Rails running at Dreamhost part 2

This weekend I moved two of my newer Ruby on Rails based sites (http://windandtides.com and http://gearandboats.com) over to Dreamhost. You simply can’t beat their prices for hosting a small site (if you prepay for 2 years and use coupon code GABRITO to save $50 the total is $140.80) and I’ve found their customer service to be pretty responsive too.

I followed their Wiki based Ruby on Rails instructions and had the sites up and running with Capistrano in no time. However, I soon discovered that their process monitor likes to kill off long running processes (effectively taking my sites offline without warning) and the only indication of what’s wrong is this error in the apache error.log:

FastCGI: comm with (dynamic) server "/path/to/dispatch.fcgi" aborted: (first read)
  idle timeout (60 sec), referer: http://gearandboats.com
FastCGI: incomplete headers (0 bytes) received from server "/path/to/dispatch.fcgi",
  referer: http://gearandboats.com

To fix this I wrote a short ruby script fcgimaint.rb that I run once per hour via cron to restart my dispatch.fcgi processes (first with a USR1 and then a kill -9 if the USR1 didn’t work). This has the added benefit of cleaning up accumulated FastCGI processes and starting with a fresh memory footprint in case of a memory leak:

Crontab entry: 33 * * * * /home/myuser/bin/fcgimaint.rb

#!/usr/bin/ruby
def dispatchers()
  ps = `ps ux`.grep(/dispatch\.fcgi/)
  ps.map {|p| {'pid' => p.split[1], 'start' => p.split[8]}}
end
dispatchers.each { |d| `kill -USR1 #{d['pid']}` }
sleep 30
# Kill dispatchers that didn't restart nicely
dispatchers.each do |d|
  if (Time.now.hour != d['start'][0..1])
    `kill -KILL #{d['pid']}`
  end
end

One other tool I found useful (especially during this process when the sites were going down periodically) was using Geoffrey Grosenbach’s modifications to dwatch to monitor my sites and email my cell phone if there was a problem.

Caveat: if your site does more than 10,000 page views per day then a shared hosting plan is not the answer, you should be looking at dedicated or VPS hosting.

This makes it totally inadequate for running a production database server. On the forums people have argued for using replication or clustering solutions while others have advocated streaming redo logs to Amazon’s S3 service. However, let’s look at the disaster recovery scenario if all your virtual machines go down (and don’t think it can’t happen, racks overload their circuits, generators fail during a power outages, AC’s can’t keep up during heat-waves, etc…):

- With a VPS or dedicated server you startup your servers, do a data integrity check, and you’re up and running again!
- With EC2 you restart the database server image, then restore the full database backup from the previous night (which you’ve been backing up to S3), then you rerun all of the redo logs (which you’ve also been backing up to S3).

The latter scenario isn’t that bad with a small database but for with a large database it could add hours to your recovery time. I imagine Amazon will eventually add permanent storage to EC2 and if/when that happens you’d be able to compare Amazon EC2 to a VPS but right now it’s comparing apples and oranges!

In my personal experience partitioning a disk into root, boot, var, home, usr/local, tmp, var/mysql, etc… is generally overkill and has caused me far more headaches than it has prevented problems. Eventually you end up having one filesystem with gigabytes free that you really need for another filesystem which leads you to do a dump-repartition-restore. And that’s always a hassle if the machine is in use by real users during business hours so you end up having to work a night or weekend.

So, while I still like to have 4 partitions on heavy duty production servers (root, var, app-or-user-data, swap), I no longer bother with desktop, development, or light use production servers. I prefer the ultra simple 2 partition scheme of root and swap and since I started doing that 3 years ago I’ve had to do far fewer dump-repartition-restores.