To that end one of the tools I want to give a shout out to is the MySQL Performance Tuning Primer Script. You download and run it against a production system (that has preferably been running under normal load for a day or two so that it’s gathered stats). It’s a read-only script so you don’t need to worry about it changing anything but it makes some great recommendations about which tuning parameters may need adjustment. Here’s a snippet from from a production server:

KEY BUFFER
Current MyISAM index space = 173 M
Current key_buffer_size = 1 G
Key cache miss rate is 1 : 1003
Key buffer fill ratio = 7.00 %
Your key_buffer_size seems to be too high.
Perhaps you can use these resources elsewhere
…
…
…

It’s no replacement for a DBA but if you want to get a somewhat sane my.cnf going for your particular application it’s a great place to start!

Update 7/8/2009: With the latest batch of contributed patches standalone migrations now works just like Rails migrations
Update 12/26/2008: I switched standalone migrations to use a Rakefile instead of a Ruby script.

In my work managing websites I end up working in Ruby, Java, and PHP. In everything but Rails managing the schema requires rolling your own solution. As a result I’ve started using Rails migrations in non-Rails projects to manage the schema. It’s not much code but I figured others might benefit from it so I created a little Github project called standalone migrations.

It’s based on Lincoln Stoll’s blog post titled Stand-alone ActiveRecord migrations and David Welton’s blog post titled Using Migrations Outside of Rails.

Assuming you have Ruby and Gem installed on your machine, here’s how to use it:

gem install -y activerecord rake mysql
wget http://github.com/thuss/standalone-migrations/zipball/master (or fetch it using git)
unzip it, and mv to something like my_non_rails_project/db
cd my_non_rails_project/db/ (or wherever you put it)
cp config/database_sample.yml config/database.yml
vi config/database.yml
./new_migration some_user_story
vi migrations/*_some_user_story.rb
rake db:migrate (this applies your newly created migration)

This makes it totally inadequate for running a production database server. On the forums people have argued for using replication or clustering solutions while others have advocated streaming redo logs to Amazon’s S3 service. However, let’s look at the disaster recovery scenario if all your virtual machines go down (and don’t think it can’t happen, racks overload their circuits, generators fail during a power outages, AC’s can’t keep up during heat-waves, etc…):

- With a VPS or dedicated server you startup your servers, do a data integrity check, and you’re up and running again!
- With EC2 you restart the database server image, then restore the full database backup from the previous night (which you’ve been backing up to S3), then you rerun all of the redo logs (which you’ve also been backing up to S3).

The latter scenario isn’t that bad with a small database but for with a large database it could add hours to your recovery time. I imagine Amazon will eventually add permanent storage to EC2 and if/when that happens you’d be able to compare Amazon EC2 to a VPS but right now it’s comparing apples and oranges!

I had one minor problem and it was that we connect to multiple databases so we have multiple session factories that interact with EhCache. Hibernates org.hibernate.cache.EhCacheProvider now expects one configuration per sesssion factory which I wasn’t too excited about given we have 51 databases (1 database per state plus Washington DC) and it’s been working fine for 2 years with one cache provider. The problem caused this exception:

 Invocation of init method failed; nested exception is java.lang.NullPointerException
    [java] Caused by: org.springframework.beans.factory.BeanCreationException:
       Error creating bean with name 'surveysSessionFactory' defined in class path resource
          [gs/data/dao/hibernate/applicationContext-hibernate.xml]:
          Invocation of init method failed;
          nested exception is java.lang.NullPointerException
    [java] Caused by: java.lang.NullPointerException
    [java] at net.sf.ehcache.CacheManager.detectAndFixDiskStorePathConflict(CacheManager.java:269)
    ...

However, I wasn’t the only person who had this problem so a quick search turned up that EhCache now offers the net.sf.ehcache.hibernate.SingletonEhCacheProvider and switching the hibernate.cache.provider_class property to this new provider solved the problem and allows us to continue using a single cache provider.

whereas this one would:

mysql -uuser -ppass dbname
> insert into tablename values (1, 2, 3, 4)”

so we ended up with inconsistent data between the master and slave servers. Turns out this only happens if you don’t specify a default database (either on the command line or via a use dbname statement) AND you have a binlog-ignore-db statement in your my.cnf such as:

binlog-ignore-db=mysql

The logic for determining whether to replicate an SQL statement can be found here in the MySQL online manual. Personally I lean more on the side of this being a bug than a feature because the logic for determining what gets written to the binlog being based on the default database is flawed if you can override the database name in an SQL statement. In any case we’re going to be removing the binlog-ignore-db from our my.cnf files, however, not everyone can do that as easily if you want to have different permissions in mysql.user between master and slave databases.

1. Using your database’s custom enumeration data-type if it has one
2. Creating an int or char column and let the mapping live in the code
3. Creating a lookup table

I generally lean towards the lookup table because:

1. It’s highly portable
2. It keeps your database self documenting
3. It works well with any OR mapping layer
4. It allows you to add new types via an administrative interface or SQL statement without altering the schema

The only downside is performance which usually makes caching of enumerated type lookup tables a necessity!

In Java/Hibernate land I handle the performance issue with a 2nd level cache such as ehCache. When working in Ruby on Rails I’ve started using the Acts as Enumerated plugin written by Trevor Squires‘ which handles the caching for you and is simple and elegant to use.

1. Use a MySQL or PostgreSQL dump that gets imported before the tests are run. I’ve never seen this approach used with Oracle, most likely because it’s such a PITA to export to text (or at least it used to be)
2. Just create the schema before the tests and then populate the test data in code
3. Use a database test fixture tool such as the XML based DBUnit for Java or Ruby’s YAML based fixtures

I have a strong preference for database test fixture tools because they work across databases (often a requirement at ISV’s) and generally offers tools to easily export fixtures in addition to importing them. However, the ability to export fixtures from your database to a fixture file is sometimes overlooked but it’s vitally important as populating those fixtures gets really time consuming if you need to type it all in. The most common use case I run into is when I pull rows of data from production into my local database and once I’m happy with the sample data want to export it into a fixture file.

At GreatSchools we’re a Java shop and we use DBUnit which does a fantastic job of exporting and importing database test fixtures. Now that I’m using Ruby on Rails on outside projects though I was starting to burn out having to manually write the fixtures.yml files by hand because Rails doesn’t offer an export facility out of the box. In this particular case I was parsing NOAA weather forecasts which I do using script/runner via cron to retrieve over HTTP and populate the database with. They can be fairly lengthy and I needed dozens of records for the tests and the thought of having to write the YAML fixtures file had me scared!

Sure enough after a little web searching I found Chris McGrath and Nathaniel Brown’s export fixtures plugin and I’m a happy camper again!

Here’s how you install it:

Then to use it I populate the tables in question with sample data and then run:

It’s a little shy on giving you feedback but once it completes you’ll find table1.yml and table2.yml in the test/fixtures directory with all of the data from your tables in it. They also have other options to export data from another database such as production or export data selectively based on a query.

If you need to move big tables live you basically have 2 options:

1. Schedule downtime: boo!
2. Import the new data into a table with an alternate name, then drop the old table and rename the new table to the old. The latter part of the operation is extremely quick under MySQL and we do it all the time while serving over a thousand page views a minute with nary a 500 error to show.

The standard version of mysqldump doesn’t offer an option to dump a table and import it into the new database under a temporary name and then quickly replace the old table. However, JDD (GreatSchools former sysadmin extraordinaire who now works for MySQL) submitted a patch a long while back that does exactly this. The patch offers a -R option such that when it writes out the dump file it imports the data into a table with a temporary name and then it does the almost instantaneous operation of dropping the old table and renaming the new table to replace it.

The patch currently only works on MySQL 4.0 but we have an in-house version we’ve ported to MySQL 5.0 (which I can post here once we’ve determined it’s stable). Now that we’re used to being able to move gigabytes of data live during production hours, I can’t imagine going back to having to regularly work late nights with scheduled downtime just to get new data live!

For the JDBC driver you’ll need to use the

for the connection string, set a property

to true, and use

as the connection string. Once you have a connection that you’ll be using for read-only purposes do a

and you’re off and running!

I’ve also started using this approach in conjunction with Spring/Hibernate and so far it looks like it will work fine. We’re going to be doing some more extensive testing and load testing of this feature before we put it into production though!

Postgres is one of the few databases I’m aware of that currently solves the problem. You can see an example at Joseph Scott’s Blog of using a regex in a check constraint.

Sure, with Active Record in Ruby I can easily do this in code or in Java using the Hibernate validator but it’s a different story when you’ve got a database with multiple legacy applications and data loading systems running against it. In those cases the database if the final line of defense!

This is just a rant but I’m shocked that this situation hasn’t markedly improved over the past 10 years!