To that end one of the tools I want to give a shout out to is the MySQL Performance Tuning Primer Script. You download and run it against a production system (that has preferably been running under normal load for a day or two so that it’s gathered stats). It’s a read-only script so you don’t need to worry about it changing anything but it makes some great recommendations about which tuning parameters may need adjustment. Here’s a snippet from from a production server:

KEY BUFFER
Current MyISAM index space = 173 M
Current key_buffer_size = 1 G
Key cache miss rate is 1 : 1003
Key buffer fill ratio = 7.00 %
Your key_buffer_size seems to be too high.
Perhaps you can use these resources elsewhere
…
…
…

It’s no replacement for a DBA but if you want to get a somewhat sane my.cnf going for your particular application it’s a great place to start!

Update 7/8/2009: With the latest batch of contributed patches standalone migrations now works just like Rails migrations
Update 12/26/2008: I switched standalone migrations to use a Rakefile instead of a Ruby script.

In my work managing websites I end up working in Ruby, Java, and PHP. In everything but Rails managing the schema requires rolling your own solution. As a result I’ve started using Rails migrations in non-Rails projects to manage the schema. It’s not much code but I figured others might benefit from it so I created a little Github project called standalone migrations.

It’s based on Lincoln Stoll’s blog post titled Stand-alone ActiveRecord migrations and David Welton’s blog post titled Using Migrations Outside of Rails.

Assuming you have Ruby and Gem installed on your machine, here’s how to use it:

gem install -y activerecord rake mysql
wget http://github.com/thuss/standalone-migrations/zipball/master (or fetch it using git)
unzip it, and mv to something like my_non_rails_project/db
cd my_non_rails_project/db/ (or wherever you put it)
cp config/database_sample.yml config/database.yml
vi config/database.yml
./new_migration some_user_story
vi migrations/*_some_user_story.rb
rake db:migrate (this applies your newly created migration)

This makes it totally inadequate for running a production database server. On the forums people have argued for using replication or clustering solutions while others have advocated streaming redo logs to Amazon’s S3 service. However, let’s look at the disaster recovery scenario if all your virtual machines go down (and don’t think it can’t happen, racks overload their circuits, generators fail during a power outages, AC’s can’t keep up during heat-waves, etc…):

- With a VPS or dedicated server you startup your servers, do a data integrity check, and you’re up and running again!
- With EC2 you restart the database server image, then restore the full database backup from the previous night (which you’ve been backing up to S3), then you rerun all of the redo logs (which you’ve also been backing up to S3).

The latter scenario isn’t that bad with a small database but for with a large database it could add hours to your recovery time. I imagine Amazon will eventually add permanent storage to EC2 and if/when that happens you’d be able to compare Amazon EC2 to a VPS but right now it’s comparing apples and oranges!

whereas this one would:

mysql -uuser -ppass dbname
> insert into tablename values (1, 2, 3, 4)”

so we ended up with inconsistent data between the master and slave servers. Turns out this only happens if you don’t specify a default database (either on the command line or via a use dbname statement) AND you have a binlog-ignore-db statement in your my.cnf such as:

binlog-ignore-db=mysql

The logic for determining whether to replicate an SQL statement can be found here in the MySQL online manual. Personally I lean more on the side of this being a bug than a feature because the logic for determining what gets written to the binlog being based on the default database is flawed if you can override the database name in an SQL statement. In any case we’re going to be removing the binlog-ignore-db from our my.cnf files, however, not everyone can do that as easily if you want to have different permissions in mysql.user between master and slave databases.

Today I had to get an Excel document that was loaded with ODBC database queries that connect to multiple MySQL databases working on a new machine. The problem I ran into is that if there are any MySQL permission issues such as a missing grant, authentication problems, etc… you’ll always get the following completely meaningless error message:

[MySQL][ODBC 3.51 Driver]Could not find driver {MySQL ODBC 3.51 Driver} in system information.

For the life of me I couldn’t even figure out how to determine what host/database/user/password a particular query was trying to use in Excel because if you click on Edit Query it would tell you it can’t find the driver. It’s hard to troubleshoot when you don’t even know which machine a particular query is trying to connect and it won’t even show you the SQL for the query if it can’t connect!

So to finally figure out what ODBC connections this Excel document was using I scp’d it over to a Unix machine and used strings:

strings document.xls | grep DRIVER | sort | uniq

which will give you a list something like this:

DRIVER={MySQL ODBC 3.51 Driver};DESC=;DATABASE=dbname;SERVER=hostname;UID=user;PASSWORD=password;PORT=0;OPTION=0;STMT=;
DRIVER={MySQL ODBC 3.51 Driver};DESC=;DATABASE=dbname2;SERVER=hostname;UID=user;PASSWORD=password;PORT=0;OPTION=0;STMT=;
DRIVER={MySQL ODBC 3.51 Driver};DESC=;DATABASE=dbname3;SERVER=hostname;UID=user;PASSWORD=password;PORT=0;OPTION=0;STMT=;

That allowed me to see all of the database connections the Excel document was using so I could go in and fix the permission issues. If anyone knows an easier way, let me know.

# Set MySQL to use a strict sql mode for all connections
class ActiveRecord::ConnectionAdapters::MysqlAdapter
  alias :connect_no_sql_mode :connect
  def connect
    connect_no_sql_mode
    execute("SET sql_mode = 'ansi,strict_trans_tables'")
  end
end

What really appeals to me about this approach is that the application configures MySQL so that no matter where it runs ( on my servers, on my laptop, on another developers laptop, or on a shared server where you can’t change the global sql-mode), that MySQL will behave consistently without needing to configure each database installation. You could also use this approach to implement David Wheeler’s recommended character set and timezone settings.

To make matters worse it seems MySQL may have different out of the box settings per platform and per version. For example a friends 5.0.21 install on Windows XP had the sql-mode set with relatively strict settings whereas 5.0.18 on my Mac had an empty sql-mode. This only illustrates the necessity of always explicitly defining these settings with each install yourself so that you get consistent behavior.

If you’re configuring MySQL for a new greenfield project there are 2 things that you should absolutely always do in your my.cnf (or my.ini in Windows):

1. Use InnoDB for your tables so that foreign keys will work and you’ll get row level locking instead of MyISAM’s table level locking: default-storage-engine=InnoDB
2. Use a strict SQL mode that insures if you try and insert an invalid value or null into a non-nullabel field that you get an error: sql-mode=ansi,strict_trans_tables.

Setting the character set to utf8 and timezone to utc is also a smart thing as recommended here by David Wheeler.

Where it gets tricky is when you work on multiple projects on your laptop where some legacy code depends on MySQL’s old settings. I handle this by leaving the sql-mode as ansi/strict and then downgrading when I need to by executing the command set @@global.sql_mode=”; and then when I’m done going back with set @@global.sql_mode=’ansi,strict_trans_tables’.

1. Use a MySQL or PostgreSQL dump that gets imported before the tests are run. I’ve never seen this approach used with Oracle, most likely because it’s such a PITA to export to text (or at least it used to be)
2. Just create the schema before the tests and then populate the test data in code
3. Use a database test fixture tool such as the XML based DBUnit for Java or Ruby’s YAML based fixtures

I have a strong preference for database test fixture tools because they work across databases (often a requirement at ISV’s) and generally offers tools to easily export fixtures in addition to importing them. However, the ability to export fixtures from your database to a fixture file is sometimes overlooked but it’s vitally important as populating those fixtures gets really time consuming if you need to type it all in. The most common use case I run into is when I pull rows of data from production into my local database and once I’m happy with the sample data want to export it into a fixture file.

At GreatSchools we’re a Java shop and we use DBUnit which does a fantastic job of exporting and importing database test fixtures. Now that I’m using Ruby on Rails on outside projects though I was starting to burn out having to manually write the fixtures.yml files by hand because Rails doesn’t offer an export facility out of the box. In this particular case I was parsing NOAA weather forecasts which I do using script/runner via cron to retrieve over HTTP and populate the database with. They can be fairly lengthy and I needed dozens of records for the tests and the thought of having to write the YAML fixtures file had me scared!

Sure enough after a little web searching I found Chris McGrath and Nathaniel Brown’s export fixtures plugin and I’m a happy camper again!

Here’s how you install it:

Then to use it I populate the tables in question with sample data and then run:

It’s a little shy on giving you feedback but once it completes you’ll find table1.yml and table2.yml in the test/fixtures directory with all of the data from your tables in it. They also have other options to export data from another database such as production or export data selectively based on a query.

If you need to move big tables live you basically have 2 options:

1. Schedule downtime: boo!
2. Import the new data into a table with an alternate name, then drop the old table and rename the new table to the old. The latter part of the operation is extremely quick under MySQL and we do it all the time while serving over a thousand page views a minute with nary a 500 error to show.

The standard version of mysqldump doesn’t offer an option to dump a table and import it into the new database under a temporary name and then quickly replace the old table. However, JDD (GreatSchools former sysadmin extraordinaire who now works for MySQL) submitted a patch a long while back that does exactly this. The patch offers a -R option such that when it writes out the dump file it imports the data into a table with a temporary name and then it does the almost instantaneous operation of dropping the old table and renaming the new table to replace it.

The patch currently only works on MySQL 4.0 but we have an in-house version we’ve ported to MySQL 5.0 (which I can post here once we’ve determined it’s stable). Now that we’re used to being able to move gigabytes of data live during production hours, I can’t imagine going back to having to regularly work late nights with scheduled downtime just to get new data live!