Todd Huss

This tongue in cheek post by Dion about Ruby not scaling made me chuckle but also got me thinking!

The whole question of whether a modern language scales or not is really the wrong question to ask. Almost any language will scale horizontally. Here’s one anecdote as an example: when we were going through the language selection process at GreatSchools several years back we rewrote some key pages in PHP, Java, and Perl (mod_perl) to try them out. Not surprisingly (more…)

I wrote a tongue in cheek humor piece on April fools day last year about ditching Java and switching to .BAT. Earlier this year it hit the front page of Reddit which was very exciting since my little server stood up to the pounding pretty well. Then more recently it hit the front page of Digg (but only briefly). Talk about two totally different experiences! (more…)

After listening to this weeks Ruby on Rails podcast where Geoffrey Grosenbach interviewed Bruce Tate, it got me to thinking about why Ruby on Rails appeals to me. For me as a Java person, the real appeal of Ruby lies in Rails and here’s why: (more…)

In our production environment at GreatSchools we have 3 production MySQL database servers: 1 read-write master and 2 read-only slaves. In moving our site from Perl to Java we need to load balance read-only connections across the read-only slave servers so that our read-write master doesn’t get overwhelmed under heavy load. As our systems administrators discovered there’s a solution to load balancing read-only connection in the MySQL JDBC driver!

For the JDBC driver you’ll need to use the com.mysql.jdbc.ReplicationDriver for the connection string, set a property roundRobinLoadBalance to true, and use jdbc:mysql://master,slave1,slave2/dbname as the connection string. Once you have a connection that you’ll be using for read-only purposes do a conn.setReadOnly(true) and you’re off and running!

I’ve also started using this approach in conjunction with Spring/Hibernate and so far it looks like it will work fine. We’re going to be doing some more extensive testing and load testing of this feature before we put it into production though!

Validating XHTML pages is an interesting subject because no high traffic site I’ve ever tried it on actually successfully validates XHTML against a W3C validator. Most folks I know take it for granted that it’s unrealistic to write large sites with 100% W3C validating XHTML. However, that presents a real problem on the software engineering side where we want to be able to rely on our development environment or continuous integration to tell us whether everything is truly working and valid! (more…)

Dions thought provoking blog entry on Refactoring tools are fantastic and overrated inspired me to think more in-depth about the deploy/redeploy nature of Java web development. He makes the point that while tools like IDEA have great refactoring, that other factors such as the pain of constantly redeploying in Java containers outweighs the benefits of these tools. I totally agree in theory, however, in practice I've found that the pain of redeploying has a very nice side benefit (at least for me).

Before I go any further let me add that we develop in Tomcat with class reloading enabled so when you change a class you just hit refresh in the browser. However, as Dion points out that doesn't solve non-trivial changes such as configuration changes which do require a Tomcat reload.

Anyhow, to my point, I personally like the pain of redeploying because it forces me to develop using TDD for just about everything up to and including the controller. Then I generally just make minor code tweaks and the HTML changes in a live Tomcat (where a simple browser refresh will give me the feedback). If a bigger code change is involved I go back to TDD to drive the controller or business logic until I get them working as desired.

On the other hand with Ruby and Perl I often find myself starting to get lazy by developing from a page centric point of view because it's much easier to just hit refresh in the browser. It's easy to get carried away by relying on the browser for the feedback, even as deeper business logic changes creep into the picture. When this happens I come back after an hour of coding only to realize I now have a bunch of out of date unit tests that need to be updated and the test coverage is never quite as good then.

Maybe it's just me, but since TDD is so much easier than constantly redeploying I actually practice better TDD under Java than I do in Ruby.

One of the tough decisions many people face regularly with DAOs and business facades is how best to test them. At work our current solution is that we let DAOs interact with the database which we've prepopulated with DBUnit and in many cases we let business facades interact with real DAOs. However, you could correctly argue that the test is no longer a unit test. Terminology aside though, the question remains, which approach is better for testing your DAO and business layer.

On the positive side, as long as your DAO and business facade tests can interact with the database it makes writing tests much easier and therefor I believe encourages developers to write more tests which is a good thing.

The downside though is that as your unit tests add up, running the full test suite can take a very long time. In terms of speed, having the majority of your JUnit tests be true “unit” tests that only test the unit of work you are concerned with is optimal. However, in cases where you have numerous or complex dependencies as is common in a business facade, your unit tests can get quite unruly and lengthy just to mock the dependencies.

Therefore the guidelines I work by are:

1. DAOs always get to interact with a real database prepopulated with DBUnit. I just don't have it in me to mock a Jdbc layer or Hibernate session and quite frankly I don't think it makes for as useful of a test when those layers are mocked.
2. The business layer should use a mocked DAO layer when it's not too dificult and reasonable. However, when dealing with a complex object graph or dependencies that would require undo amounts of mock setup, then I prefer to just use the real thing to keep the JUnit code simple and readable.

How do you approach testing your DAO and business layer?

Dion (whose blog I enjoy) wrote an interesting blog entry about forking a dependency in Java versus Ruby.

For example say you need to patch Hibernate or ActiveRecord with a fix that hasn't been released yet. Then you want to roll out that fixed/forked version to your team, then once the new release comes out with the patch, you want to switch the team back to the main release. Of course with a team you want to make this as painless as possible, ideally, without them having to do anything.

In Java (especially using Maven) I find it relatively simple to fork a jar or plugin with my own patches and have the team start using that transparently, without them having to know or do anything beyond a CVS update. Then when a new release of say Hibernate comes out with the fix in question I just update the dependency in project.xml and the whole team automatically switches over to using the standard distribution again. Many team members may not ever know or notice the switch even happened.

Now, with Ruby I'm not even sure how I would go about transparently getting the team to use a forked version of ActiveRecord without having to go around and ask each of them to do something. Or perhaps there's an easy way to do this with Ruby that I'm not familiar with?

One of our systems administrators at work setup our zero downtime deployment solution with Tomcat which so far has worked pretty well for us. We needed a zero downtime deployment solution because we strive to practice agile development and therefor launch a new release of our site every 2 weeks, plus we need the ability to launch a bug fix without taking the site down. Lastly, we were able to do zero downtime deploys with our old Perl/Modperl code base so it wouldn’t have helped our case to port from Perl to Java if we now had to incur downtime to launch a new release.

This is one area where Perl, Ruby, Pathon, et al shine because you can cut over to a new release in a matter of seconds by doing “mv website website.old; mv website.new website; cluster apachectl graceful”. You can also patch a file in the release branch and just move that file live without having to redploy the whole release. Lastly you can add debug log messages to a live class file just by editing it on the production site when the fit really hits the shan and you can’t reproduce the bug on your dev or staging environments. Java has a lot of other merits though such as excellent IDE’s with extermely powerful refactoring abilities, OR mapping, plethora of MVC frameworks, etc… which is why we chose it to replace Perl.

As it stands with our current setup we can launch a new release of the site with zero downtime unless incompatible database schema changes need to go live. In those cases we’ll schedule a maintenance window late at night to launch a release. Nobody likes staying up late at night though so I also view zero downtime deploys as advantageous for employee retention.

Here’s our current setup for zero downtime deploys:

1. We use replicated sessions in Tomcat so that the load balancer can bounce users around from one app server to another as it sees fit.
2. Our configuration is such that each physical server runs Apache, mod_jk (not mod_jk2), and a Tomcat instance.
3. Each mod_jk is configured to favor using the Tomcat on the local machine but can failover to a Tomcat running on another machine if the local instance gets shutdown.

To deploy a new WAR we have a script we call that we pass the path to the WAR file to. In the future we want to add functionality to this script to automatically SCP the WAR over from our staging/qa server, checksum it, and then deploy it. Here’s what the script currently does:

1. It SSHs to each application server sequentially (using public/private key pairs to authenticate without a password).
2. Shuts down Tomcat and then Apache’s mod_jk automatically fails over to another still runnning Tomcat on another machine.
3. Drops the new WAR into place and starts up Tomcat and then waits until Tomcat is again listening on port 8009. At that point mod_jk on the local server will start using the local Tomcat again.
5. Moves onto the next application server and repeats until it’s gone through all the app servers and our upgrade is complete.

Building on my most recent post, another objective we have at work is that each build process (which maps to one CVS module) generates one and only one artifact (e.g. a single WAR, plain old Jar, or a Javaapp Jar). There are certainly dependencies between modules but any deployable artifact that gets built (website.war, batchjobs.jar, xmlfeeds.jar, etc…) should be self-contained and include all of its dependencies such as Spring, Hibernate, etc.

This brings me to the Javapp Jar which I'm a huge fan of. A WAR is great because it can contain class files, JSP's, jar dependencies, and what have you. When you want to deploy it into production you have one self contained file that the release engineer can move live. A Javapp Jar is basically the same thing, it's a Jar file that contains your classes and files as well as all of those of your dependencies. For non Maven users you can also do this easily with ant using the zipgroupfileset task.

This makes the job of the release engineer much easier. For example rather than having to deploy batchjobs.jar, spring.jar, hibernate.jar and 10 other dependencies to our production batch processing server, instead we have a single batchjobs.jar that gets moved to production. That single Jar file contains our batch job classes, our bizlogic jar, spring, hibernate, and everything else in a single self contained file.