Using log4j to monitor web application errors

When monitoring a production website (especially with a dozen or so application servers) you don’t want to rely on combining the logs and reviewing them manually for exceptions, you want the servers to notify you when there’s a problem. There are really 3 pieces to doing this properly and they are:

1. Configuring an SMTP appender to email you when an error is logged or using a custom appender to send a message to your monitoring server (e.g. NagiosAppender for Nagios)
2. Implementing a ServletFilter or other interceptor that logs Exceptions that would otherwise go to the top of the application server stack. You can do this directly with most application servers but I prefer using a ServletFilter because you have access to the HttpServletRequest so you can write out additional information such as the referrer, user-agent, remote ip, cookies, requested URL, etc… along with the stack trace.
3. Configuring error pages for 404’s and 500’s so your users never see a stack trace.

In this post we’ll just look at configuring the SMTP appender and then I’ll do a followup post with an example ServletFilter and configuring the error pages in Tomcat.

Configuring log4j to send email when an error is logged (or warning if you prefer) is easy. Just add mail to the log4j.rootCategory in log4j.properties and add the following to log4j.properties:

log4j.appender.mail=org.apache.log4j.net.SMTPAppender
log4j.appender.mail.To=app-errors@yourdomain.com
log4j.appender.mail.From=app-errors@yourdomain.com
log4j.appender.mail.SMTPHost=mail.yourdomain.com
log4j.appender.mail.Threshold=ERROR
log4j.appender.mail.BufferSize=1
log4j.appender.mail.Subject=Application Error
log4j.appender.Mail.layout=org.apache.log4j.PatternLayout
log4j.appender.Mail.layout.ConversionPattern=%d %-5p %c %x - %m%n

I prefer the non-XML properties version of log4j configuration but if you want the XML syntax for a log4j SMTP appender here it is.

Now when you log an error it will be emailed to you. It’s critical though that you tune the site to only log errors when there’s a real problem and use a group email alias that you can disable until you fix the issue if too many messages are coming in. Otherwise, people will add rules to their email clients to send the messages to a separate folder and just ignore them which defeats the purpose.

The most common mistake I’ve seen when consulting at other companies is that programmers catch unrecoverable exceptions in their application code, log them, and rethrow them resulting in duplicate messages. Or even worse, they catch them and display their own page specific error page instead of using a site wide configurable 500 error page offered by all application servers.

You should generally never have to catch and rethrow an unrecoverable exception in your application code unless you want to add a specific message for the generic 500 page to show! Unrecoverable exceptions SHOULD go to the top of the stack and that’s what the ServletFilter and configurable error pages are for that we’ll talk about in parts 2 and 3.

Related reading: This ONJava article describes configuring java.util.logging and log4j with smtp appenders, Matt Raible’s entry on using Log4j SMTP appenders

This entry was posted in Java, Software Engineering, Systems Administration. Bookmark the permalink.

3 Responses to Using log4j to monitor web application errors

  1. Mail appender is great if there are not much errors : but in general, things are worst than expected…
    Then you have to check logs and review them, day after day. I know that this is an exceptionnaly boring task, but it has to be done.

    But you can ease your life with tools : I wrote LogDistiller (http://logdistiller.sf.net/) exactly for this sort of need, because I was in this exact situation. Just take a look and see if it might help you too…

  2. JF Gauthier says:

    For your information, there is an other way to monitor web application errors.
    Download and install XLog-Solution. It’s available for many platform and it’s also available as a VM Appliance (trouble free).
    All your logs will be stored in the same database and accessible from your browser.
    You will get email alerts to scheduled people.
    If you use Java with log4j, you can use XLog4j appender to forward all (and I mean ALL) application logs to the server and control filters from the XLog-Dashboard live (without having to restart your application). So, if you get an error message you can go back to XLog-Dashboard, change logger filters and see what is going on inside all connected application.
    This product fetch logs from Apache HTTP service, syslog and many others.

  3. Mariano Ruiz says:

    If you want monitor at runtime the log4j log file with a browser, can use my open source tool: Log4j Web Tracker.
    It is a small module (.jar) that integrates in your application, and allow you to change at runtime the level of appenders, or view the log content in the browser.
    The site to download an read the documentation is http://www.log4jwebtracker.com

Leave a Reply

Your email address will not be published. Required fields are marked *