Zero downtime deploy script for Jetty

One of the challenges when developing Java web applications is to deploy new versions of the app without any perceptible downtime by the end users – in fact, this impacts virtually any platform, although in Java it could be trickier than for PHP or Rails, for example. The problem is that most servlet containers need to first shutdown the context in order to load it again, an operation that can take several seconds to complete (or, in a worst scenario, several minutes, depending of how your webapp is built).

When you have a cluster of servers serving the same app it may not be such a big problem, as one possible approach is to deploy the new version one box at a time. On the other hand, it is fairly common to have a single machine (despite its size) with a single webserver to do all the work, and there lies a monster.

In order to address such issue, I have created a bash script that does some tricks with Jetty and Apache configuration files that allows us to deploy a new version of the application and switch to it (as well switch back to the older version if necessary) with no noticeable downtime. Although it was created with the environment we have in production there where I work in mind, it is easy to adapt it to your needs (or vice-versa). The script assumes the following:

  • Jetty’s hot deploy feature should be disabled (basically, set “scanInterval” to 0 in jetty-contexts.xml)
  • Apache is in front of Jetty through mod_proxy
  • Your app is deployed as an open directory (e.g, not as a war), ideally using Capistrano or another similar tool
  • The ports 8080 and 8081 are available
  • The environment variable JETTY_HOME points to the Jetty installation directory
  • The environment variable APACHE_HOST_CONF points to the Apache configuration file for the host you are dealing (ideally not httpd.conf, but “example.conf”)
It works this way: you use the script “” as workhorse in place of the usual “”. To start a new instance, run “ start_new”, and the script will change the proper configuration files to listen on the “opposite” port (e.g, 8081 if 8080 is the current one, or vice-versa), start a new Jetty server and wait until the context fully starts. After that it will restart Apache, which will then proxy all requests to the new jetty server. If something goes wrong you can use “ rollback”, and if everything is OK, you can stop the previous and old instance by running “ stop_previous”. Simple as that.
The project is freely available at, and please make sure you read the instructions in the file “jetty_config”. In fact, it is advisable to either understand how “” does its job.

Setting up JMX for JConsole / VisualVM on EC2, plus Jetty Configuration

Inspecting applications via JMX can be a very valuable tool in order to keep track of Memory, CPU and any other kind of information that are provided by the MBeans of the remote Java application. However, to get it working correctly can be tricky, specially if you haven’t play with it before, as the documentation is a little sparse.

Before going into the details, here are some very valuable links:

In order to get VisualVM or JConsole working with remote applications you will need to set some JVM properties on the application you want to track. In the list below are the ones that worked for me, but keep in mind that you may have different results depending on the application and the Java version (yes, it happens). In a nutshell, you’ll need to pass these properties to the JVM:<port><policy file>
 -Djava.rmi.server.hostname=<your public hostname><password_file>

“ requires the port number to listen for remote RMI connections. Make sure you specify an unused one.

java.rmi.server.hostname” is the public hostname (like of your server, without HTTP or anything else. You cannot use an internal address, otherwise you won’t be able to remotely access the service.” will require an username and password if set to true, otherwise anyone that knows (or discovers) the addres will be able to connect to your JMX registry.” is only necessary if you set authenticate=true, and you should specify the full path of a text file with the username and passwords (in plain text) that are allowed to connect to your instance. More information about the sintaxt at” is the path of a Java Policy File with the rules you need. For the sake of simplicity, you can create one with the following contents:

grant {

You may or may not need to set the policy file (I didn’t).

You don’t need all of these properties in order to the remote JMX working. As I said before, it will depend of your luck. In my very own case, I used only these:

You should pass these values when starting your java application, like

java \ [.....] -jar myapp.jar

Setting up EC2
The most important thing to set when running JMX on Amazon’s EC2 is to open the TCP ports to the IPs you will connect from, otherwise the client application won’t be able to reach the Agent. For the sake of simplicity, open all TCP ports from 0 to 65535, get it working and then keep open only the necessary ports. This approach is easier because the RMI registry that JMX uses (may?) create random ports of its communication, so you’ll have to inspect it (I am no expert on the subject, in other words).

On EC2 you configure the ports in the “Security Groups” section of the AWS Management Console.

Setting up Jetty
As a bonus, here’s how to get JMX working with Jetty. The process is fairly simple, it’s just a matter of adding the configuration keys to the right files.

First of all, you need to edit the file “start.ini” (located in Jetty’s root directory) and uncomment the line that has “etc/jetty-jmx.xml“, which is probably located at the end of the file, as shown in below:

Then, add each of the JVM properties in a different line in the same file (it doesn’t matter where). It should look like this:

Please note that “–exec” is a property specific to Jetty, and necessary to make it understand the other VM properties. “-Xshare:off” disables class data sharing, which is necessary to make VisualVM work correctly. The other settings are the ones described earlier.

The last step is to configure the file “etc/jetty-jmx.xml” to instruct it to automatically create a RMI registry and set the RMI connection properties, which are essential to remotely connect to it. My configuration file (based on Jetty 8) is below:

<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "">
<Configure id="Server" class="org.eclipse.jetty.server.Server">
  <Call id="MBeanServer" class="" name="getPlatformMBeanServer"/>
  <New id="MBeanContainer" class="org.eclipse.jetty.jmx.MBeanContainer">
      <Ref id="MBeanServer"/>
  <Get id="Container" name="container">
    <Call name="addEventListener">
        <Ref id="MBeanContainer"/>
  <Call name="addBean">
      <Ref id="MBeanContainer"/>
  <Get id="Logger" class="org.eclipse.jetty.util.log.Log" name="log"/>
  <Ref id="MBeanContainer">
    <Call name="addBean">
        <Ref id="Logger"/>
  <Call name="createRegistry" class="java.rmi.registry.LocateRegistry">
    <Arg type="java.lang.Integer">1099</Arg>
    <Call name="sleep" class="java.lang.Thread">
      <Arg type="java.lang.Integer">1000</Arg>
  <New id="ConnectorServer" class="org.eclipse.jetty.jmx.ConnectorServer">
      <New class="">
        <Arg type="java.lang.String">rmi</Arg>
        <Arg type="java.lang.String"></Arg>
        <Arg type="java.lang.Integer">0</Arg>
        <Arg type="java.lang.String">/jndi/rmi://</Arg>
    <Call name="start"/>

That’s all. When you start Jetty, it will log (to the console or file, depending of how you configured it) the JMX address you will need to use in order to connect to it from your local machine. Please see the following image:


In my case, the address that I should use in VisualVM is the address of my EC2 instance (

If the connection is successful, then you can right click on the remote host name and insert the address of the RMI registry, like shown below:

If you were lucky enough, you should be able to instrument Jetty remotely:

Character encoding issues with Jetty 8

After deploying a webapp to the production server, all pages showed character encoding problems (aka, an accented letter would display “strange” data instead). In my development box it worked like a charm, but I couldn’t get it to work in the live server. I had all files saved as UTF-8, as well calls to  ”<%@ page contentType=”text/html;charset=UTF-8″ pageEncoding=”UTF-8″ %>” in JSP files, “<page-encoding>UTF-8</page-encoding>” in WEB.xml etc…, but nevertheless, it simply didn’t work. I even used an encoding filter as suggested by a friend, without any luck.

What actually solved the problem (if I can call it that way) was a Jetty downgrade from version 8 to version 7. Simple as that. I don’t know what changed in version 8, and maybe it is just a matter of changing a setting somewhere, but until I find which one, sticking with Jetty 7 works great.