Zero downtime deploy script for Jetty

One of the challenges when developing Java web applications is to deploy new versions of the app without any perceptible downtime by the end users – in fact, this impacts virtually any platform, although in Java it could be trickier than for PHP or Rails, for example. The problem is that most servlet containers need to first shutdown the context in order to load it again, an operation that can take several seconds to complete (or, in a worst scenario, several minutes, depending of how your webapp is built).

When you have a cluster of servers serving the same app it may not be such a big problem, as one possible approach is to deploy the new version one box at a time. On the other hand, it is fairly common to have a single machine (despite its size) with a single webserver to do all the work, and there lies a monster.

In order to address such issue, I have created a bash script that does some tricks with Jetty and Apache configuration files that allows us to deploy a new version of the application and switch to it (as well switch back to the older version if necessary) with no noticeable downtime. Although it was created with the environment we have in production there where I work in mind, it is easy to adapt it to your needs (or vice-versa). The script assumes the following:

  • Jetty’s hot deploy feature should be disabled (basically, set “scanInterval” to 0 in jetty-contexts.xml)
  • Apache is in front of Jetty through mod_proxy
  • Your app is deployed as an open directory (e.g, not as a war), ideally using Capistrano or another similar tool
  • The ports 8080 and 8081 are available
  • The environment variable JETTY_HOME points to the Jetty installation directory
  • The environment variable APACHE_HOST_CONF points to the Apache configuration file for the host you are dealing (ideally not httpd.conf, but “example.conf”)
It works this way: you use the script “” as workhorse in place of the usual “”. To start a new instance, run “ start_new”, and the script will change the proper configuration files to listen on the “opposite” port (e.g, 8081 if 8080 is the current one, or vice-versa), start a new Jetty server and wait until the context fully starts. After that it will restart Apache, which will then proxy all requests to the new jetty server. If something goes wrong you can use “ rollback”, and if everything is OK, you can stop the previous and old instance by running “ stop_previous”. Simple as that.
The project is freely available at, and please make sure you read the instructions in the file “jetty_config”. In fact, it is advisable to either understand how “” does its job.

Fixing java.lang.OutOfMemoryError: PermGen space in Tomcat

If you have ever used Tomcat for development purposes for more than 10 minutes (especially within Eclipse), you certainly have encountered the infamous messag “OutOfMemoryError: PermGen space”, and the only solution was to restart Tomcat.

Despite the fact that you should be aware that such message may indicate a deeper problem, there is an easy’n dirty solution via JVM options. I have come to the fix after browsing the Interwebs for a while, and your luck may vary. The solution consists in setting up the following set of JVM arguments:


I have added them all for the sake of laziness, although the last three are said to be used with caution (especially the last one)

Setting up within Eclipse

Even in 2012 I still use the Sysdeo Tomcat Plugin because of its simpliciy over Eclipse’s WTP Servers. Open Window -> Preferences (Command + , if you are using a Mac) and go to Tomcat -> JVM Settings -> Append to JVM parameters.

Setting up Tomcat standalone

Another approach is to add the arguments directly to Locate it in Tomcat’s bin directory and set the JAVA_OPTS variable, like this:

JAVA_OPTS="-XX:+DisableExplicitGC -XX:MaxPermSize=256m -XX:PermSize=256m 
-XX:MaxNewSize=256m -XX:NewSize=256m -XX:+UseConcMarkSweepGC
-XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled"

Just restart Tomcat and you should be good to go.

Setting up JMX for JConsole / VisualVM on EC2, plus Jetty Configuration

Inspecting applications via JMX can be a very valuable tool in order to keep track of Memory, CPU and any other kind of information that are provided by the MBeans of the remote Java application. However, to get it working correctly can be tricky, specially if you haven’t play with it before, as the documentation is a little sparse.

Before going into the details, here are some very valuable links:

In order to get VisualVM or JConsole working with remote applications you will need to set some JVM properties on the application you want to track. In the list below are the ones that worked for me, but keep in mind that you may have different results depending on the application and the Java version (yes, it happens). In a nutshell, you’ll need to pass these properties to the JVM:<port><policy file>
 -Djava.rmi.server.hostname=<your public hostname><password_file>

“ requires the port number to listen for remote RMI connections. Make sure you specify an unused one.

java.rmi.server.hostname” is the public hostname (like of your server, without HTTP or anything else. You cannot use an internal address, otherwise you won’t be able to remotely access the service.” will require an username and password if set to true, otherwise anyone that knows (or discovers) the addres will be able to connect to your JMX registry.” is only necessary if you set authenticate=true, and you should specify the full path of a text file with the username and passwords (in plain text) that are allowed to connect to your instance. More information about the sintaxt at” is the path of a Java Policy File with the rules you need. For the sake of simplicity, you can create one with the following contents:

grant {

You may or may not need to set the policy file (I didn’t).

You don’t need all of these properties in order to the remote JMX working. As I said before, it will depend of your luck. In my very own case, I used only these:

You should pass these values when starting your java application, like

java \ [.....] -jar myapp.jar

Setting up EC2
The most important thing to set when running JMX on Amazon’s EC2 is to open the TCP ports to the IPs you will connect from, otherwise the client application won’t be able to reach the Agent. For the sake of simplicity, open all TCP ports from 0 to 65535, get it working and then keep open only the necessary ports. This approach is easier because the RMI registry that JMX uses (may?) create random ports of its communication, so you’ll have to inspect it (I am no expert on the subject, in other words).

On EC2 you configure the ports in the “Security Groups” section of the AWS Management Console.

Setting up Jetty
As a bonus, here’s how to get JMX working with Jetty. The process is fairly simple, it’s just a matter of adding the configuration keys to the right files.

First of all, you need to edit the file “start.ini” (located in Jetty’s root directory) and uncomment the line that has “etc/jetty-jmx.xml“, which is probably located at the end of the file, as shown in below:

Then, add each of the JVM properties in a different line in the same file (it doesn’t matter where). It should look like this:

Please note that “–exec” is a property specific to Jetty, and necessary to make it understand the other VM properties. “-Xshare:off” disables class data sharing, which is necessary to make VisualVM work correctly. The other settings are the ones described earlier.

The last step is to configure the file “etc/jetty-jmx.xml” to instruct it to automatically create a RMI registry and set the RMI connection properties, which are essential to remotely connect to it. My configuration file (based on Jetty 8) is below:

<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "">
<Configure id="Server" class="org.eclipse.jetty.server.Server">
  <Call id="MBeanServer" class="" name="getPlatformMBeanServer"/>
  <New id="MBeanContainer" class="org.eclipse.jetty.jmx.MBeanContainer">
      <Ref id="MBeanServer"/>
  <Get id="Container" name="container">
    <Call name="addEventListener">
        <Ref id="MBeanContainer"/>
  <Call name="addBean">
      <Ref id="MBeanContainer"/>
  <Get id="Logger" class="org.eclipse.jetty.util.log.Log" name="log"/>
  <Ref id="MBeanContainer">
    <Call name="addBean">
        <Ref id="Logger"/>
  <Call name="createRegistry" class="java.rmi.registry.LocateRegistry">
    <Arg type="java.lang.Integer">1099</Arg>
    <Call name="sleep" class="java.lang.Thread">
      <Arg type="java.lang.Integer">1000</Arg>
  <New id="ConnectorServer" class="org.eclipse.jetty.jmx.ConnectorServer">
      <New class="">
        <Arg type="java.lang.String">rmi</Arg>
        <Arg type="java.lang.String"></Arg>
        <Arg type="java.lang.Integer">0</Arg>
        <Arg type="java.lang.String">/jndi/rmi://</Arg>
    <Call name="start"/>

That’s all. When you start Jetty, it will log (to the console or file, depending of how you configured it) the JMX address you will need to use in order to connect to it from your local machine. Please see the following image:


In my case, the address that I should use in VisualVM is the address of my EC2 instance (

If the connection is successful, then you can right click on the remote host name and insert the address of the RMI registry, like shown below:

If you were lucky enough, you should be able to instrument Jetty remotely:

Changing the mouse cursor in Swing for the entire application

It is quite easy to change the mouse cursor in a Java Swing application, but most documentation and examples available only demonstrate how to change the cursor for a specific component. However, in an application I am developing – which has a fairly complex component hierarchy – I needed to ensure that the cursor would remain the one I set no matter where the mouse were (the most common example is the “busy / waiting” cursor, although that was not my case).

To change the cursor in a application wide fashion, you have to access the Glass Pane and set its cursor, and then make the glass pane visible – otherwise, it has no effect at all.

Take the following code as example:

int cursorType = Cursor.S_RESIZE_CURSOR;
Component glassPane = ((RootPaneContainer)someComponent.getTopLevelAncestor()).getGlassPane();
glassPane.setVisible(cursorType != Cursor.DEFAULT_CURSOR);

In the previous code, “someComponent” is any JComponent you have, it doesn’t matter which one (can be a JLabel, JButton, JPanel.. just take one). The trick is to access the RootPaneContainer, and then get the glass pane from it.

The last part is to make the glass pane visible only when the cursor is not the default one. That’s all.

The status of Swing in 2011

I always have been a Web developer in my career – well, most of the time, as since the iPad was released I have been doing some heavy development on iOS, specially in the field of digital readers. Nevertheless, is the Web platform that I am most familiar with. Being a solution centric professional, it was not surprise that last month I was requested to develop a new tool that could run on Mac and Windows desktops, in order to allow newspaper and magazine publishers to create rich content issues for mobile devices and possible web readers.

The company I work for was a long time user of Microsoft solutions (only because that when they started over 11 years ago, it was easier to find Action Script developers that could do some ASP coding, thus the migration to the .NET platform was some kind of natural for them), but for the past couple of years the transition to Rails and now Java is taking place over .NET (which is something to be discussed another time), and using Java for the new tool seemed the best way to go through, as Java is a very mature platform, with plenty of developers and documentation available, and not to have to maintain totally different base codes of Mac and Windows was a very good argument.

However, I didn’t have so many good opinions about Swing (the Java GUI Toolkit), even though it is the standard approach to do desktop programming with Java. Sure that there are other toolkits out there, like SWT and QT, but I had to consider that other people would be working on the software later, doing bug fixing and general improvements, thus if it required extra training it could be a problem.

All my experience with Swing was limited to some small programs or books I read, but it was enough to let me know that:
– I’d need to deal with layout managers
– That layout managers can get quite complex or verbose very easily (or both)
– That Swing didn’t have any major upgrade for years (if any)
– I had a clue that there weren’t so many third party components
– And definitely that I wasn’t willing to do hand coding of the GUIs. A GUI designer was vital.

I have mixed feelings about layout managers. I understand why they were created, but you had to know them all in order to make a fairly complex program, as each one of them accomplish a very specific task. Also, any documentation (including books) date back to 2004 or 2005 at most, which leads one to think that it was abandoned, but this feeling goes away after some time, as Swing has almost all (if not all) major components one may need to create any program, and it is possible to create your own components if needed. Swing is mature as well, and the fact it does not get a major upgrade in years may only means that there isn’t that many stuff to have.

Ok, I may being a little too optimistic here, as certainly the API could be improved a lot and more components could be available by default, but Sun was never good at designing easy to use APIs, far from that, and the failed Java FX thing only proves that the guys over there had no clue at all of what to do.

When searching for documentation, you often will find articles dated back to ’98, ’02. This frequently helps to keep alive that feeling that you are dealing with a dinosaur and using something that will die soon – even it will not (I hope). Sure that any example code that it’s not for Java 1.1 is very welcome when you know nothing.

The complexity of doing even simple things is something you will face constantly. The Java API was never known for its friendliness – it appears that very different teams at Sun, that did not talk to each other, were developing the base class library, thus creating a lot of inconsistency across the API -, but Swing goes a step further to make your life harder. Maybe it’s because I am more confortable with web programming, but I have done a lot of WinForms and Silverlight (.NET) coding, and by far Swing is harder. Take for example Drag and Drop over a JList or (worst) a JTree: it is insanely hard to make it work, even after reading hundreds of source code lines from open source projects and discussion forums. The event dispatching system is another thing that the more I read the less I understand, and many times I found myself doing some sort of implementation of the observer pattern instead of sticking with Swing’s own classes (I ended with a somewhat simple but very effective ‘EventCentral’ class).

As for third party components, there are some, but not many, not to mention those that does not work very well. Conversely, the WinForms API of .NET (here I go again, sorry) still get plenty of stuff today, although not as many as it did once, and you can’t stop thinking about it occasionally.

In conclusion, at the end of the day Swing it’s the only choice you have if you are going to use Java for desktop development, and albeit it has a lot of bugs and a weak, non-active community, it gets the job done. I have looked at SWT and JavaFX, but the former it still even more restricted, and the latter’s first release was a fiasco, but apparently Oracle is trying to respawn it with a 2.0 version.

Omit unexpected XML Elements with XStream

XStream is a great library to create XML from objects and vice-versa, and one of the many areas I use it is to store configurations. One problem is that XStream has the hability to ignore fields when serializing objects, but not the opposite – e.g., if it finds a tag that does not have a corresponding property in your class, it will throw an exception.

It is not clear to me why they still don’t have anything to handle such situation, which is fairly common to happen. For example, you may have a XML and only needs a small fraction of its data, or maybe you changed the way you store configurations and removed some properties. In both cases you will have to map the entire object graph, even if you don’t want it. Their FAQ page states this:

If a field is removed from the class, deserializing an old version that contains the field will cause an exception. Leaving the field in place but declaring it as transient will avoid the exception, but XStream will not try to deserialize it.

I find this behavior pretty odd and annoying, but luckily there is a workaround: the XStream class has a protected method named wrapMapper(MapperWrapper next) that subclasses may implement in order to tell the library if a given class or property should be considered. In this method you can add verifications for the fields or classes that you don’t want to bother, and return false for them.

Check out a working example:

XStream x = new XStream() {
	protected MapperWrapper wrapMapper(MapperWrapper next) {
		return new MapperWrapper(next) {
			public boolean shouldSerializeMember(Class definedIn, String fieldName) {
				if (fieldName.equals("shouldCopyWithProject")) {
					return false;

				return super.shouldSerializeMember(definedIn, fieldName);

In the previous example, I had a field named “shouldCopyWithProject” that was removed in newer versions of my application, but such refactoring resulted in crashes when opening the files from older versions of the app, so I had to handle the situation manually.

It would be great if XStream had a cleaner solution for this, but I think it is very unlikely to happen. Nevertheless, the approach here described works very well.


As Chris have pointed out in the comments, it is much easier to just use the omitField(class, fieldName) method, as in

// .....
XStream x = new XStream();
x.omitField(A.class, "shouldCopyWithProject");
// .....

I came across this method several times, but for some reason I thought it only worked for serialization. Anyway, thanks to Chris for simplifying our lives 🙂

Character encoding issues with Jetty 8

After deploying a webapp to the production server, all pages showed character encoding problems (aka, an accented letter would display “strange” data instead). In my development box it worked like a charm, but I couldn’t get it to work in the live server. I had all files saved as UTF-8, as well calls to  “<%@ page contentType=”text/html;charset=UTF-8″ pageEncoding=”UTF-8″ %>” in JSP files, “<page-encoding>UTF-8</page-encoding>” in WEB.xml etc…, but nevertheless, it simply didn’t work. I even used an encoding filter as suggested by a friend, without any luck.

What actually solved the problem (if I can call it that way) was a Jetty downgrade from version 8 to version 7. Simple as that. I don’t know what changed in version 8, and maybe it is just a matter of changing a setting somewhere, but until I find which one, sticking with Jetty 7 works great.

Correctly loading resources (images and files) in Java

To load a file (be it a text file or image) in Java is fairly easy, and most of the time all you need if the path to the resource. However, the filename approach does not work so well when your data is inside a JAR file. JAR files are basically ZIP with a different extension, and a regular filename don’t make sense when the resources are inside the jar.

This is a pretty common problem that many developers face when their app goes into production (including Swing apps), because when they are developing all files are exploded (e.g., not inside a jar), and thus loading images and files using paths like the example below work fine

// This will work when your app and resources
// are not contained inside a jar
String path = getClass().getResource("/path/to/image.jpg").getFile();
Image image = new ImageIcon(path).getImage();

Note the call to getFile(). However, getFile() will return invalid paths once you pack your application, resulting in errors.

In order to make it work right, you should use the getResource() method of Class (or getResourceAsStream()) without the call to getFile(), and instead use the URL or Stream directly. The previous example will then be converted to

// ImageIcon, like many others, can receive an URL as argument,
// which is a better approach to load resources that are
// contained in the classpath
URL url = getClass().getResource("/path/to/image.jpg");
Image image = new ImageIcon(url).getImage();

The very same approach works for files, but instead you’ll use getResourceAsStream().

Very useful reflection library for Java

Reflection in Java is a subject as old as Java itself, and likewise is the kind of thing that you can easily forget the details after some time without dealing with it. For example, to get a field from a given class, a call to getClass().getField(fieldName) will do the job, but only if the field is not private, which then you should use getDeclaredField() and call setAccessible(true). But again, only if such field is declared in the same class you are querying – e.g, it does not apply to the super class.

The list goes on and on.

There are many libraries out there, from the very low level to the high level. One good example of the latter is ReflectUtils, a very convenient and easy utility class.

Sample code:

    • Getting a value from an object field
Object value = ReflectUtils.getInstance().getFieldValue(someInstance, "fieldName");
    • Setting a value on an object field
TestEntity thing = new TestEntity();
ReflectUtils.getInstance().setFieldValue(thing, "entityId", 33);
// value of thing.getEntityId() will be "33", value is autoconverted into the right type
    • Setting a nested value on an object field
Object thing = new HashMap(); // using a hashmap for simplicity here, could easily be nested POJOs
ReflectUtils.getInstance().setFieldValue(thing, "", "aaronz");
// the value of the name field which is on the object in the contactInfo field which is on the object in the person field on the thing object is set to "aaronz"

The library is available at


Setting up the JAR file search location for the java command line tool

Before I forget it again: usually, Command-Line Java applications set up their classpath using the -classpath argument, which requires you to specify each single jar, among the root package directory where .class files are in. I find this method boring and slow to set up, specially when you have dozens of files.

A much better approach is to use the option -Djava.ext.dirs=<directory> instead, as in

java -Djava.ext.dirs=/path/to/all/jars

Just point to the root directory where all your jars are located. Way more useful and using the -classpath argument.