Tag Archives: nginx

Zero downtime deploy script for Jetty

Posted on by
Reply

One of the challenges when developing Java web applications is to deploy new versions of the app without any perceptible downtime by the end users – in fact, this impacts virtually any platform, although in Java it could be trickier than for PHP or Rails, for example. The problem is that most servlet containers need to first shutdown the context in order to load it again, an operation that can take several seconds to complete (or, in a worst scenario, several minutes, depending of how your webapp is built).

When you have a cluster of servers serving the same app it may not be such a big problem, as one possible approach is to deploy the new version one box at a time. On the other hand, it is fairly common to have a single machine (despite its size) with a single webserver to do all the work, and there lies a monster.

In order to address such issue, I have created a bash script that does some tricks with Jetty and Apache configuration files that allows us to deploy a new version of the application and switch to it (as well switch back to the older version if necessary) with no noticeable downtime. Although it was created with the environment we have in production there where I work in mind, it is easy to adapt it to your needs (or vice-versa). The script assumes the following:

  • Jetty’s hot deploy feature should be disabled (basically, set “scanInterval” to 0 in jetty-contexts.xml)
  • Apache is in front of Jetty through mod_proxy
  • Your app is deployed as an open directory (e.g, not as a war), ideally using Capistrano or another similar tool
  • The ports 8080 and 8081 are available
  • The environment variable JETTY_HOME points to the Jetty installation directory
  • The environment variable APACHE_HOST_CONF points to the Apache configuration file for the host you are dealing (ideally not httpd.conf, but “example.conf”)
It works this way: you use the script “jetty_deploy.sh” as workhorse in place of the usual “jetty.sh”. To start a new instance, run “jetty_deploy.sh start_new”, and the script will change the proper configuration files to listen on the “opposite” port (e.g, 8081 if 8080 is the current one, or vice-versa), start a new Jetty server and wait until the context fully starts. After that it will restart Apache, which will then proxy all requests to the new jetty server. If something goes wrong you can use “jetty_deploy.sh rollback”, and if everything is OK, you can stop the previous and old instance by running “jetty_deploy.sh stop_previous”. Simple as that.
The project is freely available at https://github.com/rafaelsteil/jetty-zero-downtime-deploy, and please make sure you read the instructions in the file “jetty_config”. In fact, it is advisable to either understand how “jetty_deploy.sh” does its job.

Handling the “open() 11 resource temporarily unavailable” error with Nginx

Posted on by

While developing a website, I started receiving the error “open() 11 resource temporarily unavailable” after we migrated from Apache to Nginx. The symptom was that, after any change to a CSS or JS file (in fact, to any file directly served by Nginx) the first request would crash, but any subsequent request did work. However, after performing any kind of change in one of those files, the problem happened again.

One important thing to notice is that I was developing through Samba – more specifically, I had the Nginx server running on a local Linux box, and the files were being manipulated in a text editor in Windows.

What I found out is that, if I edited the files directly in the Linux box, the problem didn’t happen. Going a little bit further, it was clear that changing to another text editor in Windows would not trigger the problem. After some Googleing, a post in the Nginx users list mentioned that this issue may arrive due to the fact that some text editors perform some kind of temporary lock at Samba in the files being edited (although I couldn’t find anything visually, at least), which caused Nginx to not being able to open the file even for reading.

So, if you have an environment like mine and are running into this issue, try swapping to another editor. You may have luck.