Profiling a JVM in EC2 with VisualVM

Every so often I find myself needing to connect VisualVM, a JVM profiling and debugging tool, to a JVM running remotely in EC2. Despite a few attempts to get it working using a SOCKS proxy following one of the many guides available, I’ve never managed to crack it and always resorted to temporarily opening all ports from within our VPN. It’s generally quite a frustrating process, so I decided to spend a few minutes figuring it out once and for all. After all, when you’re knee-deep working out what’s going awry with a production process, the last thing you want to have to do is fiddle around with ports and JVM args. Thankfully, that paid off and we’ve now distilled it into an option on our JVM startup scripts to do the business.

guess that port

VisualVM uses RMI to connect to the remote JVM, which has some challenges if trying to connect through a firewall. If it’s been as long for you as it has for me since last using RMI, allow me to very briefly recap on how ports are used. The remote JVM has things running on two ports: one for the RMI registry, from where RMI objects are discovered, and one for actually connecting to said objects. The latter generally runs on an arbitrary high port number, whose value clients get told by the registry.

The RMI handshake goes something like this:

• Client to RMI registry: “I’d like an object of type x please”
• RMI registry: “Sure, here. You’ll want to communicate with it at hostname x and port n”
• Client [to host x on port n]: “Hello?”

Now we could, of course, continue doing what I’ve resorted to in the past and open all ports on the server. This has obvious security concerns—somewhat mitigated by restricting to our VPN in our case—but is also a massive inconvenience in having to both do it and, more importantly, remembering to undo it when we’re finished. Much better to use SSH port forwarding if we could.

To do this, we need to forward both the registry and object ports to the remote host. That was a problem until relatively recently, since the latter port could not be fixed. However, recent Java 7 versions allow you to fix that port through a JVM arg, com.sun.management.rmi.port.

That sorted, the next problem is the hostname. Remember, this needs to be the hostname that the client uses to connect to the server. If we’re to tunnel this over SSH it’s therefore the local network interface. Another JVM arg to the rescue, this time java.rmi.server.hostname. Setting that to 127.0.0.1 does the trick.

flying the flags

After all that, all we need to do is add the following flags to the JVM command-line on the remote server we want to debug:

and create an SSH tunnel from our local machine to the server:

Et voila! We can add a new local JMX connection on the port we’re forwarding to the server by adding a new local JMX connection to localhost:7091 :

As I said, we’ve wrapped this all up as a command-line argument to our JVM startup script which now looks something like:

If you’ve had more luck with SOCKS proxies than me, I would love to hear how you cracked it as I tried again while writing this post, still with no joy!