First8 staat voor vakmanschap. Al onze collega’s zijn een groot aanhanger van Open Source en in het bijzonder het Java-platform. Wij zijn gespecialiseerd in het pragmatisch ontwikkelen van bedrijfskritische Java toepassingen waarbij integratie van systemen, hoge eisen aan beveiliging en veel transacties een belangrijke rol spelen. Op deze pagina vind je onze blogs.

Profiling remote JVM using VisualVM

Profiling is essential when you want to improve the performance of an application. To properly profile, you should minimize the differences between an actual production situation and the situation you are profiling.

This means that your test data should be the same and the hardware should be as similar as possible. The best possible scenario is to actually profile the production environment. You can run most profiling tools (e.g. VisualVM) on your local development machine and have it create a connection to the production environment.

To do so, you will need to expose the JVM somehow so you can reach it from your local machine. There are two ways of doing it:

  • by starting your application with some special JMX properties,
  • or by running jstatd on that machine.

If you can modify the start up scripts of your application and actually restart your application, the JMX option is there. However, in my experience, it is often quite difficult to modify start-up scripts as they are almost always managed by a different team. And, repeatedly restarting the application for the new parameters to take effect is not feasible.

Another route is running jstatd. This standard java application runs an rmi registry and allows VisualVM to connect to it. In theory it is just two simple steps:

  1. On the remote machine, start jstatd
  2. On your dev machine, start VisualVM and add a new remote host

On some operating systems (e.g. Ubuntu Linux) jstatd won’t start up but instead throws an exception like:

Could not create remote object
access denied ("java.util.PropertyPermission" "java.rmi.server.ignoreSubClasses" "write")
java.security.AccessControlException: access denied ("java.util.PropertyPermission" "java.rmi.server.ignoreSubClasses" "write")

This can be fixed (as described by the man page of jstatd) by creating a security policy file, giving jstatd all access to everything. Needless to say, if this is your production environment, you might want to be a bit careful and make sure it is not accessible for the whole world. Create a jstatd.all.policy file with the following content:

grant codebase "file:/opt/java/jdk1.7.0_21/lib/tools.jar" {
permission java.security.AllPermission;
};

Note that in my experience relative paths to tools.jar will not work, hence the hardcoded path to the running jvm.

You should then start jstatd using this policy file:

jstatd -J-Djava.security.policy=/home/user/jstatd.all.policy

If VisualVM doesn’t list the remote jvm’s, here are some tips in getting it to work:

  • Firewall issues: can you telnet from your dev machine to the jstatd port (default 1099)? If not, you’ll probably need to create ssh tunnels. Don’t forget that jstatd opens a second (randomly assigned) port as well which you’ll need to forward.
  • Binding issues: sometimes jstatd doesn’t bind to the correct ip address. You can force binding it to a specific ip using something like -J-Djava.rmi.server.hostname=10.1.1.123.
  • IPv6 problems: On rare occassions it can help to force jstatd and/or VisualVM to use IPv4: -J-Djava.net.preferIPv4Stack=true
  • Enable logging: If the steps above don’t work, you could try to enable some extra logging to jstatd to see if you can trace the problem: -J-Djava.rmi.server.logCalls=true

Some references: