JMX Secure Connections / Avoiding Java System Properties, RMI

I spent much of my weekend working on adding support for Java Management Extensions (JMX) into a large enterprise application. Security was appropriately a primary concern, and I needed to ensure that all connections were properly encrypted. The most significant observation I've made during this work is that Java system properties are often overly depended upon / misused. Dependencies within RMI are a prime example of where the use of system properties cause some severe limitations, and are an area that probably could certainly use some improvement.

System properties are global to a JVM. Especially in a large application, conflicts can quickly arise if alternate configuration methods aren't available. For example, a necessary configuration may require one section of code or a referenced library to have a given system property set to one value, while another section or library requires the same system property set to another value. This is possible if they are split into separate programs, running on separate JVMs, but not within the same JVM. System properties can be set and changed at runtime by calling System.setProperty(…), but this should not be taken lightly and should usually be avoided. When they do need to be set outside of the java command-line, system properties should only be set within an applications "main" method, or other top-level code. I previously had to fix an issue where a JSP was switching the "javax.xml.transform.TransformerFactory" value between the interpretive and compiled (XSLTC) versions, which caused interesting issues (a.k.a. failures) elsewhere throughout the application, as the switch was causing different processors to be used for various functions, depending upon the timing between calls to the JSP (review: concurrency, thread safety). The same primary issue is shared with environment variables, as they are also globally shared by the JVM (or any process), However, unlike Java's system properties, the environment variables of the current runtime are not modifiable by Java.

Specific to my work were the Java Secure Socket Extension (JSSE) customizations, particularly the "javax.net.ssl.keyStore*" and "javax.net.ssl.trustStore*" system properties. Note that these are referred to as "defaults", with some default values provided for the default parameters, meaning that there should be a way to use a non-default value when needed. Another limitation of these specific customizations is that they are only single-valued, with no way to provide support for multiple key stores, etc., short of providing an overridden implementation class, which is full of issues in itself. Especially in a large enterprise application, calls need to be made to different services that require different certificates, and especially with limitations around automatic certificate selection, there needs to be a way to hook into this through flexible code when required.

HttpsURLConnection is an excellent example of a class that provides exactly this. In addition to the static (JVM-global) setDefaultSSLSocketFactory(…) method, it also provides a setSSLSocketFactory(…) method that can be used to provide customized SSL socket factories on a per-connection basis.

Unfortunately, JMX and RMI currently provide no such hooks, relying exclusively on system properties or the default socket factory. In the case of RMI, things only get more interesting and complicated. Everything exposed through the RMI public API is protocol-generic, with no concept of TCP, IP addresses, or port numbers. These are only handled by internal UnicastRef, LiveRef, and eventually TCPEndpoint classes. Only a "stub" is communicated to the client, through a registry or other means, and this stub contains an optional, serilaized RMICClientSocketFactory for creating a connection from the client to the host. That's right - the server controls the socket factory that will be used by the client to connect back to the server. I haven't found any clean way to override this behavior at the client. For JMX over RMI, both the server and client factories are set through an environment map with property keys defined on RMIConnectorServer. (RMI is the only JMX connector included with Java 5 and 6.)

For the application I was working on, changing the system properties to control the certificates used for secure communications is simply not feasible, if even an option at all. So I created an alternate SslRMIClientSocketFactory implementation, as even mentioned in the JDK's source code:

// If someone needs to have different SslRMIClientSocketFactory factories
// with different underlying SSLSocketFactory objects using different key
// and trust stores, he can always do so by subclassing this class and
// overriding createSocket(String host, int port).

This alternate implementation returned sockets from a socket factory on a custom-initialized SSLContext. This worked great when connecting between different instances of the same application (different JVMs on the same node, as well as different nodes). However, this requires the alternate class (and any and all other references classes) to be available on the classpath of any client making the connection - making things difficult for connections from other standard clients such as JConsole or VisualVM. It is possible, however, by setting the classpath with the "-J-Djava.class.path=…" arguments, which work the same for both JConsole and VisualVM. Both these programs utilize native launchers, so the "-J" prefix means to pass the trailing argument to the JVM and not the native launcher itself. When doing this, "<java.home>/lib/jconsole.jar" must be included as well for JConsole, or JConsole won't even start. tools.jar is also necessary for connecting to local processes and possibly other features, but apparently isn't required for remote connections like those being attempted here.

The solution I'm proceeding with is two-fold. First, I'm registering two JMXConnectorServers per JVM - one that uses the standard SslRMIClientSocketFactory, and one that uses a customized factory class. The first can be used by any client, assuming that the certificates are valid in the default trust store, or that the proper references using system properties can be made. The second can be used by any client that has the customized class available on the classpath, including other instances of the application itself. Fortunately, this does not require any additional network ports to be kept open. Each instance shares the same port, with a non-visible (at least not publicly or easily) ObjID being used to distinguish between them - which is also included in the serialized connection stub used by the client.

For my customized SslRMIClientSocketFactory class, simplest is best. By using only classes native to the JDK, only the one class is necessary to be available on the client's classpath so that it can be deserialized from the RMI stub. To avoid the issues with global system properties as described above, it also needed to be customizable, ideally being able to provide alternate socket factory implementations from within the client. Unfortunately, even if this class had a setSocketFactory method available, the rest of the RMI and JMX API doesn't provide for any apparent opportunity to call such a method. While a bit of a hack, my solution was to use a ThreadLocal. Here is my entire class:

import java.io.IOException;
import java.io.Serializable;
import java.net.Socket;
import java.rmi.server.RMIClientSocketFactory;

import javax.rmi.ssl.SslRMIClientSocketFactory;

public class ThreadLocalSslRmiClientSocketFactory
    implements RMIClientSocketFactory, Serializable{
  
  private static final long serialVersionUID = 1L;
  
  public static final ThreadLocal<RMIClientSocketFactory> SOCKET_FACTORY
      = new InheritableThreadLocal<RMIClientSocketFactory>(){
    @Override
    protected RMIClientSocketFactory initialValue(){
      return new SslRMIClientSocketFactory();
    }
  };
  
  public Socket createSocket(String host, int port) throws IOException{
    return SOCKET_FACTORY.get().createSocket(host, port);
  }
  
}

This allows for the socket factory to be configured by calling ThreadLocalSslRmiClientSocketFactory.SOCKET_FACTORY.set(…) somewhere within the current thread prior to making the connection, and without having requirements on or otherwise impacting the rest of the application. If this customization is not called, then the default SslRMIClientSocketFactory is used, and the global system properties should be referenced to determine the trust store, etc. Starting the servers then looks like this:

import java.lang.management.ManagementFactory;
import java.util.HashMap;
import java.util.Map;

import javax.management.MBeanServer;
import javax.management.remote.JMXConnectorServer;
import javax.management.remote.JMXConnectorServerFactory;
import javax.management.remote.JMXServiceURL;
import javax.management.remote.rmi.RMIConnectorServer;
import javax.rmi.ssl.SslRMIClientSocketFactory;

import org.slf4j.Logger;

// …

MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
JMXServiceURL url = new JMXServiceURL("rmi", null, 0);

Map<String, ? super Object> serverEnv = new HashMap<String, Object>();
serverEnv.put(
  RMIConnectorServer.RMI_SERVER_SOCKET_FACTORY_ATTRIBUTE,
  new JmxSslRmiServerSocketFactory());
serverEnv.put(
  RMIConnectorServer.RMI_CLIENT_SOCKET_FACTORY_ATTRIBUTE,
  new SslRMIClientSocketFactory());

JMXConnectorServer sslConnector = JMXConnectorServerFactory.newJMXConnectorServer(url, serverEnv, mbs);
sslConnector.start();
LOGGER.info("JMX SSL server started: {}", sslConnector.getAddress());

serverEnv.put(
  RMIConnectorServer.RMI_CLIENT_SOCKET_FACTORY_ATTRIBUTE,
  new ThreadLocalSslRmiClientSocketFactory());

JMXConnectorServer threadLocalSslConnector = JMXConnectorServerFactory.newJMXConnectorServer(url, serverEnv, mbs);
threadLocalSslConnector.start();
LOGGER.info("JMX thread-local SSL server started: {}", threadLocalSslConnector.getAddress())

What about other connectors? As listed in the connector chapter of the JMX overview documentation:

An optional part of the JMX Remote API, which is not included in the Java SE platform, is a generic connector. This connector can be configured by adding pluggable modules to define the following:

• The transport protocol used to send requests from the client to the server, and to send responses and notifications from the server to the clients
• The object wrapping for objects that are sent from the client to the server and whose class loader can depend on the target MBean

The JMX Messaging Protocol (JMXMP) connector is a configuration of the generic connector where the transport protocol is based on TCP and the object wrapping is native Java serialization. Security is more advanced than for the RMI connector. Security is based on the Java Secure Socket Extension (JSSE), the Java Authentication and Authorization Service (JAAS), and the Simple Authentication and Security Layer (SASL).

The generic connector and its JMXMP configuration are optional, which means that they are not always included in an implementation of the JMX Remote API. The Java SE platform does not include the optional generic connector.

There is also a Web Services (WS) connector being worked on, as described in JSR 262: Web Services Connector for Java Management Extensions (JMX) Agents and the reference implementation project at https://ws-jmx-connector.dev.java.net/. I've read that the WS connector is planned to be included with Java 7. Fortunately, it appears that the WS connector supports a custom SSLContext, configurable on the client, as shown in Securing JMX Web Services Connector (Jean-Francois Denise, 2007-08-16, blogs.sun.com).

Other related JMX references that have already proved useful:

• Sun Developer Network's Java Management Extensions Tutorial
• Luis-Miguel Alventosa's Blog on blogs.sun.com:
  • Mimicking the out-of-the-box JMX management agent in J2SE 5.0 (2006-06-02)
  • Authentication and Authorization in JMX RMI connectors (2006-09-25)
• Chapter 3. JSR 160 (JMX Remoting) Explained in the MX4J documentation.
• I avoided using an RMI registry completely here, but if I had, Eamonn McManus's Blog on java.net:
  • Securing the RMI registry (2006-12-29)
  • Reimplementing the RMI protocol (2007-01-11)