A faster, "complete" Java HttpServletRequest.getRequestUrl() replacement

As another performance-focused method for reuse: It should be easier to obtain the "complete" URL from a HttpServletRequest in Java. This is even a popular question on Stack Overflow: HttpServletRequest to complete URL - though I have various issues with each of the current answers.

Following is a self-written version that I've been using for a number of years now. I previously had posted it as a Gist, and since incorporated it into my MarkUtils-Web library in WebUtils - Checkstyle-approved and JUnit-tested:

MarkUtils-CLI: Annotations (and more) for Apache Commons CLI

So much of Java development in the enterprise often seems to be focused around web applications and other aspects of JEE. Sometimes it is almost comical to watch another developer who has typically been focused on this type of work try to develop a stand-alone Java program. One of the challenges faced here is typically proper acceptance, handling, and validation of command-line arguments.

Fortunately, Apache Commmons CLI exists to help with this effort. The summary from their home page:

The Apache Commons CLI library provides an API for parsing command line options passed to programs. It's also able to print help messages detailing the options available for a command line tool.

Commons CLI supports different types of options:

• POSIX like options (ie. tar -zxvf foo.tar.gz)
• GNU like long options (ie. du --human-readable --max-depth=1)
• Java like properties (ie. java -Djava.awt.headless=true -Djava.net.useSystemProxies=true Foo)
• Short options with value attached (ie. gcc -O2 foo.c)
• long options with single hyphen (ie. ant -projecthelp)

If anything, I feel that the Apache Commons CLI project is selling themselves short. I've found it to be a very comprehensive, well-designed library for effectively parsing the command-line. The only shortcoming I've observed is that the project was developed before Java 5 - and annotations - were available. As such, the library doesn't offer support for any features that annotations have to offer.

Introducing the latest addition to MarkUtils: MarkUtils-CLI is a library that provides an effective bridge between Apache Commons CLI and Java annotations - without replacing the mature Commons CLI library. Originally developed in 2013 with Commons CLI 1.2, the stability of and between both libraries across multiple releases has been a proven success. Like all of the MarkUtils libraries, there are minimal project dependencies: just commons-cli and slf4j-api.

Of the three stages to command line processing, MarkUtils-CLI should most commonly be used to:

• Replace the "Definition Stage" - wrapping it with functionality driven by annotations.
• Leave the "Parsing Stage" intact - while also offering a convenient wrapper to help "glue" everything together into a seamless process.
• Wrap the "Interrogation Stage" - automatically setting the annotated fields or calling the annotated methods as configured with the values received from the command line.

classParser

As with most languages implementing a "main" method, all command-line arguments coming into a Java program are received simply as Strings. Apache Commons CLI doesn't provide any assistance with parsing or converting these String values to the other data types that may be expected or required by the program being developed. MarkUtils-CLI contains a child "classParser" library to effectively bridging this gap - automatically converting one or more String values to the value type being expected, including:

• All primitive types: boolean, char, byte, short, int, long, float, and double.
• Number Object types: Byte, Short, Integer, Long, Float, Double, BigInteger, BigDecimal.
• Direct assignment to the abstract Number type - using BigDecimal unless a number with a radix != 10 is specified.
• Support for alternate number bases for all integral types listed above:
  • Hexadecimal: Prefixes 0x, 0X, 0h, or 0H.
  • Octal: Prefix 0o.
• Boolean
• Character
• String and CharSequence
• java.io.File
• java.net.URI
• java.net.URL
• java.util.UUID
• Identifiers to java.nio.charset.Charset
• Identifiers to java.util.TimeZone
• Enum types, using the enum type's named identifiers.
• Class (using forName)
• Object (using default instantiation of the passed-in class name, similar to Class above - 0-parameter or default constructor required.)
• java.net.InetAddress and java.net.InetSocketAddress, and their respective subtypes - such as java.net.Inet4Address and java.net.Inet6Address.
  • InetSocketAddresses are expected in the form of <InetSocketAddress>:<int port>. For example: "127.0.0.1:80" or "::1:80".

Boolean (either to the primitive or Object forms) conversion builds upon the default Boolean.valueOf logic, which evaluates to true if and only if the value is case-insensitive equals to "true". The conversion accepts the following values:

• true: Y, YES, T, TRUE, 1, -1, X
• false: N, NO, F, FALSE, 0
• Any other values will result in a detailed ClassParseException.

The parsers are split into a few different groups, with each class implementing IClassParser. ClassParserChain simply chains together these groups, and also implements IClassParser itself. Each parser may either return a definitive result (including null), throw an Exception which will bubble-up to the caller, or return IClassParser.NO_RESULT to defer processing to a later parser in the chain. UnhandledParser is a parser that always throws a detailed "Unhandled type" ClassParseException. ClassParserChain defines a default chain that can be obtained by calling getDefault(), and contains all available parsers within the library (those listed above) - ending with UnhandledParser.

If something in the default parse chain doesn't work for you (or if you simply don't like it) - simply create and use your own chain. For simplicity, the default chain could be obtained and prepended to in order to add a custom parser that takes precedence over all the following (default) parsers - or appended in order to handle any conversions that aren't already handled by the former (default) parsers.

Nothing in the "classParser" package is tied to any of the CLI logic - either that of Commons CLI, or the CLI-specific logic within MarkUtils-CLI. As such, this code may be appropriately reused for other purposes beyond parsing of command lines. (If appropriate, this code may be factored into a separate project outside of MarkUtils-CLI, which would then become a dependency of MarkUtils-CLI.)

cli

Now that we have a comprehensive set of code to handle the String-to-type conversions, it's time to put it to work.

The Parameter annotation can be assigned to any field or method - and provides attributes that map to most of the available options in the Commons CLI Option class. If a parameter name is not specifically defined, the base field/method name is used by default - following standard JavaBean conventions. Additional ParameterGroup and ParameterGroups annotations are used to work with the Commons CLI OptionGroup functionality. To eliminate needing to deal with security manager concerns, all fields or methods must be declared public to be accessed by this library. Proper design of the application code - including using separate "plain-old Java objects" ("POJOs") for containing the mapped command-line arguments - should eliminate most concerns with this.

Mappings support 1 or more argument values, acceptable by a suitable Array type as either a field, or an equivalent set method - including allowing for varargs. If the annotated class wishes to add any special processing - such as additional validations, or storing multiple values in a Collection type instead of an Array - simply annotate an appropriate set method instead of a field, and implement accordingly.

Most of the actual "magic" happens within ClassOptions. To allow for future extensibility and customization, all methods here are non-static, so an instance of this class must be created before use (using the default constructor). General usage is as follows:

1. ClassOptionsData get(Class<?>)
   • Pass-in the type of the "config" class, containing the @Parameter annotations. The returned ClassOptionsData instance contains mappings from each Commons CLI Option to the reflection Member for efficiency - especially if multiple executions are to be evaluated from within the same program (sure, this may be rare - but it is accounted for). Use of this composite class also allows for future, internal extensions - hopefully without breaking the public API.
   • This essentially wraps the Commons CLI "Definition Stage".
2. Options getOptions(ClassOptionsData cod)
   • Using the ClassOptionsData obtained above, returns a fresh Commons CLI Options instance - for use by the Commons CLI CommandLineParser - completing the Commons CLI "Parsing Stage".
3. void autoMap(Object target, ClassOptionsData, CommandLine, IClassParser)
   • Given an instance of the actual "config" class, along with the ClassOptionsData, a Commons CLI CommandLine, and an IClassParser - will read the values from the parsed command-line and automatically map the values to the annotated fields / methods in the "config" target.
   • This completed the Commons CLI "Interrogation Stage" - but besides simply providing the necessary details from the command-line, this should hopefully eliminate most of the "boilerplate" coding that would have to be done otherwise.

CliRunner

As part of the cli package, the CliRunner class aims to further simplify the above. While the separate calls to the various ClassOptions methods provide for almost unbounded flexibility, and may still be required in some cases, CliRunner should help further eliminate any remaining "boilerplate" code.

Click here to view the CliRunner Javadoc. All that is required is instantiation using one of the constructors, and calling its run method. As such, the complete code necessary to handle command-line processing - using both Apache Commons CLI and MarkUtils-CLI - can be as simple as:

package com.ziesemer.utils.cli;

import java.util.concurrent.Callable;

public class CliRunnerTestConfigTarget implements Callable<Integer>{
  
  @Parameter
  public int testReturn;

  public Integer call() throws Exception{
    return testReturn;
  }
  
}

package com.ziesemer.utils.cli.examples;

import org.apache.commons.cli.PosixParser;

import com.ziesemer.utils.cli.CliRunner;
import com.ziesemer.utils.cli.CliRunnerTestConfigTarget;

public class CliRunnerExample{

  public static void main(String[] args){
    CliRunner<CliRunnerTestConfigTarget> runner = new CliRunner<CliRunnerTestConfigTarget>(
      new PosixParser(), CliRunnerTestConfigTarget.class);
    CliRunnerTestConfigTarget configTarget = new CliRunnerTestConfigTarget();
    runner.run(configTarget, configTarget, args);
  }

}

Note that here, "configTarget" is being used both as the "configuration" object - as well as the target to be executed. I.E., CliRunnerTestConfigTarget has @Parameter annotations, and implements Callable<Integer>. The Integer result returned by the callable is used as the return code / exit status for the program (assuming everything else executes "normally" - further details below).

There are a significant number of getters and setters provided that can be used to customize the operation of CliRunner. Any unit of significant processing within CliRunner is factored into its own method - allowing for sub-classing and overriding as desired / required. Many of these methods are specifically documented as extension points. Please view the Javadocs (and possibly the source code) for additional details.

As noted above, an additional advantage of using CliRunner is that it can handle the return code / exit status of the program without any additional coding. As noted in the Javadocs, the following codes are used by default (all of which can be overridden):

• If the command line is successfully parsed, the result returned by the target. Otherwise:
  • 1 if the default help was requested.
  • 254 if a ParseException is generated.
  • 255 for any other exception (considered the "worst").

Note that to properly handle the return code / exit status for most cases, System.exit() will be called (by default).

Download

com.ziesemer.utils.cli is available on java.ziesemer.com under the GPL license, complete with source code, a compiled .jar, generated Javadocs, and a suite of 95+ JUnit tests. Please report any bugs or feature requests on the GitHub Issue Tracker.

MarkUtils Refresh, New Hosting

It's been a few years, but I just completed a much-needed refresh to my open source Java libraries ("MarkUtils").

The code is vastly unchanged, and has held up incredibly well over the Java 1.6, 1.7, and 1.8 upgrades (and soon, 1.9) - probably mostly due to 90%+ unit test coverage on everything. The main driving factor here was to get things re-hosted following the shutdown of java.net. In the process, I rebased against against all the latest dependencies and Maven plugins, and issued a coordinated release. Everything is built on the latest version of Java (and only publicly available secure version), 1.8.0_131 as of this writing - but should continue to run fine on older runtimes as documented and as needed for other purposes. Change logs are included in the Maven site reports of each project. Also included are some new projects, as well as some past efforts not previously hosted - neither of which have been blogged about in further detail here yet.

Downloads and additional details here: https://java.ziesemer.com. HTTPS/TLS/SSL, DNSSEC, CDN, and IPv6 support all now included.

As time permits, I'll work through my collection of old blog posts here to update any broken links to java.net - as well as to the legacy downloads across java.net and Google Code. Future efforts may include making the source code available directly on GitHub, as well as an accessible online Maven repository - either as a hosted directory at the above site, or possibly through Bintray (any input is welcome).

MarkUtils-JMX

Following my recent post on JMX Secure Connections / Avoiding Java System Properties, I am making another addition to MarkUtils: MarkUtils-JMX.

JMX Management Bean Metadata

My primary inspiration for this library was that JMX provides a generous amount of metadata along with each management bean, attribute, operation, and parameter - including names, descriptions, and impacts (INFO, ACTION_INFO, ACTION, or UNKNOWN) for operations. Parameter names must be provided through metadata, otherwise only the generated defaults of p0, p1, etc. are displayed. Unfortunately, associating this data with a management bean is currently a pain, and the only support available for this through the JDK itself is by subclassing StandardMBean, or completely instantiating a MBeanInfo yourself, and returning from a subclass of StandardMBean or from a custom DynamicMBean implementation.

MarkUtils-JMX provides a MBeanInfoBuilder class that serves as an alternative to the non-public classes used to create MBeanInfo's within the JDK. The most significant feature of MBeanInfoBuilder is support for including names, descriptions, and impacts that are read from Java annotations. The supported annotations are provided as part of this library in the com.ziesemer.utils.jmx.beanInfo package. Similar functionality is planned for release with Java 7 as part of JMX 2.0 / JSR 255, as detailed by Eamonn McManus in Playing with the JMX 2.0 API (2008-08-06, weblogs.java.net). Unfortunately, while I'm sure the annotations will not be compatible with my own (even just due to mine being in a com.ziesemer package), mine are available immediately, and support both Java 1.5 / 5.0 and Java 1.6 / 6.

MBeanInfoBuilder is not intended to be a complete replacement, as it currently doesn't support constructors, notifications, and descriptors, mostly as I currently have no use for them. However, these limitations are almost completely mitigated by MBeanInfoCombiner, which allows the metadata from a custom MBeanInfo built by MBeanInfoBuilder to supplement the default MBeanInfo built by the JDK's StandardMBean - effectively adding support for the custom annotations.

SimpleMBean, also included, is a alternative to the JDK's StandardMBean (as with MBeanInfoBuilder, and doesn't depend upon non-public classes. Along with utilizing MBeanInfoBuilder by default, this allows for easy extension / customization of the implementation. Additionally, both MBeanInfoBuilder and SimpleMBean are designed and written with performance as a primary focus. Performance is more of a concern for the actual MBean implementation than the building of the MBeanInfo, as the information should only have to be built once, where as the MBean implementation will likely be called repeatedly and often throughout the lifetime of the hosting application.

Authentication and Authorization

Also included in this library are classes to handle most of the work around securing JMX access by providing authentication and authorization.

Authentication is the simpler half, and is provided by com.ziesemer.utils.jmx.server.authentication.BaseJmxAuthenticator. This abstract class handles all the validation and breaking-down of the Object input parameter. An implementing class must only implement Subject authenticate(String username, String password). The authentication implementation is then registered as a value to the JMXConnectorServer.AUTHENTICATOR property, and passed-in as part of the environment map to JMXConnectorServerFactory.newJMXConnectorServer.

Authorization is a little more complex. Limited support is built-in to the JDK through Java's security policy and MBeanPermission. Unfortunately, the JDK approach seems overly-complex and did not meet several of my requirements. This library provides a high-performance and flexible alternative, BaseJmxAuthorizer.

BaseJmxAuthorizer and the other classes in com.ziesemer.utils.jmx.server.authorization are based on "Use Case 2" in Luis-Miguel Alventosa's blog post, Authentication and Authorization in JMX RMI connectors (2006-09-25, blogs.sun.com), where an InvocationHandler is used to selectively proxy requests to the management bean. Again, BaseJmxAuthorizer was designed and written with performance as a primary focus, along with flexibility. It is based on a Map (implemented with a HashMap), associating incoming method names (e.g. getAttribute or invoke) with an associated handler. This Map is publicly exposed, allowing for easy customization through modifications made to the map. Several default handlers, including the simple and unconditional AllowHandler and DenyHandler, are included in the package.

A ReadOnlyInvokeHandler is also provided, and allows all calls to operations / methods that are considered to be "read-only". The request impact (MBeanOperationInfo.getImpact()) is first consulted, and the request is allowed if INFO is returned. The request is otherwise denied, unless the impact returned is UNKNOWN, in which case an attempt to determine if the method has a "read-only" name is made, which checks for if the method name starts with "get", "is", "list", "query", or is equal to "hashCode". By default, it also allows for calls to ThreadMXBean.dumpAllThreads. These allowances allow for full and easy use of all "read-only" functionality provided by JConsole by default, without compromising on security.

Unfortunately, under Java 1.5 / 5.0, this currently denies access to get* methods on ThreadMXBean and LoggingMXBean, as these operations are marked as ACTION_INFO rather than UNKNOWN (or more preferably / correctly, INFO). This appears to have been somewhat fixed in Java 1.6 / 6, as per Sun Bug 6320104, it seems that UNKNOWN was simply not supported by the infrastructure in 1.5 / 5.0 at the time. A ReadOnlyJavaFixInvokeHandler is provided in the package as a work-around for use under Java 1.5 / 5.0. I reported Sun Bug 6933325 to address that these operations should now return INFO instead of ACTION_INFO, now that it is possible. Please view the Javadocs and/or source for additional details.

BaseJmxAuthorizer handles basic logging of method invocations through SLF4J by default.

This library's BaseJmxAuthorizer also simplifies authentication and improves performance by associating permissions with a subclass of JMXPrincipal, JmxPermissionPrincipal. This allows permissions to be obtained for a user once during authorization, without having to re-check permissions on each JMX call. Just be sure that this caching is accounted for, such that a user doesn't maintain unattended access if / once a permission is removed by the underlying security directory. A few options could be implementing a session timeout, or adding a call to a customized implementation to also remove the permissions from the runtime if / once removed from the user.

JmxPermissionPrincipal maintains an instance of a subclass of PermissionCollection, JmxPermissions, which contains one or more JmxPermission (subclass of Permission instances. Default provided JmxPermission's are CONNECT, READ_EX, INVOKE, and ADMIN. Other subclasses may be created and used, but should be kept as singletons. Please view the Javadocs and/or source for additional details.

Packaging with Apache Maven, Java versions

As with all my other MarkUtils libraries, MarkUtils-JMX is configured, compiled, tested, and packaged using Apache Maven. This library posed a little bit of a challenge, however, as I wanted to provide 2 versions, one built for Java 1.5 / 5.0, and one for Java 1.6 / 6. This is one of the few times I wish that Java had standard support for conditional compilation directives, such as those supported by the C preprocessor.

The primary reason for requiring dual versions is for Java 6's support of MXBean's, which were not supported in Java 5. This includes use of the additional 2-arg constructor added to StandardMBean, which as far as I can see, is impossible to implement without being available at compile-time - even if considering reflection tricks.

The solution I decided on for now is to offer dual packages / builds: com.ziesemer.utils.jmx.java5 and com.ziesemer.utils.jmx.java6. The Java 6 version includes the source paths from the Java 5 version. To clarify, only the Java 5 or the Java 6 version is required, as the Java 6 version is a superset of the Java 5 version. The Java 5 version includes a StandardMBeanInfoCombiner5 class that is a subclass of StandardMBean and utilizes MBeanInfoBuilder and MBeanInfoCombiner. The Java 6 version includes a StandardMBeanInfoCombiner6, which is the same as the Java 5 version, but exposes the added constructor and allows use of MXBeans. This setup actually worked with Maven fairly well, including the ability to automatically run Maven builds against both versions in one operation, and including packaging the build outputs together into the final distribution.

The Java 6 version also includes a DefaultRegistration class, which registers a few additional custom MXBeans that I found helpful and that provide functionality not currently offered by the MXBeans provided by the JDK (ManagementFactory). These include my custom CharsetMXBean, SecurityMXBean, and SystemMXBean. (As these custom beans themselves don't require any Java 6-specific features or calls at compile time, they are actually included in the Java 5 package. They just require Java 6 to be registered as MXBeans rather than regular MBeans.)

Download

com.ziesemer.utils.jmx is available on ziesemer.dev.java.net under the GPL license, complete with source code, a compiled .jar, generated JavaDocs, and a suite of 40+ JUnit tests. Download the com.ziesemer.utils.jmx-*.zip distribution from here. Please report any bugs or feature requests on the java.net Issue Tracker.

MarkUtils-IO: Performant Java Streams, Readers, and Writers

MarkUtils-IO is another high-performance addition to MarkUtils, and is a collection of utility classes that I've found myself frequently reusing over the past number of years.

• AppendableWriter: A faster alternative to StringWriter. Uses a non-synchronized StringBuilder by default instead of the synchronized StringBuffer, but also accepts any Appendable as the underlying destination.
• CharSequenceReader: An alternative to StringReader, but supports reading from any CharSequence in addition to String. Optimizations included for String, StringBuilder, StringBuffer, and CharBuffer.
• NoFlushBufferedOutputStream, NoFlushBufferedWriter: Fixes / work-arounds for https://issues.apache.org/jira/browse/XALANJ-2500. Overrides flush() to do nothing, unless called as flush(boolean) with true. (Previously discussed: Xalan-J Serialization Performance hindered by Flushing.)
• OutputStreamSplitter, WriterSplitter: Writes to zero or more OuputStreams or Writers as one.
• RandomAccessFileInputStream, RandomAccessFileOutputStream: Provides an InputStream or an OutputStream from a RandomAccessFile.

Work on this library was also the driving factor for another recent post concerning some of Java's built-in code, Redundant argument validation code in Java IO classes.

com.ziesemer.utils.io is available on ziesemer.java.net under the GPL license, complete with source code, a compiled .jar, generated JavaDocs, and a suite of 40+ JUnit tests. Download the com.ziesemer.utils.io-*.zip distribution from here. Please report any bugs or feature requests on the java.net Issue Tracker.

Capturing complete HTTP requests - Echo Server

Background

I recently had a need to capture and inspect a complete HTTP request in preparation for developing a new web service. The main reason for this is that there were no real requirements for the requested service. It wasn't clear which parameters would be sent in the request, or exactly how the parameters would be named. It also wasn't clear how the parameters would be split between GET and POST parameters, or even additional HTTP request headers. Additionally, there was also some history with issues around various character encodings, so I needed to be able to capture a byte-accurate copy of the entire request, including the headers and the body.

Initially, I did not have good luck finding an existing tool or solution for this. My first attempt was to just host a basic web server, then to capture the data using Wireshark. Unfortunately, Wireshark primarily works with Ethernet packets. It supports higher-level viewing of many protocols including HTTP. It even includes options for re-assembly of both HTTP headers and bodies, re-assembly of chuncked transfer-coded bodies, and decompression of entity bodies. However, while I'm sure there are additional options and methods for getting it to work more like I desired, it just didn't seem like the right tool for this particular job - and that is no fault of Wireshark.

My other early attempt was to use the mod_dumpio module in Apache HTTP Server. The first issue with this was that all the output from all requests is simply mixed-in to the same error log file (along with other debugging / outputs), which would make the data very difficult for proper extraction. The second issue was that at least as far as I can tell, there can only be one error log file per <VirtualHost/>, which would have resulted in an excessive amount of data being captured.

I then started to look at a simple Java HTTP server to capture the desired data. I've written trivial HTTP servers before, but it quickly becomes non-trivial to properly handle and respond to all the possible options and variations - even just to accept a complete request (including body) from a client. Trying to avoid duplicating previous work, I looked at a number of existing web servers including Apache Tomcat, but did not find any that provided the desired logging options.

My solution

I started looking further into Jetty. (I previously used and blogged about Jetty in regards to a test platform for my MarkUtils-Web project.) I found that I could intercept the incoming requests byte-by-byte by extending Jetty's default Connector - SelectChannelConenctor, and then overloading the newEndPoint(…) method to return an extended SelectChannelEndPoint. Hooking into the SelectChannelEndPoint's fill(Buffer buffer) method allows for capturing of the complete HTTP request. Kudos to the Jetty developers for not making this difficult or impossible by marking everything as private or otherwise overly-restricted, as compared to an unfortunate practice followed by many other projects and companies!

With only a little extra code, each HTTP request is logged to a chosen directory as a pair of files, grouped by a time-based session ID. The first is a "meta" file that contains details that would not ordinarily be captured as part of the HTTP capture, including the session ID, server date, and remote address, host, and port. The second is the "content" file that contains the actual byte-by-byte capture of the HTTP request. While it is named as a ".txt" file for easy viewing, it is treated as binary and will accurately capture all requests, including those with binary payloads. The format also allows for easily re-playing the request to a server for additional testing, debugging, or analysis.

Finally, by implementing and registering an associated Handler, the request is not only captured, but is efficiently echoed back to the client - without ever needing to buffer or store the entire request. This echoed response starts with the contents of the "meta" file, including the session ID that can be used by the client to easily refer to the saved log file back on the server. The contents of a sample echoed response are shown below, and would appear in the body of a viewing web browser:

Session ID: 124f67a451d-6313f5e0
Date: Sun Nov 15 00:14:18 CST 2009
remoteAddr: 127.0.0.1
remoteHost: 127.0.0.1
remotePort: 23349
==========
POST /someUrlPath?someGetKey=someGetValue HTTP/1.1
Host: localhost:8080
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5 GTB5 (.NET CLR 3.5.30729)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Content-Type: application/x-www-form-urlencoded
Content-Length: 25

somePostKey=somePostValue

On the server-side, the first portion (above the '=' divider) is saved as 124f67a451d-6313f5e0-meta.txt, with the last portion (below the divider) saved as 124f67a451d-6313f5e0-content.txt.

The main class for this project is com.ziesemer.httpEchoServer.HttpEchoServer. It is written to be suitable for inclusion into other projects or uses, as visible from the included JUnit test. It also includes a main() method for direct use from the command-line, supporting arguments to control the port to listen on ("--port") and the directory to use to store the log files ("--logDir"). By default, Jetty is configured to listen on port 8080. If the specified port is unavailable, add "--allowDynamicPort" to configure the process to fall-back to a dynamically-chosen port if the specified port is already in-use.

Fiddler: Another alternative

Another alternative I later considered was Microsoft's Fiddler, a HTTP Debugging Proxy. While not open-source, it is freeware and extensible. It is also certainly a better match for my requirements than either Wireshark or Apache's mod_dumpio, and arguably even my solution described here. However, Fiddler still requires a server to answer the requests for it can monitor the traffic, and doesn't support echoing the request to the response. Fiddler does have many other features to offer that may prove useful, and is at least worth testing out.

Download

com.ziesemer.httpEchoServer is available on ziesemer.java.net under the GPL license, complete with source code, a compiled .jar, generated JavaDocs, and JUnit tests. Download the com.ziesemer.httpEchoServer-*.zip distribution from here. Please report any bugs or feature requests on the java.net Issue Tracker.

MarkUtils-Codec: Base64, URL, and other byte/char conversions

This is an overdue introduction of my latest addition to MarkUtils. MarkUtils-Codec could be considered a high-performance replacement for Apache Commons Codec. Like Commons Codec, this implementation has support for Base64, URL (a.k.a. Percent, and covered previously), and Hexadecimal encodings and decodings. Also like Commons Codec, this implementation utilizes a number of interfaces that allow various codecs to be used interchangeably. Unlike Commons Codec, this implementation is designed to be higher performing, as it is written for streaming use with the Buffer classes. The most significant advantage to this design is lower memory requirements and usage, especially when working with longer lengths of data.

MarkUtils-Codec is really a follow-up to one of my previous posts, Improving URLEncoder/URLDecoder Performance in Java. While the API I proposed and sample code I provided solved an immediate need, the lack of proper interfaces made it difficult to replace with other codecs, such as Base64. The options to plug-in to other standard streaming classes was also limited. For example, there was no clear way to create an InputStream that would read decoded data from encoded data. This library is meant as a complete replacement, as I have placed the "urlCodec" library in archival status.

Until I have a suitable place to host the Javadocs online, please reference them in the downloads available at ziesemer.dev.java.net.

The highest-level API interface is com.ziesemer.utils.codec.ICoder. Verbatim from the Javadoc, this is the "Base API for high-performance encoding and decoding between various Buffers. Supports conversions between ByteBuffers and CharBuffers through the IByteToCharEncoder and ICharToByteDecoder child interfaces. This API is similar in design to CharsetEncoder and CharsetDecoder."

Do note that the relation to the Charset classes may seem a bit backwards. When a character set is decoded, the input is bytes and the output is characters. The purpose of this library is to encode any data (as bytes) into character data that can safely be sent through various non-byte transports, e.g. HTTP forms. For this purpose, decoding takes characters and input and produces bytes as output.

Here is a simple example of supported direct usage, taking no advantage of streaming capabilities. This is included as one of the JUnit tests within the com.ziesemer.utils.codec.DemoTest class:

/**
 * Simple usage, taking no advantage of streaming capabilities.
 */
@Test
public void testDirectSimple() throws Exception{
  IByteToCharEncoder encoder = new URLEncoder();
  ICharToByteDecoder decoder = new URLDecoder();
  
  // Random test data.
  byte[] rawData = new byte[1 << 10];
  new Random().nextBytes(rawData);
  
  // Encode.
  CharBuffer cbOut = encoder.code(ByteBuffer.wrap(rawData));
  
  // Decode (round-trip).
  ByteBuffer bbOut = decoder.code(cbOut);
  
  // Verify.
  byte[] result = new byte[bbOut.remaining()];
  bbOut.get(result);
  Assert.assertArrayEquals(rawData, result);
}

Or an even simpler example, using convenience methods. Note that the Base64 codec can be replaced with URL (percent), Hex, or another provided codec:

byte[] sampleBytes = new byte[]{0, 1, 2, 3};
String enc = new Base64Encoder().encodeToString(sampleBytes);
System.out.println(enc); // Yields: AAECAw==
byte[] dec = new Base64Decoder().decodeToBytes(enc);
System.out.println(Arrays.equals(sampleBytes, dec)); // Yields: true

A number of input/output wrappers are also included in the "com.ziesemer.utils.codec.io" package, allowing for transparent use as a standard Java IO reader, writer, or stream. The signatures of the required constructors are also shown. Each class also provides an alternate constructor that can be used to fine-tune the read buffer size.

• CharDecoderInputStream(ICharToByteDecoder decoder, Reader reader)

  Reads raw bytes from encoded characters. Counter-part to CharEncoderReader. This is a pull-interface; CharDecoderWriter is the equivalent push-interface.

  Can be adapted to read characters to the consumer (instead of raw bytes) by wrapping in a InputStreamReader. This is only valid if the decoded form of the data is known to only contain valid characters. Can also be adapted to read bytes from a provider (instead of characters) by using a InputStreamReader as the Reader.

• CharDecoderWriter(ICharToByteDecoder decoder, OutputStream outputStream)

  Accepts encoded characters, and writes the raw bytes. Counter-part to CharEncoderOutputStream. This is a push-interface; CharDecoderInputStream is the equivalent pull-interface.

  Can be adapted to accept bytes from a provider (instead of characters) by wrapping in a OutputStreamWriter.

• CharEncoderOutputStream(IByteToCharEncoder encoder, Writer writer)

  Accepts raw bytes, and writes the encoded characters. Counter-part to CharDecoderWriter. This is a push-interface; CharEncoderReader is the equivalent pull-interface.

  Can be adapted to write bytes to the consumer (instead of characters) by using a OutputStreamWriter as the Writer. Can also be adapted to accept characters from the provider (instead of raw bytes) by wrapping in a OutputStreamWriter.

• CharEncoderReader(IByteToCharEncoder encoder, InputStream reader)

  Reads encoded characters from raw bytes. Counter-part to CharDecoderInputStream. This is a pull-interface; CharEncoderOutputStream is the equivalent push-interface.

  Can be adapted to read characters to the consumer (instead of raw bytes) by wrapping in a InputStreamReader.

Also included are a number of "character lists" (in the com.ziesemer.utils.codec.charLists package), particularly to support the different Base64 variations.

Please refer to the included JUnit tests (currently 169) for usage examples.

Download

com.ziesemer.utils.codec is available on ziesemer.java.net under the GPL license, complete with source code, a compiled .jar, generated JavaDocs, and a suite of JUnit tests. Download the com.ziesemer.utils.codec-*.zip distribution from here. Please report any bugs or feature requests on the java.net Issue Tracker.

MarkUtils-PacProxySelector for Java

Many computer networks make use of proxy servers for web and Internet connectivity. This is especially true for business and other organization networks, where their use is required by security and other policies. Due to the typical use of proxy servers, they are often thought of in terms of "restricting access". Instead, they should be thought of in the proper terms of a means of "providing access". Even outside of typical corporate environments, proxy servers can be invaluable for testing and debugging, as well as used as a type of VPN between private networks.

Most web browsers and other networked applications support directing traffic through one or more proxy servers. The typical configuration dialog looks like this, as shown from Mozilla Firefox:

Of the "manual" options, "HTTP" and "SSL" (TLS) are the most basic and common, followed by "FTP". "Gopher" is rarely used anymore - it has already been dropped by Microsoft Internet Explorer, and may be dropped in Mozilla Firefox 4.0. "SOCKS" is arguably the most powerful, supporting any of the above protocols in addition to any other TCP- or UDP-based protocol. (If SOCKS is configured and supported, none of the other protocols need to be configured.)

Unfortunately, directing an entire networks' traffic through a single proxy server can quickly cause a bottle-neck and a common point for failure. This is especially true when all LAN traffic is also sent through the proxy. (In ideal network traffic patterns, there should be many more multiples of LAN traffic over Internet traffic.) Some of the performance and availability concerns can be addressed through DNS or other load balancing. Another common attempt is to configure the "no proxy for" / exception list. Unfortunately, there are some severe limitations to this design. First, the list must be kept up-to-date as the network configuration changes. (There are various tools for this.) More significantly, there are many desired configurations that cannot be accounted for. For example, what if a list is needed for the servers that should be sent to a proxy server, rather than skipping the proxy server (reverse logic)? Or what if traffic must be split among multiple proxy servers, depending upon the destination or other parameters?

The solution to all of the above and other similar concerns is through the use of the last option shown above, and probably the most overlooked: "Automatic proxy configuration URL". This option is also known as proxy auto-config, or PAC, and was introduced into Netscape Navigator as early as 1996. A PAC file only needs to contain an implementation of a JavaScript function, FindProxyForURL(url, host). From here, the full power of JavaScript can be used, including regular expressions, associative arrays, and closures, as well as a number of predefined helper functions specific to PAC. Within the PAC function, various load balancing and black- or white-listing tasks can be performed, optionally by maintaining internal state. A list of multiple proxies may also be returned for attempts by the client. The PAC file is loaded from a URL (including local file:// URLs), where it can be centrally maintained and updated. The PAC file may be cached by the web browser or other client, but should respect the cache settings sent in the HTTP headers if retrieved through HTTP. Alternatively, a chrome:// URL can even be used, allowing for the PAC file to be maintained within a Firefox extension, and updated through Firefox's standard auto-update process for extensions.

Java support

Java supports many of the same above proxy options, mostly through the use of system properties. For full details, see the tech note at java.sun.com, Java Networking and Proxies. These settings affect any communications made through URLConnection, Socket, and possibly other network-related classes. Previously, the proxy configuration options were limited to the "manual" options listed above, with separate options for HTTP, HTTPS, FTP, and SOCKS. However, Java 1.5/5.0 introduced the Proxy and ProxySelector classes. A default ProxySelector can be configured for the current JVM by calling ProxySelector.setDefault(ProxySelector).

Unfortunately, Java does not currently provide any visible support for proxy auto-config (PAC) files. However, the ProxySelector's List<Proxy> select(URI uri) method looks and works very similar to the PAC's FindProxyForURL(url, host) function. The most notable difference is that it is strongly-typed to standard Java classes. As part of my MarkUtils collection, I created MarkUtils-PacProxySelector to provide a ProxySelector implementation that works with PAC files.

Since the PAC files are based on JavaScript, the ability to evaluate JavaScript is required. Fortunately, this is easily done through Java, especially with the introduction of the Java Scripting API in Java 1.6/6.0 (JSR-223). Java 1.6 bundles an internal version of the Mozilla Rhino implementation of JavaScript for Java, based on 1.6R2. Unfortunately, Java doesn't expose all the features of JavaScript or Rhino directly through the scripting API, some of which are required to implement the PAC functionality in a compatible fashion. This includes defining top-level bindings in the JavaScript environment to Java functions, which is directly supported in Rhino by adding a binding to a FunctionObject - a class to which there is no publicly visible match in the JDK. While it is probably possible to hack a work-around to this, my current implementation utilizes Rhino directly. Besides taking advantage of the improvements in the latest version of Rhino (currently 1.7R2), this allows the utility to be easily used with both Java 1.5/5.0 and 1.6/6.0. (However, note that JSR-223 is unofficially supported under Java 1.5/5.0 as well by downloading and including the .jar's from the reference implementation.) Using Rhino directly also avoids some potential security issues, which I reported in Sun Bug 6782031 and Mozilla Bug 468385.

As commented in the pom.xml file, Mozilla Rhino is currently not available through the central repository, a Mozilla repository, or any other "official" repository. I've added a dependency to it as "org.mozilla.javascript : rhino : 1.7R2". For this to work properly, Rhino will need to be downloaded and installed into a local repository as named above.

In addition to the standard PAC methods, PacProxySelector supports an added function called "connectFailed" to take advantage of the connectFailed(URI, SocketAddress, IOException) functionality on ProxySelector. The JavaScript method is called with the same arguments as on ProxySelector, just with the .toString() representations of each of the three parameters. The PAC file could then store this information within internal state to possibly affect future calls to FindProxyForURL.

For the most flexibility, the constructor to PacProxySelector accepts a Reader, which should read from a PAC file. There is also a public static configureFromProperties() method that returns a ProxySelector, assuming that the path to a PAC file is stored as either a Java system or environment property named "proxy.autoConfig", similar to the other network properties. After obtaining an instance from either the constructor or the method, it should be passed to ProxySelector.setDefault(ProxySelector), unless otherwise used directly. Alternatively, a setDefaultFromProperties() convenience method is provided to do this in one call.

I wrote this in mind for plugging into other Java applications. Ideally, the JDK would provide a system property that accepts the classname for the default ProxySelector or some other method for setting the default outside of a function call within the code, but this is currently not the case. However, all that has to be done is finding a way to execute one of the above configuration options from the desired Java application before network access is attempted. I've successfully written a plugin for Oracle SQL Developer that does exactly this. The same is also possible for Eclipse, though it requires patching of some of the plugins due to the current infrastructure. (See Eclipse bug 257443.) Alternatively, PacProxySelector provides a main method that calls setDefaultFromProperties() before chaining execution to another program's main method. See the included Javadoc for details.

Download

com.ziesemer.utils.pacProxySelector is available on ziesemer.java.net under the GPL license, complete with source code, a compiled .jar, generated JavaDocs, and a suite of JUnit tests. Download the com.ziesemer.utils.pacProxySelector-*.zip distribution from here. Please report any bugs or feature requests on the java.net Issue Tracker.

Handling XML Encodings with MarkUtils

Especially in today's focus on higher-level languages, lower-level details are often overlooked. However, character encodings are one such detail that must be remembered, particularly when working with interfaces such as web services, which may be communicating with a large variety of different platforms and languages - both programming and written.

US-ASCII is probably one of the most significant character encodings, but not the earliest. ASCII is the predecessor to the ISO/IEC 646 standard, and is a subset of the Unicode standard, particularly UTF-8. (See also: English in computing.) US-ASCII by itself is unable to represent more than its defined 95 printable characters - 62 of these being [A-Za-z0-9], with the remainder being used for punctuation and other miscellaneous symbols. This limited character set makes it impossible to accurately represent other Latin-based languages such as Spanish, French, and German. "Extended ASCII" such as ISO/IEC 8859-1 provide complete or near-complete coverage of these and other alphabets, but still fail to account for other characters. Using a character encoding that provides full support for the Universal Character Set (UCS) such as UTF-8 is the recommended solution to this and other related issues.

Many character encodings share the same byte representations for the basic English character set [A-Za-z] and [0-9]. These include US-ASCII, ISO-8859-1, UTF-8, and others. For example, in these character encodings, the following mappings are always true:

Character	Decimal	Hexadecimal
0	48	0x30
9	57	0x39
A	65	0x41
Z	90	0x5A
a	97	0x61
z	122	0x7A

Other character encodings, such as EBCDIC, are completely different.

Even when receiving a byte stream from one of the "compatible" encodings, the correct encoding still needs to be determined, as each encoding is handled slightly differently. Certain byte sequences that may be allowed in US-ASCII variants may be invalid in UTF-8. UTF-16 and UTF-32 are easily converted to and from UTF-8, but may be sent with different byte orderings - either big-endian or little-endian.

Encodings in XML

In the W3C's recommendation for XML (essentially the XML standard), the use of character encodings is specifically addressed in section 4.3.3. However, since the encoding is defined with the XML, the encoding declaration itself is also encoded. This is also addressed in the appendix, Autodetection of Character Encodings (Non-Normative):

The XML encoding declaration functions as an internal label on each entity, indicating which character encoding is in use. Before an XML processor can read the internal label, however, it apparently has to know what character encoding is in use—which is what the internal label is trying to indicate. In the general case, this is a hopeless situation. It is not entirely hopeless in XML, however, because XML limits the general case in two ways: each implementation is assumed to support only a finite set of character encodings, and the XML encoding declaration is restricted in position and content in order to make it feasible to autodetect the character encoding in use in each entity in normal cases.

Essentially, the character encoding in use may be determined by a combination of the Byte-order mark (if present), and the value of the "encoding" attribute in the XML declaration of the prolog. However, there may also be external encoding information available from higher-level protocols that must be factored into the determination. This includes the MIME Content-Type sent over HTTP, as defined in RFC 3023.

In Apache Xerces2-J, this is mostly handled within the implementation, particularly by org.apache.xerces.impl.XMLEntityManager.createReader(…). Similar code is also in org.apache.xerces.xinclude.XIncludeTextReader, which also accounts for at least most of the RFC 3023 rules, but only if the input source is a "system Id" which results in the use of a URLConnection.

Addition to MarkUtils-XML

I had a need to meet these same requirements on an input stream to determine the encoding, but before handing off the processing directly to Xerces or another XML processor. I also wanted a flexible solution that would follow the RFC 3023 recommendation, but would be able to work with a variety of sources - including a URLConnection or a HttpServletRequest. I found no existing and public API that met these requirements, so I made my own - and am making it available for public use as part of MarkUtils-XML.

Below is an outline of the public API. While it may first appear to be a bit lengthy, it is designed to be flexible and usable in high-performance environments. None of the Javadoc documentation is included here for brevity. The complete source code, along with a compiled .jar, generated Javadocs, and a comprehensive suite of JUnit tests are available in the com.ziesemer.utils.xml-*.zip distribution on ziesemer.java.net, with XmlEncoding available starting with version 2009-05-20. The tests include all 16 usable MIME Content-Type examples from http://tools.ietf.org/html/rfc3023#section-8.

public class XmlEncoding{
  
  public static final String US_ASCII_NAME = "US-ASCII";
  public static final Charset US_ASCII;
  public static final String UTF_8_NAME = "UTF-8";
  public static final Charset UTF_8;
  public static final String UTF_16BE_NAME = "UTF-16BE";
  public static final Charset UTF_16BE;
  public static final String UTF_16LE_NAME = "UTF-16LE";
  public static final Charset UTF_16LE;
  
  // Below are not guaranteed to be in both Java 1.5/5.0 and 1.6/6.0.
  public static final String UTF_32BE_NAME = "UTF-32BE";
  public static final String UTF_32LE_NAME = "UTF-32LE";
  public static final String IBM037_NAME = "IBM037";
  
  public static InputStream createBufferedStream(InputStream is) throws IOException{…}
  
  public static String determineFromBOM(InputStream is) throws IOException{…}
  
  public static String guessFromDeclaration(InputStream is) throws IOException{…}
  
  public static String determineFromDeclaration(InputStream is, Charset guessed) throws IOException{…}
  public static String determineFromDeclaration(InputStream is, Charset guessed, int readSize) throws IOException{…}
  
  public static String calculate(InputStream is) throws IOException{…}
  public static String calculate(InputStream is, String contentTypeMime, String contentTypeEncoding) throws IOException{…}
  public static String calculate(InputStream is, String contentType) throws IOException{…}
  
  public static InputSource createInputSource(InputStream is) throws IOException{…}
  public static InputSource createInputSource(InputStream is, Charset charset) throws IOException{…}
  public static InputSource createInputSource(InputStream is, String contentTypeMime, String contentTypeEncoding) throws IOException{…}
  public static InputSource createInputSource(InputStream is, String contentType) throws IOException{…}
  
  public static String getContentTypeMime(String contentType){…}
  
  public static String getContentTypeEncoding(String contentType){…}
  
  public static boolean isContentTypeTextXml(String mime){…}
  
  public static boolean isContentTypeApplicationXml(String mime){…}
}

In most instances, only one of the createInputSource methods will be necessary. For example, from a HttpServletRequest:

HttpServletRequest req = …;
InputSource iSource = XmlEncoding.createInputSource(
  req.getInputStream(), req.getContentType());

All other non-createInputSource(…) methods in this class that accept an InputStream require that mark(int) and reset() are supported. This allows for bytes to be read and "unread", allowing for read portions to be re-read by the XML processor as appropriate. For example, determineFromBOM(InputStream) will consume the BOM bytes if it can be successfully determined, while the guessFromDeclaration(…) and determineFromDeclaration(…) methods will always reset the InputStream to its original position. createBufferedStream(InputStream) can be used to appropriately wrap an InputStream if required. See the included Javadocs for complete details.

XML Property Expansion with MarkUtils-XML

XML is an increasingly popular format for configuration files. Unfortunately, XML doesn't have a built-in standard for variable substitution or property expansion. This addition to my MarkUtils-XML package is one solution, included as of with version 2009.03.14, and is available for download at http://code.google.com/p/ziesemer/downloads/list?q=label:Featured.

Background

Several existing practices make use of replacement string patterns, e.g. "${…}" described and supported by Apache Commons Digester's MultiVariableExpander. Apache Maven uses the same pattern in its pom.xml files, but doesn't appear to use Digester to implement this. Instead, Maven uses implementations of org.codehaus.plexus.component.configurator.expression.

I see a few issues with this overall approach of using replacement string patterns:

• While Maven supports resolving properties from 5 different sources, including environment variables and Java system properties, the implementation doesn't eliminate the possibility for naming collisions.
• Neither Digester nor Maven appear to allow for escaping these patterns. While not common, this can sometimes cause issues if "${" needs to be used and passed as-is.
• Neither implementation makes it clear what happens if replacement string patterns are present but not expanded due to the variable not existing. (Is an exception thrown, the replacement string pattern returned as part of the result, or the replacement string pattern expanded to nothing ("")?)

Granted, these few issues could all easily be fixed through the use of escape characters, etc. However, more similar issues will probably continue to appear. Instead, let's use XML as I think it is intended.

Consider the use of XSLT. For inserting values from referenced elements into the output, <xsl:value-of select="…"/> is used, rather than a "${…}"-style syntax. However, this isn't the best example, as the select is a XPath Expression, which then makes use of the "$" prefix as part of a VariableReference.

Solution

The solution provided in MarkUtils-XML is in 2 parts: A schema that declares the property elements, and a XmlPropertyExpander class that performs the expansions.

Schema

The schema is namespaced to avoid naming collisions, as "http://namespaces.ziesemer.com/utils.xml/propertyExpansion". It is meant for easily inclusion into other schemas through the use of the <import …/> element. The following property elements are defined, all extending "BasePropertyType", with one required "name" attribute of type NCName:

• <SystemProperty/> - Resolves Java system properties from System.getProperty(…).
• <EnvironmentProperty/> - Resolves environment variables from System.getenv(…).
• <InstanceProperty/> - Resolves properties from those set on the current instance of the property expander.
• <LocalProperty/> - Resolves properties from <LocalPropertyDef/> elements registered with the current instance of the property expander, typically from the same XML file.

A "PropertyTypeGroup" group exists that allows the above 4 property element types, as well as any elements from any other namespace - allowing for extensions or use-cases not covered by the above pre-defined elements. A "SubstitutionVariableType" group exists that allows for any elements from this group, as well as other mixed content.

For defining <LocalProperty/> values, use one ore more <LocalPropertyDef/> elements. They are of type "LocalPropertyDefType", which extend "BasePropertyType". This allows for references to other properties, including other <LocalProperty/> elements, to be used as part of the definition for a <LocalProperty/>. Just be sure to avoid creating circular references, which would normally result in a StackOverflowError but are instead caught earlier (32 recursions by default) and thrown as a RuntimeException. A default "Properties" element exists to act as a container for one or more <LocalPropertyDef/> elements, including a XSD schema <key/> element to provide a constraint against multiple local property elements with the same name.

Shown below is an example schema that allows for storing a list of file system directories. It imports the "PropertyExpansion" schema, allowing for a <Properties/> element, and the inclusion of one or more "PropertyTypeGroup" elements for property expansion. This example is included in the distribution as "DirectoriesExample.xsd" in the JUnit source:

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema
    xmlns:xs="http://www.w3.org/2001/XMLSchema"
    xmlns:pe="http://namespaces.ziesemer.com/utils.xml/propertyExpansion"
    elementFormDefault="qualified">
  
  <xs:import namespace="http://namespaces.ziesemer.com/utils.xml/propertyExpansion"/>
  
  <xs:element name="Directories">
    <xs:complexType>
      <xs:sequence>
        <xs:element ref="pe:Properties" minOccurs="0"/>
        <xs:element ref="Directory" minOccurs="0" maxOccurs="unbounded"/>
      </xs:sequence>
    </xs:complexType>
    
    <xs:keyref name="PropertyKeyRef" refer="pe:PropertyKey">
      <xs:selector xpath=".//pe:LocalProperty"/>
      <xs:field xpath="@name"/>
    </xs:keyref>
  </xs:element>
    
  <xs:element name="Directory" type="pe:SubstitutionVariableType"/>
  
</xs:schema>

Now, a simple example XML document that follows the above schema. It is also included in the distribution as "DirectoriesExample.xml" in the JUnit source:

<?xml version="1.0" encoding="UTF-8"?>
<Directories
    xmlns:pe="http://namespaces.ziesemer.com/utils.xml/propertyExpansion">
  
  <pe:Properties>
    <pe:LocalPropertyDef name="java.lib">
      <pe:SystemProperty name="java.home"/>/lib
    </pe:LocalPropertyDef>
  </pe:Properties>
  
  <Directory><pe:SystemProperty name="java.home"/>/bin</Directory>
  <Directory><pe:SystemProperty name="java.home"/>/lib</Directory>
  
  <Directory><pe:LocalProperty name="java.lib"/></Directory>
  <Directory><pe:LocalProperty name="java.lib"/>/ext</Directory>
  <Directory><pe:LocalProperty name="java.lib"/>/management</Directory>
  <Directory><pe:LocalProperty name="java.lib"/>/security</Directory>
  
</Directories>

By default, whitespace is trimmed from each text node during expansion (but not whitespace within individual text nodes). This allows for new lines or other "pretty-printing" to be used within the elements to be expanded without having the whitespace included in the expanded result. If whitespace must be maintained, use CDATA sections.

Java class

Everything needed to expand these properties from Java is contained within one class, com.ziesemer.utils.xml.XmlPropertyExpander. It is best-suited for use with XML DOM, providing one core method for expanding elements containing child properties into a String:

public String expand(Node parent);

If no <LocalPropertyDef/> elements are to be supported, XmlPropertyExpander can be instantiated using the 0-argument constructor. Otherwise, the constructor accepts one or more Nodes that contain child <LocalPropertyDef/> elements to resolve against, either as a List or as varargs.

While best-suited for use with DOM, supporting methods are publicly accessible for other use, such as SAX parsing:

public String resolveProperty(String nodeName, String propName);
public Element findLocalProperty(String name);

This assumes that <LocalPropertyDef/> elements will still be available through DOM, which should normally not be an issue as the number of local properties defined should typically be a small fraction of the number elements that refer to these properties. (Potentially parse the <Properties/> element using DOM, then the rest of a larger document with SAX.) If you would like additional SAX support, please open an enhancement request on the issue tracker at ziesemer.dev.java.net.

While not shown in the above example, <InstanceProperty/> elements are resolved through a Map set on the instance:

public void setInstanceMap(Map<String, String> instanceMap);

By default, if a standard property fails to resolve (resolves to null), nothing is output for the property - as if the property element didn't exist. A warning with the unresolved property type and name will be output through java.util.logging. (While I prefer and support SLF4J, this avoids an additional compile- and run-time dependency. java.util.logging can be forwarded to SLF4J through jul-to-slf4j.jar.) The same applies if elements from an external namespace are found during the expansion. To change this default functionality, including returning a resolved String, sub-class and override either or both of these methods:

protected String unresolvedProperty(String localName, String propName);
protected String externalNamespaceResolveProperties(Element e);

Finally, a static method is available for returning a StreamSource for the property expansion XSD schema, useful for building Schema instances with for validating instance documents.

public static StreamSource getSchemaStreamSource();

See the distribution's Javadocs or source code for details on any of the above methods. For more examples, including more features and complex use-cases, see the JUnit source.

MarkUtils-XML: NamespaceContextMap, PrettyPrint, Date Format

Adding to my collection of MarkUtils, this is my introduction of MarkUtils-XML. It is available on ziesemer.java.net under the GPL license, complete with source code, a compiled .jar, generated JavaDocs, and a suite of JUnit tests. Download the com.ziesemer.utils.xml-*.zip distribution from here.

NamespaceContextMap

I think that XML Namespaces are a great solution for avoiding naming collisions. I also think that XPath is a very useful tool for pulling data out of XML documents. Unfortunately, using XPath with XML Namespaces involves a little bit of extra work, especially in the current version of Java.

The most common issue I see other developers run into when first working with this combination is finding that their XPath isn't returning any results. This is because unless otherwise specified, the XPath only searches for nodes declared without a namespace. XML nodes declared with namespaces can be referenced using namespace prefixes, where each prefix is assigned to a specific namespace URI. It should be noted that prefixes only function as placeholders for the namespace URIs. Even though an XML document may have one prefix assigned to a given namespace, it cannot be assumed that prefix will remain unchanged. Many times, these prefixes are generated automatically and/or as needed for each namespace used in a XML document. Two XML documents should be considered equal if the only difference between them are the prefixes used for a common namespace. For example, XSLT uses a namespace URI of "http://www.w3.org/1999/XSL/Transform". It is commonly prefixed to either "xs:" or "xsl:", though other prefixes are also used and valid. As such, any application should explicitly map any desired namespaces to a local prefix that can be used to reference XML nodes declared with a namespace.

In Java, the XPath class accepts a NamespaceContext instance for resolving namespace prefixes to namespace URIs, and vice-versa. Unfortunately, Java does not currently provide an implementation of the NamespaceContext interface, as reported in Sun's bug 6376058. It is relatively easy to write a simple implementation, which can optionally be included as either an inner-class or an anonymous inner-class. However, this can quickly become quite repetitive, especially when needing to support multiple namespace mappings in the same context.

My solution is the NamespaceContextMap class. It implements both NamespaceContext and Map<String, String>, making it very easy to configure and use. It accepts both prefix/URI pairs, as well as QName instances. Lookups are first resolved against the instance's configured list of mappings, then the default mappings as defined in NamespaceContext and XMLConstants. It also follows all the guidelines listed in the interface's Javadoc.

Here is some basic, example usage:

NamespaceContextMap ncm = new NamespaceContextMap();
ncm.put("xslt", "http://www.w3.org/1999/XSL/Transform");
ncm.put("xhtml", "http://www.w3.org/1999/xhtml");

XPathFactory xpf = XPathFactory.newInstance();
XPath xpath = xpf.newXPath();
xpath.setNamespaceContext(ncm);

// Do XPath operations here...

Two maps are internally maintained for performance regardless of lookup type - one map is keyed by prefix, the other by URI. The later are stored a Set with a backing List, which guarantees that multiple prefixes are supported per URI, and that the getPrefixes(String) method returns them in the order that they were added (FIFO) - essentially, an ordered map.

In the implementation, I struggled with finding a solid way of enforcing consistency and constraints. In particular, the entrySet(), keySet(), and values() methods of the Map interface make it very difficult (but not impossible) to intercept add/remove operations, something that I previously posted about in Java Collections Listeners. For now, these methods return unmodifiable collections.

XML PrettyPrint

While XML may commonly be sent as a single-line or without indentation for compactness and efficiency, it is usually most easily viewed with increased indentation at each level, commonly referred to as "pretty printing". This styling is presented by default in most web browsers, including both Mozilla Firefox and Microsoft Internet Explorer, as well as many IDEs and text editors. However, performing this formatting from an automated fashion within Java doesn't seem to be a feature that is readily available, stable, or easy to use. See the "Java 1.5 doesn't want to indent XML output" forum thread for some related discussion, including a copy of my solution.

My solution is an XSLT that reformats the XML with indentation, accounting for existing whitespace, and without any necessary references to "xml.apache.org". It also accepts configurable XSLT parameters for the indentation and newline character sequences. A PrettyPrint class is provided that handles loading the XSLT as a class resource, and returns a reusable, thread-safe Templates instance. For some details on this, including notes on how to chain it into an existing transformation or serialization for increased performance, see my previous post: XML and XSLT Tips and Tricks for Java.

As noted in my "Tips and Tricks" post, please be sure to upgrade to the latest version of Apache Xalan, 2.7.1 or newer. Otherwise, there is a particular issue where generated comments tend to disappear. This isn't an issue specific to my transformation, and can be reproduced even with an identity transformation. See the comment at the beginning of PrettyPrint.xslt for details.

XmlDateFormat

A frequent task I encounter is generating valid XML schema dateTime-formatted values. This format is a profile of the ISO 8601 standard, and is further detailed in RFC 3339. Unfortunately, Java doesn't currently provide a standard DateFormat that matches this specification. Included in my package is a XmlDateFormat class with a getDateFormat() method that returns a properly-configured DateFormat. As with most Format instances, the returned DateFormat instance is not guaranteed to be thread-safe and should not be re-used across threads.