Capturing complete HTTP requests - Echo Server

Background

I recently had a need to capture and inspect a complete HTTP request in preparation for developing a new web service. The main reason for this is that there were no real requirements for the requested service. It wasn't clear which parameters would be sent in the request, or exactly how the parameters would be named. It also wasn't clear how the parameters would be split between GET and POST parameters, or even additional HTTP request headers. Additionally, there was also some history with issues around various character encodings, so I needed to be able to capture a byte-accurate copy of the entire request, including the headers and the body.

Initially, I did not have good luck finding an existing tool or solution for this. My first attempt was to just host a basic web server, then to capture the data using Wireshark. Unfortunately, Wireshark primarily works with Ethernet packets. It supports higher-level viewing of many protocols including HTTP. It even includes options for re-assembly of both HTTP headers and bodies, re-assembly of chuncked transfer-coded bodies, and decompression of entity bodies. However, while I'm sure there are additional options and methods for getting it to work more like I desired, it just didn't seem like the right tool for this particular job - and that is no fault of Wireshark.

My other early attempt was to use the mod_dumpio module in Apache HTTP Server. The first issue with this was that all the output from all requests is simply mixed-in to the same error log file (along with other debugging / outputs), which would make the data very difficult for proper extraction. The second issue was that at least as far as I can tell, there can only be one error log file per <VirtualHost/>, which would have resulted in an excessive amount of data being captured.

I then started to look at a simple Java HTTP server to capture the desired data. I've written trivial HTTP servers before, but it quickly becomes non-trivial to properly handle and respond to all the possible options and variations - even just to accept a complete request (including body) from a client. Trying to avoid duplicating previous work, I looked at a number of existing web servers including Apache Tomcat, but did not find any that provided the desired logging options.

My solution

I started looking further into Jetty. (I previously used and blogged about Jetty in regards to a test platform for my MarkUtils-Web project.) I found that I could intercept the incoming requests byte-by-byte by extending Jetty's default Connector - SelectChannelConenctor, and then overloading the newEndPoint(…) method to return an extended SelectChannelEndPoint. Hooking into the SelectChannelEndPoint's fill(Buffer buffer) method allows for capturing of the complete HTTP request. Kudos to the Jetty developers for not making this difficult or impossible by marking everything as private or otherwise overly-restricted, as compared to an unfortunate practice followed by many other projects and companies!

With only a little extra code, each HTTP request is logged to a chosen directory as a pair of files, grouped by a time-based session ID. The first is a "meta" file that contains details that would not ordinarily be captured as part of the HTTP capture, including the session ID, server date, and remote address, host, and port. The second is the "content" file that contains the actual byte-by-byte capture of the HTTP request. While it is named as a ".txt" file for easy viewing, it is treated as binary and will accurately capture all requests, including those with binary payloads. The format also allows for easily re-playing the request to a server for additional testing, debugging, or analysis.

Finally, by implementing and registering an associated Handler, the request is not only captured, but is efficiently echoed back to the client - without ever needing to buffer or store the entire request. This echoed response starts with the contents of the "meta" file, including the session ID that can be used by the client to easily refer to the saved log file back on the server. The contents of a sample echoed response are shown below, and would appear in the body of a viewing web browser:

Session ID: 124f67a451d-6313f5e0
Date: Sun Nov 15 00:14:18 CST 2009
remoteAddr: 127.0.0.1
remoteHost: 127.0.0.1
remotePort: 23349
==========
POST /someUrlPath?someGetKey=someGetValue HTTP/1.1
Host: localhost:8080
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5 GTB5 (.NET CLR 3.5.30729)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Content-Type: application/x-www-form-urlencoded
Content-Length: 25

somePostKey=somePostValue

On the server-side, the first portion (above the '=' divider) is saved as 124f67a451d-6313f5e0-meta.txt, with the last portion (below the divider) saved as 124f67a451d-6313f5e0-content.txt.

The main class for this project is com.ziesemer.httpEchoServer.HttpEchoServer. It is written to be suitable for inclusion into other projects or uses, as visible from the included JUnit test. It also includes a main() method for direct use from the command-line, supporting arguments to control the port to listen on ("--port") and the directory to use to store the log files ("--logDir"). By default, Jetty is configured to listen on port 8080. If the specified port is unavailable, add "--allowDynamicPort" to configure the process to fall-back to a dynamically-chosen port if the specified port is already in-use.

Fiddler: Another alternative

Another alternative I later considered was Microsoft's Fiddler, a HTTP Debugging Proxy. While not open-source, it is freeware and extensible. It is also certainly a better match for my requirements than either Wireshark or Apache's mod_dumpio, and arguably even my solution described here. However, Fiddler still requires a server to answer the requests for it can monitor the traffic, and doesn't support echoing the request to the response. Fiddler does have many other features to offer that may prove useful, and is at least worth testing out.

Download

com.ziesemer.httpEchoServer is available on ziesemer.java.net under the GPL license, complete with source code, a compiled .jar, generated JavaDocs, and JUnit tests. Download the com.ziesemer.httpEchoServer-*.zip distribution from here. Please report any bugs or feature requests on the java.net Issue Tracker.

Executing JavaScript from HTML AJAX

AJAX, by definition, is known as "asynchronous JavaScript and XML". However, the use of XMLHttpRequest is not limited to XML, with JSON being a popular alternative. This post will focus on another alternative: standard HTML. Using HTML may be necessary when the HTML is already being generated server-side, with no available XML or JSON equivalent. If the response is to be viewable by the user, HTML may already be the desired result, and can avoid additional complexities of transforming XML or JSON into HTML.

Simply retrieving HTML from an AJAX request and inserting into the existing DOM of a web page is relatively easy. For my purpose, I used the .responseText attribute of the response on the XMLHttpRequest. Once the HTML string is obtained, it can be injected into an existing DOM node in the document by assigning it to the node's .innerHTML property.

Another possibility to consider is if the returned HTML is guaranteed to be valid XHTML. In this case, it may be possible to continue treating the response as XML and using the .responseXML attribute, and inject it into an existing DOM node in the document by calling the node's .importNode(…) method.

In either case, there is the complication of scripts that may be embedded within the returned HTML. These embedded scripts are not reliably executed, especially not consistently across web browsers. This is particularly complicated by Internet Explorer's handling of the <script/> tags as "NoScope" elements, which is detailed by Microsoft's John Sudds in MSDN's .innerHTML property documentation and referenced forum thread. In summary, embedded scripts should have the "defer" property set to true, and must follow a scoped element (e.g. <input type="hidden"/>) to work with IE. I have seen several pages that suggest setting the .innerHTML on a cloned node, including the above provisions for IE, then inserting back into the HTML DOM using methods such as .appendChild(…) or .replaceChild(…) to work across browsers. However, even this is not guaranteed to always work, and fails with Safari in particular.

Current Working Solution:

Rather than using a combination of browser "hacks" and/or browser detection, this solution instead recognizes and assumes that embedded scripts will not be executed automatically by the browser. After inclusion into the document, any embedded scripts are efficiently searched for and executed through JavaScript's eval(…) method. Shown below is an example / test case:

JavaScript source code:

 // http://blogger.ziesemer.com/2008/05/javascript-namespace-function.html
 namespace("com.ziesemer.demos").htmlAjax = function(){
    var pub = {};
    var contentDiv, oldContent;
    
    var html = "<div>New 1st-level dynamic text, presumably from AJAX response.</div>"
      + "<div id=\"com.ziesemer.demos.htmlAjax.middle\">More 1st-level dynamic text that should be replaced.</div>"
      + "<div>More 1st-level dynamic text, presumably from an AJAX response.</div>"
      + "<script type=\"text/javascript\">"
      + "document.getElementById(\"com.ziesemer.demos.htmlAjax.middle\").innerHTML = "
      + "\"2nd-level dynamic text, controlled by JavaScript presumably from an AJAX response.\";"
      // http://www.wwco.com/~wls/blog/2007/04/25/using-script-in-a-javascript-literal/
      + "<" + "/script>";
    
    pub.run = function(){
      contentDiv = document.getElementById("com.ziesemer.demos.htmlAjax.output");
      oldContent = contentDiv.innerHTML;
      contentDiv.innerHTML = html;
      
      var scripts = contentDiv.getElementsByTagName("script");
      // .text is necessary for IE.
      for(var i=0; i < scripts.length; i++){
        eval(scripts[i].innerHTML || scripts[i].text);
      }
    };
    
    pub.reset = function(){
      if(contentDiv && oldContent){
        contentDiv.innerHTML = oldContent;
      }
    };
    
    return pub;
  }();

HTML source code: (equivalent to using this page's View/Source)

<div style="border:1px solid; padding:0.5em;">
  <div><b>Dynamic test area:</b></div>
  <div id="com.ziesemer.demos.htmlAjax.output" style="border:1px solid;">
    Original static HTML text that should be replaced.
  </div>
  <p>
    <input type="button" value="Go!" onclick="com.ziesemer.demos.htmlAjax.run();"/>
    <input type="button" value="Reset" onclick="com.ziesemer.demos.htmlAjax.reset();"/>
  </p>
  
  <div>
    <b>Expected result:</b> (of what the above should appear as after the &quot;Go!&quot; button is clicked)
  </div>
  <div style="border:1px solid;">
    <div>New 1st-level dynamic text, presumably from AJAX response.</div>
    <div>2nd-level dynamic text, controlled by JavaScript presumably from an AJAX response.</div>
    <div>More 1st-level dynamic text, presumably from an AJAX response.</div>
  </div>
</div>

Demo:

This was tested on Mozilla Firefox 3.5; Internet Explorer 6, 7, and 8; Google Chrome 3; and Apple Safari 4 for Windows. If both blocks of above text don't match after clicking "Go!", you've probably found an issue.

A few things to note:

• This does not work with externally sourced scripts, i.e., those with a "src" attribute. This should normally not be a concern, as there shouldn't be any reason why these types of scripts can't be included in the page before the AJAX call is made. However, if supporting this is necessary, YUI's Get Utility will probably prove useful.
• As with most similar approaches, this does not work with scripts that utilize document.write(…). If document.write is used in this method, it will likely result in all existing content of the page being removed. Again, this should normally not be a concern, as use of document.write should generally be discouraged and avoided for a number of reasons. The suggested alternative is to use existing DOM elements as placeholders, as shown in my example above.
• Note the use of the "broken" closing </script> tag. This would not be necessary if the HTML string was actually obtained externally, e.g. from an AJAX call. It is only necessary as it is contained within a JavaScript string literal (as in the above example), as excellently described by Walt Stoneburner in his blog at http://www.wwco.com/~wls/blog/2007/04/25/using-script-in-a-javascript-literal/ (2007-04-25).
• Not really specific to this example, but the for loop around the returned "scripts" array would ideally be replaced by Array.forEach(…), e.g.:

  contentDiv.getElementsByTagName("script").forEach(function(script){
    // .text is necessary for IE.
    eval(script.innerHTML || script.text);
  });

  However, .forEach isn't supported until JavaScript 1.6. Firefox 3.5 supports JS 1.8.1. Chrome 1.0 and Safari 3.2 support JS 1.7. Even version 8 of IE still only supports JS 1.5!

Additional references:

• "Help me do the right DOM thing.". Martin Honnen, et al., 2006-11-02, comp.lang.javascript.
• "Execute JavaScript injected using innerHTML attribute, even with Safari", 2007-06-22.
• "Javascript script execution in innerHTML: the revenge". Giu­liani Vito, Ivan, 2006-03-07.

MarkUtils-Codec: Base64, URL, and other byte/char conversions

This is an overdue introduction of my latest addition to MarkUtils. MarkUtils-Codec could be considered a high-performance replacement for Apache Commons Codec. Like Commons Codec, this implementation has support for Base64, URL (a.k.a. Percent, and covered previously), and Hexadecimal encodings and decodings. Also like Commons Codec, this implementation utilizes a number of interfaces that allow various codecs to be used interchangeably. Unlike Commons Codec, this implementation is designed to be higher performing, as it is written for streaming use with the Buffer classes. The most significant advantage to this design is lower memory requirements and usage, especially when working with longer lengths of data.

MarkUtils-Codec is really a follow-up to one of my previous posts, Improving URLEncoder/URLDecoder Performance in Java. While the API I proposed and sample code I provided solved an immediate need, the lack of proper interfaces made it difficult to replace with other codecs, such as Base64. The options to plug-in to other standard streaming classes was also limited. For example, there was no clear way to create an InputStream that would read decoded data from encoded data. This library is meant as a complete replacement, as I have placed the "urlCodec" library in archival status.

Until I have a suitable place to host the Javadocs online, please reference them in the downloads available at ziesemer.dev.java.net.

The highest-level API interface is com.ziesemer.utils.codec.ICoder. Verbatim from the Javadoc, this is the "Base API for high-performance encoding and decoding between various Buffers. Supports conversions between ByteBuffers and CharBuffers through the IByteToCharEncoder and ICharToByteDecoder child interfaces. This API is similar in design to CharsetEncoder and CharsetDecoder."

Do note that the relation to the Charset classes may seem a bit backwards. When a character set is decoded, the input is bytes and the output is characters. The purpose of this library is to encode any data (as bytes) into character data that can safely be sent through various non-byte transports, e.g. HTTP forms. For this purpose, decoding takes characters and input and produces bytes as output.

Here is a simple example of supported direct usage, taking no advantage of streaming capabilities. This is included as one of the JUnit tests within the com.ziesemer.utils.codec.DemoTest class:

/**
 * Simple usage, taking no advantage of streaming capabilities.
 */
@Test
public void testDirectSimple() throws Exception{
  IByteToCharEncoder encoder = new URLEncoder();
  ICharToByteDecoder decoder = new URLDecoder();
  
  // Random test data.
  byte[] rawData = new byte[1 << 10];
  new Random().nextBytes(rawData);
  
  // Encode.
  CharBuffer cbOut = encoder.code(ByteBuffer.wrap(rawData));
  
  // Decode (round-trip).
  ByteBuffer bbOut = decoder.code(cbOut);
  
  // Verify.
  byte[] result = new byte[bbOut.remaining()];
  bbOut.get(result);
  Assert.assertArrayEquals(rawData, result);
}

Or an even simpler example, using convenience methods. Note that the Base64 codec can be replaced with URL (percent), Hex, or another provided codec:

byte[] sampleBytes = new byte[]{0, 1, 2, 3};
String enc = new Base64Encoder().encodeToString(sampleBytes);
System.out.println(enc); // Yields: AAECAw==
byte[] dec = new Base64Decoder().decodeToBytes(enc);
System.out.println(Arrays.equals(sampleBytes, dec)); // Yields: true

A number of input/output wrappers are also included in the "com.ziesemer.utils.codec.io" package, allowing for transparent use as a standard Java IO reader, writer, or stream. The signatures of the required constructors are also shown. Each class also provides an alternate constructor that can be used to fine-tune the read buffer size.

• CharDecoderInputStream(ICharToByteDecoder decoder, Reader reader)

  Reads raw bytes from encoded characters. Counter-part to CharEncoderReader. This is a pull-interface; CharDecoderWriter is the equivalent push-interface.

  Can be adapted to read characters to the consumer (instead of raw bytes) by wrapping in a InputStreamReader. This is only valid if the decoded form of the data is known to only contain valid characters. Can also be adapted to read bytes from a provider (instead of characters) by using a InputStreamReader as the Reader.

• CharDecoderWriter(ICharToByteDecoder decoder, OutputStream outputStream)

  Accepts encoded characters, and writes the raw bytes. Counter-part to CharEncoderOutputStream. This is a push-interface; CharDecoderInputStream is the equivalent pull-interface.

  Can be adapted to accept bytes from a provider (instead of characters) by wrapping in a OutputStreamWriter.

• CharEncoderOutputStream(IByteToCharEncoder encoder, Writer writer)

  Accepts raw bytes, and writes the encoded characters. Counter-part to CharDecoderWriter. This is a push-interface; CharEncoderReader is the equivalent pull-interface.

  Can be adapted to write bytes to the consumer (instead of characters) by using a OutputStreamWriter as the Writer. Can also be adapted to accept characters from the provider (instead of raw bytes) by wrapping in a OutputStreamWriter.

• CharEncoderReader(IByteToCharEncoder encoder, InputStream reader)

  Reads encoded characters from raw bytes. Counter-part to CharDecoderInputStream. This is a pull-interface; CharEncoderOutputStream is the equivalent push-interface.

  Can be adapted to read characters to the consumer (instead of raw bytes) by wrapping in a InputStreamReader.

Also included are a number of "character lists" (in the com.ziesemer.utils.codec.charLists package), particularly to support the different Base64 variations.

Please refer to the included JUnit tests (currently 169) for usage examples.

Download

com.ziesemer.utils.codec is available on ziesemer.java.net under the GPL license, complete with source code, a compiled .jar, generated JavaDocs, and a suite of JUnit tests. Download the com.ziesemer.utils.codec-*.zip distribution from here. Please report any bugs or feature requests on the java.net Issue Tracker.

"networking restart" issues, VLANs under Ubuntu

For this post, I'm using Ubuntu Linux 9.04 / "Jaunty Jackalope". This is somewhat a follow-up to my Ubuntu Linux Router Upgrade Project.

Errors during "/etc/init.d/networking restart"

First, assuming a statically-configured LAN for server use, with NetworkManager disabled:

/etc/network/interfaces:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
  address 192.168.1.1
  netmask 255.255.255.0

$ sudo /etc/init.d/networking restart
 * Reconfiguring network interfaces...                                   [ OK ]

The networking simply restarts, without showing any errors or warnings. However, this quickly changes once an additional network adapter is configured. This could be an additional physical adapter, but for my purposes, I had added a virtual LAN (VLAN) to work with my Dell PowerConnect 2716. For Ubuntu, this simply requires installing the "vlan" package, and defining the virtual LAN with an additional entry in /etc/network/interfaces. Note that this is done using a "<base name>.<VLAN ID>" naming scheme. As shown below, I'm adding an interface for VLAN 2:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
  address 192.168.1.1
  netmask 255.255.255.0

auto eth0.2
iface eth0.2 inet static
  address 192.168.2.1
  netmask 255.255.255.0

Please add a comment if you can find any official documentation that documents this functionality of the interfaces file, as I can't. However, this appears to be driven by "/etc/network/if-pre-up.d/vlan" and "/etc/network/if-post-down.d/vlan".

This also requires the "8021q" module, but at least in Jaunty, it is already available by default, as shown by "lsmod | grep 8021q".

This is where I started running into errors:

$ sudo /etc/init.d/networking restart
 * Reconfiguring network interfaces...
RTNETLINK answers: No such process
Removed VLAN -:eth0.2:-
 * if-up.d/mountnfs[eth0]: waiting for interface eth0.2 before doing NFS mounts
Set name-type for VLAN subsystem. Should be visible in /proc/net/vlan/config
Added VLAN with VID == 2 to IF -:eth0:-
                                                                         [ OK ]

This doesn't actually cause any issues, but it isn't right and should be fixed. I don't see any way to make "networking restart" give more verbose output. Additionally, nothing helpful appears in the logs. However, most of what the networking script does is call "ifdown -a --exclude=lo" and "ifup -a", where "-a" is "affect all interfaces marked auto". Fortunately, repeating this with adding "-v" for verbose mode yields some details:

$ sudo ifdown -av --exclude=lo && sudo ifup -av
Configuring interface eth0=eth0 (inet)
run-parts --verbose /etc/network/if-down.d
run-parts: executing /etc/network/if-down.d/avahi-autoipd
run-parts: executing /etc/network/if-down.d/wpasupplicant

ifconfig eth0 down
run-parts --verbose /etc/network/if-post-down.d
run-parts: executing /etc/network/if-post-down.d/avahi-daemon
run-parts: executing /etc/network/if-post-down.d/bridge
run-parts: executing /etc/network/if-post-down.d/vlan
run-parts: executing /etc/network/if-post-down.d/wireless-tools
run-parts: executing /etc/network/if-post-down.d/wpasupplicant
Configuring interface eth0.2=eth0.2 (inet)
run-parts --verbose /etc/network/if-down.d
run-parts: executing /etc/network/if-down.d/avahi-autoipd
RTNETLINK answers: No such process
run-parts: executing /etc/network/if-down.d/wpasupplicant

ifconfig eth0.2 down
run-parts --verbose /etc/network/if-post-down.d
run-parts: executing /etc/network/if-post-down.d/avahi-daemon
run-parts: executing /etc/network/if-post-down.d/bridge
run-parts: executing /etc/network/if-post-down.d/vlan
Removed VLAN -:eth0.2:-
run-parts: executing /etc/network/if-post-down.d/wireless-tools
run-parts: executing /etc/network/if-post-down.d/wpasupplicant
Configuring interface eth0=eth0 (inet)
run-parts --verbose /etc/network/if-pre-up.d
run-parts: executing /etc/network/if-pre-up.d/bridge
run-parts: executing /etc/network/if-pre-up.d/dhclient3-apparmor
run-parts: executing /etc/network/if-pre-up.d/vlan
run-parts: executing /etc/network/if-pre-up.d/wireless-tools
run-parts: executing /etc/network/if-pre-up.d/wpasupplicant

ifconfig eth0 192.168.1.1 netmask 255.255.255.0       up

run-parts --verbose /etc/network/if-up.d
run-parts: executing /etc/network/if-up.d/avahi-autoipd
run-parts: executing /etc/network/if-up.d/avahi-daemon
run-parts: executing /etc/network/if-up.d/ip
run-parts: executing /etc/network/if-up.d/mountnfs
 * if-up.d/mountnfs[eth0]: waiting for interface eth0.2 before doing NFS mounts
run-parts: executing /etc/network/if-up.d/ntpdate
run-parts: executing /etc/network/if-up.d/wpasupplicant
Configuring interface eth0.2=eth0.2 (inet)
run-parts --verbose /etc/network/if-pre-up.d
run-parts: executing /etc/network/if-pre-up.d/bridge
run-parts: executing /etc/network/if-pre-up.d/dhclient3-apparmor
run-parts: executing /etc/network/if-pre-up.d/vlan
Set name-type for VLAN subsystem. Should be visible in /proc/net/vlan/config
Added VLAN with VID == 2 to IF -:eth0:-
run-parts: executing /etc/network/if-pre-up.d/wireless-tools
run-parts: executing /etc/network/if-pre-up.d/wpasupplicant

ifconfig eth0.2 192.168.2.1 netmask 255.255.255.0       up

run-parts --verbose /etc/network/if-up.d
run-parts: executing /etc/network/if-up.d/avahi-autoipd
run-parts: executing /etc/network/if-up.d/avahi-daemon
run-parts: executing /etc/network/if-up.d/ip
run-parts: executing /etc/network/if-up.d/mountnfs
run-parts: executing /etc/network/if-up.d/ntpdate
run-parts: executing /etc/network/if-up.d/wpasupplicant

So the error is apparently coming from "/etc/network/if-down.d/avahi-autoipd". (Avahi is a free Zero configuration networking (zeroconf) implementation.) I temporarily edited the "avahi-autoipd" script to add "-x" to enable debugging by changing the first line to "#!/bin/sh -ex". What is happening is that this script first checks for the existence of a route matching "169.254.0.0/16", but doesn't check for what interface it is on. If it finds a matching route, even on another interface, it makes a call to delete the route, but only for the current interface. When this combination doesn't exist, the "RTNETLINK answers: No such process" message is given.

Resolution: The scripts could just be fixed to include the interface in the search. I opened a bug report to Ubuntu on this: #425854. However, a "server"-type system shouldn't be handling these automatic types of network routings, so I just disabled these scripts:

sudo chmod -x /etc/network/if-up.d/avahi-autoipd
sudo chmod -x /etc/network/if-down.d/avahi-autoipd

Shown fixed:

$ sudo /etc/init.d/networking restart
 * Reconfiguring network interfaces...
Removed VLAN -:eth0.2:-
 * if-up.d/mountnfs[eth0]: waiting for interface eth0.2 before doing NFS mounts
Set name-type for VLAN subsystem. Should be visible in /proc/net/vlan/config
Added VLAN with VID == 2 to IF -:eth0:-
                                                                         [ OK ]

DHCP server binding issues

Another issue I noticed with running "/etc/init.d/networking restart" is that the DHCP server ("dhcpd" / "dhcp3-server") quit responding to requests. (Previous posting on dhcp3-server.) Using Wireshark, I noticed that DHCP requests were still being received, but no responses were being sent. Nothing applicable is shown in any of the logs, other than that dhcpd isn't even acknowledging the requests after the interface is brought back up.

Tools such as "lsof" "netstat" show that dhcpd is still properly bound to the interfaces. Other applications such as the SSH server (sshd) don't have this issue, and continue accepting connections even after the interface goes down and comes back up, so I'm not sure why this is an issue for the DHCP server. However, I can't get things to resume working properly short of restarting the DHCP server using "sudo /etc/init.d/dhcp3-server restart" As I'm starting to work with IPv6, I noticed the exact same issue with radvd.

I wasn't exactly sure where the best place was to automate these restarts. I could probably add a script to "/etc/network/if-up.d/", but would then have to include a check that I wasn't unnecessarily restarting the servers for every interface, as they should only be concerned with the interface they are serving requests on, e.g. eth0. For now, I just added these commands to "/etc/network/interfaces":

# …

auto eth0
iface eth0 inet static
  address 192.168.1.1
  netmask 255.255.255.0
  post-up /etc/init.d/dhcp3-server restart
  post-up /etc/init.d/radvd restart

# …

Scripted hiding of Windows Updates under Vista

Similar to my last post, here is another UI issue with Windows Vista. Fortunately, this time I have a solution to offer.

Starting with Windows Vista, the "Windows Update" functionality is provided through Control Panel rather than Internet Explorer. In both versions, there is the ability to hide updates. While hidden updates can easily be restored, this feature allows for ignoring unnecessary updates so that they don't continually count towards the number of available updates that are displayed. For me, this includes the 34 "Windows Vista Ultimate Language Packs" that are currently available. Unfortunately, multiple-selection is not enabled in the "View available updates" dialog. There are checkboxes, including a checkbox on the header that can be used to select/unselect all shown updates, but the checkbox selections are used for the "Install" button only. The other options available from the context menu - "View details", "Copy details", and "Hide update" - can currently be applied only one-at-a-time. This means that hiding just the 34 language packs would require no fewer than 68 clicks!

Originally, I assumed that these hidden updates and other preferences would be stored in the Windows registry, or possibly in a file on the file system. They are in a file, but a database-type file that isn't directly editable: %SystemRoot%\SoftwareDistribution\DataStore\DataStore.edb. Fortunately, there is a comprehensive Windows API for viewing and editing this information, and it is even easily available to scripting through the Windows Scripting Host and languages such as JScript. Microsoft's reference is located on MSDN: Windows Update Agent API.

Here is my resulting script that automatically hides all the "Windows Vista Ultimate Language Packs":

var updateSession = WScript.CreateObject("Microsoft.Update.Session");
var updateSearcher = updateSession.CreateUpdateSearcher();
updateSearcher.Online = false;

var searchResult = updateSearcher.Search("CategoryIDs Contains 'a901c1bd-989c-45c6-8da0-8dde8dbb69e0' And IsInstalled=0");

for(var i=0; i<searchResult.Updates.Count; i++){
  var update = searchResult.Updates.Item(i);
  WScript.echo("Hiding update: " + update.Title);
  update.IsHidden = true;
}

If you're not familiar with WSH, this can be simply executed as saving it as a *.js file, then double-clicking. A better option is to execute the file from a command-line with cscript. This will cause the output messages to be written to the standard output, instead of popping up a message box that must be acknowledged for each message. Also, since this script is making administrative changes to the system, it must be executed as an administrator.

"a901c1bd-989c-45c6-8da0-8dde8dbb69e0" is the ICategory.CategoryID for "Windows Vista Ultimate Language Packs". (This ICategory happens to have a .Type of "Product".) A similar script can easily be used to perform operations on other sets of updates by simply modifying the search query.

For the above example, the changes can be reverted by updating the script to executed update.IsHidden = false; (instead of true), then re-executing the script. Alternatively, here the Windows Vista GUI works a little better: By clicking on "Restore hidden updates" from the side panel in Windows Update, the "Restore" button operates on the checkbox selection - allowing all hidden updates to quickly be restored with 2 clicks if desired.

Finally, here is an extended example that doesn't change anything, but displays some of the many details that are available through this API. First, it displays all the updates grouped and nested by category. Note that some updates belong to more than one category. Finally, it displays all available updates in a "flat" view, without using categories.

var updateSession = WScript.CreateObject("Microsoft.Update.Session");
var updateSearcher = updateSession.CreateUpdateSearcher();
updateSearcher.Online = false;

var searchResult = updateSearcher.Search("IsInstalled=1 or IsInstalled=0");

var describeCategory = function(cat, depth){
  var pad = new Array(depth + 1).join("  ");
  WScript.echo(pad + depth + ": " + cat + ", " + cat.CategoryID + ", " + cat.Name + ", " + cat.Type);

  for(var i=0; i<cat.Children.Count; i++){
    var child = cat.Children.Item(i);
    describeCategory(child, depth + 1);
  }
  
  for(var i=0; i<cat.Updates.Count; i++){
    var update = cat.Updates.Item(i);
    WScript.echo(pad + "  " + describeUpdate(update, pad + "  "));
  }
};

var describeUpdate = function(update, pad){
  var u = update;
  var np = "\n" + (pad || "") + "  ";
  return u.Title
    + np + "Type: " + u.Type
    //+ np + "Description: " + u.Description
    + np + "IsInstalled: " + u.IsInstalled
    + np + "IsDownloaded: " + u.IsDownloaded
    + np + "IsHidden: " + u.IsHidden
    + np + "AutoSelectOnWebSites: " + u.AutoSelectOnWebSites;
};

for(var i=0; i<searchResult.RootCategories.Count; i++){
  var category = searchResult.RootCategories.Item(i);
  describeCategory(category, 1);
}

WScript.echo("\n");

for(var i=0; i<searchResult.Updates.Count; i++){
  var update = searchResult.Updates.Item(i);
  WScript.echo(describeUpdate(update));
}

According to the IUpdateSearcher.Search documentation, the default search criteria is "IsInstalled = 0 and IsHidden = 0". Unfortunately, there doesn't seem to be a simple option to short-circuit the evaluator to just return all available updates, e.g. "" or "1=1". So far now, "IsInstalled=1 or IsInstalled=0" results in all updates being displayed. The only other note concerning the above example is that the "description" line is commented out in the describeUpdate function only because it can be rather verbose, and make the overall output difficult to read. Feel free to uncomment it to view the details, as well as adding additional lines for all the other properties available from IUpdate.

IME and other Vista Start Menu Annoyances

Recently I noticed some context menu items under Windows Vista that I didn't previously recall. I first noticed this after right-clicking on the text box on the start menu. However, these items appear on most text-input fields, including Notepad, any of the text fields in the default File Open/Save dialog boxes from almost any application, or even right-clicking on the text while renaming a file from Windows Explorer. The complete context menu displayed is shown below:

These extra items - "Right to left Reading order", "Show Unicode control characters", "Insert Unicode control character", "Open IME", and "Reconversion" - practically double both the width and the height of the context menu. To make things even more annoying, there doesn't appear to be any way to disable even the display of these items.

IME stands for Input Method Editor, which should allow for the input of additional characters and symbols that may not be found on the keyboard. However, the "Open IME" option doesn't even appear to function - even though clicking it toggles the option between "Open IME" and "Close IME". I've also never seen the "Reconversion" item enabled, even when text is selected. I don't even know what it is for. While "IME" contains a short 1-sentence definition in the Windows Help, searching for "reconversion" doesn't give any results. (Regardless of the menu functionality, a user should still be able to input any character code through the use of "alt codes".)

At first, I thought I accidentally installed an additional language pack, which is method that would have added these items under previous operating systems such as Windows XP. Another thought was that these options were introduced with Windows Vista Service Pack 2, or that I had accidentally installed a SP2 version designed for multiple languages. However, I found that these options are immediately available in default installations of Vista, including both the English Ultimate and Business editions.

I was a bit surprised at how few results related to this menu and Windows Vista show up on Google. Most of those are several years old (around 2007), the most comprehensive and recent of which I found on this forum thread. None that I've found contain any solutions, and apparently these menu items still exist in Windows 7 as well.

Start Menu size / width

Another issue I just noticed is that Vista's new start menu is not at all resizable, and is sometimes too small / too narrow to display items with longer names. To make matters worse, there are no scroll bars or other apparent functionality for viewing the cut-off text. This is even the case with some of Microsoft's own programs and installed shortcuts. For example, the last 2 characters of "Microsoft Windows Performance Toolkit" do not fit within the fixed width of the "All Programs" menu.

Unfortunately, again, there doesn't appear to be any resolution or reasonable work-around to this issue, short of renaming the longer names to shorter ones. The two most relevant pages I found from searching the web are this forum thread, and this Microsoft Answers forum thread where a Microsoft support engineer's response is only to switch to the classic Start menu instead.

In summary...

"UI blunders"?

Alltel / Verizon Wireless Internet: Disconnections, DNS, and other issues

About 9 months ago, I made a few posts around my Ubuntu Linux router upgrade project. Since then, Ubuntu has more than met the need and the goal. While my original notes were for 8.04 ("Hardy Heron"), they have since been successfully repeated on 8.10 ("Intrepid Ibex") and 9.04 ("Jaunty Jackalope").

Unfortunately, I cannot say I've had the same success with Alltel's wireless Internet service. (Alltel has since been acquired by Verizon Wireless, though there hasn't yet been any noticeable changes as a customer.) To be fair, I've been pretty impressed with Alltel's voice service. Their data services also seem to work well for temporary / mobile usage, especially for Blackberries and other smart phones. However, I often fail to comprehend how this service can be deservingly called "Internet" given some of the issues.

Disconnections

The most frustrating issue is the frequency and apparent length of disconnections. This has been an ongoing issue since I first started using the service back in September. I've learned that there is a hard 12-hour timeout, after which any connection will be terminated regardless of usage. While this is certainly an annoyance, it is an issue that I could live with - if it was the only issue. However, on a much to frequent basis, I'll seemingly get stuck in continued rounds of disconnections - repeatedly getting disconnected after periods of anywhere from a few seconds to a few minutes. This can go on for minutes or hours, and usually happens from at least once a week to several times a day. Even going to the extreme of disconnecting the wireless modem and "giving it a rest" for 10 minutes does not help. Each disconnection is rather disruptive, especially as each new connection brings a new IP address - which makes it practically impossible to gracefully resume many types of connections.

I am not the only one having these issues. A few Google searches show a number of other users with related issues with Alltel (and also Verizon), but no currently known solutions.

This is not a signal strength issue, as the reported RSSI is almost always at one of the two best reportable values: -63 or -47 dBm, or 4/5 - 5/5 bars. I've tried adding an external antenna, which made no difference. I'm unable to find any correlations to the time of day or other environmental factors.

Alltel technical support is all to eager to blame this on my use of Linux. However, the problems are identical if not worse when using Windows and their QuickLink Mobile software (made by SmithMicro Software). If anything, Linux is better able to handle the connection and any necessary re-dialing. Trying the data card on different computers (a desktop and 3 available laptops) does not change anything. My current data card is a UTStarcom / Pantech UM175AL. At one point I tried a loaner Huawei EC228 card, which also made no difference in the issues. Worse, Alltel got the accounts mixed up and ended changing the phone and account numbers on the UM175 that I had to keep, and I received a threat of legal action in the mail for breach of contract!

I've seen a number of theories for the cause of these disconnections, and have tried various solutions to them all, but without any success. This includes setting various MTU values, asyncmap settings, adjusting for LCP echo failures / timeouts, tweaking the AT commands sent in the modem chat script, and making sure that there isn't any private-address traffic being sent over the connection.

I've contacted Alltel's technical support many times to try to resolve this issue, always speaking with their separate "data card support" department, and probably totaling at least 20 hours of phone time. The original theory was that the data card was bouncing between Alltel and Sprint towers, and loosing the connection each time this happened. They updated the preferred roaming list (PRL) on the device several times, without any affect. An interesting fix they tried was setting the device to "static" mode. Supposedly this doesn't refer to a static vs. dynamic IP address, but it did result in obtaining a persistent IP address. This also did appear to help with the connection stability - but not perfectly. Most critically, this mode caused an unfortunate side effect: I could no longer receive any incoming connections. Alltel's response was that incoming connections are not supported. Many ISP's, especially residential, block common server ports - such as 80 for HTTP and 25 for SMTP. This is a debatable practice, but doesn't affect "simple" activities such as web browsing and email. However, in this "static" mode, it was as if I was behind another firewall with all incoming ports being blocked - affecting a number of Internet uses including remote desktop applications, VPN clients, instant-messaging file transfers with family and friends, or even online games.

Some of my relatives also experience the same issues. However, with pretty much only checking their email a few times per week, they seldom notice the issues. Eventually, we moved closer to Appleton - about 15 miles closer - and while this has slightly improved things, it is still not acceptable. Now that other Internet options are available, including cable and DSL, I would have switched already if we weren't stuck with the 2 year contract, and with over a year remaining yet. We would have better service at half the price.

DNS Concerns

This is another issue I found while checking-up on some things in the aftermath of the DNS cache poisoning vulnerability reported by Dan Kaminsky. The first issue I noticed was an alert on a report from the ICSI Netalyzer hosted by UC Berkeley: No DNS Port Randomization. There have been too many issues with ISP DNS servers in the past, e.g. DNS hijacking by several companies including Charter Communications, Comcast, and Time Warner. Due to this and other various ISP issues, along with being able to improve performance through local caching, I typically run my own DNS server for my home network. I thought I may have had it set to forward all DNS requests to the DNS server address received through DHCP from the ISP, and that I'd just change it to do its own recursive resolution.

This is where things got interesting. Alltel is not only playing tricks with their own DHCP-advertised DNS servers, but are intercepting all outgoing DNS traffic (UDP port 53) and acting as a transparent proxy. This is quickly confirmed through one of the same tools that can be used to check for the DNS cache poisoning vulnerability: porttest.dns-oarc.net. On my Alltel PPP connection, the peer's IP address is currently 75.116.231.29. The client IPs seen by the port test tool are 75.116.63.155 and 75.116.63.156. Neither of these addresses are mine, which are in a completely different subnet. None of these addresses currently have pointer records (PTR) registered, though I recall seeing them resolve to windstream.net on Windstream Communications only about a week ago. ARIN WHOIS does show that all these addresses belong to Alltel Communications.

The ICSI Netalyzer results seem to reflect this as well. Two points listed under Reachability Tests / General Connectivity were "A DNS proxy or firewall caused the applet's direct DNS request to be sent from another address. Instead of your IP, the request came from 75.116.63.156" and "A DNS proxy or firewall generated a new request rather than passing the applet's request unmodified". Interestingly, both were listed simply as information rather than warnings. Many people tout OpenDNS or other 3rd-party DNS servers in situations like this. Even though OpenDNS follows some of the same practices, such as redirecting invalid lookups instead of sending NXDOMAIN by default, at least they provide options to control and disable this functionality. Unfortunately, it is impossible to query any specific DNS server - OpenDNS or otherwise - while Alltel is intercepting these requests. The only work-around I can think of short of Alltel fixing this would be finding a DNS server that provides service on a non-standard port, or sending all DNS queries through some available VPN to another location. I found this forum thread from February that showed a few other users complaining of similar issues with Alltel redirecting DNS requests. While the issue apparently had been resolved, it only appears to have been temporary. Since there is nothing as a user that I can currently do to avoid these issues, I can only hope that Alltel is on top of things, including having other measures in place for avoiding DNS poisoning.

Update (2009-08-26):

Alltel finally decided to do something for me. As previously mentioned, I've lost count of how many calls I've made to Alltel technical support. Several times, I also visited the Alltel corporate retail store, and brought with me printed connection log that showed my excessive number of connections and disconnections. About a week ago, I made one last trip to the corporate retail store. After talking with the same store manager I happened to previously speak with a few months ago, he agreed to cancel my contract with no early termination fee. It wasn't an ideal solution for either of us - I'd rather have a working, mobile connection, and I'd hope that they'd value and want to keep my business. However, at least now I don't have to continue paying for service that doesn't work. I will soon be signed-up with faster, more reliable, and less expensive DSL service.

Related postings:

• EVDOforums.com: Alltel EVDO Service: Disconnections, DNS proxying, and other issues
• WirelessAdvisor.com foruums: Pantech UM175AL Disconnect
• OpenDNS:
  • OpenDNS Community Forums: Domain Name System (DNS) troubles: Verizon broadband wireless
  • OpenDNS Idea Bank:
    • Listen on another port
    • Offer an alternate port for DNS queries
    • Alternate port
    • DNS On Nonstandard Ports (Defeat stupid ISP proxies and forced DNS)

JScript for Ping, Renew IP and Network Info / Repair

I was having a frustrating issue where a remote Windows XP computer that I need to Remote Desktop into would occasionally become unresponsive. This issue already required me to drive a number of late-night trips to temporarily remedy the issue. While I've not yet found the root cause, the main issue seems to be that something becomes "disconnected" between the network adapter and the operating system. While the Windows "Status" dialog on the network adapter shows "Connected" and packets being sent and received, I am not even able to ping the default gateway. Windows Firewall is not enabled. The only other suspect software I need to rule out yet is an anti-virus service. Rebooting the computer always helped, but is rather drastic. Choosing the "Repair" option does not work, and shows a message about failing while renewing the IP address. Interestingly, the fastest temporary "fix" I found is to run "ipconfig /release" followed by "ipconfig /renew".

While this computer is probably due for a re-installation, it is not an immediate option. In the meantime, I came up with a Windows Script Host (WSH) script to automatically detect and repair this issue.

Update: The root cause is certainly related to "Symantec Endpoint Protection" 11.0.x (Antivirus). If Symantec is completely removed, the problem disappears. When reinstalled, the problem comes back. The "Network Thread Protection Logs" would show everything being blocked without reason, until reset by an IP renew or a reboot. (The "Rule" cited is either "Block local file sharing", even though it blocks ALL ports - or "Block all other traffic".) At least at one point, this was affecting dozens of desktops. However, Symantec needs to stay per corporate policy...

As previously posted, I chose to use JScript for this task (particularly over VBScript). Everything but the logging is essentially WMI (Windows Management Instrumentation) calls. If I hadn't coded for all of the extra logging, the code would have probably been less than half the current size. It basically finds a given network adapter, and attempts to ping the configured default gateway. If unsuccessful, it will perform a DHCP release and renew operation.

/** 
 * @author Mark A. Ziesemer, www.ziesemer.com, 2009-06-25
 */
(function(){
  
  var fso = new ActiveXObject("Scripting.FileSystemObject");
  var wmi = GetObject("winmgmts://localhost/root/cimv2");
  
  var getNetworkAdapter = function(name){
    var adapters = new Enumerator(wmi.ExecQuery(
      "Select * From Win32_NetworkAdapter Where NetConnectionID='" + name + "'"));
    if(!adapters.atEnd()){
      return adapters.item();
    }
  };
  
  var getAdapterConfig = function(index){
    var configs = new Enumerator(wmi.ExecQuery(
      "Select * From Win32_NetworkAdapterConfiguration Where Index=" + index));
    if(!configs.atEnd()){
      return configs.item();
    }
  };
  
  var ping = function(host){
    var results = new Enumerator(wmi.ExecQuery(
      "Select * From Win32_PingStatus Where Address = '" + host + "'"));
    if(!results.atEnd()){
      var result = results.item();
      return result.StatusCode;
    }
  };
  
  var logWriter = fso.OpenTextFile("PingTest.log", 8, true);
  try{
    var log = function(msg){
      logWriter.WriteLine(new Date() + " - " + msg);
    };
    
    var adapter = getNetworkAdapter("Local Area Connection");
    if(adapter){
      log("Found network adapter:"
        + "\r\n\tAvailability: " + adapter.Availability
        + "\r\n\tCaption: " + adapter.Caption
        + "\r\n\tConfigManagerErrorCode: " + adapter.ConfigManagerErrorCode
        + "\r\n\tErrorDescription: " + adapter.ErrorDescription
        + "\r\n\tLastErrorCode: " + adapter.LastErrorCode
        + "\r\n\tMACAddress: " + adapter.MACAddress
        + "\r\n\tName: " + adapter.Name
        + "\r\n\tNetConnectionID: " + adapter.NetConnectionID
        + "\r\n\tNetConnectionStatus: " + adapter.NetConnectionStatus
        + "\r\n\tStatus: " + adapter.Status
        + "\r\n\tTimeOfLastReset: " + adapter.TimeOfLastReset);
    }else{
      log("No suitable network adapter found.  Quitting...");
      return;
    }
    
    var adapterConfig = getAdapterConfig(adapter.Index);
    if(adapterConfig){
      log("Found network adapter config:"
        + "\r\n\tIPAddress: " + adapterConfig.IPAddress.toArray()
        + "\r\n\tDefaultIPGateway: " + adapterConfig.DefaultIPGateway.toArray());
    }else{
      log("No suitable network adapter configuration found.  Quitting...");
      return;
    }
    
    var pingResult = ping(adapterConfig.DefaultIPGateway.toArray()[0]);
    log("Ping result: " + pingResult);
    if(pingResult){
      var releaseResult = adapterConfig.ReleaseDHCPLease();
      if(releaseResult){
        log("Unexpected ReleaseDHCPLease() result: " + releaseResult);
        return;
      }
      var renewResult = adapterConfig.RenewDHCPLease();
      if(renewResult){
        log("Unexpected RenewDHCPLease() result: " + renewResult);
        return;
      }
      log("Successfully released and renewed IP: " + adapterConfig.IPAddress.toArray());
    }
  }finally{
    logWriter.WriteLine();
    logWriter.Close();
  }
  
})();

I added it to the Windows Task Scheduler, and set it to run every 5 minutes. While not available through the Windows UI, the schtasks command allows for scheduling as the "SYSTEM" user, assuming you have administrative rights. This eliminates the need for your password, and prevents the job from breaking when you change your password:

> schtasks /create /tn "PingTask" /tr "cscript C:\<path>\PingTest.js" /sc MINUTE /mo 5 /st 00:00:00 /ru System
INFO: The Schedule Task "PingTask" will be created under user name ("NT AUTHORITY\SYSTEM").
SUCCESS: The scheduled task "PingTask" has successfully been created.

While the problem detection isn't based on the packet count statistics that I observed from the Windows UI, I thought it would also be useful information to log to see if there is a correlation to the "disconnects". Unfortunately, I couldn't find an easy way to obtain these statistics - either through WMI or the command line. I did find a few places to obtain protocol/transport-level statistics such as TCP, but nothing at the link-layer (Ethernet) level. The only involved discussion I could find on this was this newsgroup posting to microsoft.public.win32.programmer.wmi back from 2006 which implies that these statistics are only available through native code.

PuTTY and SSH port forwarding corruption

PuTTY seems to have some serious issues with SSH port forwarding. http://www.chiark.greenend.org.uk/~sgtatham/putty/wishlist/portfwd-corrupt.html seems to document exactly the same issues I've been having.

I was doing a simple read of XML data in Java from a remote HTTP server, using a SSH tunnel created by PuTTY. Commonly but sporadically, the HttpURLConnection appeared to close prematurely, resulting in only partially-received content. Switching between "dynamic" and "static" (Local) tunnel destinations did not have any effect.

I've also been having issues with PuTTY crashing on unclean closes.

The latest development snapshot (2009-05-27:r8577) does not appear to help. The "portfwd-corrupt" bug was filed in mid-2003, and there hasn't been a new release in over 2 years. (0.60 was released on 2007-04-29.) All of the PuTTY forks and variations I tried have the same issue. (See PuTTY on Wikipedia.) TeraTerm (Wikipedia) does not suffer the same issue, but does not appear to support dynamic tunneling using SOCKS.

My working solution: Use OpenSSH (Wikipedia), which also works on Windows through the use of Cygwin (Wikipedia). Just be aware that this is a command-line solution. While there are probably various front-ends available, I'd be surprised if there was one that didn't limit the available options that ssh has to offer. Ironically, I am using PuTTYcyg to run ssh, as it is much better than the Windows' command prompt. By entering ssh with the desired arguments as the command instead of the default "-" for the login shell, it saves a bash process from running.

MarkUtils-PacProxySelector for Java

Many computer networks make use of proxy servers for web and Internet connectivity. This is especially true for business and other organization networks, where their use is required by security and other policies. Due to the typical use of proxy servers, they are often thought of in terms of "restricting access". Instead, they should be thought of in the proper terms of a means of "providing access". Even outside of typical corporate environments, proxy servers can be invaluable for testing and debugging, as well as used as a type of VPN between private networks.

Most web browsers and other networked applications support directing traffic through one or more proxy servers. The typical configuration dialog looks like this, as shown from Mozilla Firefox:

Of the "manual" options, "HTTP" and "SSL" (TLS) are the most basic and common, followed by "FTP". "Gopher" is rarely used anymore - it has already been dropped by Microsoft Internet Explorer, and may be dropped in Mozilla Firefox 4.0. "SOCKS" is arguably the most powerful, supporting any of the above protocols in addition to any other TCP- or UDP-based protocol. (If SOCKS is configured and supported, none of the other protocols need to be configured.)

Unfortunately, directing an entire networks' traffic through a single proxy server can quickly cause a bottle-neck and a common point for failure. This is especially true when all LAN traffic is also sent through the proxy. (In ideal network traffic patterns, there should be many more multiples of LAN traffic over Internet traffic.) Some of the performance and availability concerns can be addressed through DNS or other load balancing. Another common attempt is to configure the "no proxy for" / exception list. Unfortunately, there are some severe limitations to this design. First, the list must be kept up-to-date as the network configuration changes. (There are various tools for this.) More significantly, there are many desired configurations that cannot be accounted for. For example, what if a list is needed for the servers that should be sent to a proxy server, rather than skipping the proxy server (reverse logic)? Or what if traffic must be split among multiple proxy servers, depending upon the destination or other parameters?

The solution to all of the above and other similar concerns is through the use of the last option shown above, and probably the most overlooked: "Automatic proxy configuration URL". This option is also known as proxy auto-config, or PAC, and was introduced into Netscape Navigator as early as 1996. A PAC file only needs to contain an implementation of a JavaScript function, FindProxyForURL(url, host). From here, the full power of JavaScript can be used, including regular expressions, associative arrays, and closures, as well as a number of predefined helper functions specific to PAC. Within the PAC function, various load balancing and black- or white-listing tasks can be performed, optionally by maintaining internal state. A list of multiple proxies may also be returned for attempts by the client. The PAC file is loaded from a URL (including local file:// URLs), where it can be centrally maintained and updated. The PAC file may be cached by the web browser or other client, but should respect the cache settings sent in the HTTP headers if retrieved through HTTP. Alternatively, a chrome:// URL can even be used, allowing for the PAC file to be maintained within a Firefox extension, and updated through Firefox's standard auto-update process for extensions.

Java support

Java supports many of the same above proxy options, mostly through the use of system properties. For full details, see the tech note at java.sun.com, Java Networking and Proxies. These settings affect any communications made through URLConnection, Socket, and possibly other network-related classes. Previously, the proxy configuration options were limited to the "manual" options listed above, with separate options for HTTP, HTTPS, FTP, and SOCKS. However, Java 1.5/5.0 introduced the Proxy and ProxySelector classes. A default ProxySelector can be configured for the current JVM by calling ProxySelector.setDefault(ProxySelector).

Unfortunately, Java does not currently provide any visible support for proxy auto-config (PAC) files. However, the ProxySelector's List<Proxy> select(URI uri) method looks and works very similar to the PAC's FindProxyForURL(url, host) function. The most notable difference is that it is strongly-typed to standard Java classes. As part of my MarkUtils collection, I created MarkUtils-PacProxySelector to provide a ProxySelector implementation that works with PAC files.

Since the PAC files are based on JavaScript, the ability to evaluate JavaScript is required. Fortunately, this is easily done through Java, especially with the introduction of the Java Scripting API in Java 1.6/6.0 (JSR-223). Java 1.6 bundles an internal version of the Mozilla Rhino implementation of JavaScript for Java, based on 1.6R2. Unfortunately, Java doesn't expose all the features of JavaScript or Rhino directly through the scripting API, some of which are required to implement the PAC functionality in a compatible fashion. This includes defining top-level bindings in the JavaScript environment to Java functions, which is directly supported in Rhino by adding a binding to a FunctionObject - a class to which there is no publicly visible match in the JDK. While it is probably possible to hack a work-around to this, my current implementation utilizes Rhino directly. Besides taking advantage of the improvements in the latest version of Rhino (currently 1.7R2), this allows the utility to be easily used with both Java 1.5/5.0 and 1.6/6.0. (However, note that JSR-223 is unofficially supported under Java 1.5/5.0 as well by downloading and including the .jar's from the reference implementation.) Using Rhino directly also avoids some potential security issues, which I reported in Sun Bug 6782031 and Mozilla Bug 468385.

As commented in the pom.xml file, Mozilla Rhino is currently not available through the central repository, a Mozilla repository, or any other "official" repository. I've added a dependency to it as "org.mozilla.javascript : rhino : 1.7R2". For this to work properly, Rhino will need to be downloaded and installed into a local repository as named above.

In addition to the standard PAC methods, PacProxySelector supports an added function called "connectFailed" to take advantage of the connectFailed(URI, SocketAddress, IOException) functionality on ProxySelector. The JavaScript method is called with the same arguments as on ProxySelector, just with the .toString() representations of each of the three parameters. The PAC file could then store this information within internal state to possibly affect future calls to FindProxyForURL.

For the most flexibility, the constructor to PacProxySelector accepts a Reader, which should read from a PAC file. There is also a public static configureFromProperties() method that returns a ProxySelector, assuming that the path to a PAC file is stored as either a Java system or environment property named "proxy.autoConfig", similar to the other network properties. After obtaining an instance from either the constructor or the method, it should be passed to ProxySelector.setDefault(ProxySelector), unless otherwise used directly. Alternatively, a setDefaultFromProperties() convenience method is provided to do this in one call.

I wrote this in mind for plugging into other Java applications. Ideally, the JDK would provide a system property that accepts the classname for the default ProxySelector or some other method for setting the default outside of a function call within the code, but this is currently not the case. However, all that has to be done is finding a way to execute one of the above configuration options from the desired Java application before network access is attempted. I've successfully written a plugin for Oracle SQL Developer that does exactly this. The same is also possible for Eclipse, though it requires patching of some of the plugins due to the current infrastructure. (See Eclipse bug 257443.) Alternatively, PacProxySelector provides a main method that calls setDefaultFromProperties() before chaining execution to another program's main method. See the included Javadoc for details.

Download

com.ziesemer.utils.pacProxySelector is available on ziesemer.java.net under the GPL license, complete with source code, a compiled .jar, generated JavaDocs, and a suite of JUnit tests. Download the com.ziesemer.utils.pacProxySelector-*.zip distribution from here. Please report any bugs or feature requests on the java.net Issue Tracker.

Handling XML Encodings with MarkUtils

Especially in today's focus on higher-level languages, lower-level details are often overlooked. However, character encodings are one such detail that must be remembered, particularly when working with interfaces such as web services, which may be communicating with a large variety of different platforms and languages - both programming and written.

US-ASCII is probably one of the most significant character encodings, but not the earliest. ASCII is the predecessor to the ISO/IEC 646 standard, and is a subset of the Unicode standard, particularly UTF-8. (See also: English in computing.) US-ASCII by itself is unable to represent more than its defined 95 printable characters - 62 of these being [A-Za-z0-9], with the remainder being used for punctuation and other miscellaneous symbols. This limited character set makes it impossible to accurately represent other Latin-based languages such as Spanish, French, and German. "Extended ASCII" such as ISO/IEC 8859-1 provide complete or near-complete coverage of these and other alphabets, but still fail to account for other characters. Using a character encoding that provides full support for the Universal Character Set (UCS) such as UTF-8 is the recommended solution to this and other related issues.

Many character encodings share the same byte representations for the basic English character set [A-Za-z] and [0-9]. These include US-ASCII, ISO-8859-1, UTF-8, and others. For example, in these character encodings, the following mappings are always true:

Character	Decimal	Hexadecimal
0	48	0x30
9	57	0x39
A	65	0x41
Z	90	0x5A
a	97	0x61
z	122	0x7A

Other character encodings, such as EBCDIC, are completely different.

Even when receiving a byte stream from one of the "compatible" encodings, the correct encoding still needs to be determined, as each encoding is handled slightly differently. Certain byte sequences that may be allowed in US-ASCII variants may be invalid in UTF-8. UTF-16 and UTF-32 are easily converted to and from UTF-8, but may be sent with different byte orderings - either big-endian or little-endian.

Encodings in XML

In the W3C's recommendation for XML (essentially the XML standard), the use of character encodings is specifically addressed in section 4.3.3. However, since the encoding is defined with the XML, the encoding declaration itself is also encoded. This is also addressed in the appendix, Autodetection of Character Encodings (Non-Normative):

The XML encoding declaration functions as an internal label on each entity, indicating which character encoding is in use. Before an XML processor can read the internal label, however, it apparently has to know what character encoding is in use—which is what the internal label is trying to indicate. In the general case, this is a hopeless situation. It is not entirely hopeless in XML, however, because XML limits the general case in two ways: each implementation is assumed to support only a finite set of character encodings, and the XML encoding declaration is restricted in position and content in order to make it feasible to autodetect the character encoding in use in each entity in normal cases.

Essentially, the character encoding in use may be determined by a combination of the Byte-order mark (if present), and the value of the "encoding" attribute in the XML declaration of the prolog. However, there may also be external encoding information available from higher-level protocols that must be factored into the determination. This includes the MIME Content-Type sent over HTTP, as defined in RFC 3023.

In Apache Xerces2-J, this is mostly handled within the implementation, particularly by org.apache.xerces.impl.XMLEntityManager.createReader(…). Similar code is also in org.apache.xerces.xinclude.XIncludeTextReader, which also accounts for at least most of the RFC 3023 rules, but only if the input source is a "system Id" which results in the use of a URLConnection.

Addition to MarkUtils-XML

I had a need to meet these same requirements on an input stream to determine the encoding, but before handing off the processing directly to Xerces or another XML processor. I also wanted a flexible solution that would follow the RFC 3023 recommendation, but would be able to work with a variety of sources - including a URLConnection or a HttpServletRequest. I found no existing and public API that met these requirements, so I made my own - and am making it available for public use as part of MarkUtils-XML.

Below is an outline of the public API. While it may first appear to be a bit lengthy, it is designed to be flexible and usable in high-performance environments. None of the Javadoc documentation is included here for brevity. The complete source code, along with a compiled .jar, generated Javadocs, and a comprehensive suite of JUnit tests are available in the com.ziesemer.utils.xml-*.zip distribution on ziesemer.java.net, with XmlEncoding available starting with version 2009-05-20. The tests include all 16 usable MIME Content-Type examples from http://tools.ietf.org/html/rfc3023#section-8.

public class XmlEncoding{
  
  public static final String US_ASCII_NAME = "US-ASCII";
  public static final Charset US_ASCII;
  public static final String UTF_8_NAME = "UTF-8";
  public static final Charset UTF_8;
  public static final String UTF_16BE_NAME = "UTF-16BE";
  public static final Charset UTF_16BE;
  public static final String UTF_16LE_NAME = "UTF-16LE";
  public static final Charset UTF_16LE;
  
  // Below are not guaranteed to be in both Java 1.5/5.0 and 1.6/6.0.
  public static final String UTF_32BE_NAME = "UTF-32BE";
  public static final String UTF_32LE_NAME = "UTF-32LE";
  public static final String IBM037_NAME = "IBM037";
  
  public static InputStream createBufferedStream(InputStream is) throws IOException{…}
  
  public static String determineFromBOM(InputStream is) throws IOException{…}
  
  public static String guessFromDeclaration(InputStream is) throws IOException{…}
  
  public static String determineFromDeclaration(InputStream is, Charset guessed) throws IOException{…}
  public static String determineFromDeclaration(InputStream is, Charset guessed, int readSize) throws IOException{…}
  
  public static String calculate(InputStream is) throws IOException{…}
  public static String calculate(InputStream is, String contentTypeMime, String contentTypeEncoding) throws IOException{…}
  public static String calculate(InputStream is, String contentType) throws IOException{…}
  
  public static InputSource createInputSource(InputStream is) throws IOException{…}
  public static InputSource createInputSource(InputStream is, Charset charset) throws IOException{…}
  public static InputSource createInputSource(InputStream is, String contentTypeMime, String contentTypeEncoding) throws IOException{…}
  public static InputSource createInputSource(InputStream is, String contentType) throws IOException{…}
  
  public static String getContentTypeMime(String contentType){…}
  
  public static String getContentTypeEncoding(String contentType){…}
  
  public static boolean isContentTypeTextXml(String mime){…}
  
  public static boolean isContentTypeApplicationXml(String mime){…}
}

In most instances, only one of the createInputSource methods will be necessary. For example, from a HttpServletRequest:

HttpServletRequest req = …;
InputSource iSource = XmlEncoding.createInputSource(
  req.getInputStream(), req.getContentType());

All other non-createInputSource(…) methods in this class that accept an InputStream require that mark(int) and reset() are supported. This allows for bytes to be read and "unread", allowing for read portions to be re-read by the XML processor as appropriate. For example, determineFromBOM(InputStream) will consume the BOM bytes if it can be successfully determined, while the guessFromDeclaration(…) and determineFromDeclaration(…) methods will always reset the InputStream to its original position. createBufferedStream(InputStream) can be used to appropriately wrap an InputStream if required. See the included Javadocs for complete details.

Xalan-J Serialization Performance hindered by Flushing

Following the "Chaining Transformations" approach I described in XML and XSLT Tips and Tricks for Java, I had developed a very performance-aware system centered around XML processing. By "pipe-lining" the various steps, less memory is required, and the execution time is reduced. The example in my previous post only used a sample final destination of System.out. Unfortunately, an issue quickly appeared once a similar approach was used in a real-world situation, where the output was a higher-latency destination. The approach was and still is correct, but a work-around is currently necessary to avoid a bug in the Apache Xalan/Serializer implementation that would otherwise cause a severe performance penalty.

As discussed between 2001 and 2003 in the XALANJ-78 bug report, there was some discussion around when flush() is called on the result. The overall consensus was that it was and should only be called from endDocument(). This would mean only one flush operation per document, which would seem acceptable.

However, I found that flush() is being called much more often, at least using versions of Xalan-J between 2.6.0 (used in Java 1.5/5.0 - 1.6/6.0) and the latest 2.7.1. It seems that any call to TransformerIdentityImpl.startPrefixMapping(…) calls ContentHandler.startPrefixMapping(…), with no overloaded methods in the public API. This is implemented by ToStream.startPrefixMapping(String prefix, String uri). This then calls the non-API method ToStream.startPrefixMapping(String prefix, String uri, boolean shouldFlush), with "shouldFlush" always true. This in itself seems to be correct, in that "shouldFlush" affects other logic beyond just flushing the output stream. However, this always calls flushPending(), which then flushes the actual output stream.

The result? The output stream or writer may be flushed as much as once per XML element written. I reported this in XALANJ-2500, along with an example that demonstrates 100 XML elements being written, and flush() being called as many times. In this particular case, using namespaced XML elements is required. However, where I first ran into this was with an XSL that utilized XML namespaces for parameter names, but the generated document was completely within the default namespace.

Assume that the output destination has a latency of even just 50ms. Writing just the small sample document of 100 elements will take 5 seconds under the given circumstances! In some related scenarios, wrapping the OutputStream or Writer in a BufferedOutputStream or BufferedWriter can improve performance by allowing the caller to write without causing a call to the underlying system for each write. Unfortunately, each call to flush() on the buffered implementations simply cause the buffers to flush to the underlying output, and for the output to be flushed as well.

The only solution I'm aware of at the moment is the one I mentioned in the bug report: Use a subclass of BufferedOutputStream or BufferedWriter, with flush() being overwritten to do essentially nothing. (See my NoFlushBufferedOutputStream and NoFlushBufferedWriter classes in MarkUtils-IO for an implementation.)