JScript for Ping, Renew IP and Network Info / Repair

I was having a frustrating issue where a remote Windows XP computer that I need to Remote Desktop into would occasionally become unresponsive. This issue already required me to drive a number of late-night trips to temporarily remedy the issue. While I've not yet found the root cause, the main issue seems to be that something becomes "disconnected" between the network adapter and the operating system. While the Windows "Status" dialog on the network adapter shows "Connected" and packets being sent and received, I am not even able to ping the default gateway. Windows Firewall is not enabled. The only other suspect software I need to rule out yet is an anti-virus service. Rebooting the computer always helped, but is rather drastic. Choosing the "Repair" option does not work, and shows a message about failing while renewing the IP address. Interestingly, the fastest temporary "fix" I found is to run "ipconfig /release" followed by "ipconfig /renew".

While this computer is probably due for a re-installation, it is not an immediate option. In the meantime, I came up with a Windows Script Host (WSH) script to automatically detect and repair this issue. I added it to the Windows Task Scheduler, and set it to run every 15 minutes.

As previously posted, I chose to use JScript for this task (particularly over VBScript). Everything but the logging is essentially WMI (Windows Management Instrumentation) calls. If I hadn't coded for all of the extra logging, the code would have probably been less than half the current size. It basically finds a given network adapter, and attempts to ping the configured default gateway. If unsuccessful, it will perform a DHCP release and renew operation.

/** 
 * @author Mark A. Ziesemer, www.ziesemer.com, 2009-06-25
 */
(function(){
  
  var fso = new ActiveXObject("Scripting.FileSystemObject");
  var wmi = GetObject("winmgmts://localhost/root/cimv2");
  
  var getNetworkAdapter = function(name){
    var adapters = new Enumerator(wmi.ExecQuery(
      "Select * From Win32_NetworkAdapter Where NetConnectionID='" + name + "'"));
    if(!adapters.atEnd()){
      return adapters.item();
    }
  };
  
  var getAdapterConfig = function(index){
    var configs = new Enumerator(wmi.ExecQuery(
      "Select * From Win32_NetworkAdapterConfiguration Where Index=" + index));
    if(!configs.atEnd()){
      return configs.item();
    }
  };
  
  var ping = function(host){
    var results = new Enumerator(wmi.ExecQuery(
      "Select * From Win32_PingStatus Where Address = '" + host + "'"));
    if(!results.atEnd()){
      var result = results.item();
      return result.StatusCode;
    }
  };
  
  var logWriter = fso.OpenTextFile("PingTest.log", 8, true);
  try{
    var log = function(msg){
      logWriter.WriteLine(new Date() + " - " + msg);
    };
    
    var adapter = getNetworkAdapter("Local Area Connection");
    if(adapter){
      log("Found network adapter:"
        + "\r\n\tAvailability: " + adapter.Availability
        + "\r\n\tCaption: " + adapter.Caption
        + "\r\n\tConfigManagerErrorCode: " + adapter.ConfigManagerErrorCode
        + "\r\n\tErrorDescription: " + adapter.ErrorDescription
        + "\r\n\tLastErrorCode: " + adapter.LastErrorCode
        + "\r\n\tMACAddress: " + adapter.MACAddress
        + "\r\n\tName: " + adapter.Name
        + "\r\n\tNetConnectionID: " + adapter.NetConnectionID
        + "\r\n\tNetConnectionStatus: " + adapter.NetConnectionStatus
        + "\r\n\tStatus: " + adapter.Status
        + "\r\n\tTimeOfLastReset: " + adapter.TimeOfLastReset);
    }else{
      log("No suitable network adapter found.  Quitting...");
      return;
    }
    
    var adapterConfig = getAdapterConfig(adapter.Index);
    if(adapterConfig){
      log("Found network adapter config:"
        + "\r\n\tIPAddress: " + adapterConfig.IPAddress.toArray()
        + "\r\n\tDefaultIPGateway: " + adapterConfig.DefaultIPGateway.toArray());
    }else{
      log("No suitable network adapter configuration found.  Quitting...");
      return;
    }
    
    var pingResult = ping(adapterConfig.DefaultIPGateway.toArray()[0]);
    log("Ping result: " + pingResult);
    if(pingResult){
      var releaseResult = adapterConfig.ReleaseDHCPLease();
      if(releaseResult){
        log("Unexpected ReleaseDHCPLease() result: " + releaseResult);
        return;
      }
      var renewResult = adapterConfig.RenewDHCPLease();
      if(renewResult){
        log("Unexpected RenewDHCPLease() result: " + renewResult);
        return;
      }
      log("Successfully released and renewed IP: " + adapterConfig.IPAddress.toArray());
    }
  }finally{
    logWriter.WriteLine();
    logWriter.Close();
  }
  
})();

While the problem detection isn't based on the packet count statistics that I observed from the Windows UI, I thought it would also be useful information to log to see if there is a correlation to the "disconnects". Unfortunately, I couldn't find an easy way to obtain these statistics - either through WMI or the command line. I did find a few places to obtain protocol/transport-level statistics such as TCP, but nothing at the link-layer (Ethernet) level. The only involved discussion I could find on this was this newsgroup posting to microsoft.public.win32.programmer.wmi back from 2006 which implies that these statistics are only available through native code.

PuTTY and SSH port forwarding corruption

PuTTY seems to have some serious issues with SSH port forwarding. http://www.chiark.greenend.org.uk/~sgtatham/putty/wishlist/portfwd-corrupt.html seems to document exactly the same issues I've been having.

I was doing a simple read of XML data in Java from a remote HTTP server, using a SSH tunnel created by PuTTY. Commonly but sporadically, the HttpURLConnection appeared to close prematurely, resulting in only partially-received content. Switching between "dynamic" and "static" (Local) tunnel destinations did not have any effect.

I've also been having issues with PuTTY crashing on unclean closes.

The latest development snapshot (2009-05-27:r8577) does not appear to help. The "portfwd-corrupt" bug was filed in mid-2003, and there hasn't been a new release in over 2 years. (0.60 was released on 2007-04-29.) All of the PuTTY forks and variations I tried have the same issue. (See PuTTY on Wikipedia.) TeraTerm (Wikipedia) does not suffer the same issue, but does not appear to support dynamic tunneling using SOCKS.

My working solution: Use OpenSSH (Wikipedia), which also works on Windows through the use of Cygwin (Wikipedia). Just be aware that this is a command-line solution. While there are probably various front-ends available, I'd be surprised if there was one that didn't limit the available options that ssh has to offer. Ironically, I am using PuTTYcyg to run ssh, as it is much better than the Windows' command prompt. By entering ssh with the desired arguments as the command instead of the default "-" for the login shell, it saves a bash process from running.

MarkUtils-PacProxySelector for Java

Many computer networks make use of proxy servers for web and Internet connectivity. This is especially true for business and other organization networks, where their use is required by security and other policies. Due to the typical use of proxy servers, they are often thought of in terms of "restricting access". Instead, they should be thought of in the proper terms of a means of "providing access". Even outside of typical corporate environments, proxy servers can be invaluable for testing and debugging, as well as used as a type of VPN between private networks.

Most web browsers and other networked applications support directing traffic through one or more proxy servers. The typical configuration dialog looks like this, as shown from Mozilla Firefox:

Of the "manual" options, "HTTP" and "SSL" (TLS) are the most basic and common, followed by "FTP". "Gopher" is rarely used anymore - it has already been dropped by Microsoft Internet Explorer, and may be dropped in Mozilla Firefox 4.0. "SOCKS" is arguably the most powerful, supporting any of the above protocols in addition to any other TCP- or UDP-based protocol. (If SOCKS is configured and supported, none of the other protocols need to be configured.)

Unfortunately, directing an entire networks' traffic through a single proxy server can quickly cause a bottle-neck and a common point for failure. This is especially true when all LAN traffic is also sent through the proxy. (In ideal network traffic patterns, there should be many more multiples of LAN traffic over Internet traffic.) Some of the performance and availability concerns can be addressed through DNS or other load balancing. Another common attempt is to configure the "no proxy for" / exception list. Unfortunately, there are some severe limitations to this design. First, the list must be kept up-to-date as the network configuration changes. (There are various tools for this.) More significantly, there are many desired configurations that cannot be accounted for. For example, what if a list is needed for the servers that should be sent to a proxy server, rather than skipping the proxy server (reverse logic)? Or what if traffic must be split among multiple proxy servers, depending upon the destination or other parameters?

The solution to all of the above and other similar concerns is through the use of the last option shown above, and probably the most overlooked: "Automatic proxy configuration URL". This option is also known as proxy auto-config, or PAC, and was introduced into Netscape Navigator as early as 1996. A PAC file only needs to contain an implementation of a JavaScript function, FindProxyForURL(url, host). From here, the full power of JavaScript can be used, including regular expressions, associative arrays, and closures, as well as a number of predefined helper functions specific to PAC. Within the PAC function, various load balancing and black- or white-listing tasks can be performed, optionally by maintaining internal state. A list of multiple proxies may also be returned for attempts by the client. The PAC file is loaded from a URL (including local file:// URLs), where it can be centrally maintained and updated. The PAC file may be cached by the web browser or other client, but should respect the cache settings sent in the HTTP headers if retrieved through HTTP. Alternatively, a chrome:// URL can even be used, allowing for the PAC file to be maintained within a Firefox extension, and updated through Firefox's standard auto-update process for extensions.

Java support

Java supports many of the same above proxy options, mostly through the use of system properties. For full details, see the tech note at java.sun.com, Java Networking and Proxies. These settings affect any communications made through URLConnection, Socket, and possibly other network-related classes. Previously, the proxy configuration options were limited to the "manual" options listed above, with separate options for HTTP, HTTPS, FTP, and SOCKS. However, Java 1.5/5.0 introduced the Proxy and ProxySelector classes. A default ProxySelector can be configured for the current JVM by calling ProxySelector.setDefault(ProxySelector).

Unfortunately, Java does not currently provide any visible support for proxy auto-config (PAC) files. However, the ProxySelector's List<Proxy> select(URI uri) method looks and works very similar to the PAC's FindProxyForURL(url, host) function. The most notable difference is that it is strongly-typed to standard Java classes. As part of my MarkUtils collection, I created MarkUtils-PacProxySelector to provide a ProxySelector implementation that works with PAC files.

Since the PAC files are based on JavaScript, the ability to evaluate JavaScript is required. Fortunately, this is easily done through Java, especially with the introduction of the Java Scripting API in Java 1.6/6.0 (JSR-223). Java 1.6 bundles an internal version of the Mozilla Rhino implementation of JavaScript for Java, based on 1.6R2. Unfortunately, Java doesn't expose all the features of JavaScript or Rhino directly through the scripting API, some of which are required to implement the PAC functionality in a compatible fashion. This includes defining top-level bindings in the JavaScript environment to Java functions, which is directly supported in Rhino by adding a binding to a FunctionObject - a class to which there is no publicly visible match in the JDK. While it is probably possible to hack a work-around to this, my current implementation utilizes Rhino directly. Besides taking advantage of the improvements in the latest version of Rhino (currently 1.7R2), this allows the utility to be easily used with both Java 1.5/5.0 and 1.6/6.0. (However, note that JSR-223 is unofficially supported under Java 1.5/5.0 as well by downloading and including the .jar's from the reference implementation.) Using Rhino directly also avoids some potential security issues, which I reported in Sun Bug 6782031 and Mozilla Bug 468385.

As commented in the pom.xml file, Mozilla Rhino is currently not available through the central repository, a Mozilla repository, or any other "official" repository. I've added a dependency to it as "org.mozilla.javascript : rhino : 1.7R2". For this to work properly, Rhino will need to be downloaded and installed into a local repository as named above.

In addition to the standard PAC methods, PacProxySelector supports an added function called "connectFailed" to take advantage of the connectFailed(URI, SocketAddress, IOException) functionality on ProxySelector. The JavaScript method is called with the same arguments as on ProxySelector, just with the .toString() representations of each of the three parameters. The PAC file could then store this information within internal state to possibly affect future calls to FindProxyForURL.

For the most flexibility, the constructor to PacProxySelector accepts a Reader, which should read from a PAC file. There is also a public static configureFromProperties() method that returns a ProxySelector, assuming that the path to a PAC file is stored as either a Java system or environment property named "proxy.autoConfig", similar to the other network properties. After obtaining an instance from either the constructor or the method, it should be passed to ProxySelector.setDefault(ProxySelector), unless otherwise used directly. Alternatively, a setDefaultFromProperties() convenience method is provided to do this in one call.

I wrote this in mind for plugging into other Java applications. Ideally, the JDK would provide a system property that accepts the classname for the default ProxySelector or some other method for setting the default outside of a function call within the code, but this is currently not the case. However, all that has to be done is finding a way to execute one of the above configuration options from the desired Java application before network access is attempted. I've successfully written a plugin for Oracle SQL Developer that does exactly this. The same is also possible for Eclipse, though it requires patching of some of the plugins due to the current infrastructure. (See Eclipse bug 257443.) Alternatively, PacProxySelector provides a main method that calls setDefaultFromProperties() before chaining execution to another program's main method. See the included Javadoc for details.

The complete source code, along with a compiled .jar, generated Javadocs, and a suite of JUnit tests are available in the com.ziesemer.utils.pacProxySelector-*.zip distribution on ziesemer.dev.java.net. Please report any bugs or feature requests on the java.net Issue Tracker.

Handling XML Encodings with MarkUtils

Especially in today's focus on higher-level languages, lower-level details are often overlooked. However, character encodings are one such detail that must be remembered, particularly when working with interfaces such as web services, which may be communicating with a large variety of different platforms and languages - both programming and written.

US-ASCII is probably one of the most significant character encodings, but not the earliest. ASCII is the predecessor to the ISO/IEC 646 standard, and is a subset of the Unicode standard, particularly UTF-8. (See also: English in computing.) US-ASCII by itself is unable to represent more than its defined 95 printable characters - 62 of these being [A-Za-z0-9], with the remainder being used for punctuation and other miscellaneous symbols. This limited character set makes it impossible to accurately represent other Latin-based languages such as Spanish, French, and German. "Extended ASCII" such as ISO/IEC 8859-1 provide complete or near-complete coverage of these and other alphabets, but still fail to account for other characters. Using a character encoding that provides full support for the Universal Character Set (UCS) such as UTF-8 is the recommended solution to this and other related issues.

Many character encodings share the same byte representations for the basic English character set [A-Za-z] and [0-9]. These include US-ASCII, ISO-8859-1, UTF-8, and others. For example, in these character encodings, the following mappings are always true:

Character	Decimal	Hexadecimal
0	48	0x30
9	57	0x39
A	65	0x41
Z	90	0x5A
a	97	0x61
z	122	0x7A

Other character encodings, such as EBCDIC, are completely different.

Even when receiving a byte stream from one of the "compatible" encodings, the correct encoding still needs to be determined, as each encoding is handled slightly differently. Certain byte sequences that may be allowed in US-ASCII variants may be invalid in UTF-8. UTF-16 and UTF-32 are easily converted to and from UTF-8, but may be sent with different byte orderings - either big-endian or little-endian.

Encodings in XML

In the W3C's recommendation for XML (essentially the XML standard), the use of character encodings is specifically addressed in section 4.3.3. However, since the encoding is defined with the XML, the encoding declaration itself is also encoded. This is also addressed in the appendix, Autodetection of Character Encodings (Non-Normative):

The XML encoding declaration functions as an internal label on each entity, indicating which character encoding is in use. Before an XML processor can read the internal label, however, it apparently has to know what character encoding is in use—which is what the internal label is trying to indicate. In the general case, this is a hopeless situation. It is not entirely hopeless in XML, however, because XML limits the general case in two ways: each implementation is assumed to support only a finite set of character encodings, and the XML encoding declaration is restricted in position and content in order to make it feasible to autodetect the character encoding in use in each entity in normal cases.

Essentially, the character encoding in use may be determined by a combination of the Byte-order mark (if present), and the value of the "encoding" attribute in the XML declaration of the prolog. However, there may also be external encoding information available from higher-level protocols that must be factored into the determination. This includes the MIME Content-Type sent over HTTP, as defined in RFC 3023.

In Apache Xerces2-J, this is mostly handled within the implementation, particularly by org.apache.xerces.impl.XMLEntityManager.createReader(…). Similar code is also in org.apache.xerces.xinclude.XIncludeTextReader, which also accounts for at least most of the RFC 3023 rules, but only if the input source is a "system Id" which results in the use of a URLConnection.

Addition to MarkUtils-XML

I had a need to meet these same requirements on an input stream to determine the encoding, but before handing off the processing directly to Xerces or another XML processor. I also wanted a flexible solution that would follow the RFC 3023 recommendation, but would be able to work with a variety of sources - including a URLConnection or a HttpServletRequest. I found no existing and public API that met these requirements, so I made my own - and am making it available for public use as part of MarkUtils-XML.

Below is an outline of the public API. While it may first appear to be a bit lengthy, it is designed to be flexible and usable in high-performance environments. None of the Javadoc documentation is included here for brevity. The complete source code, along with a compiled .jar, generated Javadocs, and a comprehensive suite of JUnit tests are available in the com.ziesemer.utils.xml-*.zip distribution on ziesemer.dev.java.net, with XmlEncoding available starting with version 2009-05-20. The tests include all 16 usable MIME Content-Type examples from http://tools.ietf.org/html/rfc3023#section-8.

public class XmlEncoding{
  
  public static final String US_ASCII_NAME = "US-ASCII";
  public static final Charset US_ASCII;
  public static final String UTF_8_NAME = "UTF-8";
  public static final Charset UTF_8;
  public static final String UTF_16BE_NAME = "UTF-16BE";
  public static final Charset UTF_16BE;
  public static final String UTF_16LE_NAME = "UTF-16LE";
  public static final Charset UTF_16LE;
  
  // Below are not guaranteed to be in both Java 1.5/5.0 and 1.6/6.0.
  public static final String UTF_32BE_NAME = "UTF-32BE";
  public static final String UTF_32LE_NAME = "UTF-32LE";
  public static final String IBM037_NAME = "IBM037";
  
  public static InputStream createBufferedStream(InputStream is) throws IOException{…}
  
  public static String determineFromBOM(InputStream is) throws IOException{…}
  
  public static String guessFromDeclaration(InputStream is) throws IOException{…}
  
  public static String determineFromDeclaration(InputStream is, Charset guessed) throws IOException{…}
  public static String determineFromDeclaration(InputStream is, Charset guessed, int readSize) throws IOException{…}
  
  public static String calculate(InputStream is) throws IOException{…}
  public static String calculate(InputStream is, String contentTypeMime, String contentTypeEncoding) throws IOException{…}
  public static String calculate(InputStream is, String contentType) throws IOException{…}
  
  public static InputSource createInputSource(InputStream is) throws IOException{…}
  public static InputSource createInputSource(InputStream is, Charset charset) throws IOException{…}
  public static InputSource createInputSource(InputStream is, String contentTypeMime, String contentTypeEncoding) throws IOException{…}
  public static InputSource createInputSource(InputStream is, String contentType) throws IOException{…}
  
  public static String getContentTypeMime(String contentType){…}
  
  public static String getContentTypeEncoding(String contentType){…}
  
  public static boolean isContentTypeTextXml(String mime){…}
  
  public static boolean isContentTypeApplicationXml(String mime){…}
}

In most instances, only one of the createInputSource methods will be necessary. For example, from a HttpServletRequest:

HttpServletRequest req = …;
InputSource iSource = XmlEncoding.createInputSource(
  req.getInputStream(), req.getContentType());

All other non-createInputSource(…) methods in this class that accept an InputStream require that mark(int) and reset() are supported. This allows for bytes to be read and "unread", allowing for read portions to be re-read by the XML processor as appropriate. For example, determineFromBOM(InputStream) will consume the BOM bytes if it can be successfully determined, while the guessFromDeclaration(…) and determineFromDeclaration(…) methods will always reset the InputStream to its original position. createBufferedStream(InputStream) can be used to appropriately wrap an InputStream if required. See the included Javadocs for complete details.

Xalan-J Serialization Performance hindered by Flushing

Following the "Chaining Transformations" approach I described in XML and XSLT Tips and Tricks for Java, I had developed a very performance-aware system centered around XML processing. By "pipe-lining" the various steps, less memory is required, and the execution time is reduced. The example in my previous post only used a sample final destination of System.out. Unfortunately, an issue quickly appeared once a similar approach was used in a real-world situation, where the output was a higher-latency destination. The approach was and still is correct, but a work-around is currently necessary to avoid a bug in the Apache Xalan/Serializer implementation that would otherwise cause a severe performance penalty.

As discussed between 2001 and 2003 in the XALANJ-78 bug report, there was some discussion around when flush() is called on the result. The overall consensus was that it was and should only be called from endDocument(). This would mean only one flush operation per document, which would seem acceptable.

However, I found that flush() is being called much more often, at least using versions of Xalan-J between 2.6.0 (used in Java 1.5/5.0 - 1.6/6.0) and the latest 2.7.1. It seems that any call to TransformerIdentityImpl.startPrefixMapping(...) calls ContentHandler.startPrefixMapping(...), with no overloaded methods in the public API. This is implemented by ToStream.startPrefixMapping(String prefix, String uri). This then calls the non-API method ToStream.startPrefixMapping(String prefix, String uri, boolean shouldFlush), with "shouldFlush" always true. This in itself seems to be correct, in that "shouldFlush" affects other logic beyond just flushing the output stream. However, this always calls flushPending(), which then flushes the actual output stream.

The result? The output stream or writer may be flushed as much as once per XML element written. I reported this in XALANJ-2500, along with an example that demonstrates 100 XML elements being written, and flush() being called as many times. In this particular case, using namespaced XML elements is required. However, where I first ran into this was with an XSL that utilized XML namespaces for parameter names, but the generated document was completely within the default namespace.

Assume that the output destination has a latency of even just 50ms. Writing just the small sample document of 100 elements will take 5 seconds under the given circumstances! In some related scenarios, wrapping the OutputStream or Writer in a BufferedOutputStream or BufferedWriter can improve performance by allowing the caller to write without causing a call to the underlying system for each write. Unfortunately, each call to flush() on the buffered implementations simply cause the buffers to flush to the underlying output, and for the output to be flushed as well.

The only solution I'm aware of at the moment is the one I mentioned in the bug report: Use a subclass of BufferedOutputStream or BufferedWriter, with flush() being overwritten to do essentially nothing.

Outlook HTML vs. Printing

Today I noticed how I haven't missed using Microsoft Outlook. I printed an email, but found that all the standard headers were missing - From, Sent, To, and Subject. Additionally, trying to troubleshoot using Print Preview also gave surprising results:

Microsoft's KB 222320 ("Some print options not available with HTML messages") does little more than acknowledging this as a problem.

Converting from HTML to Rich Text format is not an option. HTML is much more standards compliant, even with Outlook's quirks. More importantly, HTML format has much better support by other email clients, and has a smaller message size than Rich Text format.

I didn't find a "great" solution for this. The best work-around I found it to not have Outlook handle the printing of messages at all, even though it involves some extra steps. Simply save the message(s) using File, Save As, then selecting "HTML" for "Save As Type". Open the file in your favorite web browser, then preview and/or print from there.

I experienced this on Outlook 2003 with the latest service packs. The KB article states the issue also exists with 2002 (XP) and 2000. I don't have Office 2007 available to test, but I'd be curious of the results.

Dynamically Configuring Logging at Runtime

I've been using SLF4J - the Simple Logging Facade for Java, and Logback - a native and the preferred implementation, for about 3 years. Logback is an excellent replacement for the popular log4j project, development of which has mostly stalled by Apache. Both SLF4J and Logback were designed and are maintained by Ceki Gülcü, the founder of log4j. This combination of SLF4J and Logback are currently used in many significant projects, ironically including many Apache projects, as well as Hibernate, Jetty, and others. See also on Wikipedia: SLF4J and Log4j.

Like log4j and similar logging frameworks, Logback provides several, powerful options for configuration. This includes an XML configuration file that supports variable substitution, nested variables and property files, default values, and file inclusion, among other features. However, what can be done when these options aren't enough?

In particular, Logback resolves its variables from Java's system properties. These must be set when Java starts, or at another point before SLF4J (Logback) is accessed for the first time and automatically configures itself. (Logback can always be reconfigured, but this isn't exactly clean and can cause other issues.) I have seen practices where all calls to the logging framework are expected to go through a proxy class that would first perform the configuration, but this is prone to error. Even if none of multiple developers accidentally call SLF4J or Logback without going through the proxy, there may be 3rd-party libraries that wouldn't even know of the proxy. Additionally, variables by themselves may not be able to provide the dynamic configuration options desired at runtime.

Even Java's built-in java.util.logging introduced with Java 1.4 reads a java.util.logging.config.class system property that can be used to configure the logging within code. However, this again relies on the system property to be set before logging is accessed. Ideally, it would have a default value and class name that could be read from the classpath. Logback does provide a StatusManager, it is primarily meant only for receiving configuration status updates. While it could feasibly be used as a hook for further configuration, this would be an ugly hack at best, and is not the shown intent of the StatusListeners.

I was adding SLF4J and logback into a large, multi-tiered environment, with multiple application server nodes, and multiple JVMs per node. All JVMs operate from the same NAS mount, so even without considering the multiple JVMs per node, simply providing separate configuration options is not an option - not to mention the need it would introduce to maintain multiple files. Logback's FileAppender actually supports this configuration, in having multiple JVMs write to the same log files through the use of the prudent configuration property - but not without approximately tripling the cost of writing logging events. In this performance-critical environment, this increased cost is not an option - so I needed a way to quickly, easily, and reliably store separate logging files per node and JVM.

The easiest way I found to implement this was by using a sub-classed FileAppender. Here is an example of my configuration:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <rollingPolicy class="com.ziesemer.example.MAZTimeBasedRollingPolicy">
      <FileNamePattern>maz-%d-%jvm.log.gz</FileNamePattern>
      <MaxHistory>60</MaxHistory>
    </rollingPolicy>
    <layout class="ch.qos.logback.classic.PatternLayout">
      <!-- http://logback.qos.ch/manual/layouts.html#PatternLayout -->
      <Pattern>%d [%t] %-5p %c %X - %m%n</Pattern>
    </layout>
  </appender>
  <root>
    <level value="WARN"/>
    <appender-ref ref="FILE"/>
  </root>
</configuration>

And my sub-class:

package com.ziesemer.example;

import java.io.File;
import java.net.InetAddress;

import ch.qos.logback.core.rolling.TimeBasedRollingPolicy;

public class MAZTimeBasedRollingPolicy<E> extends TimeBasedRollingPolicy<E>{
  @Override
  public void setFileNamePattern(String fnp){
    try{
      fnp = fnp.replace("%jvm", System.getProperty("PROCESS_ID"));
      
      String serverName = InetAddress.getLocalHost().getHostName();
      File f = new File("/<logPath>/" + serverName);
      // Logback will create any necessary parent paths.
      super.setFileNamePattern(f.getAbsolutePath() + "/" + fnp);
    }catch(Exception ex){
      throw new RuntimeException(ex);
    }
  }
}

This assumes that the basic functionality of the time-based rolling policy and rolling file appender was still desired. Otherwise, the normal FileAppender could have been sub-classed and used instead. This also assumes that the parent process launching the Java process sets a Java system property called "PROCESS_ID". Otherwise, Igor Minar lists some options for obtaining the process ID through Java, though none are perfect or the most ideal: How a Java Application Can Discover its Process ID (PID) (blog.igorminar.com, 2007-03-03). Still otherwise, a random number or other information could potentially be used to differentiate by JVM.

Note that this implementation doesn't completely control the file name pattern - even though it could. Instead, it is using and extending the file name pattern configured in the XML. It intercepts the configured pattern through the setFileNamePattern(String) method, then calls super with the extended version. Besides adding child directories for each server, it adds support for a new, custom "%jvm" variable - in addition to the "%d" already supported by TimeBasedRollingPolicy.

Improving URLEncoder/URLDecoder Performance in Java

I had a need to do some Percent-encoding (a.k.a. "URL encoding") in Java with high-performance requirements. Java provides a default implementation of this functionality in java.net.URLEncoder and java.net.URLDecoder. Unfortunately, it is not the best performing, due to both how the API was written as well as details within the implementation. A number of performance-related bugs have been filed on sun.com in relation to URLEncoder.

There is an alternative: org.apache.commons.codec.net.URLCodec from Apache Commons Codec. (Commons Codec also provides a useful implementation for Base64 encoding.) Unfortunately, Commons' URLCodec suffers some of the same issues as Java's URLEncoder/URLDecoder.

The current sources for each are available online at jdk-jrl-sources.dev.java.net for the JDK (requires registration) and svn.apache.org for Commons. Here are some things I see that could be improved upon, especially considering the features readily-available in Java 1.5/5.0 and above:

Recommendations for the JDK:

• Use of the synchronized StringBuffer instead of the faster StringBuilder. Since these are local method variables, and will never be accessed simultaneously by multiple threads, there is no need for the synchronization overhead. (Java 1.6/6.0's "escape analysis" attempts to skip the synchronization where it is not needed, but it doesn't always work.)
  • The same applies to the CharArrayWriter instance used. While none of CharArrayWriter's methods are marked as synchronized, its write(...) methods all make use of synchronization blocks - really the same thing.

I have reported the above observations to Sun in bug 6837325.

Recommendations for both the JDK and Commons:

• When constructing any of the "buffer" classes, e.g. ByteArrayOutputStream, CharArrayWriter, StringBuilder, or StringBuffer, estimate and pass-in an estimated capacity. The JDK's URLEncoder currently does this for its StringBuffer, but should do this for its CharArrayWriter instance as well. Common's URLCodec should do this for its ByteArrayOutputStream instance. If the classes' default buffer sizes are too small, they may have to resize by copying into new, larger buffers - which isn't exactly a "cheap" operation. If the classes' default buffer sizes are too large, memory may be unnecessarily wasted.
• Both implementations are dependent on Charsets, but only accept them as their String name. Charset provides a simple and small cache for name lookups - storing only the last 2 Charsets used. This should not be relied upon, and both should accept Charset instances for other interoperability reasons as well.
• Both implementations only handle fixed-size inputs and outputs. The JDK's URLEncoder only works with String instances. Commons' URLCodec is also based on Strings, but also works with byte[] arrays. This is a design-level constraint that essentially prevents efficient processing of larger or variable-length inputs. Instead, the "stream-supporting" interfaces such as CharSequence, Appendable, and java.nio's Buffer implementations of ByteBuffer and CharBuffer should be supported.

Recommended replacement URLCodec API:

public class URLCodec{
  public static CharSequence encode(CharSequence in) throws IOException{…}
  public static void encode(CharSequence in, Appendable out) throws IOException{…}
  public static void encode(CharSequence in, Charset charset, Appendable out) throws IOException{…}
  public static void encode(ByteBuffer in, Appendable out) throws IOException{…}
  
  public static CharSequence decode(CharSequence in) throws IOException{…}
  public static void decode(CharSequence in, Appendable out) throws IOException{…}
  public static void decode(CharSequence in, Charset charset, Appendable out) throws IOException{…}
  public static byte[] decodeToBytes(CharSequence in) throws IOException{…}
  public static void decode(CharSequence in, OutputStream out) throws IOException{…}
}

public class URLEncoder implements Appendable, Flushable, Closeable{
  public URLEncoder(Appendable out){…}
  public URLEncoder(Appendable out, int bufferSize){…}
  public URLEncoder(Appendable out, int bufferSize, Charset charset){…}
  
  public Appendable append(CharSequence in) throws IOException{…}
  public Appendable append(char c) throws IOException{…}
  public Appendable append(CharSequence csq, int start, int end) throws IOException{…}
  public void close() throws IOException{…}
}

public class URLEncoderOutputStream extends OutputStream{
  public URLEncoderOutputStream(Appendable out){…}
  public void write(int b) throws IOException{…}
  public void write(byte[] b, int off, int len) throws IOException{…}
}

public class URLDecoder implements Appendable, Flushable, Closeable{
  public URLDecoder(Appendable out){…}
  public URLDecoder(Appendable out, int bufferSize){…}
  public URLDecoder(Appendable out, int bufferSize, Charset charset){…}
  
  public Appendable append(CharSequence in) throws IOException{…}
  public Appendable append(char c) throws IOException{…}
  public Appendable append(CharSequence csq, int start, int end) throws IOException{…}
  public void close() throws IOException{…}
}

The "byte[] decodeToBytes(CharSequence in)" and "void decode(CharSequence in, OutputStream out)" methods reflect that Percent-encoding can be used to encode any series of bytes - not just character representations.

Unfortunately, Java does not yet have an Appendable-equivalent interface for bytes (rather than chars). As such, there is no common interface for OutputStream and ByteBuffer. URLEncoderOutputStream is provided, since OutputStream can be continually appended to without limit. Ideally, these byte methods would also be visible on URLEncoder, but can't be done without spending the class's one option for inheritance, as OutputStream is an abstract class rather than an interface - and Java does not support multiple inheritance. (This also is visible in the implementation of "decodeUrl(CharSequence in, Charset charset, Appendable out)").

Performance

Improving performance was the original goal, so here are some quick measurements. (While not perfect, some steps were taken to avoid the typical flaws of running a flawed microbenchmark). I took a random sequence of 50 characters. Using 10,000,000 iterations per implementation and operation, I encoded the raw characters and decoded the encoded characters:

Implementation	Encode		Decode
JDK Version:	1.6/6.0	1.5/5.0	1.6/6.0	1.5/5.0
JDK URLEncoder/URLDecoder	8,817 ms	27,607	5,980 ms	23,719
Apache Commons URLCodec	9,470 ms	37,735	9,323 ms	32,625
com.ziesemer.utils.urlCodec	2,505 ms	14,934	3,875 ms	11,889

The 1.6/6.0 JDK was "Java(TM) SE Runtime Environment (build 1.6.0_13-b03), Java HotSpot(TM) 64-Bit Server VM (build 11.3-b02, mixed mode)". The 1.5/5.0 JDK was "Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_18-b02), Java HotSpot(TM) Client VM (build 1.5.0_18-b02, mixed mode, sharing)" (32-bit). All tests were run under Windows Vista 64-bit. I also tested the equivalent 32-bit version of the 1.6/6.0 JDK, and the results were all somewhere in-between the previous results. The 32-bit version did default to the client VM rather than the server VM. Forcing it to the server VM did make-up most, but not all of the differences.

Note that com.ziesemer.utils.urlCodec is over 3x as fast as the JDK URLEncoder, and over 1.5x as fast as the JDK URLDecoder. (The JDK's URLDecoder was faster than the URLEncoder, so there wasn't as much room for improvement.)

Download

Adding to my collection of MarkUtils, com.ziesemer.utils.urlCodec is available on ziesemer.dev.java.net under the GPL license, complete with source code, a compiled .jar, generated JavaDocs, and a suite of JUnit tests. Download the com.ziesemer.utils.urlCodec-*.zip distribution from here. Please report any bugs or feature requests on the java.net Issue Tracker.

HP 0xC0000005 Print Driver Errors, Works Font Cache, and more HP issues

I had a call to help my relative install something to open MS Word files. I thought this would be easy - install the free OpenOffice.org. (His computer came with a 60 day trial of MS Office 2003, which had since expired.) Unfortunately, most of the files he needed to open were actually *.wps files for Microsoft Works. Works was failing to open due to an error, and the current "vanilla" version of OpenOffice.org doesn't support the MS Works file format.

Clicking for the details reports the following:

AppName: wkgdcach.exe
AppVer: 8.4.623.0
ModName: hpz3r5mu.dll
ModVer: 61.73.241.0
Offset: 000a0490

Several searches suggested that this could be solved by forcing Windows' font cache to rebuild. Most included removing the C:\Windows\ttfCache file (not folder), which doesn't even appear to exist under current versions of Windows (XP or newer). There is a C:\windows\system32\FNTCACHE.dat file, but renaming this didn't help. What I found is that this was caused by the printing system. Temporarily stopping the "Print Spooler" Windows service (Spooler) prevented the error and allowed Works to start. After restarting the Spooler service, I found that removing the HP printer drivers also provided a work-around.

I completely removed all the HP printer drivers, then reinstalled. The Works issue with wkgdcach.exe came back, as well as the following error when trying to open the preference pages by right-clicking the printer from the "Printers and Faxes" folder and clicking Properties:

This was after installing the latest 10.0.0 (06-2008) version of the "HP Deskjet Full Feature Software and Drivers" (100_215_DJ_AIO_03_F4200_Full_NonNet_enu.exe) from HP's web site for the HP Deskjet F4240. Installing the following available updates didn't help either:

• "Critical Update to Enhance Reliability of Network and USB Connectivity and Improve System Responsiveness While Printing", 04-2009, 2.0, slp_dd_hathi_110_017.exe
• "Critical Update to Correct a PC to Printer Communication Issue", 03-2009, 1.0, ConvergedIO_HPCOM_V3.exe

Removing and installing the 11.0.0 (06-200) version of the "HP Deskjet Basic Driver" didn't help either, nor did re-installing the above updates. There were other issues with this machine, so I went ahead with a back-up and reinstall operation. Unfortunately, there was no stand-alone Windows XP installation disc with the system, just the factory restore partition - so I used that instead. (This is also a Hewlett-Packard / Compaq computer system.) It would be possible to "customize" a XP install disc that would work with the OEM key on the system, but I didn't have the time for this - though it probably would have saved me time in the long-run. After re-installing, I ended up with exactly the same issues - again.

HP Online Support Issues:

After already wasting too many hours on these issues, I thought I'd give my luck with HP support a try. I prefer using the online chat over phone support for a number of reasons. Unfortunately, HP seems to be having serious issues with that as well. On multiple computers, regardless of the web browser used, I could not get their chat program to start. Using Mozilla Firefox redirected to the following server-side error page from Microsoft ASP.NET:

  We're sorry, but an unexpected error has occurred.
The error has been logged and will be examined for further review.

2009-05-02 21:14:05Z Error on 15.201.8.161: Unhandled exception caught: Error Message: Input string was not in a correct format.
 Error Occurred On: /ChatEntry/Chat.aspx
 ExceptionType: System.FormatException
 Stack Trace:
   at System.Number.StringToNumber(String str, NumberStyles options, NumberBuffer& number, NumberFormatInfo info, Boolean parseDecimal)
   at System.Number.ParseInt32(String s, NumberStyles style, NumberFormatInfo info)
   at System.Convert.ToInt32(String value)
   at ChatEntry.Chat.IsSupportedBrowser() in C:\ASTRO\Chat\Integration\inetpub\wwwroot\chat\Chat.aspx.vb:line 346
   at ChatEntry.Chat.Page_Load(Object sender, EventArgs e) in C:\ASTRO\Chat\Integration\inetpub\wwwroot\chat\Chat.aspx.vb:line 163
   at System.Web.UI.Control.OnLoad(EventArgs e)
   at System.Web.UI.Control.LoadRecursive()
   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)

(If they're going to use ASP.NET, they could at least do themselves a favor and use C#.NET instead of VB.NET.) Using Microsoft Internet Explorer 8 at least brought up a chat dialog window, but failed with client-side scripting errors:

Webpage error details

User Agent: Mozilla/4.0 (compatible; MSIE 8.0; …)
Timestamp: Sat, 2 May 2009 21:22:27 UTC


Message: Object expected
Line: 609
Char: 2
Code: 0
URI: https://wimpro.cce.hp.com/system/LiveCustomerServlet.egain

I gave up and called the 800 technical support number. When I was connected to a representative, I explained that their online chat system appeared to be broken, and his answer was that it must be because it is the weekend (!).

The solution:

The HP representative said he recognized this error, and suggested trying to create a new user profile. I feel a bit disappointed for not trying this myself first, as it is a usual part of my troubleshooting routine. Sure enough, this solved the issue - for both the printer driver and MS Works. In this case, even the system reinstall didn't help, as HP's default user profile included in the recovery partition was already "corrupt".

Having previously searched HP's online knowledge base, there were no results for "0xC0000005" - so I strongly suggested that this be included for future searches. I was also surprised that general online searching for 0xC0000005 errors returned a number of results, but no real solutions and nothing related to the user profile.

It should be possible to correct this issue in the current user profile without starting a new, "clean" user profile. However, I haven't found any obvious fixes - such as something to reset under "Hewlett-Packard" in the registry or the user profile folder. Hopefully, as with my previous post on HP driver issues, this post will become helpful to many other users.

Finding Modifier State without Swing Event parameters

Java Swing calls a number of methods with either an InputEvent or an ActionEvent instance as a parameter. Each of these classes provide a getModifiers() method that can be used to find the current state of the CTRL, ALT, SHIFT, and META keys. Given a KeyEvent or MouseEvent subclass of InputEvent there are even more details available, such as the exact key or button that was pressed. However, what if some of these details are required without having an event instance available?

For example, I was looking to add a shortcut to a JTree. If the CTRL button is being held down while a node is expanded or collapsed, all child nodes should recursively do the same. The plan seemed fairly simple: Call JTree.addTreeExpansionListener(TreeExpansionListener), adding a listener that detects if the CTRL button is being held down, then recursively call expandPath or collapsePath>. Unfortunately, the methods on TreeExpansionListener (as well as TreeWillExpandListener) only provide a TreeExpansionEvent, which does not extend either InputEvent or ActionEvent and does not provide any of the above details.

I thought I could easily trap these details myself, through the use of addKeyListener and addMouseListener. Unfortunately, due to the event model in Swing, these listeners are not called until after the tree expand or collapse methods have been processed. Attempting to add these listeners to the component's parent also failed, as the events are first dispatched to the child. There is also no API method to get the current keyboard or mouse state "from anywhere" without having access to one of the event instances, short of writing some custom JNI code. Changing my requirements to modify the GUI instead of making the keyboard modifier work would also have been an undesirable possibility.

This situation is not particular to JTree. There are many other subclasses of AWTEventListener that pass subclasses of EventObject (other than subclasses of AWTEvent) that do not provide means for accessing the desired details.

I did find some possible solutions:

• Use of a glass pane. Basically, it is set as visible and intercepts all events. Unfortunately, this requires then forwarding all events to the desired destination - including a non-trivial amount of work to determine the original destination. This would not scale well, and does not work well with a decorator-type pattern. While it may be a good feature to remember for other specific uses, this certainly is not the first solution I would recommend for this scenario.

In each of the below possible solutions, the idea is to find the event when it is available, and keep a reference to the latest someplace for later:

1. A rather hacked solution - and another one that I do not at all recommend: Get the current EventQueue from Toolkit.getSystemEventQueue(). Then, replace it with a subclassed version using push(). In the subclassed version, override the dispatchEvent method to capture and store the state of any mouse and/or keyboard events. Note that if not done properly, this can result in serious performance and other issues.

2. Subclass JFrame or any other component that is the target for extension. Override processEvent, or more efficiently, processKeyEvent and/or processMouseEvent. Unfortunately, this does not allow for these listeners to be added as a decorator after the instance is created.

3. The winner: Add a listener using Toolkit.addAWTEventListener(AWTEventListener listener, long eventMask). This is my current favorite and go-forward approach, as it works well with decorators and is better-performing than #1. By passing an appropriate eventMask, this assures that the listener only receives events that it is remotely interested in. For example, in my scenario, I'm using AWTEvent.KEY_EVENT_MASK | AWTEvent.MOUSE_EVENT_MASK.

   The AWTEventListener added here in in effect for the entire application, and again, can result in serious performance and other issues if not done properly. Adding such a listener to store only the last keyboard- or mouse-related event should only need to be done once, and suited for reuse throughout the entire application. (Continually adding additional listeners for each use would certainly not scale well.) Look for a utility version of this functionality in an upcoming release of MarkUtils-Swing.

XML Property Expansion with MarkUtils-XML

XML is an increasingly popular format for configuration files. Unfortunately, XML doesn't have a built-in standard for variable substitution or property expansion. This addition to my MarkUtils-XML package is one solution, included as of with version 2009.03.14, and is available for download at ziesemer.dev.java.net.

Background

Several existing practices make use of replacement string patterns, e.g. "${...}" described and supported by Apache Commons Digester's MultiVariableExpander. Apache Maven uses the same pattern in its pom.xml files, but doesn't appear to use Digester to implement this. Instead, Maven uses implementations of org.codehaus.plexus.component.configurator.expression.

I see a few issues with this overall approach of using replacement string patterns:

• While Maven supports resolving properties from 5 different sources, including environment variables and Java system properties, the implementation doesn't eliminate the possibility for naming collisions.
• Neither Digester nor Maven appear to allow for escaping these patterns. While not common, this can sometimes cause issues if "${" needs to be used and passed as-is.
• Neither implementation makes it clear what happens if replacement string patterns are present but not expanded due to the variable not existing. (Is an exception thrown, the replacement string pattern returned as part of the result, or the replacement string pattern expanded to nothing ("")?)

Granted, these few issues could all easily be fixed through the use of escape characters, etc. However, more similar issues will probably continue to appear. Instead, let's use XML as I think it is intended.

Consider the use of XSLT. For inserting values from referenced elements into the output, <xsl:value-of select="..."/> is used, rather than a "${...}"-style syntax. However, this isn't the best example, as the select is a XPath Expression, which then makes use of the "$" prefix as part of a VariableReference.

Solution

The solution provided in MarkUtils-XML is in 2 parts: A schema that declares the property elements, and a XmlPropertyExpander class that performs the expansions.

Schema

The schema is namespaced to avoid naming collisions, as "http://namespaces.ziesemer.com/utils.xml/propertyExpansion". It is meant for easily inclusion into other schemas through the use of the <import .../> element. The following property elements are defined, all extending "BasePropertyType", with one required "name" attribute of type NCName:

• <SystemProperty/> - Resolves Java system properties from System.getProperty(...).
• <EnvironmentProperty/> - Resolves environment variables from System.getenv(...).
• <InstanceProperty/> - Resolves properties from those set on the current instance of the property expander.
• <LocalProperty/> - Resolves properties from <LocalPropertyDef/> elements registered with the current instance of the property expander, typically from the same XML file.

A "PropertyTypeGroup" group exists that allows the above 4 property element types, as well as any elements from any other namespace - allowing for extensions or use-cases not covered by the above pre-defined elements. A "SubstitutionVariableType" group exists that allows for any elements from this group, as well as other mixed content.

For defining <LocalProperty/> values, use one ore more <LocalPropertyDef/> elements. They are of type "LocalPropertyDefType", which extend "BasePropertyType". This allows for references to other properties, including other <LocalProperty/> elements, to be used as part of the definition for a <LocalProperty/>. Just be sure to avoid creating circular references, which would normally result in a StackOverflowError but are instead caught earlier (32 recursions by default) and thrown as a RuntimeException. A default "Properties" element exists to act as a container for one or more <LocalPropertyDef/> elements, including a XSD schema <key/> element to provide a constraint against multiple local property elements with the same name.

Shown below is an example schema that allows for storing a list of file system directories. It imports the "PropertyExpansion" schema, allowing for a <Properties/> element, and the inclusion of one or more "PropertyTypeGroup" elements for property expansion. This example is included in the distribution as "DirectoriesExample.xsd" in the JUnit source:

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema
    xmlns:xs="http://www.w3.org/2001/XMLSchema"
    xmlns:pe="http://namespaces.ziesemer.com/utils.xml/propertyExpansion"
    elementFormDefault="qualified">
  
  <xs:import namespace="http://namespaces.ziesemer.com/utils.xml/propertyExpansion"/>
  
  <xs:element name="Directories">
    <xs:complexType>
      <xs:sequence>
        <xs:element ref="pe:Properties" minOccurs="0"/>
        <xs:element ref="Directory" minOccurs="0" maxOccurs="unbounded"/>
      </xs:sequence>
    </xs:complexType>
    
    <xs:keyref name="PropertyKeyRef" refer="pe:PropertyKey">
      <xs:selector xpath=".//pe:LocalProperty"/>
      <xs:field xpath="@name"/>
    </xs:keyref>
  </xs:element>
    
  <xs:element name="Directory" type="pe:SubstitutionVariableType"/>
  
</xs:schema>

Now, a simple example XML document that follows the above schema. It is also included in the distribution as "DirectoriesExample.xml" in the JUnit source:

<?xml version="1.0" encoding="UTF-8"?>
<Directories
    xmlns:pe="http://namespaces.ziesemer.com/utils.xml/propertyExpansion">
  
  <pe:Properties>
    <pe:LocalPropertyDef name="java.lib">
      <pe:SystemProperty name="java.home"/>/lib
    </pe:LocalPropertyDef>
  </pe:Properties>
  
  <Directory><pe:SystemProperty name="java.home"/>/bin</Directory>
  <Directory><pe:SystemProperty name="java.home"/>/lib</Directory>
  
  <Directory><pe:LocalProperty name="java.lib"/></Directory>
  <Directory><pe:LocalProperty name="java.lib"/>/ext</Directory>
  <Directory><pe:LocalProperty name="java.lib"/>/management</Directory>
  <Directory><pe:LocalProperty name="java.lib"/>/security</Directory>
  
</Directories>

By default, whitespace is trimmed from each text node during expansion (but not whitespace within individual text nodes). This allows for new lines or other "pretty-printing" to be used within the elements to be expanded without having the whitespace included in the expanded result. If whitespace must be maintained, use CDATA sections.

Java class

Everything needed to expand these properties from Java is contained within one class, com.ziesemer.utils.xml.XmlPropertyExpander. It is best-suited for use with XML DOM, providing one core method for expanding elements containing child properties into a String:

public String expand(Node parent);

If no <LocalPropertyDef/> elements are to be supported, XmlPropertyExpander can be instantiated using the 0-argument constructor. Otherwise, the constructor accepts one or more Nodes that contain child <LocalPropertyDef/> elements to resolve against, either as a List or as varargs.

While best-suited for use with DOM, supporting methods are publicly accessible for other use, such as SAX parsing:

public String resolveProperty(String nodeName, String propName);
public Element findLocalProperty(String name);

This assumes that <LocalPropertyDef/> elements will still be available through DOM, which should normally not be an issue as the number of local properties defined should typically be a small fraction of the number elements that refer to these properties. (Potentially parse the <Properties/> element using DOM, then the rest of a larger document with SAX.) If you would like additional SAX support, please open an enhancement request on the issue tracker at ziesemer.dev.java.net.

While not shown in the above example, <InstanceProperty/> elements are resolved through a Map set on the instance:

public void setInstanceMap(Map<String, String> instanceMap);

By default, if a standard property fails to resolve (resolves to null), nothing is output for the property - as if the property element didn't exist. A warning with the unresolved property type and name will be output through java.util.logging. (While I prefer and support SLF4J, this avoids an additional compile- and run-time dependency. java.util.logging can be forwarded to SLF4J through jul-to-slf4j.jar.) The same applies if elements from an external namespace are found during the expansion. To change this default functionality, including returning a resolved String, sub-class and override either or both of these methods:

protected String unresolvedProperty(String localName, String propName);
protected String externalNamespaceResolveProperties(Element e);

Finally, a static method is available for returning a StreamSource for the property expansion XSD schema, useful for building Schema instances with for validating instance documents.

public static StreamSource getSchemaStreamSource();

See the distribution's Javadocs or source code for details on any of the above methods. For more examples, including more features and complex use-cases, see the JUnit source.

MarkUtils-XML: NamespaceContextMap, PrettyPrint, Date Format

Adding to my collection of MarkUtils, this is my introduction of MarkUtils-XML. It is available on ziesemer.dev.java.net under the GPL license, complete with source code, a compiled .jar, generated JavaDocs, and a suite of JUnit tests. Download the com.ziesemer.utils.xml-*.zip distribution from here.

NamespaceContextMap

I think that XML Namespaces are a great solution for avoiding naming collisions. I also think that XPath is a very useful tool for pulling data out of XML documents. Unfortunately, using XPath with XML Namespaces involves a little bit of extra work, especially in the current version of Java.

The most common issue I see other developers run into when first working with this combination is finding that their XPath isn't returning any results. This is because unless otherwise specified, the XPath only searches for nodes declared without a namespace. XML nodes declared with namespaces can be referenced using namespace prefixes, where each prefix is assigned to a specific namespace URI. It should be noted that prefixes only function as placeholders for the namespace URIs. Even though an XML document may have one prefix assigned to a given namespace, it cannot be assumed that prefix will remain unchanged. Many times, these prefixes are generated automatically and/or as needed for each namespace used in a XML document. Two XML documents should be considered equal if the only difference between them are the prefixes used for a common namespace. For example, XSLT uses a namespace URI of "http://www.w3.org/1999/XSL/Transform". It is commonly prefixed to either "xs:" or "xsl:", though other prefixes are also used and valid. As such, any application should explicitly map any desired namespaces to a local prefix that can be used to reference XML nodes declared with a namespace.

In Java, the XPath class accepts a NamespaceContext instance for resolving namespace prefixes to namespace URIs, and vice-versa. Unfortunately, Java does not currently provide an implementation of the NamespaceContext interface, as reported in Sun's bug 6376058. It is relatively easy to write a simple implementation, which can optionally be included as either an inner-class or an anonymous inner-class. However, this can quickly become quite repetitive, especially when needing to support multiple namespace mappings in the same context.

My solution is the NamespaceContextMap class. It implements both NamespaceContext and Map<String, String>, making it very easy to configure and use. It accepts both prefix/URI pairs, as well as QName instances. Lookups are first resolved against the instance's configured list of mappings, then the default mappings as defined in NamespaceContext and XMLConstants. It also follows all the guidelines listed in the interface's Javadoc.

Here is some basic, example usage:

NamespaceContextMap ncm = new NamespaceContextMap();
ncm.put("xslt", "http://www.w3.org/1999/XSL/Transform");
ncm.put("xhtml", "http://www.w3.org/1999/xhtml");

XPathFactory xpf = XPathFactory.newInstance();
XPath xpath = xpf.newXPath();
xpath.setNamespaceContext(ncm);

// Do XPath operations here...

Two maps are internally maintained for performance regardless of lookup type - one map is keyed by prefix, the other by URI. The later are stored a Set with a backing List, which guarantees that multiple prefixes are supported per URI, and that the getPrefixes(String) method returns them in the order that they were added (FIFO) - essentially, an ordered map.

In the implementation, I struggled with finding a solid way of enforcing consistency and constraints. In particular, the entrySet(), keySet(), and values() methods of the Map interface make it very difficult (but not impossible) to intercept add/remove operations, something that I previously posted about in Java Collections Listeners. For now, these methods return unmodifiable collections.

XML PrettyPrint

While XML may commonly be sent as a single-line or without indentation for compactness and efficiency, it is usually most easily viewed with increased indentation at each level, commonly referred to as "pretty printing". This styling is presented by default in most web browsers, including both Mozilla Firefox and Microsoft Internet Explorer, as well as many IDEs and text editors. However, performing this formatting from an automated fashion within Java doesn't seem to be a feature that is readily available, stable, or easy to use. See the "Java 1.5 doesn't want to indent XML output" forum thread for some related discussion, including a copy of my solution.

My solution is an XSLT that reformats the XML with indentation, accounting for existing whitespace, and without any necessary references to "xml.apache.org". It also accepts configurable XSLT parameters for the indentation and newline character sequences. A PrettyPrint class is provided that handles loading the XSLT as a class resource, and returns a reusable, thread-safe Templates instance. For some details on this, including notes on how to chain it into an existing transformation or serialization for increased performance, see my previous post: XML and XSLT Tips and Tricks for Java.

As noted in my "Tips and Tricks" post, please be sure to upgrade to the latest version of Apache Xalan, 2.7.1 or newer. Otherwise, there is a particular issue where generated comments tend to disappear. This isn't an issue specific to my transformation, and can be reproduced even with an identity transformation. See the comment at the beginning of PrettyPrint.xslt for details.

XmlDateFormat

A frequent task I encounter is generating valid XML schema dateTime-formatted values. This format is a profile of the ISO 8601 standard, and is further detailed in RFC 3339. Unfortunately, Java doesn't currently provide a standard DateFormat that matches this specification. Included in my package is a XmlDateFormat class with a getDateFormat() method that returns a properly-configured DateFormat. As with most Format instances, the returned DateFormat instance is not guaranteed to be thread-safe and should not be re-used across threads.

See all newspaper comments at once with Greasemonkey

Background

Like many web users today, I get much of my local news from the online versions of the local newspapers. One particular feature the online editions offer over the print editions is the inclusion of user comments / responses to the stories.

At least around my areas of interest in Central Wisconsin and the Fox Valley, most of the local papers are owned by the Gannett Company. This includes the Appleton Post-Crescent, the Wausau Daily Herald, and others. Much of Gannett's online presence is currently provided by Pluck Social Media's SiteLife product, particularly SiteLife Comments. Pluck even hosts a special customer profile detailing their work with Gannett Corporation. While the sites I'm working with here all happen to be owned by Gannett, it's quite possible that this will also apply to other Pluck-based sites as well.

Unfortunately, Pluck's current implementation leaves some things to be desired. I recall reading many of the negative comments left as the transition was made from the old version of the sites to the current version, which is when I believe Pluck became involved. Fortunately, they have improved some things since, and definitely seem to be fairing better than the current fiasco at Dell's online community after their so-called upgrade. However, the most annoying issue I have while reading Gannett's local news articles is that only 5 user comments are visible at a time. There is a "Full Page View" option available at the bottom, but this only increases the visible comments-per-page to 10. While all these comments are loaded in an AJAX-type fashion using JSON data, clicking to retrieve the next page still results in reloading the entire page. Even on a broadband connection, each page change requires 5+ seconds. This makes trying to read through all the comments on a popular story very frustrating, especially when there are sometimes 50 or more responses. While many of these comments are informative or insightful, having to click through and reload 5 or more pages is certainly not making the best use of web technology.

Technical Challenges

As I had done with Resizing the Blogger Edit Box, my first thought was to attempt to improve things with a Bookmarklet. Unfortunately, the task proved to be too complex, partially due to the same origin policy blocking the necessary cross-domain data. In particular, while the article page is served from a "www." host, the JSON data containing the comments is obtained from a "sitelife." host. While the current pages seem to work around this restriction through some iframe tricks, attempting to reuse that functionality would be a hack at best. Instead, I turned to a Greasemonkey-based solution. Greasemonkey provides a non-domain-restricted GM_xmlhttpRequest API method that provides access to Mozilla's chrome-privileged XMLHttpRequest object.

The pages I had to work with were not at all desirable or the easiest to work with. Each page typically includes about 20 JavaScript files, and some of the code is quite obfuscated. One of the main files, "GDSRScripts.js", is about 86 KB. The core of the Yahoo! UI Library (YUI) (yahoo-dom-event.js in 2.6.0) is not even half that, at only 31 KB. I also see no effort made at respecting the JavaScript global namespace, or use of other best practices.

The Solution

I've completed a Greasemonkey script that I've posted at userscripts.org: All Pluck Comments. Once installed and configured for one or more of the Pluck-based Gannett news sites, it will update any loaded news article by showing all available comments on the same, single page. If all the existing comments already fit within one page and the current 5-post limit, the script will exit and do nothing. Unfortunately, much of the previous waiting time doesn't seem to be in the JavaScript, but with the server responding to the JSON requests - a performance issue that can't be resolved client-side. While those requests are made, the script will show the loading status above the existing comments. Once all comment "pages" have been downloaded, the comments section is repopulated with the complete list. Additionally, changing the sort order between "Newest first" and "Oldest first" now performs instantly, without requiring additional remote requests.

Due to the number of possible supported sites, only one default URL pattern is configured to the "included pages" within the Greasemonkey script. Other desired, supported sites will need to be manually added. (This would be easier of Greasemonkey supported regular expressions for the patterns, as I requested in ticket #216.) There are two types of URLs I've observed that should be supported. The first looks like "http://<hostname>/article/<date>/<siteId>/<articleId>/". The other looks like "http://<hostname>/apps/pbcs.dll/article?AID=/<date>/<siteId>/<articleId>". The best non-regular expression pattern I can suggest to match both these patterns is "http://<hostname>/*article*", where "<hostname>" needs to be replaced with the literal host and domain name to be supported.

The only current limitation is that the per-comment controls (Recommend, New post, Reply to this Post, Report Abuse, etc.) are not regenerated. This is because it would be very difficult, if even possible, to make all of Pluck's existing JavaScript work with these enhancements. In order to use these controls, click on the "Full Page View" link that is left below the list of comments. This will bring back the limit of 10 comments per page, but the Greasemonkey script will exit without making any changes and leaving these controls intact. I seldom use these controls, so this issue isn't that important to me. However, if there is a stated interest, I may look into resolving this for a future version. Alternatively, feel free to write and submit a patch!

Technical Details

The script first waits for the existing comments to load, at which point it determines the article ID, the total number of comments available, and other information necessary for requesting the additional "pages" of comments. If it times out waiting, or determines that it is on the "Full Page View", it simply exits and does nothing. Otherwise, it makes a series of asynchronous requests to retrieve all the available comments. The responses are unnecessarily URL-encoded, and are decoded by the script using unescape(). The responses also contain an unnecessary <script> section at the beginning, which is searched for and removed. The JSON text is then "safely" evaluated to a JavaScript object using the regular expression provided in section 6 of RFC 4627. Once all responses are received, the existing comments HTML is cleared, and new comments are built and populated from the JSON data using the HTML DOM.

Some of the tools I used during this process were Firebug, JSView, and Notepad++. Some of the JavaScript practices I used include closures and other JavaScript topics I've written about.

New version of MarkUtils-Web: ZipServlet and CompressFilter

If you're not already familiar with my ZipServlet and CompressFilter, see my previous posting where I introduced these Java web utilities.

As with the previous release, the update is available on ziesemer.dev.java.net. The release folder is directly available here. "com.ziesemer.utils.web-2009.01.11.zip" contains the source code, a compiled .jar, and generated JavaDocs.

New in this release are a number of fixes and enhancements. To report any new fixes or enhancements, please use the issue tracker on ziesemer.dev.java.net.

ZipServlet combo mode

ZipServlet now provides a feature almost identical to YUI's Combo Handling. This allows for multiple files to be requested and sent joined together in one response, which can reduce the number of HTTP requests and improve performance, as detailed in this post on yuiblog.com.

This feature works particularly well for JavaScript and CSS files. However, only files of one type should be requested together, otherwise the returned MIME Content-Type HTTP header wouldn't make any sense. Unlike the current implementation provided by yahooapis.com, ZipServlet enforces this by returning a HTTP 400 error (Bad Request) if multiple files are requested that have different content types.

The combo mode is enabled by default in ZipServlet, but can be disabled by using the "comboEnabled" servlet parameter. The path on which combination requests are answered can also be configured through the "comboPath" servlet parameter. comboPath defaults to "combo", as used by Yahoo! / YUI. Additional details are listed in the JavaDocs for ZipServlet, included in the download.

A notable difference between ZipServlet and the Yahoo! / YUI implementation is that YUI files require the version to be prefixed to each requested file, e.g. "/combo?2.6.0/build/yuiloader/yuiloader-min.js&2.6.0/build/dom/dom-min.js&2.6.0/build/event/event-min.js". This presumably allows for multiple files to be requested across different versions. This doesn't make a lot of sense, and results in a longer URL as the version is prefixed before each requested file. As ZipServlet is designed to have each instance associated with one .zip file, as each version of a resource (YUI, etc.) belongs in its own file, and due to implementation details, it made sense to include the "combination" functionality directly into ZipServlet rather than as an additional filter, etc. This restricts combination requests to a given ZipServlet instance, while improving code reuse, performance, the URL length, and other aspects. The equivalent URL to the above when requested from ZipServlet, when using the example configuration in the previous post, including the "yui/" zipPrefix, would be "/combo?/build/yuiloader/yuiloader-min.js&/build/dom/dom-min.js&/build/event/event-min.js". Additionally, the leading slash before each file is optional in the ZipServlet implementation.

Unit Tests

New in this release is a complete suite of JUnit tests. I previously had not included any tests partially due to finding a unit testing solution for Java EE servlets and filters that met my requirements. I had looked at HttpUnit ServletUnit, but found a number of shortcomings. While I had previously been testing under Apache Tomcat, this required manual starting and stopping of the server, and registration of the projects. There are methods to automate this, but none that didn't seem overly complex and without their own shortcomings.

I finally decided upon using Jetty. Jetty is both free and open-source, and is 100% Java which makes it very portable. It is very representative of a production servlet engine as it is one, used by many projects including the JBoss and Apache Geronimo application servers. Jetty fully supports embedded within a Java application, which makes it very suitable for unit testing. It can be configured either declaratively or by using standard web.xml files, with details and several examples available at http://docs.codehaus.org/display/JETTY/Embedding+Jetty. As an added bonus, Jetty is readily available as an Apache Maven artifact, including the latest release and beta versions. This means that if opening the project with Maven support, such as with m2eclipse, Jetty and any other dependencies will automatically be downloaded and included on the testing classpath.

For this project, I wrote a ServletTester class that is reused by all the test classes. Each instance starts a Jetty instance on a dynamic port on the loopback address (127.0.0.1), rather than requiring that a specific be used or configured. This class then provides convenience methods for obtaining the server, connector, context, and base URL.

There are 2 types of tests that I provided for each component (ZipServlet and CompressFilter) - standard tests and configuration tests. The standard tests initializes and holds on to a Jetty instance within a ServletTester as a static field, which is reused by all the test methods. This is mainly for performance reasons, so that a new server isn't required for every test. The configuration tests deal specifically with testing the configuration options available for each component. While these configuration tests still reuse the same server instance, the server's context is restarted with a new ServletHandler for every test.

For the details of this methodology, please feel free to download the code and see for yourself. These tests also demonstrate the features and usage of the components.

XML and XSLT Tips and Tricks for Java

1. Get the latest versions

Starting with Java 1.4, the Java runtime has included a default XML parser and transformer implementation as part of the Java API for XML Processing (JAXP).

However, the included versions aren't up-to-date - not even to the latest versions available when each Java version was released.

Java Version	javax.xml.datatype.DatatypeFactory	javax.xml.parsers.DocumentBuilderFactory	javax.xml.parsers.SAXParserFactory	javax.xml.validation.SchemaFactory	javax.xml.transform.TransformerFactory	javax.xml.xpath.XPathFactory
1.4	n/a	Apache Crimson		n/a	Apache Xalan-J 2.4.1	n/a
1.5 / 5.0	Apache Xerces2-J 2.6.2				Apache Xalan-J 2.6.0
1.6 / 6.0	Apache Xerces2-J 2.6.2				Apache Xalan-J 2.6.0

As of this writing, the latest Apache Xerces2-J version is 2.9.1, and the latest Apache Xalan-J version is 2.7.1. I strongly recommend using the latest versions, as the versions built-in to Java are both somewhat limited and buggy. Xalan's FAQ page gives some strict notes concerning using a newer version under Java 1.4, due to the Endorsed Standards Override Mechanism. However, these instructions make it rather difficult - if not impossible - to use an updated library for a particular application on a shared JRE. Fortunately, these steps appear to be no longer required starting with Java 1.5 / 5.0. In these later versions, Sun has repackaged the Apache libraries into rt.jar as com.sun.org.apache.*, and properly load any desired implementation based on the "META-INF/services/javax.xml.*" files found on the classpath. Implementations including these files, including Apache Xerces2-J and Xalan-J, will automatically be used by default if included on the classpath.

2. Use Templates to reuse Transformations

Most of the JAXP interfaces are not thread-safe, including the factories and the instances obtained from them. I.E., neither instance of DocumentBuilderFactory nor DocumentBuilder should be stored statically or in another such way where they could be accessed by multiple threads.

The same applies to a Transformer. While it can be used repeatedly within a given thread, it is not thread-safe for use across multiple threads.

The solution is to use a Templates object, which can be thought of as a compiled-form of a stylesheet. Per the JavaDoc, "Templates must be threadsafe for a given instance over multiple threads running concurrently, and may be used multiple times in a given session." Additionally, use of Templates for repeated transformations will probably provide a performance improvement, as the transformation source (usually an XSLT) doesn't need to be re-read, re-parsed, and re-compiled.

Here is some simple, typical code of performing a transformation without a Templates object:

TransformerFactory tf = TransformerFactory.newInstance();
StreamSource myStylesheetSrc = new StreamSource(
  getClass().getResourceAsStream("MyStylesheet.xslt"));
Transformer t = tf.newTransformer(myStylesheetSrc);
t.transform(new StreamSource(System.in), new StreamResult(System.out));

Here is the improved code, which makes use of a reusable Templates object:

TransformerFactory tf = TransformerFactory.newInstance();
if(!tf.getFeature(SAXTransformerFactory.FEATURE)){
  throw new RuntimeException(
    "Did not find a SAX-compatible TransformerFactory.");
}
SAXTransformerFactory stf = (SAXTransformerFactory)tf;
StreamSource myStylesheetSrc = new StreamSource(
  getClass().getResourceAsStream("MyStylesheet.xslt"));
Templates templates = stf.newTemplates(myStylesheetSrc);

// templates can now be stored and re-used from practically anywhere.

Transformer t = templates.newTransformer();
t.transform(new StreamSource(System.in), new StreamResult(System.out));

3. Chaining Transformations

When multiple, successive transformations are required to the same XML document, be sure to avoid unnecessary parsing operations. I frequently run into code that transforms a String to another String, then transforms that String to yet another String. Not only is this slow, but it can consume a significant amount of memory as well, especially if the intermediate Strings aren't allowed to be garbage collected.

Most transformations are based on a series of SAX events. A SAX parser will typically parse an InputStream or another InputSource into SAX events, which can then be fed to a Transformer. Rather than having the Transformer output to a File, String, or another such Result, a SAXResult can be used instead. A SAXResult accepts a ContentHandler, which can pass these SAX events directly to another Transformer, etc.

Here is one approach, and the one I usually prefer as it provides more flexibility for various input and output sources. It also makes it fairly easy to create a transformation chain dynamically and with a variable number of transformations.

SAXTransformerFactory stf = (SAXTransformerFactory)TransformerFactory.newInstance();

// These templates objects could be reused and obtained from elsewhere.
Templates templates1 = stf.newTemplates(new StreamSource(
  getClass().getResourceAsStream("MyStylesheet1.xslt")));
Templates templates2 = stf.newTemplates(new StreamSource(
  getClass().getResourceAsStream("MyStylesheet1.xslt")));

TransformerHandler th1 = stf.newTransformerHandler(templates1);
TransformerHandler th2 = stf.newTransformerHandler(templates2);

th1.setResult(new SAXResult(th2));
th2.setResult(new StreamResult(System.out));

Transformer t = stf.newTransformer();
t.transform(new StreamSource(System.in), new SAXResult(th1));

// th1 feeds th2, which in turn feeds System.out.

Here is another approach, which makes use of XMLFilter's. This approach is also documented in Sun's J2EE 1.4 Tutorial.

SAXTransformerFactory stf = (SAXTransformerFactory)TransformerFactory.newInstance();

// These templates objects could be reused and obtained from elsewhere.
Templates templates1 = stf.newTemplates(new StreamSource(
  getClass().getResourceAsStream("MyStylesheet1.xslt")));
Templates templates2 = stf.newTemplates(new StreamSource(
  getClass().getResourceAsStream("MyStylesheet1.xslt")));

SAXParserFactory spf = SAXParserFactory.newInstance();
SAXParser parser = spf.newSAXParser();
XMLReader reader = parser.getXMLReader();

XMLFilter filter1 = stf.newXMLFilter(templates1);
XMLFilter filter2 = stf.newXMLFilter(templates2);

filter1.setParent(reader);
filter2.setParent(filter1);

Transformer t = stf.newTransformer();
t.transform(
  new SAXSource(filter2, new InputSource(System.in)),
  new StreamResult(System.out));

Note how in this later approach, the filter is applied at the source instead of the result.

4. Input Validation

Prior to Java 1.5 / 5.0, the only way to control validation through the JAXP API was to set custom attributes. This is described quite well in Sun's J2EE 1.4 Tutorial in "Validating with XML Schema", so I won't repeat it all here. However, do pay attention to the end of the page, which explains that schemas can be loaded from several different sources, including InputStreams and other InputSources - not just local Files or URLs, which many developers seem to overlook.

Starting with Java 1.5 / 5.0, the function of setValidating on DocumentBuilderFactory seems to have changed slightly. It now essentially controls only DTD validation, not modern schema validation e.g. W3C XML Schema or RELAX NG. Instead, a setSchema method is available, which accepts a compiled Schema object. Like the Templates object above, this is one of the few JAXP classes that is thread-safe and is meant for reuse.

One advantage with the DocumentBuilderFactory's setSchema method is that a document can be checked not only for well-formedness and for validity against a schema, but also for validity against a particular, pre-defined schema. Additionally, by default, the parsing process will follow URLs out to the Internet to resolve schemas, etc. Passing in a Schema object built from locally-kept files can improve performance, and eliminate the need for accessing the Internet. However, if there are additional references to be resolved, further attempts may still be made. These can be intercepted by registering an EntityResolver to the DocumentBuilder.

For ensuring that a particular DTD is used, use the extended EntityResolver2. I've found that if the DOCTYPE is missing, getExternalSubset is called. To use a particular DOCTYPE by default, this method could call and return the result from resolveEntity, after passing in the desired publicId and/or systemId. If the XML to be validated already includes a DOCTYPE, then resolveEntity will be called directly. This can be written to either throw an exception or silently return the desired entity when an unexpected entity is received.

5. XML Creation using XSLT

XSLT is a well-known method for transforming XML, but it can also be used for XML generation. The easiest way it to use XSLT as a transformation, similar to the above methods, but with an empty input source. This is additionally noted in the Transformer.transform(...) JavaDoc.

Using XSLT for XML generation works particularly well when the XML is rather static, or when the XSLT can be used as a template. The Transformer's setParameter(...) can be used to pass parameters into the transformation which can then be used as variables. To avoid possible naming collisions, especially when using larger or 3rd-party XSLTs, I strongly recommend using the namespace prefixes.

Below is a sample XSLT with namespaced parameters, then populated by Java code. It generates a valid XHTML document, with the document title and a message in the body passed-in as parameters:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:z="http://namespaces.ziesemer.com/example"
    exclude-result-prefixes="z">
    
  <xsl:output
    method="html"
    doctype-public="-//W3C//DTD XHTML 1.0 Strict//EN"
    doctype-system="http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"/>
  
  <xsl:param name="z:title"/>
  <xsl:param name="z:message"/>
  
  <xsl:template match="/">
    <html xmlns="http://www.w3.org/1999/xhtml" lang="en">
      <head>
        <title><xsl:value-of select="$z:title"/></title>
      </head>
      <body>
        <h1><xsl:value-of select="$z:title"/></h1>
        <p><xsl:value-of select="$z:message"/></p>
      </body>
    </html>
  </xsl:template>
  
</xsl:stylesheet>

final String NAMESPACE_PREFIX = "{http://namespaces.ziesemer.com/example}";

SAXTransformerFactory stf = (SAXTransformerFactory)TransformerFactory.newInstance();
Templates templates = stf.newTemplates(new StreamSource(
  getClass().getResourceAsStream("XHTMLMessage.xslt")));

// templates can now be stored and re-used from practically anywhere.

Transformer t = templates.newTransformer();
t.setParameter(NAMESPACE_PREFIX + "title",
  "Example Title");
t.setParameter(NAMESPACE_PREFIX + "message",
  "Example Message");

t.transform(new DOMSource(), new StreamResult(System.out));

This approach has a number of advantages. It is fairly easy to see what is happening, and it is easy to make changes or additions to the output. It guarantees valid XML output, as an exception will be thrown if the XSLT is invalid. It also performs quite well.

6. XSLT Inheritance

Just as common functionality can be factored out of Java classes into shared parent classes, XSLT can also make similar use of inheritance by using Stylesheet Imports. Here is an example split into a parent and child:

XHTMLTemplate.xslt:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:z="http://namespaces.ziesemer.com/example"
    exclude-result-prefixes="z">
    
  <xsl:output
    method="html"
    doctype-public="-//W3C//DTD XHTML 1.0 Strict//EN"
    doctype-system="http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"/>
  
  <xsl:param name="z:title"/>
  
  <xsl:template match="/">
    <html xmlns="http://www.w3.org/1999/xhtml" lang="en">
      <head>
        <title><xsl:value-of select="$z:title"/></title>
      </head>
      <body>
        <h1><xsl:value-of select="$z:title"/></h1>
        <xsl:call-template name="z:Message"/>
      </body>
    </html>
  </xsl:template>
  
  <!-- This should be overridden by child stylesheets. -->
  <xsl:template name="z:Message"/>
  
</xsl:stylesheet>

XHTMLMessage.xslt:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:z="http://namespaces.ziesemer.com/example"
    exclude-result-prefixes="z">
  
  <xsl:import href="XHTMLTemplate.xslt"/>
  
  <xsl:param name="z:message"/>
  
  <xsl:template name="z:Message">
    <p xmlns="http://www.w3.org/1999/xhtml">
      <xsl:value-of select="$z:message"/>
    </p>
  </xsl:template>
  
</xsl:stylesheet>

Depending upon the type of location the dependent files, a custom URIResolver will probably be needed to properly resolve the resources. In my example, I'm reading from the Java classpath. Other possibilities could include the local file system, or HTTP URLs. Only the necessary changes to the above Java code are shown below:

URIResolver resolver = new URIResolver(){
  @Override
  public Source resolve(String href, String base) throws TransformerException{
    return new StreamSource(getClass().getResourceAsStream(href));
  }};

SAXTransformerFactory stf = (SAXTransformerFactory)TransformerFactory.newInstance();
stf.setURIResolver(resolver);
Templates templates = stf.newTemplates(resolver.resolve("XHTMLMessage.xslt", null));

7. XSLT Extensions

Using parameters is a start, but the limitations are quickly visible. However, when combined with extension mechanisms, XSLT generation should be able to solve almost any requirement. Reading http://xml.apache.org/xalan-j/extensions.html is an excellent starting point. When properly used, extensions can feed into the transformation process and keep the memory footprint to a minimum.

Following is an example that uses an XSLT extension to output a variable number of messages. Additionally, this method allows for the properties to be calculated dynamically on each iteration, rather than pre-processing and storing the formatted messages - which can save memory.

XHTMLMessage.xslt:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:z="http://namespaces.ziesemer.com/example"
    xmlns:zMessageGenExt="com.ziesemer.example.MessageGenerator"
    xmlns:zMessageExt="com.ziesemer.example.Message"
    extension-element-prefixes="zMessageGenExt zMessageExt"
    exclude-result-prefixes="z zMessageGenExt zMessageExt">
    
  <xsl:output
    method="html"
    doctype-public="-//W3C//DTD XHTML 1.0 Strict//EN"
    doctype-system="http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"/>
  
  <xsl:param name="z:title"/>
  <xsl:param name="z:ext"/>
  
  <xsl:template match="/">
    <html xmlns="http://www.w3.org/1999/xhtml" lang="en">
      <head>
        <title><xsl:value-of select="$z:title"/></title>
      </head>
      <body>
        <h1><xsl:value-of select="$z:title"/></h1>
        <xsl:call-template name="z:Messages"/>
      </body>
    </html>
  </xsl:template>
  
  <xsl:template name="z:Messages">
    <xsl:variable name="z:message" select="zMessageGenExt:getNextMessage($z:ext)"/>
    <xsl:if test="string($z:message)">
      <p xmlns="http://www.w3.org/1999/xhtml">
        <b>
          <xsl:value-of select="zMessageExt:getTitle($z:message)"/>
        </b><xsl:text>: </xsl:text>
        <xsl:value-of select="zMessageExt:getDescription($z:message)"/>
      </p>
      <xsl:call-template name="z:Messages"/>
    </xsl:if>
  </xsl:template>
  
</xsl:stylesheet>

XHTMLExample.java:

t.setParameter(NAMESPACE_PREFIX + "ext",
  new MessageGenerator());

IMessage.java:

package com.ziesemer.example;

public interface IMessage{
  String getTitle();
  String getDescription();
}

MesssageGenerator.java

package com.ziesemer.example;

public class MessageGenerator{
  
  protected int index = 0;
  protected int size = 5;
  
  public IMessage getNextMessage(){
    if(index < size){
      IMessage msg = new TestMessage();
      index++;
      return msg;
    }
    return null;
  }
  
  protected class TestMessage implements IMessage{
    @Override
    public String getTitle(){
      return String.format("This is title %d.", index);
    }
    
    @Override
    public String getDescription(){
      return String.format("This is description %d.", index);
    }
  }
}

Output:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
<head>
<title>Example Title</title>
</head>
<body>
<h1>Example Title</h1>
<p>
<b>This is title 1.</b>: This is description 1.</p>
<p>
<b>This is title 2.</b>: This is description 2.</p>
<p>
<b>This is title 3.</b>: This is description 3.</p>
<p>
<b>This is title 4.</b>: This is description 4.</p>
<p>
<b>This is title 5.</b>: This is description 5.</p>
</body>
</html>

XSLT is more of a functional language than procedural, and the demonstrated use of recursion is really the only way to implement a loop. Unfortunately, this can lead to stack overflow errors within the Java implementation. This can be mitigated by increasing the stack size. While this can be set globally for the JVM, it is certainly not the best option. There is a Thread constructor that allows for the thread's stack size to be specified, but it is platform-depdendent, and is still only a mitigation. There is a good article on IBM developerWorks, "http://www.ibm.com/developerworks/xml/library/x-xslrecur/" (Jared Jackson, 2002-10-01), that specifically addresses this stack overflow issue with XSL recursion. Unfortunately, the examples provided require either a pre-known list size, and/or only calculate within the loop rather than producing output.

Here are some modifications to my above XSLT that recurses down a tree, splitting into 2 children at each level, and making the maximum necessary depth log₂n. (A traditional divide & conquer algorithm.) However, by supporting a loop of an unknown size, the depth cannot be calculated to the appropriate minimum level in advance. In my approach, a pre-defined depth of a "sufficient size" is used. I chose 32, as 2^32 = 4,294,967,296, and would be equal to Java's Integer if it were non-signed. (As Integers are signed in Java, the maximum value of an int is 2,147,483,647.) Also, my trials have shown that the default stack size supports over 1,000 recursions before overflowing, so 32 should be a more than safe value. The tree will be filled "depth-first", and the tree will then continue to grow in "width". Some additional work is done to test if each call still resulted in output. If not, an xsl-if prevents further recursion, otherwise the entire tree would still be built and traversed. At 2 billion+ potential nodes in the tree, completing the recursion would require an unacceptable amount of time.

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:z="http://namespaces.ziesemer.com/example"
    xmlns:zMessageGenExt="com.ziesemer.example.MessageGenerator"
    xmlns:zMessageExt="com.ziesemer.example.Message"
    extension-element-prefixes="zMessageGenExt zMessageExt"
    exclude-result-prefixes="z zMessageGenExt zMessageExt">
    
  <xsl:output
    method="html"
    doctype-public="-//W3C//DTD XHTML 1.0 Strict//EN"
    doctype-system="http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"/>
  
  <xsl:param name="z:title"/>
  <xsl:param name="z:ext"/>
  
  <xsl:template match="/">
    <html xmlns="http://www.w3.org/1999/xhtml" lang="en">
      <head>
        <title><xsl:value-of select="$z:title"/></title>
      </head>
      <body>
        <h1><xsl:value-of select="$z:title"/></h1>
        <xsl:call-template name="z:MessagesRecursive">
          <xsl:with-param name="z:depth" select="0"/>
        </xsl:call-template>
      </body>
    </html>
  </xsl:template>
  
  <xsl:template name="z:MessagesRecursive">
    <xsl:param name="z:depth"/>
    <xsl:variable name="x">
      <xsl:call-template name="z:Messages"/>
    </xsl:variable>
    <xsl:if test="string($x) and $z:depth &lt; 32">
      <xsl:copy-of select="$x"/>
      <xsl:call-template name="z:MessagesRecursive">
        <xsl:with-param name="z:depth" select="$z:depth + 1"/>
      </xsl:call-template>
      <xsl:call-template name="z:MessagesRecursive">
        <xsl:with-param name="z:depth" select="$z:depth + 1"/>
      </xsl:call-template>
    </xsl:if>
  </xsl:template>
  
  <xsl:template name="z:Messages">
    <xsl:variable name="z:message" select="zMessageGenExt:getNextMessage($z:ext)"/>
    <xsl:if test="string($z:message)">
      <p xmlns="http://www.w3.org/1999/xhtml">
        <b>
          <xsl:value-of select="zMessageExt:getTitle($z:message)"/>
        </b><xsl:text>: </xsl:text>
        <xsl:value-of select="zMessageExt:getDescription($z:message)"/>
      </p>
    </xsl:if>
  </xsl:template>
  
</xsl:stylesheet>

8. Beware of classloaders

One frustrating issue I recently dealt with was related to multiple classloaders, where extension classes and methods were being reported as "not found", as mentioned in the Xalan-J FAQ. Fortunately, the Xalan implementation appears to handle multiple classloaders in a quite robust fashion, by using the context ClassLoader. In the environment where I was working, the Xalan classes were in a parent classloader from the extension classes; however, this never posed to be a problem for me previously. The actual error in my particular case was that the servlet engine was old and buggy, and was not setting the context ClassLoader on new threads. I worked around this by calling setContextClassLoader(...) in my servlet's service(...) method, before calling super.service(...).

9. Using DocumentFragments

This is really the first use I found of DocumentFragment where it seemed appropriate. Even when using the XSLT approach, there may be instances where it is necessary or easier to build and include a section of XML from within Java rather than XSLT. If used excessively, fragments are counter-productive to the advantages of using XSLT. Understand that while XSLT can stream the content in a pipelined-fashion as it is prodcued, each DocumentFragment must be completely built and returned before it can be streamed, which will increase memory requirements with the size of the fragments.

As doocumented on the Xalan Extensions page, DocumentFragments are a valid return type from an extension. They are also far easier to produce than the other Node-Set types. Here is the best method I found to make use of this functionality:

Java extension method:

public DocumentFragment fill(Node n){
  Document doc = (Document)n;
  DocumentFragment df = doc.createDocumentFragment();
  
  // Append any number of children and/or sub-children...
  Element e = doc.createElement("Example");
  df.appendChild(e);
  
  return df;
}

XSLT:

<xsl:copy-of select="extensionPrefix:fill($instanceVariable, .)"/>

10. XSLT vs. JAXB and JibX, Castor, etc.

While several people I know are big fans of XML data binding frameworks, I try my best to avoid them. For almost any use that I've seen of these frameworks, I would contend that XSLT and/or one of the XML generation techniques I previously described would be a better fit. In general, these frameworks introduce additional complexities and dependencies, along with usually artificial limitations. I've seen several performance comparisons and presentations between such frameworks, but none that dare to include XSLT and the other direct approaches.

Backups, Re-installation, Free Software Checklist

I feel like a post like this is a number of years overdue for me. Especially for those of you I've done computer work for, this is basically a copy of the mental checklist I use when reformatting and reinstalling a Windows PC.

To quickly summarize, many computers can benefit from an occasional reformat and re-installation - especially if it seems to be running slow or if a computer virus or other malware is suspected. Given the complexity of much of today's malware, you can't realistically rely on your anti-virus software to catch everything, much less properly undo and repair things after an infection - no matter what the manufacturers claim. The best solution is to never get infected in the first place. Following "safe computing" practices is critical, which I'll save another post for. Even without an infection, reinstalling from the operating system-up can help by getting the latest features and other updates for all your software.

This is primarily meant as a list of items to consider and be aware of, and is not meant to be a complete step-by-step guide. Things will vary by your computer, desired software, and your needs. If you're not confident in what you're doing, get some assistance. If you don't already know someone who can help, ask around - word-of-mouth references are the best. A local computer repair shop is the next best-bet. I personally do not recommend Best Buy's Geek Squad or other similar services by other large, chain retailers. For anyone in the area of my previous home in Green Lake, WI, I'd recommend Badgerland Computer Systems. In Eau Claire, WI, I'd recommend RAM Technologies.

Backing Up

The first, and most important step is ensuring you have a proper, up-to-date backup of anything you want to save. Typically, you only need to worry about data, or files that are user-created. Programs can be reinstalled from CD/DVD or downloaded online. Backing up everything in the "Documents and Settings" folder is a good starting point, which is usually "C:\Documents and Settings", or "C:\Users" under Windows Vista.

Be aware some of these files will usually be in use, especially for the current user, and can't simply be copied while in-use. Use one of the following options:

• Boot from a Live CD, such as Ubuntu's LiveCD or BartPE (direct).
• Remove the hard drive, connect it to another computer, and use that computer's operating system to copy the files. (Don't simply boot the hard drive on another computer's hardware.) A Disk enclosure or other adapter may be helpful / required for this. Sunbeam's SATA IDE to USB Adapter has worked well for me, and can be purchased from Newegg and elsewhere for only about $20.
• Use backup software that supports Shadow Copy, which includes Microsoft's NTBackup and Vista's Windows Backup applications.
• As a last resort, backup using a secondary user account. Logging-out the first user doesn't always sufficiently close all open files, particularly ntuser.dat, so rebooting before logging in as the second user is recommended.

Writable DVDs are cheap and usually work well. Just be aware that they are quite fragile. USB flash drives and external hard drives can also be used. I've often considered trying Iomega's REV Drive (Wikipedia). Regardless of the media, consider and choose a backup rotation scheme. Especially, consider if something were to happen while a back-up was in progress - is there a previous and suitable backup available? Keeping one or more recent off-site copies is also a necessity.

Review other considerations and concepts at Wikipedia's Backup article.

Re-installation

1) Verify that you have a proper backup (above), re-installation CD/DVDs for all your programs, and any critical drivers on a CD or other removable media for reinstalling the network drivers, etc.

2) Check for and apply any BIOS updates.

3) Reinstall the operating system.

• Use a slipstreamed installation disk with the latest service pack and updates if possible. For Microsoft Windows XP, I recommend using RyanVM. Additional guides are readily available via a Google search: "slipstream windows xp sp3".
• Consider partitioning the drive into separate partitions, at least one for the operating systems / programs, and one for user data. This allows for the possibility of reformatting and reinstalling the OS partition in the future without loosing user data - though this should not replace backing up.

4) Use Windows Update/Microsoft Update to install any available security and other critical updates. The process may need to be repeated until no further updates are available.

5) Install any needed device drivers and other system utilities.

6) Re-install any needed applications / programs.

7) Restore data from backup.

Free Software Checklist

Finally, make sure your computer can do everything you expect it to do by getting the necessary software. Everything below is "free" as in zero-cost. Additionally, a significant portion of this list is Free Software / Open Source, meaning that it "is software that can be used, studied, and modified without restriction, and which can be copied and redistributed in modified or unmodified form either without restriction".

Unless otherwise noted, everything here should work on Windows XP 32-bit, and both the 32-bit and 64-bit versions of Windows Vista. (I'm using the 64-bit version of Windows Vista, and have everything below installed and working.)

First are some applications that should probably be installed on any new computer, regardless of user. These are frequently required on the web and for opening common file formats, and will eliminate a near-future need to find and install the necessary program later.

• Mozilla Firefox (web browser) [direct].
  • Especially for more technical users / developers, review my previous post of Firefox extensions.
• Adobe: (all proprietary)
  • (Acrobat) Reader [direct].
    • Alternatively, consider the free/open source Sumatra PDF viewer [direct].
    • For creating PDF documents, look at PDFCreator [direct].
  • Flash Player [direct].
  • Shockwave [direct].
• The latest Java Runtime Environment [direct].
  • Developers should also install the Java Development Kit (JDK).
• Apple QuickTime (media player) [direct] (proprietary).
• VLC media player [direct]. Plays DVDs, and just about any other media format.
• The Xvid video codec [direct], for playing Xvid, DivX, and similarly-encoded video files.
• The GIMP image editor [direct].
• 7-Zip file archiver [direct].
  • While Windows provides built-in support for regular .zip files, 7-Zip provides extended options and better performance, and supports a wide variety of additional archive formats.
• InfraRecorder CD/DVD writing (burning) software [direct].
• OpenOffice.org office suite [direct].
• Avast! anti-virus home edition [direct] (proprietary).
  • Alternatively, AVG Anti-Virus Free Edition (proprietary).
• For Windows XP only, the Microsoft PowerToys for Windows XP. Particularly: Tweak UI, Image Resizer, Power Calculator, Open Command Window Here, and the Alt-Tab replacement (all proprietary).
  • In Windows Vista, the Windows XP-version of Tweak UI will not work. Look at the basic version of TweakVI instead.
  • In Windows Vista, the "Open Command Window Here" is already included. Just hold-down the shift key while right-clicking on or in a folder.
  • In Windows Vista, the "Alt-Tab replacement" is essentially built-in to Windows Aero.
• The Sysinternals suite (proprietary). Download the "Sysinternals Suite" .zip file, and extract is somewhere like "C:\Program Files\Sysinternals". Many users will never use this, but it makes many useful Windows tools available when needed. (I use Process Explorer all the time.)

Here are some more essentials for more technical users/developers:

• Cygwin [direct].
• PuTTy [direct].
  • Use the PuTTYcyg patches for direct use with Cygwin.
• WinSCP [direct].
• FileZilla FTP/SFTP/FTPS client [direct].
• Notepad++ [direct].
  • See also my previous post: Notepad++.
• MagicDisc virtual CD/DVD manager [direct] (proprietary).
• HashCheck Shell Extension for generating and verifying CRC-32, MD4, MD5, and SHA-1 checksums, and supporting the .sfv, .md4, .md5, and .sha1 file extensions.

Resizing the Blogger Edit Box

If you compose in Blogger at all, you may be frustrated with the relatively small size of the text area provided for writing. It defaults to only about 15 lines!

Here's a fix.

I started out by using Firebug's Inspect feature to find what I wanted to resize, and to initially tweak the size on-demand. However, this is a bit tedious, and requires further tweaking if the window is resized.

I then wrote a JavaScript Bookmarklet to automate the task. When run, it automatically resizes the Blogger edit boxes for both the "Edit Html" and "Compose" tabs to just a bit smaller than the current size of the browser window, leaving just enough margin for the scrollbars and other editor components, etc. Additionally, it also listens for window resize events, and resizes the edit boxes along with the browser window.

Here is the human-friendly source code:

// Mark A. Ziesemer, www.ziesemer.com.
javascript:(
  function(){
    // http://www.quirksmode.org./js/events_advanced.html
    var attachEvent = function(obj, evType, handler){
      if(obj.addEventListener){
        obj.addEventListener(evType, handler, true);
        return true;
      }else if(obj.attachEvent){
        return obj.attachEvent("on" + evType, handler);
      }else{
        return false;
      }
    },
    getDim = function(dim){
      // http://www.howtocreate.co.uk./tutorials/javascript/browserwindow
      return window["inner" + dim] || document.documentElement["client" + dim] || document.body["client" + dim];
    },
    getH = function(){
      return getDim("Height");
    },
    getW = function(){
      return getDim("Width");
    },
    editSize = function(dim, size){
      document.getElementById("textarea").style[dim]
          = document.getElementById("richeditorframe").style[dim]
          = size + "px";
    },
    lastH, lastW,
    resize = function(){
      var newH = getH(), newW = getW();
      // http://webbugtrack.blogspot.com./2007/10/bug-104-resize-event-firing-errors-in.html
      if(newH != lastH || newW != lastW){
        editSize("height", newH - 250);
        editSize("width", newW - 100);
        lastH = getH();
        lastW = getW();
      }
    };
    resize();
    attachEvent(window, "resize", resize);
  }
)();

This is cross-browser compatible, and was tested against Mozilla Firefox 3 and Microsoft Internet Explorer 7. Not surprisingly, it could be much shorter if it didn't need to work-around a few IE bugs, as commented above:

1. Supporting a non-standards event registration model.
   • For serious use elsewhere, use a JavaScript library that already handles this properly, e.g. the Event Utility in the Yahoo! User Interface Library (YUI).
2. Difficulties in obtaining the size of the browser window, including differences between "strict" and "quirks" modes.
3. The resize event being called repeatedly, which usually resulted in infinite loops and hanging IE.

Here's the same code, condensed as a usable bookmarklet:

javascript:(function(){var%20attachEvent=function(obj,evType,handler){if(obj.addEventListener){obj.addEventListener(evType,handler,true);return%20true;}else%20if(obj.attachEvent){return%20obj.attachEvent('on'+evType,handler);}else{return%20false;}},getDim=function(dim){return%20window['inner'+dim]||document.documentElement['client'+dim]||document.body['client'+dim];},getH=function(){return%20getDim('Height');},getW=function(){return%20getDim('Width');},editSize=function(dim,size){document.getElementById('textarea').style[dim]=document.getElementById('richeditorframe').style[dim]=size+'px';},lastH,lastW,resize=function(){var%20newH=getH(),newW=getW();if(newH!=lastH||newW!=lastW){editSize('height',newH-250);editSize('width',newW-100);lastH=getH();lastW=getW();}};resize();attachEvent(window,'resize',resize);})();

(Bookmarklet - Either right-click and choose to bookmark, or drag to your bookmarks menu or links toolbar.)

I've not yet found a decent tool to compress JavaScript into bookmarklets. The best I've found is at http://chris.zarate.org./projects/bookmarkleter/. However, one issue it has in particular is failure to properly handle comments. While it is the tool I used to condense to the above, I manually removed the comments first and converted the double-quotes to single-quotes.

This script could easily be modified for similar functionality on other sites. Additionally, the need to initially enable the resizing by manually activating the bookmarklet could be avoided by converting this to a Greasemonkey script.

VMware Server 2.0 under Ubuntu Intrepid Ibex

Following my Ubuntu Linux router upgrade project, and my "Ubuntu Hardy Heron under VMware" post, I felt it was time to swap things around and run VMware Server 2.0 under Ubuntu Intrepid Ibex (8.10).

The goal is to run one of the virtual machines I need to keep running quite often on my Shuttle K-4500-N2 which is being used as my router and left running all the time anyway. While it's not exactly a high-performance machine, it should suffice for the virtual machine I'm looking to move, which operates fine with only 256 MB RAM. This will allow me to keep my regular desktop powered off more, and save some power.

vsock Installation Issue

The first issue I had with the host installation under Ubuntu Intrepid Ibex was an issue with building the "vsock" module:

None of the pre-built vsock modules for VMware Server is suitable for your
running kernel.  Do you want this program to try to build the vsock module for
your system (you need to have a C compiler installed on your system)? [yes]

Extracting the sources of the vsock module.

Building the vsock module.

Using 2.6.x kernel build system.
make: Entering directory `/tmp/vmware-config0/vsock-only'
make -C /lib/modules/2.6.27-7-generic/build/include/.. SUBDIRS=$PWD SRCROOT=$PWD/. modules
make[1]: Entering directory `/usr/src/linux-headers-2.6.27-7-generic'
  CC [M]  /tmp/vmware-config0/vsock-only/linux/af_vsock.o
  CC [M]  /tmp/vmware-config0/vsock-only/linux/driverLog.o
  CC [M]  /tmp/vmware-config0/vsock-only/linux/util.o
/tmp/vmware-config0/vsock-only/linux/util.c: In function âVSockVmciLogPktâ:
/tmp/vmware-config0/vsock-only/linux/util.c:157: warning: format not a string literal and no format arguments
  CC [M]  /tmp/vmware-config0/vsock-only/linux/vsockAddr.o
  LD [M]  /tmp/vmware-config0/vsock-only/vsock.o
  Building modules, stage 2.
  MODPOST 1 modules
WARNING: "VMCIDatagram_CreateHnd" [/tmp/vmware-config0/vsock-only/vsock.ko] undefined!
WARNING: "VMCIDatagram_DestroyHnd" [/tmp/vmware-config0/vsock-only/vsock.ko] undefined!
WARNING: "VMCI_GetContextID" [/tmp/vmware-config0/vsock-only/vsock.ko] undefined!
WARNING: "VMCIDatagram_Send" [/tmp/vmware-config0/vsock-only/vsock.ko] undefined!
  CC      /tmp/vmware-config0/vsock-only/vsock.mod.o
  LD [M]  /tmp/vmware-config0/vsock-only/vsock.ko
make[1]: Leaving directory `/usr/src/linux-headers-2.6.27-7-generic'
cp -f vsock.ko ./../vsock.o
make: Leaving directory `/tmp/vmware-config0/vsock-only'
Unable to make a vsock module that can be loaded in the running kernel:
insmod: error inserting '/tmp/vmware-config0/vsock.o': -1 Unknown symbol in module
There is probably a slight difference in the kernel configuration between the
set of C header files you specified and your running kernel.  You may want to
rebuild a kernel based on that directory, or specify another directory.

The VM communication interface socket family is used in conjunction with the VM
communication interface to provide a new communication path among guests and
host.  The rest of this software provided by VMware Server is designed to work
independently of this feature.  If you wish to have the VSOCK feature  you can
install the driver by running vmware-config.pl again after making sure that
gcc, binutils, make and the kernel sources for your running kernel are
installed on your machine. These packages are available on your distribution's
installation CD.
[ Press the Enter key to continue.]

I found a solution to this posted on the Ubuntu Forums: "VMware server 2 vsock warning", specifically the patch to "vmware-config.pl" posted by "delgurth" on 2008-11-28: http://ubuntuforums.org/showpost.php?p=6267637&postcount=17.

Web Interface

I've been using the VMware Server 1.x versions on my Windows desktop and some other machines for quite some time, and things have worked quite well. The 2.0 version is quite an upgrade, and one that I first tested on my Windows desktop. While it brings many new features, I, like many others, was disappointed in the migration to - and apparent requirement to use - the web-based interface.

Included in the issues against the web interface is an issue with SSL certificates. For one, the server certificates are self-signed, which causes various warnings and issues under any browser, but these are common and somewhat-easily worked around. (See also: "Firefox SSL-Certificate Debate Rages On", 2008-08-22, slashdot.org.)

However, there is also an issue where client certificates are continuously prompted for. On the Windows host, I simply avoided this by connecting to the HTTP port (8222) instead of HTTPS (port 8333). The Linux host installation, however, redirects any HTTP requests to HTTPS. The 2 most informative search results I found were on the VMware forums: "VI constantly asks for Client Certificates?" and "identify yourself with a certificate". Unfortunately, I didn't have success with any of the listed suggestions.

Fortunately, the "VMware Infrastructure Client" can be used to connect instead, very similar to the "VMware Server Console" in the 1.x versions. From a Windows install, the client installation is located at "\VMware\VMware Server\hostd\docroot\client\VMware-viclient.exe" underneath Program Files. In order to get it to connect, I had to suffix the port number to the host name (:8333). It also seems to connect to the HTTPS port only, not HTTP.

Unfortunately, the VMware Infrastructure Client seems to have a few issues of its own. For one, I can't keep a console window open for a VM without keeping the main "VMware Infrasturcture Client" open as well. Additionally, it seems that some settings are only available under the web interface, or at least some settings that only work underneath the web interface. For example, when trying to configure the amount of "Reserved Memory" after going into maintenance mode, I get an error: "A specified parameter was not correct. spec.virtualMachineReserved". Configuring the same option through the web UI does work, however.

I was happy to see that the web UI was standardized across operating systems since the 1.x version. Previously, Microsoft IIS was required for any web management under Windows. While this worked well for me under Windows Server 2003 environments, it didn't work so well under Windows XP Professional, where there is a limit of only one operational virtual web site at a time. (See also: "IIS5.1 in XP Pro", Brett Hill, 2002, iisanswers.com.) While this may be fine for a dedicated VMware host, it prevents the use of the "Default Web Site" or any other desired sites.

Now, Apache Tomcat & Java are used. Unfortunately, continually running the web access service consumes ~100 MB of RAM. This is to significant of a waste for me for a process that is only going to be seldom used, especially on a machine that only has a maximum of 2 GB RAM available to begin with (which includes shared video memory). If the other issues were fixed, and/or if VMware made it easy to reuse existing Java/Tomcat instances instead of installing their own, I may reconsider.

Final Thoughts

I've been hearing more and more good things about VirtualBox, now developed by Sun Microsystems. I plan on giving it a try...

Dell Latitude E6500 VID_413C PID_8149

After about 7.5 years, I finally got a new laptop - a Dell Latitude E6500.

Since I previously posted in "No 64-bit Windows for Dell", the situation has improved. See the update there for details, but Dell is now definitely making 64-bit versions of Windows available to their customers, even if not yet available as pre-installed.

Having received my Dell OEM "Reinstallation DVD" for "Windows Vista Ultimate 64BIT SP1", I began the upgrade process. Considering that the system came with an apparently complete set of recovery DVDs and CDs, I wasn't too concerned about wiping the hard drive and starting from scratch. However, as a precaution, I pulled the hard drive and imaged the first 2 partitions, "Diagnostic - DellUtility" and "Recovery".

The recovery partition looked rather interesting. The original partition was somewhere between 1-2 GB, but wasn't even half-full. The copied, compressed image is only 383 MB. According to Dell's documentation, "Dell Factory Image Restore (Microsoft Windows Vista only) returns your hard drive to the operating state it was in when your purchased the computer.". 383 MB seems rather impossible for that task, and probably not enough enough to reinstall Windows, much less any of the Dell drivers or applications. To no surprise, manually searching through the recovery partition didn't yield anything Dell-specific.

I was able to reinstall all the drivers with minimal issues, except one. The main issue is that I don't know what it is! There is one remaining device in the Windows Device Manager that simply lists as "Unknown device", as shown:

All this really shows is that the mysterious device is connected via USB.

Before I reformatted, I worked with the pre-installed image for a bit just to see how things originally were. In hindsight, I wish I would have imaged the whole drive, or for this particular issue, at least taken some "before" screenshots from the Device Manager. The one thing that I do remember, which was a bit of a surprise, was that my system apparently has Wireless USB support, as was indicated by an applet in the notification area (system tray).

As I no longer see this after the reinstall, and since I only have the one unknown device, it would seem that this unknown device must be related to the Wireless USB. Additionally, the Dell Bluetooth device is a "Dell Wireless 410 Bluetooth Module with UWB", where UWB is Ultra-wideband, which Wireless USB is based on.

However, none of the Dell Wireless 410 Bluetooth drivers (specifically R197543.exe) are recognized as valid for the unknown device.

Looking at the details tab of the device properties in Device Manager, the following details are shown:

Hardware Ids

USB\VID_413C&PID_8149&REV_0100
USB\VID_413C&PID_8149

Compatible Ids

USB\Class_FE&SubClass_01&Prot_02
USB\Class_FE&SubClass_01
USB\Class_FE

The only Google result after searching for the above VID and PID to date is a forum post by "Justin77" on 2008-10-16, where he seems to have the same question: http://forum.notebookreview.com/showpost.php?p=4041896&postcount=599. (I'm guessing this will quickly become 1 of 2 Google results for the search!)

The Linux USB Project has a the most comprehensive list of USB Vendor IDs and Product IDs I've seen at http://www.linux-usb.org./usb.ids. This clearly shows that the 413C Vendor ID belongs to "Dell Computer Corp.". While it doesn't currently list a match for the Product ID of 8149, it does show that PID 8140 is "Wireless 360 Bluetooth", and that PID 8142 is "Mobile 360 in DFU". If Dell has any method to assigning their Product IDs, it would seem that PID 8149 should be related to the Dell Bluetooth device.

The listed Compatible IDs, however, seem to hint away from Wireless USB. http://www.usb.org/developers/defined_class lists Class FE as "Application Specific", and SubClass 01 as "Device Firmware Upgrade" - though it lists this next to Protocol 01, where mine is is 02. This also seems to relate to the above 8142 PID - "Mobile 360 in DFU", or "Device Firmware Upgrade".

However, another forum post by "somms" on 2008-11-08, seems to confirm that I should have a "Dell Wireless 410 Wireless USB Device", as shown in the included screenshot at http://forum.notebookreview.com/showpost.php?p=4133796&postcount=21.

Just in case this was an issue with the 64-bit version of Windows, I dual-booted another install of the 32-bit version, and had exactly the same issue. None of the drivers installed there resolved this unknown device, either.

I'm starting to wonder if this whole issue is at all related to "USB mode switching", as was the case with my Alltel UM175AL USB EVDO device.

That's all I have for now. I have an escalated support request open with Dell. Hopefully I'll hear something back during the next week after the holiday weekend is over. Besides resolving this "Unknown device" issue, I hope to find out what the situation was with the recovery partition. I'll add an update here with anything I find out.

In the meantime, time to give Ubuntu Linux a try on the Latitude E6500...

Update (2008-11-30):

This "Unknown device" is definitely a "Dell Wireless 410 Bluetooth & UWB Mini-card". User "somms" posted a response at http://forum.notebookreview.com/showthread.php?t=297416&page=3, containing a link to a torrent containing a WiQuest driver. After a lengthy download, I found that "Drivers\Wiquest\Vista\Delwusb.inf" contained the following:

;.------------------------------------------------------------------.
;|      Dell Wireless 410 Bluetooth & UWB Mini-card                 |
;'------------------------------------------------------------------'
;       Dell MO9 WUSB HWA

%LOADER_HWA_BT_DESC_DELL%     = WQ_LDRHWA.Dev,  USB\VID_413C&PID_8149
%UWB_HWA_BT_DESC_DELL%        = WQ_USBUWB.Dev,  USB\VID_413C&PID_8150
%HWA_GENERIC_BT_DESC_DELL%    = WQ_USBHWA.Dev,  USB\VID_413C&PID_8150&MI_00

After loading the driver, anything with "PID_8149" no longer appears in the Device Manager. Instead, there are the following devices:

• "Dell Wireless 410 Wireless USB Device"
  • Hardware Ids

    USB\VID_413C&PID_8150&REV_:513
    USB\VID_413C&PID_8150
    

  • Compatible Ids

    USB\Class_E0&SubClass_02&Prot_01
    USB\Class_E0&SubClass_02
    USB\Class_E0
    

• "Dell Wireless 410 Wireless USB Host Wire Adapter"
  • Hardware Ids

    USB\VID_413c&PID_8150&MI_00
    

  • Compatible Ids

    USB\Class_e0&SubClass_02&Prot_01
    USB\Class_e0&SubClass_02
    USB\Class_e0
    

So it definitely seems that there is some sort of USB mode switching in effect.

Now, I'd just like to find the official source of these drivers. I can't find anything on Dell's website. While the drivers are listed as from "WiQuest Communications, Inc.", they are Dell-branded - both in the installer and in the application.

Note that WiQuest folded at the end of October (PC Magazine 2008-11-4, EE Times 2008-10-31), which should make things interesting.

The installation provided by "somms" contained the drivers and applet for the WUSB as well as the Bluetooth. However, the driver version for the Bluetooth was 6.1.0.4500 (2008-03-07), where the latest provided by Dell is 6.1.0.4100 (2008-01-31). This makes me wonder if there is a slightly newer version of the WUSB driver available as well.

Regardless, at least now I have the functionality working that was shipped with the laptop, whether or not WUSB is "dead". At least now I can disable the WUSB radio with some certainty and maybe save some power and battery life!

IBM PK70653 WebSphere XSLT Empty DOM Debug Failure

I'm pleased to see that the details of a PMR I had opened with IBM over 3 months ago now are now publicly acknowledged as PK70653: XSLT TRANSFORMATION IN DEBUG MODE FAILS ON AN EMPTY INPUT XML DOCUMENT WITH MESSAGE [FATAL ERROR] PREMATURE END OF FILE.

There are some details which seem to have been left out of the above APAR / PK that may be useful:

• No application Throwable is ever thrown. The described error is generated only to stderr.
• The transformation is aborted and nothing is ever set on the result. If passing in a DOMResult as the "requestResult" listed in the APAR, setNode(...) is never called, and getNode() will return null - potentially leading to additional application errors, probably as a NullPointerException if unchecked.
• As documented in the Transformer's transform(...) method, passing in an empty Source is explicitly allowed by the API.
• There is a temporary work-around. Follow the instructions at http://www-01.ibm.com/support/docview.wss?uid=swg21258865 to disable the XSLT Debug Adapter, either by setting the "com.ibm.debug.attach.agent.xslt.enabled" JVM custom property to "false", or by modifying a preference in Rational Application Developer (RAD) if using the WebSphere Application Server (WAS) Test Environment.

Finally, just for reference purposes, here is the error that may be seen in stderr if experiencing this issue (also as shown in the APAR):

[Fatal Error]  :-1:-1: Premature end of file.
SystemId Unknown;Line #-1; Column #-1; Premature end of file.

Kudos to Dorine, Bruce, Samantha, and the others involved at IBM for taking attention to and resolving this issue!

Configuring LAN DCHP and Dynamic DNS under Ubuntu Linux

(This post is the 6th part of my Ubuntu Linux Router Upgrade Project.)

After configuring routing and NAT in my last post, while the Internet access may be shared, other clients won't know about it unless manually configured, requiring at least an IP address and subnet, local gateway, and one or more DNS servers for proper access.

DHCP

As DHCP is used to retrieve IP configuration information for the router from the Internet, it can also be served from the router to the LAN, and is probably the easiest way to configure networking within a LAN.

In OpenWrt, Dnsmasq was used for this purpose. However, it is relatively simple, and part of this project was to learn other Linux standards. For this project, I chose ISC's DHCP Distribution (a.k.a. "dhcpd", "dhcp3-server"), which seems to be the most popular and full-featured choice for a DHCP server under Linux. (See also dhcp3-server on help.ubuntu.com.)

ISC's DHCP Distribution is available as an Ubuntu-maintained package, "dhcp3-server". See "InstallingSoftware" on help.ubuntu.com for notes on installing software. A good write-up on installing and configuring dhcp3-server in particular is available at http://www.howtoforge.com/dhcp_server_linux_debian_sarge.

One of the main prerequisites is to have a static IP address on the interface that dhcp3-server will be bound to and serving IP addresses on. For this project, this was already done as part disabling NetworkManager in Configuring Persistent PPP.

The server then needs to be configured, using "/etc/dhcp3/dhcpd.conf". Shown below is mine:

# option definitions common to all supported networks...
option domain-name "lan.ziesemer.com";

default-lease-time 691200; #600;
max-lease-time 691200; #7200;

# If this DHCP server is the official DHCP server for the local
# network, the authoritative directive should be uncommented.
authoritative;

# Use this to send dhcp log messages to a different log file (you also
# have to hack syslog.conf to complete the redirection).
log-facility local7;

subnet 192.168.###.0 netmask 255.255.255.0 {
 range 192.168.###.### 192.168.###.###;
 option routers 192.168.###.1;
 option domain-name-servers 192.168.###.1; 
 ddns-domainname "lan.ziesemer.com.";
}

host static-host-name {
 hardware ethernet xx:xx:xx:xx:xx:xx;
 fixed-address 192.168.###.###;
}

DNS

In order for clients to resolve domain names to IP addresses, DNS Servers need to be defined. For many ISP consumers, DNS servers provided by the ISP are used. Unfortunately, more and more ISPs, including Charter Communications, feel obliged to break standard Internet protocol (RFC 2308) and hijack these requests, posing security and other implications. (See also "DNS hijacking" on Wikipedia.) As such, running a local resolver becomes rather desirable, which can also offer performance advantages, and the ability to offer DNS services for the local domain.

As with DHCP, in OpenWrt, Dnsmasq was also used for this purpose. Again, I chose something a bit more "heavy-weight" and popular, BIND (a.k.a. "named"). (See also: http://en.wikipedia.org./wiki/BIND and https://help.ubuntu.com/community/BIND9ServerHowto.) Again, "bind9" will have to be installed.

There are 3 main files for configuring BIND: "/etc/bind/named.conf", "/etc/bind/named.conf.options", and "/etc/bind/named.conf.local", the later two of which are both included by the first.

There is really nothing needed for a basic configuration. However, adding the RFC 1918 zones into named.conf.local as commented in the file is recommended. However, as I'm using 192.168.###.###, I didn't want to uncomment the line to include the entire file. Instead, I did a "literal" include by copying in the contents of the "/etc/bind/zones.rfc1918" file, except for the last line that contained zone "168.192.in-addr.arpa".

Making this DNS server known to the LAN was already accomplished in the dhcpd configuration above by "option domain-name-servers".

Local DNS and Dynamic Updates

Now for the advanced part - configuring a local domain with dynamic updates. Two other posts I reviewed were http://www.semicomplete.com/articles/dynamic-dns-with-dhcp/ and http://www.debian-administration.org/articles/343, neither of which seemed completely correct, comprehensive, or up-to-date - though this post isn't exactly meant to be completely comprehensive, either.

The first thing is to generate a secure key used to allow dhcpd to send updates to named. "/usr/sbin/rndc-confgen" can be used to generate this key context. Running it without any arguments will output 2 sections that are meant to be copied and pasted into the named files, "rndc.conf" and "named.conf". Alternatively, running it with the "-a" argument will generate just the "key" clause, and write it to the default key file ("(/etc/bind/rndc.key", run with sudo). It seems that I needed all 3 parts, but running both versions would result in separate keys. I used the "-a" argument, then copied out the key for manually updating the other 2 files. This assumes that both dhcpd and named are running on the same server, which isn't necessary.

Assume the following was output to "/etc/bind/rndc.key" using "sudo /usr/sbin/rndc-confgen -a":

key "rndc-key" {
 algorithm hmac-md5;
 secret "hp83n+fUOC2LdoDF9S12wg==";
};

The following should then be added to "/etc/bind/named.conf" (at no particular location):

include "/etc/bind/rndc.key";
controls {
 inet 127.0.0.1 allow {
  localhost;
 }
 keys {
  rndc-key;
 };
};

Next, the following should then be added to "/etc/dhcp3/dhcpd.conf" (at no particular location):

ddns-update-style interim;

key "rndc-key" {
 algorithm hmac-md5;
 secret hp83n+fUOC2LdoDF9S12wg==;
};

Note that quotes around the secret are not allowed here. Including quotes will produce errors.

At this point, both dhcpd and named now share a "secret" key used to authenticate dhcpd to named for updating DNS entries.

Now, we need to define a DNS zone in "/etc/bind/named.conf.local" for the LAN:

zone "lan.ziesemer.com" {
 type master;
 file "/var/lib/bind/lan.ziesemer.com.zone";
 allow-update {
  key "rndc-key";
 };
 notify yes;
};

zone "##.168.192.in-addr.arpa" {
 type master;
 file "/var/lib/bind/##.168.192.in-addr.arpa.zone";
 allow-update {
  key "rndc-key";
 };
 notify yes;
};

The "allow-update" sections with the "key" are references to the key defined in "/etc/bind/rndc.key" (and included from "/etc/bind/named.conf") that is allowed to make updates for the zones.

Note the location of the zone files. Originally, I had these in "/etc/bind" as well, but ran into "permission denied" errors whenever updates were attempted, even after editing permissions on the files. It seems that even with opened permissions, the files are protected by AppArmor. This is further described in the "bind / dhcp jnl failure - permission denied" thread on ubuntuforums.org.

The referenced zone files then need to be created, the process of which is outside the scope of this post. Refer to the above linked articles for further reference. For the purpose of dynamically updated zones, these files should be chmod'd to 755, with both the user and group set to "bind". (I don't recall doing this, so they may be automatically set at some point?)

Finally, the following should be included in the subnet block declared in "/etc/dhcp3/dhcpd.conf" (above):

subnet 192.168.###.0 netmask 255.255.255.0 {
 zone lan.ziesemer.com. {
  primary 127.0.0.1;
  key "rndc-key";
 }

 zone ##.168.192.in-addr.arpa. {
  primary 127.0.0.1;
  key "rndc-key";
 }
}

Restart the services for a fully functional LAN:

sudo /etc/init.d/dhcp3-server restart
sudo /etc/init.d/bind9 restart

Enabling routing and NAT with iptables

(This post is the 5th part of my Ubuntu Linux Router Upgrade Project.)

After my last post, I now have a persistent Internet connection configured through PPP. Now it's time to make the connection available to the rest of my Local Area Network.

Enabling Routing

IPv4 (and also IPv6) routing is already built-in to the Linux kernel (at least under Ubuntu Hardy Heron), and simply needs to be enabled. The easiest way to do this is to edit "/etc/sysctl.conf", which like many of the other configuration changes made in this project, needs to be edited as root. (Use sudo.)

At least under Ubuntu Hardy Heron, a sysctl.conf file already exists with a few default options, and many other commented and documented options. The option that needs to be enabled is "net.ipv4.ip_forward=1", either by removing the prefixed "#" comment character, or by adding the line somewhere in the file.

This setting can then be immediately placed into effect without rebooting, etc., by executing "/sbin/sysctl -p".

Note that there are many related pages on the web that refer to echoing a "1" into "/proc/sys/net/ipv4/ip_forward". While this may work temporarily, it will most likely be set during reboot, so the above method should almost always be used instead.

The case for NAT

At this point, assuming there is both a public / WAN (Internet) interface and a private / LAN interface, Linux will route between them. However, assuming the LAN is configured with RFC1918 non-routable addresses e.g. 192.168.*.* (see also Private network on Wikipedia), these addresses aren't recognized on the Internet, and Linux shouldn't (and probably doesn't) even try to forward traffic from them. Even if this wasn't the case and traffic from the LAN destined for the Internet was forwarded, no other router on the Internet would know where to send the responses back to, without having configured routing rules back to the local router.

Most ISP consumers only receive one public IP address, which is typically dynamic and changes regularly. NAT, or Network Address Translation is the most commonly used method to allow multiple computers / devices on a LAN to share this one address. (This is the same method that is used by almost all SOHO routers, e.g. Linksys, Netgear, and D-Link.)

Configuring iptables and NAT

NAT is commonly implemented in Linux within iptables. At least under Ubuntu Hardy Heron, iptables is already installed and running, just using a default set of rules that effectively allows everything. (This is not necessarily an issue for a new installation as there are no services installed and running to connect to that can be compromised.)

iptables is typically configured using the "iptables" command (which is actually located at "/sbin/iptables"). Usually several steps are needed for a desired configuration, and are typically entered as separate calls to iptables. These are typically consolidated into a shell script.

Here is the simplest configuration using iptables necessary to enable sharing from a WAN to a LAN:

iptables -t nat -A POSTROUTING -o ppp0 -j MASQUERADE

Again, since this alters the configuration of the system, the commands or script containing the commands need to be run with sudo. If forgotten, an error containing "Permission denied (you must be root)" should be displayed.

At least under the default iptables configuration in Ubuntu Hardy Heron, this is all that is needed as the default policy for everything is ACCEPT. This command basically changes the source address of any outgoing packets (in this case, on ppp0) to the outgoing interface's address, so return packets are sent to the router's public / WAN address. This is really the only valid option, as the original private / LAN addresses are invalid, as described above. These masqueraded connections are then tracked, such that when a response is received, the process can be reversed, and the response is sent to the original requesting device.

Here is a more comprehensive example, one that I saved as "/etc/iptables.conf":

#!/bin/sh

logger -i $0 Starting...
iptables -F
iptables -F -t nat

#LAN_IF=eth0
WAN_IF=ppp0

if [ -d "/proc/sys/net/ipv4/conf/$WAN_IF" ]; then

 #WAN_IP=$(ifconfig $WAN_IF | sed -n 's/ *inet addr:\([0-9.]*\).*/\1/p')

 iptables -t nat -A POSTROUTING -o $WAN_IF -j MASQUERADE

fi

logger -i $0 Complete.

• This version first flushes all existing rules ("iptables -F"), ensuring a consistent configuration.
• The WAN-dependent portions run only if the WAN interface exists.
• It also logs to the system logger using logger, for future diagnosis.
• Variables e.g. "WAN_IF" are used for maintainability, making it easier to update when the configuration changes. "WAN_IP" and "LAN_IF" are currently not used, but shown for example.

This is not the most secure configuration by any means, but it's a simple solution for solving the immediate need that can be extended upon. Some additional details are available in the "Masquerading Made Simple HOWTO".

Making it Automatic

So far, everything configured with iptables will be lost on reboot. The best way I found to handle this is by placing a link to the above "/etc/iptables.conf" in "/etc/network/if-up.d". Use "ln -s" to do this, or write a new script that calls the above. The same may be desirable for capturing when the interface goes down, using "/etc/network/if-down.d". As documented in run-parts, ensure that any desired files are marked executable, and follow the proper naming (consist entirely of upper and lower case letters, digits, underscores, and hyphens - no periods!).

For connections that utilize IP leases using DHCP, and if utilizing a DHCP IP address within iptables, it is probably also necessary to make sure that the rules are updated by re-running iptables when a lease is renewed. Similar to the above, scripts can be put in or linked to from the "dhclient-enter-hooks.d" and/or "dhclient-exit-hooks.d" directories, which executed by the default "/sbin/dhclient-script" script when using dhclient.

As I'm currently using a PPP connection which doesn't utilize DHCP, this wasn't necessary for me. Instead, I tied into the PPP connection process by linking to "/etc/iptables.conf" in "/etc/ppp/ip-up.d", same as described above.

To be continued...

Next up: Configuring LAN DHCP and Dynamic DNS.

Configuring Persistent PPP under Ubuntu

(This post is the 4th part of my Ubuntu Linux Router Upgrade Project.)

After my last post, Alltel UM175AL USB EVDO under Ubuntu Hardy Heron, I now have a usable serial device to connect to with PPP.

Initial Trials

At this point, Jason Costomiris's instructions at HOWTO: Verizon UM175 USB EVDO Card under Ubuntu Hardy work perfectly. The only difference is that being on Alltel instead of Verizon, the username is "<Modem'sPhoneNumber>@alltel.net", and the password is "alltel".

Disabling NetworkManager

The next improvement is to upgrade the NetworkManager utility that is part of the ubuntu-desktop package, in order for it to recognize and work with the CDMA connection. This is detailed in Jason's next post, Using NetworkManager with Pantech UM175 under Ubuntu Hardy, and allows NetworkManager to respond with the proper connection status when queried by other applications. This also makes it easier to establish the connection automatically.

Unfortunately, NetworkManager does not play well with the goals of a server or router configuration. As listed on its package page at http://packages.ubuntu.com/hardy/base/network-manager:

NetworkManager attempts to keep an active network connection available at all times. It is intended only for the desktop use-case, and is not intended for usage on servers. The point of NetworkManager is to make networking configuration and setup as painless and automatic as possible. If using DHCP, NetworkManager is _intended_ to replace default routes, obtain IP addresses from a DHCP server, and change nameservers whenever it sees fit.

As suggested, I disabled NetworkManager. There are instructions at https://help.ubuntu.com/community/NetworkManager#Disabling%20NetworkManager. However, at least under 8.10 (Intrepid Ibex), placing the "exit" scripts in "/etc/default" didn't have any effect. Instead, I just disabled the startup scripts, by calling "update-rc.d -f NetworkManager remove" (again, using sudo).

The next step is to manually configure the network settings in "/etc/network/interfaces". (See "man interfaces" for details.) Primarily, ensure that the desired interfaces are brought up automatically on startup (using the "auto" stanza), and for the purposes of a router, assigning the internal interface a static IP address:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
address 192.168.###.1
netmask 255.255.255.0

pppd Configuration

This is fully documented in the pppd man page. You could even use pppconfig to help with the initial setup, but I opted to work with the actual configuration files directly.

Here is what I saved as "/etc/ppp/peers/Alltel":

/dev/ttyACM0
lock

persist
#debug
hide-password

noauth
user ##########@alltel.net

defaultroute
#usepeerdns #Using bind9 instead.

init "/usr/bin/logger -i /etc/ppp/peers/Alltel Calling..."
connect "/usr/sbin/chat -Vf /etc/chatscripts/Alltel 2>/var/log/Alltel.log"

Here is what I saved as "/etc/chatscripts/Alltel":

ABORT 'BUSY'
ABORT 'NO CARRIER'
ABORT 'ERROR'
'' 'AT'
'OK' 'ATQ0V1E0'
'OK' 'ATZ'
'OK' 'AT&F'
'OK' 'AT+CSQ'
'OK' 'ATDT#777'
CONNECT CLIENT

This and other chatscripts are used by and documented in chat.

Finally, the password for the user identified in the peers file needs to be added to "/etc/ppp/pap-secrets":

"##########@alltel.net" Alltel "alltel"

The connection can then manually be initiated using "pppd call <name>", where "<name>" is one of the scripts configured under "/etc/ppp/peers". Note that the "nodetach debug" options are only to diagnose the initial connection, and the "nodetach" option will prevent the call from returning. (Closing the terminal window will disconnect the connection.) Note also that this needs to be run with "sudo" otherwise pppd won't have access to the above configured scripts, which should also be saved with restricted permissions to "root" (or another restricted user account).

$ sudo pppd call Alltel nodetach debug
Serial port initialized.
Serial connection established.
using channel 34
Using interface ppp0
Connect: ppp0 <--> /dev/ttyACM0
sent [LCP ConfReq id=0x1 <asyncmap 0x0> <magic 0x63dc86e0> <pcomp> <accomp>]
sent [LCP ConfReq id=0x1 <asyncmap 0x0> <magic 0x63dc86e0> <pcomp> <accomp>]
rcvd [LCP ConfReq id=0x1 <asyncmap 0x0> <auth chap MD5> <magic 0xfd80aab9> <pcomp> <accomp>]
sent [LCP ConfNak id=0x1 <auth pap>]
rcvd [LCP ConfAck id=0x1 <asyncmap 0x0> <magic 0x63dc86e0> <pcomp> <accomp>]
rcvd [LCP ConfReq id=0x2 <asyncmap 0x0> <auth pap> <magic 0xfd80aab9> <pcomp> <accomp>]
sent [LCP ConfAck id=0x2 <asyncmap 0x0> <auth pap> <magic 0xfd80aab9> <pcomp> <accomp>]
sent [LCP EchoReq id=0x0 magic=0x63dc86e0]
sent [PAP AuthReq id=0x1 user="##########@alltel.net" password=<hidden>]
rcvd [LCP EchoRep id=0x0 magic=0xfd80aab9]
rcvd [PAP AuthAck id=0x1 ""]
PAP authentication succeeded
sent [CCP ConfReq id=0x1 <deflate 15> <deflate(old#) 15> <bsd v1 15>]
sent [IPCP ConfReq id=0x1 <compress VJ 0f 01> <addr 0.0.0.0>]
rcvd [IPCP ConfReq id=0x1 <addr ###.###.###.###>]
sent [IPCP ConfAck id=0x1 <addr ###.###.###.###>]
rcvd [LCP ProtRej id=0x4 80 fd 01 01 00 0f 1a 04 78 00 18 04 78 00 15 03 2f]
Protocol-Reject for 'Compression Control Protocol' (0x80fd) received
rcvd [IPCP ConfRej id=0x1 <compress VJ 0f 01>]
sent [IPCP ConfReq id=0x2 <addr 0.0.0.0>]
rcvd [IPCP ConfNak id=0x2 <addr ###.###.###.###>]
sent [IPCP ConfReq id=0x3 <addr ###.###.###.###>]
rcvd [IPCP ConfAck id=0x3 <addr ###.###.###.###>]
Cannot determine ethernet address for proxy ARP
local  IP address ###.###.###.###
remote IP address ###.###.###.###
Script /etc/ppp/ip-up started (pid 12340)
Script /etc/ppp/ip-up finished (pid 12340), status = 0x0

Once the connection is successfully established without "nodetach", there doesn't appear to be a "hangup" option to pair with "call". It seems that using "kill" to kill the associated pppd process is the most common approach. The "pon" and "poff" scripts may also be used, but internally, "poff" just calls "kill" as well.

Integrating with /etc/network/interfaces

By adding a reference to the above pppd configuration in "/etc/network/interfaces", the connection can be easily managed like any of the other interfaces with commands like "ifup" and "ifdown" (both of which are actually the same program, but called with different names).

auto ppp0
iface ppp0 inet ppp
 provider Alltel

Bringing up the connection then yields an error:

$ sudo ifup ppp0
ppp0: ERROR while getting interface flags: No such device

While this doesn't look good, the connection is actually successfully established. This can be verified by seeing a connection for "ppp0" using "ifconfig". Calling ifup with the "-v" flag for verbose mode shows some detail as to where the error is occurring:

$ sudo ifup -v ppp0
Configuring interface ppp0=ppp0 (inet)
run-parts --verbose /etc/network/if-pre-up.d
run-parts: executing /etc/network/if-pre-up.d/vlan
run-parts: executing /etc/network/if-pre-up.d/wireless-tools
ppp0: ERROR while getting interface flags: No such device
run-parts: executing /etc/network/if-pre-up.d/wpasupplicant
pon Alltel
run-parts --verbose /etc/network/if-up.d
run-parts: executing /etc/network/if-up.d/avahi-autoipd
run-parts: executing /etc/network/if-up.d/avahi-daemon
run-parts: executing /etc/network/if-up.d/ip
run-parts: executing /etc/network/if-up.d/mountnfs
run-parts: executing /etc/network/if-up.d/wpasupplicant

It is actually due to the "wireless-tools" script, apparently on line #11 where it calls "$IFCONFIG "$IFACE"" up". It seems that "$IFACE" resolves to "" at this point, as the "/etc/network/if-*up.d/" scripts are called before pppd actually brings up the connection's interface. The entire script could probably be selectively disabled, but it isn't hurting anything for now.

At another point, I added a link to an iptables script in "/etc/network/if-up.d/". Similar to the above situation, "ppp0" isn't yet available at this point. The script called ifconfig to determine ppp0's IP address, which resulted in the following error:

ppp0: error fetching interface information: Device not found

That's it. At this point, I have a functional, persistent Internet connection over PPP for the local computer.

To be continued...

Next up: Enabling routing and NAT with iptables.

Alltel UM175AL USB EVDO under Ubuntu Hardy Heron

(This post is the 3rd part of my Ubuntu Linux Router Upgrade Project.)

Note that all Linux commands here assume use of Ubuntu Linux 8.04 ("Hardy Heron"), with all updates installed as of 2008-09-30.

The device I purchased for Alltel's Wireless Internet service (3G 1xEV-DO) was a UTStarcom UM175, purchased for $100 before a $100 mail-in rebate on a 2-year contract.

Some specs as printed on the box and from Alltel's website:

• USB modem with swivel connector
• Compatible with Windows XP, Vista, Mac OS X
• 1xRTT/EVDO Rev 0 and Rev A Ready
• CDMA 800/1900MHz

Also Known As

The "Alltel Part#" listed on the box label is "UM175ALA". The P/N on the device label is "UM175AL". (I'm assuming the "AL"/"ALA" suffixes are specific to Alltel?) The FCC ID is PP4PX-700. Also listed on the device label: "Distributed by UTStarcom Personal Communications. Made in Korea by PANTECH. QUALCOMM 3G CDMA."

When connected to Windows before installing the drivers, the following "Hardware Ids" are listed in Device Manager where it is recognized as a "USB Mass Storage Device", with a vendor ID of 0x106c and a product ID of 0x3b03:

USB\Vid_106c&Pid_3b03&Rev_0100
USB\Vid_106c&Pid_3b03

And here is the output of "/usr/sbin/lsusb -l" under Linux:

Bus 001 Device 006: ID 106c:3b03 Curitel Communications, Inc.
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               1.10
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0
  bDeviceProtocol         0
  bMaxPacketSize0        64
  idVendor           0x106c Curitel Communications, Inc.
  idProduct          0x3b03
  bcdDevice            1.00
  iManufacturer           1 PANTECH
  iProduct                2 USB MMC Storage
  iSerial                 3 000000000002
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           32
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0
    bmAttributes         0xc0
      Self Powered
    MaxPower              100mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass         8 Mass Storage
      bInterfaceSubClass      6 SCSI
      bInterfaceProtocol     80 Bulk (Zip)
      iInterface              0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x83  EP 3 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040  1x 64 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x05  EP 5 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040  1x 64 bytes
        bInterval               0
Device Status:     0x0001
  Self Powered

Note that the above shows a vendor ID of 0x106c ("Curitel Communications, Inc.") and a product ID of 0x3b03.

Should be Easy (or not)

Even before I made the purchase, I ran across a blog post by Jason Costomiris, "HOWTO: Verizon UM175 USB EVDO Card under Ubuntu Hardy". Reading it gave me a good vote of confidence that my plan should work. He had success using the "same" device under the same distribution of Linux that I was planning on. Verizon and Alltel use the same CDMA technology (and Verizon is buying Alltel). The instructions made it almost appear as simple as "plug & play".

Unfortunately, this is where things got tricky. The device is not even shipped with drivers, at least not on a CD. Instead, the device doubles as a USB mass storage device (USB MCS / UMS), and is recognized as an Autorun-enabled CD-Rom with drivers for Windows and Mac OS. Under Windows, after the drivers are installed, the device is listed as a "PANTECH UM175AL Composite Device", a "UM175AL CD-ROM USB Device", a "PANTECH UM175AL Diagnostic Port", and a "PANTECH UM175AL" modem.

By default, Linux only sees this device as a USB CD-Rom. As such, no serial port (e.g. /dev/ttyACM0) is ever created, leaving PPP nothing to connect to.

I need to thank Jason for responding to the comment I left on his post asking for help. Unfortunately, it appears that my "UM175AL" is a slightly different revision than what Jason used with so few issues.

As a sidenote, be aware of this CD-Rom emulation functionality while trying to boot from a real CD/DVD. At least under my Shuttle K-4500-N2, while my UM175 was connected, an external USB DVD-RW that was also connected wasn't given the chance to be used as a boot device. Instead, a boot attempt was made only on the UM175, which of course failed as there is nothing to boot from.

lsusb Note

Many pages I found while researching this issue referenced the output of "/proc/bus/usb/devices". This does not exist under recent versions of Ubuntu Linux. lsusb appears to be one of the preferred replacements. Some details are available in Ubuntu bug #156085 on launchpad.

USB_ModeSwitch

The closest I found to anyone having a similar issue was a post by "theosib" on the Ubuntu Forums with a post less than a month old, "Total disaster trying to set up Verizon EVDO device". I replied with some details of my issue, after which "theosib" replied with what would turn out to be my solution - USB_ModeSwitch.

Be sure to read the main content on USB_ModeSwitch's main page at http://www.draisberghof.de/usb_modeswitch/, as it is a pretty good summary of the issue.

Note that in my response on the UbuntuForums, I stated that when the device is properly recognized by Windows, the Vendor ID was 0x3715, compared to the current 0x3b03. So the goal is getting this to "switch" to 0x3715, at which point it should be (and is) properly recognized as a USB modem by Linux.

The download of USB_ModeSwitch comes with a precompiled executable, but I'm guessing it was compiled for 32-bit, as it wasn't recognized on my system. I recompiled it from the included source without any issues. Just make sure the "build-essential" and "libusb-dev" packages are installed, as this isn't mentioned on the page. Following the notes on the USB_ModeSwitch page, I placed a copy of the output in "/sbin" as "/sbin/usb_modeswitch" using sudo.

Given the about output from "lsusb", I already had the "DefaultVendor" and "DefaultProduct" values needed to configure USB_ModeSwitch. Using these successfully detached the storage driver, but didn't have any effect on the Vendor ID or any apparent change on the output listed by lsusb. Following the documentation, this most likely meant that I still needed to send an additional special command to make the device "switch", using the "MessageEndpoint" and "MessageContent" parameters.

The preferred tool to use to find these parameters mentioned in the USB_ModeSwitch documentation is "SniffUSB". The link is for version 1.8. For me, it would constantly and uncontrollably refresh the screen, making it impossible to scroll the device list and pretty much making it unusable. Fortunately, I found an updated version 2.0, based on the same source, at http://www.pcausa.com/Utilities/UsbSnoop/default.htm. (Source code is readily available for both versions.)

To sniff the necessary commands from this device, I found the VID/PID with the "unswitched" ID's. Specifically, "USB\Vid_106c&Pid_3b03&Rev_0100" ("PANTECH UM175AL Composite Device"). I then installed the filter, then removed and reinserted the device. The following is my captured "UsbSnoop.log", with the necessary message portions highlighted:

[3 ms] UsbSnoop - FilterAddDevice(9a47f748) : DriverObject 89773a48, pdo 8992a9d8
[3 ms] UsbSnoop - FilterDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_QUERY_LEGACY_BUS_INFORMATION)
[3 ms] UsbSnoop - FdoHookDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_QUERY_LEGACY_BUS_INFORMATION)
[4 ms] UsbSnoop - FilterDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_QUERY_RESOURCE_REQUIREMENTS)
[4 ms] UsbSnoop - FdoHookDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_QUERY_RESOURCE_REQUIREMENTS)
[4 ms] UsbSnoop - FilterDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_FILTER_RESOURCE_REQUIREMENTS)
[4 ms] UsbSnoop - FdoHookDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_FILTER_RESOURCE_REQUIREMENTS)
[4 ms] UsbSnoop - FilterDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_START_DEVICE)
[4 ms] UsbSnoop - FdoHookDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_START_DEVICE)
[5 ms] UsbSnoop - FilterDispatchAny(9a47afd2) : IRP_MJ_SYSTEM_CONTROL
[8 ms] UsbSnoop - FilterDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_QUERY_CAPABILITIES)
[8 ms] UsbSnoop - FdoHookDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_QUERY_CAPABILITIES)
[8 ms] UsbSnoop - FilterDispatchAny(9a47afd2) : IRP_MJ_INTERNAL_DEVICE_CONTROL
[8 ms] UsbSnoop - FdoHookDispatchInternalIoctl(9a47b1ea) : fdo=8992a9d8, Irp=888c49d0, IRQL=0
[8 ms]  >>>  URB 1 going down  >>> 
-- URB_FUNCTION_GET_DESCRIPTOR_FROM_DEVICE:
  TransferBufferLength = 00000012
  TransferBuffer       = 89a098f8
  TransferBufferMDL    = 00000000
  Index                = 00000000
  DescriptorType       = 00000001 (USB_DEVICE_DESCRIPTOR_TYPE)
  LanguageId           = 00000000
[11 ms] UsbSnoop - MyInternalIOCTLCompletion(9a47b126) : fido=00000000, Irp=888c49d0, Context=89951008, IRQL=2
[11 ms]  <<<  URB 1 coming back  <<< 
-- URB_FUNCTION_CONTROL_TRANSFER:
  PipeHandle           = 89673020
  TransferFlags        = 0000000b (USBD_TRANSFER_DIRECTION_IN, USBD_SHORT_TRANSFER_OK)
  TransferBufferLength = 00000012
  TransferBuffer       = 89a098f8
  TransferBufferMDL    = 8979e558
    00000000: 12 01 10 01 00 00 00 40 6c 10 03 3b 00 01 01 02
    00000010: 03 01
  UrbLink              = 00000000
  SetupPacket          =
    00000000: 80 06 00 01 00 00 12 00
[11 ms] UsbSnoop - FilterDispatchAny(9a47afd2) : IRP_MJ_INTERNAL_DEVICE_CONTROL
[11 ms] UsbSnoop - FdoHookDispatchInternalIoctl(9a47b1ea) : fdo=8992a9d8, Irp=888c49d0, IRQL=0
[11 ms]  >>>  URB 2 going down  >>> 
-- URB_FUNCTION_GET_DESCRIPTOR_FROM_DEVICE:
  TransferBufferLength = 00000009
  TransferBuffer       = 896cab78
  TransferBufferMDL    = 00000000
  Index                = 00000000
  DescriptorType       = 00000002 (USB_CONFIGURATION_DESCRIPTOR_TYPE)
  LanguageId           = 00000000
[15 ms] UsbSnoop - MyInternalIOCTLCompletion(9a47b126) : fido=00000000, Irp=888c49d0, Context=89951008, IRQL=2
[15 ms]  <<<  URB 2 coming back  <<< 
-- URB_FUNCTION_CONTROL_TRANSFER:
  PipeHandle           = 89673020
  TransferFlags        = 74f06e2f (USBD_TRANSFER_DIRECTION_IN, USBD_SHORT_TRANSFER_OK)
  TransferBufferLength = 00000009
  TransferBuffer       = 896cab78
  TransferBufferMDL    = 8898c160
    00000000: 09 02 20 00 01 01 00 c0 32
  UrbLink              = 00000000
  SetupPacket          =
    00000000: 80 06 00 02 00 00 09 00
[15 ms] UsbSnoop - FilterDispatchAny(9a47afd2) : IRP_MJ_INTERNAL_DEVICE_CONTROL
[15 ms] UsbSnoop - FdoHookDispatchInternalIoctl(9a47b1ea) : fdo=8992a9d8, Irp=888c49d0, IRQL=0
[15 ms]  >>>  URB 3 going down  >>> 
-- URB_FUNCTION_GET_DESCRIPTOR_FROM_DEVICE:
  TransferBufferLength = 00000020
  TransferBuffer       = 89873cc8
  TransferBufferMDL    = 00000000
  Index                = 00000000
  DescriptorType       = 00000002 (USB_CONFIGURATION_DESCRIPTOR_TYPE)
  LanguageId           = 00000000
[19 ms] UsbSnoop - MyInternalIOCTLCompletion(9a47b126) : fido=00000000, Irp=888c49d0, Context=89951008, IRQL=2
[19 ms]  <<<  URB 3 coming back  <<< 
-- URB_FUNCTION_CONTROL_TRANSFER:
  PipeHandle           = 89673020
  TransferFlags        = 74f06e2f (USBD_TRANSFER_DIRECTION_IN, USBD_SHORT_TRANSFER_OK)
  TransferBufferLength = 00000020
  TransferBuffer       = 89873cc8
  TransferBufferMDL    = 8898c160
    00000000: 09 02 20 00 01 01 00 c0 32 09 04 00 00 02 08 06
    00000010: 50 00 07 05 83 02 40 00 00 07 05 05 02 40 00 00
  UrbLink              = 00000000
  SetupPacket          =
    00000000: 80 06 00 02 00 00 20 00
[19 ms] UsbSnoop - FilterDispatchAny(9a47afd2) : IRP_MJ_INTERNAL_DEVICE_CONTROL
[19 ms] UsbSnoop - FdoHookDispatchInternalIoctl(9a47b1ea) : fdo=8992a9d8, Irp=888c49d0, IRQL=0
[19 ms]  >>>  URB 4 going down  >>> 
-- URB_FUNCTION_SELECT_CONFIGURATION:
  ConfigurationDescriptor = 0x89873cc8 (configure)
  ConfigurationDescriptor : bLength             = 9
  ConfigurationDescriptor : bDescriptorType     = 0x00000002
  ConfigurationDescriptor : wTotalLength        = 0x00000020
  ConfigurationDescriptor : bNumInterfaces      = 0x00000001
  ConfigurationDescriptor : bConfigurationValue = 0x00000001
  ConfigurationDescriptor : iConfiguration      = 0x00000000
  ConfigurationDescriptor : bmAttributes        = 0x000000c0
  ConfigurationDescriptor : MaxPower            = 0x00000032
  ConfigurationHandle     = 0x00000000
  Interface[0]: Length            = 56
  Interface[0]: InterfaceNumber   = 0
  Interface[0]: AlternateSetting  = 0
[57 ms] UsbSnoop - MyInternalIOCTLCompletion(9a47b126) : fido=00000000, Irp=888c49d0, Context=89951008, IRQL=0
[57 ms]  <<<  URB 4 coming back  <<< 
-- URB_FUNCTION_SELECT_CONFIGURATION:
  ConfigurationDescriptor = 0x89873cc8 (configure)
  ConfigurationDescriptor : bLength             = 9
  ConfigurationDescriptor : bDescriptorType     = 0x00000002
  ConfigurationDescriptor : wTotalLength        = 0x00000020
  ConfigurationDescriptor : bNumInterfaces      = 0x00000001
  ConfigurationDescriptor : bConfigurationValue = 0x00000001
  ConfigurationDescriptor : iConfiguration      = 0x00000000
  ConfigurationDescriptor : bmAttributes        = 0x000000c0
  ConfigurationDescriptor : MaxPower            = 0x00000032
  ConfigurationHandle     = 0x888d14d0
  Interface[0]: Length            = 56
  Interface[0]: InterfaceNumber   = 0
  Interface[0]: AlternateSetting  = 0
  Interface[0]: Class             = 0x00000008
  Interface[0]: SubClass          = 0x00000006
  Interface[0]: Protocol          = 0x00000050
  Interface[0]: InterfaceHandle   = 0x88bcece0
  Interface[0]: NumberOfPipes     = 2
  Interface[0]: Pipes[0] : MaximumPacketSize = 0x00000040
  Interface[0]: Pipes[0] : EndpointAddress   = 0x00000083
  Interface[0]: Pipes[0] : Interval          = 0x00000000
  Interface[0]: Pipes[0] : PipeType          = 0x00000002 (UsbdPipeTypeBulk)
  Interface[0]: Pipes[0] : PipeHandle        = 0x88bcecfc
  Interface[0]: Pipes[0] : MaxTransferSize   = 0x00001000
  Interface[0]: Pipes[0] : PipeFlags         = 0x00000000
  Interface[0]: Pipes[1] : MaximumPacketSize = 0x00000040
  Interface[0]: Pipes[1] : EndpointAddress   = 0x00000005
  Interface[0]: Pipes[1] : Interval          = 0x00000000
  Interface[0]: Pipes[1] : PipeType          = 0x00000002 (UsbdPipeTypeBulk)
  Interface[0]: Pipes[1] : PipeHandle        = 0x88bced1c
  Interface[0]: Pipes[1] : MaxTransferSize   = 0x00001000
  Interface[0]: Pipes[1] : PipeFlags         = 0x00000000
[57 ms] UsbSnoop - FilterDispatchAny(9a47afd2) : IRP_MJ_INTERNAL_DEVICE_CONTROL
[57 ms] UsbSnoop - FdoHookDispatchInternalIoctl(9a47b1ea) : fdo=8992a9d8, Irp=88da7770, IRQL=0
[57 ms]  >>>  URB 5 going down  >>> 
-- URB_FUNCTION_BULK_OR_INTERRUPT_TRANSFER:
  PipeHandle           = 88bced1c [endpoint 0x00000005]
  TransferFlags        = 00000000 (USBD_TRANSFER_DIRECTION_OUT, ~USBD_SHORT_TRANSFER_OK)
  TransferBufferLength = 0000001f
  TransferBuffer       = 88c4b790
  TransferBufferMDL    = 00000000
    00000000: 55 53 42 43 90 4e d6 8a 24 00 00 00 80 00 08 ff
    00000010: 02 44 45 56 43 48 47 00 00 00 00 00 00 00 00
  UrbLink              = 00000000
[57 ms] UsbSnoop - FilterDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_QUERY_CAPABILITIES)
[57 ms] UsbSnoop - FdoHookDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_QUERY_CAPABILITIES)
[58 ms] UsbSnoop - FilterDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_QUERY_PNP_DEVICE_STATE)
[58 ms] UsbSnoop - FdoHookDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_QUERY_PNP_DEVICE_STATE)
[58 ms] UsbSnoop - FilterDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_QUERY_DEVICE_RELATIONS)
[58 ms] UsbSnoop - FdoHookDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_QUERY_DEVICE_RELATIONS)
[58 ms] UsbSnoop - MyInternalIOCTLCompletion(9a47b126) : fido=00000000, Irp=88da7770, Context=89951008, IRQL=2
[58 ms]  <<<  URB 5 coming back  <<< 
-- URB_FUNCTION_BULK_OR_INTERRUPT_TRANSFER:
  PipeHandle           = 88bced1c [endpoint 0x00000005]
  TransferFlags        = 00000000 (USBD_TRANSFER_DIRECTION_OUT, ~USBD_SHORT_TRANSFER_OK)
  TransferBufferLength = 0000001f
  TransferBuffer       = 88c4b790
  TransferBufferMDL    = 8979e558
  UrbLink              = 00000000
[165 ms] UsbSnoop - FilterDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_QUERY_DEVICE_RELATIONS)
[165 ms] UsbSnoop - FdoHookDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_QUERY_DEVICE_RELATIONS)
[165 ms] UsbSnoop - FilterDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_QUERY_DEVICE_RELATIONS)
[165 ms] UsbSnoop - FdoHookDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_QUERY_DEVICE_RELATIONS)
[166 ms] UsbSnoop - FilterDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_SURPRISE_REMOVAL)
[166 ms] UsbSnoop - FdoHookDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_SURPRISE_REMOVAL)
[240 ms] UsbSnoop - FilterDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_REMOVE_DEVICE)

Given the above information, I then saved the following as "/etc/usb_modeswitch.conf":

DefaultVendor = 0x106c
DefaultProduct = 0x3b03

MessageEndpoint = 0x05
MessageContent = "55534243904ed68a24000000800008ff024445564348470000000000000000"

The following command can then be used to "switch" the mode of the device so that it is recognized as a USB modem. Note that as documented on the USB_ModeSwitch page, most programs using libusb need to be run as root:

sudo /sbin/usb_modeswitch -c /etc/usb_modeswitch.conf

Here is the updated output of "/usr/sbin/lsusb -l":

Bus 001 Device 006: ID 106c:3715 Curitel Communications, Inc.
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               1.10
  bDeviceClass            2 Communications
  bDeviceSubClass         0
  bDeviceProtocol         0
  bMaxPacketSize0        64
  idVendor           0x106c Curitel Communications, Inc.
  idProduct          0x3715
  bcdDevice            1.00
  iManufacturer           1 PANTECH
  iProduct                2 PANTECH USB MODEM
  iSerial                 0
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength          113
    bNumInterfaces          4
    bConfigurationValue     1
    iConfiguration          0
    bmAttributes         0x80
      (Bus Powered)
    MaxPower              500mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         2 Communications
      bInterfaceSubClass      2 Abstract (modem)
      bInterfaceProtocol      1 AT-commands (v.25ter)
      iInterface              0
      CDC Header:
        bcdCDC               1.09
      CDC Call Management:
        bmCapabilities       0x03
          call management
          use DataInterface
        bDataInterface          1
      CDC ACM:
        bmCapabilities       0x0f
          connection notifications
          sends break
          line coding and serial state
          get/set/clear comm features
      CDC Union:
        bMasterInterface        0
        bSlaveInterface         1
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0010  1x 16 bytes
        bInterval              32
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass        10 CDC Data
      bInterfaceSubClass      0 Unused
      bInterfaceProtocol      0
      iInterface              3 Data Interface
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x82  EP 2 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040  1x 64 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x02  EP 2 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040  1x 64 bytes
        bInterval               0
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        2
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass       255 Vendor Specific Class
      bInterfaceSubClass      0
      bInterfaceProtocol      0
      iInterface              0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x84  EP 4 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040  1x 64 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x04  EP 4 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040  1x 64 bytes
        bInterval               0
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        3
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass         8 Mass Storage
      bInterfaceSubClass      6 SCSI
      bInterfaceProtocol     80 Bulk (Zip)
      iInterface              0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x83  EP 3 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040  1x 64 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x05  EP 5 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040  1x 64 bytes
        bInterval               0
Device Status:     0x0001
  Self Powered

At this point, I was finally able to establish a connection using PPP. This setting appears to even persist across reboots, as long as the USB device doesn't loose power. Following the examples on the USB_ModeSwitch page, I saved the following as "/etc/udev/rules.d/99-usb_modeswitch.rules" which will automatically run the above command whenever the device is detected:

SUBSYSTEM=="usb", ATTR{idProduct}=="3b03", ATTR{idVendor}=="106c", \
        RUN+="/sbin/usb_modeswitch -c /etc/usb_modeswitch.conf"

Note that there is a "README" file in the "/etc/udev/rules.d" directory. Additional information can be found in the man page for udev ("man udev").

Accessing the Modem

The "cdc_acm" kernel module should now detect a connected device, and connect it to "/dev/ttyACM0".

Before making the Internet connection with PPP, the modem can be communicated with as a serial device. I'm not sure what the preferred method is for doing this under Linux, but the "screen" command (located in "/usr/bin") worked for me:

/usr/bin/screen /dev/ttyACM0

AT
OK
AT+GMM
UM175AL

OK
ATI
Manufacturer: UTStarcom communication Inc.
Model: UM175AL
Revision: D0700ALM01_5.226  1  [Mar 21 2008 06:00:00] [Jul 29 2008 15:10:28]
ESN: 0x83AF77E
+GCAP: +CIS707-A, CIS-856, CIS-856-A

OK
AT+CSQ
31, 0

OK

(If you don't recognize the above, the Hayes command set on Wikipedia may be a good starting reference.)

Signal Strength

The "AT+CSQ" returns the current signal strength as a value between 0-31, with higher being better. According to Ken Kinder, this number indicates the signal strength above -109 dBm in 2 dBm increments.

I know the Windows client software displays a regularly updated signal strength indicator, even while the connection is active. It would be nice to have this functionality available under Linux, too, but it doesn't look easy, if even possible.

I found one forum thread with one response on the topic: http://fixunix.com/ppp/62263-commad-mode-gprs-same-time.html. Pretty much, it seems that the data stream would have to be paused, the modem dropped to command mode, the "AT+CSQ" command issued and the value retrieved, then sending "ATO" to go back online. The forum response suggested that a modified pppd would be needed. I'm wondering if a "wrapper" tty couldn't be written that would be passed to pppd, and query the signal strength whenever the connection is idle. Alternatively, I'd be curious if the "diagnostic port" as shown in Windows would provide similar functionality without having to interrupt the data line? Maybe this is available by one of the other interface descriptors shown in the lsusb output (above)? Maybe "bInterfaceClass 10 CDC Data"?

Update (2008-10-08): Sysinternal's Portmon shows that under Windows, the The Alltel "QuickLink Mobile" Wireless Connection Manager communicates with the "diagnostic port". Unfortunately, unlike the text-based "AT" commands used above, this communication looks completely binary. It also doesn't look to be easily decipherable, or to use any known standard. I've sent an email to UTStarcom, and can only hope for an unlikely helpful response.

Here's the output from about 5 seconds of communication with QuickLink Mobile and the "diagnostic port" using Portmon:

0  0.00000000  QuickLink Mobil  IRP_MJ_CREATE  PTDLserd0  Options: Open 
0  0.00584593  SUCCESS  
1  0.00000000  QuickLink Mobil  IOCTL_SERIAL_SET_DTR  PTDLserd0  
1  0.00000206  SUCCESS  
2  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_PROPERTIES  PTDLserd0  
2  0.00000052  SUCCESS  
3  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_BAUD_RATE  PTDLserd0  
3  0.00000072  SUCCESS  
4  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_LINE_CONTROL  PTDLserd0  
4  0.00000064  SUCCESS  
5  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_CHARS  PTDLserd0  
5  0.00000048  SUCCESS  
6  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_HANDFLOW  PTDLserd0  
6  0.00000046  SUCCESS  
7  0.00000000  QuickLink Mobil  IOCTL_SERIAL_SET_QUEUE_SIZE  PTDLserd0  InSize: 4096 OutSize: 4096
7  0.00000044  SUCCESS  
8  0.00000000  QuickLink Mobil  IOCTL_SERIAL_SET_TIMEOUTS  PTDLserd0  RI:-1 RM:0 RC:0 WM:0 WC:0
8  0.00000109  SUCCESS  
9  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_MODEMSTATUS  PTDLserd0  
9  0.00000130  SUCCESS  
10  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_PROPERTIES  PTDLserd0  
10  0.00000053  SUCCESS  
11  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_BAUD_RATE  PTDLserd0  
11  0.00000093  SUCCESS  
12  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_LINE_CONTROL  PTDLserd0  
12  0.00000112  SUCCESS  
13  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_CHARS  PTDLserd0  
13  0.00000051  SUCCESS  
14  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_HANDFLOW  PTDLserd0  
14  0.00000045  SUCCESS  
15  0.00000000  QuickLink Mobil  IOCTL_SERIAL_PURGE  PTDLserd0  Purge: TXABORT TXCLEAR 
15  0.00000196  SUCCESS  
16  0.00000000  QuickLink Mobil  IOCTL_SERIAL_SET_QUEUE_SIZE  PTDLserd0  InSize: 4096 OutSize: 4096
16  0.00000096  SUCCESS  
17  0.00000000  QuickLink Mobil  IOCTL_SERIAL_SET_TIMEOUTS  PTDLserd0  RI:-1 RM:0 RC:0 WM:0 WC:0
17  0.00000124  SUCCESS  
18  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_MODEMSTATUS  PTDLserd0  
18  0.00000117  SUCCESS  
19  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_PROPERTIES  PTDLserd0  
19  0.00000065  SUCCESS  
20  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_BAUD_RATE  PTDLserd0  
20  0.00000109  SUCCESS  
21  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_LINE_CONTROL  PTDLserd0  
21  0.00000061  SUCCESS  
22  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_CHARS  PTDLserd0  
22  0.00000050  SUCCESS  
23  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_HANDFLOW  PTDLserd0  
23  0.00000045  SUCCESS  
24  0.00000000  QuickLink Mobil  IOCTL_SERIAL_PURGE  PTDLserd0  Purge: TXABORT TXCLEAR 
24  0.00000186  SUCCESS  
25  0.00000000  QuickLink Mobil  IOCTL_SERIAL_SET_QUEUE_SIZE  PTDLserd0  InSize: 4096 OutSize: 4096
25  0.00000181  SUCCESS  
26  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_BAUD_RATE  PTDLserd0  
26  0.00000140  SUCCESS  
27  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_LINE_CONTROL  PTDLserd0  
27  0.00000083  SUCCESS  
28  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_CHARS  PTDLserd0  
28  0.00000087  SUCCESS  
29  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_HANDFLOW  PTDLserd0  
29  0.00000065  SUCCESS  
30  0.00000000  QuickLink Mobil  IOCTL_SERIAL_SET_BAUD_RATE  PTDLserd0  Rate: 115200
30  0.00000137  SUCCESS  
31  0.00000000  QuickLink Mobil  IOCTL_SERIAL_SET_DTR  PTDLserd0  
31  0.00000072  SUCCESS  
32  0.00000000  QuickLink Mobil  IOCTL_SERIAL_SET_LINE_CONTROL  PTDLserd0  StopBits: 1 Parity: NONE WordLength: 8
32  0.00000115  SUCCESS  
33  0.00000000  QuickLink Mobil  IOCTL_SERIAL_SET_CHAR  PTDLserd0  EOF:0 ERR:0 BRK:0 EVT:0 XON:11 XOFF:13
33  0.00000053  SUCCESS  
34  0.00000000  QuickLink Mobil  IOCTL_SERIAL_SET_HANDFLOW  PTDLserd0  Shake:9 Replace:80 XonLimit:8192 XoffLimit:51200
34  0.00000224  SUCCESS  
35  0.00000000  QuickLink Mobil  IOCTL_SERIAL_SET_TIMEOUTS  PTDLserd0  RI:-1 RM:0 RC:0 WM:0 WC:0
35  0.00000049  SUCCESS  
36  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_MODEMSTATUS  PTDLserd0  
36  0.00000088  SUCCESS  
37  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_PROPERTIES  PTDLserd0  
37  0.00000040  SUCCESS  
38  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_BAUD_RATE  PTDLserd0  
38  0.00000070  SUCCESS  
39  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_LINE_CONTROL  PTDLserd0  
39  0.00000048  SUCCESS  
40  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_CHARS  PTDLserd0  
40  0.00000045  SUCCESS  
41  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_HANDFLOW  PTDLserd0  
41  0.00000044  SUCCESS  
42  0.00000000  QuickLink Mobil  IOCTL_SERIAL_PURGE  PTDLserd0  Purge: TXABORT TXCLEAR 
42  0.00000182  SUCCESS  
43  0.00000000  QuickLink Mobil  IOCTL_SERIAL_SET_QUEUE_SIZE  PTDLserd0  InSize: 2048 OutSize: 2048
43  0.00000203  SUCCESS  
44  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_BAUD_RATE  PTDLserd0  
44  0.00000223  SUCCESS  
45  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_LINE_CONTROL  PTDLserd0  
45  0.00000106  SUCCESS  
46  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_CHARS  PTDLserd0  
46  0.00000066  SUCCESS  
47  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_HANDFLOW  PTDLserd0  
47  0.00000069  SUCCESS  
48  0.00000000  QuickLink Mobil  IOCTL_SERIAL_SET_BAUD_RATE  PTDLserd0  Rate: 115200
48  0.00000098  SUCCESS  
49  0.00000000  QuickLink Mobil  IOCTL_SERIAL_SET_DTR  PTDLserd0  
49  0.00000127  SUCCESS  
50  0.00000000  QuickLink Mobil  IOCTL_SERIAL_SET_LINE_CONTROL  PTDLserd0  StopBits: 1 Parity: NONE WordLength: 8
50  0.00000107  SUCCESS  
51  0.00000000  QuickLink Mobil  IOCTL_SERIAL_SET_CHAR  PTDLserd0  EOF:0 ERR:0 BRK:0 EVT:0 XON:11 XOFF:13
51  0.00000052  SUCCESS  
52  0.00000000  QuickLink Mobil  IOCTL_SERIAL_SET_HANDFLOW  PTDLserd0  Shake:9 Replace:80 XonLimit:8192 XoffLimit:51200
52  0.00000193  SUCCESS  
53  0.00000000  QuickLink Mobil  IOCTL_SERIAL_SET_TIMEOUTS  PTDLserd0  RI:-1 RM:0 RC:0 WM:0 WC:0
53  0.00000049  SUCCESS  
54  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_MODEMSTATUS  PTDLserd0  
54  0.00000090  SUCCESS  
55  0.00000000  QuickLink Mobil  IRP_MJ_WRITE  PTDLserd0  Length 5: C8 0D C8 D0 7E 
56  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
56  0.00000087  SUCCESS  
55  0.00074312  SUCCESS  
57  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
57  0.00000118  SUCCESS  
58  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
58  0.00000049  SUCCESS  
59  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 9
59  0.00000169  SUCCESS  Length 9: C8 0D 00 00 00 00 D0 AD 7E 
60  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
60  0.00000082  SUCCESS  
61  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
61  0.00000311  SUCCESS  
62  0.00000000  QuickLink Mobil  IRP_MJ_WRITE  PTDLserd0  Length 5: C8 06 1B 6E 7E 
63  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
63  0.00000173  SUCCESS  
62  0.00034797  SUCCESS  
64  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
64  0.00000113  SUCCESS  
65  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
65  0.00000077  SUCCESS  
66  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
66  0.00000048  SUCCESS  
67  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 533
67  0.00000255  SUCCESS  Length 533: C8 06 09 02 0A 08 05 0A 09 06 05 05 00 00 00 00 00 00 00 00 00 00 00 00 7D 5E F7 3A 08 01 55 54 53 74 61 72 63 6F 6D 20 49 6E 63 2E 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
68  0.00000000  QuickLink Mobil  IRP_MJ_WRITE  PTDLserd0  Length 5: C8 06 1B 6E 7E 
69  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
69  0.00000084  SUCCESS  
68  0.00111850  SUCCESS  
70  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
70  0.00000095  SUCCESS  
71  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
71  0.00000076  SUCCESS  
72  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
72  0.00000048  SUCCESS  
73  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 533
73  0.00000207  SUCCESS  Length 533: C8 06 09 02 0A 08 05 0A 09 06 05 05 00 00 00 00 00 00 00 00 00 00 00 00 7D 5E F7 3A 08 01 55 54 53 74 61 72 63 6F 6D 20 49 6E 63 2E 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
74  0.00000000  QuickLink Mobil  IRP_MJ_WRITE  PTDLserd0  Length 5: C8 06 1B 6E 7E 
75  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
75  0.00000127  SUCCESS  
74  0.00036960  SUCCESS  
76  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
76  0.00000076  SUCCESS  
77  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
77  0.00000085  SUCCESS  
78  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
78  0.00000047  SUCCESS  
79  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 533
79  0.00000188  SUCCESS  Length 533: C8 06 09 02 0A 08 05 0A 09 06 05 05 00 00 00 00 00 00 00 00 00 00 00 00 7D 5E F7 3A 08 01 55 54 53 74 61 72 63 6F 6D 20 49 6E 63 2E 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
80  0.00000000  QuickLink Mobil  IRP_MJ_WRITE  PTDLserd0  Length 5: C8 06 1B 6E 7E 
81  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
81  0.00000095  SUCCESS  
80  0.00067287  SUCCESS  
82  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
82  0.00000109  SUCCESS  
83  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
83  0.00000071  SUCCESS  
84  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 533
84  0.00000212  SUCCESS  Length 533: C8 06 09 02 0A 08 05 0A 09 06 05 05 00 00 00 00 00 00 00 00 00 00 00 00 7D 5E F7 3A 08 01 55 54 53 74 61 72 63 6F 6D 20 49 6E 63 2E 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
85  0.00000000  QuickLink Mobil  IRP_MJ_WRITE  PTDLserd0  Length 5: C8 0A 77 A4 7E 
86  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
86  0.00000263  SUCCESS  
85  0.00083082  SUCCESS  
87  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
87  0.00000139  SUCCESS  
88  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
88  0.00000059  SUCCESS  
89  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 25
89  0.00000212  SUCCESS  Length 25: C8 0A 60 8B 00 00 F1 83 00 00 0F 21 00 00 76 0B 00 00 04 C0 0B 00 01 A0 7E 
90  0.00000000  QuickLink Mobil  IRP_MJ_WRITE  PTDLserd0  Length 5: C8 0B FE B5 7E 
91  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
91  0.00000176  SUCCESS  
90  0.00071377  SUCCESS  
92  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
92  0.00000117  SUCCESS  
93  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
93  0.00000082  SUCCESS  
94  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 75
94  0.00000291  SUCCESS  Length 75: C8 0B 07 00 00 00 09 01 D8 07 0A 00 13 00 0D 00 0A 00 37 00 41 03 D4 FE FF FF 5D 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 67 00 00 00 02 00 00 00 E4 3C 7E 
95  0.00000000  QuickLink Mobil  IRP_MJ_WRITE  PTDLserd0  Length 5: C8 06 1B 6E 7E 
96  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
96  0.00000122  SUCCESS  
95  0.00088541  SUCCESS  
97  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
97  0.00000116  SUCCESS  
98  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
98  0.00000075  SUCCESS  
99  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 533
99  0.00000256  SUCCESS  Length 533: C8 06 09 02 0A 08 05 0A 09 06 05 05 00 00 00 00 00 00 00 00 00 00 00 00 7D 5E F7 3A 08 01 55 54 53 74 61 72 63 6F 6D 20 49 6E 63 2E 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
100  0.00000000  QuickLink Mobil  IRP_MJ_WRITE  PTDLserd0  Length 10: C8 24 01 00 00 00 00 0F 8D 7E 
101  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
101  0.00000119  SUCCESS  
100  0.00042567  SUCCESS  
102  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
102  0.00000094  SUCCESS  
103  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
103  0.00000047  SUCCESS  
104  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 9
104  0.00000161  SUCCESS  Length 9: C8 24 00 00 00 00 25 9C 7E 
105  0.00000000  QuickLink Mobil  IRP_MJ_WRITE  PTDLserd0  Length 10: C8 24 02 00 00 00 00 C3 90 7E 
106  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
106  0.00000114  SUCCESS  
105  0.00036322  SUCCESS  
107  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
107  0.00000118  SUCCESS  
108  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
108  0.00000067  SUCCESS  
109  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 9
109  0.00000213  SUCCESS  Length 9: C8 24 00 00 00 00 25 9C 7E 
110  0.00000000  QuickLink Mobil  IRP_MJ_WRITE  PTDLserd0  Length 10: C8 24 03 00 00 00 00 87 9B 7E 
111  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
111  0.00000094  SUCCESS  
110  0.00046675  SUCCESS  
112  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
112  0.00000155  SUCCESS  
113  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
113  0.00000062  SUCCESS  
114  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 9
114  0.00000206  SUCCESS  Length 9: C8 24 00 00 00 00 25 9C 7E 
115  0.00000000  QuickLink Mobil  IRP_MJ_WRITE  PTDLserd0  Length 10: C8 24 04 00 00 00 00 5B AB 7E 
116  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
116  0.00000120  SUCCESS  
115  0.00061533  SUCCESS  
117  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
117  0.00000103  SUCCESS  
118  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
118  0.00000078  SUCCESS  
119  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 9
119  0.00000175  SUCCESS  Length 9: C8 24 00 00 00 00 25 9C 7E 
120  0.00000000  QuickLink Mobil  IRP_MJ_WRITE  PTDLserd0  Length 10: C8 24 05 00 00 00 00 1F A0 7E 
121  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
121  0.00000125  SUCCESS  
120  0.00081854  SUCCESS  
122  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
122  0.00000147  SUCCESS  
123  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
123  0.00000050  SUCCESS  
124  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 9
124  0.00000193  SUCCESS  Length 9: C8 24 00 00 00 00 25 9C 7E 
125  0.00000000  QuickLink Mobil  IRP_MJ_WRITE  PTDLserd0  Length 10: C8 24 09 00 00 00 00 2F D7 7E 
126  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
126  0.00000104  SUCCESS  
125  0.00089310  SUCCESS  
127  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
127  0.00000126  SUCCESS  
128  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
128  0.00000069  SUCCESS  
129  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 9
129  0.00000203  SUCCESS  Length 9: C8 24 00 00 00 00 25 9C 7E 
130  0.00000000  QuickLink Mobil  IRP_MJ_WRITE  PTDLserd0  Length 10: C8 24 0B 00 00 00 00 A7 C1 7E 
131  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
131  0.00000117  SUCCESS  
130  0.00108786  SUCCESS  
132  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
132  0.00000096  SUCCESS  
133  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
133  0.00000047  SUCCESS  
134  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 9
134  0.00000125  SUCCESS  Length 9: C8 24 00 00 00 00 25 9C 7E 
135  0.00000000  QuickLink Mobil  IRP_MJ_WRITE  PTDLserd0  Length 10: C8 24 0D 00 00 00 00 3F FA 7E 
136  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
136  0.00000093  SUCCESS  
135  0.00020632  SUCCESS  
137  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
137  0.00000208  SUCCESS  
138  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
138  0.00000050  SUCCESS  
139  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 9
139  0.00000217  SUCCESS  Length 9: C8 24 01 00 00 00 9E 80 7E 
140  0.00000000  QuickLink Mobil  IRP_MJ_WRITE  PTDLserd0  Length 10: C8 24 0E 00 00 00 00 F3 E7 7E 
141  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
141  0.00000094  SUCCESS  
140  0.00032217  SUCCESS  
142  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
142  0.00000114  SUCCESS  
143  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
143  0.00000047  SUCCESS  
144  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 9
144  0.00000144  SUCCESS  Length 9: C8 24 01 00 00 00 9E 80 7E 
145  0.00000000  QuickLink Mobil  IRP_MJ_WRITE  PTDLserd0  Length 10: C8 24 0F 00 00 00 00 B7 EC 7E 
146  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
146  0.00000116  SUCCESS  
145  0.00047807  SUCCESS  
147  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
147  0.00000118  SUCCESS  
148  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
148  0.00000047  SUCCESS  
149  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 9
149  0.00000209  SUCCESS  Length 9: C8 24 01 00 00 00 9E 80 7E 
150  0.00000000  QuickLink Mobil  IRP_MJ_WRITE  PTDLserd0  Length 5: C8 0C 41 C1 7E 
151  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
151  0.00000103  SUCCESS  
150  0.00036696  SUCCESS  
152  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
152  0.00000161  SUCCESS  
153  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
153  0.00000049  SUCCESS  
154  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 10
154  0.00000218  SUCCESS  Length 10: C8 0C 02 00 00 00 00 FB 34 7E 
155  0.00000000  QuickLink Mobil  IRP_MJ_WRITE  PTDLserd0  Length 9: C8 11 01 00 00 00 1B 72 7E 
156  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
156  0.00000077  SUCCESS  
155  0.00070517  SUCCESS  
157  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
157  0.00000151  SUCCESS  
158  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
158  0.00000047  SUCCESS  
159  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 9
159  0.00000209  SUCCESS  Length 9: C8 11 00 00 00 00 A0 6E 7E 
160  0.00000000  QuickLink Mobil  IRP_MJ_WRITE  PTDLserd0  Length 5: C8 06 1B 6E 7E 
161  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
161  0.00000263  SUCCESS  
160  0.00093037  SUCCESS  
162  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
162  0.00000083  SUCCESS  
163  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
163  0.00000048  SUCCESS  
164  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 533
164  0.00000199  SUCCESS  Length 533: C8 06 09 02 0A 08 05 0A 09 06 05 05 00 00 00 00 00 00 00 00 00 00 00 00 7D 5E F7 3A 08 01 55 54 53 74 61 72 63 6F 6D 20 49 6E 63 2E 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
165  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
165  0.00000216  SUCCESS  
166  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
166  0.00000051  SUCCESS  
167  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 150
167  0.00000215  SUCCESS  Length 150: C8 0B 01 00 00 00 09 01 D8 07 0A 00 13 00 0D 00 0A 00 38 00 04 03 D4 FE FF FF 61 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 67 00 00 00 02 00 00 00 9C 7C 7E C8 0B 01 00 00 00 09 01 D8 07 
168  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
168  0.00000056  SUCCESS  
169  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
169  0.00000154  SUCCESS  
170  0.00000000  QuickLink Mobil  IRP_MJ_WRITE  PTDLserd0  Length 5: C8 0A 77 A4 7E 
171  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
171  0.00000186  SUCCESS  
170  0.00085352  SUCCESS  
172  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
172  0.00000127  SUCCESS  
173  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
173  0.00000048  SUCCESS  
174  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 25
174  0.00000177  SUCCESS  Length 25: C8 0A D1 92 00 00 62 8B 00 00 3E 22 00 00 76 0B 00 00 04 C0 0B 00 21 42 7E 
175  0.00000000  QuickLink Mobil  IRP_MJ_WRITE  PTDLserd0  Length 5: C8 06 1B 6E 7E 
176  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
176  0.00000227  SUCCESS  
175  0.00058334  SUCCESS  
177  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
177  0.00000230  SUCCESS  
178  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
178  0.00000237  SUCCESS  
179  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
179  0.00000075  SUCCESS  
180  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 533
180  0.00000325  SUCCESS  Length 533: C8 06 09 02 0A 08 05 0A 09 06 05 05 00 00 00 00 00 00 00 00 00 00 00 00 7D 5E F7 3A 08 01 55 54 53 74 61 72 63 6F 6D 20 49 6E 63 2E 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
181  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
181  0.00000379  SUCCESS  
182  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
182  0.00000275  SUCCESS  
183  0.00000000  QuickLink Mobil  IRP_MJ_WRITE  PTDLserd0  Length 5: C8 0A 77 A4 7E 
184  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
184  0.00000208  SUCCESS  
185  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
185  0.00000094  SUCCESS  
186  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 101
186  0.00000292  SUCCESS  Length 101: C8 0B 07 00 00 00 09 01 D8 07 0A 00 13 00 0D 00 0A 00 39 00 7D 5E 01 D4 FE FF FF 5F 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 66 00 00 00 02 00 00 00 A7 07 7E C8 0A 4D 9A 00 00 DE 92 00 
183  0.00095781  SUCCESS  
187  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
187  0.00000205  SUCCESS  
188  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
188  0.00000049  SUCCESS  
189  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 100
189  0.00000220  SUCCESS  Length 100: C8 0A 84 9A 00 00 15 93 00 00 AD 25 00 00 76 0B 00 00 04 C0 0B 00 35 18 7E C8 0B 07 00 00 00 09 01 D8 07 0A 00 13 00 0D 00 0A 00 3A 00 0C 00 D4 FE FF FF 5F 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
190  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
190  0.00000074  SUCCESS  
191  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
191  0.00000094  SUCCESS  
192  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
192  0.00000077  SUCCESS  
193  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
193  0.00000217  SUCCESS  
194  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
194  0.00000090  SUCCESS  
195  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
195  0.00000060  SUCCESS  
196  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
196  0.00000264  SUCCESS  
197  0.00000000  QuickLink Mobil  IRP_MJ_WRITE  PTDLserd0  Length 5: C8 0A 77 A4 7E 
198  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
198  0.00000227  SUCCESS  
197  0.00051973  SUCCESS  
199  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
199  0.00000158  SUCCESS  
200  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
200  0.00000071  SUCCESS  
201  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 25
201  0.00000228  SUCCESS  Length 25: C8 0A 44 A6 00 00 D5 9E 00 00 AD 25 00 00 76 0B 00 00 04 C0 0B 00 A8 53 7E 
202  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
202  0.00000346  SUCCESS  
203  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
203  0.00000115  SUCCESS  
204  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
204  0.00000070  SUCCESS  
205  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
205  0.00000235  SUCCESS  
206  0.00000000  QuickLink Mobil  IRP_MJ_WRITE  PTDLserd0  Length 10: C8 24 01 00 00 00 01 86 9C 7E 
207  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
207  0.00000220  SUCCESS  
206  0.00100681  SUCCESS  
208  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
208  0.00000202  SUCCESS  
209  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
209  0.00000049  SUCCESS  
210  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 9
210  0.00000179  SUCCESS  Length 9: C8 24 00 00 00 00 25 9C 7E 
211  0.00000000  QuickLink Mobil  IRP_MJ_WRITE  PTDLserd0  Length 10: C8 24 02 00 00 00 01 4A 81 7E 
212  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
212  0.00000091  SUCCESS  
211  0.00057449  SUCCESS  
213  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
213  0.00000101  SUCCESS  
214  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
214  0.00000074  SUCCESS  
215  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 9
215  0.00000172  SUCCESS  Length 9: C8 24 00 00 00 00 25 9C 7E 
216  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
216  0.00000103  SUCCESS  
217  0.00000000  QuickLink Mobil  IRP_MJ_WRITE  PTDLserd0  Length 10: C8 24 03 00 00 00 01 0E 8A 7E 
217  0.00063347  SUCCESS  
218  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
218  0.00000171  SUCCESS  
219  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
219  0.00000050  SUCCESS  
220  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 9
220  0.00000160  SUCCESS  Length 9: C8 24 00 00 00 00 25 9C 7E 
221  0.00000000  QuickLink Mobil  IRP_MJ_WRITE  PTDLserd0  Length 10: C8 24 04 00 00 00 01 D2 BA 7E 
222  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
222  0.00000117  SUCCESS  
221  0.00093368  SUCCESS  
223  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
223  0.00000107  SUCCESS  
224  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
224  0.00000048  SUCCESS  
225  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 9
225  0.00000138  SUCCESS  Length 9: C8 24 00 00 00 00 25 9C 7E 
226  0.00000000  QuickLink Mobil  IRP_MJ_WRITE  PTDLserd0  Length 10: C8 24 05 00 00 00 01 96 B1 7E 
227  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
227  0.00000096  SUCCESS  
226  0.00111167  SUCCESS  
228  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
228  0.00000141  SUCCESS  
229  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
229  0.00000048  SUCCESS  
230  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 9
230  0.00000194  SUCCESS  Length 9: C8 24 00 00 00 00 25 9C 7E 
231  0.00000000  QuickLink Mobil  IRP_MJ_WRITE  PTDLserd0  Length 10: C8 24 09 00 00 00 01 A6 C6 7E 
232  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
232  0.00000139  SUCCESS  
231  0.00120009  SUCCESS  
233  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
233  0.00000169  SUCCESS  
234  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
234  0.00000048  SUCCESS  
235  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 9
235  0.00000194  SUCCESS  Length 9: C8 24 00 00 00 00 25 9C 7E 
236  0.00000000  QuickLink Mobil  IRP_MJ_WRITE  PTDLserd0  Length 10: C8 24 0B 00 00 00 01 2E D0 7E 
237  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
237  0.00000143  SUCCESS  
236  0.00095763  SUCCESS  
238  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
238  0.00000110  SUCCESS  
239  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
239  0.00000076  SUCCESS  
240  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 9
240  0.00000349  SUCCESS  Length 9: C8 24 00 00 00 00 25 9C 7E 
241  0.00000000  QuickLink Mobil  IRP_MJ_WRITE  PTDLserd0  Length 10: C8 24 0D 00 00 00 01 B6 EB 7E 
242  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
242  0.00000091  SUCCESS  
241  0.00039045  SUCCESS  
243  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
243  0.00000138  SUCCESS  
244  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
244  0.00000047  SUCCESS  
245  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 9
245  0.00000172  SUCCESS  Length 9: C8 24 01 00 00 00 9E 80 7E 
246  0.00000000  QuickLink Mobil  IRP_MJ_WRITE  PTDLserd0  Length 10: C8 24 0E 00 00 00 01 7A F6 7E 
247  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
247  0.00000095  SUCCESS  
246  0.00057285  SUCCESS  
248  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
248  0.00000123  SUCCESS  
249  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
249  0.00000048  SUCCESS  
250  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 9
250  0.00000158  SUCCESS  Length 9: C8 24 01 00 00 00 9E 80 7E 
251  0.00000000  QuickLink Mobil  IRP_MJ_WRITE  PTDLserd0  Length 10: C8 24 0F 00 00 00 01 3E FD 7E 
252  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
252  0.00000136  SUCCESS  
251  0.00070754  SUCCESS  
253  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
253  0.00000132  SUCCESS  
254  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
254  0.00000046  SUCCESS  
255  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 9
255  0.00000192  SUCCESS  Length 9: C8 24 01 00 00 00 9E 80 7E 
256  0.00000000  QuickLink Mobil  IRP_MJ_WRITE  PTDLserd0  Length 5: C8 0E 53 E2 7E 
257  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
257  0.00000095  SUCCESS  
256  0.00084635  SUCCESS  
258  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
258  0.00000123  SUCCESS  
259  0.00000000  QuickLink Mobil  IOCTL_SERIAL_GET_COMMSTATUS  PTDLserd0  
259  0.00000072  SUCCESS  
260  0.00000000  QuickLink Mobil  IRP_MJ_READ  PTDLserd0  Length 9
260  0.00000206  SUCCESS  Length 9: C8 0E 66 00 00 00 62 62 7E 
261  0.00000000  QuickLink Mobil  IOCTL_SERIAL_PURGE  PTDLserd0  Purge: TXABORT TXCLEAR 
261  0.00000271  SUCCESS  
262  0.00000000  QuickLink Mobil  IOCTL_SERIAL_CLR_DTR  PTDLserd0  
262  0.00000225  SUCCESS  
263  0.00000000  QuickLink Mobil  IRP_MJ_CLEANUP  PTDLserd0  
263  0.00265768  SUCCESS  
264  0.00000000  QuickLink Mobil  IRP_MJ_CLOSE  PTDLserd0  
264  0.00000111  SUCCESS  

To be continued...

Next up: Configuring Persistent PPP under Ubuntu.

Shuttle K-4500-N2

(This post is the 2nd part of my Ubuntu Linux Router Upgrade Project.)

I chose Shuttle's KPC4500 model, purchased from Newegg as the K-4500-N2 (#N82E16883104036) for about $210.

Surprisingly, I currently have the only posted customer review for this product on Newegg.

Specs

While the K4500 certainly isn't a top-of-the-line system, it's definitely suitable for this task. Included is a 2.0 GHz 64-bit processor, 160 GB SATA hard drive, 10/100/1000 Mbps Ethernet, and integrated video & audio. It has 4 USB ports, as well as COM/serial, LPT/parallel and PS/2. Internally, there is room for a 2nd HD, with an 1 additional SATA and 1 IDE port available, as well as some extra USB headers.

The K-4500-N2 came configured with 512 MB RAM, which is shared by the on-board video card. The sharing can be configured in the BIOS for a dynamic (using DVMT) or fixed partition. While this should be plenty for the setup, maxing out the supported memory is relatively cheap - so I decided to do it right away instead of later. I couldn't find a common product with appropriate specs that was in the memory support list for the K45 on Shuttle's web site and that Newegg carried, but a Kingston 2x1 GB pair (Newegg # N82E16820134117, ~$30) is working well. While this required removing the supplied 512 MB chip, this had the added benefit of enabling dual-channel mode.

With a largest dimension of 11", it is space-saving. Though I haven't measured A/C usage, it also seems that it is also rather energy efficient - with only a 100W power supply, and very few extra components to power.

One peripheral to note as missing is an optical drive, though there is one available on the K4800 model. However, short of reinstalling the operating system, I rarely expect a need for one. It does support an external USB DVD drive I have without any apparent issues, and worked well for reinstalling Linux.

Operating System

I was not impressed with the supplied installation of Foresight Linux. Several menu options only produced errors - not a good first impression. Fortunately, it's free to begin with, so I had nothing to loose by simply doing a clean install of the latest Ubuntu x64 - which works great.

Other Thoughts

Especially without optimal viewing conditions, the power button on the front can easily be mistaken for a USB port. Considering how there are already spare USB headers on the motherboard and available space on the front panel, at least one USB port on the front panel would be nice for connecting flash drives, etc.

There's a clear spot on the back plate for a 2nd Ethernet jack, which could have been nice for use as a router. However, it's nothing that VLAN support and/or an external USB adapter can't resolve.

To be continued...

Next up: Alltel UM175AL USB EVDO under Ubuntu Hardy Heron.