How to (not) Validate an Email Address

A common requirement to a software developer often follows the lines of "accept a valid email address". Somewhat appropriately, use of a regular expression is typically considered to fulfill this requirement. Unfortunately, many times this is made more complicated than necessary - and only causes additional issues due to becoming too restrictive.

This scenario is best explained by "I Knew How To Validate An Email Address Until I Read The RFC" (Phil Haack, 2007-08-21, haacked.com). I highly encourage you to read through the entire article, but here are a few significant quotes:

What I found out was surprising. Nearly 100% of regular expressions on the web purporting to validate an email address are too strict.

These are all valid email addresses!

"Abc\@def"@example.com
"Fred Bloggs"@example.com
"Joe\\Blow"@example.com
"Abc@def"@example.com
customer/department=shipping@example.com
$A12345@example.com
!def!xyz%abc@example.com
_somename@example.com

The article goes on to suggest the following regular expression (complete with unit tests!):

^(?!\.)("([^"\r\\]|\\["\r\\])*"|([-a-z0-9!#$%&'*+/=?^_`{|}~] |(?@[a-z0-9][\w\.-]*[a-z0-9]\.[a-z][a-z\.]*[a-z]$

However - this still misrepresents the real issue. There is only one way to validate an email address: Send something containing a cryptographically secure, one-time token to the email address in question, and prompt the user to provide it (either by input back to the web page, and/or by clicking on a link to invoke the validation). If and only if the user is able to complete this step, it can be reasonably assured that the user owns or at least maintains control of the provided email address.

As such, any related requirement should be relaxed to only ensure that whatever is being entered at least "looks" like an email address. This isn't so much for the purpose of ensuring that the address is valid - but that the proper data is being entered into the correct fields, such that an email address is being entered when prompted, instead of a name or a phone number, for example. To accomplish this, the following regular expression is sufficient:

.+@.+\..+

This basically indicates that in order for a given text to match against this pattern, it must contain one or more characters, followed by the '@' sign, followed by one or more characters, followed by a dot ('.'), followed by one or more characters. (Any of the "one or more characters" will also allow for one or more additional '@' signs or dots ('.').)

So for example, the below is an example validation implemented in JavaScript that "makes sense":

var emailValidation = /.+@.+\..+/;
console.log(emailValidation.exec("user@example.com") != null); // Result: "true"
console.log(emailValidation.exec("www.example.com") != null); // Result: "false"

The only exception to this should probably be for a system that needs to validate email addresses for registration into an email system itself. I.E., for registration of an address into a company's webmail system, and it is well-known that this particular system doesn't accept special characters in the email address, for example. Otherwise, why should a given site be coded to reject what otherwise would be a perfectly valid and legitimate email address, just because the local system's requirements don't "like" certain characters or formats? (Hint: If this is the case, the local system's requirements are probably due for review.)

Real-world issues

One of my reminders of this issue, and part of this inspiration of this post, comes from Liferay Portal. In older versions of Liferay, this was the regular expression (implemented within Java code) for validating an email address:

"([\\w-]+\\.)*[\\w-]+@([\\w-]+\\.)+[A-Za-z]+

This incorrectly caused email addresses containing a '+' sign to be rejected - a practice somewhat common with Gmail, for example. As an enterprise Liferay customer, I requested a patch, which replaced the above with the following regular expression to provide support for the '+' sign (and some other things):

([\\w!#%&-/=_`~\\Q.$*+?^{|}\\E]+)*@([\\w-]+\\.)+[A-Za-z]+

Unfortunately, this patch only led to worse issues. At least with the newer version of the pattern, it is a non-optimized regular expression and is subject to catastrophic backtracking. (Please refer to http://www.regular-expressions.info/catastrophic.html and/or http://www.codinghorror.com/blog/2006/01/regex-performance.html for additional details regarding catastrophic backtracking.) In particular, in these situations, each additional character that needs to be backtracked over will cause an exponential growth in required CPU iterations. This expression seems to "fall apart" starting at about 25 characters. This caused us to run into periodic instances (both production and non-production) where our Liferay JVM would completely hang, causing 100% CPU utilization. I captured a stack trace that a provided to them - but the JVM was basically getting stuck on the following call: com.liferay.portal.kernel.util.Validator.isEmailAddress(String).

I proceeded to escalate this as a critical security issue, due to the potential of an easy-to-execute denial-of-service (DOS) attack for anyone running versions of Liferay using this patch or newer versions containing this code.

A minimal example that can be used to reproduce the issue outside of Liferay - but using the same later regular expression shown above:

 Pattern emailAddressPattern = Pattern.compile("([\\w!#%&-/=_`~\\Q.$*+?^{|}\\E]+)*@([\\w-]+\\.)+[A-Za-z]+");
 // "Accidentally" type an email address in with the wrong punctuation.
 Matcher m = emailAddressPattern.matcher("mark.ziesemer.myexamplecompany@com");
 System.out.println(m.matches());

I'm not even exactly sure what this new regular expression was aiming to accomplish, but quite honestly, I believe Liferay went the wrong direction / took the wrong approach with this modification. I also provided them with what I've provided above, including the minimal regular expression. My advice was not immediately accepted. The latest patch uses the following pattern:

[\w!#$%&'*+/=?^_`{|}~-]+(?:\.[\w!#$%&'*+/=?^_`{|}~-]+)*@(?:[\w](?:[\w-]*[\w])?\.)+[\w](?:[\w-]*[\w])?

... which is almost as bad as Phil Haack's example (above) - actually, probably worse, since I doubt it was tested to the same standards as his (if Liferay even has unit tests around this at all...) To their credit, they did file a feature request around this (LPS-30849) - though it doesn't include any of the rationale for its existence, nor has any effort been demonstrated on the ticket yet.

Improving code and quality with Checkstyle

I've always been picky about the quality my code and the code that I work with. For good reason. It makes the code easier to read and work with. Consistency makes the program flow easier to understand, bugs and other potential issues easier to spot, and difference comparisons between files and versions more effective. Sun Microsystems (original designer of the Java platform, now part of Oracle) seems to agree. From their "Code Conventions for the Java Programming Language":

This Code Conventions for the Java Programming Language document contains the standard conventions that we at Sun follow and recommend that others follow. It covers filenames, file organization, indentation, comments, declarations, statements, white space, naming conventions, programming practices and includes a code example.

• 80% of the lifetime cost of a piece of software goes to maintenance.
• Hardly any software is maintained for its whole life by the original author.
• Code conventions improve the readability of the software, allowing engineers to understand new code more quickly and thoroughly.

For years, I haven't really used any special tools to assist with code formatting and standards, other than the Eclipse Java Formatter for automatically and completely reformatting code as needed, and Eclipse's Compiler Errors/Warnings for pointing out potential issues. One limitation with the compiler errors/warnings in Eclipse is that they really don't include any formatting-specific checks.

I've had experience with Checkstyle in the past (official site | Wikipedia). However, I'm not sure how much thought was put into the configuration of the standards that were in place, and combined with a high level of false positives, most of the developers found ourselves working against the tool rather than working with it. Standards are great, but they need to be thought through, and have buy-in from the project team.

Over the past few months, I rediscovered a need for Checkstyle for the group of developers I was working with. We were doing an satisfactory job addressing the compiler errors and warnings, but the code itself was a sore sight to look at. The code formatting was addressed during code reviews, but without having the formatting issues also appear as warnings in the Eclipse "Problems" view as it was being developed, the formatting issues were quickly "out of sight, out of mind", and quickly forgotten by much of the team. (I'm not sure how the group would have managed with a language like Python, where proper indentation of the code is actually part of the language syntax.)

Fortunately, since I last worked with Checkstyle, the tools have only improved. eclipse-cs now integrates seamlessly with both Eclipse and the Checkstyle XML configurations, and is fully configurable to projects on both local and global levels - using either internal, external, remote, or project-relative configurations.

For my own reference, as well as others to benefit from, I'm sharing the Checkstyle configuration I've worked with over the past few months. The surprising thing is how many inconsistencies were reported and that I was able to fix even in my own code after running the Checkstyle validations.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE module PUBLIC "-//Puppy Crawl//DTD Check Configuration 1.3//EN" "http://www.puppycrawl.com/dtds/configuration_1_3.dtd">

<!--
    This configuration file was written by the eclipse-cs plugin configuration editor
-->
<!--
    Checkstyle-Configuration: MAZ
    Description: none
-->
<module name="Checker">
  <property name="severity" value="warning"/>
  <module name="TreeWalker">
    <property name="tabWidth" value="2"/>
    <module name="JavadocMethod">
      <property name="severity" value="ignore"/>
      <property name="suppressLoadErrors" value="true"/>
      <metadata name="net.sf.eclipsecs.core.lastEnabledSeverity" value="inherit"/>
    </module>
    <module name="JavadocType">
      <property name="severity" value="ignore"/>
      <metadata name="net.sf.eclipsecs.core.lastEnabledSeverity" value="inherit"/>
    </module>
    <module name="JavadocVariable">
      <property name="severity" value="ignore"/>
      <metadata name="net.sf.eclipsecs.core.lastEnabledSeverity" value="inherit"/>
    </module>
    <module name="JavadocStyle">
      <property name="checkFirstSentence" value="false"/>
    </module>
    <module name="ConstantName"/>
    <module name="LocalFinalVariableName"/>
    <module name="LocalVariableName"/>
    <module name="MemberName"/>
    <module name="MethodName"/>
    <module name="PackageName"/>
    <module name="ParameterName"/>
    <module name="StaticVariableName">
      <property name="severity" value="ignore"/>
      <metadata name="net.sf.eclipsecs.core.lastEnabledSeverity" value="inherit"/>
    </module>
    <module name="TypeName"/>
    <module name="AvoidStarImport"/>
    <module name="IllegalImport"/>
    <module name="RedundantImport"/>
    <module name="UnusedImports"/>
    <module name="LineLength">
      <property name="severity" value="ignore"/>
      <metadata name="net.sf.eclipsecs.core.lastEnabledSeverity" value="inherit"/>
    </module>
    <module name="MethodLength">
      <property name="severity" value="ignore"/>
      <metadata name="net.sf.eclipsecs.core.lastEnabledSeverity" value="inherit"/>
    </module>
    <module name="ParameterNumber">
      <property name="severity" value="ignore"/>
      <metadata name="net.sf.eclipsecs.core.lastEnabledSeverity" value="inherit"/>
    </module>
    <module name="EmptyForIteratorPad"/>
    <module name="MethodParamPad"/>
    <module name="NoWhitespaceAfter">
      <property name="tokens" value="BNOT,DEC,DOT,INC,LNOT,UNARY_MINUS,UNARY_PLUS"/>
    </module>
    <module name="NoWhitespaceBefore"/>
    <module name="OperatorWrap"/>
    <module name="ParenPad"/>
    <module name="TypecastParenPad"/>
    <module name="WhitespaceAfter">
      <property name="tokens" value="COMMA,SEMI"/>
    </module>
    <module name="WhitespaceAround">
      <property name="tokens" value="BAND,BAND_ASSIGN,BOR,BOR_ASSIGN,BSR,BSR_ASSIGN,BXOR,BXOR_ASSIGN,COLON,DIV,DIV_ASSIGN,EQUAL,GE,GT,LAND,LE,LOR,LT,MINUS,MINUS_ASSIGN,MOD,MOD_ASSIGN,NOT_EQUAL,PLUS,PLUS_ASSIGN,QUESTION,SL,SL_ASSIGN,SR,SR_ASSIGN,STAR,STAR_ASSIGN,TYPE_EXTENSION_AND"/>
    </module>
    <module name="ModifierOrder"/>
    <module name="RedundantModifier"/>
    <module name="AvoidNestedBlocks">
      <property name="severity" value="ignore"/>
      <metadata name="net.sf.eclipsecs.core.lastEnabledSeverity" value="inherit"/>
    </module>
    <module name="EmptyBlock">
      <property name="option" value="text"/>
      <property name="tokens" value="LITERAL_DO,LITERAL_ELSE,LITERAL_FINALLY,LITERAL_IF,LITERAL_FOR,LITERAL_TRY,LITERAL_WHILE,STATIC_INIT"/>
    </module>
    <module name="LeftCurly"/>
    <module name="NeedBraces"/>
    <module name="RightCurly"/>
    <module name="AvoidInlineConditionals">
      <property name="severity" value="ignore"/>
      <metadata name="net.sf.eclipsecs.core.lastEnabledSeverity" value="inherit"/>
    </module>
    <module name="DoubleCheckedLocking"/>
    <module name="EmptyStatement"/>
    <module name="EqualsHashCode"/>
    <module name="HiddenField">
      <property name="severity" value="ignore"/>
      <metadata name="net.sf.eclipsecs.core.lastEnabledSeverity" value="inherit"/>
    </module>
    <module name="IllegalInstantiation"/>
    <module name="InnerAssignment">
      <property name="severity" value="ignore"/>
      <metadata name="net.sf.eclipsecs.core.lastEnabledSeverity" value="inherit"/>
    </module>
    <module name="MagicNumber">
      <property name="severity" value="ignore"/>
      <metadata name="net.sf.eclipsecs.core.lastEnabledSeverity" value="inherit"/>
    </module>
    <module name="MissingSwitchDefault">
      <property name="severity" value="ignore"/>
      <metadata name="net.sf.eclipsecs.core.lastEnabledSeverity" value="inherit"/>
    </module>
    <module name="RedundantThrows">
      <property name="suppressLoadErrors" value="true"/>
    </module>
    <module name="SimplifyBooleanExpression"/>
    <module name="SimplifyBooleanReturn"/>
    <module name="DesignForExtension">
      <property name="severity" value="ignore"/>
      <metadata name="net.sf.eclipsecs.core.lastEnabledSeverity" value="inherit"/>
    </module>
    <module name="FinalClass"/>
    <module name="HideUtilityClassConstructor">
      <metadata name="net.sf.eclipsecs.core.comment" value="MAZ: Disabled, until this module provides an option to ignore abstract classes."/>
      <property name="severity" value="ignore"/>
      <metadata name="net.sf.eclipsecs.core.lastEnabledSeverity" value="inherit"/>
    </module>
    <module name="InterfaceIsType"/>
    <module name="VisibilityModifier">
      <property name="severity" value="ignore"/>
      <metadata name="net.sf.eclipsecs.core.lastEnabledSeverity" value="inherit"/>
    </module>
    <module name="ArrayTypeStyle"/>
    <module name="FinalParameters">
      <property name="severity" value="ignore"/>
      <metadata name="net.sf.eclipsecs.core.lastEnabledSeverity" value="inherit"/>
    </module>
    <module name="TodoComment">
      <property name="severity" value="ignore"/>
      <metadata name="net.sf.eclipsecs.core.lastEnabledSeverity" value="inherit"/>
    </module>
    <module name="UpperEll"/>
    <module name="RegexpSinglelineJava">
      <metadata name="net.sf.eclipsecs.core.comment" value="MAZ"/>
      <property name="format" value="^\t* +\t*\S"/>
      <property name="message" value="Line has leading space characters; indentation should be performed with tabs only."/>
      <property name="ignoreComments" value="true"/>
    </module>
    <module name="RegexpSinglelineJava">
      <metadata name="net.sf.eclipsecs.core.comment" value="MAZ"/>
      <property name="format" value="^[^&quot;]*(catch|for|if|switch|synchronized|while)\s+\("/>
      <property name="message" value="No whitespace after block keyword."/>
      <property name="ignoreComments" value="true"/>
    </module>
    <module name="RegexpSinglelineJava">
      <metadata name="net.sf.eclipsecs.core.comment" value="MAZ"/>
      <property name="format" value="^\s*(&quot;([^&quot;])*(?&lt;!\\)&quot;|[^&quot;\t])+\)\s+\{"/>
      <property name="message" value="No whitespace before '{'."/>
      <property name="ignoreComments" value="true"/>
    </module>
    <module name="RegexpSinglelineJava">
      <metadata name="net.sf.eclipsecs.core.comment" value="MAZ"/>
      <property name="format" value="^\s*(&quot;([^&quot;])*(?&lt;!\\)&quot;|[^&quot;\t])+(\t|  )"/>
      <property name="message" value="No tabs or multiple whitespace beyond indentation."/>
      <property name="ignoreComments" value="true"/>
    </module>
    <module name="AnnotationUseStyle">
      <metadata name="net.sf.eclipsecs.core.comment" value="javac currently does not accept a trailing comma (http://bugs.sun.com/view_bug.do?bug_id=6337964)."/>
      <property name="severity" value="error"/>
      <property name="elementStyle" value="ignore"/>
      <property name="closingParens" value="ignore"/>
    </module>
    <module name="MissingDeprecated"/>
    <module name="MissingOverride"/>
  </module>
  <module name="JavadocPackage">
    <property name="severity" value="ignore"/>
    <metadata name="net.sf.eclipsecs.core.lastEnabledSeverity" value="inherit"/>
  </module>
  <module name="NewlineAtEndOfFile">
    <property name="lineSeparator" value="lf"/>
  </module>
  <module name="Translation"/>
  <module name="FileLength">
    <property name="severity" value="ignore"/>
    <metadata name="net.sf.eclipsecs.core.lastEnabledSeverity" value="inherit"/>
  </module>
  <module name="FileTabCharacter">
    <property name="severity" value="ignore"/>
    <metadata name="net.sf.eclipsecs.core.lastEnabledSeverity" value="inherit"/>
  </module>
  <module name="RegexpSingleline">
    <metadata name="net.sf.eclipsecs.core.comment" value="MAZ"/>
    <property name="format" value="[^\s]+\s+$"/>
    <property name="message" value="Line has trailing whitespace."/>
  </module>
</module>

This configuration is based off of the "Sun Checks" configuration, with a few enhancements, and tweaks for personal preference.

One of the noteworthy settings is "trailingArrayComma" setting under Annotations/Annotation Use Style. Per the Java Language Specification (JLS), a trailing comma in an annotation array should be allowed / ignored. The Eclipse compiler respects this, but attempting to use javac (e.g. as part of a Maven build) with a trailing comma results in a "illegal start of expression" failure, due to an acknowledged bug with javac. Using Checkstyle here helps to ensure consistent, stable builds across compilers.

Additionally, I extended the configuration to perform limited checks on non-Java files as well. First, there is a "exclude from checking" configuration section that includes a "all files types except:" rule that is enabled by default, and set to "java, properties". This filter needs to be expanded to include any file types that should even be considered for processing by Checkstyle. For basic web development, I added "htm, html, js, css, jsp". Next, disable the "Use simple configuration" option, which enables "Advanced - configure file sets to control which files get checked by which configuration". I then configured 3 file sets:

• Java - Uses the configuration shown above. Includes regular expression matches for "\.java$" and "\.properties$".
• Web - Provides basic whitespace rules for non-Java files. Includes regular expression matches for each of the "web development" types mentioned above, or "\.(html?|js|jsp|css)$" as a version to accomplish it with one expression.
• JSP - Provides additional checks specific to the scriptlet tags used in JSP files. Includes a regular expression match for "\.jsp$". Note that the above Web file set is configured to also apply to JSP files.

In each of these configurations are a number of additional regular expression checks, that primarily focus on whitespace rules that aren't currently checked for or configurable by the other Checkstyle rules:

File Set	Module	Pattern	Message
Java	RegexpSingleLineJava	^\t* +\t*\S	Line has leading space characters; indentation should be performed with tabs only.
Java	RegexpSingleLineJava	^\s*("([^"])*(?<!\\)"|[^"\t])+(\t|  )	No tabs or multiple whitespace beyond indentation.
Java	RegexpSingleLine	[^\s]+\s+$	Line has trailing whitespace.
Java	RegexpSingleLineJava	^\s*("([^"])*(?<!\\)"|[^"\t])+\)\s+\{	No whitespace before '{'.
Java	RegexpSingleLineJava	^\s*}\s+	No whitespace after '}'.
Java	RegexpSingleLineJava	^[^"]*(catch|for|if|switch|synchronized|while)\s+\(	No whitespace after block keyword.
Web	RegexpSingleLine	^\t* +(?!\*)+\t*\S	Line has leading space characters; indentation should be performed with tabs only.
Web	RegexpSingleLine	^\s*(?!\*)([^\s"]|"[^"]*(?<!\\)")+(\t|  )	No tabs or multiple whitespace beyond indentation.
Web	RegexpSingleLine	[^\s]+\s+$	Line has trailing whitespace.
JSP	RegexpSingleLine	(?<=.+)(?<!%>)<%(?!=|.+;%>)	Scriptlet begin tag must be at beginning of line.
JSP	RegexpSingleLine	<%[^@=](?!;%>)+$	Content after scriptlet begin tag must be on new line.

These rules are not all infallible. They are designed to catch the majority of the targeted issues, but without causing false positives in the process. (For example, ignoring spacing within multi-line comments in JSP files - as Checkstyle's Java interpretation doesn't work in JSP files, and forbidden sequences that are allowed within String constants are a bit tricky to work with.) In order to prevent regressions as the rules are expanded upon, all of these regular expressions are included in a local suite of JUnit tests that I plan to make available at a future date. There are currently 22 tests in 3 test classes (one per file set), executing 33 base comparisons that result in a total of 993 assertions. Recently, some issues with the JSP rules were observed, causing Checkstyle to fail on "minified" JavaScript files that consisted of single lines of 10,000+ characters. Recursion within Java's regular expression support resulted in stack overflow exceptions, until I tweaked the JSP expressions to make use of different zero-width lookahead and lookbehind constructs. A test case on 10,000+ characters in a single line still takes several seconds to execute, but at least it consistently completes without exception now.

After getting the source code and formatting in order (and to not drown developers in too many warnings at once), additional tools can be considered for use, e.g. FindBugs (official site | Wikipedia). While I have nothing against FindBugs, I just haven't seen as urgent of a need for it. Most of the bugs I've seen are either due to things that are already covered by properly-configured errors and warnings in the Eclipse compiler, or are functional type issues (not properly following business requirements) or other higher-level issues that not even FindBugs can do a satisfactory job with detecting - but are things that are better caught during a proper code review process and comprehensive unit testing. Additionally, FindBugs only searches against the compiled Java bytecode, so it can't help report the source code level issues, like Checkstyle.

YUI "website top nav" Menu from JavaScript only

I recently had the opportunity to observe someone running into some difficulties trying to implement a YUI 2 Menu. The menu items were to be retrieved from a database, and needed to be dynamically updatable through an AJAX call - so using JSON rather than HTML to build the menu made sense in this case. According to YUI's menu page, "Menus can be created from simple, semantic markup on the page or purely through JavaScript".

In this case, getting the menu built "purely through JavaScript" was looking to be a little bit of a challenge. The "Website Top Nav With Submenus" was the type of menu desired. YUI's available menu examples included this type of menu built both "From Markup" and "From JavaScript". However, the "From JavaScript" version still built the top-level menu with HTML rather than JavaScript. While certainly possible, dynamically generating both HTML and JavaScript from the server-side certainly didn't seem ideal.

Recognizing that YUI is a very robust JavaScript library, I knew that there had to be a way to build the menu with only JavaScript. After starting with a new test script, I quickly found the 2 main parts to the solution:

1. The MenuBar constructor accepts an "itemData" property as part of its 2nd "config" argument. The menu structure can be configured here for the top level, in a fashion identical to the "aSubmenuData" array of objects visible in the original YUI example.

2. All menu items - including the top-level menus, submenus, and the menu bar itself - must have a unique ID for the HTML DOM, otherwise the item will not activate. For example, if a top-level menu is missing an ID, it will display, but not activate to display the submenus.

   Ideally, it'd be nice to have the "id" property optional. However, the "YAHOO.util.Dom.generateId()" function works well for this - and at least for the purposes of this example, I aliased it as simply "id".

The complete updated example is shown below. For lack of better data, the same example data from the original YUI example is used:

JavaScript source code:

  YAHOO.util.Event.onDOMReady(function(){
    
    var id = YAHOO.util.Dom.generateId;
    
    var menuData = [
      {
        text: "Communication",
        submenu: {id: id(), itemdata: [ 
          {text: "360", url: "http://360.yahoo.com"},
          {text: "Alerts", url: "http://alerts.yahoo.com"},
          {text: "Avatars", url: "http://avatars.yahoo.com"},
          {text: "Groups", url: "http://groups.yahoo.com"},
          {text: "Internet Access", url: "http://promo.yahoo.com/broadband"},
          {
            text: "PIM", 
            submenu: { 
              id: id(), 
              itemdata: [
                {text: "Yahoo! Mail", url: "http://mail.yahoo.com"},
                {text: "Yahoo! Address Book", url: "http://addressbook.yahoo.com"},
                {text: "Yahoo! Calendar",  url: "http://calendar.yahoo.com"},
                {text: "Yahoo! Notepad", url: "http://notepad.yahoo.com"}
              ] 
            }
          }, 
          {text: "Member Directory", url: "http://members.yahoo.com"},
          {text: "Messenger", url: "http://messenger.yahoo.com"},
          {text: "Mobile", url: "http://mobile.yahoo.com"},
          {text: "Flickr Photo Sharing", url: "http://www.flickr.com"},
        ]}
      },
      {
        text: "Shopping",
        submenu: {id: id(), itemdata: [
          {text: "Auctions", url: "http://auctions.shopping.yahoo.com"},
          {text: "Autos", url: "http://autos.yahoo.com"},
          {text: "Classifieds", url: "http://classifieds.yahoo.com"},
          {text: "Flowers & Gifts", url: "http://shopping.yahoo.com/b:Flowers%20%26%20Gifts:20146735"},
          {text: "Real Estate", url: "http://realestate.yahoo.com"},
          {text: "Travel", url: "http://travel.yahoo.com"},
          {text: "Wallet", url: "http://wallet.yahoo.com"},
          {text: "Yellow Pages", url: "http://yp.yahoo.com"}
        ]}
      },
      {
        text: "Entertainment",
        submenu: {id: id(), itemdata: [
          {text: "Fantasy Sports", url: "http://fantasysports.yahoo.com"},
          {text: "Games", url: "http://games.yahoo.com"},
          {text: "Kids", url: "http://www.yahooligans.com"},
          {text: "Music", url: "http://music.yahoo.com"},
          {text: "Movies", url: "http://movies.yahoo.com"},
          {text: "Radio", url: "http://music.yahoo.com/launchcast"},
          {text: "Travel", url: "http://travel.yahoo.com"},
          {text: "TV", url: "http://tv.yahoo.com"}
        ]}
      },
      {
        text: "Information",
        submenu: {id: id(), itemdata: [
          {text: "Downloads", url: "http://downloads.yahoo.com"},
          {text: "Finance", url: "http://finance.yahoo.com"},
          {text: "Health", url: "http://health.yahoo.com"},
          {text: "Local", url: "http://local.yahoo.com"},
          {text: "Maps & Directions", url: "http://maps.yahoo.com"},
          {text: "My Yahoo!", url: "http://my.yahoo.com"},
          {text: "News", url: "http://news.yahoo.com"},
          {text: "Search", url: "http://search.yahoo.com"},
          {text: "Small Business", url: "http://smallbusiness.yahoo.com"},
          {text: "Weather", url: "http://weather.yahoo.com"}
        ]}
      }
    ];
    
    new YAHOO.util.YUILoader({require: ["menu"], onSuccess: function(){
      var oMenuBar = new YAHOO.widget.MenuBar(id(), {
        autosubmenudisplay: true, 
        hidedelay: 750, 
        itemdata: menuData});
      
      oMenuBar.render("com.ziesemer.demos.yuiMenuJsOnly.global_menu_parent");
    }}).insert();
  });

HTML source code:

  <div class="yui-skin-sam">
    <div id="com.ziesemer.demos.yuiMenuJsOnly.global_menu_parent" class="yuimenubarnav"></div>
  </div>

Output:

Updated Blogger Tools

I've updated the "Blog Archive" and "Labels" gadgets on this site - as visible in the right-hand margin of this page. Please leave a comment with any issues or suggestions.

Over a year ago, I had already replaced Blogger's default Labels gadget with the Yahoo! UI Library (YUI)'s TreeView component, which provided:

• A view that is collapsed by default, saving screen space for other features unless the Labels are clicked and expanded for use.
• A complete and compact list of tagged posts under each label, without having to request a new label search page - while still providing these links.

This tree is populated with an AJAX request to Google's GData Blogger API, using the JSON Alt Type and YUI's Get Utility. Since the returned data is a list of posts, a limited amount of JavaScript is needed to group and sort the posts into the labels. As part of this last update, a label will now again automatically expand to show the related posts if the current page being viewed is a label search page. (This is similar to the default functionality, but is determined by the current URL.)

Also now just completed, I've also replaced Blogger's default Blog Archive gadget with another YUI TreeView, providing:

• A consistent look-and-feel between the Blog Archive and Labels features.
• A more compact display, while maintaining readability.
• Post titles were previously truncated after 50 characters. Complete post titles are now displayed, along with the specific date posted.
• As the same data retrieved to populate the Labels is re-used, the page loading performance should actually be improved as the HTML is now dynamically generated client-side - instead of downloading what was essentially duplicate information as additional HTML from the server.

As with the Labels, the archive will also automatically expand to show the related posts for the time period being viewed - similar to the default functionality, but again determined by the current URL.

Also just added are "loading bar" graphics for the short period of time between the initial page load and the population of the trees.

A few other quick things to note:

• The source JavaScript code used for these features is available for reference in this page's HTML source (among the rest of Blogger's default and somewhat cluttered code), using your browser's View/Source feature. Even though it is only ~250 lines or ~7 KB of code, this is not the most efficient as this code is reloaded with every page load. I'd like to be able to properly move this code into a separate and cacheable *.js file - but can't currently find a usable solution that matches the reliability and pricing (free) of the rest of Blogger. I attempted using versions stored on Google Code and Google Sites, but each had various issues - including being able to properly make updates, as well as having download landing pages interfere - partially complicated by not being hosted on the blogspot.com domain. Amazon's S3 storage service looks like a promising option that I may still consider. However, while it would likely only cost pennies / month to use for this, it's another bill to take care of.

• The JSON data feed received is a bit large - currently ~200 KB - but unlike the JavaScript code, this data is returned with proper HTTP headers by Google, and should be cacheable by the browser. Much of this data is information that is not necessary or used, such as post summaries and comments. Hopefully, these extra fields will soon be selectively disabled - as soon as the Blogger GData API supports partial responses.

• As visible in the source code, I also needed to implement a custom "clickEvent" handler for YUI's TreeView - otherwise the HTML links contained within the tree nodes were not clickable, as any clicks would instead expand/collapse the node instead of activating the link.

A few ideas for future enhancements:

• Options to alternatively sort the labels, including by post count.
• Having the displayed labels under each post interact with the Labels tree.

JavaScript conditional oversights and solutions

While this is nothing new or particularly noteworthy, I felt this post would be good to share after fixing a number of conditionals in various JavaScript code - as well as having received some interesting and surprising feedback after suggesting the same fixes to others.

The code in question looked something like this:

var x;
// …
if(x == ""){
  // …
}

While the exact validity of this code depends on the exact requirements being implemented, there are a few potential issues in what is otherwise a very simple block of code:

• null is not checked for, which is often desirable as - while accepted as a best programming practice or not - null and an empty string ("") are usually treated (or expected to be treated) the same.
• Looking from the other side of things, the if block will also be entered if x is any of the following (not guaranteed to be a complete list): 0 (a number), "0" (a string representation of the number), or false (a boolean).

Stated another way, when using the standard equals operator (==) in JavaScript, all of ["", 0, "0", false] are considered equal. The same applies with a separate set of [null, (undefined)], though no elements between sets are considered equal.

Verify the results for yourself, by running this code in your browser:

log(false == false);
log("" == false);
log(0 == false);
log("0" == false);

var x = {};
// x.y is undefined:
log(x.y == null);
// Another way is to obtain an undefined result is to use the void operator:
log(void(0) == null);

// Show that items between these 2 groups are not considered equal:
log(null != false);

My recommendation to all JavaScript developers is to instead use one of the following approaches:

• To really test against an empty string (""), and only an empty string, use the strict equals operator (===). For example:

  var x;
  // …
  if(x === ""){
    // …
  }

  The same also applies for testing against 0, "0", or false.
• From the other side of things, don't use an equals operator at all. Just use the variable in a conditional without an explicit operator or comparison, which is equivalent to wrapping with the Boolean(…) function. To test for a defined, non-null, and non-"false" value, the following 4 statements are all functionally identical, with the first and highlighted version being the preferred version:

  var x;
  // …
  if(x){
    // …
  }
  if(Boolean(x)){ /*…*/ }
  if(x != null && x == true){ /*…*/ }
  if(x !== null && typeof(x) != "undefined" && x !== "" && x !== 0 && x !== "0" && x !== false){ /*…*/ }

  Or for the reverse, by using the negated form:

  if(!x){ /*…*/ }
  if(!Boolean(x)){ /*…*/ }
  if(x == null || x == false || isNaN(x)){ /*…*/ }
  if(x === null || typeof(x) == "undefined" || x === "" || x === 0 || x === "0" || x === false || isNaN(x)){ /*…*/ }

  Assuming one of these code blocks meet the functional requirement, the first line forms are the preferred usage.

Executing JavaScript from HTML AJAX

AJAX, by definition, is known as "asynchronous JavaScript and XML". However, the use of XMLHttpRequest is not limited to XML, with JSON being a popular alternative. This post will focus on another alternative: standard HTML. Using HTML may be necessary when the HTML is already being generated server-side, with no available XML or JSON equivalent. If the response is to be viewable by the user, HTML may already be the desired result, and can avoid additional complexities of transforming XML or JSON into HTML.

Simply retrieving HTML from an AJAX request and inserting into the existing DOM of a web page is relatively easy. For my purpose, I used the .responseText attribute of the response on the XMLHttpRequest. Once the HTML string is obtained, it can be injected into an existing DOM node in the document by assigning it to the node's .innerHTML property.

Another possibility to consider is if the returned HTML is guaranteed to be valid XHTML. In this case, it may be possible to continue treating the response as XML and using the .responseXML attribute, and inject it into an existing DOM node in the document by calling the node's .importNode(…) method.

In either case, there is the complication of scripts that may be embedded within the returned HTML. These embedded scripts are not reliably executed, especially not consistently across web browsers. This is particularly complicated by Internet Explorer's handling of the <script/> tags as "NoScope" elements, which is detailed by Microsoft's John Sudds in MSDN's .innerHTML property documentation and referenced forum thread. In summary, embedded scripts should have the "defer" property set to true, and must follow a scoped element (e.g. <input type="hidden"/>) to work with IE. I have seen several pages that suggest setting the .innerHTML on a cloned node, including the above provisions for IE, then inserting back into the HTML DOM using methods such as .appendChild(…) or .replaceChild(…) to work across browsers. However, even this is not guaranteed to always work, and fails with Safari in particular.

Current Working Solution:

Rather than using a combination of browser "hacks" and/or browser detection, this solution instead recognizes and assumes that embedded scripts will not be executed automatically by the browser. After inclusion into the document, any embedded scripts are efficiently searched for and executed through JavaScript's eval(…) method. Shown below is an example / test case:

JavaScript source code:

 // http://blogger.ziesemer.com/2008/05/javascript-namespace-function.html
 namespace("com.ziesemer.demos").htmlAjax = function(){
    var pub = {};
    var contentDiv, oldContent;
    
    var html = "<div>New 1st-level dynamic text, presumably from AJAX response.</div>"
      + "<div id=\"com.ziesemer.demos.htmlAjax.middle\">More 1st-level dynamic text that should be replaced.</div>"
      + "<div>More 1st-level dynamic text, presumably from an AJAX response.</div>"
      + "<script type=\"text/javascript\">"
      + "document.getElementById(\"com.ziesemer.demos.htmlAjax.middle\").innerHTML = "
      + "\"2nd-level dynamic text, controlled by JavaScript presumably from an AJAX response.\";"
      // http://www.wwco.com/~wls/blog/2007/04/25/using-script-in-a-javascript-literal/
      + "<" + "/script>";
    
    pub.run = function(){
      contentDiv = document.getElementById("com.ziesemer.demos.htmlAjax.output");
      oldContent = contentDiv.innerHTML;
      contentDiv.innerHTML = html;
      
      var scripts = contentDiv.getElementsByTagName("script");
      // .text is necessary for IE.
      for(var i=0; i < scripts.length; i++){
        eval(scripts[i].innerHTML || scripts[i].text);
      }
    };
    
    pub.reset = function(){
      if(contentDiv && oldContent){
        contentDiv.innerHTML = oldContent;
      }
    };
    
    return pub;
  }();

HTML source code: (equivalent to using this page's View/Source)

<div style="border:1px solid; padding:0.5em;">
  <div><b>Dynamic test area:</b></div>
  <div id="com.ziesemer.demos.htmlAjax.output" style="border:1px solid;">
    Original static HTML text that should be replaced.
  </div>
  <p>
    <input type="button" value="Go!" onclick="com.ziesemer.demos.htmlAjax.run();"/>
    <input type="button" value="Reset" onclick="com.ziesemer.demos.htmlAjax.reset();"/>
  </p>
  
  <div>
    <b>Expected result:</b> (of what the above should appear as after the &quot;Go!&quot; button is clicked)
  </div>
  <div style="border:1px solid;">
    <div>New 1st-level dynamic text, presumably from AJAX response.</div>
    <div>2nd-level dynamic text, controlled by JavaScript presumably from an AJAX response.</div>
    <div>More 1st-level dynamic text, presumably from an AJAX response.</div>
  </div>
</div>

Demo:

This was tested on Mozilla Firefox 3.5; Internet Explorer 6, 7, and 8; Google Chrome 3; and Apple Safari 4 for Windows. If both blocks of above text don't match after clicking "Go!", you've probably found an issue.

A few things to note:

• This does not work with externally sourced scripts, i.e., those with a "src" attribute. This should normally not be a concern, as there shouldn't be any reason why these types of scripts can't be included in the page before the AJAX call is made. However, if supporting this is necessary, YUI's Get Utility will probably prove useful.
• As with most similar approaches, this does not work with scripts that utilize document.write(…). If document.write is used in this method, it will likely result in all existing content of the page being removed. Again, this should normally not be a concern, as use of document.write should generally be discouraged and avoided for a number of reasons. The suggested alternative is to use existing DOM elements as placeholders, as shown in my example above.
• Note the use of the "broken" closing </script> tag. This would not be necessary if the HTML string was actually obtained externally, e.g. from an AJAX call. It is only necessary as it is contained within a JavaScript string literal (as in the above example), as excellently described by Walt Stoneburner in his blog at http://www.wwco.com/~wls/blog/2007/04/25/using-script-in-a-javascript-literal/ (2007-04-25).
• Not really specific to this example, but the for loop around the returned "scripts" array would ideally be replaced by Array.forEach(…), e.g.:

  contentDiv.getElementsByTagName("script").forEach(function(script){
    // .text is necessary for IE.
    eval(script.innerHTML || script.text);
  });

  However, .forEach isn't supported until JavaScript 1.6. Firefox 3.5 supports JS 1.8.1. Chrome 1.0 and Safari 3.2 support JS 1.7. Even version 8 of IE still only supports JS 1.5!

Additional references:

• "Help me do the right DOM thing.". Martin Honnen, et al., 2006-11-02, comp.lang.javascript.
• "Execute JavaScript injected using innerHTML attribute, even with Safari", 2007-06-22.
• "Javascript script execution in innerHTML: the revenge". Giu­liani Vito, Ivan, 2006-03-07.

Scripted hiding of Windows Updates under Vista

Similar to my last post, here is another UI issue with Windows Vista. Fortunately, this time I have a solution to offer.

Starting with Windows Vista, the "Windows Update" functionality is provided through Control Panel rather than Internet Explorer. In both versions, there is the ability to hide updates. While hidden updates can easily be restored, this feature allows for ignoring unnecessary updates so that they don't continually count towards the number of available updates that are displayed. For me, this includes the 34 "Windows Vista Ultimate Language Packs" that are currently available. Unfortunately, multiple-selection is not enabled in the "View available updates" dialog. There are checkboxes, including a checkbox on the header that can be used to select/unselect all shown updates, but the checkbox selections are used for the "Install" button only. The other options available from the context menu - "View details", "Copy details", and "Hide update" - can currently be applied only one-at-a-time. This means that hiding just the 34 language packs would require no fewer than 68 clicks!

Originally, I assumed that these hidden updates and other preferences would be stored in the Windows registry, or possibly in a file on the file system. They are in a file, but a database-type file that isn't directly editable: %SystemRoot%\SoftwareDistribution\DataStore\DataStore.edb. Fortunately, there is a comprehensive Windows API for viewing and editing this information, and it is even easily available to scripting through the Windows Scripting Host and languages such as JScript. Microsoft's reference is located on MSDN: Windows Update Agent API.

Here is my resulting script that automatically hides all the "Windows Vista Ultimate Language Packs":

var updateSession = WScript.CreateObject("Microsoft.Update.Session");
var updateSearcher = updateSession.CreateUpdateSearcher();
updateSearcher.Online = false;

var searchResult = updateSearcher.Search("CategoryIDs Contains 'a901c1bd-989c-45c6-8da0-8dde8dbb69e0' And IsInstalled=0");

for(var i=0; i<searchResult.Updates.Count; i++){
  var update = searchResult.Updates.Item(i);
  WScript.echo("Hiding update: " + update.Title);
  update.IsHidden = true;
}

If you're not familiar with WSH, this can be simply executed as saving it as a *.js file, then double-clicking. A better option is to execute the file from a command-line with cscript. This will cause the output messages to be written to the standard output, instead of popping up a message box that must be acknowledged for each message. Also, since this script is making administrative changes to the system, it must be executed as an administrator.

"a901c1bd-989c-45c6-8da0-8dde8dbb69e0" is the ICategory.CategoryID for "Windows Vista Ultimate Language Packs". (This ICategory happens to have a .Type of "Product".) A similar script can easily be used to perform operations on other sets of updates by simply modifying the search query.

For the above example, the changes can be reverted by updating the script to executed update.IsHidden = false; (instead of true), then re-executing the script. Alternatively, here the Windows Vista GUI works a little better: By clicking on "Restore hidden updates" from the side panel in Windows Update, the "Restore" button operates on the checkbox selection - allowing all hidden updates to quickly be restored with 2 clicks if desired.

Finally, here is an extended example that doesn't change anything, but displays some of the many details that are available through this API. First, it displays all the updates grouped and nested by category. Note that some updates belong to more than one category. Finally, it displays all available updates in a "flat" view, without using categories.

var updateSession = WScript.CreateObject("Microsoft.Update.Session");
var updateSearcher = updateSession.CreateUpdateSearcher();
updateSearcher.Online = false;

var searchResult = updateSearcher.Search("IsInstalled=1 or IsInstalled=0");

var describeCategory = function(cat, depth){
  var pad = new Array(depth + 1).join("  ");
  WScript.echo(pad + depth + ": " + cat + ", " + cat.CategoryID + ", " + cat.Name + ", " + cat.Type);

  for(var i=0; i<cat.Children.Count; i++){
    var child = cat.Children.Item(i);
    describeCategory(child, depth + 1);
  }
  
  for(var i=0; i<cat.Updates.Count; i++){
    var update = cat.Updates.Item(i);
    WScript.echo(pad + "  " + describeUpdate(update, pad + "  "));
  }
};

var describeUpdate = function(update, pad){
  var u = update;
  var np = "\n" + (pad || "") + "  ";
  return u.Title
    + np + "Type: " + u.Type
    //+ np + "Description: " + u.Description
    + np + "IsInstalled: " + u.IsInstalled
    + np + "IsDownloaded: " + u.IsDownloaded
    + np + "IsHidden: " + u.IsHidden
    + np + "AutoSelectOnWebSites: " + u.AutoSelectOnWebSites;
};

for(var i=0; i<searchResult.RootCategories.Count; i++){
  var category = searchResult.RootCategories.Item(i);
  describeCategory(category, 1);
}

WScript.echo("\n");

for(var i=0; i<searchResult.Updates.Count; i++){
  var update = searchResult.Updates.Item(i);
  WScript.echo(describeUpdate(update));
}

According to the IUpdateSearcher.Search documentation, the default search criteria is "IsInstalled = 0 and IsHidden = 0". Unfortunately, there doesn't seem to be a simple option to short-circuit the evaluator to just return all available updates, e.g. "" or "1=1". So far now, "IsInstalled=1 or IsInstalled=0" results in all updates being displayed. The only other note concerning the above example is that the "description" line is commented out in the describeUpdate function only because it can be rather verbose, and make the overall output difficult to read. Feel free to uncomment it to view the details, as well as adding additional lines for all the other properties available from IUpdate.

JScript for Ping, Renew IP and Network Info / Repair

I was having a frustrating issue where a remote Windows XP computer that I need to Remote Desktop into would occasionally become unresponsive. This issue already required me to drive a number of late-night trips to temporarily remedy the issue. While I've not yet found the root cause, the main issue seems to be that something becomes "disconnected" between the network adapter and the operating system. While the Windows "Status" dialog on the network adapter shows "Connected" and packets being sent and received, I am not even able to ping the default gateway. Windows Firewall is not enabled. The only other suspect software I need to rule out yet is an anti-virus service. Rebooting the computer always helped, but is rather drastic. Choosing the "Repair" option does not work, and shows a message about failing while renewing the IP address. Interestingly, the fastest temporary "fix" I found is to run "ipconfig /release" followed by "ipconfig /renew".

While this computer is probably due for a re-installation, it is not an immediate option. In the meantime, I came up with a Windows Script Host (WSH) script to automatically detect and repair this issue.

Update: The root cause is certainly related to "Symantec Endpoint Protection" 11.0.x (Antivirus). If Symantec is completely removed, the problem disappears. When reinstalled, the problem comes back. The "Network Thread Protection Logs" would show everything being blocked without reason, until reset by an IP renew or a reboot. (The "Rule" cited is either "Block local file sharing", even though it blocks ALL ports - or "Block all other traffic".) At least at one point, this was affecting dozens of desktops. However, Symantec needs to stay per corporate policy...

As previously posted, I chose to use JScript for this task (particularly over VBScript). Everything but the logging is essentially WMI (Windows Management Instrumentation) calls. If I hadn't coded for all of the extra logging, the code would have probably been less than half the current size. It basically finds a given network adapter, and attempts to ping the configured default gateway. If unsuccessful, it will perform a DHCP release and renew operation.

/** 
 * @author Mark A. Ziesemer, www.ziesemer.com, 2009-06-25
 */
(function(){
  
  var fso = new ActiveXObject("Scripting.FileSystemObject");
  var wmi = GetObject("winmgmts://localhost/root/cimv2");
  
  var getNetworkAdapter = function(name){
    var adapters = new Enumerator(wmi.ExecQuery(
      "Select * From Win32_NetworkAdapter Where NetConnectionID='" + name + "'"));
    if(!adapters.atEnd()){
      return adapters.item();
    }
  };
  
  var getAdapterConfig = function(index){
    var configs = new Enumerator(wmi.ExecQuery(
      "Select * From Win32_NetworkAdapterConfiguration Where Index=" + index));
    if(!configs.atEnd()){
      return configs.item();
    }
  };
  
  var ping = function(host){
    var results = new Enumerator(wmi.ExecQuery(
      "Select * From Win32_PingStatus Where Address = '" + host + "'"));
    if(!results.atEnd()){
      var result = results.item();
      return result.StatusCode;
    }
  };
  
  var logWriter = fso.OpenTextFile("PingTest.log", 8, true);
  try{
    var log = function(msg){
      logWriter.WriteLine(new Date() + " - " + msg);
    };
    
    var adapter = getNetworkAdapter("Local Area Connection");
    if(adapter){
      log("Found network adapter:"
        + "\r\n\tAvailability: " + adapter.Availability
        + "\r\n\tCaption: " + adapter.Caption
        + "\r\n\tConfigManagerErrorCode: " + adapter.ConfigManagerErrorCode
        + "\r\n\tErrorDescription: " + adapter.ErrorDescription
        + "\r\n\tLastErrorCode: " + adapter.LastErrorCode
        + "\r\n\tMACAddress: " + adapter.MACAddress
        + "\r\n\tName: " + adapter.Name
        + "\r\n\tNetConnectionID: " + adapter.NetConnectionID
        + "\r\n\tNetConnectionStatus: " + adapter.NetConnectionStatus
        + "\r\n\tStatus: " + adapter.Status
        + "\r\n\tTimeOfLastReset: " + adapter.TimeOfLastReset);
    }else{
      log("No suitable network adapter found.  Quitting...");
      return;
    }
    
    var adapterConfig = getAdapterConfig(adapter.Index);
    if(adapterConfig){
      log("Found network adapter config:"
        + "\r\n\tIPAddress: " + adapterConfig.IPAddress.toArray()
        + "\r\n\tDefaultIPGateway: " + adapterConfig.DefaultIPGateway.toArray());
    }else{
      log("No suitable network adapter configuration found.  Quitting...");
      return;
    }
    
    var pingResult = ping(adapterConfig.DefaultIPGateway.toArray()[0]);
    log("Ping result: " + pingResult);
    if(pingResult){
      var releaseResult = adapterConfig.ReleaseDHCPLease();
      if(releaseResult){
        log("Unexpected ReleaseDHCPLease() result: " + releaseResult);
        return;
      }
      var renewResult = adapterConfig.RenewDHCPLease();
      if(renewResult){
        log("Unexpected RenewDHCPLease() result: " + renewResult);
        return;
      }
      log("Successfully released and renewed IP: " + adapterConfig.IPAddress.toArray());
    }
  }finally{
    logWriter.WriteLine();
    logWriter.Close();
  }
  
})();

I added it to the Windows Task Scheduler, and set it to run every 5 minutes. While not available through the Windows UI, the schtasks command allows for scheduling as the "SYSTEM" user, assuming you have administrative rights. This eliminates the need for your password, and prevents the job from breaking when you change your password:

> schtasks /create /tn "PingTask" /tr "cscript C:\<path>\PingTest.js" /sc MINUTE /mo 5 /st 00:00:00 /ru System
INFO: The Schedule Task "PingTask" will be created under user name ("NT AUTHORITY\SYSTEM").
SUCCESS: The scheduled task "PingTask" has successfully been created.

While the problem detection isn't based on the packet count statistics that I observed from the Windows UI, I thought it would also be useful information to log to see if there is a correlation to the "disconnects". Unfortunately, I couldn't find an easy way to obtain these statistics - either through WMI or the command line. I did find a few places to obtain protocol/transport-level statistics such as TCP, but nothing at the link-layer (Ethernet) level. The only involved discussion I could find on this was this newsgroup posting to microsoft.public.win32.programmer.wmi back from 2006 which implies that these statistics are only available through native code.

MarkUtils-PacProxySelector for Java

Many computer networks make use of proxy servers for web and Internet connectivity. This is especially true for business and other organization networks, where their use is required by security and other policies. Due to the typical use of proxy servers, they are often thought of in terms of "restricting access". Instead, they should be thought of in the proper terms of a means of "providing access". Even outside of typical corporate environments, proxy servers can be invaluable for testing and debugging, as well as used as a type of VPN between private networks.

Most web browsers and other networked applications support directing traffic through one or more proxy servers. The typical configuration dialog looks like this, as shown from Mozilla Firefox:

Of the "manual" options, "HTTP" and "SSL" (TLS) are the most basic and common, followed by "FTP". "Gopher" is rarely used anymore - it has already been dropped by Microsoft Internet Explorer, and may be dropped in Mozilla Firefox 4.0. "SOCKS" is arguably the most powerful, supporting any of the above protocols in addition to any other TCP- or UDP-based protocol. (If SOCKS is configured and supported, none of the other protocols need to be configured.)

Unfortunately, directing an entire networks' traffic through a single proxy server can quickly cause a bottle-neck and a common point for failure. This is especially true when all LAN traffic is also sent through the proxy. (In ideal network traffic patterns, there should be many more multiples of LAN traffic over Internet traffic.) Some of the performance and availability concerns can be addressed through DNS or other load balancing. Another common attempt is to configure the "no proxy for" / exception list. Unfortunately, there are some severe limitations to this design. First, the list must be kept up-to-date as the network configuration changes. (There are various tools for this.) More significantly, there are many desired configurations that cannot be accounted for. For example, what if a list is needed for the servers that should be sent to a proxy server, rather than skipping the proxy server (reverse logic)? Or what if traffic must be split among multiple proxy servers, depending upon the destination or other parameters?

The solution to all of the above and other similar concerns is through the use of the last option shown above, and probably the most overlooked: "Automatic proxy configuration URL". This option is also known as proxy auto-config, or PAC, and was introduced into Netscape Navigator as early as 1996. A PAC file only needs to contain an implementation of a JavaScript function, FindProxyForURL(url, host). From here, the full power of JavaScript can be used, including regular expressions, associative arrays, and closures, as well as a number of predefined helper functions specific to PAC. Within the PAC function, various load balancing and black- or white-listing tasks can be performed, optionally by maintaining internal state. A list of multiple proxies may also be returned for attempts by the client. The PAC file is loaded from a URL (including local file:// URLs), where it can be centrally maintained and updated. The PAC file may be cached by the web browser or other client, but should respect the cache settings sent in the HTTP headers if retrieved through HTTP. Alternatively, a chrome:// URL can even be used, allowing for the PAC file to be maintained within a Firefox extension, and updated through Firefox's standard auto-update process for extensions.

Java support

Java supports many of the same above proxy options, mostly through the use of system properties. For full details, see the tech note at java.sun.com, Java Networking and Proxies. These settings affect any communications made through URLConnection, Socket, and possibly other network-related classes. Previously, the proxy configuration options were limited to the "manual" options listed above, with separate options for HTTP, HTTPS, FTP, and SOCKS. However, Java 1.5/5.0 introduced the Proxy and ProxySelector classes. A default ProxySelector can be configured for the current JVM by calling ProxySelector.setDefault(ProxySelector).

Unfortunately, Java does not currently provide any visible support for proxy auto-config (PAC) files. However, the ProxySelector's List<Proxy> select(URI uri) method looks and works very similar to the PAC's FindProxyForURL(url, host) function. The most notable difference is that it is strongly-typed to standard Java classes. As part of my MarkUtils collection, I created MarkUtils-PacProxySelector to provide a ProxySelector implementation that works with PAC files.

Since the PAC files are based on JavaScript, the ability to evaluate JavaScript is required. Fortunately, this is easily done through Java, especially with the introduction of the Java Scripting API in Java 1.6/6.0 (JSR-223). Java 1.6 bundles an internal version of the Mozilla Rhino implementation of JavaScript for Java, based on 1.6R2. Unfortunately, Java doesn't expose all the features of JavaScript or Rhino directly through the scripting API, some of which are required to implement the PAC functionality in a compatible fashion. This includes defining top-level bindings in the JavaScript environment to Java functions, which is directly supported in Rhino by adding a binding to a FunctionObject - a class to which there is no publicly visible match in the JDK. While it is probably possible to hack a work-around to this, my current implementation utilizes Rhino directly. Besides taking advantage of the improvements in the latest version of Rhino (currently 1.7R2), this allows the utility to be easily used with both Java 1.5/5.0 and 1.6/6.0. (However, note that JSR-223 is unofficially supported under Java 1.5/5.0 as well by downloading and including the .jar's from the reference implementation.) Using Rhino directly also avoids some potential security issues, which I reported in Sun Bug 6782031 and Mozilla Bug 468385.

As commented in the pom.xml file, Mozilla Rhino is currently not available through the central repository, a Mozilla repository, or any other "official" repository. I've added a dependency to it as "org.mozilla.javascript : rhino : 1.7R2". For this to work properly, Rhino will need to be downloaded and installed into a local repository as named above.

In addition to the standard PAC methods, PacProxySelector supports an added function called "connectFailed" to take advantage of the connectFailed(URI, SocketAddress, IOException) functionality on ProxySelector. The JavaScript method is called with the same arguments as on ProxySelector, just with the .toString() representations of each of the three parameters. The PAC file could then store this information within internal state to possibly affect future calls to FindProxyForURL.

For the most flexibility, the constructor to PacProxySelector accepts a Reader, which should read from a PAC file. There is also a public static configureFromProperties() method that returns a ProxySelector, assuming that the path to a PAC file is stored as either a Java system or environment property named "proxy.autoConfig", similar to the other network properties. After obtaining an instance from either the constructor or the method, it should be passed to ProxySelector.setDefault(ProxySelector), unless otherwise used directly. Alternatively, a setDefaultFromProperties() convenience method is provided to do this in one call.

I wrote this in mind for plugging into other Java applications. Ideally, the JDK would provide a system property that accepts the classname for the default ProxySelector or some other method for setting the default outside of a function call within the code, but this is currently not the case. However, all that has to be done is finding a way to execute one of the above configuration options from the desired Java application before network access is attempted. I've successfully written a plugin for Oracle SQL Developer that does exactly this. The same is also possible for Eclipse, though it requires patching of some of the plugins due to the current infrastructure. (See Eclipse bug 257443.) Alternatively, PacProxySelector provides a main method that calls setDefaultFromProperties() before chaining execution to another program's main method. See the included Javadoc for details.

Download

com.ziesemer.utils.pacProxySelector is available on ziesemer.java.net under the GPL license, complete with source code, a compiled .jar, generated JavaDocs, and a suite of JUnit tests. Download the com.ziesemer.utils.pacProxySelector-*.zip distribution from here. Please report any bugs or feature requests on the java.net Issue Tracker.

See all newspaper comments at once with Greasemonkey

Background

Like many web users today, I get much of my local news from the online versions of the local newspapers. One particular feature the online editions offer over the print editions is the inclusion of user comments / responses to the stories.

At least around my areas of interest in Central Wisconsin and the Fox Valley, most of the local papers are owned by the Gannett Company. This includes the Appleton Post-Crescent, the Wausau Daily Herald, and others. Much of Gannett's online presence is currently provided by Pluck Social Media's SiteLife product, particularly SiteLife Comments. Pluck even hosts a special customer profile detailing their work with Gannett Corporation. While the sites I'm working with here all happen to be owned by Gannett, it's quite possible that this will also apply to other Pluck-based sites as well.

Unfortunately, Pluck's current implementation leaves some things to be desired. I recall reading many of the negative comments left as the transition was made from the old version of the sites to the current version, which is when I believe Pluck became involved. Fortunately, they have improved some things since, and definitely seem to be fairing better than the current fiasco at Dell's online community after their so-called upgrade. However, the most annoying issue I have while reading Gannett's local news articles is that only 5 user comments are visible at a time. There is a "Full Page View" option available at the bottom, but this only increases the visible comments-per-page to 10. While all these comments are loaded in an AJAX-type fashion using JSON data, clicking to retrieve the next page still results in reloading the entire page. Even on a broadband connection, each page change requires 5+ seconds. This makes trying to read through all the comments on a popular story very frustrating, especially when there are sometimes 50 or more responses. While many of these comments are informative or insightful, having to click through and reload 5 or more pages is certainly not making the best use of web technology.

Technical Challenges

As I had done with Resizing the Blogger Edit Box, my first thought was to attempt to improve things with a Bookmarklet. Unfortunately, the task proved to be too complex, partially due to the same origin policy blocking the necessary cross-domain data. In particular, while the article page is served from a "www." host, the JSON data containing the comments is obtained from a "sitelife." host. While the current pages seem to work around this restriction through some iframe tricks, attempting to reuse that functionality would be a hack at best. Instead, I turned to a Greasemonkey-based solution. Greasemonkey provides a non-domain-restricted GM_xmlhttpRequest API method that provides access to Mozilla's chrome-privileged XMLHttpRequest object.

The pages I had to work with were not at all desirable or the easiest to work with. Each page typically includes about 20 JavaScript files, and some of the code is quite obfuscated. One of the main files, "GDSRScripts.js", is about 86 KB. The core of the Yahoo! UI Library (YUI) (yahoo-dom-event.js in 2.6.0) is not even half that, at only 31 KB. I also see no effort made at respecting the JavaScript global namespace, or use of other best practices.

The Solution

I've completed a Greasemonkey script that I've posted at userscripts.org: All Pluck Comments. Once installed and configured for one or more of the Pluck-based Gannett news sites, it will update any loaded news article by showing all available comments on the same, single page. If all the existing comments already fit within one page and the current 5-post limit, the script will exit and do nothing. Unfortunately, much of the previous waiting time doesn't seem to be in the JavaScript, but with the server responding to the JSON requests - a performance issue that can't be resolved client-side. While those requests are made, the script will show the loading status above the existing comments. Once all comment "pages" have been downloaded, the comments section is repopulated with the complete list. Additionally, changing the sort order between "Newest first" and "Oldest first" now performs instantly, without requiring additional remote requests.

Due to the number of possible supported sites, only one default URL pattern is configured to the "included pages" within the Greasemonkey script. Other desired, supported sites will need to be manually added. (This would be easier of Greasemonkey supported regular expressions for the patterns, as I requested in ticket #216.) There are two types of URLs I've observed that should be supported. The first looks like "http://<hostname>/article/<date>/<siteId>/<articleId>/". The other looks like "http://<hostname>/apps/pbcs.dll/article?AID=/<date>/<siteId>/<articleId>". The best non-regular expression pattern I can suggest to match both these patterns is "http://<hostname>/*article*", where "<hostname>" needs to be replaced with the literal host and domain name to be supported.

The only current limitation is that the per-comment controls (Recommend, New post, Reply to this Post, Report Abuse, etc.) are not regenerated. This is because it would be very difficult, if even possible, to make all of Pluck's existing JavaScript work with these enhancements. In order to use these controls, click on the "Full Page View" link that is left below the list of comments. This will bring back the limit of 10 comments per page, but the Greasemonkey script will exit without making any changes and leaving these controls intact. I seldom use these controls, so this issue isn't that important to me. However, if there is a stated interest, I may look into resolving this for a future version. Alternatively, feel free to write and submit a patch!

Technical Details

The script first waits for the existing comments to load, at which point it determines the article ID, the total number of comments available, and other information necessary for requesting the additional "pages" of comments. If it times out waiting, or determines that it is on the "Full Page View", it simply exits and does nothing. Otherwise, it makes a series of asynchronous requests to retrieve all the available comments. The responses are unnecessarily URL-encoded, and are decoded by the script using unescape(). The responses also contain an unnecessary <script> section at the beginning, which is searched for and removed. The JSON text is then "safely" evaluated to a JavaScript object using the regular expression provided in section 6 of RFC 4627. Once all responses are received, the existing comments HTML is cleared, and new comments are built and populated from the JSON data using the HTML DOM.

Some of the tools I used during this process were Firebug, JSView, and Notepad++. Some of the JavaScript practices I used include closures and other JavaScript topics I've written about.

New version of MarkUtils-Web: ZipServlet and CompressFilter

If you're not already familiar with my ZipServlet and CompressFilter, see my previous posting where I introduced these Java web utilities.

As with the previous release, the update is available on ziesemer.java.net. The release folder is directly available here. "com.ziesemer.utils.web-2009.01.11.zip" contains the source code, a compiled .jar, and generated JavaDocs.

New in this release are a number of fixes and enhancements. To report any new bugs or enhancement requests, please use the issue tracker on ziesemer.dev.java.net.

ZipServlet combo mode

ZipServlet now provides a feature almost identical to YUI's Combo Handling. This allows for multiple files to be requested and sent joined together in one response, which can reduce the number of HTTP requests and improve performance, as detailed in this post on yuiblog.com.

This feature works particularly well for JavaScript and CSS files. However, only files of one type should be requested together, otherwise the returned MIME Content-Type HTTP header wouldn't make any sense. Unlike the current implementation provided by yahooapis.com, ZipServlet enforces this by returning a HTTP 400 error (Bad Request) if multiple files are requested that have different content types.

The combo mode is enabled by default in ZipServlet, but can be disabled by using the "comboEnabled" servlet parameter. The path on which combination requests are answered can also be configured through the "comboPath" servlet parameter. comboPath defaults to "combo", as used by Yahoo! / YUI. Additional details are listed in the JavaDocs for ZipServlet, included in the download.

A notable difference between ZipServlet and the Yahoo! / YUI implementation is that YUI files require the version to be prefixed to each requested file, e.g. "/combo?2.6.0/build/yuiloader/yuiloader-min.js&2.6.0/build/dom/dom-min.js&2.6.0/build/event/event-min.js". This presumably allows for multiple files to be requested across different versions. This doesn't make a lot of sense, and results in a longer URL as the version is prefixed before each requested file. As ZipServlet is designed to have each instance associated with one .zip file, as each version of a resource (YUI, etc.) belongs in its own file, and due to implementation details, it made sense to include the "combination" functionality directly into ZipServlet rather than as an additional filter, etc. This restricts combination requests to a given ZipServlet instance, while improving code reuse, performance, the URL length, and other aspects. The equivalent URL to the above when requested from ZipServlet, when using the example configuration in the previous post, including the "yui/" zipPrefix, would be "/combo?/build/yuiloader/yuiloader-min.js&/build/dom/dom-min.js&/build/event/event-min.js". Additionally, the leading slash before each file is optional in the ZipServlet implementation.

Unit Tests

New in this release is a complete suite of JUnit tests. I previously had not included any tests partially due to finding a unit testing solution for Java EE servlets and filters that met my requirements. I had looked at HttpUnit ServletUnit, but found a number of shortcomings. While I had previously been testing under Apache Tomcat, this required manual starting and stopping of the server, and registration of the projects. There are methods to automate this, but none that didn't seem overly complex and without their own shortcomings.

I finally decided upon using Jetty. Jetty is both free and open-source, and is 100% Java which makes it very portable. It is very representative of a production servlet engine as it is one, used by many projects including the JBoss and Apache Geronimo application servers. Jetty fully supports embedded within a Java application, which makes it very suitable for unit testing. It can be configured either declaratively or by using standard web.xml files, with details and several examples available at http://docs.codehaus.org/display/JETTY/Embedding+Jetty. As an added bonus, Jetty is readily available as an Apache Maven artifact, including the latest release and beta versions. This means that if opening the project with Maven support, such as with m2eclipse, Jetty and any other dependencies will automatically be downloaded and included on the testing classpath.

For this project, I wrote a ServletTester class that is reused by all the test classes. Each instance starts a Jetty instance on a dynamic port on the loopback address (127.0.0.1), rather than requiring that a specific be used or configured. This class then provides convenience methods for obtaining the server, connector, context, and base URL.

There are 2 types of tests that I provided for each component (ZipServlet and CompressFilter) - standard tests and configuration tests. The standard tests initializes and holds on to a Jetty instance within a ServletTester as a static field, which is reused by all the test methods. This is mainly for performance reasons, so that a new server isn't required for every test. The configuration tests deal specifically with testing the configuration options available for each component. While these configuration tests still reuse the same server instance, the server's context is restarted with a new ServletHandler for every test.

For the details of this methodology, please feel free to download the code and see for yourself. These tests also demonstrate the features and usage of the components.