Scraping Sun's bug database with NekoHTML

Sun has a rather comprehensive bug database at http://bugs.sun.com/, which is probably most notably used to track Java bugs. Unfortunately, at least with the interface and options presented to the general public (such as myself), some aspects are rather lacking. Beyond being able to vote for a bug (only up to 3), and creating a single "watch list", it appears to be a far cry from Bugzilla.

The feature I've been missing the most, however, is an easy way to keep track of bugs of interest, and by extension, being able to share bug lists / searches. Bugzilla makes this incredibly easy by offering a number of export options, including XML, CSV, and RSS feeds. As of Bugzilla 3.0, saved searches can also be shared, as an effort of Bugzilla Bug 69,000, and is demonstrated in Bugzilla's public test environment, http://landfill.bugzilla.org/bugzilla-3.0-branch/.

As a side note, while both https://bugs.eclipse.org/bugs/ and https://bugzilla.mozilla.org/ appear to be running version 3.0+ of Bugzilla, neither currently seem to have these shared queries working/enabled. (I've opened bug 223594 on Eclipse to hopefully address this.)

Java HTML Parsers

Unfortunately, as I previously mentioned, Sun isn't using Bugzilla, and there are no export options of any type. As far as I can tell, this really only leaves parsing the HTML as the only option.

If the HTML returned by Sun on their bug pages was valid XHTML, this would be an easier task. Just write a few XPaths to find the necessary fields, format the data as desired, and another task complete. Unfortunately, the pages are returned simply as regular HTML, not the XML-compatible variety. At least unlike Blogger, Sun's pages are at least returned with a valid identifying type that actually matches the page, in this case a doctype of "-//W3C//DTD HTML 4.01 Transitional//EN".

There are many ways to parse HTML. While I know a lot of people like Perl, I prefer to stick with the Java approach. In any language, there are also many existing tools and frameworks to help with the task, so starting from scratch would probably be a waste of time. The non-XML-based version of HTML can be rather messy, with unbalanced tags, missing escape sequences, and other issues that will quickly lead to headaches - and these tools will help "clean" the input.

I recently found a blog posting about a similar goal: "Showdown - Java HTML Parsing Comparison" (Ben McCann, 2008-02-02, lumidant.com). He demonstrates 4 Java libraries for parsing HTML, including NekoHTML, TagSoup, jTidy, and HTMLCleaner. (Unlike the referenced blog, I've linked these utilities to their official sites for your convenience.) The referenced post favored HTMLCleaner, as it was supposedly the only tool to successfully extract 10/10 documents. I've had better luck with NekoHTML. I'd be curious to see the rest of the code that was used, as I suspect that case sensitivity and a few other issues may have played into the results, and at least with NekoHTML, can easily be normalized with a configuration option or two.

NekoHTML

In the past, I utilized HttpUnit as such a tool. It actually uses NekoHTML as the parser, then provides a HTML-specific API for navigating/querying the "cleaned" document - as well as performing actions, for HttpUnit's real purpose as a testing framework.

Of the four parsers listed above, NekoHTML definately seems to be the most comprehensive and as of recently, actively maintained. After a period without any releases between June 2005 and December 2007, it was relaunched on SourceForge. Beware of version 1.9.6.2, however, as there seems to be a rather severe regression bug with the handling of single quotes, as I reported in issue 1922810.

For starting with NekoHTML, I recommend the following configuration:

import org.cyberneko.html.parsers.DOMParser;
// …
DOMParser domParser = new DOMParser();
domParser.setFeature("http://cyberneko.org/html/features/insert-namespaces", true);
domParser.setProperty("http://cyberneko.org/html/properties/names/elems", "lower");

See http://nekohtml.sourceforge.net/settings.html for the details of these options. The insert-namespaces feature basically utilizes the XHTML namespace, http://www.w3.org/1999/xhtml, to all HTML content, allowing for distinction between HTML content and other possible content defined in an alternate namespace within the document. The names/elems property instructs NekoHTML to convert all tag names to lower case, which matches the XHTML specification (compared to upper-case for HTML). Setting the insert-namespaces feature seems to be a pre-requisite for the names/elems property.

The only down-side to enabling XML namespaces is slightly complicating the use of XPath. In order to properly query elements in XML namespaces with XPath, a javax.xml.namespace.NamespaceContext implementation needs to be registered to the XPath using XPath.setNamespaceContext(…), which maps prefixes to namespaces, etc. (See NamespaceContextMap in MarkUtils-XML.)

Parsing Sun's bug pages

For extracting the fields from one of Sun's bug pages, I formulated two XPath expressions:

//html:table[preceding::html:a[@name='skip2content']]//html:table/html:tbody

This expression finds the body of the table after an anchor that separates the desired content from the rest of the page headers, navigation, etc.

html:tr[html:td//text()=$pageLabel]/html:td[position()=2]

This expression finds the value of a desired field on the page, e.g. "Bug ID:", "Synopsis", or "Category", as currently represented by $pageLabel above. The expression finds the first <td/> node matching the desired label, then returns the following <td/> node that contains the desired value.

Use a XPathVariableResolver to handle the variable in the 2nd expression. If multiple bug pages are to be processed, these expressions should probably be compiled to XPathExpressions for repeated use.

Here is my export of the bugs I'm currently "watching" / am interested in:

Bug ID	Synopsis	Category	Reported Against	Release Fixed	State	Priority	Submit Date
4079882	Request for JTristateCheckbox implementation	java:classes_swing	1.3.1 , 1.4.1 , 1.1fcs		In progress, request for enhancement	4-Low	1997-09-17
4187336	ServletResponse.setContentLength(Long)	javax_servlet:api	1.1fcs		Closed, will not be fixed	4-Low	1998-11-05
4526561	File system change notification events should be supported	java:classes_io	merlin-beta2		In progress, request for enhancement	4-Low	2001-11-13
4652184	please compile j2sdk rt.jar with -g (all options)	java:build	1.4.2 , 1.4.2_04 , merlin-rc1 , tiger-beta , tiger-beta2	mustang(b28)	Closed, fixed	4-Low	2002-03-13
4782054	Allow for comments in the MANIFEST.MF file	java:jar	1.4.1		In progress, request for enhancement	4-Low	2002-11-20
4787931	System property "user.home" does not correspond to "USERPROFILE" (win)	java:classes_lang	1.3 , 1.4.1 , 1.4.2		In progress, bug	3-Medium	2002-12-03
4838318	(str) Substitute CharSequence for String arguments wherever possible	java:classes_lang	1.4.1 , 1.4.2		In progress, request for enhancement	4-Low	2003-03-27
4880234	ServiceUI needs a printDialog method wtih a Component parameter	java:classes_2d	1.4.1		In progress, request for enhancement	4-Low	2003-06-18
4983159	Typedef (alias)	java:specification	tiger-beta		In progress, request for enhancement	4-Low	2004-01-24
5018574	Unable to set focus to another component in JOptionPane	java:classes_swing	tiger		In progress, bug	3-Medium	2004-03-23
5043696	StringReader should be allow a String{Buffer,Builder} to be the backing store	java:classes_io	1.4.2		In progress, request for enhancement	4-Low	2004-05-07
5096679	PIT:PrintDialog is not positioned properly on multi-mon, when coords are invalid	java:classes_2d	mustang		In progress, bug	4-Low	2004-09-03
5109347	PrinterJob.printDialog() does not support multi-mon, always displayed on primary	java:classes_2d	1.4		In progress, bug	4-Low	2004-09-30
6192554	Need generic factory interface.	java:classes_util			In progress, request for enhancement	4-Low	2004-11-09
6212751	DOC: ServiceUI.printDialog() need to enhance the description for X,Y coordinates	java:classes_2d	1.4		In progress, bug	4-Low	2004-12-27
6214380	Quality setting is disabled and always set to Normal in Print Dialog	java:classes_2d			In progress, request for enhancement	4-Low	2005-01-05
6215174	Can't force layout of non-showing component	java:classes_awt	5.0		In progress, request for enhancement	4-Low	2005-01-07
6312085	The for/in statement should support Iterators	java:specification	tiger-beta		In progress, request for enhancement	4-Low	2005-08-17
6325564	(str) Provide CharSequenceReader with sub-sequence capability	java:classes_lang			In progress, request for enhancement	4-Low	2005-09-19
6358852	Add methods on concurrent data structures that interrupt blocked threads	java:classes_util_concurrent			In progress, request for enhancement	4-Low	2005-12-05
6400189	raw types and inference	java:compiler			In progress, bug	4-Low	2006-03-17
6476646	(str) Make AbstractStringBuilder class public	java:classes_lang			In progress, request for enhancement	5-Very Low	2006-09-29

If I can find the time, a complete sample code download may also follow.

Sun's version of "640k": 2GB ServletResponse Content-Length

Whether or not Bill Gates ever actually said that 640k should be enough for everyone, Sun certainly seems to have recreated their own version with javax.servlet.ServletResponse.

I was working on a custom Java servlet to return files out of a zip file stored on a web server (ZipServlet), was trying to implement RFC 2616 as completely and accurately as possible, and noticed an issue with the setContentLength(int len) method. Java's int data type is 32 bits, and all Java data types are signed, meaning that this imposes a maximum declared response length of 2,147,483,647 bytes, or 2 GB.

To be fair, this was probably generous back in 1998. I believe the first 2GB+ file I ever downloaded was a Linux installation DVD .iso, and even today, this type of use is still probably a bit out-of-place for a Java Servlet.

What makes this interesting is the number of other Java classes that already utilize longs (64-bit signed) over ints. java.io.File and java.util.zip.ZipEntry are two examples that come to mind.

I did find a bug entry on this issue, http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4187336, opened in 1998. The evaluator's comment is a bit discouraging. I understand the need to maintain API compatibility, but this didn't seem to prevent Sun from fixing a similar bug elsewhere. (I recalled - but am now unable to locate - another java.* class with 2 methods something like int getLength() and long getLongLength(), the later having been introduced later to overcome the size limitation of an int.)

Fortunately, I did find one possible work-around:

javax.servlet.http.HttpServletResponse response;
long length;
…
if(length <= Integer.MAX_VALUE){
  response.setContentLength((int)length);
}else{
  response.addHeader("Content-Length", Long.toString(length));
}

Unfortunately, it probably only serves as informational to the client. The servlet container probably won't make use of this, and resort to sending the content as chunked.

Sun Download server blocking US users?

I just witnessed something a bit peculiar while trying to download the latest Java 6 JDK. Headed to http://java.sun.com/javase/downloads/index.jsp, and clicked on the Download button for "JDK 6 Update 4". This directed me to a 2nd page to "Select Platform for your download", the URL beginning with https://cds.sun.com/is-bin/INTERSHOP.enfinity/WFS/CDS-CDS_Developer-Site/en_US/-/USD/ViewFilteredProducts-SingleVariationTypeFilter.

No matter what platform I choose, the same page returns an error: "Your download transaction cannot be approved. Contact Customer Service."

This error is listed in Sun's Download Center FAQ at http://www.sun.com/download/faq.xml:

Q: Why am I stopped from downloading and getting the message, "Your download transaction cannot be approved. Contact Customer Service."

A: The Sun Download Center contains important safeguards to ensure compliance with the export regulations of the United States and with Sun's own policies regarding international distribution of software and source code. Sun also enforces all U.S. mandated embargoes and trade sanctions with regards to certain countries.

Your account most likely was flagged for review. Please contact Customer Service with your full contact information (name, company, address, phone, email address) and the name of the product you wish to download. We will review your account and reply.

Being a citizen of the United States, and receiving the error from a download request through Charter Communications in Wausau, WI, I find this very interesting. My IP resolves to x.x.x.x.dhcp.stpt.wi.charter.com. It also doesn't appear to be an issue with the browser or cookies, as I receive the error in both Firefox and Internet Explorer. I also tried the download request through a Charter business account, with an IP that resolves to x.x.x.x.static.stpt.wi.charter.com.

I followed the error link, submitted a download inquiry form, and promptly received an auto reply stating that the issue is under review and should probably be resolved within one business day.

Waiting to see what happens. Interesting...

Update:

I've yet to figure out exactly how these versions are supposed to related, but some have claimed that downloading from http://download.java.net/jdk6/binaries/ is the same thing. However, according to http://java.sun.com/javase/6/webnotes/ReleaseNotes.html, for version 1.6.0_04, "The full internal version number for this update release is 1.6.0_04-b12". However, what is posted on java.net is b11... (Please comment if you can help me clear up this confusion.)

Alternatively, you can try your luck with Google's search results.

Update:

Posted a question to the java.net forums regarding the JDK versions available on the jdk6.dev.java.net site: http://forums.java.net/jive/thread.jspa?threadID=37203

Also, according to Google Analytics, this site has received over 130 visits so far today - a new record. It appears most were related to this particular post, and that the requests came from 32 different countries! Interestingly, I've not seen any news from Sun about the issue nor have I received an email reply. One would think they would've at least put a note on their site, something to the effect of "We're aware of the issue and should have the issue resolved shortly."

Update (2008-02-27):

Received an email back from Sun Customer Support: "The Sun Download Center was experiencing technical difficulties but it is now functioning properly. Please try your download again again and let us know if you are still experiencing any problems. We apologize for the inconvenience."

Update (2008-03-17):

Ray Gans, one of the maintainers of the jdk6 project on dev.java.net, finally replied to my Forum post with some insightful information regarding the apparently missing source and binary releases: http://forums.java.net/jive/thread.jspa?threadID=37203