Alltel UM175AL USB EVDO under Ubuntu Hardy Heron

(This post is the 3rd part of my Ubuntu Linux Router Upgrade Project.)

Note that all Linux commands here assume use of Ubuntu Linux 8.04 ("Hardy Heron"), with all updates installed as of 2008-09-30.

The device I purchased for Alltel's Wireless Internet service (3G 1xEV-DO) was a UTStarcom UM175, purchased for $100 before a $100 mail-in rebate on a 2-year contract.

Some specs as printed on the box and from Alltel's website:

• USB modem with swivel connector
• Compatible with Windows XP, Vista, Mac OS X
• 1xRTT/EVDO Rev 0 and Rev A Ready
• CDMA 800/1900MHz

Also Known As

The "Alltel Part#" listed on the box label is "UM175ALA". The P/N on the device label is "UM175AL". (I'm assuming the "AL"/"ALA" suffixes are specific to Alltel?) The FCC ID is PP4PX-700. Also listed on the device label: "Distributed by UTStarcom Personal Communications. Made in Korea by PANTECH. QUALCOMM 3G CDMA."

When connected to Windows before installing the drivers, the following "Hardware Ids" are listed in Device Manager where it is recognized as a "USB Mass Storage Device", with a vendor ID of 0x106c and a product ID of 0x3b03:

USB\Vid_106c&Pid_3b03&Rev_0100
USB\Vid_106c&Pid_3b03

And here is the output of "/usr/sbin/lsusb -l" under Linux:

Bus 001 Device 006: ID 106c:3b03 Curitel Communications, Inc.
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               1.10
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0
  bDeviceProtocol         0
  bMaxPacketSize0        64
  idVendor           0x106c Curitel Communications, Inc.
  idProduct          0x3b03
  bcdDevice            1.00
  iManufacturer           1 PANTECH
  iProduct                2 USB MMC Storage
  iSerial                 3 000000000002
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           32
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0
    bmAttributes         0xc0
      Self Powered
    MaxPower              100mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass         8 Mass Storage
      bInterfaceSubClass      6 SCSI
      bInterfaceProtocol     80 Bulk (Zip)
      iInterface              0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x83  EP 3 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040  1x 64 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x05  EP 5 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040  1x 64 bytes
        bInterval               0
Device Status:     0x0001
  Self Powered

Note that the above shows a vendor ID of 0x106c ("Curitel Communications, Inc.") and a product ID of 0x3b03.

Should be Easy (or not)

Even before I made the purchase, I ran across a blog post by Jason Costomiris, "HOWTO: Verizon UM175 USB EVDO Card under Ubuntu Hardy". Reading it gave me a good vote of confidence that my plan should work. He had success using the "same" device under the same distribution of Linux that I was planning on. Verizon and Alltel use the same CDMA technology (and Verizon is buying Alltel). The instructions made it almost appear as simple as "plug & play".

Unfortunately, this is where things got tricky. The device is not even shipped with drivers, at least not on a CD. Instead, the device doubles as a USB mass storage device (USB MCS / UMS), and is recognized as an Autorun-enabled CD-Rom with drivers for Windows and Mac OS. Under Windows, after the drivers are installed, the device is listed as a "PANTECH UM175AL Composite Device", a "UM175AL CD-ROM USB Device", a "PANTECH UM175AL Diagnostic Port", and a "PANTECH UM175AL" modem.

By default, Linux only sees this device as a USB CD-Rom. As such, no serial port (e.g. /dev/ttyACM0) is ever created, leaving PPP nothing to connect to.

I need to thank Jason for responding to the comment I left on his post asking for help. Unfortunately, it appears that my "UM175AL" is a slightly different revision than what Jason used with so few issues.

lsusb Note

Many pages I found while researching this issue referenced the output of "/proc/bus/usb/devices". This does not exist under recent versions of Ubuntu Linux. lsusb appears to be one of the preferred replacements. Some details are available in Ubuntu bug #156085 on launchpad.

USB_ModeSwitch

The closest I found to anyone having a similar issue was a post by "theosib" on the Ubuntu Forums with a post less than a month old, "Total disaster trying to set up Verizon EVDO device". I replied with some details of my issue, after which "theosib" replied with what would turn out to be my solution - USB_ModeSwitch.

Be sure to read the main content on USB_ModeSwitch's main page at http://www.draisberghof.de/usb_modeswitch/, as it is a pretty good summary of the issue.

Note that in my response on the UbuntuForums, I stated that when the device is properly recognized by Windows, the Vendor ID was 0x3715, compared to the current 0x3b03. So the goal is getting this to "switch" to 0x3715, at which point it should be (and is) properly recognized as a USB modem by Linux.

The download of USB_ModeSwitch comes with a precompiled executable, but I'm guessing it was compiled for 32-bit, as it wasn't recognized on my system. I recompiled it from the included source without any issues. Just make sure the "build-essential" and "libusb-dev" packages are installed, as this isn't mentioned on the page. Following the notes on the USB_ModeSwitch page, I placed a copy of the output in "/sbin" as "/sbin/usb_modeswitch" using sudo.

Given the about output from "lsusb", I already had the "DefaultVendor" and "DefaultProduct" values needed to configure USB_ModeSwitch. Using these successfully detached the storage driver, but didn't have any effect on the Vendor ID or any apparent change on the output listed by lsusb. Following the documentation, this most likely meant that I still needed to send an additional special command to make the device "switch", using the "MessageEndpoint" and "MessageContent" parameters.

The preferred tool to use to find these parameters mentioned in the USB_ModeSwitch documentation is "SniffUSB". The link is for version 1.8. For me, it would constantly and uncontrollably refresh the screen, making it impossible to scroll the device list and pretty much making it unusable. Fortunately, I found an updated version 2.0, based on the same source, at http://www.pcausa.com/Utilities/UsbSnoop/default.htm. (Source code is readily available for both versions.)

To sniff the necessary commands from this device, I found the VID/PID with the "unswitched" ID's. Specifically, "USB\Vid_106c&Pid_3b03&Rev_0100" ("PANTECH UM175AL Composite Device"). I then installed the filter, then removed and reinserted the device. The following is my captured "UsbSnoop.log", with the necessary message portions highlighted:

[3 ms] UsbSnoop - FilterAddDevice(9a47f748) : DriverObject 89773a48, pdo 8992a9d8
[3 ms] UsbSnoop - FilterDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_QUERY_LEGACY_BUS_INFORMATION)
[3 ms] UsbSnoop - FdoHookDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_QUERY_LEGACY_BUS_INFORMATION)
[4 ms] UsbSnoop - FilterDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_QUERY_RESOURCE_REQUIREMENTS)
[4 ms] UsbSnoop - FdoHookDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_QUERY_RESOURCE_REQUIREMENTS)
[4 ms] UsbSnoop - FilterDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_FILTER_RESOURCE_REQUIREMENTS)
[4 ms] UsbSnoop - FdoHookDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_FILTER_RESOURCE_REQUIREMENTS)
[4 ms] UsbSnoop - FilterDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_START_DEVICE)
[4 ms] UsbSnoop - FdoHookDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_START_DEVICE)
[5 ms] UsbSnoop - FilterDispatchAny(9a47afd2) : IRP_MJ_SYSTEM_CONTROL
[8 ms] UsbSnoop - FilterDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_QUERY_CAPABILITIES)
[8 ms] UsbSnoop - FdoHookDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_QUERY_CAPABILITIES)
[8 ms] UsbSnoop - FilterDispatchAny(9a47afd2) : IRP_MJ_INTERNAL_DEVICE_CONTROL
[8 ms] UsbSnoop - FdoHookDispatchInternalIoctl(9a47b1ea) : fdo=8992a9d8, Irp=888c49d0, IRQL=0
[8 ms]  >>>  URB 1 going down  >>> 
-- URB_FUNCTION_GET_DESCRIPTOR_FROM_DEVICE:
  TransferBufferLength = 00000012
  TransferBuffer       = 89a098f8
  TransferBufferMDL    = 00000000
  Index                = 00000000
  DescriptorType       = 00000001 (USB_DEVICE_DESCRIPTOR_TYPE)
  LanguageId           = 00000000
[11 ms] UsbSnoop - MyInternalIOCTLCompletion(9a47b126) : fido=00000000, Irp=888c49d0, Context=89951008, IRQL=2
[11 ms]  <<<  URB 1 coming back  <<< 
-- URB_FUNCTION_CONTROL_TRANSFER:
  PipeHandle           = 89673020
  TransferFlags        = 0000000b (USBD_TRANSFER_DIRECTION_IN, USBD_SHORT_TRANSFER_OK)
  TransferBufferLength = 00000012
  TransferBuffer       = 89a098f8
  TransferBufferMDL    = 8979e558
    00000000: 12 01 10 01 00 00 00 40 6c 10 03 3b 00 01 01 02
    00000010: 03 01
  UrbLink              = 00000000
  SetupPacket          =
    00000000: 80 06 00 01 00 00 12 00
[11 ms] UsbSnoop - FilterDispatchAny(9a47afd2) : IRP_MJ_INTERNAL_DEVICE_CONTROL
[11 ms] UsbSnoop - FdoHookDispatchInternalIoctl(9a47b1ea) : fdo=8992a9d8, Irp=888c49d0, IRQL=0
[11 ms]  >>>  URB 2 going down  >>> 
-- URB_FUNCTION_GET_DESCRIPTOR_FROM_DEVICE:
  TransferBufferLength = 00000009
  TransferBuffer       = 896cab78
  TransferBufferMDL    = 00000000
  Index                = 00000000
  DescriptorType       = 00000002 (USB_CONFIGURATION_DESCRIPTOR_TYPE)
  LanguageId           = 00000000
[15 ms] UsbSnoop - MyInternalIOCTLCompletion(9a47b126) : fido=00000000, Irp=888c49d0, Context=89951008, IRQL=2
[15 ms]  <<<  URB 2 coming back  <<< 
-- URB_FUNCTION_CONTROL_TRANSFER:
  PipeHandle           = 89673020
  TransferFlags        = 74f06e2f (USBD_TRANSFER_DIRECTION_IN, USBD_SHORT_TRANSFER_OK)
  TransferBufferLength = 00000009
  TransferBuffer       = 896cab78
  TransferBufferMDL    = 8898c160
    00000000: 09 02 20 00 01 01 00 c0 32
  UrbLink              = 00000000
  SetupPacket          =
    00000000: 80 06 00 02 00 00 09 00
[15 ms] UsbSnoop - FilterDispatchAny(9a47afd2) : IRP_MJ_INTERNAL_DEVICE_CONTROL
[15 ms] UsbSnoop - FdoHookDispatchInternalIoctl(9a47b1ea) : fdo=8992a9d8, Irp=888c49d0, IRQL=0
[15 ms]  >>>  URB 3 going down  >>> 
-- URB_FUNCTION_GET_DESCRIPTOR_FROM_DEVICE:
  TransferBufferLength = 00000020
  TransferBuffer       = 89873cc8
  TransferBufferMDL    = 00000000
  Index                = 00000000
  DescriptorType       = 00000002 (USB_CONFIGURATION_DESCRIPTOR_TYPE)
  LanguageId           = 00000000
[19 ms] UsbSnoop - MyInternalIOCTLCompletion(9a47b126) : fido=00000000, Irp=888c49d0, Context=89951008, IRQL=2
[19 ms]  <<<  URB 3 coming back  <<< 
-- URB_FUNCTION_CONTROL_TRANSFER:
  PipeHandle           = 89673020
  TransferFlags        = 74f06e2f (USBD_TRANSFER_DIRECTION_IN, USBD_SHORT_TRANSFER_OK)
  TransferBufferLength = 00000020
  TransferBuffer       = 89873cc8
  TransferBufferMDL    = 8898c160
    00000000: 09 02 20 00 01 01 00 c0 32 09 04 00 00 02 08 06
    00000010: 50 00 07 05 83 02 40 00 00 07 05 05 02 40 00 00
  UrbLink              = 00000000
  SetupPacket          =
    00000000: 80 06 00 02 00 00 20 00
[19 ms] UsbSnoop - FilterDispatchAny(9a47afd2) : IRP_MJ_INTERNAL_DEVICE_CONTROL
[19 ms] UsbSnoop - FdoHookDispatchInternalIoctl(9a47b1ea) : fdo=8992a9d8, Irp=888c49d0, IRQL=0
[19 ms]  >>>  URB 4 going down  >>> 
-- URB_FUNCTION_SELECT_CONFIGURATION:
  ConfigurationDescriptor = 0x89873cc8 (configure)
  ConfigurationDescriptor : bLength             = 9
  ConfigurationDescriptor : bDescriptorType     = 0x00000002
  ConfigurationDescriptor : wTotalLength        = 0x00000020
  ConfigurationDescriptor : bNumInterfaces      = 0x00000001
  ConfigurationDescriptor : bConfigurationValue = 0x00000001
  ConfigurationDescriptor : iConfiguration      = 0x00000000
  ConfigurationDescriptor : bmAttributes        = 0x000000c0
  ConfigurationDescriptor : MaxPower            = 0x00000032
  ConfigurationHandle     = 0x00000000
  Interface[0]: Length            = 56
  Interface[0]: InterfaceNumber   = 0
  Interface[0]: AlternateSetting  = 0
[57 ms] UsbSnoop - MyInternalIOCTLCompletion(9a47b126) : fido=00000000, Irp=888c49d0, Context=89951008, IRQL=0
[57 ms]  <<<  URB 4 coming back  <<< 
-- URB_FUNCTION_SELECT_CONFIGURATION:
  ConfigurationDescriptor = 0x89873cc8 (configure)
  ConfigurationDescriptor : bLength             = 9
  ConfigurationDescriptor : bDescriptorType     = 0x00000002
  ConfigurationDescriptor : wTotalLength        = 0x00000020
  ConfigurationDescriptor : bNumInterfaces      = 0x00000001
  ConfigurationDescriptor : bConfigurationValue = 0x00000001
  ConfigurationDescriptor : iConfiguration      = 0x00000000
  ConfigurationDescriptor : bmAttributes        = 0x000000c0
  ConfigurationDescriptor : MaxPower            = 0x00000032
  ConfigurationHandle     = 0x888d14d0
  Interface[0]: Length            = 56
  Interface[0]: InterfaceNumber   = 0
  Interface[0]: AlternateSetting  = 0
  Interface[0]: Class             = 0x00000008
  Interface[0]: SubClass          = 0x00000006
  Interface[0]: Protocol          = 0x00000050
  Interface[0]: InterfaceHandle   = 0x88bcece0
  Interface[0]: NumberOfPipes     = 2
  Interface[0]: Pipes[0] : MaximumPacketSize = 0x00000040
  Interface[0]: Pipes[0] : EndpointAddress   = 0x00000083
  Interface[0]: Pipes[0] : Interval          = 0x00000000
  Interface[0]: Pipes[0] : PipeType          = 0x00000002 (UsbdPipeTypeBulk)
  Interface[0]: Pipes[0] : PipeHandle        = 0x88bcecfc
  Interface[0]: Pipes[0] : MaxTransferSize   = 0x00001000
  Interface[0]: Pipes[0] : PipeFlags         = 0x00000000
  Interface[0]: Pipes[1] : MaximumPacketSize = 0x00000040
  Interface[0]: Pipes[1] : EndpointAddress   = 0x00000005
  Interface[0]: Pipes[1] : Interval          = 0x00000000
  Interface[0]: Pipes[1] : PipeType          = 0x00000002 (UsbdPipeTypeBulk)
  Interface[0]: Pipes[1] : PipeHandle        = 0x88bced1c
  Interface[0]: Pipes[1] : MaxTransferSize   = 0x00001000
  Interface[0]: Pipes[1] : PipeFlags         = 0x00000000
[57 ms] UsbSnoop - FilterDispatchAny(9a47afd2) : IRP_MJ_INTERNAL_DEVICE_CONTROL
[57 ms] UsbSnoop - FdoHookDispatchInternalIoctl(9a47b1ea) : fdo=8992a9d8, Irp=88da7770, IRQL=0
[57 ms]  >>>  URB 5 going down  >>> 
-- URB_FUNCTION_BULK_OR_INTERRUPT_TRANSFER:
  PipeHandle           = 88bced1c [endpoint 0x00000005]
  TransferFlags        = 00000000 (USBD_TRANSFER_DIRECTION_OUT, ~USBD_SHORT_TRANSFER_OK)
  TransferBufferLength = 0000001f
  TransferBuffer       = 88c4b790
  TransferBufferMDL    = 00000000
    00000000: 55 53 42 43 90 4e d6 8a 24 00 00 00 80 00 08 ff
    00000010: 02 44 45 56 43 48 47 00 00 00 00 00 00 00 00
  UrbLink              = 00000000
[57 ms] UsbSnoop - FilterDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_QUERY_CAPABILITIES)
[57 ms] UsbSnoop - FdoHookDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_QUERY_CAPABILITIES)
[58 ms] UsbSnoop - FilterDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_QUERY_PNP_DEVICE_STATE)
[58 ms] UsbSnoop - FdoHookDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_QUERY_PNP_DEVICE_STATE)
[58 ms] UsbSnoop - FilterDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_QUERY_DEVICE_RELATIONS)
[58 ms] UsbSnoop - FdoHookDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_QUERY_DEVICE_RELATIONS)
[58 ms] UsbSnoop - MyInternalIOCTLCompletion(9a47b126) : fido=00000000, Irp=88da7770, Context=89951008, IRQL=2
[58 ms]  <<<  URB 5 coming back  <<< 
-- URB_FUNCTION_BULK_OR_INTERRUPT_TRANSFER:
  PipeHandle           = 88bced1c [endpoint 0x00000005]
  TransferFlags        = 00000000 (USBD_TRANSFER_DIRECTION_OUT, ~USBD_SHORT_TRANSFER_OK)
  TransferBufferLength = 0000001f
  TransferBuffer       = 88c4b790
  TransferBufferMDL    = 8979e558
  UrbLink              = 00000000
[165 ms] UsbSnoop - FilterDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_QUERY_DEVICE_RELATIONS)
[165 ms] UsbSnoop - FdoHookDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_QUERY_DEVICE_RELATIONS)
[165 ms] UsbSnoop - FilterDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_QUERY_DEVICE_RELATIONS)
[165 ms] UsbSnoop - FdoHookDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_QUERY_DEVICE_RELATIONS)
[166 ms] UsbSnoop - FilterDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_SURPRISE_REMOVAL)
[166 ms] UsbSnoop - FdoHookDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_SURPRISE_REMOVAL)
[240 ms] UsbSnoop - FilterDispatchPnp(9a47f45c) : IRP_MJ_PNP (IRP_MN_REMOVE_DEVICE)

Given the above information, I then saved the following as "/etc/usb_modeswitch.conf":

DefaultVendor = 0x106c
DefaultProduct = 0x3b03

MessageEndpoint = 0x05
MessageContent = "55534243904ed68a24000000800008ff024445564348470000000000000000"

The following command can then be used to "switch" the mode of the device so that it is recognized as a USB modem. Note that as documented on the USB_ModeSwitch page, most programs using libusb need to be run as root:

sudo /sbin/usb_modeswitch -c /etc/usb_modeswitch.conf

Here is the updated output of "/usr/sbin/lsusb -l":

Bus 001 Device 006: ID 106c:3715 Curitel Communications, Inc.
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               1.10
  bDeviceClass            2 Communications
  bDeviceSubClass         0
  bDeviceProtocol         0
  bMaxPacketSize0        64
  idVendor           0x106c Curitel Communications, Inc.
  idProduct          0x3715
  bcdDevice            1.00
  iManufacturer           1 PANTECH
  iProduct                2 PANTECH USB MODEM
  iSerial                 0
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength          113
    bNumInterfaces          4
    bConfigurationValue     1
    iConfiguration          0
    bmAttributes         0x80
      (Bus Powered)
    MaxPower              500mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         2 Communications
      bInterfaceSubClass      2 Abstract (modem)
      bInterfaceProtocol      1 AT-commands (v.25ter)
      iInterface              0
      CDC Header:
        bcdCDC               1.09
      CDC Call Management:
        bmCapabilities       0x03
          call management
          use DataInterface
        bDataInterface          1
      CDC ACM:
        bmCapabilities       0x0f
          connection notifications
          sends break
          line coding and serial state
          get/set/clear comm features
      CDC Union:
        bMasterInterface        0
        bSlaveInterface         1
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0010  1x 16 bytes
        bInterval              32
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass        10 CDC Data
      bInterfaceSubClass      0 Unused
      bInterfaceProtocol      0
      iInterface              3 Data Interface
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x82  EP 2 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040  1x 64 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x02  EP 2 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040  1x 64 bytes
        bInterval               0
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        2
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass       255 Vendor Specific Class
      bInterfaceSubClass      0
      bInterfaceProtocol      0
      iInterface              0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x84  EP 4 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040  1x 64 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x04  EP 4 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040  1x 64 bytes
        bInterval               0
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        3
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass         8 Mass Storage
      bInterfaceSubClass      6 SCSI
      bInterfaceProtocol     80 Bulk (Zip)
      iInterface              0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x83  EP 3 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040  1x 64 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x05  EP 5 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040  1x 64 bytes
        bInterval               0
Device Status:     0x0001
  Self Powered

At this point, I was finally able to establish a connection using PPP. This setting appears to even persist across reboots, as long as the USB device doesn't loose power. Following the examples on the USB_ModeSwitch page, I saved the following as "/etc/udev/rules.d/99-usb_modeswitch.rules" which will automatically run the above command whenever the device is detected:

SUBSYSTEM=="usb", ATTR{idProduct}="3b03", ATTR{idVendor}=="106c", \
        RUN+="/sbin/usb_modeswitch -c /etc/usb_modeswitch.conf"

Note that there is a "README" file in the "/etc/udev/rules.d" directory. Additional information can be found in the man page for udev ("man udev").

Accessing the Modem

The "cdc_acm" kernel module should now detect a connected device, and connect it to "/dev/ttyACM0".

Before making the Internet connection with PPP, the modem can be communicated with as a serial device. I'm not sure what the preferred method is for doing this under Linux, but the "screen" command (located in "/usr/bin") worked for me:

/usr/bin/screen /dev/ttyACM0

AT
OK
AT+GMM
UM175AL

OK
ATI
Manufacturer: UTStarcom communication Inc.
Model: UM175AL
Revision: D0700ALM01_5.226  1  [Mar 21 2008 06:00:00] [Jul 29 2008 15:10:28]
ESN: 0x83AF77E
+GCAP: +CIS707-A, CIS-856, CIS-856-A

OK
AT+CSQ
31, 0

OK

(If you don't recognize the above, the Hayes command set on Wikipedia may be a good starting reference.)

Signal Strength

The "AT+CSQ" returns the current signal strength as a value between 0-31, with higher being better. According to Ken Kinder, this number indicates the signal strength above -109 dBm in 2 dBm increments.

I know the Windows client software displays a regularly updated signal strength indicator, even while the connection is active. It would be nice to have this functionality available under Linux, too, but it doesn't look easy, if even possible.

I found one forum thread with one response on the topic: http://fixunix.com/ppp/62263-commad-mode-gprs-same-time.html. Pretty much, it seems that the data stream would have to be paused, the modem dropped to command mode, the "AT+CSQ" command issued and the value retrieved, then sending "ATO" to go back online. The forum response suggested that a modified pppd would be needed. I'm wondering if a "wrapper" tty couldn't be written that would be passed to pppd, and query the signal strength whenever the connection is idle. Alternatively, I'd be curious if the "diagnostic port" as shown in Windows would provide similar functionality without having to interrupt the data line? Maybe this is available by one of the other interface descriptors shown in the lsusb output (above)? Maybe "bInterfaceClass 10 CDC Data"?

Update 2008-10-08: Sysinternal's Portmon shows that under Windows, the The Alltel Wireless Connection Manager communicates with the "diagnostic port". Unfortunately, unlike the text-based "AT" commands used above, this communication looks completely binary. I will include some of the logging output here shortly. It also doesn't look to be easily decipherable, or to use any known standard. I've sent an email to UTStarcom, and can only hope for an unlikely helpful response.

To be continued...

Next up: Configuring PPP.

Shuttle K-4500-N2

(This post is the 2nd part of my Ubuntu Linux Router Upgrade Project.)

I chose Shuttle's KPC4500 model, purchased from Newegg as the K-4500-N2 (#N82E16883104036) for about $210.

Surprisingly, I currently have the only posted customer review for this product on Newegg.

Specs

While the K4500 certainly isn't a top-of-the-line system, it's definitely suitable for this task. Included is a 2.0 GHz 64-bit processor, 160 GB SATA hard drive, 10/100/1000 Mbps Ethernet, and integrated video & audio. It has 4 USB ports, as well as COM/serial, LPT/parallel and PS/2. Internally, there is room for a 2nd HD, with an 1 additional SATA and 1 IDE port available, as well as some extra USB headers.

The K-4500-N2 came configured with 512 MB RAM, which is shared by the on-board video card. The sharing can be configured in the BIOS for a dynamic (using DVMT) or fixed partition. While this should be plenty for the setup, maxing out the supported memory is relatively cheap - so I decided to do it right away instead of later. I couldn't find a common product with appropriate specs that was in the memory support list for the K45 on Shuttle's web site and that Newegg carried, but a Kingston 2x1 GB pair (Newegg # N82E16820134117, ~$30) is working well. While this required removing the supplied 512 MB chip, this had the added benefit of enabling dual-channel mode.

With a largest dimension of 11", it is space-saving. Though I haven't measured A/C usage, it also seems that it is also rather energy efficient - with only a 100W power supply, and very few extra components to power.

One peripheral to note as missing is an optical drive, though there is one available on the K4800 model. However, short of reinstalling the operating system, I rarely expect a need for one. It does support an external USB DVD drive I have without any apparent issues, and worked well for reinstalling Linux.

Operating System

I was not impressed with the supplied installation of Foresight Linux. Several menu options only produced errors - not a good first impression. Fortunately, it's free to begin with, so I had nothing to loose by simply doing a clean install of the latest Ubuntu x64 - which works great.

Other Thoughts

Especially without optimal viewing conditions, the power button on the front can easily be mistaken for a USB port. Considering how there are already spare USB headers on the motherboard and available space on the front panel, at least one USB port on the front panel would be nice for connecting flash drives, etc.

There's a clear spot on the back plate for a 2nd Ethernet jack, which could have been nice for use as a router. However, it's nothing that VLAN support and/or an external USB adapter can't resolve.

To be continued...

Next up: Getting Alltel's wireless card to work under Linux.

Ubuntu Linux Router Upgrade Project

As much as I've enjoyed using OpenWrt for my home networking, I've decided to "upgrade" to a fuller Linux distribution running on a more "traditional" computer rather than an embedded device.

Internet via Cell Phone

One factor encouraging this transition is that there is currently no cable or DSL Internet service to my current residence. To remain connected, I'm using Alltel's Wireless Internet service (3G 1xEV-DO). Unfortunately, the associated access devices are essentially modems that connect by either USB, PCMCIA, or ExpressCard - none of which can be directly connected to the standard Ethernet ports on my OpenWrt-enabled Linksys router.

There are a number of devices available to bridge mobile broadband to a home or other small LAN, otherwise known as cellular routers. One such device is the Linksys WRT54G3G. However, it seems specifically targeted towards Sprint/Nextel, and none of the listed supported data cards are offered by Alltel. Also, retailing at ~$130, it costs almost twice as much as my existing WRT54GL while not even offering the same quality 802.11 wireless.

Linux opportunity

While I don't consider myself a stranger to Linux, my use of it at home has been limited to OpenWrt and virtual machines. This will be an opportunity for me to run it natively and have an always-on instance to utilize as needed.

My choice of distribution for this project is the latest version of Ubuntu, 8.04 ("Hardy Heron"), based on its release schedule, stability, features, popularity, and availability of online community support. Version 8.10 ("Intrepid Ibex") is due at the end of the month (2008-10-30). Rather than waiting, I'll utilize the remainder of the month to experiment and polish my setup. My plan is to then hopefully repeat the same success with minimal effort after a clean install of 8.10 once released.

New Hardware

I considered using an old laptop for the task. Even though the screen has serious issues, I'd usually just have it connected to my KVM switch or use SSH. However, considering that I have a Gigabit network switch and matching link to my primary desktop, I wanted a device that would also match speeds.

To be continued...

I'll be following up with a number of posts. To be included:

• My choice of hardware
• Getting Alltel's wireless card to work under Linux
• Configuring PPP
• Enabling routing and NAT with iptables
• Setting up local DHCP and DNS services
• Additional challenges and resolutions

Google Sites: Limitations

As excited as I was to start posting content on Google Sites (formerly JotSpot), there are a number of issues / limitations that I think are show-stoppers for now.

Google Sites offers a number of advantages, including a wiki format and interface, and as part of its inclusion into Google Apps, can be hosted on a custom domain.

However, after working with a number of different wiki products, both free and commercial, Google Sites seems excessively simple, to the point of restrictive. One example is when creating a new page, a type of page must be chosen. There then seems to be no option for changing that page's type, short of deleting it and creating a new one.

Deleting pages, however, is what I see to be one of the biggest issue with the current version of Sites. While a page or attachment can be "deleted", either seems to be infinitely recoverable, with no option to "permanently delete". As discussed in this thread, Steven Hind ("Visionary") wrote:

The ability to recover deleted pages is absolutely necessary as Google Sites allows collaboration, and a collaborator may delete a page (unintentionally or deliberately) and the owner may need to get it back.

While that is definitely a valid use-case, site owners should at least be given the option to permanently purge / remove "deleted" items. There are usability reasons, such as difficulty in recreating pages with the same name. More seriously are the security concerns, e.g. if something accidentally gets posted that shouldn't have been. Granted, even if something disappears from Sites, it may still be obtainable through Google Cache or the Internet Archive: Wayback Machine. However, even these services properly respect robot exclusions.

Working with included page images is a similar but even more mysterious issue. Images inserted into a page without being included "as an attachment" seem to function as "hidden attachments". Even when removed from the page, they remain available for later repeated use, with no visible method of removal.

Searching for "delete page" on the Site's Help returns dozens of results, with many users sharing the same concern. Hopefully this is something Google will address in the near future.

In the meantime, I'm still looking for a decent solution where I can share some of my code and projects, as open source, but under "my control". While I'll continue to use dev.java.net, it's really only purposed for Java projects. I have at least one Microsoft .NET project that I'd like to share. Google Code and SourceForge.net are two other options I explored, but again, leave very little control left with the owner. Especially when built on Subversion, the Subversion Obliterate issue, it becomes practically impossible to make a removal when needed.

OpenOffice issues - Dictionary, Impress Hyperlink Wrapping

As I'm trying to move to OpenOffice.org, I just finished dealing with 2 issues under 3.0 beta 2 worth documenting.

English Dictionaries

After working with a previous OpenOffice.org version, I was surprised to see that the spell checking was apparently broken after installing the latest beta. The "English spelling and hyphenation dictionaries and thesaurus", version 20051213, appeared under the Extension Manager. Everything appeared in-order under Tools / Options / Language Settings.

I eventually found an answer buried in the OpenOffice.org Community Forum by juanon on 2008-08-06 (which I then echoed).

Basically, it appears that the current version of the "English spelling and hyphenation dictionaries and thesaurus" extension (version 20051213) is missing files en_US.dic and en_US.aff, which can be downloaded from http://ftp.services.openoffice.org/pub/OpenOffice.org/contrib/dictionaries/en_US.zip (linked to from the Dictionaries page on the OpenOffice.org Wiki.)

Hyperlink Wrapping in Impress

I was pretty impressed with OpenOffice.org's presentation product, Impress. It stands to compete with Microsoft PowerPoint. The one issue I was running into was getting Hyperlinks to wrap lines. I finally found an existing issue logged for this: 43494 - "hyperlinks should support word wrap".

Java Collections Listeners

Maybe I've been looking at various JavaScript frameworks, etc., for too long, but I'm surprised that there's really no positive way to listen for add/remove/update events within Java's Collections Framework.

Assumptions / Use Case:

1. Implement the Map interface. This allows for passing into other methods that only accept an implementation of Map, as well as providing a consistent and familiar API to other developers.
2. Add some additional processing, e.g.:
   • Disallowing certain keys or values from being inserted.
   • Keeping one or more associated maps or sets for performance, e.g. a map of lists when an entry needs to be obtained by value rather than key.
   • Extending HashMap into an "OrderedMap", where unique keys are still guaranteed, but the order of insertions is also kept.

Considerations

Clearly public interfaces are designed to be implemented, and public abstract classes are available to extend. Sun's "Custom Implementations" lesson in the Collections tutorial demonstrates this. As the lesson describes, there are several abstract classes available to serve as a starting point for a custom collection implementation. However, extending these to match the full functionality and efficiency of the existing concrete implementations is not a small effort, especially for Map implementations. The most significant features to note are the methods that return a "view" to a different part or representation of the map, e.g. keySet(), values(), and entrySet(). All are documented to return a view of the Map, such that changes in the view are reflected in the map, and vice-versa.

If that's not already a lot of extra work, consider the methods that are available on the returned view that should also remain functional, such as the iterator()'s remove() method, or worse, the List.listIterator() method. Sure, many of these are listed as "optional operations", which may simply throw an UnsupportedOperationException for simple implementations. However, this can be frustrating to developers attempting to use these methods, and some code may depend on these methods to function.

One solution to save a lot of work is to extend an existing concrete implementation, e.g. HashMap. However, there always seems to be some debate over extending a non-abstract class. My view is that if the class wasn't meant to be extended, it should be marked final or have less-than-public constructors. (As a compromise, the class could remain open to extension, while protecting various methods by marking them final.) java.util.Properties is one example of a public class that extends java.util.Hashtable, another concrete, public class.

Considerable progress seems possible on a HashMap subclass by simply overriding the put(K, V), remove(Object), and clear() methods. However, as described above, the "view" methods provide alternate access points to modify the underlying collection data, not all of which chain down to the overridden methods. This can quickly lead to an incomplete and buggy class.

My working solution

The best approach I've found is to follow the recommendation in the tutorial, and to extend AbstractMap. A child HashMap can then be held as an instance variable, with most of the AbstractMap methods delegated or proxied to it. For returning the "view" methods, they can be made "unmodifiable" by wrapping them in calls to the Collections.unmodifiable*(...) wrapper methods. While this may lead to some frustrations defined above, it provides a solid class that still properly implements the core collection methods.

I did find one Sun bug report that is a request for enhancement regarding this issue: 5078552 - "(coll) ChangeListener, VetoableChangeListener for Collections, Lists and others". It was submitted in July 2004, and hasn't yet received any attention since.

Gmail migration with IMAP and the Java Mail API

As I previously posted, Gmail has IMAP support. I've been using it for a while now. It's a great way to use Gmail with a a desktop email client such as Mozilla Thunderbird, as well as having offline access to your email.

IMAP also offers another significant feature to Gmail users: The ability to easily migrate messages between accounts - either between different Gmail accounts or from another source into Gmail. Google offers their own tool for this, but it's only available for commercial and educational accounts - not for their personal accounts like the ones I have.

I attempted to run my transfer using the configured accounts in Microsoft Outlook, very much like Ashish Mohta described on his blog. Things were looking good until I started getting the following error:

The current command did not succeed.
The mail server responded: Unable to append message to folder (Failure).

The first thing I suspected was an Outlook issue, so I switched to Thunderbird but got the same result. I suspected I may be hitting some sort of transfer limit, so I tried downloading everything from my first account to a local folder first. That worked, but then I still wasn't able to upload from the local folder into the second account. A Google search shows a number of other people having the same issue. Google help has two (1, 2) answers that primarily blame formatting incompatibilities, but as indicated by by other results in the Google search, this definitely appears to be the result of exceeding some unposted transfer limits.

There didn't seem to be any way to configure either Outlook or Firebird to "slow things down", without manually moving only a few messages at a time - which would be tedious and prone to error. Additionally, I guessed that both clients were performing additional tasks behind the scenes that were making the issue worse, such as needlessly refreshing folders.

JavaMail API

I figured that my best chance of getting everything how I wanted it was to script the transfer. For this, I turned to the JavaMail API. I've used this before, but mostly just for sending messages over SMTP. Surprisingly, use with Gmail is even listed in the JavaMail API FAQ.

Connecting was quite simple. The only property that I needed to pass into Session.getInstance(...) was a value of "imaps" for "mail.store.protocol". For simplicity, I then used the Store's connect(String host, String user, String password) method to connect.

To open the "All Mail" folder in Gmail, either .getFolder("[Gmail]").getFolder("All Mail") or .getFolder("[Gmail]/"All Mail") works.

I did my transfer in two separate steps - the download than the upload. This was mostly because I also wanted a local, backup copy to keep. By iterating through each Message from the Folder's getMessages(), the Message's writeTo(OutputStream) method allows for easily saving the entire message - including headers and attachments - in MIME format. Similarly, the messages can then be recreated using the MimeMessage(Session, InputStream) constructor, then imported into an IMAP folder using the appendMessages(Message[]) method.

Alternatively - though I haven't tried it - opening a session to each account and using the Folder's copyMessages(Message[], Folder) method looks like it would also be convenient. Note that the "this" Folder is the source, and the Folder parameter is the destination. However, while the method is convenient for copying multiple messages at once, it may need to be coded for only one message at a time, so a pause can be inserted between transfers to account for the Gmail restrictions described above.

To slow down the transfers to keep the Gmail servers happy, after each download or upload I simply called Thread.sleep(long millis). I started out at 2,500 ms, but had this down to 500 ms by the time I finished without any further issues.

Sent Mail

Another issue I had was that I wanted/needed to re-populate my "Sent Mail" folder. This may not always be an issue, but due to any number of reasons in my case, Gmail no longer recognized any of my sent messages as "sent". While Gmail seems to treat "Sent Mail" just like any other "label", the Gmail interface doesn't provide any options to add or remove messages from it, short of deletion. However, it can be modified through IMAP. While copying in each new message, I checked to see if the getSender() method matched one of email addresses. If so, it should be copied into the "Sent Mail" folder rather than "All Items". Note the difference in using the sender rather than the "From" attribute from getFrom(). The later would typically include items that were actually received, probably from an automated sender, where it was only made to appear "from" the same email address.

Originally, I simply imported all mail into the "All Mail" folder, then copied it into "Sent Mail" as needed. However, passing the same MimeMessage into Folder.appendMessages(Message[]) then again to copyMessages(Message[], Folder) for "Sent Mail" results in a ClassCastException as copyMessages expects a IMAPMessage variant of a message that already exists in the IMAP store. The solution would be to obtain a proper reference to the inserted message to copy. The Message[] com.sun.mail.imap.IMAPFolder#addMessages(Message[]) method would allow for exactly this, but it appears that Gmail doesn't currently support the required UIDPLUS extension from RFC 2359. The only apparent remaining option would be to recall getMessages(), then iterate through and find the message matching the one inserted, probably by getMessageID().

Even after all this work, I still ran into an issue with the Gmail web interface showing all my messages in my "Sent Items" as "To: me", rather than the actual recipients. The issue wasn't related to anything with the transfer, but in my Gmail account settings. I had assumed that Gmail calculated "me" as mail sent to any address listed in the "Send mail as" list. Instead - and probably more reasonable - it is almost the opposite. If the email is sent to any address not listed in the "Sent mail as" list, it is assumed by Gmail to be "To: me".

Other tools

After I successfully finished my migration using the above method, I did find some other tools that appear to solve many of the same issues. 2 that I found off of related Google searches are imapsync and IMAPSize. (I've not downloaded or tried either of these.)

PayPal's refusal to recognize my marriage

In an apparent show of security theater, PayPal is essentially refusing to honor one of the benefits of marriage.

Having been married for just over a year now, Sarah and I have been rather blessed. However, while I don't think either of us even seriously considered against having her last name changed, it is definitely not a stress-free process. I share in her frustration as I observe the difficulties.

So far, PayPal has definitely posed the most difficulty. While PayPal never requires any identity verification when signing up for an account - which can be a security concern in itself - they require a faxed copy of a driver's license or marriage certificate for a name change. Especially today, most people don't even have a fax machine at home, or even one readily accessible.

More frustratingly are PayPal's policies against joint accounts. They only allow each bank / credit card account to be registered to one PayPal account - without exception. While this policy may be useful for detecting potential security issues, there should be an exception process for joint accounts, such as the ones that Sarah & I share, both at the bank and with our credit cards. Neither our bank nor any of our credit card companies had any real issues with converting our accounts into join accounts. Why should PayPal make things any more difficult? So unless we open and maintain a balance in a 2nd checking account just for PayPal, one of us may as well just close our PayPal account.

An alternative option would be to allow Sarah & I to share the same PayPal account. Unfortunately, PayPal doesn't allow that either. So now I'll be sharing my PayPal username and password with Sarah, which is arguably worse than any of PayPal's above security concerns. Though I trust her completely, this means that whenever the password needs to be changed, we have to communicate it to each other, which would normally result in it being written down someplace. This also has the disadvantage in only one of our names being displayed on the account. More significantly, the person not listed probably won't be able to contact PayPal support, etc.

Following are two responses I got from PayPal after emailing them with the above concerns:

...

Unfortunately you can only have one name on one PayPal account and you can't use the same financial information on another active account.

...

Thank you for contacting PayPal with your concern.

The short answer is "no;" we cannot have the same financial information on more than one account. Please read below for a more detailed explanation.

Our security procedures limit credit card registrations to one PayPal account at a time. If your credit card was previously registered to another PayPal account, you will need to regain access to that account in order to use the credit card on a different PayPal account.

...

Yahoo! User Interface Library (YUI)

As I previously posted, I've switched to the Yahoo! User Interface Library (YUI).

Some key links:

• Official web page: http://developer.yahoo.com/yui/
• SourceForge project: http://sourceforge.net/projects/yui
• Forums: http://tech.groups.yahoo.com/group/ydn-javascript/
• Blog: http://yuiblog.com/

I think YUI is a great library for many of the same reasons I had liked Ext JS:

• Respect for the global namespace and base language, without polluting it with numerous function and variable names. (See my JavaScript namespace function, Respecting the JavaScript global namespace and JavaScript Closures posts.)
• Capable DOM manipulation. (See Dom.)
• Capable event listeners / event management. (See Event.)

YUI includes a number of useful and versatile UI components. The ones I use the most are DataTable and Calendar.

A non-UI component that I've been quite pleased with is the YUI Loader Utility. Instead of including the entire YUI library across an entire web site, it allows for safe, easy, and relatively efficient partial loading, including reliable, sorted loading of dependencies.

Unlike Ext JS, there is excellent, comprehensive documentation available by component. Not only the API docs, but "getting started" pages complete with numerous examples and notes as well. In addition to all this, there is also the forum and blog.

There are a few bugs and feature requests that I've submitted on YUI's SourceForge trackers that may be of interest.

I have some additional YUI-related notes to share that will follow as future posts. Just watch for the "Yahoo UI Library" label.

Ubuntu Hardy Heron under VMware

I just got the latest Ubuntu Linux distribution (8.04, "Hardy Heron") working quite well under VMware.

I found a few issues during the process specific to running under VMware that required some attention. Google helped me find the solution for the first two both on Peter Cooper's blog:

Mouse scroll wheel

First, the scroll wheel on the mouse wasn't working. Not so surprising, as I've typically had various mouse problems in the past working with Linux versions under VMware. The solution Peter found was to add the following highlighted three lines in "/etc/X11/xorg.conf":

Section "InputDevice"
Identifier "Configured Mouse"
Driver "vmmouse"
Option "CorePointer"
Option "Device" "/dev/input/mice"
Option "Protocol" "ImPS/2"
Option "Buttons" "5"
Option "ZAxisMapping" "4 5"
EndSection

For the original with full details, see How To Enable Mouse Wheel Scrolling in Ubuntu Hardy on VMware Fusion (Peter Cooper, 2008-04-26). While written for VMware Fusion, the solution works equally well for the free VMware Server.

VMware Tools

Next were some issues getting the VMware tools to install properly. The basic GUI vmware-toolbox worked, but no precompiled modules for the file sharing or network driver could be found for the current kernel. When it attempted to compile appropriate ones, gcc would error out with "conflicting types" messages.

This solution is a little more lengthy. Peter's instructions work quite well: How to Install VMware Tools on Ubuntu Hardy 8.04 under VMware Fusion (Peter Cooper, 2008-04-26).

The only alteration I made is using the most recent version of open-vm-tools from SourceForge - 2008.05.15 as of this writing. Again, this works just as well for VMware Server as for VMware Fusion, other than that VMware Server doesn't have support for the "Shared Folders" feature.

Unnecessary Services

Another optimization that can be made under VMware is disabling a number of services that don't function / simply don't exist under VMware.

Services can quickly be disabled using the "Services" applet under the System / Administration menu. The services I disabled:

• Bluetooth device management (bluetooth)
• CPU Frequency manager (powernowd)
• Power management (acpid)
• Power management (apmd)

Boot Details

Another non-VMware related change I wanted was to see more details during the boot process, which I was used to from previous versions.

Part of the challenge was finding the proper search terms. Ubuntu uses "Usplash" (see also USplash on help.ubuntu.com, and somewhat related to "Splashy") as the default "Bootsplash".

The best write-up I found related to this is Quickzi: Get rid of the Ubuntu splash screen during boot (author unknown, 2007-10-27, foogazi.com).

Essentially, to adjust the process permanently (at least until the next kernel upgrade), the file to edit is "/boot/grub/menu.lst". Find the "kernel" line in the config you are booting with and remove the "quiet" and/or "splash" options. My preference is to leave "splash" and just remove the "quiet", which scrolls the boot progress on the lower portion of the splash. (I wouldn't recommend removing "splash" and leaving "quiet".)

"Open Terminal" on desktop/context menu

Previous Linux distributions I had included a "Open Terminal" option on the desktop/context menu, and I found myself using it quite frequently and missing it in Ubuntu. To obtain this functionality, install "nautilus-open-terminal".

JavaScript namespace function

As a follow-up to my "Respecting the JavaScript global namespace" post, here's a function that can be used to quickly and easily create namespaces:

String.prototype.namespace = function(separator){
  var ns = this.split(separator || '.')
  var o = window;
  for(var i=0, len=ns.length; i<len; i++){
    o = o[ns[i]] = o[ns[i]] || {};
  }
  return o;
};

I'm generally not a fan of modifying the base JavaScript language through prototypes - but in this case, it's rather useful. Alternatively, this could be modified to take in the namespace string as the 1st parameter, and the function assigned to some other object (to respect the global namespace.)

See also:

• Namespacing made easy (kangax, 2008-01-30, thinkweb2.com)
• JavaScript Namespaces (Remy Sharp, 2008-02-04, remysharp.com)
• [1944624] Allow YAHOO.namespace To Create Arbitrary Namespaces (sourceforge.net/projects/yui/)

Unlike the "Namespacing made easy" post, this solution doesn't require Prototype. This solution also prevents overwriting existing objects and returns the created "namespace".

Update (2008-05-13):

The concern was raised that the "len" variable in my for-loop is being created in the global scope / "namespace", which would contradict the entire purpose of this post! :-)

Fortunately, according to developer.mozilla.org, all variables declared after "var" are created within the same scope - and if declared within a function, the scope is that current function.

Here's my "test case". Use Firebug. First check that neither the "i" nor "len" variables already exist. Then run the following code, and note that neither variable is created in the global scope.

(function(){
  for(var i=0, len=0;false;){};
})();

Compared to this version, which does populate "len" into the global scope:

(function(){
  len = 0;
  for(var i=0;false;){};
})();

Maybe surprisingly, this makes "len" local again, even though it is assigned to before being declared with "var". This is because JavaScript does not have block scope.

(function(){
  len = 0;
  for(var i=0, len=0;false;){};
})();

This general practice is documented at http://www.prototypejs.org/api/array, where it is also explained how caching the length property is an aid to performance. (Though for the stated purpose of creating namespaces, which will probably only be handling single-digit lengths for the majority of the cases, the savings will be practically zero.)

Farewell, Ext JS

Though I was previously impressed with the Ext JavaScript library, it's time to move on.

Everything I previously wrote about Ext JS is still true. It is a very capable, comprehensive, well-written JavaScript library. Unfortunately, the management of the project doesn't at all seem to live up to the same standards.

There were always a few minor ongoing issues, which still exist as of this posting. One is the lack of bug / feature request trackers, e.g. Bugzilla. Another is the restriction of read-only access to the Subversion source code repository to only premium (paying) members.

The main issue, however, is the license change that was announced with the release of Ext JS 2.1 on 2008-04-21. Ext JS was always dual licensed under both the LGPL license as well as a commercial license. This meant that anyone was free to do almost anything with the library. The only recognized restriction was using Ext JS in a "software development library or toolkit", when the commercial license would have been required.

Introducing the GPL

With the license change, the dual licensing model still exists, but uses GPL instead of LGPL. I and many other users didn't see this as an issue for our uses. Unfortunately, there was little more than confusion on Ext's "License Change?" forum, which as of this writing topped 650 posts and 61,000 views.

I generally support open source. Even corporate development commonly makes uses of open source software, licensed under either LGPL, GPL, or dozens of other "open source / free" licenses. The one restriction with using GPL-licensed software is that if the parent application is to be redistributed, the source code usually has to be redistributed as well. I don't have a problem with this for any of my personal applications.

While many businesses work with proprietary applications that simply are not meant to be shared, they can even utilize GPL-licensed software as long as their application is not "redistributed". For companies who only develop and host a web site rather than developing and selling software, it would seem that there would be not redistribution and no GPL requirements. Even Microsoft uses a GPL-licensed JavaScript library on their web site, and they don't appear to be under any requirement to release any source code.

The GPL license makes a distinction between propagate and convey:

To "propagate" a work means to do anything with it that, without permission, would make you directly or secondarily liable for infringement under applicable copyright law, except executing it on a computer or modifying a private copy. Propagation includes copying, distribution (with or without modification), making available to the public, and in some countries other activities as well.

To "convey" a work means any kind of propagation that enables other parties to make or receive copies. Mere interaction with a user through a computer network, with no transfer of a copy, is not conveying.

From all definitions / explanations I've seen, simply hosting a web site (utilizing Ext or whatever) would be classified as propagation, not conveying. To convey, one would have to do something similar to zipping up the web site and redistributing it for others to host as well (regardless of terms).

Ext Indecisions

This is where everything gets muddy really fast with Ext.

On 2007-03-27, one of the lead Ext developers agreed with the statement that "For the purposes of Ext, hosting an application is not considered distribution, so the license does not apply".

Less than a month later, the Ext team then claims that by simply hosting an Ext-enabled web application, the application is being conveyed as well as propagated. They claim that all the source for all related JavaScript code in the application would then also be covered by the GPL, and must have the source code be made available. Since JavaScript is an interpreted rather than a compiled language, the argument could be made that this requirement is already fulfilled - by running the application, the source is already provided. However, the same lead developer further stretches a claim that not only must the sources be made available for the client-side code, but portions of any server-side code as well.

Please understand - the Ext authors probably have the right to distribute Ext JS under whatever license they choose. The issue is that they are claiming the GPL license, but requiring different terms. As I and others have suggested, it seems what the Ext team really wants is the AGPL license. From http://www.fsf.org/licensing/licenses/:

Its (the AGPL) terms effectively consist of the terms of GPLv3, with an additional paragraph in section 13 to allow users who interact with the licensed software over a network to receive the source for that program. We recommend that developers consider using the GNU AGPL for any software which will commonly be run over a network.

Again, if Ext, LLC does not wish to license Ext JS under either the LGPL, the GPL, or the AGPL, they are free to create their own license. Just don't claim to honor the GPL and then dispute the terms.

To clarify, I don't see any issues using Ext JS in an "open source", GPL'd application. However, there is so much confusion over the license terms that I don't see how any business could risk playing this license game, short of paying for the commercial license (which still leaves unanswered questions and concerns). Additionally, why choose the commercial license when other genuinely-free alternatives are readily available?

Related Posts

Here are a few other related posts on this subject:

• ExtJS: When open source is not open at all (Matt Asay, 2008-02-25, cnet.com)
• Stephan Schmidt on stephan.reposita.org:
  • John Resig on ExtJS, the GPL fiasco and open source community style (2008-04-28)
  • More cluelessness from ExtJS lead developer Jack on the GPL issue (2008-04-24)
  • When people don’t understand GPL and LGPL - or - ExtJS is history (2008-04-22)
• ExtJS 2.1 AJAX Library Switches To GPL (2008-04-22, slashdot.org)

Switching to YUI

As I posted to the Ext "License Change?" forum, I've since switched to the Yahoo! User Interface Library (YUI). Beyond being BSD licensed, YUI has:

• equally capable event listeners/management,
• official & structured bug / feature request / patch trackers on SourceForge,
• no forum threads restricted to premium/paying members,
• commitment to make the SVN repository publicly available,
• and fixes / inclusions of many of the bugs / enhancement requests reported here against Ext JS without action.

I was using Ext JS only for 2-3 odd bits of functionality I found in it almost a year ago. I didn't see them in YUI at the time, either because they've since been added, or that I just missed them the first time around (more likely).

Over the past few weeks, I've found that working with YUI performs better, and is actually easier and faster to work with. (Details in a future post.)

YUI isn't the only alternative out there. The Dojo Toolkit is one other that I've been considering.

MarkUtils-Web: ZipServlet and CompressFilter

"MarkUtils-Web" is an initial collection of web-related Java utilities that I wrote and am releasing on ziesemer.dev.java.net for public use.

The source code, a compiled .jar, and generated JavaDocs are all included in "MarkUtils-Web.zip". (Download)

At the moment, there are 2 components:

• ZipServlet: A HttpServlet to serve content from within a specified Zip (or Jar) file.
• CompressFilter: A Filter to compress outgoing content, using either GZIPOutputStream or DeflaterOutputStream.

ZipServlet

My idea for ZipServlet came while working with Ext JS in a J2EE application. Typically, when including Java components, one or few JAR libraries are imported into the application. No real equivalent really exists for web content.

For full support of Ext JS 2.0.2, including the several required .js scripts along with .css and image resources, this amounts to 1,818 files across 174 folders, totaling 21.9 MB. While Ext JS provides some other options, including custom builds, each poses dependency and other challenges. Even including only the core 2 .js files, 1 ext-all.css, and possibly 196 image files still seems a bit excessive.

Including any large number of files in a project can quickly lead to difficulties, particularly with source control. While utilizing the extracted layout would easily allow developers to patch and modify the included project, such practice is typically not a good idea and would likely lead to additional upgrade complexities as previous changes would have to be found and migrated. Just the upgrade itself would require the replacement of potentially hundreds of files!

My more ideal solution was to simply include Ext JS's distribution .zip into my project's WebContent/WEB-INF folder. My ZipServlet then serves any Ext-related files requested by the client directly out of the .zip file. While this leads to additional uncompression that can require additional CPU time, I've not encountered any performance issues. Client-side caching is usually "on" by default. Additional server-side caching could additionally be used. Another option would be to re-package the .zip file without compression. (The size of the .zip file would be larger, but it would still be a single .zip file.)

Shown below are the relevant sections out of my web.xml file, allowing ext-2.0.2.zip to appear just like a folder stored on my web server. For example, with the configuration listed below, "<contextRoot>/ext-2.0.2/ext-all.js" will return Ext JS's primary JavaScript file:

<servlet>
 <servlet-name>ext-2.0.2</servlet-name>
 <servlet-class>com.ziesemer.utils.web.ZipServlet</servlet-class>
 <init-param>
  <param-name>file</param-name>
  <param-value>/WEB-INF/ext-2.0.2.zip</param-value>
 </init-param>
 <init-param>
  <param-name>zipPrefix</param-name>
  <param-value>ext-2.0.2/</param-value>
 </init-param>
</servlet>

<servlet-mapping>
 <servlet-name>ext-2.0.2</servlet-name>
 <url-pattern>/ext-2.0.2/*</url-pattern>
</servlet-mapping>

See the included JavaDocs for further details on these options.

ZipServlet follows the standards described in RFC 2616 as applicable. In particular, this implementation supports last modified checking and partial content requests.

Similar to Ext JS, some other libraries I've included this way include Firebug Lite, The Yahoo! User Interface Library (YUI), and The Dojo Toolkit.

CompressFilter

The idea behind CompressFilter is explained fairly well in "Two Servlet Filters Every Web Application Should Have" (Jayson Falkner, 2003-11-19, ONJava.com). The sample code included on ONJava.com is a bit lacking and requires some improvement, but to be fair, it is over 4 years old.

Notable features of my CompressFilter include:

• Following the standards described in RFC 2616 as applicable, particularl