Gmail has IMAP!

After hearing the promising news on Slashdot, I'm excited to learn that my Google Mail / Gmail account now has IMAP support!

More information is available on a Google Help Page, Getting Started with IMAP for Gmail.

I've been hosting my own email and server for a while now, mainly because I refused to live with one of the disadvantages typical of webmail - not being able to access email when disconnected from the Internet. With support to synchronize messages between Gmail and any IMAP client, including Microsoft Outlook and Mozilla Thunderbird, this is looking to be a very promising replacement to my current setup!

TimeTool L

Background

I'm finally hoping to solve one deficiency in my audio/video work - sync!

Synchronization is something taken for granted in a closed system, e.g. a consumer camcorder, where the audio and video is recorded simultaneously within the same device. Unfortunately, this simplicity also poses a number of limitations, such as being able to record additional channels of audio.

There are many flavors of synchronization in the professional media industry, e.g. the LTC flavor of SMPTE timecode, MTC (MIDI timecode), Word Clock, and ADAT Sync. In the end, they all serve the same purpose - to keep multiple devices moving along at the same rate. When you see a video where people's lips don't match up to their voice, that is a synchronization problem.

The Problem

My problem has been that I'm pretty well equipped for making digital audio recordings, being able to record from at least 10 analog sources (multi track). My MOTU 828mkII does this nicely, along with a multitude of other features. Unfortunately, I don't do enough video work to justify the expense of a camcorder of the same professional grade.

My current camcorder is a MiniDV Sony DCR-TRV30, which was at the high-end of the consumer line when I purchased it. (Unfortunately, it's not 3CCD, but again, I couldn't justify the additional price.) It does well for the purpose of recording digital video, but like most consumer camcorders, recording decent audio is nearly impossible.

This camcorder and most similar models have a video (S-Video and composite) and stereo audio (RCA) line-in/out jacks of relatively decent quality. Unfortunately, the RCA audio line-in jacks are only available for use in VCR mode. I.E., you can't use them to record external audio while taping. There is an external microphone jack, reportedly even stereo. Again, unfortunately, it is meant to be simple, not to provide even near-professional results. It has an auto-gain control which can't be adjusted or disabled. While it may be desired for a novice recording in a noisy room, it is far from ideal for hooking into a professional audio system. (Any attempts result in an unacceptable amount of noise and buzzing.)

Getting a camcorder with professional XLR audio connections would require something like the Canon XL2, at a price tag around $4,500. Even then, it's limited to 2 channels of audio, or 4 in slightly lower quality (from 16-bit to 12-bit audio).

The Goal

The idea is to use my MOTU 828mkII to record the audio, and just let the camcorder record the video. I should then be able to take both and lay them back together on a timeline in the computer for production to a DVD, etc. Unfortunately, without anything in place to keep synchronization, the audio and the video eventually drift out of sync. While maybe not noticeable for a 5 minute clip, it definitely becomes apparent after recording for 1-2 hours. I have previously done this and manually corrected for the drift, but it is time consuming and a bit of guesswork, without having a continuous point of on-screen reference to sync against.

The MOTU 828mkII and many other computer audio digital interfaces have the ability to use an external sync source. The 828mkII accepts every type of sync source listed above! The problem is getting the sync off the camcorder. Many professional cameras provide and accept external sync, but not even the $4,500 Canon XL2. Instead, something like the $9,000 Canon XLH1 would be needed.

Fortunately, my DCR-TRV30 has a LANC interface jack on it. However, up until now, I had no way to make use of this, as it is a different protocol and not directly usable by the 828mkII. Enter the TimeTool L from Fisher Robotics Inc. It is designed to read from the LANC interface and produce SMPTE LTC timecode, which should then be usable by the 828mkII.

Caution / Update: The legitimacy of the Fisher Robotics business is in question, as reported by the Anonymous comment below. An open letter has been posted to the rec.arts.movies.production.sound newsgroup, started 2009-09-23. Please use your best judgment before placing any orders or doing other business with Fisher Robotics.

I just placed my order for one of these $250 gadgets. I look forward to testing everything out once I receive the unit, and will post my results here. If successful, it will provide me with the professional functionality normally only available on $9,000+ units for only a small fraction of the cost.

Update: (2007-10-29)

I received my TimeTool L after 2-day USPS priority mail on the 23rd. I've since tried getting my Motu 828mkII to lock-on sync to my camcorder through the TimeTool with no success - but the I'm hopeful that this should be quickly and easily resolved in the coming days.

I've been in fairly regular email communication with Fisher Robotics, and they reply quite promptly to my emails. On the 24th, I was told that "it does appear that this is an anomaly - however it can (and will) be fixed shortly...", though no details were provided.

In the meantime, I've continued to do some further investigation. I recorded waveforms of the SMPTE output of both the TimeTool L, as well as SMPTE generated by my Motu 828mkII. My 828mkII is able to successfully lock-on to a playback of the recording from the 828mkII, but not from the TimeTool L, even when the audio levels were matched.

I then wanted to decode and see exactly what was being sent in these waveforms. The SMPTE bit codes are described on Wikipedia. I just needed an easy and quick way to read the Manchester-type encoding into the individual bits. Staring at the visual representation of the waveform, counting long and short periods, and recording with a pencil sure didn't seem like a workable approach, with 80 bits for every frame of data, or 2,400 bits for 1 second. (I figured I'd need at least 10 seconds to verify proper codes, and ideally 30+ minutes to verify that drop-code was working to-spec.)

In a candidate for another post, just over 100 lines of Java code later (making use of the Java Sound API), I had a program to read the recorded .wav files and output the raw bits along with decoded representations of hours/minutes/seconds/frames and the additional SMPTE codes.

What I found is that the SMPTE timecode from my 828mkII is properly counting from frames 0-29, for 30 frames every second, except for frames 0-1 being dropped in the first second of every non-10th minute, which matches the specification for 29.97 fps drop-frame timecode. The TimeTool, on the other hand, consistently counts from frames 0-28, resulting in only 29 frames every second. I assume my 828mkII fails after properly expecting and waiting for the missing frame 29.

Hopefully this is what Fisher Robotics was already aware of, and is working on for their next firmware release. For good measure, I just emailed them these details. I'm looking forward to their response and a hopeful fix, and will post back with an update once I know more!

Update: (2007-10-30)

I'll post more details soon, but after applying the 1.002 firmware update (just released), success! I think it's a keeper!!

Update: (2007-11-03)

My only remaining problem was adjusting the audio level for the Motu 828mkII to accept on its SMPTE input jack. Fisher Robotics kindly posted some additional specifications addressing this. Photo album of my assembly.

.innerHTML and .innerText slow performance under Internet Explorer

Background

Here's another one of those situations where you write some HTML/JavaScript using a standards-compliant web browser such as Firefox, then have to correct any issues that appear under Microsoft's Internet Explorer.

In this particular case, I was working on a JavaScript-based search form that performed special search functions on data in an IFRAME, supporting restrictions on location and the use of regular expressions. This involved traversing the HTML DOM tree, very much like the high-performance approach described in Java: Iterating over XML DOM Nodes, then using the .innerHTML and/or .textContent || .innerHTML properties to further analyze - you guessed it - the text content.

.textContent and .innerText

The .textContent / .innerHTML properties are a bit of an issue themselves. .textContent is "standards-compliant", defined by the W3C. As Microsoft admits on their MSDN page, .innerText has "no public standard that applies to this property".

To make matters worse, many developers seem to find obscure ways of detecting which property to use, often by using browser detection or other similarly flawed methods like detecting for other browser properties - e.g. using document.all - which aren't even related to the issue. It doesn't help that even the most popular JavaScript libraries don't provide a helper method for this issue. IMO, ".textContent || .innerText" is the best fix. This works due to the conditional logic in JavaScript. If .textContent returns a value that can be evaluated to true, it will be used as the result. Otherwise, .innerText will be returned. 0, -0, null, false, NaN, undefined, or the empty string ("") all evaluate to false. In Internet Explorer, .textContent returns undefined, so .innerText is used.

The Performance Issue

An almost equal performance issue with both .innerText and .innerHTML quickly became apparent under Internet Explorer. Basic testing showed that iterative use of these properties took on the order of 100x longer than the .textContent / .innerHTML properties under Firefox. (Firefox seems to work almost instantaneously, so increasing the length of the iteration quickly yields an exponential growth in the difference. Additionally, these properties seem to be internally cached by Internet Explorer, such that calling the property repeatedly on the same node results in almost immediate results after the initial call - important for testing this.)

Unfortunately, there don't appear to be any good (and free) JavaScript profilers for Internet Explorer. (Firebug and the "Venkman" JavaScript Debugger extensions exist for Firefox.)

Below is some sample JavaScript to demonstrate the issue. It will generate the specified number of <span> elements within a <div>, then time itself as it calls the selected property on each node.

On my machine, Firefox consistently completes any of the function selections in less than 500 ms (1/2 second) for 100,000 items. Internet Explorer consistently takes > 30,000 ms (30 seconds) except for the "null" selection (a dummy function that simply "returns null"), which performs as fast as Firefox.

Items: Function: node.innerHTML node.textContent || node.innerText null

 

---

 

(As much as I'd like to include this as an attached page, Blogger doesn't currently support non-image file attachments.)

Work-around

The only work-around I know of at the moment is to "write your own" method to retrieve the text content. .nodeValue may work for text nodes. Issues include making sure a text node is being referenced, and concatenating the values from all the children (and their children...) if needed, which matches the functionality of the above built-in properties. Fortunately, in the controlled data I'm working with, this is not the case so a simple call to .nodeValue is working... for now.

Unable to round-trip characters in XML and JavaScript under Firefox

(See the submitted bug report to Mozilla: Bug 398272 – Unable to round-trip tab character using XMLHttpRequest and XMLSerializer)

I just finished working on a Firefox extension, after battling with Firefox's XML support. Don't get me wrong - it's a rather good implementation, just with a few issues such as this one.

I was attempting to use an XML file as backup storage for some of Firefox's configuration options when making other changes from within the extension. Some of these values consist of tab-separated values. Unfortunately, the current XML support doesn't allow for a successful round-trip of these tab characters between a save using XMLSerializer and a read using XMLHttpRequest. Part of the reason seems to be that XMLHttpRequest normalizes the attributes on read, and the tabs are read-back as spaces. It seems that the best resolution would be for the XMLSerializer to somehow escape the tabs (and possibly other similar characters) with XML character entities, e.g. 	 for the tab character, which assures that the characters are read back as intended.

Fortunately, Firefox is open-source and is constantly improving, so hopefully this will be resolved in the near future.

My work-around at the moment is to store the data within CDATA elements, rather than XML attributes, which properly maintains all whitespace - including the tab characters. Another option would have been splitting the value and storing the individual parts as separate attributes or elements. However, this would have resulted in keying to the specifics of the values, while I want to avoid such strong coupling and work with the values more generically. (This way, if the values later change to use spaces, commas, or some other separation, no change is required for my extension.)

Respecting the JavaScript global namespace

Here's my plea to all JavaScript developers, whether for web pages, Firefox extensions, or otherwise: Please respect the global namespace.

I've not seen a lot written about this, but as fast as new uses of JavaScript are appearing on web pages, and with how many extensions are being developed for Firefox, function and variable naming collisions and conflicts are a real potential issue.

In a web page, all functions and variables share the same global namespace - that of the Window object. In a Firefox extension, all declared functions and variables become properties of the ChromeWindow.

(Note that in JavaScript, a function is basically just another variable. function someFunction(…){…} and var someFunction = function(…){…}; are identical, and someFunction can be passed and reassigned just like someString from var someString = "Hello, World!". As such, the same issue pertains to both functions and other variables.)

(Don't) prefix your variable names

One primitive approach to preventing problems is to prefix variable names, as discussed at http://forums.mozillazine.org/viewtopic.php?t=129485. While this may slightly help to lessen the chance of naming collisions, it is not the best solution. There is no system in place to prevent 2 authors or code libraries from choosing the "abc_" prefix. The best alternative to such an approach would be to choose a prefix that is more guaranteed to be unique, such as a utilizing an owned Internet domain name, such as with Java packages. For example, I might prefix all my JavaScript code with "com_ziesemer_", to separate it from other JavaScript libraries I may utilize either now or in the future. But there are still better options!

JavaScript is an object-oriented language

Contrary to some popular beliefs, JavaScript is very much an object-oriented language. As such, I feel that an object-oriented solution is in order.

One of the best posts I've found about this is from Douglas Crockford, a senior JavaScript Architect at Yahoo!, and well known for introducing JavaScript Object Notation(JSON). In "Global Domination" on the Yahoo! User Interface Blog, "Global variables are evil". His post is later referred to by Eric Miraglia, also of Yahoo!, in "A JavaScript Module Pattern", including a useful code example.

Another posting I found, "Namspacing your JavaScript", I'm listing more for the comments that follow than the original post. (Choosing "DED" as a top-level variable name is a prime example of choosing a bad name.)

To clearly state the desired trend, most of the best examples and methods make use of JSON and/or JavaScript closures.

Package it up

Further following the appearance of Java packages, I recommend something like the following, again, utilizing JavaScript as an OO language:

if(!com) var com={};
if(!com.ziesemer) com.ziesemer={};

com.ziesemer.myPackage = {
  name: 'visitor',
  location: 'someplace unknown',
  alertHello: function(){
    alert('Hello, ' + this.name
      + ", from " + this.location + '!');
  }
}

(See JavaScript Closures for more context.)

The if checks at the top prevent for wiping out the objects if they already exist, and allow for either creating each level as a new object, or extending an existing object at that level.

Update (2008-05-12): See my JavaScript namespace function for a function to handle all of this in one call, e.g.
namespace("com.ziesemer.myPackage").alertHello = { /* … */ };

Update (2009-12-02): See my proposal to add the above namespace function to the base ECMAScript / JavaScript language.

I am also honored to see that this page itself has been linked to from the Mozilla Firefox Add-on Review Process.

The other advantage of this design over prefixing is that each package can easily be aliased, e.g.:

var czmp = com.ziesemer.myPackage;
czmp.name = 'Mark';
czmp.location = 'Wausau, WI';
czmp.alertHello();

Note that such use should only be done from within another function or another scope limitation, otherwise "czmp" would become a member of the global namespace, which is exactly what this is trying to prevent. An alternative would be to use the JavaScript with statement: (However, this is only meant as a possibility, not a recommendation. I would nest the above in a closure function instead. See Douglas Crockford's "with Statement Considered Harmful" for details.)

with(com.ziesemer.myPackage){
  name = 'Mark';
  location = 'Wausau, WI';
  alertHello();
}

Checking Firefox and extensions

To see how populated/polluted your Firefox global namespace is, follow the instructions at JavaScript debugging in Firefox extensions to setup the JavaScript Debugger/Venkman to inspect the browser window. Then run the following two commands at the debugger's Interactive Session window to get an idea of all the variables created at the global level:

var vars=[]; for(var v in this){vars.push(v);} vars.sort();

vars.length;

On my current Firefox install with less than 20 extensions installed, this yields more than 1,500 variables! (439 of them begin with "webdeveloper_" from the Web Developer extension.)

Update (2009-12-02): See Mats Bryntse's "Global is the new private". It shows the number of global variables used by 30+ popular JavaScript frameworks, from YUI (currently 1), to Microsoft VirtualEarth (currently 880!). It also allows for analysis of any given script (available somewhere online). If that wasn't enough, it even calculates namespace collisions between chosen frameworks.

Update (2008-12-19):

This is in response to Arturo's comment. I'm replying here so that I can properly format my code.

I'm not sure how you got "hello content". Given your code above, I'd expect "hello undefined".

I think avoiding the JSON syntax would probably be the best bet with what you're attempting to do.

Also, beware that your example declares "com.ziesemer.myPackage" twice, which is unnecessary.

Here's what I would recommend, using closures:

if(!com) var com={};
if(!com.ziesemer) com.ziesemer={};

com.ziesemer.myPackage = function(){
  var pub = {};
  pub.name = "visitor";
  pub.greeting = "hello " + pub.name;
  pub.alertHello = function(){
    alert(pub.greeting);
  };
  return pub;
}();

com.ziesemer.myPackage.alertHello();

You can tweak this to not assign everything to "pub", essentially making some things private. You could also somewhat combine both methods, utilizing the "with" within the closure - though that could be somewhat confusing.

BTW, please don't use "com.ziesemer" - that namespace is registered to me. :-)

JavaScript debugging in Firefox extensions

I've been working on a few Firefox extensions. One of the most difficult things I found was a good way to debug JavaScript code.

Starting out by reading MDC's "Setting up extension development environment" page is a great first step. Enabling the javascript.options.showInConsole and javascript.options.strict options are a good help. Also enabling nglayout.debug.disable_xul_cache and browser.dom.window.dump.enabled doesn't hurt anything, though I don't consider these options as useful.

Of the listed "helpful extensions", JavaScript Debugger, a.k.a. "Venkman" is definitely the most useful. It took me a few minutes how to set the scope to the browser window: Find the "Open Windows" view, right-click on "Browser Window", and select "Set as Evaluation Object".

DOM Inspector

The DOM Inspector, sometimes a.k.a. "DOMi" is great for inspecting the Chrome layout.

For Firefox 2.x, this is not a separate extension. Choose to install with a Firefox custom install. For Firefox 3.x, it is available at https://addons.mozilla.org/en-US/firefox/addon/6622. (See also bug 339229 and bug 271812 for details on the move.)

The DOM Inspector can also be used for inspecting the JavaScript of the ChromeWindow in a pinch, though it's not readily apparent how. (The lack of any real documentation doesn't help much.) "Mossop" was kind enough to help me out at irc.mozilla.org#extdev by pointing me to the defaultView property. In summary:

1. Open DOM Inspector
2. Select the "#document" nodeName from the left-hand pane.
3. Select to inspect the "Javascript Object" using the drop-down above the right-hand pane.
4. Expand "Subject", then "defaultView".

JavaScript can then be executed on defaultView or any children by right-clicking on the desired property, and choosing "Evaluate JavaScript...".

Unfortunately, in Firefox 2.x, the property tree is not sorted. This was fixed with bug 226819, but isn't included until the 2.0.0 version for Firefox 3.x. (The defaultView was broken in Firefox 3.0a8 in that it won't expand. I had reported this in bug 398285.)

Console² and Chrome List are also useful and recommended. The Extension Developer's Extension may prove useful to some - though I'm not currently one of them.

Firebug

I do think that Firebug is a great extension, but it's main use is mostly focused on debugging and analyzing loaded web pages within the main browser window, rather than the Firefox browser itself (i.e., the Chrome). An interesting hack is to open chrome://browser/content/browser.xul as a web page. This essentially opens an instance of Firefox from inside the browser window. (See "Fun with Firefox Chrome URLs".) Unfortunately, this can cause a number of side effects, particularly with various add-ons. (Read more at Issue 194 - Firebug should be able to inspect chrome and "Firebug and Firefox".)

Stack Traces

The other issue I had was finding stack traces, or at least the source file and line number of an error. Most of this seems to come naturally for debugging web pages with Firebug, but can be trickier to track down within a Firefox extension.

As listed above, upgrading to the Console² is a good upgrade to and even fixes a few bugs with Firefox's built-in "Error Console". Unfortunately, it still doesn't display all of the available debugging information from a JavaScript error. See the FAQ at http://console2.mozdev.org./ as well as Mozilla bugs 228205 and 125612 for details.

The first thing to check is that the only unhandled objects being thrown are based on the global Error object. Unlike Java, where only an instance of a class extending Throwable can be thrown, JavaScript allows any instance of an object to be thrown. I.E., throwing a string, e.g. throw "This is an error!" is completely valid. However, only objects built from the Error object contain the populated details, e.g. fileName, lineNumber, and stack.

Even with Console² installed and throwing Error objects, I still find the following helpful - at least until Firefox/Console² supports the Error's stack by default:

try{
  // Do, or call to, all your code here.
}catch(e){
  try{
    if(e.stack){
      Components.utils.reportError(e.stack);
    }
    // Show the error console.
    toJavaScriptConsole();
  }finally{
    throw e;
  }
}

This will actually generate seemingly duplicate messages to the console: first one with the additional detail, then the one that would be generated by default - but I find that it's the safest way, and still allows for showing non-Error objects that were thrown by the default method.

While I'm on the topic, before continuing work on your next extension, please also read Respecting the JavaScript global namespace.

Updated: 2007-10-09, 2008-05-13 - Information on DOM Inspector

Notepad++

I just found a new favorite text editor: Notepad++.

I've been doing quite a bit of JavaScript coding lately, and the desire for a good code editor quickly becomes apparent.

The two main features I was looking for were capable syntax highlighting and code folding. (While other languages such as Java and C++ are commonly supported, many other editors seem to lack good support for JavaScript.)

Notepad++ does both of these nicely, does them by default, and does them well. Additionally, it is free and open source, with a GNU GPL license.

Other useful features included are:

• Search and replace using Regular Expressions.
• "Find in Files". Handy for searching for a string or pattern across multiple files and directories.
• Block-select mode / rectangular block selection. Select and manipulate columns of characters, without selecting the entire row(s).
• Bracket matching, auto indentation, and compiler integration.

Notepad++ even has somewhat stated support for Linux, through the use of Wine: details here and/or here.

Read more from Wikipedia: Notepad++ and Comparison of text editors.

My 2nd choice would be SciTE. While it has native support for both Win32 and X11 for Linux, the default options aren't the most favorable, and most preferences have to be set through a config file only, since the GUI options are a bit limited. (Read more on Wikipedia: SciTE.)

Both Notepad++ and SciTE seem far better than two other editors I had been using: TextPad (Wikipedia) and EditPlus (Wikipedia). Neither are free nor open source. Neither support code folding. EditPlus supports basic syntax highlighting by default, as does TextPad, but only after adding a syntax definition file.