I bought VMware Workstation several years back. Used virtual machines quite a lot in developing and testing software. The company I work for licensed the enterprise version of VMware, at least in part due to developer experience using VMware. A few of the VMs currently hosted under the enterprise version of VMware started out running under VMware on my desktop.

A few years back – well after EMC acquired VMware – I got the distinct impression that the desktop version of VMware was somewhat neglected. I was losing increasing amounts of time to locked-up VMs, messed up keymaps, and even main desktop lockup. This provided an incentive to try Sun’s VirtualBox, which proved more reliable for my usage.

Oracle gained ownership of the VirtualBox software with the Sun acquisition. Is there any strong reason for Oracle to offer and develop the “free” version of VirtualBox, or will the desktop version of VirtualBox suffer the same apparent neglect as did the the desktop version of VMware?

Of late I was reviewing the documentation we sent to our customers with the last software release, as we are rolling up to a small update in the near future, and a larger release a bit later. In the documentation we send to customers, we have a prominent declaration of support for running under VMware. This was an easy declaration to make, as developers used VMware heavily in development and testing. We knew our product ran under VMware.

Now this is somewhat less true. Certainly the more elaborate test setups (mine) are all run under VirtualBox. I expect things all still work under VMware, but I personally no longer have a basis to make that assertion. For the next release, we will likely add an statement of support for VirtualBox.

As to the obvious bit I had missed:

Many of our customers are still not using virtualization software. Generally customers move from a physical machine to a VM at the time of a major version upgrade. At the next major version upgrade, we will offer a recommendation for VirtualBox.

Oh. Right. I had forgotten about that multiplier. Selling enterprise virtualization to our customers represents many more opportunities than any in-house usage at our development outfit. Nevermind my misgivings – offering free software to developers is about the best advertising Oracle could buy.

Does Oracle know this?

One surprise in the experience was the difficulty in getting the domain controller (and DHCP setup) properly working. Admittedly, setting up a domain controller is a somewhat unusual task. As a rare task perhaps not so important to Microsoft as a process that needs the user experience optimized. Still, given I have long been well-acquainted with how this all works at a network level, I ended up wasting a surprising amount of time trying to find the right knobs to turn. (Much searching with Google to get a key Microsoft product to work … the irony.)

Oddly, all the nice GUI interfaces to administering a domain server actually make the process more obscure than the equivalent activity on Unix (where I am accustomed to configuration kept in text files). I do now better understand my past conversations with network administrators who did not know how to do (what I considered) simple tasks with Microsoft’s domain controller … and why they often got it wrong.

Since my aim was testing, I wrote programs to populate Active Directory with large/variable numbers of users and groups, and to cleanup after. Though you could make direct Active Directory or LDAP calls, I find Microsoft’s ldifde utility is quite useful for this purpose. Note that Active Directory uses tombstones to support distributed replication (which is good), and that scripted changes will leave behind many tombstones – which may perturb your testing. You can limit the scope of the impact by changing the tombstone lifetime – the period of time tombstones are kept before cleanup – from the default down to 2(?) days. (Finding and changing that single setting ate up the large part of day.)

(Had a flashback to the old original text-based Adventure game. “You are in a maze of twisty passages, all alike.”)

Another approach is to take a snapshot of the VM when Active Directory is in a “clean” state, and restore the snapshot later. In principle this could be simpler/faster than the scripted restore (using delete operations fed via ldifde). Found a quite thorough article on virtualizing Active Directory, and ran across the USN ROLLBACK problem.

To aid in distributed replication, Active Directory keeps a count of local changes (the USN number) to track replication of remote changes. All quite usual and normal for this problem domain. Given that Active Directory has been around for a decade, you would expect the obvious use-cases to be covered, but not quite….

Turns out if you distribute changes among more than one Active Directory controller, you have a problem in the not-unusual case of taking a backup (or a snapshot) and restoring it later. Thus the many items returned by the above web search, including developers at Microsoft.

Um … what?!? Seems to me this is a pretty common case. That Microsoft would have got this wrong in the first version back in 2000 is only slightly disappointing. That this is still a problem a decade later is very surprising indeed.

Given that I may for future test purposes need to add a second domain controller VM, this looks to be another problem to avoid.

At first, there was no word from Oracle on a real fix … then I received this email.

Hi Preston,
Good to hear from you. Have I told you what a fine group of folks there is at Oracle Java? No? That’s because there isn’t.
[snip]

Anyway I ended up talking with some people in Oracle (our SES in DC called the president of Oracle) and several people later a VP of security called me. Anyway the fix will contain 3 implementations low, medium and high security and he claims it will take a significant amount of work in the application software to implement any of them. The release is expected by September. The gotcha is the VP told me that they will only release the JRE to licensed JRE users. I told him the JRE has always been a free download and he responded with “free download not free use”. In order to use the JRE you must have a support contract in place and it appeared to him the DoD was a couple million licenses out of compliance. Apparently it was only free until they came up with this Java for Business thing they have now. In my experience it’s a very Oracle response. Quite the opposite to my dealings with Microsoft.

I guess time will tell….
[snip]

(Emphasis is mine. All identifying information removed.)

Well, we have an answer. Yikes.

To be clear, I believe Oracle is entitled to do whatever they want to make money from Java. Oracle bought Sun, and now they own Java. Sun had a different approach, but Sun failed. Also what Oracle does in the long term may or may not be different from what the above message indicates.

On the flip side, I have to re-consider whether it makes sense to write code for the Java platform. If Oracle is fragmenting the Java platform, and changing the cost to customers … using Java may no longer make sense. This may be the point where I start planning to move all future work off Java and away from the JVM.

Of course, there is open-source Java, but I am not convinced there is the critical mass of interest required to make open-source Java a viable option. I would very much like to be wrong on this point, but for now, I am skeptical.

Is Google’s Dalvik VM potentially a viable option? Might IBM’s investment in Java translate into backing for open-source Java? Is it time to move off the Sun/Oracle JVM entirely?

Time to re-visit choices made long ago….

When was the last time you used a computer with a floppy disk drive? When did you last save a file to a floppy disk?

Seems rather past time for a new icon.

Funny bit is that the square-with-notched-corner shape of a floppy looks rather like an SD memory card. Still not a good choice for file-save, as more likely use is to copy files from the memory card.

Checked to see if Microsoft had any good ideas. Office 2007 uses a image of a floppy for “Save…” and “Save As…” – adding a pencil writing on the floppy label (an antique behavior) for the “Save As…” icon. Internet Explorer changes “Save As…” so the pencil is writing on the sliding metal cover that protected the magnetic film (not really a good idea).

Went searching through GUI Style Guides looking to see if Microsoft (nope) or Apple (nope) had an updated replacement. Along the way ran across the style guide sections on designing for differing display resolutions. Display resolution is measured in DPI (dots-per-inch) and display sizes are measured in inches. Because both Microsoft Windows and Apple Macintosh software were originally developed on the west coast of the United States, entire generations of programmers and users have learned to think about screen sizes in inches.

Have you noticed that all the HDTVs sold have sizes measured in inches, not in Metric System units?

This is odd, as only the United States still holds firmly to the old “British Imperial System” of measures, and HDTVs are not made here. Even odder, in countries that do use the Metric measures, the HDTV sizes are still expressed in inches. (Don’t believe me? Check out the Amazon site in France – home of the Metric System.)

If only the French had developed Windows … then the Metric System would have had a chance.

I expect WordPress to be insecure. Looked at the source code early on. Like most PHP applications, the potential attack surface is very large.

Will be a bit before things are entirely in order. (Ick. Using a stock WordPress theme.)

Sun was always a bit of a mixed bag. Some of their work was absolutely brilliant, and some – especially with software – was amazingly dumb. Never did really understand why this was the case. (Lots of theories, of course, but no certain grasp.)

Oracle is a very different mixed bag. The Oracle database is both a very solid piece of work, and old not-quite-irrelevant to the present and future (an intentional over-simplification.) The Oracle database is core to Oracle the company. On the other hand, Oracle (the company) is also heavily invested in applications and software that cluster around the Oracle database. Much of this software is written in Java (for very logical reasons). This would seem to guarantee the future of Java (as a platform).

But … the Oracle-core has a diminishing (if very fat) future. That fat future can stretch out quite a while – beyond obvious reason – as the IBM mainframe market proved.

As a developer, I find that I crossed a threshold some time back, and am impatient with Java when I can express more elegant solutions (to complex problems) in Javascript. (Yes, really – Javascript. May have something to do with early heavy exposure to Lisp in school.) Hearing that C# incorporates lambdas … is C# a better base than Java (with appropriate wariness of stealth Microsoft patents), or rather an extraneous step before bundling a Javascript interpreter?

Do you trust Oracle?

Use of multi-column lists

One of my favorite annoyances is multi-column lists. When they first appeared in the early 1990′s, multi-column lists were cool. Add in user-sortable and re-sizable columns, and your application could offer better eye-candy than the competitors.

The usual problem with multi-column lists was that the columns came up the wrong size. Some applications were smart enough to auto-size columns – but not all – and auto-sizing was not always what the user wanted. The next logical enhancement was to save user preferences for column sizes, so that the next time the application came up, the user’s prior manual adjustments to column width were preserved. If the most-useful column widths change with different data, the users spend a lot of time manually fiddling with column widths.

As an example, we have the (really lame) Services view in Microsoft Windows. When this first appeared (in the mid-1990′s) the columns both (1) came up the wrong size, (2) in the wrong order (another topic), and (3) did not save user preferences. At the time, I figured this was a work in progress, and the Microsoft folk would fix this in a couple years.

The screenshot below is from Windows 7, so … apparently not.

The Eclipse IDE uses multi-column lists for compiler error messages (and the like). Eclipse might be my current-favorite IDE, but this is one of my least-favorite aspects.

Note that basically all the columns are the wrong size. The error message text is most important, and is too small, with only a fraction visible. That makes any less-important column too-large, by definition. The column with the most excess whitespace – “Type” – is one of the least important. The “Path” column occupies a lot of space, but is of at best secondary importance.

The most important bit of the “Problems” panel is tiny.

Note also that in the above screenshot, the overhead elements in the panel eat up about half the vertical space – far more space than the most-meaningful message texts.

This is not efficient … on a large 1920×1200 panel. On smaller panels, things are much worse.

Presenting program texts

The main payload in an IDE is the display of program text. An IDE may put up supplemental panels to help discover structure and aid navigation within the program, but the central bit is the program text. In Eclipse only a fraction of the screen is usable for the display of program text.

Current screens are more than wide enough for pretty much all program texts, but those same texts are far taller than they are wide. As screens are far shorter than program texts, you want to make the best use of the vertical space available. Only about 2/3 of the screen height is text, in the typical example below.

With Eclipse you can “maximize” the text panel, but there is still substantial vertical space lost to overhead elements.

Note also that while there is a shortage of vertical space, there is an excess of unused horizontal space.

This is not efficient … on a large 1920×1200 panel. On smaller panels, things are much worse.

Alternate presentations

To make the best use of scarce vertical space – especially on smaller panels – you want to excise vertical overhead. Program texts tend to be dense on the left edge, and sparse on the right. For the display of error messages or navigation aids, if you place panels on the right, there is good chance you will overlap little or no program text. If you use fly-away panels, you can quickly view even very long lines of program text on the smallest-current laptop displays.

There is a simple/clever way around the presentation of variable-sized columns – concatenate all the variable size bits into one column. You can keep the fixed sized bits as columns (or not). If the resulting text is too wide for the available column space, allow the text to wrap. Excise the less-useful fields, and present on explicit user action (like hovering or clicking on the row).

Note that too-wide message texts can be hard (or slow) to read. (Ever lose track of the line you are on when looking across a very wide display?) There is a good argument for limiting the width of message texts to about the width of a typical piece of paper. (Funny how that works out – common paper width tends to be about the efficient maximum for text width.)

Combine the notions for maximal program text and complete-but-constrained message texts, and you might end up with something like this simple IDE mock-up. (Note this is very much a quick-and-dirty first iteration.)

At 1920×1200 – on a 24″ desktop panel.

(Best to pull this up in Google Chrome, as only Chrome does a really good maximum size full-screen. Works in Safari. Renders in Firefox, though event handling is wrong. IE is pretty much hopeless. As this is a mockup, my aim was not to make it work across web browsers.)

Note that as the screen size drops, the use of screen space becomes radically better than conventional IDEs.

At 1440×900 – the resolution of my 17″ laptop panel.

(Imagine the vertical overhead of Chrome as a part of the text-space.)

At 1280×800 – the resolution of common basic laptop panels.

Note that on this smallish laptop we are getting both more vertical program text, and more complete message texts, than on my big 1920×1200 panel using Eclipse.

At 1024×600 – the resolution of a netbook panel.

With the fly-away message panel toggled off, even a netbook screen shows a good amount of program text.

Topic for later – navigation

After using Eclipse for several years, I have become pretty comfortable. Oddly, I have come to realize that in one aspect – navigation – Eclipse is inferior to my old use of Emacs with ctags/mkid/calls/grep. Under the hood, the functions for introspecting program texts are a dream compared to what I had twenty years back … but navigation still feels a bit clunky.

The next exercise will be to try and map out what is different, and take a stab at how to do better.

Most FastCGI backends seems to be written for non-multiplexed connections. (Much simpler, so understandable.) The IIS FastCGI connector apparently does not support multiplexed connections.

Writing a FastCGI backend that allows for multiplexed connections. Would be a waste of time if not supported by a frontend, or if existing frontends are buggy.

(Not really expecting a response, but have to state the question.)

The HTML Working Group at W3C is … far too much noise. The HTML5 “standard” is going to be a bloated monster, and there is no chance I can change that. Time to stop the pretense of trying.

Not that all the work is bad, or that there is any shortage of well-intentioned folk. What the group lacks is any sense of minimalism, and enough strong voices able to say “no”. What we will get is going to be even harder to digest than HTML4. This is sad. Future developers are going to have an even harder time, for no good reason.

The wildcard here is that the mainstream browser implementations may not follow all the half-thought ideas thrown in by the HTML5 Working Group. No idea how this will work out.

At the core HTML is pretty damn simple – or could be. HTML4 got stuffed with a bunch of half-thought notions, most of which have since proved of no value, and were ignored by developers. Ideally we would learn from experience, omit the fuzzy disused bits, and trim HTML down to the useful core. There are well-known (though not universally known) means to achieve this aim. This is not going to happen.

I cannot keep up with the herd of Energizer-bunnies eager to make their mark, and with too-limited experience.
Time to stop pretending.

The use of UTF-8 on the web is common and increasing. Lots of data comes in as UTF-8, and inefficiency in UTF-8 data handling is going to have pretty pervasive impact.

On the flip side, the creators of UTF-8 did a good job. There is nothing really complicated about the UTF-8 format, and processing is simple.

So I was surprised (or rather shocked) to find in an earlier experiment that Java performed UTF-8 conversion slowly. In fact, I was able to write a faster UTF-8 decoder in Java than the stock decoder. This is just plain wrong. Conversion between encodings is a primitive/simple operation best written in C/C++ and run as native code (and this is the sort of processing where C/C++ is probably always going to be much faster than Java).

There is a problem in that malicious external parties can send oddly-encoded UTF-8, and bypass simple-minded malware detection software. Ordinary ASCII characters can coded as an alternate multi-byte sequences, and simple scanners miss the alternate encoding.

This is a problem. There is a simple solution. One of a set of principles I adopted a long time ago is “convert at the edges”. If you have data coming in from an untrusted source, then you perform conversion and validation at the “edge” where the data is first received.

In the case of UTF-8 coming from an untrusted source, to make all later processing simpler, you must recode to eliminate any alternate encodings. This is quite simple, as recoded UTF-8 will always be the same size or smaller, and so can be done in-place. The prior experiment measured the cost of UTF-8 recoding. Looks like we can drive a 1-Gbit network link at full speed (with efficient code), while recoding the entire contents. Since UTF-8 data usually represents a smaller portion of traffic, and since other processing tends to take the larger part of the load, there is no reason to not perform recoding on any UTF-8 data coming from an untrusted source.

Combine “convert at the edges” with UTF-8 recoding, and we lose the basis for the requirement in RFC 3629 for detection of “illegal” UTF-8 code sequences. In addition we allow all downstream processing to be simpler and more efficient … and we also can be tolerant of imperfect upstream software. (Yes, I am going to invoke Postel, again.)

The basis is good (secure processing with untrusted sources), but the requirement for detection of “illegal” sequences is not necessary and (most definitely!) not optimal.

Example of UTF-8 recoding (from String.cpp).

void UTF8::String::recode() {
    // Iterate until all UTF8 characters are normalized.
    // UTF8 in canonical form can only be smaller, so work in-place.
    char* p1 = pBuffer;
    char* p2 = pBuffer;
    char* pEOS = pBuffer + nContent;
    while (p1 < pEOS) {
        int c = 255 & *p1++;
        if (c < 0x80) {
            *p2++ = c;
            continue;
        }
        if (c < 0xE0) {
            c = (31 & c) << 6;
            c |= 63 & *p1++;
        } else if (c < 0xF0) {
            c = (15 & c) << 12;
            c |= (63 & *p1++) << 6;
            c |= 63 & *p1++;
        } else if (c < 0xF8) {
            c = (7 & c) << 18;
            c |= (63 & *p1++) << 12;
            c |= (63 & *p1++) << 6;
            c |= 63 & *p1++;
        } else if (c < 0xFC) {
            c = (3 & c) << 24;
            c |= (63 & *p1++) << 18;
            c |= (63 & *p1++) << 12;
            c |= (63 & *p1++) << 6;
            c |= 63 & *p1++;
        } else {
            c = (1 & c) << 30;
            c |= (63 & *p1++) << 24;
            c |= (63 & *p1++) << 18;
            c |= (63 & *p1++) << 12;
            c |= (63 & *p1++) << 6;
            c |= 63 & *p1++;
        }
        if (c < 0x80) {
            *p2++ = c;
        } else if (c < 0x800) {
            *p2++ = 0xC0 | (c >> 6);
            *p2++ = 0x80 | (63 & c);
        } else if (c < 0x10000) {
            *p2++ = 0xE0 | (c >> 12);
            *p2++ = 0x80 | (63 & (c >> 6));
            *p2++ = 0x80 | (63 & c);
        } else if (c < 0x200000) {
            *p2++ = 0xF0 | (c >> 18);
            *p2++ = 0x80 | (63 & (c >> 12));
            *p2++ = 0x80 | (63 & (c >> 6));
            *p2++ = 0x80 | (63 & c);
        } else if (c < 0x4000000) {
            *p2++ = 0xF8 | (c >> 24);
            *p2++ = 0x80 | (63 & (c >> 18));
            *p2++ = 0x80 | (63 & (c >> 12));
            *p2++ = 0x80 | (63 & (c >> 6));
            *p2++ = 0x80 | (63 & c);
        } else {
            *p2++ = 0xFC | (1 & (c >> 30));
            *p2++ = 0x80 | (63 & (c >> 24));
            *p2++ = 0x80 | (63 & (c >> 18));
            *p2++ = 0x80 | (63 & (c >> 12));
            *p2++ = 0x80 | (63 & (c >> 6));
            *p2++ = 0x80 | (63 & c);
        }
    }
    nContent = (int) (p2 - pBuffer);
}

UCS (Unicode) to UTF8 conversion, and the reverse, when efficiently coded in C++ clocks in well above 100MB/s on current generation CPUs. If you are getting something much less – enough to be a problem – then there are questions you should ask. The following run spans 1 to 6 byte UTF8 encodings.

Recoding a UTF8 string to a normalized form is a bit slower at 122-87 MB/s.

preston@athena:~/workspace/json-c$ time Release/json
base 00000010 :  229.9 MB/s UCS to UTF8 conversion
base 00000010 :  253.7 MB/s UTF8 to UCS conversion
base 00000010 :  122.0 MB/s UTF8 recode
base 02000010 :  196.8 MB/s UCS to UTF8 conversion
base 02000010 :  177.1 MB/s UTF8 to UCS conversion
base 02000010 :  100.6 MB/s UTF8 recode
base 04000010 :  173.8 MB/s UCS to UTF8 conversion
base 04000010 :  157.1 MB/s UTF8 to UCS conversion
base 04000010 :   89.2 MB/s UTF8 recode
base 06000010 :  174.0 MB/s UCS to UTF8 conversion
base 06000010 :  158.5 MB/s UTF8 to UCS conversion
base 06000010 :   89.2 MB/s UTF8 recode
base 08000010 :  174.0 MB/s UCS to UTF8 conversion
base 08000010 :  154.5 MB/s UTF8 to UCS conversion
base 08000010 :   89.6 MB/s UTF8 recode
base 0a000010 :  174.1 MB/s UCS to UTF8 conversion
base 0a000010 :  156.5 MB/s UTF8 to UCS conversion
base 0a000010 :   89.8 MB/s UTF8 recode
base 0c000010 :  174.0 MB/s UCS to UTF8 conversion
base 0c000010 :  155.1 MB/s UTF8 to UCS conversion
base 0c000010 :   89.6 MB/s UTF8 recode
base 0e000010 :  174.0 MB/s UCS to UTF8 conversion
base 0e000010 :  158.8 MB/s UTF8 to UCS conversion
base 0e000010 :   87.4 MB/s UTF8 recode
base 10000010 :  170.8 MB/s UCS to UTF8 conversion
base 10000010 :  158.2 MB/s UTF8 to UCS conversion
base 10000010 :   89.7 MB/s UTF8 recode
base 12000010 :  174.0 MB/s UCS to UTF8 conversion
base 12000010 :  158.8 MB/s UTF8 to UCS conversion
base 12000010 :   86.5 MB/s UTF8 recode
base 14000010 :  171.5 MB/s UCS to UTF8 conversion
base 14000010 :  153.9 MB/s UTF8 to UCS conversion
base 14000010 :   87.1 MB/s UTF8 recode
base 16000010 :  172.1 MB/s UCS to UTF8 conversion
base 16000010 :  158.1 MB/s UTF8 to UCS conversion
base 16000010 :   87.5 MB/s UTF8 recode
base 18000010 :  172.1 MB/s UCS to UTF8 conversion
base 18000010 :  158.2 MB/s UTF8 to UCS conversion
base 18000010 :   86.9 MB/s UTF8 recode
base 1a000010 :  171.3 MB/s UCS to UTF8 conversion
base 1a000010 :  158.2 MB/s UTF8 to UCS conversion
base 1a000010 :   86.5 MB/s UTF8 recode
base 1c000010 :  169.5 MB/s UCS to UTF8 conversion
base 1c000010 :  158.5 MB/s UTF8 to UCS conversion
base 1c000010 :   86.5 MB/s UTF8 recode
base 1e000010 :  173.0 MB/s UCS to UTF8 conversion
base 1e000010 :  157.8 MB/s UTF8 to UCS conversion
base 1e000010 :   86.1 MB/s UTF8 recode
base 20000010 :  173.1 MB/s UCS to UTF8 conversion
base 20000010 :  158.4 MB/s UTF8 to UCS conversion
base 20000010 :   87.2 MB/s UTF8 recode
base 22000010 :  173.1 MB/s UCS to UTF8 conversion
base 22000010 :  158.2 MB/s UTF8 to UCS conversion
base 22000010 :   88.2 MB/s UTF8 recode
base 24000010 :  173.3 MB/s UCS to UTF8 conversion
base 24000010 :  158.8 MB/s UTF8 to UCS conversion
base 24000010 :   87.6 MB/s UTF8 recode
base 26000010 :  173.8 MB/s UCS to UTF8 conversion
base 26000010 :  158.1 MB/s UTF8 to UCS conversion
base 26000010 :   87.9 MB/s UTF8 recode
base 28000010 :  172.0 MB/s UCS to UTF8 conversion
base 28000010 :  158.7 MB/s UTF8 to UCS conversion
base 28000010 :   87.6 MB/s UTF8 recode
base 2a000010 :  172.0 MB/s UCS to UTF8 conversion
base 2a000010 :  158.1 MB/s UTF8 to UCS conversion
base 2a000010 :   86.1 MB/s UTF8 recode
base 2c000010 :  172.1 MB/s UCS to UTF8 conversion
base 2c000010 :  158.5 MB/s UTF8 to UCS conversion
base 2c000010 :   86.3 MB/s UTF8 recode
base 2e000010 :  173.8 MB/s UCS to UTF8 conversion
base 2e000010 :  158.2 MB/s UTF8 to UCS conversion
base 2e000010 :   89.3 MB/s UTF8 recode
base 30000010 :  170.0 MB/s UCS to UTF8 conversion
base 30000010 :  158.7 MB/s UTF8 to UCS conversion
base 30000010 :   87.2 MB/s UTF8 recode
base 32000010 :  173.8 MB/s UCS to UTF8 conversion
base 32000010 :  158.4 MB/s UTF8 to UCS conversion
base 32000010 :   87.8 MB/s UTF8 recode
base 34000010 :  173.5 MB/s UCS to UTF8 conversion
base 34000010 :  158.5 MB/s UTF8 to UCS conversion
base 34000010 :   88.0 MB/s UTF8 recode
base 36000010 :  173.1 MB/s UCS to UTF8 conversion
base 36000010 :  158.4 MB/s UTF8 to UCS conversion
base 36000010 :   88.4 MB/s UTF8 recode
base 38000010 :  172.5 MB/s UCS to UTF8 conversion
base 38000010 :  158.5 MB/s UTF8 to UCS conversion
base 38000010 :   88.7 MB/s UTF8 recode
base 3a000010 :  170.5 MB/s UCS to UTF8 conversion
base 3a000010 :  158.4 MB/s UTF8 to UCS conversion
base 3a000010 :   87.4 MB/s UTF8 recode
base 3c000010 :  169.3 MB/s UCS to UTF8 conversion
base 3c000010 :  154.7 MB/s UTF8 to UCS conversion
base 3c000010 :   86.3 MB/s UTF8 recode
base 3e000010 :  172.0 MB/s UCS to UTF8 conversion
base 3e000010 :  156.9 MB/s UTF8 to UCS conversion
base 3e000010 :   87.9 MB/s UTF8 recode
base 40000010 :  171.8 MB/s UCS to UTF8 conversion
base 40000010 :  148.9 MB/s UTF8 to UCS conversion
base 40000010 :   83.7 MB/s UTF8 recode
base 42000010 :  173.1 MB/s UCS to UTF8 conversion
base 42000010 :  149.3 MB/s UTF8 to UCS conversion
base 42000010 :   88.1 MB/s UTF8 recode
base 44000010 :  173.7 MB/s UCS to UTF8 conversion
base 44000010 :  157.7 MB/s UTF8 to UCS conversion
base 44000010 :   87.4 MB/s UTF8 recode
base 46000010 :  172.1 MB/s UCS to UTF8 conversion
base 46000010 :  152.2 MB/s UTF8 to UCS conversion
base 46000010 :   87.3 MB/s UTF8 recode
base 48000010 :  174.0 MB/s UCS to UTF8 conversion
base 48000010 :  142.7 MB/s UTF8 to UCS conversion
base 48000010 :   87.8 MB/s UTF8 recode
base 4a000010 :  173.5 MB/s UCS to UTF8 conversion
base 4a000010 :  150.2 MB/s UTF8 to UCS conversion
base 4a000010 :   87.7 MB/s UTF8 recode
base 4c000010 :  174.0 MB/s UCS to UTF8 conversion
base 4c000010 :  158.4 MB/s UTF8 to UCS conversion
base 4c000010 :   88.4 MB/s UTF8 recode
base 4e000010 :  174.0 MB/s UCS to UTF8 conversion
base 4e000010 :  156.9 MB/s UTF8 to UCS conversion
base 4e000010 :   88.4 MB/s UTF8 recode

real	2m0.751s
user	2m0.540s
sys	0m0.190s

The above run is on a AMD Phenom(tm) II X4 955 Processor running at the stock clock rate.

Code is in: http://svn.bannister.us/public/json-c/
(A start on an experiment in fastest-possible JSON conversion.)

Note that I intentionally allow proper conversion of “invalid” UTF8 code strings. I completely understand the reason for the disallowed conversions, and I disagree.

Update: Converted to use pointer arithmetic, rather than array and index. Was not sure pointer math was still a win on current CPUs and compilers. Got a big boost in throughput, so it is!

The lightweight string class is simple enough to be reproduced from memory (nothing over-complicated), but can make quite a difference in performance. I had hoped to make a point.

Since posting, after the initial burst, the articles have collected what looks like a few new hits, every day, for the past five years. Counting hits is a pretty foggy indicator, but it seems possible quite a few folk have read the article (though a relatively small fraction of the programming community). Some may have copied the C++ string class in their code. (What I hoped!)

But there is no sure way of knowing. What is the cumulative impact of this steady trickle of readers?

Three distinct usages I’d like to cover: optimal performance,Windows web server (IIS) interoperable, and webhosting.

In addition, there are three interesting aspects of optimal performance: throughput, scalability, and stability.

For serving static content, I really like the model of a single-threaded non-blocking web server, of which thttpd was an early example, and for which the C10K question clarified the need. A small/simple web server has a much better chance to being very reliable. With the single-threaded non-blocking model, massive scalability is possible.

For serving dynamic content, I really like the isolation and load distribution possible with the FastCGI model (or the like). Dynamic code tends to be complex. Javascript interpreters are complex. Complex code tends to fail more often. Complex code can use more compute throughput than possible on a single box. For intranet applications, a single front-end web server is often preferable, and load distribution via FastCGI offers more headroom. All of which tends to argue for the FastCGI model, with isolation from the front-end web service, and potential distribution of load across more than one machine.

For the widest possible usage, in additional to optimal deployments (when there is no restriction on the front-end web server), the engine on which the application runs should be deployable behind IIS (for Windows-only organizations), and at common web-hosting services (like Dreamhost). Microsoft’s recent support of FastCGI with IIS is a big help.

At that time (three years back), none of the solutions were really optimal – and in fact were pretty far from optimal. The Java-based RhinoJavascript interpreter was easiest to embed, but failed the webhosting case. The C++ based JavaScript interpreters were a pain to embed, and offered good (but not great) performance.

Fast forward to the present, and Google offers the V8 JavaScript Engine that offers great performance, and is easy to embed. (Google as the good guys, riding to the rescue once again … you’d think they have white hats superglued to their brains.) Suddenly we have lots of projects embedding the V8 engine. In addition, seems most all the single-threaded non-blocking web servers have picked up support for FastCGI.

Oh … and I am pretty much fed up with the Java Servlet model. After considerable time with the problem, I am of the opinion that the servlet model chose the wrong abstractions, and this makes for awkward solutions. (Of course, the servlet model appeared very early in the history of web applications, so the mistake is easy to understand.)

Which means the model offered by node.js makes a lot of sense. I like the notion of a naked node (running JavaScript on the V8 engine) performing request dispatch without any extra layers or abstractions. The main lack with node.js is the ability to work via FastCGI (and thus no means to be deployed behind IIS on Windows).

But there are as yet items unresolved and/or unclear.

• Projects like v8cgi offer the V8 JavaScript Engine connected via FastCGI.
• The node.js project offers a single-threaded non-blocking web server … but can it work behind FastCGI?
• Is the environment for server-side JavaScript the same (or sufficiently similar) between node.js and v8cgi?
• Comet is still a question. Can FastCGI work well with long-outstanding requests from applications?

The good news is that we seem a lot closer to attractive and well-supported server-side JavaScript for web applications … but it seems we are not quite fully there, as yet.

Went with the Intel adaptor when I ordered this notebook, in part as Intel seems to be actively supporting the development of Linux drivers. In practice, my laptop wireless connection is mostly unreliable, and often near-useless.

I used to think the problem was interference from other nearby wireless routers, but my daughter’s laptop seems to work well when mine does not. I have a simple “bounce” test for network performance. My daughter’s cheap Toshiba, on the same “bounce” page, gets steady/fast performance when side-by-side to my laptop … which gets slower/unsteady/unreliable performance.

Not specific to the wireless router or crowded neighborhood either – I get similar poor performance when at my father’s place in Colorado (different brand and generation of router, and fewer/further neighbors).

After the prior results, I wanted to see if a less general-purpose implementation would perform better. Given enough advantages, could a Trie out-perform a hash table? (Again, this visits to some extent the question asked in a prior discussion.)

For the current exercise I built two Trie implementations. The LinkedTrie is cheap to build, minimal in use of memory, but not especially fast to access. The FixedTrie implementation should be pretty close to optimal in access time (for a Trie), but expensive to build (in fact the FixedTrieBuilder transforms a LinkedTrie into an optimized FixedTrie).

As before, the sources are in (as an Eclipse project): http://svn.bannister.us/public/Trie/.

The performance numbers make sense. The older TallTrie and WideTrie implementations (that traded increased memory use for speed) are indeed faster, though the LinkedTrie uses much(!) less memory. The new FixedTrie is fastest (hurrah!) and uses the least memory (a shade less than LinkedTrie).

But even the FixedTrie is slower than a generic hash table, with read-only access about 3 times as expensive.

Sample measurements…

=== words
21 ms - 98569 words loaded

=== hash
3935/second - hash map loaded {4007 ms, 15771040 operations = 254 ns/op}
8554/second - hash map re-loaded {4010 ms, 34302012 operations = 116 ns/op}
10768/second - access each item in hash map {4000 ms, 43074653 operations = 92 ns/op}

=== trie (linked)
870/second - loaded trie (linked) {4078 ms, 3548484 operations = 1149 ns/op}
1204/second - re-loaded trie (linked) {4010 ms, 4829881 operations = 830 ns/op}
1431/second - access each item in trie (linked) {4063 ms, 5815571 operations = 698 ns/op}
98569 slots of trie (linked)
225791 nodes of trie (linked)

=== build fixed trie

=== trie (fixed)
85779/second - loaded trie (fixed) {4000 ms, 343118689 operations = 11 ns/op}
87652/second - re-loaded trie (fixed) {4000 ms, 350609933 operations = 11 ns/op}
3438/second - access each item in trie (fixed) {4013 ms, 13799660 operations = 290 ns/op}
98569 slots of trie (fixed)
225791 nodes of trie (fixed)

=== trie (wide)
65/second - loaded trie (wide) {4502 ms, 295707 operations = 15224 ns/op}
1659/second - re-loaded trie (wide) {4040 ms, 6702692 operations = 602 ns/op}
1672/second - access each item in trie (wide) {4007 ms, 6702692 operations = 597 ns/op}
41533124 slots of trie (wide)
225890 nodes of trie (wide)

=== trie (tall)
682/second - loaded trie (tall) {4043 ms, 2759932 operations = 1464 ns/op}
1733/second - re-loaded trie (tall) {4036 ms, 6998399 operations = 576 ns/op}
1913/second - access each item in trie (tall) {4019 ms, 7688382 operations = 522 ns/op}
6982694 slots of trie (tall)
446075 nodes of trie (tall)

=== string to UTF8 conversion
1393/second - word to UTF8 (stock) {4033 ms, 5618433 operations = 717 ns/op}
9585/second - word to UTF8 (fast) {4000 ms, 38343341 operations = 104 ns/op}

(Note that the FixedTrie ignores re-load, so the times for load and re-load are bogus.)

I suspect a C++ Trie implementation could do a bit better … but not necessarily outperform … compared to hash tables.

Looks very much like even a specialized read-only Trie cannot match the performance of a generic hash table (at least in Java).

After getting stuck on the latest problem, I finally built a “PrintExplorer” test program. The sources are up on Kenai, and on my website. In both cases the sources are in a Mercurial repository, and I “push” changes to both (one of the nice features of Mercurial). The project was built using NetBeans 6.7, and you can download the built PrintExplorer.jar file.

Run as usual:

java -jar PrintExplorer.jar

(On Windows you can just double-click the JAR file.)

I filed two reports (so far, with review ids: 1660182 and 1660234) with bugs.sun.com, though for all I know that may be a black hole (or not … I have no idea). With the OpenJDK and JDK7 work there may be better places to report/record this … I do not know. (If you know a better path, please let me know.)

Found so far:

• PrinterJob.getPageDialog() makes invalid change to page size. (log)
• Toolkit.getPrintJob() changes JobAttributes.getPrinter() to default(?) printer. (log)
• Quarter-sized pages sometimes printed on an HP DeskJet 5550 on Windows 7. (log)

I am sure there are more. :/

Note that PrintExplorer always generates a log file. The log file contains the parameter values and print operations performed. (You need to be acquainted with Java Printing, otherwise the test program will be entirely obscure.) If you can reproduce a problem, then you can see in the log file exactly what you did, and what was returned. (Useful when you lost track of the exact sequence of steps.) When you send a log file, anyone can verify the exact sequence of steps you followed (a mistake in the problem description can be caught).

At this point the DocPrinterJob works for an empty Doc and a Pageable Doc, but not for a Reader Doc. (Unclear if the last is my bug, or a Java bug.)

Update – 2010.02.01
Today the filed reports became public-visible bugs 6921596 and 6921597.

My original query as represented in Canon’s support system (and trimmed):

Original Message Follows:
————————-
Email Support Form Message
Product Type: PIXMA MX860
Product Serial Number: absx22394
Date of Purchase: 11/18/2009
First Name: Preston
Last Name: Bannister
Address: 14 Vallecito
City: Foothill Ranch
State: CA
Zip: 92610
Phone Number: 9495880872
Email Address: preston@bannister.us
INQUIRY: Need drivers for 64-bit Ubuntu (Debian Linux).
Found the 32-bit drivers and sources on the Canon European site, but the sources are incomplete, so I cannot compile a 64-bit version.
Either compiled 64-bit drivers, or a complete set of sources would be acceptable.

The response was pretty predictable, if not exactly addressing the question as posed.

On Fri, Nov 20, 2009 at 5:09 AM, Canon Support – MFP <mfp @cits.canon.com> wrote:

Dear Preston Bannister:

Thank you for contacting Canon product support. We value you as a Canon customer and appreciate the opportunity to assist you. We regret the issues experienced with locating Linux drivers for the MX860.

Canon USA does not support Linux with its consumer products. The only recommendation I can offer would be to check the various Linux enthusiast websites to see if anyone has successfully attempted writing a third party driver.

We apologize for any inconvenience. Please let us know if we can be of any further assistance with your MX860.

Sincerely,

James
Technical Support Representative

Since there is always a chance (if small) of a constructive outcome, I tried again.

What I did was find that the Canon Europe and Canon Australia sites distribute Canon-developed Linux drivers for the MX860 (and other printers).

The software is pretty well done, has an active documented development history that goes at least to the year 2000, and is distributed as both binaries and source.

Odd that Canon USA does not distribute Canon-developed drivers (in which Canon has made a significant investment).

This is a case of so-close-but-not-quite there. The effort required for Canon to have a complete offering is pretty small, if someone can find the right branch of the company hierarchy to nudge.

Any chance of finding who within Canon sponsored development of the existing drivers for Linux?

The last response, which sounds dangerously close to “Fuck you. Go away.” (Though I would gladly be proven wrong.)

Dear Preston Bannister:

We appreciate your continued correspondence regarding Linux drivers for your PIXMA MX860.

I have forwarded your comments to Canon USA through our Customer Feedback process. This process allows us to capture important feedback from our valued customers. As we constantly strive to improve our products and services, your comments are vital to our continued success.

We hope this information is helpful to you. Please let us know if we can be of any further assistance with your PIXMA MX860.

Thank you for choosing Canon.

Sincerely,

George
Technical Support Representative

Had to try….

• Conversations are a first-class concept, and each message is a part of a conversation.
• Each message can be presented by a different application.

Because Wave is on the web, the set of applications is come from all applications in the world (on the web).

E-mail clients can present “threads” that are something like conversations, but this requires a bit of guessing and fakery on the part of the client, and is easily confused. IM clients can present a log of messages – that looks something like a conversation – but does not work when you switch between devices. Given that email and IM servers know nothing about conversations, the clients do pretty much the best they can. If you know about conversations on the server, you can do a better job on the client.

When conversations are a first-class concept, there are lots of new possible functions.

On the flip side, the Google Wave folk have made some (common) missteps.

• Using new names for existing notions is sometimes useful, but more often not. Calling a conversation a “wave”, calling items (messages) within a conversation “blits” … I find this more confusing than useful. This reminds me more of talk from marketing folk than engineers.
• It’s Talk, on the web. As a programmer, I get it – this is kewl! Writing a web application that allows all participants to see each character typed – is tricky. As a demo of tricky programming, this is sure to impress. As a useful feature in a general use application – not so much. Character-at-a-time message traffic is a huge load increase on the server, and a disservice to the users. Think about:
  • Locus of attention – the on-screen activity from updates to other users’ incomplete messages is sure to distract.
  • Quality of expression – how often have you started writing a message, then revised what you said and how you said it, before sending? Do you really want everyone to see your missteps … always?

  Note that both are a disservice to all participants – cool in a programming demo, not cool in a deployed application.

Other bits….

• It looks like Outlook. This was new and cool in 1997, but not so much now. Not at all sure this layout is efficient, especially given the changes in most-common screen layouts. In 1997 you would target 640×480, 800×600, and 1024×768 on 14″ to 17″ as most common screen sizes. Today you are looking 1280×800 on 14″ or 15″ as your minimum size, and 1920×1080 at 20″ or bigger as common. Present and future screens have not so much new height, with lots of new width – in fact too much new width for text content. Seems a different presentation is in order.
• Mixed applications in one web page is cool and exciting as ActiveX was in the web browser in 1996. Viewed in terms of the good things it can do, ActiveX allows an open-ended mix of applications to present within a web page – very cool. The Microsoft folk were blindsided (as constructive, well-intentioned folk) as they did not consider all the bad things that could be done (and with ActiveX, very bad things are possible). The Google folk have lots of history to make them wary, and the scope for harm is much smaller, so likely little bad stuff get through. Still – do not underestimate the black hats. There are bound to be some exploits (though likely few and quickly suppressed).

In sum, a decent start on a great idea. I’d want to be pretty ruthless about overhead that takes height away from the conversation view. I’d drop the per-character updates. I’d drop the use of funky names for existing ideas. None are fundamental problems, and all are easy to address.

Fixing up an old Java desktop application for viewing old “green-bar” reports. Should be pretty simple – the reports are all fixed pitch text. Given I like to do things that “just work” (from the users perspective), I’d added live “smart” definition of controllable font-auto-sizing, empty page suppression, and margins. Given that old printed reports may be formatted for paper sizes different than in your desktop (or network) printer, this was slightly tricky, but only that.

Of course, many of those old reports – meant for old line-printer green-bar paper – are best printed in landscape orientation. To my surprise, I find when in landscape orientation, the return from PrinterJob.pageDialog(PageFormat) consistently alters the margins, even when no changes were made, and the error is cumulative.

Um, what?

At first I assumed the error was in my code, but this bit of logged output caught the problem:

PageFormat validate::
	paper dx: 792.0 dy: 612.0
	margins top: 72 left: 72 bottom: 72 right: 72
	orientation : LANDSCAPE
PageFormat after pageDialog::
	paper dx: 794.0 dy: 613.0
	margins top: 72 left: 74 bottom: 73 right: 72
	orientation : LANDSCAPE

The first is the contents of the PageFormat before calling PrinterJob.pageDialog(PageFormat), and the second is the new PageFormat returned. Note the +1 dy bump and +2 dx bump to the page extents (and the matching errors introduced to the left/bottom margins).

The code I used to remove the error:

            controller.pageFormat = job.pageDialog(controller.pageFormat);

            // Work around an interesting(?) bug in PrinterJob.pageDialog().
            // For some reason the returned Paper is over-sized. (Why? Dunno.)
            // Use the Paper returned from PrinterJob.defaultPage() to get a proper size.
            // If the Java folk remove the bug, this code should change nothing.
            PageFormat pf = job.defaultPage();
            Paper paper1 = controller.pageFormat.getPaper();
            Paper paper2 = pf.getPaper();
            double x = paper1.getImageableX();
            double y = paper1.getImageableY();
            double dx = paper1.getImageableWidth();
            double dy = paper1.getImageableHeight();
            paper2.setImageableArea(x,y,dx,dy);
            controller.pageFormat.setPaper(paper2);

The bug was encountered using Java 1.6.0_15 (64-bit) on Ubuntu (Linux 2.6.28-15-generic #52-Ubuntu SMP Wed Sep 9 10:48:52 UTC 2009 x86_64 GNU/Linux).

The results are not encouraging. From a test run:

=== words
14 ms - 98569 words loaded

=== hash
3837/second - hash map loaded {4007 ms, 15376764 operations = 260 ns/op}
7953/second - hash map re-loaded {4003 ms, 31837787 operations = 125 ns/op}
12958/second - access each item in hash map {4001 ms, 51847294 operations = 77 ns/op}

=== trie (wide)
343/second - loaded trie (wide) {4022 ms, 1379966 operations = 2914 ns/op}
2089/second - re-loaded trie (wide) {4010 ms, 8378365 operations = 478 ns/op}
2165/second - access each item in trie (wide) {4005 ms, 8674072 operations = 461 ns/op}
41533124 slots of trie (wide)
225890 nodes of trie (wide)

=== trie (tall)
2254/second - loaded trie (tall) {4022 ms, 9068348 operations = 443 ns/op}
2446/second - re-loaded trie (tall) {4029 ms, 9856900 operations = 408 ns/op}
2429/second - access each item in trie (tall) {4017 ms, 9758331 operations = 411 ns/op}
6982694 slots of trie (tall)
446075 nodes of trie (tall)

=== string to UTF8 conversion
4708/second - word to UTF8 (stock) {4019 ms, 18925248 operations = 212 ns/op}
10503/second - word to UTF8 (fast) {4007 ms, 42088963 operations = 95 ns/op}

The test data is a file of 98569 words found in Linux installation. The words are read from a file, and loaded into an array of strings. The words are loaded into a HashMap (a library class) for comparison, and loaded into both a “tall” and “wide” Trie (differing space/time trade-offs).

The notion used to generate a “fast” Trie is for each node in the Trie to index off a byte in the UTF8 encoding of key String. Along the way, found that the stock String to UTF8 conversion to be a bit slow, so wrote a faster version. Even with the faster encoding, measured Trie performance is not good.

(Bit of a shock to find the stock String-to-UTF8 conversion slow in Java, as this is used a lot – especially in web applications – and may be significant enough to effect benchmarks. Java has been around long enough, so that I expected better. In fact, I expected UTF8 to be in native code in the JVM. Byte-bashing code is better suited to C/C++.)

If strings were already in UTF8 (or ASCII) format – as in C/C++ code – a similar Trie implementation would perform somewhat better, but still use too much memory. Whether a Trie in C++ could more closely approach the performance of a hash table is a bit of an open question.

Not sure why the “wide” Trie (using one node to index one UTF8 byte) is slightly slower than the “tall” Trie (using two nodes to index 4-bits each from one UTF8 byte). Expected the “wide” version to be faster. Suspect the larger memory use is enough to bust the CPU cache, and on smaller data the “wide” variant may be faster. Also saw a lot of jitter in the times – presumably due to Java GC activity (garbage collection) – though the relative results were consistent.

Clearly this “fast, general-purpose” Trie implementation is not overly useful – at ~5 times slower than a hash table, and huge use of memory. Alternate forms can be more space-efficient, but are likely to be slower.

As a guess, a Trie is not going to find much use in my code.

There are good reasons to take a run at the problem. Physics is changing what we can expect from future computers. Starting a few years back, and barring any unexpected shifts in technology, the rate at which a CPU can process a single instruction stream will increase only slowly. The economics of chip fabrication allow us to build a CPU with multiple cores. The physics of power consumption tell us that we can get more computing done per watt with slimmer cores at slightly lower clock rates. All of which argues for fabricating CPU chips with a slowly increasing number of cores, and slowly increasing clock rates.

All of which means that to make full use of present and future CPUs, there has to be a lot of concurrent computing.

Concurrency in programming is tricky, and often got wrong. This is nothing new. My first job out of college (so many years ago) was to work of an Ada compiler (a computer language with direct support for multi-threaded interaction) on a product (the Pascal Microengine) that had thread support built into the CPU’s microcode. There was then much talk of how to do concurrent programming.

What we learned then and in the time since is that fine-grained multi-threaded concurrent programming is tricky, and very easy to get wrong. For the bulk of programs and programmers, there is very limited need for this sort of concurrency. All things considered, this is probably a good thing.

Tim Bray – and the folk responding to his post – are mainly focusing on a programming language for concurrent programs. I suspect this is a mistake. Maybe some new (or newish) programming language will make bug-free concurrent programs easy, but I do not think this is likely.

I think we already have the bulk solution. Web-scale applications (at least those that work well) make use of large numbers of CPUs, with a huge amount of concurrent execution. Web-scale programming is mostly about swarms of small-scale execution single-threaded programs (not necessarily small), well isolated from other threads. Many of the attributes Tim lists are true – or mostly-true, or should-be-true – of web scale applications.

Clearly the web-scale application approach does not work for all applications – though it may work for more than you expect. There are always going to be some applications that need fine-grained threading … but I suspect this group is very small. For the bulk of programming we want to allow for massive concurrency, but well-isolated and coarse-grained.

Why the focus on programming languages that support fine-grained concurrent programs? What problem – in the application space – does that solve? Are more than a tiny fraction of applications in that space?

My answer to Tim’s question is to point at the concerns of web-scale applications and cloud-computing. The problem does not drive an interest in new programming languages. The answer to large-scale concurrent execution is – for most applications – large numbers of single-threaded programs, responding to requests. Tim’s list of characteristics – in part – is useful for that sort of programming. Editing down the list:

• Message passing
  and Dataflow
• Immutable data,
  and Transactional memory,
  and Tuple space

(Have to admit, I am not sure the above grouped notions are distinct, when viewed at this level.)

For the bulk of programmers and applications, the main needed change is finding the simplest possible adaptation to the needs of web-scale programming. Once done, concurrency and threading are solved problem. We do not want or need fine-scale single-name/address space multi-threading – that way lies madness. We do need well-isolated single threads, in mass numbers, cooperating across web-scale process, machine, and network boundaries.

This is out of our fields, but interesting.

http://www.edn.com/article/CA6686259.html?nid=2435&rid=8150303

I liked the one comment: “the generally horrible code from the Microsoft world”.

My characteristically reserved response:

Yes, it is usually cute when hardware guys talk about software.

As I remember, the result came from the large number of branches found in “the generally horrible code from the Microsoft world that dominated tasks in those days”. Except this characteristic is common to most modern code, not just in “Microsoft world” code. It comes from complex behaviors – an increasing and increasingly common characteristic in contemporary code. Outside a small set of niche applications (with large number crunching), you are going to have a lot of branches. Branching limits the value of concurrent instruction dispatch, and speculative execution (as we might guess). Results pointed at more than ~3-way dispatch as on the wrong side of the “knee” of the curve.

I think the chip design offers a hint as to their expected application domain:

Each of the Power7′s cores has 12 execution units, including two each load/store and fixed-point, four double-precision floating-point, and a decimal floating-point unit. The core can dispatch six instructions per cycle.

With four floating-point execution units, my guess is that they are aiming at niche mass number-crunching applications, not general purpose applications. Combined with the large shared-cache, these should do well on problems that allow highly concurrent bulk number crunching. Might see a lot of these to run weather simulations (to predict Global Warming), and some sorts of large engineering design calculations/simulations.

As a general purpose CPU, my guess is that benchmarks (when and if they appear) will be unimpressive.

Add to this the fact that most big new web applications are moving (have moved, in fact) to a no-shared-memory model – lots of single-board nodes with 1 to 8 CPU sockets, no shared memory between processes, connected via ethernet to large-scale high-performance redundant datastores (specialized database/filesystems).

Which makes the really cool design aspects of the Power7 pretty much useless. This is not the sort of CPU that would ever see large-scale deployment in a web-service datacenter.

Too bad the writer of the article does not have a clue about software, but no surprise from a hardware magazine.

Oh wait! I take it all back! This CPU could be terrific at the more compute-intense computer games! Perhaps Microsoft will pick this up for the next XBox? Super high frame rates in MegaBlaster2010! … except highly concurrent shared memory applications are a real bitch to write, even worse to debug, so better make that MegaBlaster2012, or 2013, or 2015, or … (hey! they never did ship!?!).

… maybe not.

Was out in the garage last night, planning the next bit, when my daughter popped out of the house. “Oh, you’re thinking again, aren’t you?” She was not surprised to find me sitting and staring for an hour, before starting on the next bit.

Seems I use roughly the same approach whether cooking, building things out of wood and metal, or writing software. When cooking I never use recipes, I am always changing something, and usually get good results. When building I never use plans (other than occasional rough sketches), will think through the problem and build the next bit in my mind, before starting manufacture – and usually get good results. When writing software I do not have much use for design documents or formal processes, but cannot start until I have thought through the problem, and usually get good results.

In each case I can visualize the end result before I start. In each case I am working without plans (written plans or plans from others), yet … I always have a plan.

As I learned back in school, this pattern of work does not work at all for some of my peers. My upper-division Physics lab class was the first clear example. The lab was – at base – one student alone in a room with a pile of equipment, and a set experiment to complete in a couple weeks. The lab had a new instructor when I took the class, and he was not familiar with the equipment. Also, for some reason, none of the equipment worked. So with little or no instructions, I had to figure out what the equipment did, how it should work, and what needed to be fixed.

Getting the experiments to work was both frustrating, and fun!

Midway through the course, the instructor assigned another student to work with me, as this guy was not doing well by himself. I knew the guy, counted him as a bit of a friend, and thought he was at least as good as I at the theory classes. But in the lab, he drove me nuts. I needed silence so I could visualize how things should work. For the parts I did not understand well enough to visualize, I would disassemble and play with the equipment, until I better understood how it worked.

I suppose it looked to the other guy like I did things in no particular order, while ignoring anything he said. After an indeterminate period of seemingly aimless activity, he’d suddenly hear “Ah … I get it!”, followed by an ordered, logical explanation, and a burst of activity – after which the equipment would (usually) work.

Bet I drove him a bit nuts, as he did to me.

Also, up to that point in the lab class, I thought I was doing poorly. Most times – when the instructor dropped by – the equipment was not working, and I had no coherent explanation (or so I felt) of what to do next. Once the equipment was working, I could – in short order – perform the experiment, and move to the next lab.

Bet also my more present co-workers see something similar – an indeterminate period of seemingly aimless activity, followed suddenly by a burst of activity and plans. Probably drives some folk – of differing nature – slightly nuts.

So my daughter caught me planning, again …

The main limitation of the Pearson hash is that it only computes an 8-bit value (i.e. the range 0..255). For some problems a bigger range of values would be good. The means offered in Pearson’s original paper (as I recall – threw out the old magazine long ago, and the paper is not on the web) was to run the algorithm more than once to add additional 8-bit chunks to the hash.

Just now I’m feeling like a bit of an idiot, as it seems the Pearson hash could be easily extended up to 16-bits (or a bit beyond) with a ridiculously small change.

The original Pearson algorithm:

static unsigned char T[] = {
    // The values 0..255 shuffled into random order.
};

static inline unsigned hashOf(unsigned h,const char* s) {
    for (int c = *s++; c; c = *s++) {
        h = T[h ^ (255 & c)];
    }
    return h;
}

The algorithm modified to generate a 16-bit hash:

static unsigned short T[] = {
    // The values 0..65535 shuffled into random order.
};

static inline unsigned hashOf(unsigned h,const char* s) {
    for (int c = *s++; c; c = *s++) {
        h = T[h ^ (255 & c)];
    }
    return h;
}

Note that only the type and size of the table T changed!

With the large cache sizes of current CPUs, the 128KB table will fit nicely into cache, and perform very well. On CPUs with smaller caches, you might want to use a smaller table. Also clearly the table size for a 32-bit hash would be huge. You might want some sort of test for degenerate patterns in T[] (though I’d guess the probability of getting a bad table is very small).

This notion is so simple, I am a bit surprised that no one (so far as I can tell) suggested this approach before.

For large numbers of keys, each bit added to the hash value saves the equivalent of a key comparison. Adding 8-bits to the key saves 8 key comparisons. If the expense added to the hash function is less than the cost of one key comparison (certainly true of string compares), then you are ahead from the first bit added.

I had looked at this over a year ago, but the SIPE support for Pidgin did not support the options needed for the IM service as configured by the company. On checking again, it seemed possible that the current version might work, but I needed the most recent version of Pidgin and the SIPE plugin.

So … for future reference.

Building Pidgin and the SIPE plugin from sources on Ubuntu

1. Install the dependencies needed to build from sources.

   sudo apt-get build-dep pidgin pidgin-sipe
   

2. Remove any existing installation.

   sudo apt-get remove pidgin pidgin-sipe libpurple-dev
   

3. Download Pidgin sources.
4. Change to the directory in which you unpacked the Pidgin sources, and build.

   ./configure
   make
   sudo make install
   

5. Download Pidgin SIPE plugin.
6. Change to the directory in which you unpacked the SIPE plugin sources, and build.

   ./configure
   make
   sudo make install
   

7. Now run Pidgin. It should appear in the usual Gnome menu, or can be run from the command line. Given that figuring out the right options for you company’s setup can be tricky (it was for me), you might want to do the initial connection setup with debug output from the command line.

   pidgin -d
   

   Not strictly necessary, as the “Debug Window” (got to via the “Help” menu) offers the same(?) information.

   You might also want to (temporarily) disable any other IM accounts you have configured.

Figuring out the right options to use with your company IM service … you are on your own.

As a regular practice, before generating a build to go to customers, I always carefully review all the changes to the program sources since the last customer release. Using CVS the command line is simple (run from the root directory for the component):

cvs diff -t WIDGET_v301

Reviewing the differences helps both to write up a customer-oriented list of changes, and also helps insure that nothing unintended (say, left over experiments or debug code) makes it into the customer release.

With Subversion under Eclipse the equivalent comparison is easy – just select the trunk and tags/v301 and a compare gets the same result as CVS.

With Subversion on the command line, the equivalent comparison is ridiculously tedious.

svn diff http://hostname/path/widget/tags/v301

Um, hello? Subversion knows the http://hostname/path/ part. If I followed the conventional layout, Subversion can deduce that the current directory is trunk and from that the path to tags. Yes, I know the repository layout (in terms of branches and tags) under Subversion can be arbitrary (and this is all very cool), but optimizing for the most-common case is always good.

Figuring that I must have missed something, I went digging, and came up with this gem:

Branko Äibej – Re: svn diff branch woprking copy against mainline?
This seems to be a common misconception. The important thing to remember here is that there is no separate namespace for labels and branches in SVN, and that the layout of the repository is arbitrary. IOW, the fact that you have branches in /branches is a convention, not something imposed by the SVN server.

The writer is almost exactly wrong. There are a separate namespaces in the Subversion repository. The namespaces are established by convention. The recommended PROJECT/{trunk,tags,branches} can be recognized by default, and any local variant specified by user-set properties in Subversion. This is easy.

I expected Subversion to be a nearly uniform improvement over CVS, not stuck for years on how to cover simple usage.

Note that the default VM memory size suggested by VirtualBox is 512MB (where 256MB is recommended for 2003 Server) – and for good reason! At least during the (usual) install and patch phase, it looks like 512MB is not enough the keep the VM away from swapping. Have to note – while the eye-candy is better installing 2008 Server, a Ubuntu server is a lot less trouble. As a sysadmin installing your third, tenth, or hundredth server, the eye-candy loses all appeal.

This leads to an interesting question. If Windows 2008 Server is run directly on the hardware, current server hardware is likely a quad processor with at least 4GB. On hardware the 2008 Server needs are a complete non-issue. On the other hand, if 2008 Server is run on a VM (and at this point I would expect most Windows Server instances to be VMs), then a 512MB baseline requirement for the VM may be excessive.

This has got to be a pretty problem for Microsoft.

Naturally Microsoft would like to sell the OS installed on the hardware. Lots of money to be got there. At the same time, smart customers are more likely to want something else (less compulsive expense) as the base OS, and push Windows Server instances into VMs. This creates a conflict. If Windows 2008 Server is too fat to run in a VM, that puts a bit more pressure on Windows business customers to either stay with Windows 2003 Server (or earlier), or to move off Windows servers entirely.

So … Microsoft could squeeze more money out of business customers, at the risk of losing those same customers … and that loss could prove permanent. Tricky. I would hate to be the guy at Microsoft making that call.

No … that is not right. I’d not have any trouble making that call. The customer is always right. I would be looking to keep the customers happy, and making the latest server OS a non-issue. Give customers no incentive to move off Windows Server. (Which is going to conflict with the MS folks still wanting lock-in.)

The install and upgrade UI on 2008 Server are a bit more “webby” … but it seems the Microsoft folk have not yet entirely got the message. While a distinct improvement, the 2008 UI still feels a bit dated.

Right. I’m describing an MS product as “dated”. You have no idea how odd that feels. Windows has gone from the extreme cutting edge (as when I went to a Windows developer conference in 1984), to “dated”?! Very, very strange ….

Back in December I got a Google Android phone – specifically a T-Mobile G1 phone. I happened to walk into the local T-Mobile store just out of curiosity. They had the phone in-stock and for a good price – and I was thoroughly disgusted with my Motorola Razor from Verizon. The fact that this was the first phone where I could freely (almost) install software was a clincher.

Compared to every phone I have used prior, the G1 is without doubt a solid improvement. The effect of the unified IM/text/email notifications is distinctly Pavlovian – my phone blurps, and I check GMail on my computer. At the same time, there are bits about the Android phone that are not ideal. The touch-selection algorithm needs work. I am never certain whether my touch will cause selection, activation, dragging, or …. I dunno. As a guess, it does what I wanted a bit less than half the time.

The GPS/location feature is very cool – except when it decides I am somewhere else. Again, an algorithm that needs work.

I have fat-fingered the phone and invoked the wrong actions when answering a call … less often than with the Razor, but still too often. Again, something that needs work.

Overall, I am distinctly encouraged. More work is needed, but this seems the best work as yet.

I have a recurring problem with sound on Ubuntu 8.10. On my desktop sound almost always stops working after a few hours (or sooner). On my laptop sound is less trouble, but still occasionally does stop working.

Also the volume on the my laptop was far too low. Found by opening the volume control applet and running all the sliders up to the max, I could get the volume to a decent level. One problem solved – after a fashion. Clearly there is an issue if the master volume control is defeated by a non-obvious secondary setting.

Lots of folks are having trouble with sound on Ubuntu, but a fair number are due to configuration issues. Wading through far too many notes, I found something that works:

Re: no sound on Ubuntu 8,10
try to restart alsa

sudo /etc/init.d/alsa-utils stop
sudo alsa force-reload
sudo /etc/init.d/alsa-utils start

Yep, that worked. Had to manually kill rhythmbox (which was playing music when sound died), and had to re-start the volume control applet. Bit crude.

Seems the base problems are more-or-less well known. Also seems at least one guy came up with instructions for diagnosing audio problems (though more for no-sound-ever problems). The PulseAudio software seems to be at the heart of the problem.

When low-end laptops ship with 1280×800 panels, large headers above tall/skinny content seem a very poor design. Designs that suit common screen sizes seem a little scarce.

Was hoping for inspiration, but left the above-mentioned CSS example sites feeling very uninspired.

Whenever I go into a new situation, I expect to find someone much smarter. It never seems to work out that way. Often I find folk as-smart, but never (yet) anyone I thought clearly smarter. (Worse, I tend to over-estimate the abilities of folk I know.)

Yet my expectation persists.

I have had some occasional odd interactions with folk in the past, that I did not understand at the time. Easy to forget that other folk have insecurities of their own. Years after one episode, I figured out the most likely and unexpected explanation.

My career-path meant I got heavy exposure to client-server programming very early. I tend to think through the issues fairly throughly (perhaps I have an odd notion of “fun”), when presented with an interesting problem. A decade later, I started at a company that had a client-server product. The product had clearly got some of the principles I had arrived at, badly wrong. After starting at Quest, and seeing the current-generation product, I heard there was a guy working on the next-generation of that same product. So I went to the guy and outlined all the things I thought he should do for the next generation product. On one point he said something like “we thought about doing that, but it was too hard”, for which my answer was “we did that on my last project, and it worked out very well”. I think the guy’s name was Jeremy.

I was just trying to be helpful.

What I did not know at the time was that what I had seen demonstrated was in fact the next-generation product, the result of two years of Jeremy’s work. So instead of criticizing the old generation to offer advice for the new generation … it must have sounded to Jeremy like this new guy was tearing apart his best work. The product was called “Vista”.

Then things got much worse.

Not many folk are intimately familiar with both Windows and Unix programming. Vista was a Unix-only product. Most folk in that outfit were Unix-only. Since I knew both platforms, I got tasked with porting Vista to Windows. Another guy – a long-term hand at the company – was tasked to work with me. The other guy – John? – I knew and liked. He was pretty sharp, but had other obligations. After a few weeks he bowed out, claiming that he was only going to slow me down.

I am a sucker for challenging problems – and always have been.

There were about a half-dozen programmers working on the Vista product. I had to keep up with their changes, while building a port to Windows, against an ever-diverging source-base. After six months of very intense work, I had a dual port. The same source base could be compiled to generate either a Windows or Unix version of the product. To keep up the necessary pace, I had isolated myself from most of the daily workings of the group. Along the way I found some perfectly astonishing bad code. I had assumed that programmers with equivalent levels of experience, but who had spent their entire careers doing only Unix programming, would be far better than I in that domain. Working through the Vista source code, I found out that I was wrong – and very badly wrong. Since I was under huge time pressure, I took an extremely pragmatic approach. When I found a malfunctioning section of code, I first tried to read through the code. If I was unclear what the code was meant to do, I re-wrote the section of code for clarity. At this point I usually realized the original code contained a number of errors. Code-level errors do not necessarily translate to customer-level errors, and the pressure on my time was enormous, so I made no effort to report my findings.

When I had finished, the group cut-over to using my source-base. They were a bit surprised to find that I had kept current with all their changes. (I had a secret weapon. They were using RCS-based source code control. I used CVS to track their changes.) What was worse was that I had almost-accidentally killed off a large part of their bug list. The code inspection and re-coding for clarity was just to make the port work, but had the side-effect of eliminating a lot of existing bugs. By this time Jeremy had been made manager for the group, and he called me in and yelled(!) at me, while admitting this. I was confused.

What I did not realize – until years later – is how this looked from Jeremy’s point of view.

Imagine a new guy at your company walks into your office, and details all the things wrong with the project you have worked on for the past two years. Imagine he claims to have done something you thought was too hard to do. Now imagine that the new guy is tasked with something you cannot do. You expect him to fail (and I know he did). Instead of failing he delivers a result that not only works, but is much improved over your best work.

Did I mention that Jeremy was made manager for the Vista group, and that I reported to him?

Midway through the project, I realized each client instance of the server in the Unix implementation of the Vista server was started via inetd. Since the Vista server-side incorporated Ghostscript, and the Ghostscript initialization took 20-30 seconds (loading a large amount of font information, mainly), this put a serious crimp in per-client startup time, and a pretty serious load on the server when there was a lot of clients. Since the Ghostscript initialization was the same for all clients, if the Vista server was run as a server, the per-client startup could be reduced to milliseconds, and the total load placed on the server hugely reduced. I went to Jeremy to explain how this could be done. He angrily rejected my suggestion, for no logical reason that I could discern. I was puzzled, but went back to the all-consuming task of completing the port.

Imagine that annoying new-guy offers a suggestion (which he has already implemented and demonstrated) that improves the startup time of your pet project by a few-thousandfold. At this point Jeremy really did not like me, and I had no clue as to why.

Naturally I had taken the same approach with the Windows port, so the per-client startup time was almost instantaneous.

After finishing the Vista port (to the great pleasure of the Marketing folk, who could now sell Vista into the Windows market), I got tasked with another port. I had a conversation with a distinctly grumpy Jeremy, who had to admit that the Vista port worked, and since I was apparently some sort of legend (not to my knowledge), I was tasked to port another product.

The next product was called “Shareplex”, and was a database replication product for Oracle databases.

I had used Oracle as a programmer/user a fair amount while at FileNet, so I was pretty familiar with Oracle at a user-level. Knowledge of Oracle at the reversed-engineering level needed for third-party database replication was entirely outside what I knew. Moreover the guy in charge of the group had written a book on Oracle performance tuning, so I felt entirely out-classed on both fronts.

Again, I was one guy trying to keep up with an active development group of about a half-dozen, while performing a port from Unix to Windows. It quickly became plain that I could not perform testing in the same labor-intensive manner as the existing group. (It also struck me as odd that there were no comprehensive test suites for a potentially mission-critical product.) So I came up with a series of Perl scripts to perform a series of automated tests against the database-replication product. The tests created, exercised, and destroyed a series of progressive, then semi-random, series of structures, data, and operations.

At the end of the port – another six months of very intense work – I had another dual-port. The same sources could be compiled for either Windows or Unix. The tests I had built proved that the port worked just as well as the original Unix code – and also proved that the original code did not work.

I was stunned. I had no idea how to proceed from that point.

What I did not realize at the time was that I had given two important group managers reason to hate me. A few weeks later, Charles – the Shareplex group manager – escorted me to a meeting, where I was fired.

Much later, I learned that the Shareplex product was withdrawn from the market, and took another 2-3 years of rewrites before it became a marketable product. The conclusions I derived from my tests were right, but I did not, at the time, know what to do with the information.

Without knowing, I had intimidated co-workers, by doing things that they could not.

This morning I made a different connection along the same lines….

When I worked at FileNet, the company hired in a new graduate from CalTech. Management treated him as if he was a living legend. Dan was a fairly smart guy, but I kind of doubted his ability to walk on water. Fairly early on, he did a performance analysis in the area in which I was working, and made a couple fairly elementary mistakes (for a first-year Physics student), and circulated some bogus results. I sent out a correction, but did nothing else.

What I did not take into account was his likely insecurity as a guy promoted well past his peers. Any sort of criticism was likely seen as a serious threat.

Fast-forward a bit over a decade. A marketing guy at my then-current employer wants to go talk to the folk at FileNet. I don’t see a likely gain to the exercise, but went along to support the marketing guy. We end up waiting in the entrance lobby for an oddly long time. Dan walks through the lobby. Dan claims not to know me, with a hint of anger. Odd. To put this in context – the lobby is not on a natural path between any two points within the building (I used to work there). That Dan would walk through the lobby at the time I was there was … odd. That Dan would not remember me (we worked together for about seven years), is … odd.

Since the episode was of no importance, the odd bit stuck, but was otherwise forgotten. In retrospect, if I had indeed made Dan feel threatened, then the combination of oddities add up to a probability. Is it possible I had made Dan feel threatened? Is it possible that we were made to wait until Dan could walk through the lobby? At some point Dan was promoted to CTO. Was the odd delay, odd diversion, and odd lack of recognition … not odd at all?

This morning’s amusement – of no further import.

Now … I could be wrong about any of the above. On the other hand, even if right, it seems unlikely that anyone involved would confirm anything. Oh well.

Started writing a simple cxi (for CGI-like Xml Interchange, perhaps) program as suggested in a prior scripting-oriented article. Pretty straight-forward coding (with some added clever notions that occurred to me later), right up until I got stuck on how to get the child command started.

Commands to be run by cxi fall into two main groups.

• Generators – commands that generate output, without reading input.
• Filters – commands that read input and generate output.

Filters are the easier case.

1. Read environment for HTTP_ACCEPT.
2. Read the input for headers.
3. Read the command line for options.
4. Based on the last found, select the command to run, and set HTTP_ACCEPT for the child process.

Note that the input must be scanned for headers before the child process can be started. Generators are tricky. We cannot read the input for headers (since there is no input), as this would block, and the child process would never get started. We could require pre-knowledge to distinguish generators from filters (a sort of database of command metadata), but this strikes me as inelegant. Traditionally in Unix there is no way to distinguish between commands that are filters and generators. Any given command could be either.

Need a simple convention to mark a generator. I’m inclined to conflate two needs and address with single solution. At the start of a pipeline of cxi commands, you need to establish the desired intermediate data type, as well as indicate that the command is a generator. Since this a usual case, the presence of a command option could indicate both. So a pipeline of cxi commands would look like:

cxi -t xml ps | cxi foo | cxi bar

In the less-usual case, where the initial command is a filter, and addition option could override and force filter-behavior:

cxi -t xml -f ps | cxi foo | cxi bar

In the less-usual case, where the type is indicated in the environment, then another option to indicate the generator:

export HTTP_ACCEPT=text/xml
cxi -g ps | cxi foo | cxi bar
cxi -g ps | cxi foo | cxi bar

Can’t think of anything more elegant, so will go with the above.

As a slightly amusing sidelight – turns out the GNU getopt is too smart for my purposes, and I will have to use an older “dumber” implementation of getopt. The GNU implementation will pick out options anywhere in the command line, and I want it to stop at the first non-option argument.

Munich’s Metro Stressful, But It Goes Everywhere | Autopia from Wired.com
Riders purchase tickets at self-service kiosk priced by zone — a ticket covering most of the inner city costs €2.30 ($2.95) — and before boarding, stamp them with an old-school time clock. It's possible to score a free ride by “forgetting” to stamp your ticket, but at the risk of getting caught by plainclothes agents making random checks. They’ll slap you with a big fine, and the “I'm just a confused tourist” won't get you anywhere.

Bruce Schneier frequently repeats the notion that security should be cost-effective (counting both material and non-material costs). Whomever came up with this approach for the Munich Metro … the result is brilliant. The investment in infrastructure for fee collection should be less. The Metro can employ fewer folk in less repetitive, more intelligent jobs. (I would bet the plainclothes agents are cross-trained to look for other criminal behaviors.) Peak traffic flows are less likely to be impeded by “security” measures.

Incidentally, I do believe that Schneier is having a large positive effect on application of concepts for security, directly or indirectly. Seems more and more often I am running into fragments of his thoughts in other venues. I know enough about the principles to use security effectively in my working domain (which makes me the “security expert”, though I would not claim that title). When folk want or need know more I almost always point them to Schneier’s writings. I suspect there are other folk who also point to Schneier as the standard-bearer for security-related issues. Over time it looks as though the cumulative effect is significant.

Heard this first applied to theories in Physics. Any not-over-complicated theory that effectively explained the facts was – as I understood – considered an “elegant” theory. (My original interest – and college degree – was in Physics. I wanted to build starships.)

Came up, long ago, with rules for “elegant” distributed applications. The first significant distributed application I spent time with was the early FileNet system. This was well over twenty years ago. Strong interest in scaling to large (by the standards of the time) deployments, forced careful thought about performance. Pretty much everything I have worked on before and since has had a network in the middle, so I’ve had time and reason to think about the subject. Changing technology does not change the inherent nature of distributed systems, so the relevant set of notions will pretty much always apply.

(What continues to surprise me is that individuals and outfits still get these same bits wrong!)

In the interest of hitting the usual points once….

When building distributed applications, there are some basic principles you should always keep in mind.

Minimize the amount of data crossing the network.

The capacity of the network is always limited. Capacity is large in the usual development setup, when both server and client are on the same segment. In real use the available capacity is almost always less, and sometimes much less.

Minimize the number of round-trips across the network.

Networks always have latency. This has nothing to do with the speed of a network (in terms of bits/second). I have written about this before. For an interactive application the ideal is one round-trip per user action.

Shift computation from the server to the clients, where practical

There are almost always more clients than servers. If you can shift computation to the clients, you will get better overall throughput.

From the above principles, for building web applications you can derive further guidelines.

Code complexity belongs on the server. Code in the client should be simple.

Code to be executed on the client must be shipped across the network. The less you have to ship across the network, the better. If you have an application that calls for code-complexity on the client, you want to ship the code once, and should be looking at solutions like Java WebStart. Otherwise you want to ship as little code as possible across the network. This fits perfectly with the use of compact scripts in the client using Javascript.

The fact that Javascript is interpreted on each and every load (and not compiled) serves only to reinforce this guideline.

Large iterations belong mainly on the server, not the client.

Compiled code is usually much more efficient than interpreted code. The server can (or should) use compiled code. In the case of web applications, the client code is interpreted code.

Large data belongs on the server, not the client.

The (sometimes) narrow network channel, and the relative efficiency of compiled code – both argue for keeping large data on the server.

Use the strengths of the web browser.

Native code is faster than interpreted code. The web browser is smart, and incorporates a large set of behaviors in native code. Use of built-in behaviors can mean smaller Javascript and faster execution.

In the present, Javascript offers an elegant solution to the need for a scripting language in both client and server. Made the mistake(?) of responding to a recent post. Seems that each time something like this comes up, we have an almost fixed set of notions coming back. After a few iterations – just not very interesting.

Some quotes, with names omitted to protect the guilty.

Arrays have no semantics. They are not first-class collections. Do not use them in any public API, regardless of the language you use. Wrap arrays with a public type that exposes semantics.

The semantics of arrays as collections and iterations are simple and perfectly suited for small scripts. For the most part, you do not need anything more. The domain for scripting is small, concise, and hideously flexible code. More elaborate solutions might make sense in large server-side code. Client-side script or structures shipped between client and server should only be as elaborate as is needed – and no more.

You’re right in the sense that JS is “good enough” for most of basic usages, but almost useless for writing bigger software. It’s the reason why there’s been recently a lot of higher level languages that generates JS code. Either Java (GWT) or haXe (http://haxe.org)

You should not be writing large code in Javascript. You must use Javascript in the client (the web browser), but in-browser script should not be large. You should consider Javascript as “glue” code on the server-side – small code, few iterations, with huge flexibility – to allow special-case customization without re-coding. Javascript (as with ELisp in Emacs and AutoLisp in AutoCAD) is a scripting language. You do not want to write the bulk of a large application in Javascript. Large code is a non-goal for a scripting language.

I still dislike JavaScript, and likely always will. It has some pretty fundamental flaws.

Javascript evolved almost as a hack (if not quite). Early versions were less capable. Early examples were uninspired (or worse). The present iteration preserves past mistakes. Ignore the mistakes, and use the good parts. The good parts are … very good, as suits a scripting language.

JavaScript has no place on the server.

Quite the opposite – as the scripting language known to the largest group, and fated to be well-known over a long period – Javascript can serve exceptionally in the role that scripting languages have long met in large, successful applications. Not for large use, but with a valued place. Often when faced with the need to adapt a large application to a specific customer/site needs, there are always cases when a simple list of options is not enough (and rarely-used options serve only to make the application obscure to all customers). There is always a part of the problem space best met by a scripting language deeply integrated with your application.

Historically we have always had a zoo of suitable scripting languages, with insufficient reason to choose between them. In an odd way, the rise of Javascript in the web browser does us a favor, as Javascript is sufficient, and now most widely known of all scripting languages. There are times when a single logical solution is best for all involved.

Javascript is good enough.

Yes, this is a many-times-solved problem – though not always well solved. Seems like most everything I’ve done in the last twenty-odd years (or more) has had a network in the middle. From that experience, a few guidelines…

Guidelines for network protocols used in distributed applications

1. Design for differing versions on the client and server.

   In some limited cases you can keep the client and server versions in sync. For the most part – over time and over growing deployments – you cannot. Assume that the client version will often differ from the server (in either direction).

2. Convert at the edges.

   The types inside your application may change many times (for good reason). The structures that cross the network should change slowly (if at all). In your application, convert from internal to protocol structures at the edge.

3. Version your protocol.

   This could be as simple as a single version number offered by the client to the server, and the server to the client, at the beginning of the conversation. You could do finer-grained versioning, but this seems rarely needed.

4. Do not be over-specific in your protocol types.

   When passing a number across the net, use a general number type. You application might internally use a small number type (8-bit or 16-bit), and you may think that is all you will ever need. Some of those numbers will grow. At the edge, when ingesting incoming data, check for values outside the range you can handle.

   The same argument applies to strings, arrays, and the like.

5. Design for upwards and downwards compatibility.

   When the client asks a question of the server, older servers may not understand the question, or return a smaller answer. When the client is older, the server may have a larger answer, or the question may be smaller. Most of the time, changes are incremental, and both server and client can handle the difference without explicit reference to the protocol version. Sooner or later someone will goof, and you will need to add a special case for a specific version.

This all might sound complex or difficult – but is simpler than the alternative.

A few years back a group at Microsoft came up with Monad … which was a pretty cool idea, if a little too fat. Took years before they eventually shipped (renamed as Powershell).

Unix has tended to be about simple ideas with a lot of mileage. While the notion of building something like Monad on Unix is pretty interesting (I’d use Javascript via Rhino on the JVM), what I really wanted was a simpler notion that better “fit” the Unix-tools mindset.

One of the really cool bits about Monad was the ability to stream structured data – typically XML. Unix tools typically work on character streams – and only that. The character streams could quite naturally include structured data … but there seemed to be something missing.

Turns out there is a simple/elegant solution that could be retrofit to existing shells and tools, and fits very well within the usual Unix-ish way of doing things. The notion is so simple, it is a bit funny. This makes a round-trip of sorts….

The first generation of web servers were almost entirely running on Unix. The first common-model for dynamic content was CGI, which was simply (and logically) a slightly warmed-over of Unix shell scripting. Running a shell script for each incoming web request was not especially efficient, so we got a whole zoo of alternatives (ASP, JSP, mod_perl, etc.) but the programming model is still based on CGI – which in turn is derived from Unix shell scripting.

When a web browser makes a request of a web server, the request includes a “Content-Type” header to indicate the MIME type of the request data, and a set of “Accept” headers to indicate acceptable data-types for the response.

When a shell script is used for CGI, the HTTP headers turn into environment variables – for example:

HTTP_ACCEPT=text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
HTTP_ACCEPT_LANGUAGE=en-us,en;q=0.5
HTTP_ACCEPT_ENCODING=gzip,deflate
HTTP_ACCEPT_CHARSET=ISO-8859-1,utf-8;q=0.7,*;q=0.7

Unix tools that got used for CGI applications (Perl, Python, PHP, etc.) were all taught to ingest HTTP requests and generate HTTP responses. The code exists and is generally quite mature.

How could we pipe structured data between processes, automatically negotiating the data-type where possible?
In exactly same way a web browser negotiates the content-type with the web server!

There are a relatively small number of missing bits needed to make shell run scripts/tools in a similar fashion to CGI.

1. The shell needs to read HTTP response headers from the output of a prior process, and translate the headers into environment variables, before invoking the next process in a pipeline.
2. Some convention so tools know whether to generate a plain character stream, or structured data. The presence of the HTTP_ACCEPT environment variable is probably sufficient.
3. … er, maybe that’s all.

There are many ways we could enable this behavior, but it could be as simple as:

HTTP_ACCEPT=text/xml ps -e | fold | spindle | whatever

… to invoke use of XML in a single pipeline, or:

export HTTP_ACCEPT=text/xml
ps -e | fold | spindle | whatever

… to invoke use of XML over an entire script.

Could use a shim without changing the shell. Something like:

HTTP_ACCEPT=text/xml ps -e | cgi perl a.pl | cgi php -f b.php

An exercise for another day….

Why? Because when you subscribe to a feed from a website, you are often offered a list of choices.

RSS-0.92
RSS-2.0
Atom

This is bad user interface design. The end user – who has no deep notion of what these things are and how they differ – has no means of knowing which to pick. A well-designed user interface will hide the redundant choices, and select the most appropriate (probably Atom, going forward). Over the entire universe of applications, we can pretty much count on the fact that not all user interfaces are well-designed. If we can help the end user, we should.

The naming used should result in the list of choices:

RSS-0.92
RSS-2.0
RSS-Atom

Now even with indifferently-done user interfaces the user is given a sufficient clue. Clearly the choices are all of the same “kind”. Sorted alphabetically “RSS-Atom” will appear after any “RSS-version-number, which is a clue to the user that Atom is a better choice.

A small aid to hundreds of millions (soon billions?) of users is a big deal.

You can grab the sources for the test CSV-parser-1 program from Subversion. The test program does a full CSV parse as described in RFC 4180 (including handling quoted fields with embedded line breaks) and a bit more, but does nothing with the parsed data.

Results from test runs – on my HP laptop (Intel T9300 Core 2 Duo CPU @ 2.5GHz with 4GB memory):

preston@mercury:~/workspace/CSV-parser-1$ time Release/CSV-parser-1 -n 0 in/1g.txt
TIME Sun Dec 21 16:45:43 2008
Scanning: in/1g.txt
Done with: in/1g.txt
TIME Sun Dec 21 16:45:47 2008
Elapsed (ms): 4249, total (MB): 981
Scanned 230 MB/s

real	0m4.254s
user	0m3.500s
sys	0m0.656s

The above is for a ~1GB file, fully cached in memory (do repeated test runs until the times stabilize).

preston@mercury:~/workspace/CSV-parser-1$ time Release/CSV-parser-1 -n 0 in/4g.txt
TIME Sun Dec 21 16:49:31 2008
Scanning: in/4g.txt
Done with: in/4g.txt
TIME Sun Dec 21 16:51:02 2008
Elapsed (ms): 91449, total (MB): 3944
Scanned 43 MB/s

real	1m31.455s
user	0m41.271s
sys	0m10.549s

The above is for a ~4GB file, not cached in memory. The result is very clear – an efficient CSV file parser can ingest data much faster than the data can be read off ordinary disks (a bit over five times faster). Even a fast RAID would be hard-pressed to deliver data faster than it could be parsed.

Of course, in “real” applications, any processing performed on the parsed CSV data will likely dominate the runtime. Application-specific processing could easily saturate more the one CPU. The problem partitions into most-efficient read-and-parse of CSV data from disk (which is what this example does), and distribution of application-specific processing across multiple CPUs (which this example can do in the same manner as feed-workers … and which may or may not suit your application).

Insert the usual caveats here. The example program has seen only basic testing. There were other applications (minimally) active. The C++ code was written for reuse, and has not been run through a profiler. You could tweak the code to get slightly better performance, but probably not any large improvements.

The same test run on a desktop (slower CPUs, faster disk):

preston@brutus:~/workspace/CSV-parser-1$ time Release/CSV-parser-1 -n 0 in/1g.txt
TIME Sun Dec 21 17:21:19 2008
Scanning: in/1g.txt
Done with: in/1g.txt
TIME Sun Dec 21 17:21:27 2008
Elapsed (ms): 7530, total (MB): 981
Scanned 130 MB/s

real	0m7.535s
user	0m6.136s
sys	0m1.388s

The above times are for a file cached in memory.

preston@brutus:~/workspace/CSV-parser-1$ time Release/CSV-parser-1 -n 0 in/4g.txt
TIME Sun Dec 21 17:23:31 2008
Scanning: in/4g.txt
Done with: in/4g.txt
TIME Sun Dec 21 17:25:00 2008
Elapsed (ms): 89020, total (MB): 3944
Scanned 44 MB/s

real	1m29.084s
user	0m26.994s
sys	0m7.004s

The above times are for a file not cached in memory. The results are entirely consistent with the first set of runs.

Clearly, an efficient CSV file parser can process data faster than a single disk can deliver. The SSD’s (solid-state disks) currently on the market seem to manage sustained read rates in the range of 40-100MB/s, so a single-process parser should be able to fully saturate the disk.

If you are doing large-scale processing of CSV data, your most-efficient approach is most likely to use a single (efficient!) reader-parser thread, and then roughly as many application-specific processing threads (or processes) as you have CPUs.

I have used Microsoft Project, on and (mostly) off over a couple decades. Each time I’ve come back to using Microsoft Project, the experience is less than satisfactory. I seem to detect an anti-pattern. Those folks that are hopelessly invested in the use of Microsoft Project offer a weak defense of the product. Pretty much everyone else thinks the product a poor choice. The single-copy price has always been somewhat high, so relatively few folk have tried to use MS Project.

In the last encounter with project planning software (several years back), I became convinced that a web-based solution was the right solution to the for this sort of problem. Schedules are inherently a shared thing, and passing file copies around via email was a poor substitute. A few years later Microsoft came up with a web-based variant of Project, but by then I was too busy (fortunately) to mess around with project-planning software. (Some other poor guy had inherited the job of making and updating the MS Project schedules – a vast and singularly non-productive time sink.)

Still … there was a chance Microsoft could have done good things since the last time I had used Project. Turning out a web-based application is very different from turning out a desktop application. With the forced change there is the opportunity to re-think the design, and perhaps to make things better.

Nope.

The web-based version of Project is unbelievably lame. I could make a list … but it would be too long. (Hey, this is my personal weblog, not a paid-for review.) The static screenshots look nice, but as a usable web application, this is not. For the design aspects, the words that come to mind are “completely clueless”. (Again, I am puzzled that Microsoft – with all their vast resources – could do anything so badly. Though from my own experience at other companies, I might have a pretty good idea why this can happen.)

Did a bit of searching to see what others were saying about the product, and mainly found folk selling training or consulting – a sign of a pain-in-the-ass to use product. I’m guessing that anyone who has a choice and a bit of intelligence uses something else.

Most of the information we present through software (content, not decorations) is organized to present well on an ordinary letter-sized piece of paper. Once the width of a screen became wider than letter-sized paper, we no longer needed to be severely constrained in our use of horizontal space. We crossed a threshold. Now even the lowest-end screens (at 1280×800) are much wider than paper.

Why should an ancient medium like paper have anything to do with modern user interface design? Common paper sizes are – in effect – the result of a centuries-long experiment in user interface design. Paper can easily be fabricated in practically any size and shape. The paper sizes we use are the result of human choice, not a limitation somehow imposed by the manufacturing process. The range of paper sizes we most commonly use today are the result of generations of humans choosing what works best.

If you compare screen size to paper size, the old trade-offs make a lot more sense. There was for a long time – as long as 800×600 (or smaller) screens were significant – a very strong motivation to place overhead elements on the top and bottom of the screen, and not on the sides. For those small screens the cases where content was wider than the screen were not rare. For anything text-like, usage that requires horizontal scrolling is horribly inefficient (for the human, not for the computer). Any design element that took away from horizontal space for content was going to force an ever larger fraction of usage into horizontal scrolling. Put simply, screens were effectively narrower than paper and best design required preserving as much horizontal space as possible for content.

The old trade-offs did in the past make a lot of sense.

A low-end 1280×800 14-inch laptop screen is about 11.9 inches wide and 7.4 inches tall. Given the content area on letter-sized is about 7.5 inches wide and 10 inches tall, the screen about 2.6 inches too short and 4.4 inches too wide. Hold a piece of paper up against the screen, and the problem is pretty clear.

Have to admit: when “widescreen” panels first started to appear on laptops, I was hoping this was an aberration, not a trend. Unless you spend all your time watching movies and playing video games, most of the content you look at is text-like, and “widescreen” is not efficient for this sort of content. By now it seems very clear that we are going to be stuck with this form-factor for a very long time, and must adjust our design-habits to match.

There is a good deep/underlying basis that justified the old design idioms on old screens. That same basis on today’s “widescreen” panels makes those old idioms horribly wrong.