Simple basic facts about aeroplanes: long thin wings tend to more efficient (aerodynamically, not structurally) than wider/thicker/shorter wings. For much the same reason – propellers are more efficient than jets. Ducted fans are less efficient than propellers, but more efficient than pure-jet engines. Turbofan engines are basically ducted fan turboprops. Gains in jet efficiency over than past few decades are in part due to higher bypass turbofans (basically moving from pure jets closer to propellers).

Even propellers are not ideal. Swirling a couple curved sticks of metal through the airstream at high speeds is going to chew up energy without adding to propulsion. Many-bladed turbofans chewing through the airstream have got to be worse. Lots of energy wasted – could there be a more efficient way? Nothing especially obvious … or something better would be in practice.

Ideally we would like a way to throw back the bulk of an airstream without lots of extraneous physical churning. The only certain way we know is to use propellers – like oars in water. Could there be another way?

There is a well-known phenomena in Physics known as “electric wind”, that moves air without physical contact, but is by no measure efficient. Is there any way this could be used?

Is there any way to efficiently push an airstream without physical contact?

The “electric wind” is a stream of charged particles. A magnetic field deflects an charged particle moving through. The deflection exerts a force on the magnet (assuming the force is not sufficient to capture the charge). Movement against that force consumes energy. That energy presumably could accelerate the charged particles.

The mean free path of a charged particle at normal atmospheric pressures is short. Any accelerated ion would give up about half it’s energy at each collision. Short mean free paths mean many collisions. The net result would be (presumably) to accelerate a bulk of the airstream. Maybe.

Magnetic fields deflect charged particles. Strong magnets rotating on opposite directions could deflect and accelerate, then re-deflect and further accelerate charged particles – maybe. Would the result be significant? Would the result be efficient? I have no idea.

This might be an approach only possible if the “controller” is sufficiently smart, and with quite intense magnets (superconducting?). Matching the acceleration and deflection of charged particles through alternating magnetic fields through changing atmospheric conditions may not be possible with simpler control.

Is a “magnetic propeller” is practical possibility?

#!/bin/sh
gnome-open "https://mail.google.com/mail?extsrc=mailto&url=$*"

Make the file executable.

On Ubuntu Linux (using the Gnome desktop), go to:

System > Preferences > Preferred Applications

Under Internet / Mail Reader select “Custom” and enter the command:

/home/preston/bin/mailto %s

(Replace “/home/preston” with your $HOME.)

This should open GMail in your default web browser, composing a new message, with the recipient set.

Microsoft Windows [Version 6.1.7600]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

Yep. Windows 7 is in fact Windows version 6. Gotta love those Marketing folk.

Most FastCGI backends seems to be written for non-multiplexed connections. (Much simpler, so understandable.) The IIS FastCGI connector apparently does not support multiplexed connections.

Writing a FastCGI backend that allows for multiplexed connections. Would be a waste of time if not supported by a frontend, or if existing frontends are buggy.

(Not really expecting a response, but have to state the question.)

The HTML Working Group at W3C is … far too much noise. The HTML5 “standard” is going to be a bloated monster, and there is no chance I can change that. Time to stop the pretense of trying.

Not that all the work is bad, or that there is any shortage of well-intentioned folk. What the group lacks is any sense of minimalism, and enough strong voices able to say “no”. What we will get is going to be even harder to digest than HTML4. This is sad. Future developers are going to have an even harder time, for no good reason.

The wildcard here is that the mainstream browser implementations may not follow all the half-thought ideas thrown in by the HTML5 Working Group. No idea how this will work out.

At the core HTML is pretty damn simple – or could be. HTML4 got stuffed with a bunch of half-thought notions, most of which have since proved of no value, and were ignored by developers. Ideally we would learn from experience, omit the fuzzy disused bits, and trim HTML down to the useful core. There are well-known (though not universally known) means to achieve this aim. This is not going to happen.

I cannot keep up with the herd of Energizer-bunnies eager to make their mark, and with too-limited experience.
Time to stop pretending.

The use of UTF-8 on the web is common and increasing. Lots of data comes in as UTF-8, and inefficiency in UTF-8 data handling is going to have pretty pervasive impact.

On the flip side, the creators of UTF-8 did a good job. There is nothing really complicated about the UTF-8 format, and processing is simple.

So I was surprised (or rather shocked) to find in an earlier experiment that Java performed UTF-8 conversion slowly. In fact, I was able to write a faster UTF-8 decoder in Java than the stock decoder. This is just plain wrong. Conversion between encodings is a primitive/simple operation best written in C/C++ and run as native code (and this is the sort of processing where C/C++ is probably always going to be much faster than Java).

There is a problem in that malicious external parties can send oddly-encoded UTF-8, and bypass simple-minded malware detection software. Ordinary ASCII characters can coded as an alternate multi-byte sequences, and simple scanners miss the alternate encoding.

This is a problem. There is a simple solution. One of a set of principles I adopted a long time ago is “convert at the edges”. If you have data coming in from an untrusted source, then you perform conversion and validation at the “edge” where the data is first received.

In the case of UTF-8 coming from an untrusted source, to make all later processing simpler, you must recode to eliminate any alternate encodings. This is quite simple, as recoded UTF-8 will always be the same size or smaller, and so can be done in-place. The prior experiment measured the cost of UTF-8 recoding. Looks like we can drive a 1-Gbit network link at full speed (with efficient code), while recoding the entire contents. Since UTF-8 data usually represents a smaller portion of traffic, and since other processing tends to take the larger part of the load, there is no reason to not perform recoding on any UTF-8 data coming from an untrusted source.

Combine “convert at the edges” with UTF-8 recoding, and we lose the basis for the requirement in RFC 3629 for detection of “illegal” UTF-8 code sequences. In addition we allow all downstream processing to be simpler and more efficient … and we also can be tolerant of imperfect upstream software. (Yes, I am going to invoke Postel, again.)

The basis is good (secure processing with untrusted sources), but the requirement for detection of “illegal” sequences is not necessary and (most definitely!) not optimal.

Example of UTF-8 recoding (from String.cpp).

void UTF8::String::recode() {
    // Iterate until all UTF8 characters are normalized.
    // UTF8 in canonical form can only be smaller, so work in-place.
    char* p1 = pBuffer;
    char* p2 = pBuffer;
    char* pEOS = pBuffer + nContent;
    while (p1 < pEOS) {
        int c = 255 & *p1++;
        if (c < 0x80) {
            *p2++ = c;
            continue;
        }
        if (c < 0xE0) {
            c = (31 & c) << 6;
            c |= 63 & *p1++;
        } else if (c < 0xF0) {
            c = (15 & c) << 12;
            c |= (63 & *p1++) << 6;
            c |= 63 & *p1++;
        } else if (c < 0xF8) {
            c = (7 & c) << 18;
            c |= (63 & *p1++) << 12;
            c |= (63 & *p1++) << 6;
            c |= 63 & *p1++;
        } else if (c < 0xFC) {
            c = (3 & c) << 24;
            c |= (63 & *p1++) << 18;
            c |= (63 & *p1++) << 12;
            c |= (63 & *p1++) << 6;
            c |= 63 & *p1++;
        } else {
            c = (1 & c) << 30;
            c |= (63 & *p1++) << 24;
            c |= (63 & *p1++) << 18;
            c |= (63 & *p1++) << 12;
            c |= (63 & *p1++) << 6;
            c |= 63 & *p1++;
        }
        if (c < 0x80) {
            *p2++ = c;
        } else if (c < 0x800) {
            *p2++ = 0xC0 | (c >> 6);
            *p2++ = 0x80 | (63 & c);
        } else if (c < 0x10000) {
            *p2++ = 0xE0 | (c >> 12);
            *p2++ = 0x80 | (63 & (c >> 6));
            *p2++ = 0x80 | (63 & c);
        } else if (c < 0x200000) {
            *p2++ = 0xF0 | (c >> 18);
            *p2++ = 0x80 | (63 & (c >> 12));
            *p2++ = 0x80 | (63 & (c >> 6));
            *p2++ = 0x80 | (63 & c);
        } else if (c < 0x4000000) {
            *p2++ = 0xF8 | (c >> 24);
            *p2++ = 0x80 | (63 & (c >> 18));
            *p2++ = 0x80 | (63 & (c >> 12));
            *p2++ = 0x80 | (63 & (c >> 6));
            *p2++ = 0x80 | (63 & c);
        } else {
            *p2++ = 0xFC | (1 & (c >> 30));
            *p2++ = 0x80 | (63 & (c >> 24));
            *p2++ = 0x80 | (63 & (c >> 18));
            *p2++ = 0x80 | (63 & (c >> 12));
            *p2++ = 0x80 | (63 & (c >> 6));
            *p2++ = 0x80 | (63 & c);
        }
    }
    nContent = (int) (p2 - pBuffer);
}

UCS (Unicode) to UTF8 conversion, and the reverse, when efficiently coded in C++ clocks in well above 100MB/s on current generation CPUs. If you are getting something much less – enough to be a problem – then there are questions you should ask. The following run spans 1 to 6 byte UTF8 encodings.

Recoding a UTF8 string to a normalized form is a bit slower at 122-87 MB/s.

preston@athena:~/workspace/json-c$ time Release/json
base 00000010 :  229.9 MB/s UCS to UTF8 conversion
base 00000010 :  253.7 MB/s UTF8 to UCS conversion
base 00000010 :  122.0 MB/s UTF8 recode
base 02000010 :  196.8 MB/s UCS to UTF8 conversion
base 02000010 :  177.1 MB/s UTF8 to UCS conversion
base 02000010 :  100.6 MB/s UTF8 recode
base 04000010 :  173.8 MB/s UCS to UTF8 conversion
base 04000010 :  157.1 MB/s UTF8 to UCS conversion
base 04000010 :   89.2 MB/s UTF8 recode
base 06000010 :  174.0 MB/s UCS to UTF8 conversion
base 06000010 :  158.5 MB/s UTF8 to UCS conversion
base 06000010 :   89.2 MB/s UTF8 recode
base 08000010 :  174.0 MB/s UCS to UTF8 conversion
base 08000010 :  154.5 MB/s UTF8 to UCS conversion
base 08000010 :   89.6 MB/s UTF8 recode
base 0a000010 :  174.1 MB/s UCS to UTF8 conversion
base 0a000010 :  156.5 MB/s UTF8 to UCS conversion
base 0a000010 :   89.8 MB/s UTF8 recode
base 0c000010 :  174.0 MB/s UCS to UTF8 conversion
base 0c000010 :  155.1 MB/s UTF8 to UCS conversion
base 0c000010 :   89.6 MB/s UTF8 recode
base 0e000010 :  174.0 MB/s UCS to UTF8 conversion
base 0e000010 :  158.8 MB/s UTF8 to UCS conversion
base 0e000010 :   87.4 MB/s UTF8 recode
base 10000010 :  170.8 MB/s UCS to UTF8 conversion
base 10000010 :  158.2 MB/s UTF8 to UCS conversion
base 10000010 :   89.7 MB/s UTF8 recode
base 12000010 :  174.0 MB/s UCS to UTF8 conversion
base 12000010 :  158.8 MB/s UTF8 to UCS conversion
base 12000010 :   86.5 MB/s UTF8 recode
base 14000010 :  171.5 MB/s UCS to UTF8 conversion
base 14000010 :  153.9 MB/s UTF8 to UCS conversion
base 14000010 :   87.1 MB/s UTF8 recode
base 16000010 :  172.1 MB/s UCS to UTF8 conversion
base 16000010 :  158.1 MB/s UTF8 to UCS conversion
base 16000010 :   87.5 MB/s UTF8 recode
base 18000010 :  172.1 MB/s UCS to UTF8 conversion
base 18000010 :  158.2 MB/s UTF8 to UCS conversion
base 18000010 :   86.9 MB/s UTF8 recode
base 1a000010 :  171.3 MB/s UCS to UTF8 conversion
base 1a000010 :  158.2 MB/s UTF8 to UCS conversion
base 1a000010 :   86.5 MB/s UTF8 recode
base 1c000010 :  169.5 MB/s UCS to UTF8 conversion
base 1c000010 :  158.5 MB/s UTF8 to UCS conversion
base 1c000010 :   86.5 MB/s UTF8 recode
base 1e000010 :  173.0 MB/s UCS to UTF8 conversion
base 1e000010 :  157.8 MB/s UTF8 to UCS conversion
base 1e000010 :   86.1 MB/s UTF8 recode
base 20000010 :  173.1 MB/s UCS to UTF8 conversion
base 20000010 :  158.4 MB/s UTF8 to UCS conversion
base 20000010 :   87.2 MB/s UTF8 recode
base 22000010 :  173.1 MB/s UCS to UTF8 conversion
base 22000010 :  158.2 MB/s UTF8 to UCS conversion
base 22000010 :   88.2 MB/s UTF8 recode
base 24000010 :  173.3 MB/s UCS to UTF8 conversion
base 24000010 :  158.8 MB/s UTF8 to UCS conversion
base 24000010 :   87.6 MB/s UTF8 recode
base 26000010 :  173.8 MB/s UCS to UTF8 conversion
base 26000010 :  158.1 MB/s UTF8 to UCS conversion
base 26000010 :   87.9 MB/s UTF8 recode
base 28000010 :  172.0 MB/s UCS to UTF8 conversion
base 28000010 :  158.7 MB/s UTF8 to UCS conversion
base 28000010 :   87.6 MB/s UTF8 recode
base 2a000010 :  172.0 MB/s UCS to UTF8 conversion
base 2a000010 :  158.1 MB/s UTF8 to UCS conversion
base 2a000010 :   86.1 MB/s UTF8 recode
base 2c000010 :  172.1 MB/s UCS to UTF8 conversion
base 2c000010 :  158.5 MB/s UTF8 to UCS conversion
base 2c000010 :   86.3 MB/s UTF8 recode
base 2e000010 :  173.8 MB/s UCS to UTF8 conversion
base 2e000010 :  158.2 MB/s UTF8 to UCS conversion
base 2e000010 :   89.3 MB/s UTF8 recode
base 30000010 :  170.0 MB/s UCS to UTF8 conversion
base 30000010 :  158.7 MB/s UTF8 to UCS conversion
base 30000010 :   87.2 MB/s UTF8 recode
base 32000010 :  173.8 MB/s UCS to UTF8 conversion
base 32000010 :  158.4 MB/s UTF8 to UCS conversion
base 32000010 :   87.8 MB/s UTF8 recode
base 34000010 :  173.5 MB/s UCS to UTF8 conversion
base 34000010 :  158.5 MB/s UTF8 to UCS conversion
base 34000010 :   88.0 MB/s UTF8 recode
base 36000010 :  173.1 MB/s UCS to UTF8 conversion
base 36000010 :  158.4 MB/s UTF8 to UCS conversion
base 36000010 :   88.4 MB/s UTF8 recode
base 38000010 :  172.5 MB/s UCS to UTF8 conversion
base 38000010 :  158.5 MB/s UTF8 to UCS conversion
base 38000010 :   88.7 MB/s UTF8 recode
base 3a000010 :  170.5 MB/s UCS to UTF8 conversion
base 3a000010 :  158.4 MB/s UTF8 to UCS conversion
base 3a000010 :   87.4 MB/s UTF8 recode
base 3c000010 :  169.3 MB/s UCS to UTF8 conversion
base 3c000010 :  154.7 MB/s UTF8 to UCS conversion
base 3c000010 :   86.3 MB/s UTF8 recode
base 3e000010 :  172.0 MB/s UCS to UTF8 conversion
base 3e000010 :  156.9 MB/s UTF8 to UCS conversion
base 3e000010 :   87.9 MB/s UTF8 recode
base 40000010 :  171.8 MB/s UCS to UTF8 conversion
base 40000010 :  148.9 MB/s UTF8 to UCS conversion
base 40000010 :   83.7 MB/s UTF8 recode
base 42000010 :  173.1 MB/s UCS to UTF8 conversion
base 42000010 :  149.3 MB/s UTF8 to UCS conversion
base 42000010 :   88.1 MB/s UTF8 recode
base 44000010 :  173.7 MB/s UCS to UTF8 conversion
base 44000010 :  157.7 MB/s UTF8 to UCS conversion
base 44000010 :   87.4 MB/s UTF8 recode
base 46000010 :  172.1 MB/s UCS to UTF8 conversion
base 46000010 :  152.2 MB/s UTF8 to UCS conversion
base 46000010 :   87.3 MB/s UTF8 recode
base 48000010 :  174.0 MB/s UCS to UTF8 conversion
base 48000010 :  142.7 MB/s UTF8 to UCS conversion
base 48000010 :   87.8 MB/s UTF8 recode
base 4a000010 :  173.5 MB/s UCS to UTF8 conversion
base 4a000010 :  150.2 MB/s UTF8 to UCS conversion
base 4a000010 :   87.7 MB/s UTF8 recode
base 4c000010 :  174.0 MB/s UCS to UTF8 conversion
base 4c000010 :  158.4 MB/s UTF8 to UCS conversion
base 4c000010 :   88.4 MB/s UTF8 recode
base 4e000010 :  174.0 MB/s UCS to UTF8 conversion
base 4e000010 :  156.9 MB/s UTF8 to UCS conversion
base 4e000010 :   88.4 MB/s UTF8 recode

real	2m0.751s
user	2m0.540s
sys	0m0.190s

The above run is on a AMD Phenom(tm) II X4 955 Processor running at the stock clock rate.

Code is in: http://svn.bannister.us/public/json-c/
(A start on an experiment in fastest-possible JSON conversion.)

Note that I intentionally allow proper conversion of “invalid” UTF8 code strings. I completely understand the reason for the disallowed conversions, and I disagree.

Update: Converted to use pointer arithmetic, rather than array and index. Was not sure pointer math was still a win on current CPUs and compilers. Got a big boost in throughput, so it is!

The lightweight string class is simple enough to be reproduced from memory (nothing over-complicated), but can make quite a difference in performance. I had hoped to make a point.

Since posting, after the initial burst, the articles have collected what looks like a few new hits, every day, for the past five years. Counting hits is a pretty foggy indicator, but it seems possible quite a few folk have read the article (though a relatively small fraction of the programming community). Some may have copied the C++ string class in their code. (What I hoped!)

But there is no sure way of knowing. What is the cumulative impact of this steady trickle of readers?

There is almost no chance I can move the working group is a useful direction. Time to disconnect.

This is all rather discouraging. The HTML working group will proceed. Some of the work is worthwhile. Much (measured by volume of email list traffic) is not. What mix will make it into the generated proposed “standard” is sure to be a mess. Not sure how to change any of this.

My status as an “Invited Expert” is up for renewal. With extreme reluctance … my judgement is that I cannot make a useful contribution, and should disassociate from the HTML working group. Of course, they will continue on the present course, in my absence. My withdrawal makes no difference of significance. There is a fair chance the body of work from this working group will be adopted, imperfect as it is. The existing body of work is … badly skewed by an imperfect process.

Nothing meaningful I can do. The result will be a mess, and will create a mess for years after. Time to disengage.

Funny bit – I do not see a way to force a disconnect.

Three distinct usages I’d like to cover: optimal performance,Windows web server (IIS) interoperable, and webhosting.

In addition, there are three interesting aspects of optimal performance: throughput, scalability, and stability.

For serving static content, I really like the model of a single-threaded non-blocking web server, of which thttpd was an early example, and for which the C10K question clarified the need. A small/simple web server has a much better chance to being very reliable. With the single-threaded non-blocking model, massive scalability is possible.

For serving dynamic content, I really like the isolation and load distribution possible with the FastCGI model (or the like). Dynamic code tends to be complex. Javascript interpreters are complex. Complex code tends to fail more often. Complex code can use more compute throughput than possible on a single box. For intranet applications, a single front-end web server is often preferable, and load distribution via FastCGI offers more headroom. All of which tends to argue for the FastCGI model, with isolation from the front-end web service, and potential distribution of load across more than one machine.

For the widest possible usage, in additional to optimal deployments (when there is no restriction on the front-end web server), the engine on which the application runs should be deployable behind IIS (for Windows-only organizations), and at common web-hosting services (like Dreamhost). Microsoft’s recent support of FastCGI with IIS is a big help.

At that time (three years back), none of the solutions were really optimal – and in fact were pretty far from optimal. The Java-based RhinoJavascript interpreter was easiest to embed, but failed the webhosting case. The C++ based JavaScript interpreters were a pain to embed, and offered good (but not great) performance.

Fast forward to the present, and Google offers the V8 JavaScript Engine that offers great performance, and is easy to embed. (Google as the good guys, riding to the rescue once again … you’d think they have white hats superglued to their brains.) Suddenly we have lots of projects embedding the V8 engine. In addition, seems most all the single-threaded non-blocking web servers have picked up support for FastCGI.

Oh … and I am pretty much fed up with the Java Servlet model. After considerable time with the problem, I am of the opinion that the servlet model chose the wrong abstractions, and this makes for awkward solutions. (Of course, the servlet model appeared very early in the history of web applications, so the mistake is easy to understand.)

Which means the model offered by node.js makes a lot of sense. I like the notion of a naked node (running JavaScript on the V8 engine) performing request dispatch without any extra layers or abstractions. The main lack with node.js is the ability to work via FastCGI (and thus no means to be deployed behind IIS on Windows).

But there are as yet items unresolved and/or unclear.

• Projects like v8cgi offer the V8 JavaScript Engine connected via FastCGI.
• The node.js project offers a single-threaded non-blocking web server … but can it work behind FastCGI?
• Is the environment for server-side JavaScript the same (or sufficiently similar) between node.js and v8cgi?
• Comet is still a question. Can FastCGI work well with long-outstanding requests from applications?

The good news is that we seem a lot closer to attractive and well-supported server-side JavaScript for web applications … but it seems we are not quite fully there, as yet.

Went with the Intel adaptor when I ordered this notebook, in part as Intel seems to be actively supporting the development of Linux drivers. In practice, my laptop wireless connection is mostly unreliable, and often near-useless.

I used to think the problem was interference from other nearby wireless routers, but my daughter’s laptop seems to work well when mine does not. I have a simple “bounce” test for network performance. My daughter’s cheap Toshiba, on the same “bounce” page, gets steady/fast performance when side-by-side to my laptop … which gets slower/unsteady/unreliable performance.

Not specific to the wireless router or crowded neighborhood either – I get similar poor performance when at my father’s place in Colorado (different brand and generation of router, and fewer/further neighbors).

After the prior results, I wanted to see if a less general-purpose implementation would perform better. Given enough advantages, could a Trie out-perform a hash table? (Again, this visits to some extent the question asked in a prior discussion.)

For the current exercise I built two Trie implementations. The LinkedTrie is cheap to build, minimal in use of memory, but not especially fast to access. The FixedTrie implementation should be pretty close to optimal in access time (for a Trie), but expensive to build (in fact the FixedTrieBuilder transforms a LinkedTrie into an optimized FixedTrie).

As before, the sources are in (as an Eclipse project): http://svn.bannister.us/public/Trie/.

The performance numbers make sense. The older TallTrie and WideTrie implementations (that traded increased memory use for speed) are indeed faster, though the LinkedTrie uses much(!) less memory. The new FixedTrie is fastest (hurrah!) and uses the least memory (a shade less than LinkedTrie).

But even the FixedTrie is slower than a generic hash table, with read-only access about 3 times as expensive.

Sample measurements…

=== words
21 ms - 98569 words loaded

=== hash
3935/second - hash map loaded {4007 ms, 15771040 operations = 254 ns/op}
8554/second - hash map re-loaded {4010 ms, 34302012 operations = 116 ns/op}
10768/second - access each item in hash map {4000 ms, 43074653 operations = 92 ns/op}

=== trie (linked)
870/second - loaded trie (linked) {4078 ms, 3548484 operations = 1149 ns/op}
1204/second - re-loaded trie (linked) {4010 ms, 4829881 operations = 830 ns/op}
1431/second - access each item in trie (linked) {4063 ms, 5815571 operations = 698 ns/op}
98569 slots of trie (linked)
225791 nodes of trie (linked)

=== build fixed trie

=== trie (fixed)
85779/second - loaded trie (fixed) {4000 ms, 343118689 operations = 11 ns/op}
87652/second - re-loaded trie (fixed) {4000 ms, 350609933 operations = 11 ns/op}
3438/second - access each item in trie (fixed) {4013 ms, 13799660 operations = 290 ns/op}
98569 slots of trie (fixed)
225791 nodes of trie (fixed)

=== trie (wide)
65/second - loaded trie (wide) {4502 ms, 295707 operations = 15224 ns/op}
1659/second - re-loaded trie (wide) {4040 ms, 6702692 operations = 602 ns/op}
1672/second - access each item in trie (wide) {4007 ms, 6702692 operations = 597 ns/op}
41533124 slots of trie (wide)
225890 nodes of trie (wide)

=== trie (tall)
682/second - loaded trie (tall) {4043 ms, 2759932 operations = 1464 ns/op}
1733/second - re-loaded trie (tall) {4036 ms, 6998399 operations = 576 ns/op}
1913/second - access each item in trie (tall) {4019 ms, 7688382 operations = 522 ns/op}
6982694 slots of trie (tall)
446075 nodes of trie (tall)

=== string to UTF8 conversion
1393/second - word to UTF8 (stock) {4033 ms, 5618433 operations = 717 ns/op}
9585/second - word to UTF8 (fast) {4000 ms, 38343341 operations = 104 ns/op}

(Note that the FixedTrie ignores re-load, so the times for load and re-load are bogus.)

I suspect a C++ Trie implementation could do a bit better … but not necessarily outperform … compared to hash tables.

Looks very much like even a specialized read-only Trie cannot match the performance of a generic hash table (at least in Java).

After getting stuck on the latest problem, I finally built a “PrintExplorer” test program. The sources are up on Kenai, and on my website. In both cases the sources are in a Mercurial repository, and I “push” changes to both (one of the nice features of Mercurial). The project was built using NetBeans 6.7, and you can download the built PrintExplorer.jar file.

Run as usual:

java -jar PrintExplorer.jar

(On Windows you can just double-click the JAR file.)

I filed two reports (so far, with review ids: 1660182 and 1660234) with bugs.sun.com, though for all I know that may be a black hole (or not … I have no idea). With the OpenJDK and JDK7 work there may be better places to report/record this … I do not know. (If you know a better path, please let me know.)

Found so far:

• PrinterJob.getPageDialog() makes invalid change to page size. (log)
• Toolkit.getPrintJob() changes JobAttributes.getPrinter() to default(?) printer. (log)
• Quarter-sized pages sometimes printed on an HP DeskJet 5550 on Windows 7. (log)

I am sure there are more. :/

Note that PrintExplorer always generates a log file. The log file contains the parameter values and print operations performed. (You need to be acquainted with Java Printing, otherwise the test program will be entirely obscure.) If you can reproduce a problem, then you can see in the log file exactly what you did, and what was returned. (Useful when you lost track of the exact sequence of steps.) When you send a log file, anyone can verify the exact sequence of steps you followed (a mistake in the problem description can be caught).

At this point the DocPrinterJob works for an empty Doc and a Pageable Doc, but not for a Reader Doc. (Unclear if the last is my bug, or a Java bug.)

Update – 2010.02.01
Today the filed reports became public-visible bugs 6921596 and 6921597.

My original query as represented in Canon’s support system (and trimmed):

Original Message Follows:
————————-
Email Support Form Message
Product Type: PIXMA MX860
Product Serial Number: absx22394
Date of Purchase: 11/18/2009
First Name: Preston
Last Name: Bannister
Address: 14 Vallecito
City: Foothill Ranch
State: CA
Zip: 92610
Phone Number: 9495880872
Email Address: preston@bannister.us
INQUIRY: Need drivers for 64-bit Ubuntu (Debian Linux).
Found the 32-bit drivers and sources on the Canon European site, but the sources are incomplete, so I cannot compile a 64-bit version.
Either compiled 64-bit drivers, or a complete set of sources would be acceptable.

The response was pretty predictable, if not exactly addressing the question as posed.

On Fri, Nov 20, 2009 at 5:09 AM, Canon Support – MFP <mfp @cits.canon.com> wrote:

Dear Preston Bannister:

Thank you for contacting Canon product support. We value you as a Canon customer and appreciate the opportunity to assist you. We regret the issues experienced with locating Linux drivers for the MX860.

Canon USA does not support Linux with its consumer products. The only recommendation I can offer would be to check the various Linux enthusiast websites to see if anyone has successfully attempted writing a third party driver.

We apologize for any inconvenience. Please let us know if we can be of any further assistance with your MX860.

Sincerely,

James
Technical Support Representative

Since there is always a chance (if small) of a constructive outcome, I tried again.

What I did was find that the Canon Europe and Canon Australia sites distribute Canon-developed Linux drivers for the MX860 (and other printers).

The software is pretty well done, has an active documented development history that goes at least to the year 2000, and is distributed as both binaries and source.

Odd that Canon USA does not distribute Canon-developed drivers (in which Canon has made a significant investment).

This is a case of so-close-but-not-quite there. The effort required for Canon to have a complete offering is pretty small, if someone can find the right branch of the company hierarchy to nudge.

Any chance of finding who within Canon sponsored development of the existing drivers for Linux?

The last response, which sounds dangerously close to “Fuck you. Go away.” (Though I would gladly be proven wrong.)

Dear Preston Bannister:

We appreciate your continued correspondence regarding Linux drivers for your PIXMA MX860.

I have forwarded your comments to Canon USA through our Customer Feedback process. This process allows us to capture important feedback from our valued customers. As we constantly strive to improve our products and services, your comments are vital to our continued success.

We hope this information is helpful to you. Please let us know if we can be of any further assistance with your PIXMA MX860.

Thank you for choosing Canon.

Sincerely,

George
Technical Support Representative

Had to try….

Billed To:
preston@bannister.us
Preston Bannister
14 Vallecito
Foothill Ranch, CA 92610

Order Number: MGLN2G836G
Receipt Date: 11/19/09
Order Total: $4.99
Billed To: MasterCard …. 1234

Item Number
1

Description
변환기, v1.0, Seller: Wang Xi|299225365 (4+)

Unit Price
$4.99

Write a Review
Report a Problem

Subtotal: $4.99
Tax: $0.00
Order Total: $4.99

Please retain for your records.
Please See Below For Terms And Conditions Pertaining To This Order.

Apple Inc.
You can find the iTunes Store Terms of Sale and Sales Policies by launching your iTunes application and clicking on Terms of Sale or Sales Policies

Answers to frequently asked questions regarding the iTunes Store can be found at http://www.apple.com/support/itunes/store/

I do not have iTunes installed. Both my desktop and laptop are running Ubuntu Linux. Also I cannot read Chinese Korean, so I do not know what was ordered. The “Report a Problem” link asks me to install iTunes.

I am guessing the iTunes store has a (big!) problem.

Update:
No charges have appeared on my credit card (so far).

I eventually found a “support” link for the iTunes site where you fill out a form to send email. Had to lie a bit as you cannot complete the form without indicating the OS on which you are running iTunes. Linux is not offered as a choice. The first email from Apple was not useful (“run iTunes to check on your account”, grrr). I replied with a copy of the receipt, and they did … something.

If no credit card charge appears, my guess is the Apple caught the problem part-way through.

Got an email from another iTunes customer who also got a receipt with the exact same mysterious item. Interesting.

Update (2):
The charge just appeared on my credit card, so it is not clear the exchange with iTunes Support addressed the problem.

Update (3):
Shortly after my last email (where I again described the problem, and added the other guy’s receipt), Apple Support sent an unexpected email refunding the charge to the other guy! Right. Half-successful?

China to investigate U.S. car subsidies
China is preparing to launch a trade investigation into whether US carmakers are being unfairly subsidised by the US government, according to people familiar with the matter.

The move comes at a time of heightened trade tensions between the two countries after the US imposed duties on Chinese tyres last month. Many warned this would prompt Beijing to retaliate.

Interesting. Is this shallow, or deep?

China is using extraordinary measures to keep the dollar high and the yuan low. This has huge effect on trade. Words from the US to China have no leverage, at least compared to China’s near-term benefit from continued exports.

What has leverage? An import duty on tires … really … why tires? Is there some shallow US domestic political motivation? Or is this an opening move with enough lead time prior to Obama visiting China?

The Chinese government used to subsidize all sorts of industries, and they probably still do to some degree. At least some of that goes into exports. Tagging the US for subsidizing an industry – one that likely does more imports from China than exports to China – is odd. If the yuan rises relative to the dollar, then US exports to China become more competitive. Bit of a doubled impact on the Chinese economy (less exports and more imports). Discouraging imports from the US through import duties could lessen that impact. Is this a pretext for the logical counter-move (but not a strong move). Again, is this shallow or deep?

In this game, China is on the wrong end of the lever. Less trade between the US and China is likely to hurt China far more than the US. There may be some point in the future when they can absorb the loss of exports to the US. Is China at that point now? Or more relevant, do the Chinese leadership believe China is at that point?

My bet is that the Chinese leadership is too conservative, too cautious to want to risk too much change. China is a country of “rule by men”, not “rule by law”. Obama meeting with the Chinese leaders would be the point where – privately – the game is made plain … if this the deeper game. If China allows the yuan to rise after the meeting – that would be a big clue. (Likely China will keep buying dollars, but at a lesser rate – a sort of compromise.)

Is this shallow, or deep?

The problem is buttons. A change in GTK+ 2.18 broke Eclipse, and anything based on Eclipse.

There are a number of bug reports tracking this problem. On the Eclipse site look at:
Bug 291257 – [Widgets] Buttons functionality problem with GTK+ 2.18

There is a work-around offered on the above ticket (setting the GTK_NATIVE_WINDOWS environment variable). It seems to work for some. For others – myself included – it causes another severe problem (for which there are other bug reports).

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f30eb1e7f7a, pid=4766, tid=139848919398672
#
# JRE version: 6.0_16-b01
# Java VM: Java HotSpot(TM) 64-Bit Server VM (14.2-b01 mixed mode linux-amd64 )
# Problematic frame:
# C  [libpango-1.0.so.0+0x24f7a]  pango_layout_new+0x2a
#

Strictly speaking, this is a bug in GTK. A new version of GTK shipped that was meant to be compatible, but instead broke existing applications (at the very least those based on Eclipse). The application use of GTK might (or might not) be considered in some way improper, but practical considerations should be dominant.

Fixing this in GTK will in one step remove the problem in all currently impacted applications. Fixing this in Eclipse will only help new and newly updated Eclipse installations. Existing Eclipse-based applications will continue to be impacted until (and if) updated by their respective developers.

Other related/duplicate bug reports:
Buttons in Eclipse not working correctly with GTK+ 2.18.1-1
Eclipse loses button events with swt-gtk in karmic
GDK_NATIVE_WINDOWS=1 still required for some dialogs
Breaks Default button in some eclipse dialogs (and rcp apps)

What’s Wrong with the Culture of Wall Street? – TIME
Before this crazy crash of 2008, bankers always landed on their feet, almost always. Job insecurity isn't the same thing for the average American worker. They often experience downward mobility or don't land on their feet.

New Device Extracts Oxygen from Moon Rocks | Universe Today
Fray anticipates that three reactors, each a meter high, would be enough to generate a ton of oxygen per year on the Moon. Three tons of rock are needed to produce a ton of oxygen, and in tests the team saw almost 100% recovery of oxygen, he says.

The comments got the math badly wrong. Poking around the web I got some rough numbers:

• ~0.25 liters / minute – oxygen consumed for human at rest.
• ~5 liters / minute – oxygen consumed for athlete / heavy exercise.
• ~1.43 grams / liter – oxygen gas at Earth sea level pressure.

Barring any dumb math errors on my part, that translates to….

• ~636,000 liters / year – the ton of oxygen produced by the above-mentioned generator.
• ~131,500 liters / year – oxygen consumed by a person at rest.
• ~4.8 person-at-rest-years of oxygen generated every year.

Assuming that a “real” space habitat would also have the means to recycle oxygen from exhaled CO², that is a lot of oxygen!

What is the real need for that much oxygen? Outside use as rocket fuel (or other manufacturing process), the main need would be to supply air for expanded living space. Assuming ~3 meter ceilings, we need ~3000 liters air for each square meter of living space. So one generator would allow expanding the living space by ~2000 square meters / year.

Not bad. And that is just one reactor (and essentially a prototype, at that).

Only one part of the equation (for habitat supply needs), of course. For practical purpose, would be nice to also have some nitrogen in the air. Not sure where that would come from. Do any moon rocks contain nitrogen? Water is also needed. Barring any fortunate discovery of water on the moon, we could synthesize water given a source of hydrogen. Do any moon rocks contain hydrogen?

• Conversations are a first-class concept, and each message is a part of a conversation.
• Each message can be presented by a different application.

Because Wave is on the web, the set of applications is come from all applications in the world (on the web).

E-mail clients can present “threads” that are something like conversations, but this requires a bit of guessing and fakery on the part of the client, and is easily confused. IM clients can present a log of messages – that looks something like a conversation – but does not work when you switch between devices. Given that email and IM servers know nothing about conversations, the clients do pretty much the best they can. If you know about conversations on the server, you can do a better job on the client.

When conversations are a first-class concept, there are lots of new possible functions.

On the flip side, the Google Wave folk have made some (common) missteps.

• Using new names for existing notions is sometimes useful, but more often not. Calling a conversation a “wave”, calling items (messages) within a conversation “blits” … I find this more confusing than useful. This reminds me more of talk from marketing folk than engineers.
• It’s Talk, on the web. As a programmer, I get it – this is kewl! Writing a web application that allows all participants to see each character typed – is tricky. As a demo of tricky programming, this is sure to impress. As a useful feature in a general use application – not so much. Character-at-a-time message traffic is a huge load increase on the server, and a disservice to the users. Think about:
  • Locus of attention – the on-screen activity from updates to other users’ incomplete messages is sure to distract.
  • Quality of expression – how often have you started writing a message, then revised what you said and how you said it, before sending? Do you really want everyone to see your missteps … always?

  Note that both are a disservice to all participants – cool in a programming demo, not cool in a deployed application.

Other bits….

• It looks like Outlook. This was new and cool in 1997, but not so much now. Not at all sure this layout is efficient, especially given the changes in most-common screen layouts. In 1997 you would target 640×480, 800×600, and 1024×768 on 14″ to 17″ as most common screen sizes. Today you are looking 1280×800 on 14″ or 15″ as your minimum size, and 1920×1080 at 20″ or bigger as common. Present and future screens have not so much new height, with lots of new width – in fact too much new width for text content. Seems a different presentation is in order.
• Mixed applications in one web page is cool and exciting as ActiveX was in the web browser in 1996. Viewed in terms of the good things it can do, ActiveX allows an open-ended mix of applications to present within a web page – very cool. The Microsoft folk were blindsided (as constructive, well-intentioned folk) as they did not consider all the bad things that could be done (and with ActiveX, very bad things are possible). The Google folk have lots of history to make them wary, and the scope for harm is much smaller, so likely little bad stuff get through. Still – do not underestimate the black hats. There are bound to be some exploits (though likely few and quickly suppressed).

In sum, a decent start on a great idea. I’d want to be pretty ruthless about overhead that takes height away from the conversation view. I’d drop the per-character updates. I’d drop the use of funky names for existing ideas. None are fundamental problems, and all are easy to address.

Fixing up an old Java desktop application for viewing old “green-bar” reports. Should be pretty simple – the reports are all fixed pitch text. Given I like to do things that “just work” (from the users perspective), I’d added live “smart” definition of controllable font-auto-sizing, empty page suppression, and margins. Given that old printed reports may be formatted for paper sizes different than in your desktop (or network) printer, this was slightly tricky, but only that.

Of course, many of those old reports – meant for old line-printer green-bar paper – are best printed in landscape orientation. To my surprise, I find when in landscape orientation, the return from PrinterJob.pageDialog(PageFormat) consistently alters the margins, even when no changes were made, and the error is cumulative.

Um, what?

At first I assumed the error was in my code, but this bit of logged output caught the problem:

PageFormat validate::
	paper dx: 792.0 dy: 612.0
	margins top: 72 left: 72 bottom: 72 right: 72
	orientation : LANDSCAPE
PageFormat after pageDialog::
	paper dx: 794.0 dy: 613.0
	margins top: 72 left: 74 bottom: 73 right: 72
	orientation : LANDSCAPE

The first is the contents of the PageFormat before calling PrinterJob.pageDialog(PageFormat), and the second is the new PageFormat returned. Note the +1 dy bump and +2 dx bump to the page extents (and the matching errors introduced to the left/bottom margins).

The code I used to remove the error:

            controller.pageFormat = job.pageDialog(controller.pageFormat);

            // Work around an interesting(?) bug in PrinterJob.pageDialog().
            // For some reason the returned Paper is over-sized. (Why? Dunno.)
            // Use the Paper returned from PrinterJob.defaultPage() to get a proper size.
            // If the Java folk remove the bug, this code should change nothing.
            PageFormat pf = job.defaultPage();
            Paper paper1 = controller.pageFormat.getPaper();
            Paper paper2 = pf.getPaper();
            double x = paper1.getImageableX();
            double y = paper1.getImageableY();
            double dx = paper1.getImageableWidth();
            double dy = paper1.getImageableHeight();
            paper2.setImageableArea(x,y,dx,dy);
            controller.pageFormat.setPaper(paper2);

The bug was encountered using Java 1.6.0_15 (64-bit) on Ubuntu (Linux 2.6.28-15-generic #52-Ubuntu SMP Wed Sep 9 10:48:52 UTC 2009 x86_64 GNU/Linux).

The results are not encouraging. From a test run:

=== words
14 ms - 98569 words loaded

=== hash
3837/second - hash map loaded {4007 ms, 15376764 operations = 260 ns/op}
7953/second - hash map re-loaded {4003 ms, 31837787 operations = 125 ns/op}
12958/second - access each item in hash map {4001 ms, 51847294 operations = 77 ns/op}

=== trie (wide)
343/second - loaded trie (wide) {4022 ms, 1379966 operations = 2914 ns/op}
2089/second - re-loaded trie (wide) {4010 ms, 8378365 operations = 478 ns/op}
2165/second - access each item in trie (wide) {4005 ms, 8674072 operations = 461 ns/op}
41533124 slots of trie (wide)
225890 nodes of trie (wide)

=== trie (tall)
2254/second - loaded trie (tall) {4022 ms, 9068348 operations = 443 ns/op}
2446/second - re-loaded trie (tall) {4029 ms, 9856900 operations = 408 ns/op}
2429/second - access each item in trie (tall) {4017 ms, 9758331 operations = 411 ns/op}
6982694 slots of trie (tall)
446075 nodes of trie (tall)

=== string to UTF8 conversion
4708/second - word to UTF8 (stock) {4019 ms, 18925248 operations = 212 ns/op}
10503/second - word to UTF8 (fast) {4007 ms, 42088963 operations = 95 ns/op}

The test data is a file of 98569 words found in Linux installation. The words are read from a file, and loaded into an array of strings. The words are loaded into a HashMap (a library class) for comparison, and loaded into both a “tall” and “wide” Trie (differing space/time trade-offs).

The notion used to generate a “fast” Trie is for each node in the Trie to index off a byte in the UTF8 encoding of key String. Along the way, found that the stock String to UTF8 conversion to be a bit slow, so wrote a faster version. Even with the faster encoding, measured Trie performance is not good.

(Bit of a shock to find the stock String-to-UTF8 conversion slow in Java, as this is used a lot – especially in web applications – and may be significant enough to effect benchmarks. Java has been around long enough, so that I expected better. In fact, I expected UTF8 to be in native code in the JVM. Byte-bashing code is better suited to C/C++.)

If strings were already in UTF8 (or ASCII) format – as in C/C++ code – a similar Trie implementation would perform somewhat better, but still use too much memory. Whether a Trie in C++ could more closely approach the performance of a hash table is a bit of an open question.

Not sure why the “wide” Trie (using one node to index one UTF8 byte) is slightly slower than the “tall” Trie (using two nodes to index 4-bits each from one UTF8 byte). Expected the “wide” version to be faster. Suspect the larger memory use is enough to bust the CPU cache, and on smaller data the “wide” variant may be faster. Also saw a lot of jitter in the times – presumably due to Java GC activity (garbage collection) – though the relative results were consistent.

Clearly this “fast, general-purpose” Trie implementation is not overly useful – at ~5 times slower than a hash table, and huge use of memory. Alternate forms can be more space-efficient, but are likely to be slower.

As a guess, a Trie is not going to find much use in my code.

There are good reasons to take a run at the problem. Physics is changing what we can expect from future computers. Starting a few years back, and barring any unexpected shifts in technology, the rate at which a CPU can process a single instruction stream will increase only slowly. The economics of chip fabrication allow us to build a CPU with multiple cores. The physics of power consumption tell us that we can get more computing done per watt with slimmer cores at slightly lower clock rates. All of which argues for fabricating CPU chips with a slowly increasing number of cores, and slowly increasing clock rates.

All of which means that to make full use of present and future CPUs, there has to be a lot of concurrent computing.

Concurrency in programming is tricky, and often got wrong. This is nothing new. My first job out of college (so many years ago) was to work of an Ada compiler (a computer language with direct support for multi-threaded interaction) on a product (the Pascal Microengine) that had thread support built into the CPU’s microcode. There was then much talk of how to do concurrent programming.

What we learned then and in the time since is that fine-grained multi-threaded concurrent programming is tricky, and very easy to get wrong. For the bulk of programs and programmers, there is very limited need for this sort of concurrency. All things considered, this is probably a good thing.

Tim Bray – and the folk responding to his post – are mainly focusing on a programming language for concurrent programs. I suspect this is a mistake. Maybe some new (or newish) programming language will make bug-free concurrent programs easy, but I do not think this is likely.

I think we already have the bulk solution. Web-scale applications (at least those that work well) make use of large numbers of CPUs, with a huge amount of concurrent execution. Web-scale programming is mostly about swarms of small-scale execution single-threaded programs (not necessarily small), well isolated from other threads. Many of the attributes Tim lists are true – or mostly-true, or should-be-true – of web scale applications.

Clearly the web-scale application approach does not work for all applications – though it may work for more than you expect. There are always going to be some applications that need fine-grained threading … but I suspect this group is very small. For the bulk of programming we want to allow for massive concurrency, but well-isolated and coarse-grained.

Why the focus on programming languages that support fine-grained concurrent programs? What problem – in the application space – does that solve? Are more than a tiny fraction of applications in that space?

My answer to Tim’s question is to point at the concerns of web-scale applications and cloud-computing. The problem does not drive an interest in new programming languages. The answer to large-scale concurrent execution is – for most applications – large numbers of single-threaded programs, responding to requests. Tim’s list of characteristics – in part – is useful for that sort of programming. Editing down the list:

• Message passing
  and Dataflow
• Immutable data,
  and Transactional memory,
  and Tuple space

(Have to admit, I am not sure the above grouped notions are distinct, when viewed at this level.)

For the bulk of programmers and applications, the main needed change is finding the simplest possible adaptation to the needs of web-scale programming. Once done, concurrency and threading are solved problem. We do not want or need fine-scale single-name/address space multi-threading – that way lies madness. We do need well-isolated single threads, in mass numbers, cooperating across web-scale process, machine, and network boundaries.

In contrast to the MSNBC and WSJ sites, the videos came across as politically neutral, and not so much into the financial orthodoxy. Seemed aimed roughly at the interests of middle management.

Not getting a lot of insight, but also not as much propaganda.

In comparison to MSNBC, the voices were at a normal level, and the sound effects hinted at “we are tasteful and sophisticated”.

Recurring memes I could identify …

• Republicans are good, Democrats are bad.
• Government is always less efficient than business.
• Government – not business – is somehow responsible to the current economic face-plant.
• Not sure how to describe … the Wall Street variant of the New England mentality?

Not sure that the videos are representative of the written/print edition of the WSJ.

Has the WSJ changed since it was sold?

First impression: Why all the shouting? Kind of the equivalent of TYPING IN CAPITAL LETTERS ALL THE TIME!! Add the dramatic special effects noise. WITH BOLD AND UNDERLINE!! Did someone conclude that SHOUTING the news made it more believable? Noisy. Should change the name to the Shouting News Network (SNN).

Second: They sure do a lot of product placements disguised as news reports.

Third: How do they pick their “outside-expert” guys to supplement the usual talking-heads? Watching, I often could not figure out what criteria lead to their choices.

Fourth: There must be some company edict to use the words “monetize” and “free market” a lot. Did not seem to matter so much whether it made sense in context. Caused the “outside-experts” some grief, as they tried to come up with sensible responses to not-quite meaningful questions.

Kind of funny, in a really odd way. Don’t think I’ll be watching MSNBC regularly.

Should try the same experiment with the other news networks.

This is out of our fields, but interesting.

http://www.edn.com/article/CA6686259.html?nid=2435&rid=8150303

I liked the one comment: “the generally horrible code from the Microsoft world”.

My characteristically reserved response:

Yes, it is usually cute when hardware guys talk about software.

As I remember, the result came from the large number of branches found in “the generally horrible code from the Microsoft world that dominated tasks in those days”. Except this characteristic is common to most modern code, not just in “Microsoft world” code. It comes from complex behaviors – an increasing and increasingly common characteristic in contemporary code. Outside a small set of niche applications (with large number crunching), you are going to have a lot of branches. Branching limits the value of concurrent instruction dispatch, and speculative execution (as we might guess). Results pointed at more than ~3-way dispatch as on the wrong side of the “knee” of the curve.

I think the chip design offers a hint as to their expected application domain:

Each of the Power7’s cores has 12 execution units, including two each load/store and fixed-point, four double-precision floating-point, and a decimal floating-point unit. The core can dispatch six instructions per cycle.

With four floating-point execution units, my guess is that they are aiming at niche mass number-crunching applications, not general purpose applications. Combined with the large shared-cache, these should do well on problems that allow highly concurrent bulk number crunching. Might see a lot of these to run weather simulations (to predict Global Warming), and some sorts of large engineering design calculations/simulations.

As a general purpose CPU, my guess is that benchmarks (when and if they appear) will be unimpressive.

Add to this the fact that most big new web applications are moving (have moved, in fact) to a no-shared-memory model – lots of single-board nodes with 1 to 8 CPU sockets, no shared memory between processes, connected via ethernet to large-scale high-performance redundant datastores (specialized database/filesystems).

Which makes the really cool design aspects of the Power7 pretty much useless. This is not the sort of CPU that would ever see large-scale deployment in a web-service datacenter.

Too bad the writer of the article does not have a clue about software, but no surprise from a hardware magazine.

Oh wait! I take it all back! This CPU could be terrific at the more compute-intense computer games! Perhaps Microsoft will pick this up for the next XBox? Super high frame rates in MegaBlaster2010! … except highly concurrent shared memory applications are a real bitch to write, even worse to debug, so better make that MegaBlaster2012, or 2013, or 2015, or … (hey! they never did ship!?!).

… maybe not.

Was out in the garage last night, planning the next bit, when my daughter popped out of the house. “Oh, you’re thinking again, aren’t you?” She was not surprised to find me sitting and staring for an hour, before starting on the next bit.

Seems I use roughly the same approach whether cooking, building things out of wood and metal, or writing software. When cooking I never use recipes, I am always changing something, and usually get good results. When building I never use plans (other than occasional rough sketches), will think through the problem and build the next bit in my mind, before starting manufacture – and usually get good results. When writing software I do not have much use for design documents or formal processes, but cannot start until I have thought through the problem, and usually get good results.

In each case I can visualize the end result before I start. In each case I am working without plans (written plans or plans from others), yet … I always have a plan.

As I learned back in school, this pattern of work does not work at all for some of my peers. My upper-division Physics lab class was the first clear example. The lab was – at base – one student alone in a room with a pile of equipment, and a set experiment to complete in a couple weeks. The lab had a new instructor when I took the class, and he was not familiar with the equipment. Also, for some reason, none of the equipment worked. So with little or no instructions, I had to figure out what the equipment did, how it should work, and what needed to be fixed.

Getting the experiments to work was both frustrating, and fun!

Midway through the course, the instructor assigned another student to work with me, as this guy was not doing well by himself. I knew the guy, counted him as a bit of a friend, and thought he was at least as good as I at the theory classes. But in the lab, he drove me nuts. I needed silence so I could visualize how things should work. For the parts I did not understand well enough to visualize, I would disassemble and play with the equipment, until I better understood how it worked.

I suppose it looked to the other guy like I did things in no particular order, while ignoring anything he said. After an indeterminate period of seemingly aimless activity, he’d suddenly hear “Ah … I get it!”, followed by an ordered, logical explanation, and a burst of activity – after which the equipment would (usually) work.

Bet I drove him a bit nuts, as he did to me.

Also, up to that point in the lab class, I thought I was doing poorly. Most times – when the instructor dropped by – the equipment was not working, and I had no coherent explanation (or so I felt) of what to do next. Once the equipment was working, I could – in short order – perform the experiment, and move to the next lab.

Bet also my more present co-workers see something similar – an indeterminate period of seemingly aimless activity, followed suddenly by a burst of activity and plans. Probably drives some folk – of differing nature – slightly nuts.

So my daughter caught me planning, again …

If you visit towns in western Colorado and south-eastern Utah, you can see a boom-and-bust pattern of development for the 1970’s and 80’s. In the 1970’s the notion of extracting shale-oil got a lot of investment, and caused a modest boom in development. The low oil prices of the 1980’s killed investment in (then uneconomic) shale oil extraction, and development slowed. Investment in shale-oil production since that time has proved a bit slow, as investors are not entirely confident the world oil prices will stay high enough for profitable production.

Keeping dollars in America means more productive economic activity within the country. Domestic oil production should help stabilize consumer prices. How can we fix the cost of imported oil high enough so that domestic production stays profitable?

We could place a tax on imported oil, but this seems likely to become a mess. How would we tax different grades of oil? What about refined products? Is politics likely to grant different rates to different nations and oil companies? We would need a new bureaucracy to set tax rates, would most likely get things slightly wrong, and create new nonconstructive opportunities for politics and corruption.

Better to not create new opportunities for bad behavior.

A far simpler approach would be to place a large retail tax on gasoline, and offer an equally large tax credit for domestic oil production. The idea is to have no net effect on the retail cost of gasoline produced from domestic sources, but make non-domestic sources more expensive – enough so that domestic production is profitable, and investment in domestic production is safe.

I once heard that shale-oil production is profitable when the world oil price is above $70 per barrel. The low point in world oil prices (that killed development of shale-oil production) was around $30 per barrel. A tax of about $40 per barrel (or $1 per gallon of gasoline) should be enough to insure that domestic production is assured profitable. Given assured profits, all feasible forms of oil production (including sustainable production from biologic sources) should be able to attract the level of investment that they need.

Put differently, by making one simplest-possible change in the ground rules, we can let the market solve the production problem, without more-complicated government intervention.

The main limitation of the Pearson hash is that it only computes an 8-bit value (i.e. the range 0..255). For some problems a bigger range of values would be good. The means offered in Pearson’s original paper (as I recall – threw out the old magazine long ago, and the paper is not on the web) was to run the algorithm more than once to add additional 8-bit chunks to the hash.

Just now I’m feeling like a bit of an idiot, as it seems the Pearson hash could be easily extended up to 16-bits (or a bit beyond) with a ridiculously small change.

The original Pearson algorithm:

static unsigned char T[] = {
    // The values 0..255 shuffled into random order.
};

static inline unsigned hashOf(unsigned h,const char* s) {
    for (int c = *s++; c; c = *s++) {
        h = T[h ^ (255 & c)];
    }
    return h;
}

The algorithm modified to generate a 16-bit hash:

static unsigned short T[] = {
    // The values 0..65535 shuffled into random order.
};

static inline unsigned hashOf(unsigned h,const char* s) {
    for (int c = *s++; c; c = *s++) {
        h = T[h ^ (255 & c)];
    }
    return h;
}

Note that only the type and size of the table T changed!

With the large cache sizes of current CPUs, the 128KB table will fit nicely into cache, and perform very well. On CPUs with smaller caches, you might want to use a smaller table. Also clearly the table size for a 32-bit hash would be huge. You might want some sort of test for degenerate patterns in T[] (though I’d guess the probability of getting a bad table is very small).

This notion is so simple, I am a bit surprised that no one (so far as I can tell) suggested this approach before.

For large numbers of keys, each bit added to the hash value saves the equivalent of a key comparison. Adding 8-bits to the key saves 8 key comparisons. If the expense added to the hash function is less than the cost of one key comparison (certainly true of string compares), then you are ahead from the first bit added.

I had looked at this over a year ago, but the SIPE support for Pidgin did not support the options needed for the IM service as configured by the company. On checking again, it seemed possible that the current version might work, but I needed the most recent version of Pidgin and the SIPE plugin.

So … for future reference.

Building Pidgin and the SIPE plugin from sources on Ubuntu

1. Install the dependencies needed to build from sources.

   sudo apt-get build-dep pidgin pidgin-sipe
   

2. Remove any existing installation.

   sudo apt-get remove pidgin pidgin-sipe libpurple-dev
   

3. Download Pidgin sources.
4. Change to the directory in which you unpacked the Pidgin sources, and build.

   ./configure
   make
   sudo make install
   

5. Download Pidgin SIPE plugin.
6. Change to the directory in which you unpacked the SIPE plugin sources, and build.

   ./configure
   make
   sudo make install
   

7. Now run Pidgin. It should appear in the usual Gnome menu, or can be run from the command line. Given that figuring out the right options for you company’s setup can be tricky (it was for me), you might want to do the initial connection setup with debug output from the command line.

   pidgin -d
   

   Not strictly necessary, as the “Debug Window” (got to via the “Help” menu) offers the same(?) information.

   You might also want to (temporarily) disable any other IM accounts you have configured.

Figuring out the right options to use with your company IM service … you are on your own.

On a related note, why are all scammers in Nigeria? Has anyone received the scam emails from ANY other country? I'm frankly dissapointed with the criminals in other countries. Where are they and why aren't they trying to scam us too?

via Consumerist – It’s Now Completely Impossible To Sell A Laptop On Ebay – eBay.

(An odd trap, when deductions – of a sort – lead to speculation that perceived reality may not follow a deductive model.)

What do I mean by a “deductive model of reality”?

If I cannot find object, search repeatedly in all the locations I expect, and eventually find the object in plain sight in a place I looked many times before – I assume by deduction that the object was in fact there all along, and I somehow missed it. By somewhat-conscious choice the episode is recorded in my memory as “the object was always there, and I somehow failed to see it”. By choice I reject from recall an instance of “the object appeared where I had looked before” – as impossible.

Along the same lines – if you and I have different memories of an event, I assume that one (or both) of us were mis-remembering, and there was – by deduction – a single true objective reality of the event.

A good practice in science is to occasionally question (and re-verify) your assumptions. What could I expect if my speculation about a “granularity” of reality might be somewhat true? With that question in mind I now more often tend to mark the mismatch between memory and “objective reality”.

An aside – At one point I wondered if the deductive the model of reality was a learned behavior – and that question suddenly invoked a faint memory of adopting a choice of interpretation, as a child. Is the memory real? I do not know.

Again, without any sort of test, there is no reason to treat this speculation as anything more. To be clear, I most firmly believe in the deterministic model, and not the speculation. But I also now have this persistent ghost of a question….

As a regular practice, before generating a build to go to customers, I always carefully review all the changes to the program sources since the last customer release. Using CVS the command line is simple (run from the root directory for the component):

cvs diff -t WIDGET_v301

Reviewing the differences helps both to write up a customer-oriented list of changes, and also helps insure that nothing unintended (say, left over experiments or debug code) makes it into the customer release.

With Subversion under Eclipse the equivalent comparison is easy – just select the trunk and tags/v301 and a compare gets the same result as CVS.

With Subversion on the command line, the equivalent comparison is ridiculously tedious.

svn diff http://hostname/path/widget/tags/v301

Um, hello? Subversion knows the http://hostname/path/ part. If I followed the conventional layout, Subversion can deduce that the current directory is trunk and from that the path to tags. Yes, I know the repository layout (in terms of branches and tags) under Subversion can be arbitrary (and this is all very cool), but optimizing for the most-common case is always good.

Figuring that I must have missed something, I went digging, and came up with this gem:

Branko Äibej – Re: svn diff branch woprking copy against mainline?
This seems to be a common misconception. The important thing to remember here is that there is no separate namespace for labels and branches in SVN, and that the layout of the repository is arbitrary. IOW, the fact that you have branches in /branches is a convention, not something imposed by the SVN server.

The writer is almost exactly wrong. There are a separate namespaces in the Subversion repository. The namespaces are established by convention. The recommended PROJECT/{trunk,tags,branches} can be recognized by default, and any local variant specified by user-set properties in Subversion. This is easy.

I expected Subversion to be a nearly uniform improvement over CVS, not stuck for years on how to cover simple usage.

An observation – a number of the small countries with outsize rankings are noted as having high taxes. So high taxes need not harm – and when the tax revenues are used effectively, may benefit – competitiveness (at least as measured by this group).

On a more speculative note, Britain is noted for “sophisticated financial markets”, and also as no longer in the top ten … and falling. Is there a correlation?

While the above video is only slightly interesting, of note is the guy in the suit – Neal Kelley – as after he was appointed Registrar, his department did a noticeably better job. The organization of supplies and instructions from the OCROV to the polling sites have improved every year. I give the guy a lot of credit for a job well done.

We have an unresolved problem. At the start of life and career, we want rough equality of opportunity. But ability is not equal. What each citizen could achieve, and will achieve … differs. If great achievement brings great wealth … I think that is right. For each guy who added much to society’s real wealth, given wealth, the odds are good that guy will do interesting things with that wealth.

On the other hand, some acquire great wealth without doing anything particularly constructive. Great wealth acquired but not productively invested leads to leads to huge instabilities in the economy – as in our present trouble. There is some sense in having a recognized “class” system – but only if the averaged result rewards real productivity (in any various sense).

For a long-term stable and productive society, we do not want vast unearned wealth passing between generations. There is only a slight chance the next generations will be as productive the first wealth-acquiring individual. This suggests that – as a society, and in the case of great wealth – we must limit the amount of owned wealth that can be passed between generations.

(Yes, a “death tax” – so colorfully named – makes sense for massive wealth.)

On the flip side, I do not especially trust government – or any large organization – to control or direct the economy. Central planning does not often work. The intelligence of a committee tends to sum up to an average of the individual participants
… or worse.

So how do you reward those of a different “class” (in terms of real achievement), tax those of great wealth (and no achievement), while using the least amount of “government”?

Perhaps the British approach to creating new “peerage” is a clue.

In the usual case, and as our society and economy is currently organized, we want great tax on great wealth. As currently organized, only a very small proportion is able to acquire great wealth. Of that portion that gathers great wealth, only a small portion can put that wealth to good use.

We need another filter – something that separates passive-rich from the worthy-rich.

Could we use something like the British peerage system? Those publicly recognized as exceptional should get a lower (or no) tax rate. Recognition could come from the government, or from a small number of entitled organizations. The number of entitlements should be limited, and some measure put on the organization (including and especially government agencies) allowed to grant entitlements.

An interesting exercise would be to express this notion in terms of a series of empirical measures. A healthy sum would include both individual or group judgment in granting economic privilege.

I have a clear notion. There are two parts. The first part is a pattern of thought. The second is depth of training in a particular area of expertise. My college degree is in Physics, but only a four-year degree. In Physics a four-year degree is only a fraction of the way to competence, so I am not a Physicist.

There is a pattern of thought essential to a good scientist (or a really good engineer). Of this I am quite certain. You need to be a good observer. You need to be a good skeptic. You need to be thoroughly analytical. This part is easy for me. My father was a good engineer. I read quite a lot, and before college had learned to be a good, skeptical, and analytical observer. The aim in college is to teach you this new mode of thought, and beyond this to transfer some of the wisdom acquired through experience by your instructors.

Of course, following the usual human tendency, I assumed my peers were at least as good at this mode of thought as was I.

In college, I did not need to learn a new mode of thought. The time in college I very much valued, as it seems that the hints from experience offered by my instructors fully sunk home. Those same lessons became so deeply embedded in my way of thought that I could not imagine a true “scientist” applying anything less.

What later I found is that “scientists” very often miss the obvious – or what I thought should be obvious. The lessons from my college professors (in Physics – admittedly the “hardest” science), suggest an analytical path that many scientists do not fully follow. This I have the hardest time understanding. Could it be that not having to acquire a new mode of thought meant I absorbed more from my instructors?

I do not know. By my standards, I am not a scientist. But … I very often come up with (what seems to me) obvious questions, on reading about the works of some scientists. Even more surprising – it seems very often to turn out that my (distinctly amateur) questions lead to interesting answers. How can that be so often true?

I am not a scientist, but perhaps I am a bit more than an intelligent layman. Maybe that long-ago acquired mode of thought made more of a difference than I expect.

I can also tell you that there is something on the underside of the mixer that can lacerate your finger, when you wash off the bits of dough left on the mixer (if mixing a bit too much dough, perhaps).

The reality of working at Burroughs in the early 1980’s was a bit more of a mixed bag. There were a lot of smart folk working at Burroughs, in that time. The company had all the potential needed to do great things. Yet … there was something wrong. The company was – for the most part – not doing great things. The Burroughs of that time had inherited a lot of loyal customers, and customers were loyal to Burroughs because of some brilliant work and products done in the past. Somewhere between the 1960’s and the 1980’s … something had gone badly wrong with management.

At the time, I did not have enough experience to know what was wrong, or even to be certain that anything was wrong (though I did eventually have suspicions).

I guess the interviews did me some good. I later learned that I’d landed a position that many folk within Burroughs very much wanted (and this was my second job out of college!).

Burroughs was a very early OEM partner for Microsoft Windows. I got sent by Burroughs to a Windows Developer conference in Seattle in 1984. During a lunch break, Bill Gates sat at our table, and suggested that we send in resumes to Microsoft. I was rather surprised at this. My employer had spent money to send me to the developer conference. For me to jump ship afterward seemed – rather rude. (I was very tempted, but my then-fiancée did not want to leave southern California. Turns out that was a very bad judgment on my part – both in the choice of wife, and in ignoring the opportunity.)

I may have underestimated Bill Gates. I thought his sitting at our table during lunch – and the offer – was essentially random. On reflection … of the tens of thousands of programmers at Burroughs, I was the representative they sent – a kid just out of college. If I were Bill Gates and looking for a useful filter – a guy just out of college who ended up at the Windows Developer conference through the second largest computer company in the world – that would be about as good as anyone could get.

The other way of reading this is that when given the perfect opportunity, you should not trust my judgment.