random memes }

Scripting inspired by Monad - for Unix

Scripting on Windows has always been pretty lame compared to Unix. The usual command.exe or cmd.exe shells on Windows are pretty pathetic compared to the Bourne Shell (which was released back in 1977!). Lacking a good shell, and with a population of GUI developers less familiar with the command line, Microsoft never really got a clue. The Cygnus ports (and the like) of GNU command line utilities and bash likely lessened the demand somewhat.

A few years back a group at Microsoft came up with Monad ... which was a pretty cool idea, if a little too fat. Took years before they eventually shipped (renamed as Powershell).

Unix has tended to be about simple ideas with a lot of mileage. While the notion of building something like Monad on Unix is pretty interesting (I'd use Javascript via Rhino on the JVM), what I really wanted was a simpler notion that better "fit" the Unix-tools mindset.

One of the really cool bits about Monad was the ability to stream structured data - typically XML. Unix tools typically work on character streams - and only that. The character streams could quite naturally include structured data ... but there seemed to be something missing.

Turns out there is a simple/elegant solution that could be retrofit to existing shells and tools, and fits very well within the usual Unix-ish way of doing things. The notion is so simple, it is a bit funny. This makes a round-trip of sorts....

The first generation of web servers were almost entirely running on Unix. The first common-model for dynamic content was CGI, which was simply (and logically) a slightly warmed-over of Unix shell scripting. Running a shell script for each incoming web request was not especially efficient, so we got a whole zoo of alternatives (ASP, JSP, mod_perl, etc.) but the programming model is still based on CGI - which in turn is derived from Unix shell scripting.

When a web browser makes a request of a web server, the request includes a "Content-Type" header to indicate the MIME type of the request data, and a set of "Accept" headers to indicate acceptable data-types for the response.

When a shell script is used for CGI, the HTTP headers turn into environment variables - for example:

HTTP_ACCEPT=text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8 HTTP_ACCEPT_LANGUAGE=en-us,en;q=0.5 HTTP_ACCEPT_ENCODING=gzip,deflate HTTP_ACCEPT_CHARSET=ISO-8859-1,utf-8;q=0.7,*;q=0.7

Unix tools that got used for CGI applications (Perl, Python, PHP, etc.) were all taught to ingest HTTP requests and generate HTTP responses. The code exists and is generally quite mature.

How could we pipe structured data between processes, automatically negotiating the data-type where possible? In exactly same way a web browser negotiates the content-type with the web server!

There are a relatively small number of missing bits needed to make shell run scripts/tools in a similar fashion to CGI.

  1. The shell needs to read HTTP response headers from the output of a prior process, and translate the headers into environment variables, before invoking the next process in a pipeline.
  2. Some convention so tools know whether to generate a plain character stream, or structured data. The presence of the HTTP_ACCEPT environment variable is probably sufficient.
  3. ... er, maybe that's all.

There are many ways we could enable this behavior, but it could be as simple as:

HTTP_ACCEPT=text/xml ps -e | fold | spindle | whatever

... to invoke use of XML in a single pipeline, or:

export HTTP_ACCEPT=text/xml ps -e | fold | spindle | whatever

... to invoke use of XML over an entire script.

Could use a shim without changing the shell. Something like:

HTTP_ACCEPT=text/xml ps -e | cgi perl a.pl | cgi php -f b.php

An exercise for another day....