Darius Bacon

The web speaks

2009-03-04T09:47:39Z

Here's the 'best' of the first half-dozen 'poems' from my new program: it babbles at random according to a Markov model using the most common bigrams from Google's whole-web corpus, constrained to a Shakespearean sonnet.

Agreed on their weight for this piece of them
For your position and became a trip
To add to you to do we must hold em
In self adhesive tape is more or zip
Code is to being with your bill in less
Than just eight points out in a way to get
A business days per month of your success
Of anal hardcore porno film is set
In prison for improved upon it will
Have seen as well to view cart is an end
Of firms that was extremely slow but still
Free web site will give to protect the trend
In thread for instant credit cards were laid
In for the children will not to be paid.

Another:

With package deals for sale at times to her
Life in case for the blessed with long on this
Scope and support group for the children were
Not gonna take this reason to dismiss
The right now if at this search engine which
Are not break free porn free world wide range for
The patient with care so forth from a bitch
About the pension plan must state of more
Attractive to go with some way to take
A gift of patients have a wife on this
Research results for your own mind that make
A self esteem and has been to dismiss
The court date on effective for which is
Served basis as far as they have heard his.

A completely safe for work one:

More stable in support that there were some
Type is shown in my products and thought this
Or been in their poor in a large groups from
Their friends and most things that some are still miss
Him well aware of his words to hang out
And his concerns for front of business to
Be paid for such sale or to lie about
The comic book by any light of view
Complete this section that a named on your
Search tips and he threw it will be a thought
I hate to spin on last line shopping for
The first few years there are not have been taught
By other regions where you understand
What you can range of any other hand.

So the program mostly works from the formal point of view, though with a few terrible rhymes like them/em, some mindlessly repeated rhymes, and it never tries for feminine rhymes. I'm surprised at how decent the scansion is, from such a simple algorithm. It sounds out the words using CMU's pronouncing dictionary.

I voted.

2008-11-04T17:57:59Z

And since they're using optical scan machines, my vote probably even got counted.

This post brought to you by the letters F, U, and W and the number -8.

two completely unrelated memes

2008-09-18T10:18:05Z

1. Take a picture of yourself right now. Don’t change your clothes. Don’t fix your hair. Just take a picture. Post that picture with no editing. (Except maybe to get the image size down to something reasonable. Don’t go posting an eight megapixel image.) Include these instructions.

2. When you see this, quote Douglas Adams in your journal.

Worksop (n.) A person who never actually gets around to doing anything because he spends all his time writing out lists headed "Things to Do (Urgent)."

(From Adams and John Lloyd, The Meaning of Liff.)

Gadding about

2008-08-30T20:34:06Z

I'm visiting the SF Bay area over roughly the next week. Want to meet up? Email me or comment here. I'll be staying with a friend for the first few days, but I'd welcome the chance to crash with someone else for the rest of the stay -- wouldn't want to impose on him that long. (I'll find a hotel if there are no volunteers.)

From there I was going to take a trip around the country on Amtrak; they have several-week rail passes at an attractive price, recommended by papersky. They also, it turns out, require photo ID, which I no longer have and certainly won't get just for them. So I'm not sure -- I suppose I'll try and set up a ride to the east coast via craigslist and then get around there somehow or other -- needs some research. Are there regular east-coast trains without this internal-passport silliness?

Spirits from the vasty deep

2008-07-11T09:33:16Z

In Scott McCloud's Understanding Comics: The Invisible Art, the 'invisible art' is in the space between the panels:

I may have drawn an axe being raised, but I'm not the one who let it drop or decided how hard the blow, or who screamed, or why. That, dear reader, was your special crime, each of you committing it in your own style. All of you participated in the murder.

Writers in any medium use tricks like this to enlist you in the storytelling. They skip over actions, as above; leave you puzzles and mysteries and choices; tell the story out of order; or just plain lie. I ran into a new device of this sort in the 2006 ICFP Programming Contest.

Their webpage during the run-up blandly announced that the contest would have to do with computational archaeolinguistics. Some days before the start they released a 'codex', a big file of no known type with peculiar strings amid the random bytes, like 'novus ordo seclorum' and a GIF logo saying 'CBV'. Finally they kicked off with an urgent appeal to their colleagues:

In 1967, during excavation for the construction of a new shopping center in Monroeville, Pennsylvania, workers uncovered a vault containing a cache of ancient scrolls. . .

Based on a translation of these documents, we now know that the society, the Cult of the Bound Variable, was devoted to the careful study of computation, over two millennia before the invention of the digital computer. . .

Two weeks ago, during a visit to the excavation site for a new computer science building at CMU, workers discovered a set of inscribed tablets that proved to be the Rosetta Stone for interpreting the Monroeville codex. The tablets precisely specify the Cult's computing device, known to initiates as the "Universal Machine." Although there is still no evidence that the cult succeeded in constructing their machine, it is a reasonably simple task to emulate it on modern hardware.

. . . We invite you to participate in this investigation. The codex and a translation of the Universal Machine (UM) specification are available for download from our web site. We encourage you to implement the UM and begin your own exploration of the codex.

Well, no way could I resist -- I joined catamorphism's team and coded a buggy UM interpreter over my lunch hour at work that Friday. The rest of the team (Brandon Moore, r6, tgies, and catamorphism) independently whipped up a working, but slow, interpreter in Haskell and started exploring the codex, which proved to hold the encrypted image of a multiuser computer -- they could log in to the guest account and start looking for ways into the rest of the system. After I got home I fixed up my interpreter and we were off -- it was fast enough for real work.

The users of that ancient computer had left files around with puzzles and in-jokes and hints at a story -- who were they, this Cult of the Bound Variable -- whence this struggle with the Cult of the L-Value -- did it start with the Separating Disjunction?

The new thing to me was a story intermediated by this UM interpreter I had to write myself. It's a kind of magic, to get a few pages of code just so, to invoke spirits from a dead civilization. "Do they come when you do call them?" -- Not at all at first, but they would wait through a few rounds of debugging before I could get to an intelligible prompt; then after speeding up memory management and fixing mishandling of text vs. binary I/O there was a login screen, and there we were -- in touch at last. Further tweaks brought further speedups and smoothed the animations -- you see how the experience of the story bore a particular directness to the work I put into it, as if it were my work, only amplified by the authors. That is the new trick in a work of fiction, that could only be done with a computer.

After the contest, submitting our team's solutions, r6 and I were moved to write some 'fanfiction' -- hacking up some tools overnight just before the deadline, to create a self-extracting compressed UM image like the codex, presenting a shell prompt when run -- r6 dropped hints in the style of the work we were responding to to imply that this shell came from the Cult of the L-Value. He developed his assembler afterwards into a neat article for the Monad Reader.

There have been other writeups of the unique excellence of the 2006 contest, but none I've seen emphasizing this response to it as worldbuilding and storytelling. This year's contest starts today! If you're a programmer, do check it out; catamorphism is one of the organizers this time, together with others at Portland State and the University of Chicago.

Zen agora

2008-02-17T21:29:44Z

In the game Zendo, a master makes up an arbitrary, secret rule, and students compete with each other to be the first to guess it. The rule classifies configurations of pyramids into two types. We call a configuration of pyramids a 'koan'. An example rule: a koan 'has Buddha-nature' if it has a red pyramid pointing at a green one.

Play starts with the master setting out two example koans, one of each type, labeled accordingly; then the master picks a student to move first.

The student builds a new koan. Then, if that student wishes, all students separately and simultaneously guess the koan's type, and the master awards a green stone to each student who was correct. Whether there was a guessing round or not, the master labels the koan with its type.

Also, if the student whose move it is has a green stone, they may give it back to the master and guess the rule. A correct guess wins the game. For a wrong guess, the master builds and labels a koan that the guess classifies incorrectly. The student may keep choosing to guess for as long as they have stones. Play passes to the next student on the left.

So that's the game; there are a few more details of play and terminology in the official rules, e.g. the type labels are white and black stones, they use Zen terms instead of my boring English, etc. I've never played it -- I was just reading its very interesting design history last night, via Chris Okasaki.

This morning I thought of a variant: instead of the master picking an initial student, and instead of play passing to the next student on the left, there's an auction on each move for the right to move. (With real money or play money, whichever. A second-price auction with sealed bids seems the right thing, although you'd want to avoid the auction machinery overwhelming the actual play.) The winner of the auction pays the auction-assigned value to the previous mover, then moves. At the end, the student who guessed the rule gets awarded some money; this money was collected from all the players at the beginning of the round when the master made up the rule. I guess for the very first auction, before there's any mover to pay the auction-value to, the payment should go into the prize pot. (Maybe in subsequent rounds the first payment goes to the winner of the previous round.)

The rationale is to reward players for inventing insightful koans whose answer is likely to bring the solution closer. If you think of a 'good experiment', the value of the next move will be higher than you had to pay, and you'll make a profit. Also -- while it's not apparent from my description which deemphasized the "Mondo" and the "Buddha nature" and such -- I think it's hilarious to mix capitalism and Zen.

(This was suggested by Eric Baum's reinforcement learning auction ideas. I suppose you could make a game this way out of any competitive machine-learning algorithm.)

ETA: This has the obvious defect that the first mover can hold onto ownership for the rest of the game. There needs to be some cost to stop that, something like the ante in poker, I guess. Oops. If per-auction antes went to the master, that'd encourage the master to make overcomplicated rules and counterexamples, so that's not the answer. Hmm. Maybe just bite the bullet and say the house takes a cut? Vegas Zen Agora. But then it needs gambling! Argh! Zendo really does sound elegant, doesn't it?

So, another attempt:

We play with funny money managed by a neutral banker. At the start of a session each player gets a stash. Each move, the banker collects 'rent' from each player for the privilege of staying on the trading floor. There's an auction for the move; the price is paid to the previous mover. The new mover builds a koan, which the master labels. The mover then may guess the rule, once. A correct guess wins the set prize money, and ends the round. A wrong guess produces a counterexample from the master, and ends the move.

This variant eliminates the green guessing stones and the all-students-guess option, since they seem sort of redundant with the auction. We could potentially take out the student-built koans, too, and have them just guess the rule every move, but that may be going too far.

Endorsement

2008-02-05T10:06:08Z

The darius endorsement for U.S. president goes to Barack Obama. As pnh puts it:

He’s not an insurgent; he’s the standardbearer for a faction of the country’s political elite. I believe that, on balance, this particular faction happens to comprise many of the the smartest and most conscientious individuals from within that elite. So I’m supporting Obama and his train, people like Samantha Power and Robert Malley and Lawrence Lessig, just as a peasant might cheer for an aristocratic faction made up of reasonably decent individuals against other factions made up of out-and-out thugs. Not because the peasant doesn’t know the game is rigged, or doesn’t have the wit to imagine a better world. But because incremental change matters, and because the right incremental changes can lead, like water flowing downhill, to bigger and more profound ones.

Also, while I am a radical in analysis, I am an incrementalist in practice, because life is short.

(Though I disagree with the reason "because life is short". In genetic algorithms it's important for your fitness function to be reasonably discriminating even in the vast area of the space that's far from optimum. Recent elections brought home to me how much that principle applies to politics.)

ETA: Peter Norvig posted another, quite different endorsement worth reading.

Beethoven/Tolkien

2008-01-24T03:16:19Z

I met Beethoven's piano sonatas and The Lord of the Rings at about the same age, around 10, though they were never especially linked in my mind before.

I've always loved the sound of the piano; in second grade after it was first announced that there'd be lessons after school, I skipped back home with the signup form, playing air keyboard. The class was taught by a sprightly old gent named Fekko von Ompteda, and I enjoyed it, unlike school in general: it was just hard enough to be interesting, and there we were making actual music with our hands on heavy hulking mysterious machines -- plus an old Moog once -- and learning bits of mathy theory and an esoteric visual code. I liked the teacher too: he was an immigrant and charmingly eccentric in his own right, though after mumbledy years I've quite forgotten why I thought so. It seems he was "a very gifted and amazingly progressive musician", and, sadly, is no more; I wish I'd come back to Toronto for a visit before he died.

Moving to California ended all that. But my parents gave me for Christmas, at some point, a multi-record set of Beethoven played by Ivan Moravec, who as far as I'm concerned defines Beethoven -- see, I imprinted on it. The cycle of piano sonatas is my ultimate comfort music now.

This time, as I happened to come back to The Fellowship of the Ring at the same time, I was struck by how much sonata #20 sounds like the Shire. It could be dance music at Bilbo's birthday party. You might plump for the 'Pastoral' sonata instead, but I say no, that one's too Elvish. :)

But that wasn't the really striking discovery. I'd like you to play this clip, the theme of the last movement of sonata #4, and read the opening of Chapter 8, Fog on the Barrow-Downs:

http://www.accesscom.com/~darius/tmp/fgc.mp3

That night they heard no noises. But either in his dreams or out of them, he could not tell which, Frodo heard a sweet singing running in his mind; a song that seemed to come like a pale light behind a grey rain-curtain, and growing stronger to turn the veil all to glass and silver, until at last it was rolled back, and a far green country opened before him under a swift sunrise.

I doubt the young Beethoven could have captured the feeling of the passage this foreshadows at the end of The Return of the King, though; that would be a job for late Beethoven, who, alas, never came back to LotR for inspiration.

Sorry for the lousy fade to end the clip; it's the first time I've tried to use a sound editor. I'm afraid I didn't note down who performed this.

darius @ 2008-01-21T14:23:00

2008-01-21T22:26:31Z

Thanks, y'all, for your comments. They were helpful, and now I've gotten in touch with my gut feeling and made up some rationalizations for it.

Any snark used in the production of this post was directed solely at the poster.

Dear lazyweb,

2008-01-21T08:18:07Z

I'm a California member of the Decline-To-State Party. I have till Tuesday to change my registration. While voting is one of the least influential things you can do politically, it's sort of bad to just skip it. Should I:

1. Register Democratic and vote for Obama? He seems like the least-bad leading candidate. (Some evidence. The linked post takes this as evidence of effectiveness; I'm more impressed by the choice of what to be effective about.)

2. Register Republican and vote for Ron Paul? There will be a Republican Party for the foreseeable future, that even wins some elections, so its future character matters a lot. Paul, for all his, ah, imperfections, is the only candidate to really repudiate the Bush administration (besides repudiations like it's too liberal or not warlike enough); a good enough showing might make some marginal difference.

3. Register Republican and strategically vote for Romney? I suppose keeping McCain out of the general election, which he might win with the press fluffing him, would make a bigger difference than influencing the Democratic choice. Downside: this is dishonest.

QOTD

2008-01-19T08:04:50Z

Quoth papersky on reading a biography of George Eliot:

In one section, she states that some well-regarded people think Middlemarch the best novel in the world, ever. I stopped and looked suspiciously at this, turned the idea around a few times, and cautiously considered that in fact perhaps Middlemarch did deserve to be considered in the same company as Lord of the Rings, Cyteen, A Fire Upon the Deep, The Disposessed and Stars in My Pocket Like Grains of Sand. (That grinding sound you hear? F.R. Leavis turning in his grave?) But you know, not really. Because it's just an awful lot easier if you get the world ready made for you. That's my main objection to people who say mainstream and fanfic can be as good as original SF. People can juggle two balls awfully well, and Middlemarch and Dark Reflections both do that, in their different ways, about as well as it can be done. But that still can't really compare to people who are juggling four.

Mostly I just wanted to share this because it made me smile, but it did trigger some thoughts:

Seeing A Fire Upon the Deep on this list took me aback a bit because I'd give higher accolades to its prequel, A Deepness in the Sky. But this choice does make better sense here: aDitS is a story of superlatives, and one of these is the blackest irony, that all the brilliance and hopeful plans are lost in the Slow Zone, with no character having the least hint of a clue of this. (Almost.) You need to have read aFutD to get that -- in that sense the book's worldbuilding doesn't stand alone like the books above.

This difference of genre has an analog in the programming world with works written for the mainstream ecosystem, fitting into a gigantic tangle of shared assumptions, versus more self-contained systems like Smalltalk... growing their own tangles. Funny how I lean towards the latter in this world, too; I wonder how much the reasons are the same.

I've wondered how we might take more advantage of allusion in programming -- in 'literate programming' can't the code itself be more literary? Today's new thought: maybe Vernor Vinge was thinking along the same lines with his 'analogical programming' in True Names.

And I really should check out Middlemarch.

2007 books

2008-01-16T22:03:02Z

Susan Cooper	The Grey King
Susan Cooper	Greenwitch
Larry Niven & Edward Lerner	Fleet of Worlds
Frederick Winsor & Marian Parry	The Space Child's Mother Goose
Ursula LeGuin	Jane On Her Own
Susan Cooper	Over Sea, Under Stone
Roald Dahl	The BFG
Ursula Vernon	It Made Sense at the Time...
Susan Cooper	The Dark is Rising
Roald Dahl	The Witches
Darby Conley	Get Fuzzy 2: Fuzzy Logic
Tanya Huff	Summon the Keeper
Vernor Vinge	A Deepness in the Sky [reread]
John M. Ford	The Scholars of Night
Jerry Scott & Jim Borgman	Zits
Berke Breathed	Billy and the Boingers Bootleg
Vernor Vinge	A Fire Upon the Deep [reread]
Olivia Judson	Dr. Tatiana's Sex Advice To All Creation
Steven Brust	Dragon [reread]
Scott Lynch	The Lies of Locke Lamora
Lois McMaster Bujold	Diplomatic Immunity [reread]
Joe Haldeman	The Accidental Time Machine
Douglas Coupland	The Gum Thief
Francesca Lia Block	The Hanged Man
Tony Auth et al.	The Gang of Eight
Poul Anderson	A Midsummer Tempest
Dominic Widdows	Geometry and Meaning
Elizabeth Barber	The Mummies of Ürümchi
Jo Walton	Ha'penny
Charles Stross	Halting State
Rafael Sabatini	Scaramouche
Dozois & Strahan [eds.]	The New Space Opera
C. J. Date	Database in Depth
Ken Macleod	The Execution Channel
Linus Torvalds and David Diamond	Just For Fun
Scott Lynch	Red Seas Under Red Skies
Ben Mezrich	Bringing Down the House
Stanley Kiesel	The War Between the Pitiful Teachers and the Splendid Kids
Scott Rosenberg	Dreaming in Code
Connie Willis	D.A.
Gregory Clark	A Farewell To Alms
William Shunn	An Alternate History of the 21st Century
Lois McMaster Bujold	The Sharing Knife: Legacy
Italo Calvino [ed.]	Italian Folktales [unfinished]
John Barnes	One For the Morning Glory [reread]
Robert Bakker	Raptor Red [reread]
Benjamin Franklin	Poor Richard's Almanack
Poul Anderson	Trader to the Stars
Edward Castronova	Synthetic Worlds
Poul Anderson	Time Patrolman
John M. Ford	Growing Up Weightless [reread]
Francois Martin Mai	Diagnosing Genius: The Life & Death of Beethoven
Elizabeth Bear	New Amsterdam
Donald Westlake	What's So Funny?
J. K. Rowling	Harry Potter and the Deathly Hallows
Neil Gaiman	Death: The Time of Your Life
Carolyn Stevermer	A College of Magics [reread]
Robert Heinlein	Between Planets [reread]
Elizabeth Bear	Carnival
J. R. R. Tolkien	The Children of Húrin
Barrington Bayley	Star Winds [reread]
Elizabeth Moon	The Speed of Dark
Pamela Dean	Tam Lin [reread]
B. Kliban	The Biggest Tongue in Tunisia
Elizabeth Willey	The Well-Favored Man [reread]
Carla Speed McNeil	The Rescuers
Carla Speed McNeil	Mystery Date
Roger Zelazny	A Night in the Lonesome October [reread]
Robert Heinlein	The Moon is a Harsh Mistress [reread]
Dan Piraro	Too Bizarro
Pamela Dean	The Dubious Hills [reread]
Robert Heinlein	The Rolling Stones [reread]
Douglas Hofstadter	I Am A Strange Loop
Beatrix Potter	Squirrel Nutkin
Carla Speed McNeil	Talisman [reread]
Dave Duncan	The Reluctant Swordsman
C. Collodi	Pinocchio
Paul Reps [compiler]	Zen Flesh, Zen Bones
Magnusson & Palsson [translators]	Njal's Saga
Joe Armstrong	Programming Erlang
Jack Vance	The Best of Jack Vance
Joan D. Vinge	Heaven Chronicles
Aubrey Beardsley	Collected Drawings
Jeff Smith	Bone vol. 9: Crown of Horns
Jeff Smith	Bone vol. 7: Ghost Circles
Jeff Smith	Bone vol. 5: Rock Jaw
Carl Sagan	The Demon-Haunted World
H. G. Wells	When the Sleeper Wakes
Michael Flynn	Eifelheim
Julie Phillips	James Tiptree, Jr.: The Double Life of Alice B. Sheldon
Rudy Rucker	Mad Professor
Peter Watts	Blindsight
Marvin Minsky	The Emotion Machine
Ted Chiang	Stories of Your Life and Others
Tom Shippey	The Road to Middle-Earth
Diana Wynne Jones	Dark Lord of Derkholm
Diana Wynne Jones	A Sudden Wild Magic
S. M. Stirling	The Sky People
J.R.R. Tolkien	The Return of the King [reread]
J.R.R. Tolkien	The Two Towers [reread]
J.R.R. Tolkien	The Fellowship of the Ring [reread]
Gerald Pollack	Cells, Gels, and the Engines of Life
Felicitas Tobien	Art Nouveau Paintings
E. Ennion and N. Tinbergen	Tracks
Walter Jon Williams	Knight Moves [reread]
Gardner Dozois [ed.]	The Year's Best Science Fiction, 14th Annual Edition [unfinished]
Doug Clapp [ed.]	The Macintosh Reader
D.H. Cannon	Scream For Jeeves
Sherman Stein	Archimedes: What Did He Do Besides Cry Eureka? [unfinished]
Sarah Susanka	The Not So Big House [unfinished]
Lois McMaster Bujold	Komarr [reread]
Freeman Dyson	The Scientist as Rebel
Art Spiegelman	In the Shadow of No Towers
Robert Harris	Imperium: A Novel of Ancient Rome
Steven Vogel	Prime Mover: A Natural History of Muscle
Seth Roberts	The Shangri-La Diet
Bill Watterson	Yukon Ho!
Peter Ward	Out of Thin Air: Dinosaurs, Birds, and Earth's Ancient Atmosphere
Octavia Butler	Mind of My Mind
Steven Johnson	The Ghost Map
Charles Stross	Toast
Michael Light	Full Moon
Neil Gaiman	Fragile Things
Lee Smolin	The Trouble With Physics
Berkeley Breathed	Opus

Reverse chronological order; more notable books that weren't rereads are bolded. It turns out my wishlist looks rather weightier than my actual reading habits, fancy that. I think I should be reading less and writing more. I considered starting by writing notes on each book, but that would take effort, and through lazy evaluation you can get the same result for books you're interested in, by forcing the thunk in the comments.

The table was generated by an awk script.

darius @ 2008-01-03T01:12:00

2008-01-03T09:54:31Z

It's often said that the U.S. can't go back to manually-counted paper ballots (like other countries use) because our ballots are so much more complicated. But couldn't we get most of the benefit of manual counting using spot checks? Like this: count with a black-box optical-scan machine at the precinct, as is done too often now, but in each precinct pick a random contest and also count it the old-fashioned way, preferably right then and there. If the discrepancy exceeds a threshold, hand-count everything and investigate what went wrong. Obviously, this guards against large-scale meddling better than local; but then large-scale hacks are the big new threat that these machines introduced.

(There are details to flesh out, like the protocol to pick a random contest -- maybe each interested party has their representative roll their own N-sided die simultaneously, and they're all added like times on a clock -- and calculating how to get a low enough probability of undetected hacking to a degree that could change results regionally or nationally, with a safety margin, etc.)

I've never volunteered at a polling place -- I'd like to learn whatever the reasons are why this is a dumb idea.

Incidentally, it looks like there's some neat research on end-to-end voting verification. What's different about this is we could do it now, in the 2008 election, as far as I can see. Except of course where we're not using paper ballots at all, and have issued an engraved "Hack Me" invitation instead. (I mean technically we could -- legally, who knows. Shouldn't there be a legal argument for recounts a priori, from all the recent studies showing that electronic voting machines should never have been certified? The certification agencies themselves have a lot of explaining to do.)

zestyping's Pvote design looks really good. I'm studying it in my spare time.

darius @ 2008-01-03T00:30:00

2008-01-03T08:51:59Z

I live within walking distance of a bunch of Hollywood studios, in Burbank. Sometimes this brings me a bemusing sight, like the giant crab atop an office building, or the barbed-wire camera-festooned hedges of Warner Bros. Ranch. Today, down the street from Legacy Media Tower -- that's what they called it. Legacy Media Tower. -- I was walking by the NBC building and saw the striking writers, who I'd managed to miss up till now. They were trudging around and around like a block-long treadmill, and I wished it were all about waving fists and singing "No Nos Moverán", impractical as that'd be to keep up.

tush: literate testing for shell scripts

2007-12-29T08:48:14Z

I've just released tush, a program-testing tool. Yes, the day after I sang the praises of 18th-century gay hobbit porn, this package rejoices in commands like tush-check and tush-bless, but honestly, it's a coincidence. I didn't want 2007 to go by without a single software release outside of work, and this one had the least left to fix. (The documentation is still in need of a wit transplant it's not going to get.)

Literate testing mixes executable tests into natural-language documents. The UpDoc page explains why:

Nonexecutable documentation tends to start out inaccurate and grow worse.
Conversely, unless tests stay intimate with documentation, they degenerate: "To keep the tests running as the software is changed, the purpose of the test is often lost, and the test is gradually changed into one that always passes."
Software can support experimental reading. In Emacs you can change tush examples and gauge the effect at the press of a key.
This extends to interactive or adversarial writing to produce the document in the first place.

The literate testing tools I know of are built around different programming languages; tush's language is the Unix shell.

To use it:

Tush looks for transcript-like lines in a file and checks them. For example:

$ echo Hello world
| Hello world

If you run tush-check README (where README is this file), it notices the above two lines, executes echo Hello world, and checks that 'Hello world' comes out on the standard output. Assuming the test passes, running tush-check succeeds silently. A failing test makes tush-check fail and output a diff.

You aren't limited to invoking the program under test; setup, clean-up, checking, etc., work the same way:

$ echo  >test.in 'here is some test input'
$ echo >>test.in 'and here is some more'
$ sort test.in | wc -l  # Check: sorting should not change the linecount.
|        2

We didn't bother to rm test.in afterwards because we have a crude kind of test isolation already: Tush makes a new temporary directory named tush-scratch, runs all the commands in the input from within it, then deletes it.

What about checking commands that should fail? There are two more special prefixes. For example:

$ cat nonesuch
@ cat: nonesuch: No such file or directory
? 1

The '@ ' line is like '| ', only for standard error instead of standard output. The '? ' line shows a nonzero exit status.

Tools:

tush-check was introduced above. It calls tush-run, which runs tush-run-raw from within a temporary tush-scratch directory.

tush-run-raw copies its input except for the special-prefixed lines introduced above: '$ ' lines are copied, too, but also executed, with their outputs/status codes inserted into the output with appropriate prefixes, so that for a successful test the output is the same as the input. Input lines starting with '| ', '@ ', or '? ' are dropped.

The above tools, like cat, take any number of files as arguments.

tush-bless foo updates foo so that tush-check foo will then pass: it changes any output/status-code lines to the actual outputs from tush-run. Use this when your program is correct but your test is wrong.

Examples:

See the *.tush files in this directory. They're a kind of self-check of the Tush implementation, though a weak one, since implementing tush-run as cat would pass it. [XXX do something about that?]

Emacs mode:

tush.el lets you pass the file you're editing through tush-run with a single keystroke. This can go nicely with interactive development combining unit tests with the code under test.

*wonders if anyone will flag this for adult content*

On First Clicking Into Frodo Hill

2007-12-27T11:41:09Z

Much had I travell'd in this realm, Live-Journal,
ere coming to its Province, teaselfiction,
whose Hobbits practice in its Jurisdiction
(the border Sign-Posts warn) the art nocturnal.
I deem'd myself secure, for howsoever
superb Exemplars of this Art there be,
an art of trifling Interest to me
(when Halflings hone it) hardly could endeavor
to gaol me there. And yet 'tis this Domain,
where Comedy, Lore, Sex, and Angst abut,
that deep-brow'd Teasel drew forth from her Brain,
that hardly can't but grab you by your Gut.
(Or other Part.) Dear Teasel, I am slain:
tied up upon a Heap of Hobbit-Smut.

I hope I'm not too embarrassed by this in the morning. Teasel, you have your revenge for my Vinge rec. :-)

N.B. This review was written after but starting on this one offering; I'm sure the others are as artful, but I haven't verified this yet, and eegh, like I have time. I meant to get some work done tonight. It looked like not all of the stories are slash; even if, like me, you're not especially into slash, do go over for the imagination and word-skill and the insight into Middle-Earth.

darius @ 2007-12-21T15:19:00

2007-12-21T07:27:37Z

Nothing special to report, except that I'm posting this with my cute new XO laptop. Wheee!

Bookworm

2007-12-15T07:51:29Z

I think it needs a little bit extra, like a pet cat or a party hat or something. But my stuffed moggie wouldn't fit on the hoard.

Do you have any books for dragons?

administrivia

2007-11-30T04:09:46Z

I just noticed that my journal's public posts had become a cesspool of comment spam. I thought this was set to screen anonymous comments and mail me notifications; I'd glance at each note and ignore the spam. This journal is still, supposedly, set that way, so I don't know what's broken, LJ or my understanding of it. I may end up disabling anonymous comments completely.

What's up

2007-09-05T23:00:08Z

I quit my job at Yahoo last month. It was a privilege to work with such smart and friendly people. Now it's a privilege to get some time on my own -- I'm living in Burbank writing some hobby-hacks -- all part of my grand vision to Change the World of Programming, of course, but the link is pretty tenuous right now.

darius @ 2005-08-09T00:53:00

2005-08-09T07:50:29Z

Walking home this evening, remembering pameladean's Sunday-night astronomy post, I kept an eye out for the moon. To my surprise it wasn't just near a couple of bright planets as she'd reported, it was right spang in between them in a straight line. If you've seen Jupiter in a telescope attended by its moons, you know how this looked, only spread out, tilted, over the western horizon: as if our moon had moons.

almost forgot

2005-06-07T19:49:04Z

Today is my 5-year LJ anniversary.

random()

2005-01-30T11:47:10Z

Out walking today, a bike brushed by on the sidewalk, its rider giving me an odd look and asking some question I missed. I shrugged apologetically, though I hadn't thought I'd been blocking the way, and thought no more of it.

Later at the library, leafing through The Joy of Music in the stacks, I was accosted by someone wanting to know if I'd accepted Jesus as my savior. It turned out this was the same guy as on the bike, and that I bore a powerful resemblance to Jesus -- not the first time I've been told so, but certainly the oddest. If we hadn't been at the library I'd have liked to preach on computational eschatology just to see what would happen. He went back for some pamphlets that I had to turn down.

When my hair was shorter nobody ever said I looked like Iain Banks, hmph.

Unrelatedly, I visited a big dotcom this week and they had everyone working in a giant open-plan office without even cubicles. How can anyone code in a place like that? Don't they need to concentrate? Maybe I'm more distractable than most? (For me even a blinking cursor is like a little man jumping up and down yelling "Look! Look!") Anyway they're hiring Erlang hackers -- if anyone's interested I can hook you up.

Since I asked for a rant from lunza it's my memetic duty to offer rants now on subjects of your choice. Expect a lecture or an essay or something instead, though, even if rants have higher fitness.

an unusual use for literature

2005-01-09T10:38:20Z

For years I've been threatening to build a user-programmable website someday, vaguely like a wiki but for code instead of text. I came up with a piece of the design tonight that I'm afraid might annoy lovers of literature. It takes some explaining: say this website were named foo.org. People come in and make new pages on it; the site assigns each page an ID, random enough to be unguessable, like https://foo.org/1e6cc1499953b4052d75. (There can be meaningful names, too, but they're less fundamental. Random unguessability is part of how we prevent abuses -- for example, the LJ hack I posted about depends on attackers knowing the URL that updates your journal.)

So we've got something like a phone number, https://foo.org/1e6cc1499953b4052d75. foo.org is a memorable enough area code, but 1e6cc1499953b4052d75... uh. Most of the time you don't care, because you just bookmark the page, or follow a link to it someone sent you, etc. Still, IDs that humans can handle would be nice to have. One choice I looked at: use dictionary words instead of numbers. We get IDs like

https://foo.org/toxon-basal-Pinal-upheld-corp-a
https://foo.org/screw-herdic-ridgil-Oglala-erose-a
https://foo.org/josser-kazoo-cause-cense-burl
https://foo.org/weal-androl-culgee-Inga-rehang-a
https://foo.org/surtax-flot-pussy-epural-lampas-aa

and what does that remind you of? That's right! Spam.

Trying again with a different word supply:

https://foo.org/awful-heave-host-loyal-slayer-heroes-decked
https://foo.org/deed-body-keels-fold-tests-giver-after
https://foo.org/both-speed-nay-blow-awful-hoard-brands
https://foo.org/dared-tore-spy-twain-spied-hale-of
https://foo.org/robed-weave-thy-third-surest-expect-sentry

Now this manages to not look like spam, I think, but I wonder: if you saw much more like this, could you ever read Beowulf without remembering that damned foo.org website? Because that was my source. I can even imagine this tainting Tolkien.

Other options: there's base32 encoding, shorter than 1e6cc1499953b4052d75 but not by enough to make it memorable. And there's random babbling within constraints to keep it more or less pronounceable -- but the result won't fit in human short-term memory at least in my case. The dictionary encoding nicely chunks a 77-bit key into 7 words, just matching the "magical 7 plus or minus 2".

It's irritating that browsers always show the full URL in the address bar, exposing it to shoulder-surfing. Presumably we need two kinds of IDs, sharable and private, and when you follow a sharable link it immediately gets redirected into the same thing in your private namespace (determined by cookie). When you want to share a link of your own you need a separate action to get a sharable ID. At least that's a natural place to add revocability... Taking this route, it seems natural to use word-encoding only for sharable IDs, to help humans notice the difference.

OTOH perhaps Javascript could deal with the shoulder-surfing, replacing the URL bar with pet names that don't leak authority, while still letting you drag and drop URLs as direct capability references. Not that Javascript's something to rely on, given a choice. I need to see if Tyler Close has anything to say about this.

Friday pieblogging with Neal Stephenson

2005-01-01T00:22:59Z

This page from The System of the World almost made me wonder if the same author writes Fafblog. I can't see him doing that, but it's not like he's incapable of the same level of lunacy:

"...and so we have made an arrangement with Mr. Party--but not disbursed any money to him, of course--nor do we expect to, until the end of this month," Daniel said. He'd given Isaac an account of the Clubb's late doings, mercilessly abbreviated because of the aroma of the mutton pies, which were waiting on a platter in his lap. The platter was a twenty-pound slab of silver done up in full Barock style and engraved with miles of tangled script: a paean to the sexual powers of Newton's niece. Here she was referred to as Aphrodite, a code that Isaac was not likely to penetrate.

In an apt demonstration of the principle of Relativity, as propounded by Galileo, the bawdy platter, and the steaming morsels thereon, remained in the same position vis-a-vis Daniel, and hence were, in principle, just as edible, as if he had been seated before, and the pies had been resting upon, a table that was stationary with respect to the fixed stars. This was true despite the fact that the carriage containing Daniel, Isaac Newton, and the pies was banging around London. Daniel guessed that they were swinging round the northern limb of St. Paul's Churchyard, but he had no real way of telling: he had closed the window-shutters, for the reason that their journey to Bedlam would take them directly across the maw of Grub Street, and he did not want to read about today's adventure in all tomorrow's papers.

Isaac, though better equipped than Daniel or any other man alive to understand Relativity, shewed no interest in his pie--as if being in a state of movement with respect to the planet Earth rendered it somehow Not a Pie. But as far as Daniel was concerned, a pie in a moving frame of reference was no less a pie than one that was sitting still: position and velocity, to him, might be perfectly interesting physical properties, but they had no bearing, no relationship to those properties that were essential to pie-ness. All that mattered to Daniel were relationships between his, Daniel's, physical state and that of the pie. If Daniel and Pie were close together both in position and velocity, then pie-eating became a practical, and tempting, possibility. If Pie were far asunder from Daniel or moving at a large relative velocity--e.g., being hurled at his face--then its pie-ness was somehow impaired, at least from the Daniel frame of reference. For the time being, however, these were purely Scholastic hypotheticals. Pie was on his lap and very much a pie, no matter what Isaac might think of it.

Mr. Cat had lent them silver table-settings, and Daniel, as he spoke, had tucked a napkin into his shirt-collar--a flag of surrender, and an unconditional capitulation to the attractions of Pie. Rather than laying down arms, he now picked them up--knife and fork.

Happy new year!