Darius Bacon (darius) wrote,
Darius Bacon
darius

tush: literate testing for shell scripts

I've just released tush, a program-testing tool. Yes, the day after I sang the praises of 18th-century gay hobbit porn, this package rejoices in commands like tush-check and tush-bless, but honestly, it's a coincidence. I didn't want 2007 to go by without a single software release outside of work, and this one had the least left to fix. (The documentation is still in need of a wit transplant it's not going to get.)

Literate testing mixes executable tests into natural-language documents. The UpDoc page explains why:
  • Nonexecutable documentation tends to start out inaccurate and grow worse.
  • Conversely, unless tests stay intimate with documentation, they degenerate: "To keep the tests running as the software is changed, the purpose of the test is often lost, and the test is gradually changed into one that always passes."
  • Software can support experimental reading. In Emacs you can change tush examples and gauge the effect at the press of a key.
  • This extends to interactive or adversarial writing to produce the document in the first place.
The literate testing tools I know of are built around different programming languages; tush's language is the Unix shell.

To use it:

Tush looks for transcript-like lines in a file and checks them. For example:

$ echo Hello world
| Hello world


If you run tush-check README (where README is this file), it notices the above two lines, executes echo Hello world, and checks that 'Hello world' comes out on the standard output. Assuming the test passes, running tush-check succeeds silently. A failing test makes tush-check fail and output a diff.

You aren't limited to invoking the program under test; setup, clean-up, checking, etc., work the same way:

$ echo  >test.in 'here is some test input'
$ echo >>test.in 'and here is some more'
$ sort test.in | wc -l  # Check: sorting should not change the linecount.
|        2


We didn't bother to rm test.in afterwards because we have a crude kind of test isolation already: Tush makes a new temporary directory named tush-scratch, runs all the commands in the input from within it, then deletes it.

What about checking commands that should fail? There are two more special prefixes. For example:

$ cat nonesuch
@ cat: nonesuch: No such file or directory
? 1


The '@ ' line is like '| ', only for standard error instead of standard output. The '? ' line shows a nonzero exit status.


Tools:

tush-check was introduced above. It calls tush-run, which runs tush-run-raw from within a temporary tush-scratch directory.

tush-run-raw copies its input except for the special-prefixed lines introduced above: '$ ' lines are copied, too, but also executed, with their outputs/status codes inserted into the output with appropriate prefixes, so that for a successful test the output is the same as the input. Input lines starting with '| ', '@ ', or '? ' are dropped.

The above tools, like cat, take any number of files as arguments.

tush-bless foo updates foo so that tush-check foo will then pass: it changes any output/status-code lines to the actual outputs from tush-run. Use this when your program is correct but your test is wrong.


Examples:

See the *.tush files in this directory. They're a kind of self-check of the Tush implementation, though a weak one, since implementing tush-run as cat would pass it. [XXX do something about that?]


Emacs mode:

tush.el lets you pass the file you're editing through tush-run with a single keystroke. This can go nicely with interactive development combining unit tests with the code under test.


*wonders if anyone will flag this for adult content*
  • Post a new comment

    Error

    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

    Your IP address will be recorded 

  • 0 comments