Sacrificing readability for automated doc tests 4
I’ve tried several times in the past to try out zc.buildout, a fairly neat sounding package that automates the buildout process for a Python app. The promise of fairly easy to write recipes that can setup external processes like nginx in addition to ensuring my webapp is put together with all the things it needs sounded great.
It occurred to me that the docs definitely didn’t help at all. In fact, they’re noticeably bizarre unless you actually realize why they’re written the way they are. Here’s a sample of the zc.buildout docs about how to make a new buildout and bootstrapping.
You’ll notice that it almost looks like command line interactions of some sort are occurring, yet the author of the docs is clearly at an interactive Python prompt. Note that none of the commands shown there will work if you copy them into your Python interpreter, nor is there any indication what you would need to do to get such commands available. As a user trying to follow the docs, that leaves me wondering… am I supposed to be in a Python interpreter? What do these variables get expanded to so that I can do that at my shell prompt? Why can’t you just give me the damn command line I’m supposed to run so I can copy/paste???
Yes, it definitely got me a bit frustrated. I believe the only logical reason the docs were in this bizarre fashion is so that they could be automatically doc-tested. Its a shame that the result of this is docs that make me want to close the web page as soon as I stumble upon the ‘samples’, since there’s no way I can handle wading through the command line abstractions.
Doctests can be useful, but turning command line interactions into a Python interactive session is a massive readability issue. People know and recognize command line interactions, lets stick with them please.
Trackbacks
Use the following link to trackback from your own site:
http://groovie.org/trackbacks?article_id=sacrificing-readability-for-automated-doc-tests&day=04&month=04&year=2008
I totally agree. That’s one of the things that excites me about Sphinx. Keep your test code in separate files and pull things in selectively when building the doc. There will be many more options for ensuring that the samples work and that they’re readable.
I believe the point here is to get complete line coverage using doctest. I can imagine that would take it’s toll on the readability/usefulness of the documentation. This is probably combining documentation/testing a little too far.
Even though I’m a big believer in doctests, I agree that often doctests don’t make for the best documentation. Jim Fulton (who wrote the zc.buildout doctests) is very hard core about them. A good reference or tutorial would be better than your typical doctest.
The arguments in favor of doctests shouldn’t be ignored though:
I think the reasons in favor of doctests often outweigh the drawbacks. That said, we should continue to think about how to write better documentation.
I should learn more about Sphinx; we’ve been considering it for the Grok documentation tool, where we’ve already been doing something similar for the Grok Tutorial.
@martijn: I agree entirely that they’re valuable. In this particular case though, the usage being documented is a command line interaction.
So to run doctests, the documentation itself is actually mangled and no longer accurate about what its proposing to document (command line interaction), and instead becomes a somewhat odd Python interpreter abstraction of a command line interaction.
Ideally, doctests should be extended to properly support running normal command line interactions so that this kind of obfuscation isn’t needed, but all the benefits of doctesting still remain.