Where's the Capistrano knock-off for us Python web devs? 19
Rails, and Ruby in general has had Capistrano for awhile now to help with the task of deployment and automating builds for servers, and even clusters of servers. Where is something like this for Python?
Now, before people note that I could easily use Capistrano for my Python project, I should note that it is rather annoying having to install yet another language. On the other hand, given that I will likely only need to install it on my development machine (which running OSX already has Ruby… and gems), it doesn’t seem too horrible to just use Capistrano and be done with it.
However, Capistrano doesn’t quite manage the Python egg’s, and the task isn’t exactly trivial. zc.buildout, which I previously ranted about due to odd docs does the management pretty well. It even results in a rather consistent build experience no matter where it occurs. Two commands, and boom, the app is ready to go.
Unfortunately, life isn’t quite that easy. When something does go wrong with buildout, trying to track it down can be exceptionally hairy. Having a tool so ‘magical’ as I’ve heard some describe it, carries its own penalties when things fail. Buildout also fails to automate the task of deploying the app itself to the other machine, which is still a manual process. It does manage egg’s rather well, though it does some very odd mangling of sys.path to accomplish this in every script.
I don’t need something as full featured as Capistrano, but I’d love to see something that has no more requirements than I’m already depending on (Python), that can handle the task of easily automating deployment of a Python application – including ensuring all the proper versions of the eggs I want are used – on a remote *nix machine. I recall seeing a post (I think by Jeff Rush) awhile back, on a system just like this that he unfortunately never released. Vellum also looks like it could be hacked further to do this task…
Is there some build/deployment tool that is just Python that I’ve missed? Something that will let me setup a script for some commands on how to deploy my app on another server and setup (hopefully in a virtualenv) the webapp so its ready-to-run (and optionally restart it/migrate the db/etc :)?
What about Fabric
@silas: The documentation is rather non-existent (the docs in the source are great though). There’s no samples. And for some reason, it doesn’t easy_install properly on my system. When I finally got it to at least install without complaining. It still drops an error everytime it runs about str object not being callable. (bizarre).
But yes, it looks like something that might just work. :)
I guess the closest to Capistrano in Python is “Pexpect”. At least, for the projects we have at work.
But believe me, Capistrano is a nightmare. We use it to deploy Python and Ruby projects and, honestly, not even the Ruby guys can stand its general brokenness.
Did anyone mention Vellum yet?
Ben – I wrote something like this for my last employer for java war deployments (in python)[0]. There’s an older version that was released as open source. I know that webapp-config (gentoo web deployment tool) was meant to be language nuetral but things get complicated fast. (Combinatoric explosion with adapters to deploy to different webservers/fcgi/proxying). What features do you want? Do you really want building in there too?
0 -http://panela.blog-city.com/warconfig__feedback_requested.htm
While Cap 1 was heavily bound to ruby, there has been a lot of work making Cap 2 much more language agnostic, and extendable to support more languages.
Thus, our capistrano is capistrano, no need for a knock-off and market fragmentation
Perhaps you should spend more time looking at zc.buildout… Writing a new recipe that does specific OS tasks is pretty trivial, and I imagine with a bit of effort you could write the missing bits that you need.
Cool, I didn’t know about Vellum, I’m going to take a look at it.
I did know about fabric but never used it, I agree that we absolutely need a capistrano like tool.
Regarding capistrano, in ruby land there is also Vlad the Deployer (what a name).
Capistrano is mostly a deployment tool, but it reaches into the build space as well, which is not so good. If you mix “my application consists of egg A 1.1, egg B 1.2 and PostgreSQL 8.1” with “I have a development server named mindy and a production server named mork” then your build tasks are only applicable to your own set of machines. Fine for 37 Signals who was deploying web apps on their own private network, but not so great when you want to share just the build tasks with developers on different networks.
zc.buildout is a great Python build tool (and I’d take it over Capistrano/Rake any day, even if there were Python ports of those tools), pairing it with a remote deployment tool would be quite nice, as long as the coupling between the tools doesn’t go beyond the deployment tool kicking off buildout commands.
Is zc.buildout ‘magical’? I would say no. The documentation can be accused of being “exhaustive” and “inscrutable” which can be quite frustrating when learning how to use the tool, yes, but it works in a straight-forward manner once you undertsand it.
The fact that the tool relays heavily on pulling packages down over the network means that it’s easy to get bit by a single-point of failure if PyPI or some other server is having a bad day. This can be an annoyance at times in development, but you can pull entirely from your own internal location for production usage. And for all those times when you pull out someone’s package fresh from SVN and let buildout put together all the dependencies required and “boom!” working application, it’s pure gold.
As for the sys.path munging, it’s ugly, but all package selection techniques in Python are ugly right now. sys.path munging is largely only an aesthetic wart though. It is a deterministic solution, as opposed to resolving eggs at run time, so this is quite nice. Path munging also let’s you pick eggs from a single cache directory, which is very nice when you have many different egg-heavy apps on the go on your development machine. Very large buildouts can be assembled from scratch in a matter of seconds since it only needs to select packages in your cache directory. Finally, sys.path munging is only an approach taken by many of the recipes used by zc.buildout, it’s not mandatory – it would be nice to have recipes for creating bash files that export a PYTHONPATH environment variable so that you can run sys.path munge free scripts with eggs selected by buildout.
One thing about Ruby, Rails, Mongrel, Rake, etc. is that the need for a Capistrano tends to be more of a necessity. I agree that Python could use a similar tool, but really, I would assume a generally better tool could be written in Python. After working with Mongrel, Monit and Rails in a production environment, the issues are huge and all revolve around threading. The reality on the threading front (when you don’t get decent out of band queuing) is nothing really can be depended on in the long run. I compare this with deploying a simple WSGI app it is a pretty stark contrast.
Again, I’m not saying that a Capistrano knock-off wouldn’t be helpful. But, the point where you require such a tool is no where close compared to Ruby/Rails (IMHO).
Check out zc.sourcerelease (http://pypi.python.org/pypi/zc.sourcerelease). With it you can make a release from a buildout that includes all the necessary source to build a project.
From a source release you can build debs, Windows installers, or RPMs (that’s what my project does).
I’ve been thinking about this problem a bit, too. I pestered a few people at PyCon about it, and was thinking of working on a tool myself…
But I’m now thinking that Vellum may very well make sense to be that tool, with a bit more work of course.
I’m a big fan of zc.buildout and think it works quite well, BUT I think that the need to create new recipes is annoying when you really only want to run an extra couple of commands somewhere. This is where I’m hoping Vellum can be put to use.
zc.buildout is basically 100% declarative. I think what you want is something that is more obviously a combination of declarative and imperative.
One plus is that zc.buildout appears to have a good library of routines that could be wrapped in Vellum.
I’m going to be experimenting in the next couple of days with making a Vellum script that builds an egg, copies it elsewhere and kicks off a buildout over there.
I do think that something like Vellum that wraps existing distutils, setuptools, paramiko, zc.buildout and maybe even virtualenv as appropriate would make for a super handy tool.
@Kevin Dangoor
You can have a buildout recipe that runs extra commands, I bumped across one (http://pypi.python.org/pypi/plone.recipe.command) and PyPI the other day.
But buildout only re-runs recipes when the configuration files change. This is useful when putting application parts together, but obviously for source code recompiliation or other tasks that you want to invoke repeatedly in “buildtool target” format then Vellum looks like it may become quite nice.
@KevinDangoor the “zc.buildout.extensions” entry points are an easy way to do brute force or mor implicit stuff without needing extra recipes. Since zc.buildout loads them implicitly (sort of a problem for finer control), you can stack them in your build if you like. Each ep is just a function that takes a buildout object.
@kevinteague re:compilation: Isn’t this what buildout install ${partname} does?
@Ben I’ve found that the -D flag (and recompiling the pyc’s so trace following in emacs works) to be a pretty good way of debugging zc.buildout. There doesn’t seem to be anything particularily magical about it beyond eggs and entry points.
It would be nice if there was some better documentation for mortals (and a recipe index). A cookbook approach is missing.
@whit
oh yeah, “buildout install [parts]” does that – it’s right there in the—help documentation.
@Ben
I wonder if people find the tool magical are referring to fact that buildout fetches the recipes as eggs over the internet. This seems to be a point not always gleaned when first using the tool, there are a number of gripes against buildout that are usually the behaviour of a particular recipe and not something that’s built into the tool itself.
@Ben I also experienced the weird str not callable error. It’s just an issue in the fab executable. It is expecting to be able to call exit(), which works fine in python2.5, without calling it from sys.
If Fabric wants to be a replacement for cap in the python world then there needs to be at least one example and a little bit of documentation.
Buildout is an awesome tool! We’ve been using it to take care of our full release cycle, from local to prod. That last missing step is the pushbutton release that cap gives you.
Regarding Fabric: I exchanged a few emails with Karmazilla (the guy developing it) and I should point out that it is at a very early stage of development.
I gather the first few releases were more prototyping than an actual example of what Fabric will become.
I think with some concerted effort, and a little cross pollination from vellum and zc.buildout, Fabric may be our answer to Capistrano. At least, I’m hoping so.
RC
Ben says about Fabric:
Not quite true anymore: http://www.nongnu.org/fab/tutorial.html
:)
But it can still get a lot better, though.
Ben, Sorry for this somewhat delayed reply, I’ve only started reading your blog recently.
Have no first hand experience with Paver but reading your post, it sounds like Paver could cut it for you.