Groovie

This is slightly old as Routes 1.4 was released about a week and a half ago, but I thought it deserved some attention. There were a handful of fixes and some slightly major feature enhancements in 1.4.

From the changelog:

• Fixed bug with map.resource related to member methods, found in Rails version.
• Fixed bug with map.resource member methods not requiring a member id.
• Fixed bug related to handling keyword argument controller.
• Added map.resource command which can automatically generate a batch of routes intended to be used in a REST-ful manner by a web framework.
• Added URL generation handling for a ‘method’ argument. If ‘method’ is specified, it is not dropped and will be changed to ‘_method’ for use by the framework.
• Added conditions option to map.connect. Accepts a dict with optional keyword args ‘method’ or ‘function’. Method is a list of HTTP methods that are valid for the route. Function is a function that will be called with environ, matchdict where matchdict is the dict created by the URL match.
• Fixed redirect_to function for using absolute URL’s. redirect_to now passes all args to url_for, then passes the resulting URL to the redirect function. Reported by climbus.

Web Resources

The map.resource command is based directly off the Simply Restful Rails plugin which adds support for various verb-oriented controller actions in a RESTful service style approach. The Simply Restful layout is more or less the exact service style laid out in the Atom Publishing Protocol.

It’s a great approach and it also meant providing a few other features to Routes that I hadn’t implemented previously, the most important being able to limit matching of a URL based on the HTTP method used. This is present in the new conditions clause for a Route:

map.connect('user/:id', controller='user', action='edit', 
    conditions={'method', ['GET', 'HEAD']})
map.connect('user/:id', controller='user', action='update',
    conditions={'method', ['PUT']})

The conditions clause can also accept your own function should you want to restrict the route to matching based off some other criteria (sub-domain, IP address, etc).

def stop_comcast(environ, match):
    if 'comcast.net' in environ['REMOTE_HOST']:
        return False
    return True

map.connect(':controller/:action/:id', conditions={'function':stop_comcast})

David Heinemeier Hansson recently posted an entry about Resources on Rails discussing how important web services are. The other key point was to make it easier to write controllers that could not only give you easy browser access to your resources, but provide a web service API as well.

The two snippets shown above give you an edit and update capability that restricts matching based off the HTTP method (verb). Writing a huge mess of those for the rest of the functions needed for a full web service API like Atom is a bit of busy-body work, so in the opinionated style of Rails a single command wraps up the whole thing. In Routes it looks like this:

map.resource('user')

That will make the two routes at the top of this entry in addition to routes that handle PUT, and DELETE. It maps them out to a set of actions in the controller, and provides the capability to easily add more methods for specific verbs.

The map.resource command is still getting tuned up, and we’re integrating the additional functionality it provides back into Pylons as well. Josh Triplett also wrote some Python code that will parse HTTP Accept headers fully so that we can add some nice functionality to use in the controller to return the appropriate data given what the client is expecting (HTML, XML, JSON, etc.)

If you’re using a Python web framework that doesn’t use Routes… maybe its time to put a request in. :)

I keep seeing blogs and blog comments pop up with some very odd notions. Many of these are summed up in this comment from a recent post by Tim on Ruby books

“It’s doubtful that the Python folks can come up with anything as compelling (or elegant) as Rails. Why? Because Ruby is so good at creating Domain Specific Langauges (DSLs). Ruby’s anonymous code blocks are a big part of what enables DSLs to be written so easily in the language. Python doesn’t have them. Python’s lambda’s (and closures in general) are crippled as well, which also doesn’t help Python’s cause.”

What is a Domain Specific Language (DSL)?

I always like to carefully define my terms before actually talking about them. So first I looked up several common definitions of a DSL. We have Martin Fowler’s entry on a DSL which describes it as:

“The basic idea of a domain specific language (DSL) is a computer language that’s targeted to a particular kind of problem, rather than a general purpose language that’s aimed at any kind of software problem.”

He further clarifies between two types of DSL:

“The most common unix-style approach is to define a language syntax and then either use code generation to a general purpose language, or write an interpreter for the DSL. Unix has many tools that make this easier. I use the term External DSL to refer to this kind of DSL.”

“The lisp and smalltalk communities also have a strong tradition of DSLs, but they tend to do it another way. Rather than define a new language, they morph the general purpose language into the DSL. (Paul Graham describes this well in Programming Bottom-Up.) This Internal DSL (also referred to as an embedded DSL) uses the constructs of the programming language itself to define to DSL.”

To summarize, an Internal DSL satisfies one of the following:

• Uses a general purpose language, then morphs it into a language to fit the domain
• Adds functionality to a general purpose language such that solving a specific domains problem is easier

An External DSL:

• Completely new language syntax
• Typically requires code generation or interpreter for DSL

Martin Fowler espouses on how very dynamic languages like Lisp, Smalltalk, and Ruby are more conducive to internal DSLs as the meta programming capabilities and dynamic nature make it easy to extend and customize the functionality for your domain. I completely agree with his contention that an Internal DSL is limited by the syntax and structure of the language.

DSL’s in Python and Ruby

While the anonymous code blocks in Ruby are useful and syntactically nice, similar functionality can be done in Python through the use of generators and iterators (more powerful generator functionality similar to Ruby’s anonymous blocks are planned for Python 2.5).

Ruby also lets you extend built-in types, which is utilized by Rails to add a few helper functions to core Ruby data types. Python and Ruby both have closures that operate slightly differently, Ruby’s closure is definitely more “full featured”. However this typically isn’t a problem because of list comprehension’s, generator expressions, and generators/iterators.

Zope is a very clear example that its quite possible to build an internal DSL in Python. Whether they built it in a clear, easy to use way is up for someone else to debate. SQLObject makes dealing with databases as easy as ActiveRecord, and there’s many other examples of internal DSL’s built in Python (Take a look at twill, a web application testing language implemented in Python).

Is Rails a DSL?

First, it should be obvious that if Rails is a DSL, its an Internal DSL. The most noticeable strides towards being a web programming DSL in Rails:

• Over a dozen core Ruby classes are extended/modified
• Dozens of helper functions
• New classes (syntactic sugar) defining database access and web paradigms

Given how loose the definition of internal DSL is, I think its clear that Rails qualifies, so does Django, TurboGears, Aquarium, etc.

If you prefer to be more strict about an internet DSL, and argue that a lay programmer needs to be able to do a reasonable amount of ‘work’ without knowing the base language implementing it, Rails starts to move apart. You can do a reasonably large amount of work in Rails without knowing Ruby. A lot can also be done in Zope without knowing Python.

In either case, you can’t really start to make advanced applications in any of the frameworks without knowing the general purpose language its built on. DSL’s can and have been built in Python, and I believe if you go out and start counting them up, you’ll see more DSL’s implemented in Python than in Ruby.

Anyways, hopefully for people that run around re-iterating something they heard somewhere about DSL’s and Python vs. Ruby, this clarified something. It’s easy to create DSL’s in Python, lots of people do, and there’s lots of them. Python and Ruby are both great for making them, outside of Rails I don’t know any other DSL’s in Ruby. Perhaps someone can compile a list of all the internal DSL’s written in both Ruby and Python?

I’ve almost got a 1.0 ready of Routes. After reading Kevin Dangoor’s post on the mysterious 1.0 I’ve come to the conclusion that Routes is 1.0 ready. In case you aren’t familiar with Routes, I’d suggest taking a look at one of my earlier posts about it.

So where is it? It’s in the latest svn for now, because I’d like to actually have a nicer site with more full fledged documentation before the release. For example, some docs on how framework creators should go about integrating it, as well as more detailed and thorough documentation on usage. I also have to finish up one little change that will make Routes fully compatible with the WSGI spec when the path/script info is split-up (which happens when a WSGI app is put under a URL-space).

Rails routes suffers from a lack of documentation as well, and the most complete source of information is buried in the test units for it. There’s also good docs in the Agile Web Dev for Rails book, but that’s not exactly a free public resource. So I’ll likely be “porting” my docs back to Rails, especially since Nicholas Seckar has been a great help while working on this project.

Whats New?

Short answer, not a lot. The main thing was actually very trivial to implement, named routes. These act essentially as a short-cut for when you want to pull some possibly long pre-defined route defaults. Short-cuts are good of course, as they save you a bit of typing and in this case also make your URL’s more flexible should you decide to change how to get to a “named” route.

Here’s what a fairly basic Route setup looks like:

m.connect(':controller/:action/:id')
m.connect('', controller='home', action='splash')

To implement Named Routes I added the ability to specify an additional string before the keywords. Here’s an example:

m.connect(':controller/:action/:id')
m.connect('home', '', controller='home',action='splash')

Now take a look at using it inside a template:

# Without named routes
url_for(controller='home',action='splash')
url_for(controller='home',action='splash', id=4)

# With named routes
url_for('home')
url_for('home', id=4)

This is slightly different from the Rails approach as I was mainly interested in keeping it “Pythonic”. So there’s no extra symbols or functions created when using a named route.

As you can see, it can save a bit of typing as using a named route is sort of like having a set of keywords inserted for you.

A 1.0 Release

Kevin’s article gave me a lot of food for thought regarding whether I should keep incrementing the Routes version. I can’t see any reason not to just go 1.0 as my version of Routes will be feature equivalent (and then some) with the Rails version, is heavily unit-tested, and is used in a production environment.

I’m confident in the reliability of the code and its being used by quite a few people in production environments (myself included). Being 1.0 doesn’t mean its done, it just means its hit a point functionality wise where I’ve accomplished everything I wanted the “finished” product to have. Even 1.0 software has bugs, and when I think of more features I’d like to add, I’ll start on my way to 1.1 or maybe even 2.0.

Hopefully I’ll have the docs ready to go sometime before November at which point a fully redesigned web-page will be put up. Until then, if any of this has you interested, head over to the Routes site or take a look at the unit tests to get a feel for using it. If you want to see it in action, try out the latest version of Myghty which contains an easy-to-use project template using Routes integration (currently with the 0.2 Routes which is lacking Named Routes).

I’ve been doing a lot of comparisons and research into various Python web frameworks lately, mainly focused on MVC oriented frameworks. Some of them have templating languages they come with, some not so much. The thing I have started to notice is how similar the Controller in them all is starting to look. First though, lets see what they all can do when it comes to the Controller.

For comparison’s sake, I’m bringing out some examples using CherryPy, Myghty, Bricks, Aquarium, Ruby on Rails, and Django.

Please note I’m not including frameworks built using any of the ones mentioned as their Controller looks pretty much the same. I’m also not an expert in all these frameworks as I only have so much time in the day, so if I seem vague on something its because I don’t know for sure. Feel free to comment (or correct) and clear up any vagueness I express.

Dispatch and Responder Styles

All the frameworks I’ve mentioned implement a style of object publishing (sometimes referred to as dispatch-style) that maps to the URL coming in. If you have another name you want to call this, go for it, I haven’t seen anything consistent on what to call this stuff.

Three different styles seem to exist that make sure an object responds to the URL:

1. User creates a regular expression to explicitly map a URL to an object (Explicit Mapping)
   • (r'^polls/(?P<poll_id>\d+)/vote/$', 'myproject.apps.polls.views.polls.vote') (from Django tutorial)
2. Object is mapped out according to the URL (Implicit Mapping)
   • /admin/login/hello/—> admin/login.py/hello() (Myghty)
3. Programmatic rule set determines object that responds (Programmatic Mapping)
   • map.connect ’:controller/:action/:id’ (Rails)
   • m.connect(’:controller/:action/:id’) (Routes)

In the Python world, three different styles also exist for how that object (or function) is defined. The first one is most popular, and is similar to how Rails sets up the Controller. They are:

1. Create a class inheriting from a Controller object, methods of the class instance are called to respond (Class-method)
2. Create a callable object that responds (Callable)
3. Create a function thats directly called (Function)

Note that the last 2 are functionally equivalent for the most part, however a callable object has the typical advantages (or disadvantages) of persistence in some web frameworks, as such if a web framework only shows one of those styles I won’t mention the other. Curious what frameworks use what styles from the above two sets?

For this listing, I’ll refer to the mapping scenario as dispatch-style and the method for defining the response as the responder-style, also please note that some frameworks support multiple dispatch-styles and responder-styles.

Aquarium:

• Dispatch-Style Mapping 1^:
  • Explicit, Implicit, Programmatic
• Responder-Styles:
  • Callable (Though it looks almost exactly like the Class-method style)

Bricks:

• Dispatch-Style Mapping:
  • Implicit
• Responder-Styles:
  • Class-method

CherryPy:

• Dispatch-Style Mapping:
  • Explicit (Not with regexp’s though, each class is attached off the root to determine what URL leads to it)
• Responder-Styles:
  • Class-method (Must be explicitly exposed with @cpg.expose)

Django:

• Dispatch-Style Mapping:
  • Explicit
• Responder-Styles:
  • Function (Packaged in the same module if they’re related)

Myghty:

• Dispatch-Style Mapping ^1:
  • Explicit, Implicit, (Programmatic with Routes in upcoming 0.99)
• Responder-Styles:
  • Class-method, Callable, Function

Rails:

• Dispatch-Style Mapping 1^:
  • Programmatic
• Responder-Styles:
  • Class-method

1^: User can create custom dispatchers to determine responder object

Who did it first?

If we throw in two more frameworks that utilize CherryPy, like Subway, and TurboGear, we can start to see that using a responder-style of Class-method is pretty popular. What’s even more interesting is that all the Class-method Controllers look remarkably similar, and I don’t think its because they’re copying each other.

The Aquarium project appears to be approximately 4 years old now, but from looking at the mailing list had little or no users until only 2-3 years ago (One poster from 2001 describes his aquarium plants turning brown…). From the sourceforge changelogs, the Controller stuff was added with 1.4 release in 2004. CherryPy has been around for 3+ years and I’m presuming it had the Controller stuff in it from the get-go as it seems a pretty core function of CherryPy. Rails has been out for a year or so, and was used in a form internally for BaseCamp for awhile predating that.

If you really want to trace back this style of Controller, you’d most likely find yourself looking at Java Python code (See Phillip J. Eby’s comment below). I’m really just trying to point out here, that even if one of these projects saw something they liked from the other, none of them were the originators. They all took a good concept, and implemented it differently, though they’ve all started to look remarkably similar.

A Controller Line-Up

So lets take a look at some of the examples on the pages of each respective website. Django is the only one that really stands out, as they’re not using a Class to contain the methods, rather they’re grouping them as plain functions inside a module.

Aquarium

class BaseController(Controller):

    def __call__(self, callNext, *args, **kargs):
        """Save a reference to myself in ctx.controller.""" 
        self._ctx.controller = self
        callNext(*args, **kargs)

    def doMessageAddAction(self):
        """Add a new message.
        ...does a bunch of stuff...

I mentioned that it was Callable but functioned in a way like a Class-method. This appears to be because Aquarium puts a context and maybe other initialization stuff per request in the __call__ bit, then has it actually calls a different method to respond.

Bricks

from bricks.controller.crud import CRUD
from bricks import user

class Test(CRUD): 
    # Override the default delete() method
    def delete(self, rowid):
        return CRUD.delete(self, rowid)
    delete.expose = user(level=1, app='app')

controller = Test('todo')

Bricks adds an interesting ability here by attaching function attributes to indicate what authorization is needed to execute the function called. Some of the other frameworks shown here do similar things through function decorators.

CherryPy

from cherrypy import cpg

class HelloWorld:
    def index(self):
        return "Hello world!" 
    index.exposed = True

cpg.root = HelloWorld()

Notice that the class instance is attached to the location off cpg.root thats desired. To map a URL to a class instance farther down, say under /booga instead of /, you’d do that last line like so: cpg.root.booga = HelloWorld()

Pretty handy I think, and it beats writing regular expressions (though doesn’t appear as flexible). One of the reasons I prefer the Programmatic dispatch-style. Moving on..

Django

from django.core.extensions import get_object_or_404, render_to_response
from django.models.polls import choices, polls
from django.utils.httpwrappers import HttpResponseRedirect

def vote(request, poll_id):
    p = get_object_or_404(polls, pk=poll_id)
    ... does a bunch more stuff to use Django-supplied functionality...

If you imagined the functions inside a class, instead of a module, it’d look rather similar to the others….

Myghty

class HelloWorld:
    def doit(self, m, **params):
        m.write("hello world!")

helloworld = HelloWorld()

At this point, the similarity between Bricks, CherryPy, and Myghty should be pretty obvious. Aquarium is close as well, while Django is only a wee farther from what’s appearing to be the norm.

Rails

class HelloController < ApplicationController
    def index
        return render :text => "hello world!" 
    end
end

Doesn’t look too different from some of our Python MVC frameworks…

What it All Means

I have no idea. Really, I just thought it was fascinating that these different frameworks started looking so similar. Some of them have been incredibly similar before others, but obviously these approaches catch on and find themselves adopted into other frameworks.

I do start to wonder at what point they’ll be similar enough that the barrier to switching web frameworks is narrowed down to only a few dozen lines of code to change here and there. Sure, going to/from Rails is going to be tough as that also means a language change, but for Python programmers tempted by a feature a different web framework offers I won’t be surprised to see more people hopping back and forth.

With regard to the Python web frameworks, none of them appear to agree on a template language. Several of them use Cheetah, some use Myghty’s template abilities, one uses Kid, and several of them have their very own template language.

It’s also interesting to consider whether or not having more possible dispatch/responder styles is beneficial or detrimental to the popularity of a specific framework. Rails creator David Hansson purposely includes only one style for each as he emphasizes with a concept known as convention over configuration. He provides defaults for pretty much everything, and following conventions makes it easier to write code as it enforces a consistent programming and naming style. I can’t help but notice that of the frameworks I mentioned above, the ones with less (apparent) options and better defaults tend to have more users (With the exception of Bricks as its still in alpha).

You might have noticed I designated Rails as being capable of user-programmed dispatching. While Rails uses its routes system by default, the thing people miss is that they’re just defaults. You can get in there and totally alter the way it works if you want, Ruby doesn’t completely close classes so you could just re-open them and override any aspect of its dispatch system you like.

I think this a valuable lesson that some of the more “verbose” examples above might want to consider. If there’s lines of code in a controller that you have to put in there every-time, find a way to make it the default case so that boiler-plate isn’t needed. It only needs to be a default, an advanced programmer will know how to override it should they need to. There’s obvious limitations to this since things work differently in Python, but there’s still Python ways to do it in a way that feels “right”.

For the Python Web Frameworks I missed

I know, I didn’t cover SkunkWeb, Spyce, Snaklets, Webware and more. Please feel free to add the relevant information regarding your framework in the comments. It’d be helpful if you could indicate what Dispatch-Styles and -Responder-Styles your framework supports.

If I failed to properly cover one of the above frameworks, bringing that to my attention is most appreciated.

The comment system supports the Textile format.

In a recent Wired article regarding One Login, reference is made to a new social style network called GoingOn. The article spends most of its time focusing on one site that hopes to aggregate functionality that currently is split between Blogger, Flickr, Friendster, and Bloglines (for the most part). However, the thing it misses is what I previously discussed regarding the lack of a working distributed identity system.

After looking around more, I’m happy to say there are indeed working identity systems out there. Unfortunately the most promised of them, the Liberty Alliance doesn’t seem to have much oomph behind it, but two others that I previously didn’t know about are now out there.

The first is from the folks at Microsoft, which they’ve called an Identity Meta-System (or something like that), which is described over at vnunet. It seems to be rather tied (or at least integrated heavily) to Microsoft technology (go figure!), and will be included in Indigo and other various Micrsoft technologies. As a mainly open-source coder, this has little appeal to me, nor am I about to start using Microsoft API’s to write my websites and web code. The standards utilized by Microsoft for their Federated Identity are generally known as WS-* for some reason I’m too lazy to investigate.

The second is much more appealing (to interested users and web developers), and has actually been around for a very long time in a primitive form (2000 is ancient by web standards). The home site appears to be the identity commons, and the current sole Identity Broker is 2idi, the organization behind the standards is XDI. They’ve made the entire code-base they run the Identity Broker on, open-source under the Affero General Public License to ensure that users are never locked into just one Identity Broker (Yea!).

If you’re curious how the Microsoft and Liberty Alliance methodology differs, idcommons has a useful FAQ addressing the differences.

The most exciting aspect for me, is that all the technology behind the XDI approach is completely open-source, and geared towards maximum user flexibility and empowerment. The user gets to move data between Identity Brokers, and every care has been made to ensure the user is never locked into a single Identity Broker. Actually, the most exciting part, is that it works right now. :)

They’re currently preparing to switch to a SAML-2.0 backed code-base, however the code they have only works from PHP, Java, and Perl. If you want to try it out, here’s how to get an i-Name, and you can try it out on those two sites. Also, a developer made a ISSO (I-name Single Sign-On) authentication system for WordPress which is pretty cool.

So what’s stopping ISSO from being used on more websites? It’s free, its open-source, its standards based, its not controlled by a commercial corporation….

It needs Python libraries!

I should mention, when I first wrote this as far as I knew, there was no Ruby version. There still isn’t a public one, but Victor Grey is fairly close to a Ruby version with a full Rails rig to go with it which I’m rather looking forward to.

Anyone want to help? I’m tired of remembering a zillion usernames and passwords, and with ISSO on the horizon I shouldn’t need to, all the Python web frameworks will be a bit better (at least the sites that use usernames/passwords) with an easy way to use ISSO.

By the way, for a useful overview of SAML, there’s a very detailed write-up of SAML2 on xml.com.