Groovie

Ruby’s anonymous block capability is probably the main feature I find myself wishing Python had on more than one occasion. While the upcoming Python 2.5 PEP 323 provides for the “with” statement which will enhance generators to get a bit closer. Though as Ryan Tomayko notes, this still doesn’t make the block available within the “generator”.

There are of course, many uses for having the block (with its closure scope), available in the function/generator you’re using. I’ve seen this used to easily register code call-backs with a CleanUp/Initializaiton manager, and other cases where its preferable to actually retain the block in its entirety for later execution. It’s also very useful when you want the function to control execution of the block, and return its output in a modified form.

In these ways, it would appear that the generator enhancements won’t quite be bringing the full power of Ruby anonymous blocks to Python. However, I recently found out that a relatively recent feature in my favorite Python template language, Myghty, implements something quite close to this.

Myghty’s Component Calls with Content

In a previous entry on Formencode and SQLObject I noted how useful the component call with content can be. What I failed to note (I didn’t know at the time as well), was that not only is the function/component able to get the content of the content “block”, but its also able to execute it again and again.

Hi, lets translate the content under:
<&| MODULE:mylib:translate, lang='es' &>
    This entire block of content will be sent in as a 
    single variable to translate.myt
    for use. This includes any <b>HTML tags and such</b> as well.
</&>

from mytranslater import translated

def translate(m, lang):
    body = "<p>The translated text is:</p>" 
    body += translated(lang, m.content())
    m.write(body)

The m.content() call can be called as many times as you want the output of it, and it retains the scope of its original location. This in many ways emulates how Ruby can yield to the block and capture its output, however it is not possible to stash the block itself (in Myghty/Python).

Next up is to actually be able to use values from the function inside the block. In Ruby this is done in a very elegant fashion letting you declare what you’ll call them, then use them, like so:

SomelistOfints.each do |item|
    item += 2
end

To emulate this behavior, our block has to make a special function call and know the name of the value in advance. Here’s an example:

% randomvar = 423
<&| MODULE:mylib:translate, lang='es' &>
    This entire block of content will be sent in 
    as a single variable to translate.myt for use. 
    This includes any <b>HTML tags and such</b> as
    well. Here's something supplied by
    translate: <% m.content_args['value'] %>.
    Of course, <% randomvar %> is still in scope too.
</&>

And just for fun, our translate function will call the block with some different values:

def translate(m, lang):
    body = "<p>The translated and repeated text is:</p>" 
    for val in range(0,4):
        body += m.content(value=val)
    body += "<p>That's it, nothing else.</p>" 
    m.write(body)

The important thing to remember that makes the concept powerful is that the block above is called in the scope you saw it in the template. Whatever variables were available there are used as normal.

Close…

This method brings us close to Ruby’s anonymous blocks, as close as might be possible in Python. Unfortunately its only usable within Myghty (if not, please let me know), and its still not true anonymous blocks. At the very least, its close enough to make me happy for now. While I could just switch to Ruby entirely, there’s still way too many things about Python that I’d miss.

From the details of PEP 343, it appears that this full capability to pass the block in was purposely avoided as having flow control in the “macro” makes the code inscrutable. Hopefully someday the utility and power such functionality provides will result in it being available in Python. Or at the very least, some clever person can try a variant in PyPy and see what sticks.

And yes, I know nested functions can be used and passed around with their closure, but its pretty annoying to be nesting functions back and forth solely for that purpose. It feels like all we have is a hammer…

Further reference:

Myghty docs on Component-Call-With-Content Arguments

In a prior post on handling form data I covered how to easily populate a form using FormEncode/SQLObject and validate/save the data.

One of the things I noted was that I had to take the HTML form output first, and use htmlfill with it to populate it with defaults and errors before displaying. This required some extra lines I wasn’t to thrilled with in the controller. After an evening of chatting with Ian Bicking about whether FormEncode could somehow be cleaned up to make this easier, Ian suggested I use a Myghty feature I had forgotten about.

Component Calling with Content blocks in Myghty

Somewhat ironic since I use Myghty all day and Ian doesn’t (afaik). It’s a very useful feature called Component Calls with Content. It’s probably easiest to understand the cool ability this provides by seeing it. In Myghty, each template is known as a component, and complex arrangements can easily be put together by component inheritance and component calls between them.

Normal Python function calls and expressions in a Myghty component (template) looks like this:

Hi everyone, 2 + 2 = <% 2+2 %> and your name is <% lookup_name() %>

To include content from other components, you can either call the component through the function call m.comp('/some/template.myt') or you can use the component call syntax:

Hi everyone
  <& /sidebar.myt &>

So here’s how we will use a component call with content:

Hi, lets translate the content under:
<&| /translate.myt, lang='es' &>
    This entire block of content will be sent in as a single variable to translate.myt
    for use. This includes any <b>HTML tags and such</b> as well.
</&>

And the translate.myt template might look like:

<p>The translated text is:</p>

<% translated(lang, content) %>

<%args>
lang
</%args>
<%init>
from mytranslater import translated
content = m.content()
</%init>

This is probably a lot to absorb, as it utilizes a few different Myghty concepts. Components can have arguments they expect, here the translate.myt component expects a lang arg which is passed in. The content block is then available as m.content() inside the template. The <%init> block is called first and variables defined there are available inside the template.

So how does this help us with the original problem?

Module Components

So far, all of these abilities are possible in Myghty’s ancestor, Mason. One more powerful construct available in Myghty however is the ability to not only call components, but Module Components.

I’ve been familiar with these for awhile as I use them for Controllers. It had never occurred to me though, that they can call any function in a module as if it was a component, not just ‘Controllers’.

Module components allow you to do a variety of interesting things, mainly, calling functions, classes, or objects that are in Python modules. Let’s take a look at the last translate.myt example using this approach instead.

Hi, lets translate the content under:
<&| MODULE:mylib:translate, lang='es' &>
    This entire block of content will be sent in as a single variable to translate.myt
    for use. This includes any <b>HTML tags and such</b> as well.
</&>

And the mylib Python module (assumed to be in the search path):

from mytranslater import translated

def translate(m, lang):
    body = "<p>The translated text is:</p>" 
    body += translated(lang, m.content())
    m.write(body)

Myghty will examine the function signature to determine what variables it wants and will search the current scope to make sure they’re passed in. Very handy. :)

Putting it All Together

So let’s see how this helps us out, first, rather than having to take the form and render it in the controller, we’ll push this into the template using a component call with content. So our new myform.myt looks like:

# myform.myt
<html>
<head><title>basic form</title></head>
<body>
<&| MODULE:mylib:formfill, defaults=defaults, errors=errors &>
<form action="/mypage" method="post">
Username: <input type="text" name="username" size="26" />
<form:error name="username">
Age: <input type="text" name="age" size="3" />
<form:error name="age">
<input type="submit" value="Send it" />
</form>
</&>
</body>
</html>

<%args>
defaults
errors
</%args>

from formencode import htmlfill

def formfill(m, defaults, errors):
    form = m.content()
    m.write(htmlfill.render(form, defaults, errors))

And the new controller:

from ourschema import UserInfoSchema
from oursqlstuff import UserInfo

def mypage(m):
    errors, defaults = {}, m.request_args
    if m.request_args:
        form, errors = UserInfoSchema(), {}
        try:
            form_result = form.to_python(m.request_args)
        except api.Invalid, e:
            errors = e.unpack_errors()
        if not errors:
            UserInfo(**form_result)  # database insert
            return m.subexec('/thankyou.myt')
    m.subexec('myform.myt', defaults=defaults, errors=errors)

We’ve cut it down 1 line, from 12 to 11, however we no longer have to go through the slightly awkward maneuver of capturing the rendered form and running htmlfill over it in the controller. This is now handled easily in the template using a component call with content. The module component we use is extremely portable and can be used to wrap any form anywhere with automatic fill-in of errors and default values.

FormEncode 0.4 also deprecates HTMLForm which we were using before, so this will help out when thats no longer available.

Module component calls are quite useful, and make it easy to group together powerful function libraries that interact with Myghty parts. Being able to pass them content allows all sorts of new constructs to be created.

If you’re using Myghty with FormEncode and SQLObject, I’d highly suggest switching to this method instead of the one I blogged about before.

TextMate is a rather slick little text editor, with a bunch of cool automation stuff thats great for programmers. You’ve probably already seen it if you’ve watched any of those demo movies for some of the latest web frameworks.

Anyways, as I mainly use Myghty, I needed a syntax highlighting mode for its format. So I made a Myghty bundle for TextMate. This should also be useful for those using Mason as only a few minor adjustments regarding the underlying language highlighter need to be tweaked (Change source.python -> source.perl). This also has a few automation snippets to speed things up.

• Unzip
• Drag into ~/Library/Application Support/TextMate/Bundles/
• Restart TextMate (Not absolutely sure this is needed though)

Enjoy!

I’ve been programming web sites for many years, and have yet to come across a templating language as appealing as Mason / Myghty. To avoid confusion I’m going to talk about Myghty, but since its a direct port of Mason (plus some MVC stuff) all of my comments apply to Mason as well (unless otherwise noted). So if you find yourself stuck using Perl (or you prefer Perl) and something here sounds appealing, by all means use Mason as I did.

Despite Myghty only having come into existence approximately 14 months ago, the code-base is stable, very quick, and has been running in production environments for over 8 months. This is mainly because it started as a very direct port of Mason to Python, it then grew a few additional features that made it great for MVC use. The methodologies present in Mason (thus Myghty as well) are known to scale to very large and complex sites, as this list of Mason-powered sites shows. But it’s the little things added up that really make Myghty my template language of choice.

This is a rather lengthy post as I highlight and explain some core concepts of Myghty, please bear with me… also, if you’d like to follow along and try the examples out, its really easy to get started using Myghty with Paste.

% for person in people:
   <b>Hi</b><% person %>
% #end

# /autohandler
<html>
<head>
<title>Sitename</title>
</head>
<body>
% m.call_next()
</body>
</html>

# /index.myt
<p>Welcome to your site!</p>

# /autohandler
<html>
  <head>
    <title><& SELF:title &></title>
  </head>
<body>
% m.call_next()
</body>
</html>

<%method title>
Your Site
</%method>

# /index.myt
<p>Welcome to your site!</p>

<%method title>
<& PARENT:title &> - Welcome Page
</%method>

# /autohandler is the same as above

# /products/autohandler
% m.call_next()

<%method title>
<& PARENT:title &> - Products
</%method>

# /products/sprocket.myt
<p>Buy a sprocket, they're great....</p>

<%method title>
<& PARENT:title &> - Sprocket Details
</%method>

# /autohandler
<html>
<head>
<title><& SELF:title &></title>
</head>
<body>
<& /sidebar.myt &>
% m.call_next()
</body>
</html>

# /sidebar.myt
<div class="sidebar">
    <ul>List of sidebar stuff</ul>
</div>

# /autohandler
<html>
<head>
<title><& SELF:title &></title>
</head>
<body>
<& /sidebar.myt, argument=variable &>
% m.call_next()
</body>
</html>

# /sidebar.myt
<h2><% argument %></h2>
<div class="sidebar">
  <ul>List of sidebar stuff</ul>
</div>
<%args>
  argument = "Default" 
</%args>

# /somefile.myt

<%python>
x = 10
y = 20
z = []
for a in range(1,100):
    z.append(x*y+a)
</%python>

# /index.myt
<p>Welcome to your site!</p>

<%method title>
<& PARENT:title &> - Welcome Page
</%method>

<%init>
if m.cache_self():
    return

do_really_slow_stuff()
</%init>

component_root = "/path/to/your/components" 

component_root = [
{'main':"/web/sites/mysite/htdocs"},
{'components':"/web/sites/mysite/components"},
{'alt':"/usr/local/lib/special-comp"}
    ]

component_root = [
{'cobrand':"/web/sites/cobrand/htdocs"},
{'main':"/web/sites/mysite/htdocs"},
{'components':"/web/sites/mysite/components"},
{'alt':"/usr/local/lib/special-comp"}
    ]

Mmm, tasty, a headline of P-word’s. Recently Ian Bicking went on a bit of a release spree with a whole bunch of goodness that I’m way too lazy to link to in this paragraph. The ones I’ve been waiting for were all in the line-up: Paste, Paste Script, and Paste Deploy. I blogged about paste and setuptools earlier and hadn’t followed up as I indicated I would partly because I was waiting on their official release.

So, what’s the big deal? Paste and its buddies solve a host of issues that commonly confront Python web developers and web administrators, the front page of Python Paste does a good job of explaining briefly why each group should care. Thus, I’ll move directly onto the fun that comes with using them.

Yes, I think its fun, but that might be just because I have a thing for installing web frameworks…. so I’ll cover using Paste as two groups of users, as a Web Developer, and as a Web Administrator.

Using Paste as a Web Developer

First off, to create a web application we need to setup a directory for the project. Since we want our web application to be Paste-enabled, the template for the web application should be using setuptools. That way when we package up our web application and give/sell it on the Internet, the people (Web Admins) using it will be able to install and run it easily.

% paster create --list-templates
Available templates:
  basic_package:            A basic setuptools-enabled package
  myghty_modulecomponents:  Module Component Template
  myghty_routes:            Routes Template
  myghty_simple:            Simple Template
  paste_deploy:             A web application deployed through paste.deploy
  pylons:                   Pylons application template
  turbogears:               TurboGear application template

Cool, right? Well, maybe its just cause I like having a big toolbelt of web frameworks sitting around. Obviously not a lot of web frameworks have paste templates available yet, but more are adding support and I’ll be rather excited when that list is 20 or more long.

If you go and install the Paste packages, your list won’t be quite as long as mine (insert evil laugh) as I’m running the CVS of Myghty (0.99 release with these templates coming shortly), along with some other stuff I’ve whipped up. Merely installing a web framework package that’s Paste-enabled and has templates available will make it show up in that list.

Another interesting thing to consider here, is that there’s no requirement that only web frameworks are allowed to create directory templates. Maybe a web application you’re working on, is so powerful you want to make it easy for end-users to extend it with their own custom add-ons. You could provide a paster template that creates a plug-in ready directory template for custom themes. There’s a ton of power packed in the various ways you can extend these tools, building on the dynamic discovery stuff in setuptools.

% paster create --template=turbogears pygo
Selected and implied templates:
  TurboPaste#turbogears  TurboGear application template

Variables:
  package:  pygo
  project:  pygo
Creating template turbogears
  Creating ./pygo/
  Recursing into +package+
    Creating ./pygo/pygo/
    Copying __init__.py to ./pygo/pygo/__init__.py
    Copying model.py to ./pygo/pygo/model.py
    Recursing into templates
      Creating ./pygo/pygo/templates/
      Copying __init__.py to ./pygo/pygo/templates/__init__.py
  Copying +package+-start.py_tmpl to ./pygo/pygo-start.py
  Copying dev.cfg_tmpl to ./pygo/dev.cfg
  Copying prod.cfg to ./pygo/prod.cfg
  Copying setup.py_tmpl to ./pygo/setup.py
Running /opt/local/bin/python setup.py egg_info

As you can see, it just setup our new project for us, which I called pygo because it was the first name that popped into my head. If you’ve actually used TurboGears, looking at that should have raised some red flags as its missing a bunch of stuff that a TurboGears project needs. This is mainly because the template was created by Ian as a demonstration of how to make a Turbogears-style template. Hopefully an upcoming version of TurboGears will be Paste-enabled so I can create a new project like this (hint hint). :)

% paster create --template=myghty_routes pygo
Selected and implied templates:
  Myghty#myghty_routes  Routes Template

Variables:
  package:  pygo
  project:  pygo
Creating template myghty_routes
  Creating ./pygo/
...Bunch more files...

~% cd pygo
~/pygo% paster serve server.conf 
Starting server in PID 6090.

[server:main]
use = egg:PasteScript#wsgiutils
host = 127.0.0.1
port = 5000

[app:main]
use = egg:pygo#paste

That’s really all the Paste you have to worry about as a web developer (though it has even more capabilities you’ll probably want to use). We can see here that its using wsgiutils from PasteScript to run the server. You can easily swap that out for any of the server support that flup offers such Fast CGI, SCGI, or AJP.

~/pygo% python setup.py bdist_egg
running bdist_egg
running egg_info
... whole bunch of stuff here...
~/pygo% ls dist/
pygo-0.0.0-py2.4.egg

That’s it. This is easy, right? You’re ready to go ahead and give your egg to anyone running Python 2.4 now (You could make a source distribution and upload it to Cheese Shop just as easily). So let’s try and totally forget that we’re a web developer, and assume a different role.

Using Paste as a Web Administrator

% sudo easy_install Pygo                                  Searching for Pygo
Reading http://www.python.org/pypi/Pygo/
Best match: Pygo 0.0
Downloading http://cheeseshop.python.org/packages/2.4/P/Pygo...
Processing pygo-0.0.0-py2.4.egg
... more stuff happens...
Installed /usr/local/lib/python2.4/site-packages/pygo-0.0.0-py2.4.egg
Processing dependencies for Pygo

Now, since I didn’t actually upload it to Cheese Shop, I’ve faked that screen. But that’s pretty close to how it would’ve looked if I had released pygo and uploaded it to Cheese Shop.

After this one command, as a web administrator, we now have the pygo application installed. There shouldn’t be any need to be installing web applications over and over for every user that wants to run it as the vast majority of web applications get their customization and settings from a database.

[server:main]
use = egg:PasteScript#flup_scgi_thread
host = 127.0.0.1
port = 3000

[composit:main]
use = egg:Paste#urlmap
/blog/fred = fredpygo
/blog/janet = janetpygo

[app:fredpygo]
use = egg:pygo
database = mysql:/username@localhost/database

[app:janetpygo]
use = egg:pygo

These configuration files are quite flexible, and allow different instances of a web application to run at different locations. In that case, one of them is running for ‘fred’ at /blog/fred. Each block for an app can have additional arguments that setup the database to use, and other settings that should probably be customized for each user. I threw in an additional database argument for one of them as an example.

Maybe you’re not a web administrator, but a web user comfortable downloading and installing many of the other webapps out there (Typo, MovableType, phpBB, etc.). Setting up your own site, using whatever Paste-enabled Python webapps you want is just as easy. Ideally ISP’s and such would just run the easyinstall command to install whatever the ‘popular’ webapps of the week are, and they can update them easily, keep old versions around if needed, etc.

Making Python Web-Application Distribution/Use Easy

This is pretty powerful stuff, and its quite easy to use. Even if you’re not making a web application with the intent to distribute it to the world, Paste is still going to help you out in a lot of ways.

An ISP or webapp end-user only needs to know how to add your webapp to their config file and installing it is a breeze thanks to setuptools. I think this has an enormous benefit for the Python web community, as it’ll significantly increase the ease of use when it comes to installing and managing web applications. Python web developers will gain some good footing to create Python Web Applications that compete with and surpass PHP webapps, especially if more ISP’s start supporting Paste-enabled applications.

Well, that’s my hope at least as I’m sure the title of this section made clear.

For the Python web framework creators out there, I like writing web applications (with various web frameworks), and I really like how easy Paste makes it to create them, package them up, and use them. Please add Paste support to your web framework, I’ll be happy to help and the Paste mail list is very responsive.