Groovie

I gave a presentation tonight on SQLObject & FormEncode for the Bay Area Python Users Group (BayPiggies). Here are the slides for the presentation, which actually have 3 extra slides I didn’t show during the presentation because I forgot they were there (doh!).

Anyways, there were rather important I think as they actually showed two common cases where you’d want to use SQLObject and FormEncode together. That is, for moving data from a CSV file to a database, and for dealing with web forms. The latter of which I previously described in much more detail on here but it would’ve been nice to go over in the presentation, ah well.

If you’re curious about FormEncode, I put a decent amount of info into it along with some examples. It’s worth checking out if you’re ever in need of data validation and converting data types.

On many occasions, its quite useful when developing web applications to have the webapp know whether its in a production/development/testing environment. Rails builds this into the framework and its rather easy to add this toggle throughout a Myghty webapp (or some other Python web framework) as well.

We’ll make use of an environment variable so that the webapp environment can be easily configured from lighttpd or Apache. This way just deploying the application under a different handler can toggle the web applications mode of operation. The other thing we’ll setup while we’re at it, is a variable to keep track of the absolute root of our web application. I’ve found it quite useful in many cases to be able to get at this information without hardcoding it in, this way its always accurate no matter where the program is.

There’s a few rather commands that’ll give you the information we’re looking for. It took me awhile to find this, so hopefully it’ll help someone else out there.

Getting our Absolute Location

Let’s assume our directory hierarchy looks like this:

webapp/
   templates/
   components/
   scripts/
      dispatch.fcgi

Maybe you have the script called by mod_python or lighttpd somewhere else, I’m assuming it’ll be inside the root of your web application somewhere. In this case, the handler called by lighttpd is dispatch.fcgi. So dispatch.fcgi needs to figure out what the absolute path of the directory is above it. Here’s the code that figures this out:

import os

# Set the prefix to our base path for the webapp
myloc = os.path.join(os.getcwd(), __file__)
prefix = os.path.normpath(myloc + '/../..')

The myloc assignment gets the absolute file-name with path of the current file, in this case dispatch.fcgi. Unlike a normal os.getcwd(), this call doesn’t care what directory we happen to be in when we import the module, it will always be the absolute file-name of that file.

In case you’re curious why this happens, __file__ will return the path of the file relative to the working environment its called from. So combining it with the full path name of the current working environment results in the complete absolute path of the module no matter what the current context or working directory.

The prefix assignment uses the normpath call to strip off the filename, and backup one directory to our webapp root. This leaves us with the absolute path to our webapp/ directory.

Setting and Using the Environment variable

Figuring out if we’re in a special runtime environment is quite easy and looks like this:

import os

MYGHTY_ENV = os.environ.get('MYGHTY_ENV') or 'development'

Now we can just test MYGHTY_ENV to determine whether to contain errors ourself, or drop them to the web (as you would want in development mode). We default to being in development mode, since this is what you’d typically run your webapp in. To switch it to production mode, here’s what the lighttpd config looks like:

fastcgi.server = ( 
   ".fcgi" => (
     "www" => (
        "min-procs" => 2,
        "max-procs" => 4,
        "socket" => "/tmp/webapp.socket",
        "bin-path" => "PATH/TO/webapp/scripts/dispatch.fcgi",
        "bin-environment" => ("MYGHTY_ENV" => "production" ),
        "idle-timeout" => 120
      )
    )
)

To set the extra environment variables with Apache, use mod_env with the SetEnv directive which would look something like this:

# Make sure you have mod_env loaded, this line assumed to be in the VirtualHost
# block of your config
SetEnv MYGHTY_ENV production

At this point, you might’ve noticed (if you’ve used Rails) how similar my Fast CGI setup with lighttpd looks when compared to the some of the Rails examples for a lighttpd + Fast CGI setup. This is intentional, as I’m adding a Routes dispatcher to Myghty so it makes sense to layout my web application in a similar directory hierarchy.

Anytime you need to toggle some behavior depending on your webapp’s runtime context, just import os and check it as I showed up above.

In a recent Wired article regarding One Login, reference is made to a new social style network called GoingOn. The article spends most of its time focusing on one site that hopes to aggregate functionality that currently is split between Blogger, Flickr, Friendster, and Bloglines (for the most part). However, the thing it misses is what I previously discussed regarding the lack of a working distributed identity system.

After looking around more, I’m happy to say there are indeed working identity systems out there. Unfortunately the most promised of them, the Liberty Alliance doesn’t seem to have much oomph behind it, but two others that I previously didn’t know about are now out there.

The first is from the folks at Microsoft, which they’ve called an Identity Meta-System (or something like that), which is described over at vnunet. It seems to be rather tied (or at least integrated heavily) to Microsoft technology (go figure!), and will be included in Indigo and other various Micrsoft technologies. As a mainly open-source coder, this has little appeal to me, nor am I about to start using Microsoft API’s to write my websites and web code. The standards utilized by Microsoft for their Federated Identity are generally known as WS-* for some reason I’m too lazy to investigate.

The second is much more appealing (to interested users and web developers), and has actually been around for a very long time in a primitive form (2000 is ancient by web standards). The home site appears to be the identity commons, and the current sole Identity Broker is 2idi, the organization behind the standards is XDI. They’ve made the entire code-base they run the Identity Broker on, open-source under the Affero General Public License to ensure that users are never locked into just one Identity Broker (Yea!).

If you’re curious how the Microsoft and Liberty Alliance methodology differs, idcommons has a useful FAQ addressing the differences.

The most exciting aspect for me, is that all the technology behind the XDI approach is completely open-source, and geared towards maximum user flexibility and empowerment. The user gets to move data between Identity Brokers, and every care has been made to ensure the user is never locked into a single Identity Broker. Actually, the most exciting part, is that it works right now. :)

They’re currently preparing to switch to a SAML-2.0 backed code-base, however the code they have only works from PHP, Java, and Perl. If you want to try it out, here’s how to get an i-Name, and you can try it out on those two sites. Also, a developer made a ISSO (I-name Single Sign-On) authentication system for WordPress which is pretty cool.

So what’s stopping ISSO from being used on more websites? It’s free, its open-source, its standards based, its not controlled by a commercial corporation….

It needs Python libraries!

I should mention, when I first wrote this as far as I knew, there was no Ruby version. There still isn’t a public one, but Victor Grey is fairly close to a Ruby version with a full Rails rig to go with it which I’m rather looking forward to.

Anyone want to help? I’m tired of remembering a zillion usernames and passwords, and with ISSO on the horizon I shouldn’t need to, all the Python web frameworks will be a bit better (at least the sites that use usernames/passwords) with an easy way to use ISSO.

By the way, for a useful overview of SAML, there’s a very detailed write-up of SAML2 on xml.com.

Two of my favorite and most often-used Python packges are formencode and sqlobject. Using these packages together is done fairly often, but I’ve rarely seen any documentation describing a full form display, validation, and insert into a database of new data from said form.

In this example, I’ve tried to wean down the code as much as possible to get a very concise snippet (only 12 lines of web page Controller code) that will:

1. Display a new form to the user
2. Handle form submission
3. Validate the form
4. Coerce form values into the proper types for the database
5. Insert the form data to the database, or
6. Display the form with error messages, and maintain their values

The nice thing about using formencode, is that 3 of the 6 steps I listed above are handled by it in a fairly automatic way. Formencode will check the data given a form schema, coerce the values into the Python types you ask for, fill in errors should they occur, and maintain the existing values.

I’ll be using Myghty for this, but since all I’m really pulling from it is the request args, it should be pretty obvious what to change for whatever web framework makes you happy.

formencode

First, lets take a look at our basic form:

# myform.myt
<html>
<head><title>basic form</title></head>
<body>
<form action="/mypage" method="post">
Username: <input type="text" name="username" size="26" />
            <form:error name="username">
Age: <input type="text" name="age" size="3" />
            <form:error name="age">
<input type="submit" value="Send it" />
</form></body></html>

To validate this, we’ll setup a formencode form schema to run this through. I should note at this point, that the formencode web documentation kind of sucks. However, the doc strings are plentiful, and extremely useful for figuring out which validator to use in addition to having examples of each. To keep things organized, I usually place related form schema classes under the same module and import it as needed.

The other thing you might notice about the form, is that it has form:error fields. These are used by the formencode parser to put in the error message that the validation triggers. This lets us put the error messages right under the boxes they occured in. The drawback is that we have to process the form before first displaying it to strip out the form:error fields.

Here’s our simple schema to validate the above form:

from formencode import schema, validators
class UserInfoSchema(schema.Schema):
    allow_extra_fields = True
    filter_extra_fields = True

    username = validators.String(not_empty = True, max = 50)
    age = validators.Int(not_empty = True)

Hopefully the above should look pretty obvious. The allow_extra_fields bit is needed so that we can pass the entire request argument dict into formencode without it tripping up if there’s “extra” keys it didn’t expect (like the submit button). Since we’re going to be passing the dict we get back from formencode directly to sqlobject, we include filter_extra_fields to remove anything that our sqlobject isn’t going to like.

The form schema needs to include all the fields the database is going to take, since we’re stripping off anything it doesn’t mention. The Int validator not only ensures that the value is an int, but will change it into a Python integer in the process.

sqlobject

Now that we’ve handled validation and value coercion, lets have a look at the sqlobject class. I’m going to “cheat”, and assume your database for this was created like so (in Postgresql):

create table user_info (
    id serial primary key,
    username varchar(50),
    age int(2)
);

Since I’m feeling lazy, we’ll rely on SQLObject to pull the table info from the database giving us a SQLObject class like so:

from sqlobject import *
class UserInfo(SQLObject):
    class sqlmeta:
        fromDatabase = True

Personally, I think if you just leave the whole thing empty and put pass in for the body, it should assume you want it populated from the database…. but the above will do the trick. Please note I’m using the sqlmeta class to define this, which is used in the recent svn builds of sqlobject. If you use the release on the site, you could replace those two lines with _fromDatabase = True instead.

Putting It All Together

Now that our form, validation, and sqlobject schema is all done its time for the meat of the matter… the web page controller. Getting this function called will vary depending on your web framework, so I’ll just assume you can figure out how to get it called, here’s what it looks like in Myghty using implicit module components:

from formencode import htmlform
from ourschema import UserInfoSchema
from oursqlstuff import UserInfo

def mypage(m):
    html = m.scomp('/myform.myt')   # load the form into a string
    form = htmlform.HTMLForm(html, UserInfoSchema())
    if m.request_args:
        form_result, errors = form.validate(m.request_args)
        if errors:
            errorForm = form.render(m.request_args, errors)
            m.write(errorForm)
        else:
            UserInfo(**form_result)  # database insert
            m.subexec('/thankyou.myt')
    else:
        m.write(form.render())

And there you have it. In a brief 12 lines, we handle displaying a new form to a user, and handle form submission, validation, and database insertion while ensuring that the string values are coerced as needed before database insertion. This task is done quite often in web sites, so making this task as painless as possible is a real time saver.

Hopefully this will help out anyone out there, who was wondering about quicker and easier ways to handle cases like this. If you have any thoughts/suggestions on how to streamline this further, be sure to leave a comment.

I’ve been thinking a lot lately about web programmers and the web frameworks they choose, or don’t choose, and why. I’m mainly going to talk about Python Web Frameworks as the majority of them have small communites, and possible reasons this could be.

I only started using Python for web development about a year ago, and it took me about a month to settle down on a web framework. In that time, I looked over at least a dozen different frameworks. There’s so many python web frameworks, quite a few people have actually setup entire pages and sections of their site just to covering them all.

I think part of the reason for the proliferation of frameworks is because of the nature of many Python programmers, as I briefly mentioned in a prior post on Making Decisions for Others.

The recent appearence of Django on the Python web framework scene I’m sure has quite a few other Python web framework developers wondering, “Why isn’t the web framework I made getting this much attention and use?”

A Common Base

Many of these same people would like to blame it on hype and good marketing. While that will certaainly boost initial usage, I don’t believe it will create a lasting user base. I think a huge driving factor behind Rails and Django, besides for the hype and marketing, is the fact that both of them make a lot of decisions for you. These decisions start the users all off at a common base of understanding.

The linear progression from:

1. Never used the framework
2. Wrote the tutorial app
3. Wrote their own basic webapp
4. Wrote an advanced web application

Makes it easy for people a step or two up, to help other new users join them. Because the steps they all take are the same steps to achieve greater understanding of the web framework, they can easily help new users get to where they are. Most, if not all the other Python web frameworks I’ve seen are so flexible its hard to have a common base of understanding amongst new users. The process looks more like this:

1. Never used the framework
2. Researched the frameworks options and choices to find a possible starting point
3. Wrote a basic web application using method X
4. Wrote an advanced web app using method X

The flexibility of the web framework becomes an obstacle to a strong user-base in this case, as it fragments the users by the methodology they’re using to build their webapp. It also reduces the common re-usable components available, since different users will utilize different options of the framework and have possibly very different starting points.

Have a Tutorial Application

Also lacking from many Python web frameworks is a clear and obvious Tutorial application. Ideally the front page of a Python web framework should be an obvious path to become an experienced user of said framework. Such as:

1. Install the framework
2. Write a basic tutorial application
3. Look here/there for instruction as need to write your own more complex application

A good tutorial should leave a user feeling confident that they know how to install and start with a common base for writing their own web applications. It’s also amazing how many problems people can have just getting a framework installed and running in a minimal configuration. Having a tutorial that leaves them with a functioning web application gives them a big leap forward.

Since many users will do the first tutorial web application, other new users can give help to even newer users that run into a problem. This is where the common base effect really provides some power.

Methods of Fragmentation

The Python frameworks I’ve tried and used have fragmented their starting points and users in various ways. All of them as a result of their “flexibility and power”. Here are a few common trends of fragmentation I’ve seen:

• Let the user choose various template language schemes (Use ZPT, or Cheetah, or…)
• Let the user choose from web paradigm (MVC, page-driven, pipelined…)
• No base or example configuration for a fully working webapp (So everyone sets up their first application slightly differently)

The last one I listed, is probably the easiest to solve, especially with useful web framework template creators like Python Paste. Obviously, removing the first two will be seen by many Python web framework developers as undesirable. I think it’d really help the users though, as it gives them more in common with each other. If they all use the same paradigm, and the same template language with your framework, their ability to help each other increases and they feel confident they made the “right” choice as well.

Assumptions

I’ve assumed for the purpose of this post, that Python web framework makers are interested in having a large user-base. This isn’t always the case, I’m sure some just want a small, very experienced user-base that isn’t going to be asking basic questions like, “I can’t connect to my database like you show in the tutorial”.

I can understand that, but for the other Python web framework makers out there, try and consider some of the things I mentioned. There are a lot of Python coders out there, and a lot of them can live without having 4 template language choices and 2 different design paradigms. So when adding that feature that’d let people get so much “power and flexibility”, will it fragment your user-base?