Groovie

No one told me that in New Hampshire, the Democrats have a hidden power. They apparently have a mysterious force known as superdelegates. Yea, I know.

I’ve never been to TED, but wow are the talks there inspiring and insightful. They also cover a wide range of topics far beyond just IT. The majority of the talks are also available online, some in high quality 480 progressive mp4 as well.

The best thing I’ve always found about going to conferences, is the inspiration that I can get and find in the talks others give. TED inspires in a different way, so here’s a few of my favorites and why:

Dr. Dean Ornish: The world now eats (and dies) like Americans Only 3 minutes, WATCH IT. The power of your diet to kill you is rather amazing, if you want an even more comprehensive background on this, read The China Study

Might You Live a Great Deal Longer? A very fascinating theme, which includes several talks, including the short 3 minute one I cited separately in the hopes more people would watch it. :)

Richard Dawkins: An atheist’s call to arms This one is sure to get a few people in a huff, for Richard Dawkins has few kind words for those who hold religious beliefs dear to their heart. But its a very interesting talk on science, and oddly enough I didn’t find it very scornful (which some do), probably my background in Philosophy.

Blaise Aguera y Arcas: Jaw-dropping Photosynth Demo This one has been posted by people before, I have to include it cause its definitely rather incredible to see. Definitely a tech theme on this one.

And finally, one of the most relevant to our current crises with regards to the environment and sustainability…

Janine Benyus: 12 sustainable design ideas from nature A very exciting talk about work being done to copy sustainable ideas from nature, also called biomimicry. If you’re looking for an upper, this one definitely left my with a smile and some hope for future developments that don’t require us to devastate places with mining, pollute the environment shipping raw materials around the world, etc.

I’m sure I missed a lot of good videos, if there’s something that really inspired or awed you that can be found in the TED talks, please share!

On the Internet, I frequently see terms misapplied to other contexts. Whether its a basic definition misapplied to a different realm, or an analogy grossly misapplied to a context that is not in fact similar. Since this particular term ‘conceptual integrity’ however was used as a beating stick on Pylons, I felt it’s something worth discussing.

First, its very useful to actually understand the context of the book the term comes from, The Mythical Man-Month. The Mythical Man-Month is about Fred Brooks observations as a manager on the OS/360 project at IBM, and the problems encountered when developing an OS for end-users. Before getting into the question of whether an OS end-user is equivalent to a web framework end-user, its useful to look at the term being referred to, ‘conceptual integrity’.

Wikipedia has a fairly good summary of the term:

Conceptual Integrity

To make a user-friendly system, the system must have conceptual integrity, which can only be achieved by separating architecture from implementation. A single chief architect (or a small number of architects), acting on the user’s behalf, decides what goes in the system and what stays out. A “super cool” idea by someone may not be included if it does not fit seamlessly with the overall system design. In fact, to ensure a user-friendly system, a system may deliberately provide fewer features than it is capable of. The point is that if a system is too complicated to use, then many of its features will go unused because no one has the time to learn how to use them.

Is there Conceptual Integrity?

The Pylons developers, including myself, spend quite a bit of time deciding what goes into Pylons, and what does not. Pylons is in fact very very small at its core due to our refusal to let ‘stuff’ into Pylons that is outside its strictly defined scope. In fact, a great many comments I get are usually about features people want, but that aren’t in the scope of Pylons to provide. I would argue that this does show thought has gone into ‘conceptual integrity’, in the explicit decisions made about what is and isn’t provided.

The scope of Pylons is to provide a small, concise, WSGI-driven, lightweight framework, that provides the flexibility for a web developer to use the best tools for the job at hand for building their web application. I would argue based on the definition, the goal of Pylons, and what Pylons has in it now, that it fully meets the design goal of having conceptual integrity with regards to feature scope.

While the original blog post I cited above starts with conceptual integrity, it delves deeply into framework design issues like, ‘Should a framework pick the ORM for you?’. This is slightly odd in the context of Pylons, as the default ORM, and template language are chosen, Pylons merely emphasizes that the choice is yours should you wish to change it (as does Django).

In another misapplied analogy, a parallel is drawn up between having an opinion on a writing style, to having an opinion on a library. Consider how odd this analogy is, that the fairly minor effect of adding a trailing ‘S’, should be compared to whether a particular ORM library is capable of meeting your demands. Should the framework authors decide whether the ORM meets your demands? Should the framework lose a significant amount of capability the second its stock ORM fails to handle your needs? These are interesting questions, well beyond an attempt to draw a parallel to adding a trailing ‘S’ to a word.

Since its clear that Pylons doesn’t encroach on conceptual integrity regarding features that are unused, the next aspect to look at is the concept of a ‘user-friendly system’. This becomes a little more difficult to analyze, because it brings into question the expectations one has of the ‘user-base’. It also shows how the original term of conceptual integrity was misapplied as another parallel is drawn, this time one that I think all web developers should be somewhat terrified by.

Website end-user == Website Programmer?

The last parallel, is to compare the programming expertise of a web developer, to the internet naivety of the website end-user. (Does that offend any other web developers?) The latter of which is well documented in various web usability books, such as the “Don’t Make Me Think” book. The underlying concept being:

• Website users have short attention spans
• Website users won’t use a website that requires thinking
• Website users have no patience

The thesis of this parallel is that a framework author is somehow in the superior position of knowing the appropriate library that will work for your task, and should therefore decide the tools for you. The parallel applied looks like:

• Website programmers have short attention spans (ok, I somewhat buy this one :)
• Website programmers won’t think about whether the tools at hand will accomplish their task
• Website programmers have no patience (Given the languages you have to learn to do anything, I seriously doubt this)

Now, there are quite a few website programmers that don’t think about whether the tools they use, work best for their task. I have seen many websites that are slow, inefficient, or just not well designed because of a lack of thinking regarding architecture, and tool choices. I would not consider these to be website programmers I would want to work with, nor are they the target audience or ‘end-user’ of Pylons. The door is purposely left open in Pylons, and lit up, with the choice available in libraries to use with Pylons.

Regarding making decisions, Pylons has choices for all the same components as Django has decided on: ORM: SQLAlchemy Templating: Mako URL’s: Routes Sessions: Beaker Caching: Beaker

The main problem apparently being, Pylons has emphasized that there is choice, that the framework is built so that they could be swapped, should one of these not be the best tool for your web application.

The Pilot System

When designing a new kind of system, a team will design a throw-away system (whether it intends to or not). This system acts as a pilot plant that reveals techniques that will subsequently cause a complete redesign of the system. This second smarter system should be the one delivered to the customer, since delivery of the pilot system would cause nothing but agony to the customer, and possibly ruin the system’s reputation and maybe even the company’s.

In response to the original post regarding conceptual integrity, I submit the issue of the pilot system. Django and other frameworks that start off with libraries you didn’t analyze in regards to your particular web application, have more than a few times led to pilot systems (I usually hear about the second incarnation, which was then done on Pylons).

Where you realize later down the road, “wow, this ORM just can’t handle the data model I have”, which then leads to, “well, since the ORM was hooked into the XML serializer, the admin UI, the form generator, etc. there’s now no competitive advantage over Pylons… which didn’t have those extras.” Systems with heavy and deep integration of libraries that may or may not meet your demands, where large parts of the system become relatively useless should you stop using one part of it, are extremely prone to being a pilot system.

When your pilot system is done, I invite you to check out Pylons, and think about what is going to work for your website application.

Notes

Please note that I take deliberate care in referring to Pylons for use with developing web applications, its my belief that Django, and similar systems in other languages with an ‘app’ centric approach like Joomla, Drupal, etc. are excellent for making web sites. While you can easily make websites with Pylons apps, this is definitely not the primary use-case emphasized and designed for, so it is harder and less obvious in many respects than Django.

The other thing everyone should’ve learned from these posts, is that The Mythical Man-Month is a rather good book worth reading. ;)

In the latest 0.9 release of Beaker, I’ve finally added cookie-side session storage. I was a little bit moved to finally do this by seeing that Rails 2.0 had added cookie-side session storage, and heck if I was going to miss out!

A few changes from how Rails 2 did it though, I was definitely not content to store all the data in an end-user visible form in the cookie. That only left encryption as the next logical choice, and that quickly led me down a path of quite a bit of cryptography research.

The world of cryptography is a constantly evolving and rapidly progressing field. New papers are coming out all the time with new research on ways to break, or ‘wound’ a particular encryption scheme. Providing a weak form of encryption in Beaker would be worse then not having it at all, since it’d lead someone to falsely believe the session data was secure.

After initially going with an RC4 cipher implementation, I got ahold of some crypto people that are actually in the field, and the unanimous opinion was to use AES encryption in Counter Mode, also referred to as AES-CTR along with a signature to prevent tampering (you’d be amazed what you can do to encrypted data, and it’d still technically decrypt). This led to a slight increase in requirements unfortunately, as pure Python based AES encryption is a bit slow. This means that using cookie-based sessions in Beaker requires the installation of PyCrypto, which includes a C extension (making cookie-based sessions faster than file-based, memcached, and db-based sessions).

The final solution in Beaker uses 256-bit AES-CTR with a 256-bit HMAC for authentication purposes. It’s fast, secure, and scales across a cluster without a problem. It’s not for everyone of course, cookies are rather severely limited in size, so if you’re just storing a few small tidbits of information in a session, for example:

• a user id
• some flags about the users status (logged in, etc)
• a flash message

Then cookie-stored sessions might be perfect for you.

Update: Forgot to mention, in the future, Beaker will probably use pycryptopp instead of PyCrypto since the PyCrypto library’s AES-CTR implementation isn’t as efficient as it could be, and will be using VMAC’s instead of HMAC’s for even more speed. Plus, apparently Andrew Kuchling isn’t maintaining PyCrypto, as there’s quite a few patches for it sitting unanswered on the sourceforge and launchpad bug trackers.

As many have blogged about, Apple has posted a rather hefty list describing all 300 new features of Leopard. Some specifics that really got the coder in me excited:

(Highlighting emphasis all mine)

Cocoa Bridges

Use Ruby and Python as first-class languages for building Cocoa applications, thanks to Objective-C bridges as well as full Xcode and Interface Builder support.

Scripting Bridge

Use Objective-C, Ruby, and Python programs to automate Mac applications. The new Scripting Bridge enables them to easily generate AppleEvents using a concise, AppleScript-like syntax.

Create Instruments with DTrace

Monitor system activity from high-level application behavior down to the operating system kernel, all thanks to the power of DTrace and the instrument builder.

DTrace

Monitor virtually any aspect of your application with DTrace, integrated into the Darwin kernel. Java, Ruby, Python, and Perl have also been extended to support DTrace, providing unprecedented access for monitoring the performance characteristics of those languages.

While some of the things can be done now (There’s PyObjC to use Python with Cocoa), the tighter integration of Python and Ruby into Leopard make it a very appealing upgrade (Even though I rather hate the new semi-transparent menubar and odd shelf nature of the Dock).