Groovie

Tonight I was drifting through Upcoming to see what’s happening in my area. Turns out the first page of 50 some events only covered today…. and its a ‘boring’ Wednesday. As I flipped through page after page, eventually getting all the way up to this upcoming Sunday… it occurred to me that there is way way too much happening within a mere 30 miles of me.

Of course, being a geek I’ve been thinking about ways to try and narrow down this list so I can actually get a useful list for what I’d prolly be interested in for the next 45 days. There’s just no way I can look through 1200 events to see what’s going on for the next month, and I like to plan ahead.

A few things I think would help to begin with would be keyword filtering, of course the next step that requires would be some way to build a useful keylist of what should be flagged for my perusal. Trying to generate a full list of every keyword I might be interested in is going to be pretty tough, though I finally thought of the one application I have which could yield the best list of keywords to filter on… Delicious Library.

It already knows the vast majority of books I own, and all my DVD’s. That’s not a bad starting point, toss in all the music I’ve bought from scouring all the id3 tags on my digital music and that’ll cover my music. I really need a way to pull all my previously watched, and upcoming movies from Netflix, in addition to mining all the data of who was in each flick (IMDB should have that). After which I should have a decent set of keywords to ensure I don’t miss either some special premier with an actor I might like, or a comedian doing standup.

Is there anything like this already that I just haven’t found? Clearly, I’d want to keep this massive keyword list personal, so the software to pull from upcoming would narrow down the result set locally. I’m still a bit too privacy obsessed to consider putting such a massive amount of information into a single vendors hands. :)

There’s been a bit of talk on the Pylons devel list regarding the recommended way to use SQLAlchemy with Pylons mainly regarding how to use SA (SQLAlchemy) in a fashion that is well documented and easy to work with (and maintain!).

Prior to Pylons 0.9.6 and SQLAlchemy 0.4 it was a bit of a mess, with the framework needing to load the config (since thats where your db settings are), then setup globals for SA…. eek. Mike Orr had a good intermediary solution for SA 0.3 called SAContext that handled many of the tricky parts. Unfortunately, this actually caused even more confusion as more ways of doing the same thing came about. SAContext solved some of the global config grabbing issues, but the additional layer of indirection made trouble shooting even harder (despite how small of a library it was).

Less is More

So the fix? Less intermediary layers, less indirection… essentially, KISS. Despite how much Pylons was attempting to help a user to get the db going, the additional layers in the end actually caused more problems then they solved. Of course, I shouldn’t have been too surprised…. Mike Bayer did warn me about many of these things at the beginning. Being overly eager to make things “easier” for new users, I ignored him. :)

This is why Pylons does not recommend Elixir, and with SA 0.4 the recommended usage of SA is to use its plain mapper functionality should you need an ORM layer. Yes, that’s right, despite almost every web framework out there pushing its ORM on you (or someone else’s ORM), there are many times when an app doesn’t even need a full-blown ORM.

Declarative vs Basic SA

For a better look at why one might consider additional layers on SA a bad thing lets compare a fairly basic table setup consisting of users and groups. Each user can be in multiple groups, and lets use proper referential integrity to ensure that groups aren’t deleted when users are still in them.

Compare the following two ways of setting up a basic many to many relation and the tables:

SQLAlchemy 0.4

from sqlalchemy import Column, ForeignKey, MetaData, Table, types
from sqlalchemy.orm import mapper, relation

metadata = MetaData()

person_table = Table('person', metadata,
    Column('id', types.Integer, primary_key=True),
    Column('name', types.String, nullable=False),
    Column('age', types.String)
)

group_table = Table('group', metadata,
    Column('id', types.Integer, primary_key=True),
    Column('name', types.String, nullable=False)
)

persongroups_table = Table('person_groups', metadata,
    Column('person_id', types.Integer, ForeignKey('person.id', ondelete='CASCADE'), primary_key=True),
    Column('group_id', types.Integer, ForeignKey('group.id', ondelete='RESTRICT'), primary_key=True),
)

class Person(object):
    pass

class Group(object):
    pass

mapper(Person, person_table, properties=dict(
       groups=relation(Group, backref='people', lazy=False)
))
mapper(Group, group_table)

Elixir

from elixir import *
class Person(Entity):
    has_field('name', String)
    has_and_belongs_to_many('groups', of_kind='Group')

class Group(Entity):
    has_field('name', String)
    has_and_belongs_to_many('people', of_kind='Person')

On first glance, its pretty obvious that everyone should love Elixir vs the obviously more tedious SA approach of layout out your tables, then mapping them to the class objects. However, look at these two examples, and try to quickly answer the following questions:

• How do you add a column to the many to many table to store an additional bit of info for the join?
• Do they both enforce referential integrity?
• How do you control whether SA is eager loading the relation? Can you restrict it to just one column of the relation?
• What are the table names used?
• How many tables are in your database?
• Where do you change the id column name?
• Which one is closer to the Zen of Python?

I think the explicit setup makes many of these questions easier to answer just at a glance. Those with enough Elixir experience can fairly easily answer most of these questions, but consider what that implies. Not only do you need to know SQLAlchemy options and parameters, but you need to know Elixir options and how they map to the SQLAlchemy functions. The desire to reduce the up-front setup of the ORM actually increases the amount of knowledge a user has to have in order to use it, and the most worrisome aspect… how to troubleshoot it.

Setups that Grow with You

With Pylons, a goal has been to provide out of the box recommendations that grow with you. That is, using the set of recommended tools may not be as apparently “easy” as some other frameworks. However, the pay-off is that you don’t hit a wall in 2 months when your application inevitably gets a little more advanced and needs to do something the simple tools either can’t do at all, or it’s incredibly difficult to do even slightly complicated things (eager load 2 columns off a related table, but not all of them). This way, the toolset you learned, you can keep using as you get more advanced and you don’t “outgrow” your tools.

While Elixir definitely appears to be easier at first glance, when you need to get more complicated you can’t exactly turn to the SA docs since Elixir has put a layer between you and SA. This can be very crippling when you eventually hit a wall, and so much ‘magic’ is wrapped up in the declarative layer that you have to troubleshoot additional layers of code when something goes wrong. The result of this is that to effectively use Elixir in complicated setups with SQLAlchemy, you need to really really know both of them which actually requires more work for a user than plain SQLAlchemy.

The SA example clearly requires a little more up-front setup, however, are you really adding tables to your database every day? How often are you going to be actually touching the table and mapper code, vs just adding domain model methods to your Person/Group class? Did the layer make it easier or harder to use multiple databases and/or put more between you and advanced SA functionality you might need later?

Adam Gomaa pointed out some interesting issues with Django’s ORM and Elixir but unfortunately tries to do the same thing Elixir and TurboEntity do…. add more layers. More layers more indirection more to wade through when you need to do something that should be pretty basic with SQLAlchemy (and is probably nicely documented on the SA site, which won’t help with these layers until you dig through the layer to find the basic SA objects the SA site refers to…).

What really makes a lot of this even more trouble-some with SA, is that when setting up complex relationships, the order of declaring table objects becomes important, since relations need to refer to them and the ORM classes. This usually results in some interesting metaclass hackery when you have these Entity’s in multiple modules, importing each other, and doing other fairly common stuff.

SQLAlchemy 0.4

In the end, I’ve been using plain SQLAlchemy 0.4 (at beta5 now, but quite stable) a lot lately, and its really great. Yes, setting up the tables (generally a one-time thing) took me probably 15 mins longer than it would’ve with Elixir. But I’m fairly certain I’ve saved myself significantly more time in the long run since I won’t need to worry about diving into Elixir code to try and find SA objects when I need a complex query, or trying to figure out how to hack Elixir’s connection should I need multiple db connections at once, etc.

So please, new users to SQLAlchemy, use just SQLAlchemy. It definitely seems daunting at first, but the flexibility and comprehensive documentation give you a solution that scales to meet your needs, with no walls in sight.

On a side-note, its interesting to compare my position on this issue to the Django team lack of AJAX helpers. The Django team rightfully claims that Javascript isn’t that hard, so “get over it” and learn a nice Javascript library so you can do powerful things. Also note that by including AJAX helpers, Pylons is encouraging one part that doesn’t scale… as the AJAX helpers will have you hitting a wall sooner or later.

I’ve been seeing more and more movies I wouldn’t mind actually buying… except that I have a 56” HD TV set. It looks amazing, pretty good with DVD’s, but really amazing with HD so if I’m going to buy a movie I sure as heck want it in HD.

Unfortunately thanks to the tech companies refusing to come up with a single standard, I can either buy a $200 HD-DVD add-on for my Xbox 360, or a PlayStation 3 which I’ve heard is the cheapest Blu-Ray DVD player available. And of course, I really need to buy both if I want HD movies since some studios are only putting their movies out on one or the other format.

The end result? I’m not buying HD movies for HD-DVD or Blu-ray because its pathetic to have to have 2 players around (no, I don’t care about universal players). In addition, I’m not buying DVD’s anymore since I know that in a year or two I’ll want to re-buy it in whatever HD format wins (please let one of them win in a year or two!!!).

What are other people doing in this situation? Buying one or the other? Still buying DVD’s? Or not buying any movies at all until they get it straight?

I remember reading a sci-fi book (David Brin’s Earth) awhile back set in the near future where a man-made black-hole fell into the center of the Earth. To say the least, it created a bit of a problem for our fictional heroes.

One of the events (I do hope I’m referring to the right book) that really stood out to me was when one of the main characters was roving around in a city, and a gang of thugs didn’t attack him/her purely because there were old retired people around hooked up with video gear that sent live feedback to the police station. These retired semi-vigilantes roamed around looking for some sort of crime to film, so they could send it in live to the police station. It was an interesting cross between the big-brother scenarios that are quite popular and vigilantism.

Ah, how fun it is as sci-fi I read years ago marches into the real-world. Earlier tonight I read a Cnet article on New York Police using cell phone photographs to help fight crime. I’m sure this will be extended to live video when cell phones and networks here can support it. I’ve found I actually prefer these kind of “big-brother” vs the type that seems to be popular in London; cameras on every street corner.

The problem with cameras all over, is that without some sort of AI, or army of people to watch them, you can only deal with logging the event and responding after the fact. It’s rare that someone watching hundreds of cameras will see some sort of crime in the act, compared to harnessing the ability of individuals to intelligently send relevant (or possibly relevant) video back to police for review.

The only thing in the article that confuses me is why it would cost the NY Police 40k to receive cell phone photos… surely there’s cheaper software that would work?

I was talking with my grandfather recently about various technology bits when he mentioned this awful trend he had started to notice on websites. Just the mention of an awful trend with the knowledge that his eyes weren’t what they used to be elicited similar anger from me, mainly, that there seems to be a significant increase in low contrast websites.

Disclaimer: I realize that this blog also commits some of these sins I’m about to rant on, and I’m in the process of changing them.

This trend is very apparent in many blogs, and some are getting so bad I want to scratch my eyes out after reading them. This isn’t the blog author’s fault, as they’re usually selecting a theme that “looks good” on a glance. Trying to read the text when the theme is in use can be a nightmare.

Consider this blog entry, it looks decent contrast-wise to me on first glance. But trying to read the text of the comments at the bottom made my eyes burn. Or the Shopify page which manages to consistently use the same color text as the background behind it. This is a pretty strict Do Not per US Section 508 Code (Web Accessibility Standards). I’m also fairly certain the W3C Accessibility standards frown on low-contrast layouts, especially if no option is given to switch to a high-contrast scheme.

This issue doesn’t just affect old people or those with handicaps. I have no color blindness or issues sorting out colors yet these themes still routinely give me headaches. The thing to keep in mind when using different different shades of the same color on top of each other is that while you might have a wonderful color-calibrated monitor, the vast majority of Internet users do not. Even those with perfect vision are typically at the mercy of the quality of their CRT/TFT and the color settings that may or may not be ‘proper’.

For the designers out there, I realize that these low contrast themes can look pretty, but please put out more high-contrast themes that look great. Now to fix-up my blog…