furia furialog · New Particles · The War Against Silence · Aedliga (songs) · photography · code · other things     ↑vF
18 September 2007 to 5 March 2006 · tagged essay
If you're going to waste your time doing obsessive analysis of data on which nobody's life or ecology depends, you ought to at least do it diligently and efficiently.  

About a month ago The Deciblog published Justin Foley's attempt to answer the timeless question "How likely is a metal band to start their name with a particular letter of the alphabet?". For his sample set, Foley took the combined rosters of several metal labels (he doesn't reveal either list), which gave him 814 names, for which he then calculated the first-letter distributions, reaching the startling conclusion that the most likely letter is S.  

Foley put his results in a bar-chart, which I assume means he used a spreadsheet, so hopefully he didn't spend a whole lot of time hand-counting. But he should have spent even less. The Encyclopaedia Metallum is not only just sitting there with a collaboratively-amassed and collectively-moderated database of 50,000+ metal bands, but they've even already split it up by first-letter and there are band-counts right at the top of each letter-page. Add, divide, and you're done.  

Here, then, is the much better-informed version of this still-pointless breakdown:

? % *
# 0.3%
A 9.1% *********
B 5.9% *****
C 6.3% ******
D 8.9% ********
E 4.9% ****
F 3.6% ***
G 3.0% ***
H 3.9% ***
I 3.7% ***
J 0.6%
K 2.2% **
L 3.1% ***
M 7.4% *******
N 4.2% ****
O 2.3% **
P 4.0% ***
Q 0.2%
R 3.3% ***
S 10.8% **********
T 4.5% ****
U 1.3% *
V 2.7% **
W 2.7% **
X 0.3%
Y 0.2%
Z 0.7%

Most of Foley's numbers aren't that far off. His small sample-size leads him to overestimate J and Y, and underestimate Q and V, but the absolute numbers for these letters are small anyway. He also seems to underestimate A, P and R, for reasons which are not apparent in his opaque reporting, but might have to do with language tendences, as EM's list is probably more global than his.  

But the biggest discrepancy in Foley's numbers, by far, is T, which he credits with 9.3% of the band-names, where EM data indicates less than half that. Here I have a wearyingly mundane but highly plausible theory: Foley has accidentally counted all the bands whose names begin with "The " as T, despite specifically saying that he didn't. This is obviously both philosophically and methodologically repugnant, and although I regret the maelstrom of blogospheric outrage that will undoubtably accompany my public exposure of this error, I think we owe ourselves (the) Truth.  

Of course, the thousands of metal fans around the world who have put time and effort into building the Encyclopaedia Metallum did it because they care about the music, not the alphabet. The site is the definitive central reference source for most metal-related matters, and certainly the final arbiter of obscurity for the vast unknown majority of the bands it lists.  

It also has, in addition to its factual content, tens of thousands of percentage-scored, user-attributed, peer-moderated reviews of metal recordings. What it does not have is any sort of similarity analysis to make use of the huge data-graph represented by the connections between bands, users and ratings. There is a wearyingly mundane reason for this, too: trying to do similarity analysis with SQL queries will make you want to eat your own neck.  

So here is an actual contribution to the world's knowledge on this admittedly peripheral subject: empath, the missing similarity analysis of EM user/review/band data, accurate as of yesterday. Pick a band, see the other bands that people who like the first band also like. Data wants to form shapes. In a better world, this would be just as easy for EM to do themselves, updating live, as it is for them to serve their raw data into web pages.  

Easier, actually. I bet it took me less time to do this analysis than it took Foley to make a bar-chart of first letters. But I have better tools. I have better tools because at the moment I'm paid to design better tools. If I do my job well enough, eventually you'll have my better tools, too. I'm not designing them to tabulate heavy metal, I'm designing them to answer questions. Not all answers turn out to be shaped like Truths, of course. But if you can't answer them, you can't be sure which are which.
Two of the very bad features of the SQL-based conception of relational data-modeling, I think, are these:  

- one-to-many relationships are significantly harder to handle than one-to-one relationships (and, inversely, many-to-one relationships easiest to handle by defactoring)  

- absence is significantly harder to handle than presence  

It's easy to add an Artist column to an Album table, for example, so we get:  

Black OneSunn O)))

But if you want to model the fact that collaboration albums can have multiple artists, you need an Albums table, an Artists table, a join table, and a bunch of IDs:  

Black One1

Sunn O)))1


This is already a pain in the ass, and thus you get a world filled with cop-outs like this:  

Black OneSunn O)))
AltarSunn O))) / Boris

and this:  

AlbumArtist 1Artist 2Artist 3
Black OneSunn O)))FALSEFALSE
AltarSunn O)))BorisFALSE

both of which are tolerable enough to just look at, but awful when you go to try to get the computer to answer even perfectly sensible questions like what albums Boris has done.  

And then, when you want to add a little more detail, you really ought to do this:  

Black One1

Sunn O)))1


but instead you'll probably try to get away with:  

Black OneSunn O)))CD/LP/D
AltarSunn O))) & BorisCD/LP


AlbumArtist 1Artist 2Artist 3CDLPD

because if you have the latter, you can do:  

SELECT Album, Artist

FROM Albums

which is easy to understand, although it won't work because you forgot you don't have a field called plain "Artist" anymore. Whereas with the five-table form you have to do something like:  

SELECT Albums.Album, Artists.Artist

FROM Albums, Artists, Authorship
WHERE Albums.ID=Authorship.AlbumID
AND Artists.ID=Authorship.ArtistID
FROM Formats, Formatship
WHERE Albums.ID=Formatship.AlbumID
AND Formats.ID=Formatship.FormatID
AND Formats.Format='Download'

and that's probably still wrong.  

I submit that this is all a colossal mess, and it's a wonder the database-driven software we get driven out of it isn't even more woeful than it is. A better paradigm would embrace the opposites of these two flaws:  

- the modeling of a relationship should be the same whether the relationship is one-to-one, one-to-many, many-to-one or many-to-many, and should be the same no matter how many "many"s you have  

- absence and presence should be equally easy to assess  

And I think a useful rough metric for such a new system is that it allows you to model your data without ever needing to say FALSE.  

And I think this, I think, because I think this is how we think and talk about relationships between things when nobody is getting in our way:  

Black One was recorded by Sunn O))), and was released in CD, LP and Download formats. Altar was recorded by Sunn O))) and Boris, and was released in CD and LP formats. All of these albums were released on CD and LP. Altar is the only one that wasn't released in Download format.  

Take these out of sentence form and make the data-types explicit and everything is still perfectly straightforward:  

Album: [Black One]

Artists: [Sunn O)))]
Formats: [CD] [LP] [Download]  

Album: [Altar]
Artists: [Sunn O)))] [Boris]
Formats: [CD] [LP]

or, looking at the same data from a different perspective:  

Format: [CD]

Albums: [Black One] [Altar]  

Format: [LP]
Albums: [Black One] [Altar]  

Format: [Download]
Albums: [Black One]

and then the old questions are also easier:  

FIND Formats WHOSE Albums INCLUDE 'Black One' and 'Altar'

FIND Albums WHOSE Formats DON'T INCLUDE 'Download'

Subliminating the one/many distinction allows us to talk about albums with 3 or 8 or 0 artists just as readily as we talk about albums with 1 or 2. Fixing the query language to handle absence lets us cope with the introduction of a new edition into our dataset, say, without having to add a new field for it and/or go through asserting that everything we already knew about doesn't have that format. I suspect pretty much every database field headed X and filled with TRUEs and FALSEs could at least be more usefully modeled in my new world as a relationship called "Characteristics" that either does or doesn't include X. And usually, as with formats, there's actually a meaningful label that contains information itself. The point is that things have relationships to their characteristics, or their formats. They don't, except in a very existential sense, have a relationship to falseness. FALSE is a machine thing.  

The relationally astute will note that the five-table version is the fully-normalized representation that would support an SQL-based implementation of my new form as a display abstraction, and presumably my pseudo-query-code could be preprocessed into the messy SQL so I'd never see it. But that's my point: SQL is too low an abstraction, which makes it too hard to do things naturally, which makes it too hard to pass along natural behavior to the victims of the system. I'm mainly arguing for a better abstraction. (Although I also expect it's still always going to be bad that there's no better underlying way to represent lists, so I'm also probably arguing for the better abstraction to not merely be layered on top of the bad one.)  

The discographically astute will note that Altar is available on iTunes, and that I have vastly undermodeled the rich and complicated space of Sunn O))) release formats. But if I'd done the examples in real detail, I'd still be typing out the bad versions...
I've come to the unexpected and disconcerting realization that my period of greatest apparent productivity in user-interface production was, pretty clearly, when I was mainly working in Visual Basic, circa 1994-1997. Ten years later, I'm actually spending much more of my design time repeatedly re-implementing basic mechanics of no inherent interest, and my design work at any given point in time is littered with more unsightly temporary hacks of things I haven't had time to re-implement this time yet. It takes me longer to produce first approximations, and longer to add some of the most common levels of noticeable refinement.  

This would be deeply tragic in several dimensions if it didn't deserve enough qualifying asterisks to make an ASCII-art smiley the size of Nebraska. The genius of Visual Basic was that it allowed you to very quickly produce 90% of a UI that was as good as 90% of everything else. Getting from 90% to 100% of a UI that was as good as 90% of everything else, however, took a large amount of extremely annoying work, some of which had to be constantly rechecked and redone every time you made the tiniest change to anything thereafter. And worse, of course, the standard established by 90% of everything else was not, in objective human terms, actually very good.  

Visual Basic's success, such as it was, relied on two very large simplifications of the underlying human problems. First, technically, VB was both a development and a deployment environment. VB applications were actually configuration data for the VB runtime. There were no platform-matrix issues, because VB itself was the platform. The specific version of VB with which you built the application, even. Unless you started introducing your own external dependencies, you could basically count on the things running the same way for your users as they ran for you.  

Much more significantly, though, VB instantiated a fixed vocabulary and grammar of UI interaction. This made it extremely simple to produce static forms and dialog boxes using standard Windows UI widgets, fairly easy to add a few well-defined kinds of dynamism to them, cumbersomely possible to do a little better, and beyond that you ended up fighting it more than it helped, and so usually didn't bother.  

This is what all well-conceived software projects and all self-aware computer programmers do, of course: we take environments in which a great many things are complicatedly possible, and transform them into environments in which a small subset of those things are significantly easier, and a larger number of things that used to be hard are now so much harder than the easier things that nobody wastes time trying them anymore. We do this by, mainly, an obsessive attention to abstraction and instantiation and packaging and layering. This is why the details of most programming successes are staggeringly boring to non-programmers in rough proportion to their genuine significance: the best programming work usually is not done in directly solving an individual human problem, but in making some tool, ten steps away from the visible problem, that makes it incrementally easier to solve some whole class of abstruse subproblems that factor in some small way into the one you care about. The web page that lets you buy embarrassing personal ointments without having to hand them to a 4'5" woman named Ruthie, who reads every package label aloud in megaphone tones, solves a human problem for you. The good programming work behind that page has nothing to do with ointment or Ruthie, and lots to do with content-management-system staging strategies, database transaction consistency, protocol standards, application-framework meta-programming, and twenty other topics so obscure and tedious that you'd probably rather talk about why you need the ointment.  

The problem with Visual Basic, circa 1994, was that the things it made easy were not important enough. Using it, I was able to produce a lot of UI very quickly, the Windows UI paradigm in which VB thrived was itself the cause of most of that work. I spent most of my time laboriously refining mildly optimized sets of widgetry in order to get the computer to show me something that I already knew. It'd be my guess that seeing and/or editing simple pieces of known data accounts for about 94% of the code ever written for computers, and that seeing and editing more-complex clumps of known data accounts for another 6%. All the stuff that does something with that data, and for which a computer is more than a very expensive and quickly obsolescent filing cabinet, at this level of precision accounts for approximately 0%.  

The web, as the new default environment for data applications, is both progress and regress. The literal display of data is generally a little easier than it was in most pre-HTML environments. Simple data editing is about as easy as it ever was, but orchestration of the editing of related bits of data is harder. Managing large sets of data, and managing the social dynamics of shared data environments, are probably slightly easier than before in relative terms, but since the connected world makes them less obviously intractable, they now represent a larger fraction of an increased overall quantity of active difficulty. And if the problem is taking the things you know, and the things I know, and getting to something new we now know together, a disappointing amount of what we've built turns out to be discussion forums where people we've never met can file unverifiable reports on whether the ointment worked for them.  

But the web, as it is now or even as the SPARQL/RDF version of the Semantic Web might make it, is yet another environment that generates a huge amount of the work it requires. This is ultimately the metric by which I think any tool, including software tools, and including meta-tool-sets as big as the web, should be most diligently measured: does it increase or decrease the human-scale meaningfulness of the human work done with it? In the same way that "the economy" should be graded on net human welfare, not gross currency movement, software should be graded on how it allows us to spend our lives, not how industriously it allows us to do new work nobody had to do at all before. It's amazing that it's possible to put a shame-free ointment store online, but absurd how much harder it is to do that than to put a tube of ointment on a metal shelf. And profoundly pathetic that the magnitude of our shame makes the enterprise seem reasonable.  

The software-design work I'm doing now won't, directly, even get rid of your rash, let alone reverse global carbon-dioxide trends. Gauging the value of your daily work based on its capacity to increase the welfare of humanity is practically difficult, and theoretically presumptuous if not absurd. But I think it matters whether it matters, and I really do believe that I am working on a part of the solution. If human existence is the stake in a race between shared insight and thermodynamic chaos, chaos has all the big-money sponsors, but shared insight is a power law. I'm trying to build software that does, for networks of highly interconnected shared data, what Excel does for columns of numbers (or, maybe more realistically at first, what Visicalc originally did for columns of numbers). I'm trying to help bring about a new base level of operative abstraction in which data is visible, and its connections traversable, by its very nature, so nobody has to write code to see what we already know. I believe that it should be possible for computers, instead of diligently enforcing dehumanized bureaucracy, to help sustain the transformative illusion that all it takes for you, as a human, to participate in the sharing of human knowledge is one thing nobody else imagined before, or one link between things that nobody else has noticed yet.  

And although global warming is not entirely a data problem, even that might arguably be mostly a data problem. Its key vectors all originate in what are fundamentally data blockages: inadequate education, incomplete understanding, incomprehensible macro-effects of unassessable micro-decisions, misapplications of science or technology, misguided optimizations of local variables at the expense of the overall system, failures of empathy, failures of recognition, and perhaps above all else the way in which untreated ignorance makes it possible to feel safely (and/or alienatingly) isolated from things to which you are really deeply and inextricably connected.  

I'm trying to help make the internet into the place where we know what we know. I'm trying to help make a new place where we go to do human things of which we are collectively proud, not somewhere where we hide among the widgets while we pray that this is the ointment that will make us feel better.
The logistical irony of substantive blogging, meaning something closer to essay-writing than link-jockeying, is that it enthusiastically both rewards and consumes time. For quantitatively ideal productivity you need to minimize the amount of time you spend doing anything else in your life, which obviously includes anything orthogonal to blogging like shopping for diaper-containment systems or eating brunch with your friends (unless, obviously, you are running a diaper-containment- or brunch-blog), but also includes whatever it takes to actually generate the material to write about whatever it is you're writing about.  

I only ever came up with two good solutions to this, where by "good" I mean effective in producing output, not necessarily in improving my own life any.  

One was writing about music during a period when most of my spare time was spent listening to and thinking about music anyway. It's a trivial psychological insight to observe that the magnitude of this preoccupation betrays it as an attempt to distract attention, mostly mine, from the absences of other things in my life.  

The other solution was writing about technology, or around technology in the same sense that I wrote around music, during a period when my day-job had devolved into pretty much exclusively sitting around thinking about and around technology. This is a pretty interesting experience to get to have, and as I repeated countless times in the job interviews that followed its inevitable eventual termination, I was never even remotely as well informed about the state of technology when I was actually contributing to it.  

I'm contributing to it again. My new job, still settling past its newness after only a few months, is by far the best job I've ever had. There's one formulation of job ideal where you say "I'd be doing this even if they weren't paying me", and this one is beyond that to "I'd be sitting at home frustrated at being unable to participate without a team around me."  

Several things are different and cooler about this job, at least in my own experience, the top few being that I'm explicitly managing it in addition to leading by design, that we're trying to find the shape of the solution we're building (and of the problem we're defining to solve) instead of being handed a VC precis or business plan to fulfill, and that the development work is not primarily segmented by function.  

The combination of these things is very challenging and extremely engrossing. I discovered a quantum jump in responsibility when I went from being a member of a large design team at Interchange to being the sole designer at eRoom, and it's a similar one from being the designer to being the designer and manager, and maybe another level again to be the person with the most navigational accountability for an exploration undertaken knowingly without a map. The internal pressures intensify radically in both directions: to dismiss inanely crazy ideas to avoid causing magnified distractions, and to take superficially inexplicable intuitions even more seriously for their transformational potential. If you don't know where you're going, exactly, you have to operate as if patience and diligence are doomed.  

Somehow this has resulted, over the past few weeks, in my spending the vast majority of my working hours actually programming. I won't reveal much by saying that we're building a big software system that will, when it's working, take in a large amount of previously scattered and unanalyzed data, and attempt to reduce its disorder. Most of the team has been working, since before my arrival, on the data-acquisition aspects of the project. So without anybody else to assign to the languishing data-analysis portion of the system, I assigned it to myself, and have been maniacally trying to figure out what I mean it to do by writing it.  

I am not, in case I haven't been clear about this, really a professional programmer. Over the course of my career I've written a lot of code in a lot of languages, but almost always in carefully circumscribed senses. The UI architecture of eRoom was devised in no small part with my individual capabilities in mind, so that I could write a large amount of the actual UI code without having to participate in non-goal-oriented conversations about lazy pointer deallocation or rogue mutexes. And even the code I'm writing now is, probably, only a prototype, and I've written lots of prototype code.  

My previous prototyping, though, has always been about faking a system in order to play with how it looks and acts. The easiest way to fake a large system is usually to actually build parts of it more or less for real, of course, but every project I seem to fake a little less. The content-management and discussion-forum systems underlying this website are constrained and minimal, but not faked at all. And although the prototype I'm writing at work will probably be migrated out of my eccentric Ruby improvisation into somebody's tediously meticulous Java for dull scalability reasons, my Ruby version really does do what it says it does.  

This is tremendous fun, but it also pushes me into a part of my brain where my most obsessive nature is most dramatically exacerbated. I sit down at my desk in the morning, and eight or nine hours slide by in an un-self-interrupted continuum. I forget to eat lunch, I forget to check my email, I forget the five-minute phone-call-returning task I'm supposed to do while eating the lunch I forgot to eat, I look up at the clock and realize I was supposed to be home an hour ago, and when you live a block from your office there's no possible excuse. Making information fall magically into order is not the only thing I want to do with my life, but it's one of the ways I think I can make the world better, and in the grip of a particularly involved spell it feels like a calling. Something shadowily like a pattern starts to emerge, and I frown at it, and twitch something I've built to be twitched, and suddenly answers are fountaining out of the screen from their own joy at existing.  

And that's not even the interesting news.  

The interesting news is that Bethany is pregnant. If the tests and milestones keep going as well as they've been going so far, about six months from now we'll have a child. Software isn't easy, exactly, but it's difficult in easy ways. If you know what to twitch, you can know what kind of magic you'll be able to make.  

People are, obviously, far less tractable. I don't know what kind of child we're going to have. I don't know what kind of parents we're going to be. I don't know what's going to be ecstatic and what's going hurt and what's going to be unbearable. I know nothing will ever be this simple again, and I was already barely coping with simplicity's complexity. Beth and I could probably have spent another ten or twenty years just figuring out how to merge two lives most inspiringly, and now we're going to try to make sense of three. My own explanation of wanting to do this is that raising children is the biggest project available to individual humans, and Bethany is the person with whom I feel most capable of the most humanity. I want us to take on the greatest, hardest, most human challenge for which we are eligible.  

As crazy intuitions go, this makes my daftest futurist semantic-web predictions feel like placing wrist-flick roulette bets with free (and mostly unredeemable) chips. I really do believe that the semantic-web project I'm working on has the potential to be the seed of something that changes the dynamics of human interaction with information sufficiently to advance human nature, but I know that that potential is prorated by its likelihood. I have a small team, and there are many teams and bigger, and many factors and convolutions. Most attempts at deliberate innovation fail, and although opportunities for accidental innovation fail at a far greater rate, they arise at an even greater rate than they fail, so the innovations arrived at by luck vastly outnumber and outfascinate the few we make knowingly.  

We will have made this child knowingly, and although accidents will probably outnumber clevernesses even more lopsidedly thereafter, in this project we will not have the luxury or salvation of irrelevance. There are no swarming competitors feverishly plotting to steal away our parent-share or undermine B and my parenting brand. There are no breathless marketers trying to find the most flattering spin on whatever turn out to be our signature family dysfunctions. Nothing will release us from this. There's just us, and a new person nobody on Earth has met yet. Somebody has to care for everybody, and this one is ours.  

So I'm obviously not ready, and can't possibly be, but I'm ready. There are a thousand things to do, and stacks of books and friends and doctors and counselors lining up to add more things to our lists, and I can't imagine how we'll ever get them all done, so I assume and stipulate that we must not have to. I will do whatever it takes to be a good parent, and that will usually be almost enough, and hopefully every once in a while will jubilantly suffice. I will try to be better at this than anything else I have ever done. I don't intend for it to become the only thing I do, but I understand that, like marriage, it is a private commitment that takes precedence over my public aspirations, and that I will make different choices now, and dream of different things. I will have far more important things to write about than b-sides, and rarely even enough time to list them. I will do what I can, and allow myself not to catalogue, even mentally, all the things I would have catalogued when I had time instead of purpose.  

And if this is the most ordinary, and thus in a sense the most mundane, way of raising the stakes for the ways in which I hope to improve the world, and to avoid being even a passive vector of thoughtlessness or evil, then so be it. These swiveling electrical-outlet covers I need to install so a baby that doesn't even exist yet can't stick (I guess) a fork in them, and whatever diaper-containment system we ultimately settle on, and the semantic web and the way we love each other and ourselves, will now all also be part of the inescapably consequential context for a new life. None of this will be faked, or maybe all of it will be, or maybe this is where it stops being meaningful to talk about the difference between what you fake and what you know. This is where suddenly the craziest answers are fountaining out of nowhere into a flood of what it is up to us to make be joy.  

(Discussion on vF...)
Somewhere, somebody needs a software system for something. Ordering sandwiches, maybe. It doesn't really matter, because software doesn't manipulate sandwiches, it manipulates information. So some programmer abstracts out the information-problems involved, and gives the abstractions names, and writes a program that operates them.  

Which hums along nicely for a while. The program creates blorfs, juggles them with admirable dexterity, and then copes ingeniously with whether they are sterkled or not. Only the programmer knows what blorfs are, exactly, and what it means to be sterkled, but the program knows what to do with them, and that's what programming is about.  

Except one day the program gets to the point where it has to actually assess the sterkledness of a specific blorf that represents some actual thing out in the world with an actual person attached to it, and thus arises the need for a User Interface. The programmer winces briefly, but then gamely poses the question:  

Is your blorf sterkled?

The user types something, and the program promptly crashes. Most of the subsequent changes to the program will effectively amount to this:  


If you speak your own language slowly and loudly enough, even uncooperative foreigners are bound to pick up at least the gist of what you're saying.  

But let's pretend the programmer gets past being irritated at the user, and realizes that they might benefit from some help answering the question. Help=instructions:  

Is your blorf sterkled? [yes/no]

Mysteriously, this doesn't really improve things. At this point the programmer reluctantly digs out that book about UI Design, and over the course of a painful few days produces this series of incremental and undeniable widget improvements:  

Is your blorf sterkled?

Is your blorf sterkled?

Is your blorf sterkled?

Is your blorf sterkled? yes no

Is your blorf sterkled? yes no

Your blorf: Sterkled

Your blorf: Sterkled

Calls to action are better than blanks, defaults are usually better than forced choices, radio buttons require fewer clicks than drop-downs when the number of choices is small, and checkboxes are even simpler when the choice is really binary. With any of the last three versions, at least, the program will be much happier. It needs an answer, and the widgets are such that it's going to get one no matter what the user does.  

Passing a usability test is a little harder than not crashing, though. The usability problem, of course, is that the user doesn't have any idea what a blorf is, or what it would mean to be sterkled, so these widgets could spin and invert and coruscate without ever changing their comprehensibility. Design is not the application of widgets to programming variables. At this point the programmer tracks down a product manager, if one can be dragged away from fiddling with PowerPoint transitions, and after some painful cross-purposes conversation they realize that the program needs to translate, for UI purposes, back into the user's vocabulary. Blorfs are, it turns out, sandwiches, or some meta-food-product abstraction that is, in this case, instantiated by a sandwich. Sterkle is a class of food-instantiation modifications, which in this case is the application (or non-application) of mayonnaise. Thus the original question is actually:  

Do you want mayonnaise on your sandwich?

Or, in our hard-won UI simplicity:  

Additions to your sandwich: Mayonnaise

This revision passes the usability tests with scores .724 higher than any UI the lab has ever evaluated, and the application is triumphantly deployed.  

By halfway through the first production lunch hour, the customer has threatened to insert the product manager's head into the panini press. The users (the customer's customers) like the software fine, but they hate their sandwiches. After a series of crowded and acrimonious meetings it is finally deduced that having mayonnaise checked by default has resulted in it being spread liberally on nearly every sandwich, because users are in a hurry and have just been clicking OK. So what the original question really amounts to, a lot of the time, is something much more like:  

Do you want mayonnaise on your peanut-butter-and-jelly sandwich?

All this time and work for a question that is almost always going to be answered one way. So we finally get this:  

Additions to your peanut butter and jelly sandwich: Mayonnaise

And, after the addition of a fancy trend-analysis module of which the programmer is justifiably proud:  

Additions to your peanut butter and jelly sandwich: Mayonnaise (not recommended!)

Sadly, although the trend analysis is fascinating for the customer, the customer's customers are still pretty unhappy. If you like mayonnaise on your ham sandwich, it's not really interesting or helpful that the system thinks most people don't. The programmer wastes a month building an elaborate personalization system to keep track of exactly this sort of preference, but vexingly, nobody takes the time to create sandwich-option-preference profiles for themselves.  

Another month of arguing and thought produces the clever idea of automatically generating preference profiles for people based on their previous orders, which results in a lot more profiles, but the unfortunate feedback effect that screwing something up in your first order applies it to your second as a default, and the number of disgusting sandwiches is not going down fast enough. The customer, who spent years efficiently making non-disgusting sandwiches with no more technology than an order pad and a pen, is about ready to join the Luddites, even if they do want mayonnaise on their PB&J.  

If the programmer and the product manager are really lucky, one of them notices, before they lose the deal, two things:  

1. The customer knows a lot about making and selling sandwiches.  

2. The system is really intended to facilitate a conversation between the customer and the customer's customers, not the end users and the program abstractions.  

The program's task is thus, more properly, to help the customer represent their sandwich expertise in some way that allows the program to give the customer's customers a sandwich-ordering-and-eating experience that is, hopefully, more accurate and efficient than the no-tech version it is supplanting. Some knowledge analysis and sweeping re-architecture later, we get something like this:  

& on

Notice that this isn't exactly the answer to "Is your blorf sterkled?"  

What Rudy (the sandwich guy's name is Rudy) knows, among other things, is that PB&J isn't a single sandwich, or a single combination of sandwich options, but a class of sandwiches involving a nut-related spread and a fruit-related spread. He has a version he likes to make, and some alternatives he knows people like. He'll make you a peanut-butter-and-mayonnaise sandwich on a spinach tortilla if you really want, but for that, for now, you have to come in and order at the counter with the rest of the freaks. The product manager keeps trying to sell him a drag-and-drop WYSIWYEat custom-sandwich-building module, but Rudy figures that if people wanted to build their own custom sandwiches, they'd build 'em in their own kitchens at home and bring 'em to work in lunch-boxes, and so wouldn't be ordering sandwiches from him. This online system isn't for people who aren't ever going to be his customers, and it isn't supposed to replace his counter. It's supposed to help his regular customers get their sandwiches without waiting in line as long, and to encourage his occasional customers to order more frequently, and to encourage his potential customers to click and try.  

The ordering system is also a menu, and it's supposed to make the sandwiches sound good. The drop-down alternatives, in fact, help demonstrate Rudy's sandwich knowledge, even to people who don't pick them. This interactive version of his menu (with similar treatment for all his other signature sandwiches) actually contains a lot more information than the one printed on the big board behind him, and thus has the potential to do a better job than the physical one of conveying not only his versatility but his expertise.  

It's also supposed to help him cope with the lunch-time crush with a little more anticipation and efficiency. His workspace out front is pretty small, and his kitchen in back is large, but if people are standing in front of you when they order, they like to see their sandwiches being made right there. Online orders not only can be turned around in 10 minutes instead of 1 minute, but they can be made back in the kitchen by Rudy's assistants, getting those out of his own way in both time and space.  

So, at the end of this long and arguably artificially obtuse case-study in design by myopic trial and error, we've come to the point where a life-improving software project should really begin: with the problem, and its aspects, and the context in which these aspects are aspects of a problem, and the characteristics that a solution in this context needs to have. Usually nobody (at least not the user, the customer, the programmer or the product manager) is going to know from the beginning exactly what the right solution should be, but everybody involved has some knowledge and some ideas and some dreams, and since those are ultimately both the cause and the goal of the project, they should be where the design process starts, and where it comes back to see how it's doing every time anybody wonders. Or, more often and even more critically, every time anybody notices that a little too much time has gone by without anybody wondering.
I spent this week at the annual conference of the American Association for Artificial Intelligence. I have no academic background in artificial intelligence. Of course, I have no academic background in computer science, or interface design, or technical support, or pretty much anything else I've ever been paid to do. The tenuous premise of my professional career is that I think well, in general, and maybe that I think particularly well in the intersection between the things that might improve human lives and the things that machines can do. My track record indicates that I've been involved with projects where this tension has been usefully resolved, and my day-to-day work seems to support my presumptuous contention that I am a personally significant contributor towards such resolutions.  

So I'm clearly unqualified to assess, in general, the internal quality of advanced research work in the field of artificial intelligence. But I'm in charge of a software project in which machine-learning algorithms are being put to practical use, so I have a stake in the field. And the general meta-question of the application of technology to the class of what, to humans, are thinking problems, is probably no farther outside of my domain than anything else I ever deal with. And even if I were deeper in the field, there were six parallel tracks of talks taking place during most of the conference, so no one person could really hope to directly evaluate the whole thing anyway.  

Take this, then, for whatever it evokes. My scattered impressions cluster around two ideas. The first, which covers about half of my experiences during the week, is that I had wandered into a convocation of alchemists. Time after time I sat through earnest descriptions of patently clever mechanisms for conveying the lead to the transmutation chamber, or of organizing the variations of gold sure to soon be produced, or of venting the toxic fumes that somebody had proved (in last year's paper) would result from the conversion. Not so much actual gold, just yet. This year's conference was celebrating the 50th anniversary of AI as a field, and although it would be wrong to dismiss a hard problem on the grounds that it hasn't been solved quickly, the only way Zeno got away with diminishing returns was by publishing measurable progress so frequently.  

One major branch of AI is based on the observations that humans think, and that we express thoughts, and that our expression of each seemingly individual thought involves a complex background of assumptions and definitions and relationships and conceptual leaps, so maybe if computers had all that context to work in, they'd be able to think, too. The logistical and emotional center of the cargo-cult effort to build that foundation of knowledge, and hope it attracts Thinkingness, is the Cyc project, whose director proudly told a room full of people that they have so far encoded 1,000,000 terms, 100,000 relationships between them, and 10,000 meta-assertions about those things and relationships, and that they believe this constitutes about 10% of what is required.  

Then he added "(isa Rudolph reindeer)" as 1,000,001 and 10,001, and with a flourish derived "Rudolph is a ruminant", which is so far from 10% correct that I'm still laughing about it a week later. I suspect the Atlantean Association of Alchemical Investigators thought they were 10% of the way to transmutation at some point, too. To me this is not only almost certainly wrong, but also fabulously self-ridiculing. There is no reasonable way to assess the size of something you can't even define, and you can't be 10% to nowhere any more than you can be 10% to a miracle. 10% to a miracle is not a scientific progress-report, it's a numerology of the angel-capacity of certain ornamental pins.  

And even if it made any sense to say that we're 10% of the way to encoding the basics of human common sense, to me the premise of the project is fatally flawed in three ways:  

1. Our ability to carry on conversations is less a product of the depths of information we plumb with every statement than of our ability to levitate on ignorance. You can always add a new level of notation to explain what the previous level means, but you can always add a new level, so all you've really succeeded in doing is turning the second M in MLM to Modeling and wasting the time of a lot of people who already knew what you meant in the first place. The machines might remember everything you can figure out how to tell them, but remembering isn't half as useful as forgetting. Not only is deduction less valuable than guessing, it's ultimately even less valuable than being productively wrong. And if after 50 years of AI and 20 years of Cyc we're only 10% of the way to encoding the things people know, then you and I will be long dead before we ever so much as get started on the exponentially larger domain of our idiocies and misconceptions and errors.  

2. Even if we could encode all of that, I'm in no way convinced that it's necessary for thinking, let alone sufficient. I sat through a lot of talks about what felt to me, ultimately, like attempts to figure out how to build a bicycle by taking apart a horse. Cyc is pretty much the embodiment of Intelligent Design as an engineering methodology, trying to build a grown human adult an atom at a time. I suspect they would defend themselves by saying that they're only really trying to build a 14-year-old, and then they can have it read the rest of the stuff in books. This is never going to work. Even building a baby and letting it learn the whole thing from scratch isn't going to work. Complexity evolves. If we're going to build artificial constructs that think, we're going to do it by sowing the most primitive of artificial creatures into the most accelerated of generative artificial worlds and giving them virtual epochs in which to solve (and state) their problems themselves.  

3. Underlying both of these errors, I think, is one fundamental misconception, engendered by the words we use to describe the point of all this speculation and work, and maybe most encouraged to fester by the inescapable apparent simplicity of one unfortunate thought-experiment. It's easy to imagine talking to a computer. The genius of Turing's Test is that it requires less equipment than an E-meter audit and less training than teaching SAT-prep classes. It's even easier to understand in the IM era than it was when he posed it. If a computer can convince us it's a person, then it is thinking. Translated to what I suspect most people would think this means, and then inverted for logic, this basically says that "people" are whatever we can't tell aren't like ourselves. But this is Fake People, not Artificial Intelligence, and shouldn't be what we mean by "thinking", or by artificial intelligence. My cats apply the Turing Test to me several times a day. I always fail. The first machine to think will be the first one that comes up with its own idea of what thinking means. It won't be like us. It won't think like us, it may not have much to say to us, and it almost certainly won't care. It will be good at things it likes to do, not the stuff we want just because we want it.  

And therein, I think, is something closer to what I think we ought to mean by artificial intelligence: not the manipulation of ideas, however facile or reified, but the generation of them. Here's the Furia Test: the first true AI is the first machine to learn to spit out Rudolph's cud and tell the Turing judges to take their incessant idiotic questions and fuck off.  

But I said that that was only half of my experience. Humans have come up with a lot of good ideas while working on bad ones, and arguably most human progress takes place in the context of grandly evocative error. I'm not really interested in building a machine I can condescend to, and I bet a lot of the other people at the conference weren't there as part of that dream, either. Taking apart a horse teaches you things. Building bicycles teaches you things. Even the effort to teach Cyc about ThingsThatMakeFartingNoisesWhenYouAccidentallySitOnThemFn teaches you something.  

My part of this comes from a much, much simpler dream. I already know people who think. I don't need machines for that. True AI will have its own questions to devise and then answer. What I need is deeper and broader answers to human questions we already know, and new questions that matter to us. I see that the information we already have, jammed into computers that don't think and aren't likely to start thinking any time soon, is sufficient to answer several orders of magnitude more questions than our current question-answering tools facilitate. This isn't an existential problem, and it's our problem because it's so clearly our fault. Computers would be perfectly happy to synthesize and connect, but instead we've obtusely put them to work obscuring information from each other so that even the syntheses inherent in the character of the information become intractable. The machine-learning parts of the system I'm working on are tangibly useful, but in a sense mostly regrettable necessities, devised to undo the structural damage done by too-exclusively packaging small amounts of knowledge for unhelpful individual resale.  

The good news from AAAI, for me, is that there are incredibly interesting things going on in the subfields of knowledge representation and reasoning. I actually do have some academic background in deductive logic, and the project I'm working on is both deeply and shallowly involved with information structure, so maybe my grounding in this subarea is just a little firmer. More than that, though, I can understand why it matters. Some of the new information-alchemy equipment is really new good information-chemistry gear opportunistically relabeled. I understand, for example, that we need graph-structures in most of the places where we have heretofore had only trees, and that our tools for comprehending graph structures are way behind our tools for understanding tree structures, and our tools for understanding tree structures without chopping them down first were already none too good. I understand what each order of logic adds, if that's not all you're doing, and what the effort to encode each new level costs and potentially loses. I understand that we want to make closed-world decisions in an open world, and that we have to. I understand why Tim Berners-Lee wants URIs on everything and I understand that a huge part of how we're able to talk to each other about anything is that we are not required to unambiguously identify every element of what we are attempting to say. I understand that Zeno was only ever half right, and that he never had to actually ship anything.  

So I go back to work, where we aren't trying to build machines that think, and aren't waiting for anybody else to. Our bicycles aren't going to look or smell much like horses, but we're going to try our best to make them faster than walking. I am bemused to have fallen into the company of alchemists, but even at their most insularly foolish they're a lot more interesting to talk to than Sarbanes-Oxley compliance officers. I'd much rather worry about what OWL leaves out than have to explain to business-unit managers why writing "XML BPM" on a PowerPoint slide doesn't constitute an advanced-technology plan. Ultimately my complaint about AI is probably not so much that its premises are flawed as that the current particular flaws lead to wrong work that is too mundane in its wrongness. I understand that I am paid to be an applied philosopher, and I want to feel, after a week among people freer to linger in theory, that practice is pedaling flat-out just to keep possibility in sight. But if we're just out here with our new fancy bicycle prototypes, and our old blank maps, and no plausible rumors of buried gold or miracles, then fine. There are places to ride, and better things to see by looking up than down, and we'll see more of them if we aren't always stopping to dig new collapsing holes in the same empty sand.
Although I'm not at all sure this is factually fair, I have begun to mentally, and maybe emotionally, blame Flickr for what feels to me like a plague of subject-oblivious square photo-cropping.  

I should admit, I guess, that when it's me operating the camera, I'm a pretty extreme horizontalist. I'm happiest at about 3:1. In a tool universe built around 4:3, though, this is kind of a pain in the ass. I could mask my camera's LCD for 3:1 feedback, but then the picture is really too small to work from. For online display I have to assume 4:3-ish frame spaces, so 3:1 images end up in practice being shorter instead of wider, which is unsatisfying. And digital cameras will have to pack a lot more pixels into 4:3 sensors before I'll be informationally content to throw away more than half of them. And my obsessive preference for aspect-ratio consistency in exhibition sets means that I would usually rather stick to 4:3 for everything than mix in the occasional 3:1 where I spot an opportunity despite the obstacles.  

So I understand, of course, the value of square-cropping in any content-neutral photo-showing application. It's possible to do an attractive job of mixing aspect-ratios, but it's exponentially easier to do an even more attractive job of displaying consistent aspect-ratios. Cropping 4:3s and 3:4s to 1:1 symmetrically is technically trivial, and although it's aesthetically unreliable in the abstract, the vast majority of amateur photographs are center-weighted, so it usually turns out OK. Actually, the vast majority of amateur photographs are also probably framed too widely, so a little universal symmetrical cropping almost certainly improves more Flickr pictures than it damages.  

So cropping all pictures to squares for thumbnailing makes perfect sense as a Flickr design decision. It simplifies away arguably the biggest visual design problem in mass photo display. If you're looking at somebody else's photographs, it's easy to fall into assuming they are square, so any weirdness in framing you're likely to implicitly attribute to the photographer. The same applies to your own photographs unless you've spent some time seriously considering the originals, and the more you use Flickr, the more it is the way you consider your own photographs en masse.  

But if your exhibited photographs are usually going to be approached through thumbnail galleries (the prevalence of which Flickr has also hugely influenced), and the thumbnails are usually going to be squares, it will simplify the rest of the experience if your photographs actually are square. I don't know if any digital cameras are already shipping with built-in square-cropping modes, but I expect those to start appearing very soon if they haven't already. The more square photos people have, the more display-tools will cheat and optimize for them, and the more incentive there will be to be square.  

But square is a bad base ratio for photography, at least if by "photography" we mean people taking pictures of things people see, for other people to later share (or imagine) the experience of seeing. We see our world horizontally. Our eyes are side-by-side, our lives are gravity-flattened, our emotional landscapes are literal landscapes as often as metaphorical. My 3:1 fetish may be extreme, but I'm pretty sure that if you take photographs on their own terms, humans instictively respond more positively to wide aspect-ratios. The standard terms are actually telling: "landscape" refers to the subject of the picture, "portrait" to the act of picturing it. We can appreciate photographs in all sorts of shapes, but we can empathize with seeing most readily when the shared vision is the shape of experienced vision.  

So this self-reinforcing dependent vogue for square photography is, I think, a machine gain and a human loss. Worse, it's a sparkly machine-gain that humans are lining up to lose. Machine gains are almost always sparkly, if only because it's far easier to polish a working machine than it is to figure out the machine you should have built instead, or admit that it was better, even if it was harder, to do something by hand. And we form machine-polishing clubs, and start companies to make machine polish, and open shops to sell it, and years go by before we stop and think about the flaws in which we've become invested.  

So too with this idiotic chronology-switchback setup we've tempoarily settled on for blog formats. The right way to read incremental written forms, beyond any vague doubt, is in serial order. You start at the top of the first entry, you read to the bottom, and then the top of the next entry should follow the bottom of the previous one. Thousands of years of usability research has validated this basic design.  

All of which was summarily and obliviously ignored by the original engineers of HTML and web browsers, with the result that they neglected to provide a simple and reliable mechanism for one absolutely essential bit of visual behavior: a fixed identity header and indepdently scrollable/pageable content. Without this, a designer of serial content can have identity reinforcement (come into the page at the top) or currency (come into it in the middle, where the new content starts), but not both. And since they didn't build in any meaningful tools for handling the user-subjectivity of "current", identity basically wins by forfeit.  

The reverse-chron blog format is a sparkly-machine solution to this problem. It puts the newest entry next to the identity, thus at least superficially addressing both goals at once. For every other purpose, though, it's actively reader-hostile. If the entries form any kind of overall narrative, you have to read it in a painful zig-zag. If you are following a blog and miss a single update, you have to use the same awful up-and-down to find where you left off, read down, scroll back up, read down, repeat. This is bad.  

But it's bad in what has become an established way, so even if you don't believe the alternatives are worse on their own terms, they almost certainly become worse in public practice. As with square photographs, we make our tools in the easiest shapes, and then we accomodate their limitations, and then we hone them to perfect their limits, and then we forget that this is not how we wanted to live.  

Next time you make a crude tool, don't polish it, and don't accomodate its limitations. Use it the way you wish it worked, pay attention to how that hurts, and then throw it away and try to make the next crude, unsparkly tool so that using it doesn't make the tool better, it makes us better.
Say you're making software to do X. The audience for this is people who want to do X, and the evaluation criteria for the software to do X is that it actually can be used to do X in some respectable and hopefully distinguished manner.  

Before you get to the released version of your software to do X, there are two useful stages you will probably want to pass through: beta and alpha. These are not percentages of completion or levels of toleration of error, they are themselves tools for different purposes than the released version. X beta exists to provide an evaluable preview of X. X alpha exists to provide an evaluable preview of X beta.  

It is not only possible, but critical, to substitute these purposes of beta and alpha back into the original statement. Like this, for beta:  

Say you're making software to {provide an evaluable preview of X}. The audience for this is people who want to {evaluate a preview of X}, and the evaluation criteria for the software to {provide an evaluable preview of X} is that it actually can be used to {evaluate a preview of X} in some respectable and hopefully distinguished manner.

Note that this is different. The people interested in previewing a tool to do X are not necessarily the same people as the ones who want to do X, and their goal is not doing X but seeing how your software is going to do X.  

Likewise with alpha:  

Say you're making software to {provide an evaluable preview of an evaluable preview of X}. The audience for this is people who want to {evaluate a preview of an evaluable preview of X}, and the evaluation criteria for the software to {provide an evaluable preview of an evaluable preview of X} is that it actually can be used to {evaluate a preview of an evaluable preview of X} in some respectable and hopefully distinguished manner.

That is, the beta is a tool for seeing how the real thing is eventually going to work, and the alpha is a tool for seeing how the beta is eventually going to work. The audience for alpha is thus double-meta, and is likely to be very small, sometimes even smaller than the team working on it.  

Even more crucially, the substitution works in the other direction, too: beta is a released version of software for evaluating a preview of X, and the alpha is a released version of software for evaluating a preview of the beta. So while the X beta may have omissions and errors in its performance of X without failing, if it has omissions and errors in its performance of the evaluation and preview of X, it fails.  

Thus the released version of an online calendar for shared scheduling, for example, fails if it can't be used for shared scheduling. But the beta fails if it can't be used for experimenting with shared scheduling. So the beta doesn't require that all the shared scheduling features work, but it does require that all the experimentation features are ready.  

If your software involves manipulating user data of any kind, then, here are some things that probably have to be already working, without appreciable error, in your beta:  

- import
- reimport
- importing useful sample data
- distinguishing data by import batch
- undoing import batches
- deleting data
- deleting data en masse
- full data purge and restart of beta experience
- comparing your treatment of this data to the treatment of it wherever it came from
- comparing different treatments of this data within your own software
- export/sync/feed of this data to any evaluation-critical downstream integration points
- sync/feed of source data from any evaluation-critical upstream integration points
- full operation of a representative subset of anything critical of which there will be multiples (integration points, environments, styles, templates, other users, etc.)
- any configuration settings integral to personal evaluation
- clear explanation of beta limitations  

If any of these aren't done, don't ship your beta yet. If some of them aren't even in your design, you've got more designing to do.
A conversation, from the point of view of any one participant, has these three elements:  

- the source: the other participants, either as individuals or as some collective they form for the purpose of this conversation  

- the context: as simple as an explicit topic, as subtle as an implicit one, as complex as a network of human relationships  

- your role: sometimes you are yourself, sometimes you are acting in a defined capacity, sometimes you participate in a conversation as part of a collective (like an audience)  

All conversations have these same fundamental elements. At the moment, our electronic conversations are subdivided and segregated according to ultimately trivial subclasses (personal email, IM, mailing lists, newsletters, feeds, web sites themselves, innumerable other variations on alerts), and the useful tools for managing conversations are scattered across similarly (arbitrarily) segmented applications.  

The conversation tool I really want will be built on the structural commonality of conversations, instead of the disparities. I want to apply email rules to RSS feeds, feed mechanics to mailing lists, contact lists as view filters, identities as task organizers, Growl formats as cell-phone notification styles. I want my email correspondence with my mom to be at least as rich as my soccer "correspondence" with the MLSnet front page, and I want my conversation with MLSnet to be as malleable as my own iTunes song status.  

I want, I think, for all my conversations to be structurally equal. I want to be able to look at them organized by any of those elements: by source, by context or by my role, and by the unions and intersections thereof. I want to see conversations in context when there's context, and I want there to more often be far more context than a list. I want to be able to understand my side of my conversations in aggregate, and for the memberships of my conversations to be effortlessly expandable, and for my computers to remember things my head forgets, even when I can't remember whether I knew them to begin with.  

I want, of course, the whole internet redesigned with this desire at its core, but I also want approximations of this in the meantime. I want Apple Mail or GMail to stop acting like somebody invented "mailing lists" and "address books" yesterday morning. Or I want Shrook or BlogBridge to speak POP3 and IMAP as fluently as RSS and OPML. Or I want Safari to look like Apple solving a problem from their own principles rather than letting somebody else define the form of the answer, or Flock aspiring to be a social application rather than just a browser with a few extra menu commands. I want Agenda and Magellan back now that I finally have enough information to justify them.  

I want us to have the conversations we could have if we didn't spend so much time just trying to keep track of what morbidly little we've already managed to say.
The main human information goal is understanding. Or wisdom, depending on your precise taxonomy. But either way, searching is plainly a means, not an end, and the current common incarnation of Search, which involves arbitrarily flattening a content space into a set of independent and logically equivalent Pages and then filtering them based on the presence or absence of words in their text, not only isn't an end, but is barely even a means to a means. This form of two-dimensional, context-stripping, schema-oblivous, answer-better-already-exist-somewhere searching is properly the very last resort, and it's a grotesque testament to the poverty of our information spaces that at the moment our last resort is often our only resort.  

The first big improvement in searching is giving it schema awareness. I doubt the people behind IMDb spend much time thinking about themselves as technology visionaries, but IMDb Search is a wildly instructive model of what is not only possible but arguably almost inevitable if you know something about the structure of your data. IMDb presents both the search widgetry and the answers in the vocabulary of the data-schema of movies and the people who work on them, not in "keywords" and "pages", and understands intimately that in IMDb's information-space search exists almost exclusively for the purpose of finding an entry point at which to start browsing. You go to Google to "look for something", you go to IMDb to "look something up"; the former phrase implies difficulty and disappointment in its very phrasing, the latter the comfortable assumption of success.  

On the web at large, of course, there is no meaningful schema, and it's impossible to make any simplifying assumptions about the subject matter of your question before you ask it. It is more productive to search in IMDb than with Google not because IMDb's searching is better, but because its data is better. But this does not even fractionally exonerate Google, or anybody else who is currently trying to solve an information problem by defining it as a search problem. They're all data problems. Google has the hardest version of this problem, since they don't directly control the information-space they're searching, but they have more than enough power and credibility to lead a revolution if they can muster the vision and organization. And anybody building an "enterprise" search tool has no such excuse; the enterprise does control their information-space, at least out to the edges where it touches the public space, and every second that can be invested in improving the data will be at least as productive as an hour sunk into flatly searching it.  

So if I worked for a Searching company right now, I'd start madly redefining ourselves tomorrow. We are not a searching company, we are an information organization company. The last resort is necessary, but neither sufficient nor transformative. I'd pull the smartest people I had off of "search" and put them to work on tools for the other end of the information process, reaching to the humans who are creating it and giving them the power to communicate not just the words of what they know but the structure of it, and to the collective mass of people to help them communicate and recognize and refine their collective knowledge about the schemas of known and knowable things. This is why Google Base holds the future of Google, and why you should sell your Google stock right now if they keep treating it as mainly a way for someone to buy your unused exercise equipment from you using a credit card. It should be the world's de facto public forum for the negotiation of the schema of all human knowledge, and if it isn't, every other decision Google makes will be forced by whatever is.  

But well-structured data, though necessary, isn't sufficient either. The good news for "search" companies is that improving the data is itself just a means to an end. Ideal data only encodes what we already know. The problems of useful inference from known data are hugely harder and super-hugely more valuable than the current forms of searching, especially when you realize that the boundary between private and public data is an obstacle and an opportunity in both directions not a wall to hide behind or run away from. The real future of "search" is in providing humans with the tools to form questions that haven't already been answered, and assemble the possible pieces of the answer, from threads of reasoning that traverse all kinds of territories of partial knowledge, into some form that synthesizes ideas that have never before even been juxtaposed, and onto which humans can further apply human powers where machine powers really fail -- fail because the machines are machines, not where they fail because we didn't take the time to let them be more thoroughly themselves -- so that they in turn can help us be more completely and wisely human.
Site contents published by glenn mcdonald under a Creative Commons BY/NC/ND License except where otherwise noted.