furia furialog · Every Noise at Once · New Particles · The War Against Silence · Aedliga (songs) · photography · other things
16 October 2008 to 24 July 2006 · tagged essay
I gave another version of my Whole Data blog post as part of a panel called "Death of the Relational Database?" at the Web 3.0 conference today.  

My answer to the question in the panel-title is that the relational database was always mostly a coping mechanism for a time of relative scarcity, when our data volume and needs exceeded our storage and processing capacities. This era is not entirely past us, but with more memory, faster data-loading, faster processing and better programming tools, there are increasingly many datasets of larger and larger size for which we can now support qualitatively richer abstractions. Where we used to have to break data apart, and thus were constrained to interact with it in fragmented and broken ways, we can now increasingly afford to keep it whole.  

And here, then, to try to explain why that matters, is my current four-point reduction of the Whole Data manifesto:  

1. What logic allows, storage may not constrain.  

The forms of the logical data-model and its inquiries are by definition independent of any extra-logical storage mechanics. That is, a proper data system exists for the express purpose of sustaining the illusion that the logical data-model is real. This must also be true over time: logically-sensible changes in the logical data model cannot be constrained by storage mechanics.  

2. All relationships are lists.  

Not only must multiple-value relationships and no-value relationships be exactly as easily expressed, manipulated and inquired about as single-value relationships, but every fundamental data operation should thrive on multiplicity and sequence.  

3. There is no up or down, only onwards from here.  

Humans impose hierarchy and directionality on data, but different humans recognize different hierarchies in the same data at the same time. The system itself must be agnostic. No fact is inherently the property of another, they are all independent and structurally equal, and although pairs of relationships can be inverses of each other, there is no absolute “up” or “down”, or “forwards” or “backwards”. A database of albums, artists and years, for example, is exactly as much a database of years as it is of artists as it is of albums, not by the generous will of its schema-creator, but by the intrinsic nature of data. Most significantly, you must be able to start at any node and see everything else in this dataset to which, and from which, it is connected. In fact, turn this around and you get the essential practical definition of a “dataset”: it is a collection of data in which all internal connections are expressed.  

4. Human inquiry follows paths; machine inquiry follows all the paths at once.  

The reason to put data into computers is so that computers can answer questions that are meaningful to humans, but faster for computers. A data model and query language exist to communicate human needs to computers, not to communicate machine preferences to humans, and the responsiveness of a data system is its qualitative ability to answer questions with human motivations, not its quantitative throughput.  
 

At the end of this talk I also did the first (very brief) public demo of Thread, the path-based query language I've written as part of the data-modeling (among other things) system I've been working on at ITA Software. I'm looking forward (to put it mildly) to being able to talk much more about this, but for now I just whirled through a fast series of mostly-unexplicated queries that at least gestured in the direction of the points in my manifesto:  

Album:Year=2000
albums from the year 2000; a one-to-one relationship in the old world, but in the new world the year 2000 is a real thing, too, making this many-to-one

Album:Artist=Nightwish
albums by Nightwish; a more familiar many-to-one relationship

Artist:Album~Dark
Artists who did an album with "Dark" in the name; a one-to-many relationship with the same syntax as above

Label:Album~Dark
Labels that put out albums with "Dark" in the name; same syntax again, but following a different path to Albums

Album:~Dark.Label
same question again, but following the path from Albums, instead of to them

Artist:(.Album.Label:=Spinefarm)
Artists who've had albums on Spinefarm; same pattern as above, but filtering on a compound relationship

Label:=Spinefarm.Album.Artist
or go the other way

Artist:(.Album:#5.Year:>2000)
Artists whose fifth album came out after 2000; multiples and missing values (one of the bands in the sample data had only four albums)

Artist:!(.Album:#5)
artists who don't have a fifth album; filtering on absence
 

I also mentioned in the talk, but didn't actually show, the hypothetical query "Who are all the living movie directors who ever directed Cary Grant?":  

Actor:=Cary Grant.Film.Director:!(.Date of Death)  

Go ahead and try to answer that in IMDB without a query language. For that matter, try to answer it in Freebase with a query language. (Or, even worse, try to answer it in Freebase yesterday, before I fixed a bunch of dates of death by hand...)  
 

There's also a short article up on semanticweb.com at the moment, which if nothing else gives a transcribed sense of how I explain these ideas differently in a phone interview than I do in writing.  
 

And there's space for discussion here.
I like CSV. It addresses a common data-exchange need, and does it with admirably little conceptual or syntactic overhead. As formats go, it's totally unglamorous, but glamor is very definitely not an end in itself. If you need to exchange a simple table of simple data, one datum per row/column, CSV is a fine tool.  

I also like JSON. Actually, I really like JSON. It has the data-modeling virtues of my old Elemental XML proposal, with a cleaner semantic distinction between hashes and arrays. I would like to see the spec amended to allow keys to be strings or numbers, rather than only strings, but other than that I think the whole thing is a solid, self-consistent, useful, appropriately scaled solution to an impressively large set of data-exchange and serialization problems. Certainly anything you could model with CSV, you could put into JSON instead, and not at all vice versa.  

But the Law of Conservation of Complexity applies here, as in most things. Although JSON syntax is pretty simple, any particular JSON data model can be arbitrarily complex, and the modeling issues in data-exchange are always far more involved than the syntactic ones. The important simplicity of CSV is in constraining the data model to be a table.  

Flexibility, however, can be employed in moderation. It is possible, for example, to render a CSV model in JSON syntax. That is, to turn this:  

Artist,Albums
Cradle of Filth,7
Nightwish,8
To/Die/For,4
 

into this:  

[
["Artist","Albums"],
["Cradle of Filth","7"],
["Nightwish","8"],
["To\/Die\/For","4"]
]
 

The advantage of doing so will, I hope, become quickly obvious once you realize that JSON then makes it trivially easy to have cell values be arrays. If we want a list of artists, each with their list of albums, CSV forces us to refactor:  

Album,Artist
Cruelty and the Beast,Cradle of Filth
Damnation and a Day,Cradle of Filth
Dusk...and Her Embrace,Cradle of Filth
Midian,Cradle of Filth
Nymphetamine,Cradle of Filth
The Principle of Evil Made Flesh,Cradle of Filth
Thornography,Cradle of Filth
Angels Fall First,Nightwish
Bless the Child,Nightwish
Century Child,Nightwish
Dark Passion Play,Nightwish
Oceanborn,Nightwish
Once,Nightwish
Over the Hills and Far Away,Nightwish
Wishmaster,Nightwish
All Eternity,To/Die/For
Epilogue,To/Die/For
IV,To/Die/For
Jaded,To/Die/For
 

This is annoying at minimum, and unworkable if we also wanted to see other properties of artists. But in JSON, or this modeling reduction of JSON that we might as well call "JSV", we can simply say:  

[
["Artist","Album"],
["Cradle of Filth",["The Principle of Evil Made Flesh","Dusk...and Her Embrace","Cruelty and the Beast","Midian","Damnation and a Day","Nymphetamine","Thornography"]],
["Nightwish",["Angels Fall First","Oceanborn","Wishmaster","Over the Hills and Far Away","Century Child","Bless the Child","Once","Dark Passion Play"]],
["To\/Die\/For",["All Eternity","Epilogue","Jaded","IV"]]
]
 

JSV retains the modeling simplicity of CSV (a single row of "fields" defined at the top), but allows a cell to be either a single value or a list of them. This is still unglamorous, but to me it's a big gain in expressivity for essentially no technical cost. I suggest that if combining lists and tables isn't a big deal to you (and you're the sort of person to whom a data-serialization format could be a big deal), maybe you've allowed tabular single-mindedness to beat all the natural human list-thinking out of you. Time to reclaim your right to multiplicity. Time to reclaim lots of your rights, really. Yet another argument for more lists, everywhere around us.
So there's "the Semantic Web", which has lots of philosophical baggage, and then there's "Linked Data", which is supposed to represent unpacking only the practical stuff from the piles of baggage. A secret password for a cult within a cult. A cult-within-a-cult big enough to have a two-day conference dedicated to it, but not yet big enough that the final panel on the second day wasn't facing a mostly empty room.  

And not quite yet a cult-within-a-cult with a coherent, useful agenda, I think. If you want to stage a grassroots revolution, you need to figure out four things:  

- What is the big change you're going to bring about?
- What's the work that has to be done?
- Who has to do the work?
- What's in it for them?  

Unfortunately, the current Linked Data agenda kind of answers these questions like this:  

- The big change is that all "data records" will have universally unique names, which will all also be web addresses so you can look them (or maybe something about them, or maybe kind of both) up with a browser, and when you look them up they will point you to other things.  

- The work to be done is that everybody must either convert their data into a list of individual subject-verb-objects assertions, including meta-assertions about those first assertions, or else at least construct the meta-assertions about the assertions and construct the unique names for the original assertions even if they don't actually exist. And all this should be done in a language (and data model) called RDF, most explanations of which begin by apologizing for it, which is understandable because it's the data-modeling equivalent of assembly language for programming, and pretty much nobody voluntarily works with nothing but levers when there's a Home Depot nearby.  

- The people who have to do the work are the owners of data.  

- The thing that's in it for them is that other people might mash up their data into something else, or discover it serendipitously, or effortlessly integrate it. Except Linked Data doesn't mean Open Linked Data, so you don't have to expose your data to the world, so in that case the benefit is, um, something something ontology something toolchain serendipitous LOOK!! OVER THERE!! IT'S A GIANT GLOBAL GRAPH EATING THE EMPIRE STATE BUILDING!!!!!! THIS IS THE WEB DONE RIGHT!!!!!  
 

This is a pretty bad plan for a revolution. It's hard to understand why the big change is big, never mind why it's good; the work is artificially difficult and deliberately obscure; the people who would have to do it are basically unprepared; and the "benefits" are peripheral more or less by definition.  
 

Let me try a different set of answers:  

- The big change is that we can finally afford to store most kinds of information in a way that separates the logical data model from the storage mechanics, so that the only thing that constrains what you can do with your data is your data.  

- The work to be done is that someone needs to build a node/arc-oriented database system, with accompanying data-model and query-language, that brings this idea up to the usability level of, say, a simple Excel spreadsheet or Google Base.  

- The people who have to do the work are data-system developers, whether new companies or existing ones.  

- The things that are in it for them are that the market of people with data is huge and growing, and the market for people with highly interconnected data is newly huge and frantically growing, and the existing RDBMS/SQL-based tools for analyzing and exploring and publishing that data are so awful that we will look back on them like we look back on COBOL and punch-cards. There is both money and human progress to be made.  
 

Mine probably initially seems like a smaller revolution. My enemy is not the web. My revolution is not about URIs or linking or dereferencing or ontology subsumption or transitive closure. I just know that we've wasted human-centuries of time fighting against internal problems introduced by relational-database implementations, and built legions of crappy, inflexible, inhumane data systems because the limitations of our databases constrained what questions we allowed humans to even ask, never mind get answered. And I know that computers are now big enough and fast enough that for increasingly many datasets, the old ugliness is no longer even remotely necessary.  

And thus the new ugliness, RDF and SPARQL, is a painfully tragic missed opportunity. I no more want to be buried in triples than I wanted to be boxed into tables. Where SQL and tables made some simple things relatively simple, some simple things very hard, and most hard things really hard, RDF and SPARQL make all things theoretically possible, but all specific trivial things barely feasible, any simple things too cumbersome to attempt without assistance, and hard things moot because the simple things raise so many epistemological issues that all actual work halts.  
 

We can do better. Let's call this new movement Whole Data, like whole food. Let our data be what it is to people, not what it's easier to be to machines. I don't claim this is any kind of new science. Object-oriented databases aren't a new idea, and arguably a binary-relation node/arc data-model is more or less what the original idea of a "relational" database was before mundane practicality spent a few decades abusing it. But we can build it now. We can build it so that people can use it, easily and without having to learn some whole esoteric new trade.  

And yes, a node/arc Whole Data database will lend itself incredibly well to publishing that data on the web. Arcs are links, and the web is made of links, and it's deliciously easy to turn links into links. Moreover, arcs are labeled links, and labeled links mean that machines can do something useful with them. But it is not the job of each data owner to rewrite their own database tools. It is not the job of each webmaster to figure out how to "semantically annotate" their pages. This is a revolution in data-modeling abstraction, and it will be fought by software designers and developers.  

And linking disparate datasets together, which is supposed to be the whole point of Linked Data, and thus the whole point of this slightly sad and more than slightly confused conference I went to last week, is only barely a technical problem to begin with, and it's not at all clear to me that the technical agenda of this subcult actually makes non-trivial linking any easier. Most of the claimed benefits, even if you don't worry about whom they're benefits for, seem to me like imaginative hand-waving.  

- Giving things unique IDs does make it easier to talk about them. But you can have IDs in table records, so that isn't new, and you can refer to things by instructions instead of IDs and most of the time it doesn't make a lot of difference, so IDs aren't even always necessary. And giving things web IDs is a totally orthogonal issue about publishing that isn't about the data itself at all, and makes no more general sense than saying that all children should be named with cell-phone numbers.  

- You can "mesh together" RDF sets. Sometimes, modulo some obtuse URI issues. And if the sets' conceptual data models are compatible enough, you might be able to get something out of the combination. But this is more or less true of relational database tables, too. Exchanging data is not difficult. CSV, JSON, XML, whatever. Agreeing what the data should be is difficult, and agreeing what it should be in triples is no easier than agreeing what it should be in tables. (Maybe easier in theory, but definitely harder in current practice.) Integration is never "effortless". It's just that sometimes somebody else has already done the effort of reaching agreement, and sometimes they haven't.  

- My adorable little FOAF file can link to yours, and we can both link ours to Tim Berners-Lee's. But this is a toy "data web". It doesn't scale. Currently it doesn't even scale technically, as the web is too slow for queries to be federated out across every foafy server on the planet, but solving the technical problem will only make it faster for us to see how useless the answers are for human reasons. Every level of remove you go through is a chance for context or trust or meaning to be lost. I don't know how you decide who goes in your file, and I certainly don't know how the people in your file decide who goes in theirs. If you find your way to me and then to Tim, it is unlikely to do you very much real good. On one hand, his office is about two blocks away from mine. On the other hand, the only reason it's becoming marginally more likely that he knows me is that I keep complaining about things he's working on. Small Pieces Loosely Joined works great when you have time to examine the joins and the pieces and figure out which ones are useful for some particular personal purpose. Carry them home, trim them, water them, love them, and you might get a garden for your efforts. Send a machine out to gather them all up at once and you'll get a mushy septic field with some fine heirloom tomatoes submerged somewhere in it.  

- As long as our cults are small, sometimes we can find each other with nothing but our code words. But thinking that URI standards will yield pervasive serendipity is socially oblivious. Nothing I do in my FOAF file will ensure that you find me when you look at Tim's. Distributed linking is inherently asymmetric, and even if you crawl the entire web to construct all the missing inverses, the numbers are implacable. You cannot center yourself in a social network without cooperation. The space of associativity is curved: you can move from the obscure to the popular easily (indeed often inexorably), and along popularity contours with some effort. The imbalance of attention is a human phenomenon, not a technical one.  
 

But my little revolution, at least, doesn't involving changing human nature. Distrust any that does. Whole Data is not magic, it's just healthier than the bleached, bromated version, and yummier than chewing raw semantic bran. All I'm trying to do is make information tools that allow humans to be less mechanical, and machines to be more helpful.  

Here are some helpfulnesses:  

- A node/arc database can handle multiple-value and no-value relationships just as easily as single-value relationships. Multiple-value relationships are 90% of the interesting structure of data, yet they're basically the bane of relational-database usability, and RDF only handles them if you don't care what order they go in or how easy it is to find out what they all are. No-value relationships are painful in relational databases, and inexpressible in RDF.  

- In a node/arc database you can make data-model modifications without touching any node whose data is not actually involved. The model does not determine the storage. In relational databases the model is the storage. Unless you already consider yourself a "working ontologist" you don't want to know what it takes to express a model in (actually for, because you can't do it in) RDF. Real models change all the time, so this can't be a half-assed after-thought.  

- In a node/arc database you can look at any data from the perspective of any other data. There's no "top", there's just "outward from here". In a relational database your tables and indexes limit your perspectives. In RDF there's no "top", but neither is there any "here"; all you can do is scan assertions until your patience runs out, looking for ones that say something else somethings this, or this somethings something else. Being able to stand anywhere and look in any direction is the difference between exploration and a zoo, and thus between discovery and watching an okapi crap.  

- A node/arc data structure can be browsed. Relational tables and RDF can mostly only be searched, and browsing is a UI illusion built on top of searching. But browsing, wandering, connecting, following, compiling, discarding: these are how humans organize and comprehend information, and we deserve a database in which those are the cheapest and sturdiest building blocks, not the most expensive and fragile fabrications.  

- In a node/arc database with a path-based query language, human browsing can be used as machine training, and human questions can be formulated for machine processing in structurally the same way they would be stated for human evaluation. We think in contexts and connections, not in subgraph matching or index inversion. We do not start every inquiry with a list of all the universe's truths. We start somewhere, and follow paths. We need to be able to send our questions down these paths, down all the paths at once, and have them come back with their own answers.  
 

The project I'm working on has a database system like I describe, and a data-model that's more usable than RDF, and a path-based query-language I wrote that's better than SPARQL. I want this thing to exist. I think we need it to exist. I'd rather it already existed, and if somebody else comes out with one tomorrow that's at least as good as mine, I'll be totally thrilled to use that instead. Mine isn't going to come out tomorrow. I don't know when it will. I don't know who will be totally thrilled, the day mine comes out, to use that one instead of the one they were working on.  

But I know these small improvements matter. Information matters, and understanding matters, and the tools for getting from one to the other matter. I think our ability to make use of information is part of how we are going to survive on Earth. So this, for the moment, is what I'm trying to do for us.
1. "Semantic". By starting the name this way, you have essentially, avoidably, uselessly doomed the whole named enterprise before it starts. Most people don't have the slightest idea what this word even means, most of the people who do have an idea think it implies pointless distinctions, and everybody left after you eliminate those two groups will still have to argue about what "semantic" means. This is a rare actual example of begging the question. Or to put it in terms you will understand: congratulations, you've introduced terminological head recursion. Any wonder the program never gets around to doing anything?  

2. "The Semantic Web". The "The" and the "Web" and the capitalization combine to suggest, even before anybody compounds the error by stating it explicitly, that this thing, which nobody can coherently explain, is intended to compete with a thing we already grok and see and fetishize. But this is totally not the point. The web is good. What we're talking about are new tools for how computers work with data. Or, really, what we're talking about are actually old tools for working with data, but ones that a) weren't as valuable or critical until the web made us more aware of our data and more aware of how badly it is serving us, and b) weren't as practical to implement until pretty recently in processor-speed and memory-size history.  

3. "FOAF". There have been worse acronyms, obviously, but this one is especially bad for the mildness of its badness. It sounds like some terrible dessert your friends pressured you to eat at a Renaissance Festival after you finally finished gnawing your baseball-bat-sized Turkey Sinew to death.  

4. FOAF as the stock example. You could have started anywhere, and almost any other start would have been better for explaining the true linked nature of data than this. "Friend" is the second farthest thing from a clean semantic annotation in anybody's daily experience. I'm barely in control of the meaning of my own friend lists, and certain wouldn't do anything with anybody else's without human context.  

5. Tagging as the stock example. "Tagged as" is the first farthest thing from a clean semantic annotation in anybody's daily experience.  

6. Blogging as the stock example. Even if your hand-typed RDFa annotations are nuggets of precious ontological purity, you can't generate enough of them by hand to matter. Your writing is for humans, not machines, and wasting brains the size of planets on chasing pingbacks is squandering electricity. We already know how to add to humanity's knowledge one fact at a time. The problem is in grasping the facts en masse, in turning data to information to knowledge to wisdom to the icecaps not melting on us.  

7. Anything AI. Natural-language-processing and entity-extraction are interesting information-science problems, and somebody, somewhere, probably ought to be working on them. But those tools are going to pretty much suck for general-purpose uses for a really long time. So keep them out of our way while we try to actually improve the world in the meantime.  

8. "Giant Global" Graph. The "Giant" and "Global" parts are menacing and unnecessary, and maybe ultimately just wrong. In data-modeling, the more giant and global you try to be, the harder it is to accomplish anything. What we're trying to do is make it possible to connect data at the point where humans want it to connect, not make all data connected. We're not trying to build one graph any more than the World Wide Web was trying to build one site.  

9. Giant Global "Graph". This is a classic jargon failure: using an overloaded term with a normal meaning that makes sense in most of the same sentences. I don't know the right answer to this one, since "web" and "network" and "mesh" and "map" are all overloaded, too. We may have to use a new term here just so people know we're talking about something new. "Nodeset", possibly. "Graph" is particularly bad because it plays into the awful idea that "visualization" is all about turning already-elusive meaning into splendidly gradient-filled, non-question-answering splatter-plots.  

10. URIs. Identifying things is a terrific idea, but "Uniform" is part of the same inane pipe-dream distraction as "Giant" and "Global", and "Resource" and the associated crap about protocols and representations munge together so many orthogonal issues that here again the discussions all end up being Zenotic debates over how many pins can be shoved halfway up which dancing angel.  

11. "Metadata". There is no such thing as "metadata". Everything is relative. Everything is data. Every bit of data is meta to everything else, and thus to nothing. It doesn't matter whether the map "is" the terrain, it just matters that you know you're talking about maps when you're talking about maps. (And it usually doesn't matter if the computer knows the difference, regardless...)  

12. RDF. It's insanely brilliant to be able to represent any kind of data structure in a universal lowest-common-denominator form. It's just insane to think that this particular brilliance is of pressing interest to anybody but data-modeling specialists, any more than hungry people want to hear your lecture about the atomic structure of food before they eat. RDF will be the core of the new model in the same way that SGML was the core of the web.  

13. The Open-World Hypothesis. See "Global", above. Acknowledging the ultimate unknowability of knowledge is a profound philosophical and moral project, but not one for which we need computer assistance. Meanwhile, computers could be helping us make use of what we do know in all our little worlds that are already more than closed enough.
There are no dead-ends in data. Everything connects to something, and if anybody tells you otherwise, you should suspect them of hoping you won't figure out the connection they've omitted.  

But we have suffered, for most of humanity's life with data, without good tools for really recognizing the connectivity of all things. We have cut down trees of wood and used them to make trees of data. Trees are full of dead-ends, of narrower and narrower branches. Books are mostly trees. Documents are usually trees. Spreadsheets tend to be trees. Speeches are trees. Trees say "We built 5 solar-power plants this year", and you either trust them, or else you break off the end of the branch and go looking for some other tree this branch could have come from. Trees are ways of telling stories that yearn constantly to end, of telling stories you can circumscribe, and saw through, and burn.  

And stories you can cut down and burn are Evil's favorite medium. Selective partial information is ignorance's fastest friend. They built 5 solar plants, they say. Is that right? And how big were they? And where? And why did they say "we built", past-tense, not "we are now running"? And how many of last year's 5 did they close this year? And how many shoddy coal plants did they bolt together elsewhere, while the PR people were shining the sun in our eyes? There are statements, and then there are facts, and then there is Truth; and Truth is always tied up in the connections.  

Not that we haven't ever tried to fix this, of course. Indices help. Footnotes help. Dictionaries and encyclopedias and catalogs help. Librarians help. Archivists and critics and contrarians and journalists help. Anything helps that lets assertions carry their context, and makes conclusions act always also as beginnings. Human diligence can weave the branches back together a little, knit the trees back into a semblance of the original web of knowledge. But it takes so much effort just to keep from losing what we already knew, effort stolen from time to learn new things, from making connections we didn't already throw away.  

The Web helps, too, by giving us in some big ways the best tools for connection that we've ever had. Now your assertions can be packaged with their context, at least loosely and sometimes, if you make the effort. Now unsupported conclusions can be, if nothing else, terms for the next Google or Wikipedia search. This is more than we had before. It is a little harder for Evil to hide now, harder to lie and get away with it, harder to control the angle from which you don't see the half-truth's frayed ends.  

But these are all still ultimately tenuous triumphs of constant human vigilance. The machines don't care what we say. The machines do not fact-check or cross-reference, of their own volition, and only barely help us when we try to do the work ourselves. The Web ought to be a web, a graph, but mostly it's just more trees. Mostly, any direction you crawl, you keep ending up on the narrowest branches, listening for the crack. All the paths of Truth may exist somewhere, but that doesn't mean you can follow them from any particular here to any specific there.  

And even if linking were thoroughly ubiquitous, and most of the Web weren't SQL dumps occasionally fogged in by tag clouds, this would still be far from enough. The links alone are nowhere near enough, and believing they are is selling out this revolution before it has deposed anything, before it has done much more than make some posters. It is not enough for individual assertions to carry their context. It is not enough for our vocabulary of connection to be reduced to "see also", even if that temporarily seems like an expansion. It is not enough to link the self-aggrandizing press-release about solar plants to the company's web site, and hope you can find their SEC filings under Investor Relations somewhere. It is not enough to link the press-release to the filings, or for your blog-post about their operations in China to make Digg for six hours, or to take down one company or expose one lie. We've built a system that fountains half-truths at an unprecedented speed, and it is nowhere near enough to complete the half-truths one at a time.  

The real revolution in information consists of two fundamental changes, neither of which have really begun yet in anything like the pervasive way they must:  

1. The standard tools and methods for representing and presenting information must understand that everything connects, that "information" is mostly, or maybe exactly, those connections. As easy as it once became to print a document, and easier than it has become to put up web-pages and query-forms and database results-lists, it must become to describe and create and share and augment sets of data in which every connection, from every point in every direction, is inherently present and plainly evident. Not better tools for making links, tools that understand that the links are already inextricably everywhere.  

2. The standard tools for exploring and consuming and analyzing connected information must move far beyond dealing with the connections one at a time. It is not enough to look up the company that built those plants. It is not enough to look up each of their yearly financial reports, one by one, for however many years you have patience to click. It's time to let the machines actually help us. They've been sitting around mostly wasting their time ever since toasters started flying, and we can't afford that any more. We need to be able to ask "What are the breakdowns of spending by plant-type for all companies that have built solar plants?", and have the machines go do all the clicking and collating and collecting. Otherwise our fancy digital web-pages might as well be illuminated manuscripts in bibliographers' crypts for all the good they do us. Linked pages must give way to linked data even more sweepingly and transformationally than shelved documents have given way to linked pages.  
 

And because we can't afford to wait until machines learn to understand human languages, we will have to begin by speaking to them like machines, like we aren't just hoping they'll magically become us. We will have to shift some of our attention, at least some of us some of the time, from writing sentences to binding fields to actually modeling data, and to modeling the tools for modeling data. From Google to Wikipedia to Freebase, from search terms to query languages to exploration languages, from multimedia to interactive to semantic, from commerce to community to evolving insight. We have not freed ourselves from the tyranny of expertise, we've freed expertise from the obscurity of stacks. Escaping from trees is not escaping from structure, it is freeing structure. It is bringing alive what has been petrified.  

There are no dead-ends in knowledge. Everything we know connects, by definition. We connect it by knowing. We connect. This is what we do, and thus what we must do better, and what we must train and allow our tools to help us do, and the only way Truth ever defeats Evil. Connecting matters. Truths, tools, links, schemata, graph alignment, ontology, semantics, inference: these things matter. The internet matters. This is why the internet matters.
If you're going to waste your time doing obsessive analysis of data on which nobody's life or ecology depends, you ought to at least do it diligently and efficiently.  

About a month ago The Deciblog published Justin Foley's attempt to answer the timeless question "How likely is a metal band to start their name with a particular letter of the alphabet?". For his sample set, Foley took the combined rosters of several metal labels (he doesn't reveal either list), which gave him 814 names, for which he then calculated the first-letter distributions, reaching the startling conclusion that the most likely letter is S.  

Foley put his results in a bar-chart, which I assume means he used a spreadsheet, so hopefully he didn't spend a whole lot of time hand-counting. But he should have spent even less. The Encyclopaedia Metallum is not only just sitting there with a collaboratively-amassed and collectively-moderated database of 50,000+ metal bands, but they've even already split it up by first-letter and there are band-counts right at the top of each letter-page. Add, divide, and you're done.  

Here, then, is the much better-informed version of this still-pointless breakdown:

? % *
# 0.3%
A 9.1% *********
B 5.9% *****
C 6.3% ******
D 8.9% ********
E 4.9% ****
F 3.6% ***
G 3.0% ***
H 3.9% ***
I 3.7% ***
J 0.6%
K 2.2% **
L 3.1% ***
M 7.4% *******
N 4.2% ****
O 2.3% **
P 4.0% ***
Q 0.2%
R 3.3% ***
S 10.8% **********
T 4.5% ****
U 1.3% *
V 2.7% **
W 2.7% **
X 0.3%
Y 0.2%
Z 0.7%


Most of Foley's numbers aren't that far off. His small sample-size leads him to overestimate J and Y, and underestimate Q and V, but the absolute numbers for these letters are small anyway. He also seems to underestimate A, P and R, for reasons which are not apparent in his opaque reporting, but might have to do with language tendences, as EM's list is probably more global than his.  

But the biggest discrepancy in Foley's numbers, by far, is T, which he credits with 9.3% of the band-names, where EM data indicates less than half that. Here I have a wearyingly mundane but highly plausible theory: Foley has accidentally counted all the bands whose names begin with "The " as T, despite specifically saying that he didn't. This is obviously both philosophically and methodologically repugnant, and although I regret the maelstrom of blogospheric outrage that will undoubtably accompany my public exposure of this error, I think we owe ourselves (the) Truth.  



Of course, the thousands of metal fans around the world who have put time and effort into building the Encyclopaedia Metallum did it because they care about the music, not the alphabet. The site is the definitive central reference source for most metal-related matters, and certainly the final arbiter of obscurity for the vast unknown majority of the bands it lists.  

It also has, in addition to its factual content, tens of thousands of percentage-scored, user-attributed, peer-moderated reviews of metal recordings. What it does not have is any sort of similarity analysis to make use of the huge data-graph represented by the connections between bands, users and ratings. There is a wearyingly mundane reason for this, too: trying to do similarity analysis with SQL queries will make you want to eat your own neck.  

So here is an actual contribution to the world's knowledge on this admittedly peripheral subject: empath, the missing similarity analysis of EM user/review/band data, accurate as of yesterday. Pick a band, see the other bands that people who like the first band also like. Data wants to form shapes. In a better world, this would be just as easy for EM to do themselves, updating live, as it is for them to serve their raw data into web pages.  

Easier, actually. I bet it took me less time to do this analysis than it took Foley to make a bar-chart of first letters. But I have better tools. I have better tools because at the moment I'm paid to design better tools. If I do my job well enough, eventually you'll have my better tools, too. I'm not designing them to tabulate heavy metal, I'm designing them to answer questions. Not all answers turn out to be shaped like Truths, of course. But if you can't answer them, you can't be sure which are which.
Two of the very bad features of the SQL-based conception of relational data-modeling, I think, are these:  

- one-to-many relationships are significantly harder to handle than one-to-one relationships (and, inversely, many-to-one relationships easiest to handle by defactoring)  

- absence is significantly harder to handle than presence  

It's easy to add an Artist column to an Album table, for example, so we get:  

AlbumArtist
Black OneSunn O)))

But if you want to model the fact that collaboration albums can have multiple artists, you need an Albums table, an Artists table, a join table, and a bunch of IDs:  

Albums
AlbumID
Black One1
Altar2

Artists
ArtistID
Sunn O)))1
Boris2

Authorship
AlbumIDArtistID
11
21
22

This is already a pain in the ass, and thus you get a world filled with cop-outs like this:  

AlbumArtist
Black OneSunn O)))
AltarSunn O))) / Boris

and this:  

AlbumArtist 1Artist 2Artist 3
Black OneSunn O)))FALSEFALSE
AltarSunn O)))BorisFALSE

both of which are tolerable enough to just look at, but awful when you go to try to get the computer to answer even perfectly sensible questions like what albums Boris has done.  

And then, when you want to add a little more detail, you really ought to do this:  

Albums
AlbumID
Black One1
Altar2

Artists
ArtistID
Sunn O)))1
Boris2

Formats
FormatID
CD1
LP2
Download3
Authorship
AlbumIDArtistID
11
21
22
Formatship
AlbumIDFormatID
11
12
13
21
22

but instead you'll probably try to get away with:  

AlbumArtistFormatCode
Black OneSunn O)))CD/LP/D
AltarSunn O))) & BorisCD/LP

or  

AlbumArtist 1Artist 2Artist 3CDLPD
Black OneSunn O)))FALSEFALSETRUETRUETRUE
AltarSunn O)))BorisFALSETRUETRUEFALSE

because if you have the latter, you can do:  

SELECT Album, Artist

FROM Albums
WHERE D=FALSE

which is easy to understand, although it won't work because you forgot you don't have a field called plain "Artist" anymore. Whereas with the five-table form you have to do something like:  

SELECT Albums.Album, Artists.Artist

FROM Albums, Artists, Authorship
WHERE Albums.ID=Authorship.AlbumID
AND Artists.ID=Authorship.ArtistID
AND NOT EXISTS (
SELECT *
FROM Formats, Formatship
WHERE Albums.ID=Formatship.AlbumID
AND Formats.ID=Formatship.FormatID
AND Formats.Format='Download'
)

and that's probably still wrong.  
 

I submit that this is all a colossal mess, and it's a wonder the database-driven software we get driven out of it isn't even more woeful than it is. A better paradigm would embrace the opposites of these two flaws:  

- the modeling of a relationship should be the same whether the relationship is one-to-one, one-to-many, many-to-one or many-to-many, and should be the same no matter how many "many"s you have  

- absence and presence should be equally easy to assess  

And I think a useful rough metric for such a new system is that it allows you to model your data without ever needing to say FALSE.  

And I think this, I think, because I think this is how we think and talk about relationships between things when nobody is getting in our way:  

Black One was recorded by Sunn O))), and was released in CD, LP and Download formats. Altar was recorded by Sunn O))) and Boris, and was released in CD and LP formats. All of these albums were released on CD and LP. Altar is the only one that wasn't released in Download format.  

Take these out of sentence form and make the data-types explicit and everything is still perfectly straightforward:  

Album: [Black One]

Artists: [Sunn O)))]
Formats: [CD] [LP] [Download]  

Album: [Altar]
Artists: [Sunn O)))] [Boris]
Formats: [CD] [LP]

or, looking at the same data from a different perspective:  

Format: [CD]

Albums: [Black One] [Altar]  

Format: [LP]
Albums: [Black One] [Altar]  

Format: [Download]
Albums: [Black One]

and then the old questions are also easier:  

FIND Formats WHOSE Albums INCLUDE 'Black One' and 'Altar'

FIND Albums WHOSE Formats DON'T INCLUDE 'Download'

Subliminating the one/many distinction allows us to talk about albums with 3 or 8 or 0 artists just as readily as we talk about albums with 1 or 2. Fixing the query language to handle absence lets us cope with the introduction of a new edition into our dataset, say, without having to add a new field for it and/or go through asserting that everything we already knew about doesn't have that format. I suspect pretty much every database field headed X and filled with TRUEs and FALSEs could at least be more usefully modeled in my new world as a relationship called "Characteristics" that either does or doesn't include X. And usually, as with formats, there's actually a meaningful label that contains information itself. The point is that things have relationships to their characteristics, or their formats. They don't, except in a very existential sense, have a relationship to falseness. FALSE is a machine thing.  
 

The relationally astute will note that the five-table version is the fully-normalized representation that would support an SQL-based implementation of my new form as a display abstraction, and presumably my pseudo-query-code could be preprocessed into the messy SQL so I'd never see it. But that's my point: SQL is too low an abstraction, which makes it too hard to do things naturally, which makes it too hard to pass along natural behavior to the victims of the system. I'm mainly arguing for a better abstraction. (Although I also expect it's still always going to be bad that there's no better underlying way to represent lists, so I'm also probably arguing for the better abstraction to not merely be layered on top of the bad one.)  
 

The discographically astute will note that Altar is available on iTunes, and that I have vastly undermodeled the rich and complicated space of Sunn O))) release formats. But if I'd done the examples in real detail, I'd still be typing out the bad versions...
I've come to the unexpected and disconcerting realization that my period of greatest apparent productivity in user-interface production was, pretty clearly, when I was mainly working in Visual Basic, circa 1994-1997. Ten years later, I'm actually spending much more of my design time repeatedly re-implementing basic mechanics of no inherent interest, and my design work at any given point in time is littered with more unsightly temporary hacks of things I haven't had time to re-implement this time yet. It takes me longer to produce first approximations, and longer to add some of the most common levels of noticeable refinement.  

This would be deeply tragic in several dimensions if it didn't deserve enough qualifying asterisks to make an ASCII-art smiley the size of Nebraska. The genius of Visual Basic was that it allowed you to very quickly produce 90% of a UI that was as good as 90% of everything else. Getting from 90% to 100% of a UI that was as good as 90% of everything else, however, took a large amount of extremely annoying work, some of which had to be constantly rechecked and redone every time you made the tiniest change to anything thereafter. And worse, of course, the standard established by 90% of everything else was not, in objective human terms, actually very good.  

Visual Basic's success, such as it was, relied on two very large simplifications of the underlying human problems. First, technically, VB was both a development and a deployment environment. VB applications were actually configuration data for the VB runtime. There were no platform-matrix issues, because VB itself was the platform. The specific version of VB with which you built the application, even. Unless you started introducing your own external dependencies, you could basically count on the things running the same way for your users as they ran for you.  

Much more significantly, though, VB instantiated a fixed vocabulary and grammar of UI interaction. This made it extremely simple to produce static forms and dialog boxes using standard Windows UI widgets, fairly easy to add a few well-defined kinds of dynamism to them, cumbersomely possible to do a little better, and beyond that you ended up fighting it more than it helped, and so usually didn't bother.  

This is what all well-conceived software projects and all self-aware computer programmers do, of course: we take environments in which a great many things are complicatedly possible, and transform them into environments in which a small subset of those things are significantly easier, and a larger number of things that used to be hard are now so much harder than the easier things that nobody wastes time trying them anymore. We do this by, mainly, an obsessive attention to abstraction and instantiation and packaging and layering. This is why the details of most programming successes are staggeringly boring to non-programmers in rough proportion to their genuine significance: the best programming work usually is not done in directly solving an individual human problem, but in making some tool, ten steps away from the visible problem, that makes it incrementally easier to solve some whole class of abstruse subproblems that factor in some small way into the one you care about. The web page that lets you buy embarrassing personal ointments without having to hand them to a 4'5" woman named Ruthie, who reads every package label aloud in megaphone tones, solves a human problem for you. The good programming work behind that page has nothing to do with ointment or Ruthie, and lots to do with content-management-system staging strategies, database transaction consistency, protocol standards, application-framework meta-programming, and twenty other topics so obscure and tedious that you'd probably rather talk about why you need the ointment.  

The problem with Visual Basic, circa 1994, was that the things it made easy were not important enough. Using it, I was able to produce a lot of UI very quickly, the Windows UI paradigm in which VB thrived was itself the cause of most of that work. I spent most of my time laboriously refining mildly optimized sets of widgetry in order to get the computer to show me something that I already knew. It'd be my guess that seeing and/or editing simple pieces of known data accounts for about 94% of the code ever written for computers, and that seeing and editing more-complex clumps of known data accounts for another 6%. All the stuff that does something with that data, and for which a computer is more than a very expensive and quickly obsolescent filing cabinet, at this level of precision accounts for approximately 0%.  

The web, as the new default environment for data applications, is both progress and regress. The literal display of data is generally a little easier than it was in most pre-HTML environments. Simple data editing is about as easy as it ever was, but orchestration of the editing of related bits of data is harder. Managing large sets of data, and managing the social dynamics of shared data environments, are probably slightly easier than before in relative terms, but since the connected world makes them less obviously intractable, they now represent a larger fraction of an increased overall quantity of active difficulty. And if the problem is taking the things you know, and the things I know, and getting to something new we now know together, a disappointing amount of what we've built turns out to be discussion forums where people we've never met can file unverifiable reports on whether the ointment worked for them.  

But the web, as it is now or even as the SPARQL/RDF version of the Semantic Web might make it, is yet another environment that generates a huge amount of the work it requires. This is ultimately the metric by which I think any tool, including software tools, and including meta-tool-sets as big as the web, should be most diligently measured: does it increase or decrease the human-scale meaningfulness of the human work done with it? In the same way that "the economy" should be graded on net human welfare, not gross currency movement, software should be graded on how it allows us to spend our lives, not how industriously it allows us to do new work nobody had to do at all before. It's amazing that it's possible to put a shame-free ointment store online, but absurd how much harder it is to do that than to put a tube of ointment on a metal shelf. And profoundly pathetic that the magnitude of our shame makes the enterprise seem reasonable.  
 

The software-design work I'm doing now won't, directly, even get rid of your rash, let alone reverse global carbon-dioxide trends. Gauging the value of your daily work based on its capacity to increase the welfare of humanity is practically difficult, and theoretically presumptuous if not absurd. But I think it matters whether it matters, and I really do believe that I am working on a part of the solution. If human existence is the stake in a race between shared insight and thermodynamic chaos, chaos has all the big-money sponsors, but shared insight is a power law. I'm trying to build software that does, for networks of highly interconnected shared data, what Excel does for columns of numbers (or, maybe more realistically at first, what Visicalc originally did for columns of numbers). I'm trying to help bring about a new base level of operative abstraction in which data is visible, and its connections traversable, by its very nature, so nobody has to write code to see what we already know. I believe that it should be possible for computers, instead of diligently enforcing dehumanized bureaucracy, to help sustain the transformative illusion that all it takes for you, as a human, to participate in the sharing of human knowledge is one thing nobody else imagined before, or one link between things that nobody else has noticed yet.  

And although global warming is not entirely a data problem, even that might arguably be mostly a data problem. Its key vectors all originate in what are fundamentally data blockages: inadequate education, incomplete understanding, incomprehensible macro-effects of unassessable micro-decisions, misapplications of science or technology, misguided optimizations of local variables at the expense of the overall system, failures of empathy, failures of recognition, and perhaps above all else the way in which untreated ignorance makes it possible to feel safely (and/or alienatingly) isolated from things to which you are really deeply and inextricably connected.  

I'm trying to help make the internet into the place where we know what we know. I'm trying to help make a new place where we go to do human things of which we are collectively proud, not somewhere where we hide among the widgets while we pray that this is the ointment that will make us feel better.
The logistical irony of substantive blogging, meaning something closer to essay-writing than link-jockeying, is that it enthusiastically both rewards and consumes time. For quantitatively ideal productivity you need to minimize the amount of time you spend doing anything else in your life, which obviously includes anything orthogonal to blogging like shopping for diaper-containment systems or eating brunch with your friends (unless, obviously, you are running a diaper-containment- or brunch-blog), but also includes whatever it takes to actually generate the material to write about whatever it is you're writing about.  

I only ever came up with two good solutions to this, where by "good" I mean effective in producing output, not necessarily in improving my own life any.  

One was writing about music during a period when most of my spare time was spent listening to and thinking about music anyway. It's a trivial psychological insight to observe that the magnitude of this preoccupation betrays it as an attempt to distract attention, mostly mine, from the absences of other things in my life.  

The other solution was writing about technology, or around technology in the same sense that I wrote around music, during a period when my day-job had devolved into pretty much exclusively sitting around thinking about and around technology. This is a pretty interesting experience to get to have, and as I repeated countless times in the job interviews that followed its inevitable eventual termination, I was never even remotely as well informed about the state of technology when I was actually contributing to it.  

I'm contributing to it again. My new job, still settling past its newness after only a few months, is by far the best job I've ever had. There's one formulation of job ideal where you say "I'd be doing this even if they weren't paying me", and this one is beyond that to "I'd be sitting at home frustrated at being unable to participate without a team around me."  

Several things are different and cooler about this job, at least in my own experience, the top few being that I'm explicitly managing it in addition to leading by design, that we're trying to find the shape of the solution we're building (and of the problem we're defining to solve) instead of being handed a VC precis or business plan to fulfill, and that the development work is not primarily segmented by function.  

The combination of these things is very challenging and extremely engrossing. I discovered a quantum jump in responsibility when I went from being a member of a large design team at Interchange to being the sole designer at eRoom, and it's a similar one from being the designer to being the designer and manager, and maybe another level again to be the person with the most navigational accountability for an exploration undertaken knowingly without a map. The internal pressures intensify radically in both directions: to dismiss inanely crazy ideas to avoid causing magnified distractions, and to take superficially inexplicable intuitions even more seriously for their transformational potential. If you don't know where you're going, exactly, you have to operate as if patience and diligence are doomed.  

Somehow this has resulted, over the past few weeks, in my spending the vast majority of my working hours actually programming. I won't reveal much by saying that we're building a big software system that will, when it's working, take in a large amount of previously scattered and unanalyzed data, and attempt to reduce its disorder. Most of the team has been working, since before my arrival, on the data-acquisition aspects of the project. So without anybody else to assign to the languishing data-analysis portion of the system, I assigned it to myself, and have been maniacally trying to figure out what I mean it to do by writing it.  

I am not, in case I haven't been clear about this, really a professional programmer. Over the course of my career I've written a lot of code in a lot of languages, but almost always in carefully circumscribed senses. The UI architecture of eRoom was devised in no small part with my individual capabilities in mind, so that I could write a large amount of the actual UI code without having to participate in non-goal-oriented conversations about lazy pointer deallocation or rogue mutexes. And even the code I'm writing now is, probably, only a prototype, and I've written lots of prototype code.  

My previous prototyping, though, has always been about faking a system in order to play with how it looks and acts. The easiest way to fake a large system is usually to actually build parts of it more or less for real, of course, but every project I seem to fake a little less. The content-management and discussion-forum systems underlying this website are constrained and minimal, but not faked at all. And although the prototype I'm writing at work will probably be migrated out of my eccentric Ruby improvisation into somebody's tediously meticulous Java for dull scalability reasons, my Ruby version really does do what it says it does.  

This is tremendous fun, but it also pushes me into a part of my brain where my most obsessive nature is most dramatically exacerbated. I sit down at my desk in the morning, and eight or nine hours slide by in an un-self-interrupted continuum. I forget to eat lunch, I forget to check my email, I forget the five-minute phone-call-returning task I'm supposed to do while eating the lunch I forgot to eat, I look up at the clock and realize I was supposed to be home an hour ago, and when you live a block from your office there's no possible excuse. Making information fall magically into order is not the only thing I want to do with my life, but it's one of the ways I think I can make the world better, and in the grip of a particularly involved spell it feels like a calling. Something shadowily like a pattern starts to emerge, and I frown at it, and twitch something I've built to be twitched, and suddenly answers are fountaining out of the screen from their own joy at existing.  
 

And that's not even the interesting news.  

The interesting news is that Bethany is pregnant. If the tests and milestones keep going as well as they've been going so far, about six months from now we'll have a child. Software isn't easy, exactly, but it's difficult in easy ways. If you know what to twitch, you can know what kind of magic you'll be able to make.  

People are, obviously, far less tractable. I don't know what kind of child we're going to have. I don't know what kind of parents we're going to be. I don't know what's going to be ecstatic and what's going hurt and what's going to be unbearable. I know nothing will ever be this simple again, and I was already barely coping with simplicity's complexity. Beth and I could probably have spent another ten or twenty years just figuring out how to merge two lives most inspiringly, and now we're going to try to make sense of three. My own explanation of wanting to do this is that raising children is the biggest project available to individual humans, and Bethany is the person with whom I feel most capable of the most humanity. I want us to take on the greatest, hardest, most human challenge for which we are eligible.  

As crazy intuitions go, this makes my daftest futurist semantic-web predictions feel like placing wrist-flick roulette bets with free (and mostly unredeemable) chips. I really do believe that the semantic-web project I'm working on has the potential to be the seed of something that changes the dynamics of human interaction with information sufficiently to advance human nature, but I know that that potential is prorated by its likelihood. I have a small team, and there are many teams and bigger, and many factors and convolutions. Most attempts at deliberate innovation fail, and although opportunities for accidental innovation fail at a far greater rate, they arise at an even greater rate than they fail, so the innovations arrived at by luck vastly outnumber and outfascinate the few we make knowingly.  

We will have made this child knowingly, and although accidents will probably outnumber clevernesses even more lopsidedly thereafter, in this project we will not have the luxury or salvation of irrelevance. There are no swarming competitors feverishly plotting to steal away our parent-share or undermine B and my parenting brand. There are no breathless marketers trying to find the most flattering spin on whatever turn out to be our signature family dysfunctions. Nothing will release us from this. There's just us, and a new person nobody on Earth has met yet. Somebody has to care for everybody, and this one is ours.  
 

So I'm obviously not ready, and can't possibly be, but I'm ready. There are a thousand things to do, and stacks of books and friends and doctors and counselors lining up to add more things to our lists, and I can't imagine how we'll ever get them all done, so I assume and stipulate that we must not have to. I will do whatever it takes to be a good parent, and that will usually be almost enough, and hopefully every once in a while will jubilantly suffice. I will try to be better at this than anything else I have ever done. I don't intend for it to become the only thing I do, but I understand that, like marriage, it is a private commitment that takes precedence over my public aspirations, and that I will make different choices now, and dream of different things. I will have far more important things to write about than b-sides, and rarely even enough time to list them. I will do what I can, and allow myself not to catalogue, even mentally, all the things I would have catalogued when I had time instead of purpose.  

And if this is the most ordinary, and thus in a sense the most mundane, way of raising the stakes for the ways in which I hope to improve the world, and to avoid being even a passive vector of thoughtlessness or evil, then so be it. These swiveling electrical-outlet covers I need to install so a baby that doesn't even exist yet can't stick (I guess) a fork in them, and whatever diaper-containment system we ultimately settle on, and the semantic web and the way we love each other and ourselves, will now all also be part of the inescapably consequential context for a new life. None of this will be faked, or maybe all of it will be, or maybe this is where it stops being meaningful to talk about the difference between what you fake and what you know. This is where suddenly the craziest answers are fountaining out of nowhere into a flood of what it is up to us to make be joy.  
 

(Discussion on vF...)
Somewhere, somebody needs a software system for something. Ordering sandwiches, maybe. It doesn't really matter, because software doesn't manipulate sandwiches, it manipulates information. So some programmer abstracts out the information-problems involved, and gives the abstractions names, and writes a program that operates them.  

Which hums along nicely for a while. The program creates blorfs, juggles them with admirable dexterity, and then copes ingeniously with whether they are sterkled or not. Only the programmer knows what blorfs are, exactly, and what it means to be sterkled, but the program knows what to do with them, and that's what programming is about.  

Except one day the program gets to the point where it has to actually assess the sterkledness of a specific blorf that represents some actual thing out in the world with an actual person attached to it, and thus arises the need for a User Interface. The programmer winces briefly, but then gamely poses the question:  

Is your blorf sterkled?

The user types something, and the program promptly crashes. Most of the subsequent changes to the program will effectively amount to this:  

IMPORTANT! IS YOUR BLORF STERKLED?!?!

If you speak your own language slowly and loudly enough, even uncooperative foreigners are bound to pick up at least the gist of what you're saying.  

But let's pretend the programmer gets past being irritated at the user, and realizes that they might benefit from some help answering the question. Help=instructions:  

Is your blorf sterkled? [yes/no]

Mysteriously, this doesn't really improve things. At this point the programmer reluctantly digs out that book about UI Design, and over the course of a painful few days produces this series of incremental and undeniable widget improvements:  

Is your blorf sterkled?

Is your blorf sterkled?

Is your blorf sterkled?

Is your blorf sterkled? yes no

Is your blorf sterkled? yes no

Your blorf: Sterkled

Your blorf: Sterkled

Calls to action are better than blanks, defaults are usually better than forced choices, radio buttons require fewer clicks than drop-downs when the number of choices is small, and checkboxes are even simpler when the choice is really binary. With any of the last three versions, at least, the program will be much happier. It needs an answer, and the widgets are such that it's going to get one no matter what the user does.  

Passing a usability test is a little harder than not crashing, though. The usability problem, of course, is that the user doesn't have any idea what a blorf is, or what it would mean to be sterkled, so these widgets could spin and invert and coruscate without ever changing their comprehensibility. Design is not the application of widgets to programming variables. At this point the programmer tracks down a product manager, if one can be dragged away from fiddling with PowerPoint transitions, and after some painful cross-purposes conversation they realize that the program needs to translate, for UI purposes, back into the user's vocabulary. Blorfs are, it turns out, sandwiches, or some meta-food-product abstraction that is, in this case, instantiated by a sandwich. Sterkle is a class of food-instantiation modifications, which in this case is the application (or non-application) of mayonnaise. Thus the original question is actually:  

Do you want mayonnaise on your sandwich?

Or, in our hard-won UI simplicity:  

Additions to your sandwich: Mayonnaise

This revision passes the usability tests with scores .724 higher than any UI the lab has ever evaluated, and the application is triumphantly deployed.  

By halfway through the first production lunch hour, the customer has threatened to insert the product manager's head into the panini press. The users (the customer's customers) like the software fine, but they hate their sandwiches. After a series of crowded and acrimonious meetings it is finally deduced that having mayonnaise checked by default has resulted in it being spread liberally on nearly every sandwich, because users are in a hurry and have just been clicking OK. So what the original question really amounts to, a lot of the time, is something much more like:  

Do you want mayonnaise on your peanut-butter-and-jelly sandwich?

All this time and work for a question that is almost always going to be answered one way. So we finally get this:  

Additions to your peanut butter and jelly sandwich: Mayonnaise

And, after the addition of a fancy trend-analysis module of which the programmer is justifiably proud:  

Additions to your peanut butter and jelly sandwich: Mayonnaise (not recommended!)

Sadly, although the trend analysis is fascinating for the customer, the customer's customers are still pretty unhappy. If you like mayonnaise on your ham sandwich, it's not really interesting or helpful that the system thinks most people don't. The programmer wastes a month building an elaborate personalization system to keep track of exactly this sort of preference, but vexingly, nobody takes the time to create sandwich-option-preference profiles for themselves.  

Another month of arguing and thought produces the clever idea of automatically generating preference profiles for people based on their previous orders, which results in a lot more profiles, but the unfortunate feedback effect that screwing something up in your first order applies it to your second as a default, and the number of disgusting sandwiches is not going down fast enough. The customer, who spent years efficiently making non-disgusting sandwiches with no more technology than an order pad and a pen, is about ready to join the Luddites, even if they do want mayonnaise on their PB&J.  

If the programmer and the product manager are really lucky, one of them notices, before they lose the deal, two things:  

1. The customer knows a lot about making and selling sandwiches.  

2. The system is really intended to facilitate a conversation between the customer and the customer's customers, not the end users and the program abstractions.  

The program's task is thus, more properly, to help the customer represent their sandwich expertise in some way that allows the program to give the customer's customers a sandwich-ordering-and-eating experience that is, hopefully, more accurate and efficient than the no-tech version it is supplanting. Some knowledge analysis and sweeping re-architecture later, we get something like this:  

& on

Notice that this isn't exactly the answer to "Is your blorf sterkled?"  

What Rudy (the sandwich guy's name is Rudy) knows, among other things, is that PB&J isn't a single sandwich, or a single combination of sandwich options, but a class of sandwiches involving a nut-related spread and a fruit-related spread. He has a version he likes to make, and some alternatives he knows people like. He'll make you a peanut-butter-and-mayonnaise sandwich on a spinach tortilla if you really want, but for that, for now, you have to come in and order at the counter with the rest of the freaks. The product manager keeps trying to sell him a drag-and-drop WYSIWYEat custom-sandwich-building module, but Rudy figures that if people wanted to build their own custom sandwiches, they'd build 'em in their own kitchens at home and bring 'em to work in lunch-boxes, and so wouldn't be ordering sandwiches from him. This online system isn't for people who aren't ever going to be his customers, and it isn't supposed to replace his counter. It's supposed to help his regular customers get their sandwiches without waiting in line as long, and to encourage his occasional customers to order more frequently, and to encourage his potential customers to click and try.  

The ordering system is also a menu, and it's supposed to make the sandwiches sound good. The drop-down alternatives, in fact, help demonstrate Rudy's sandwich knowledge, even to people who don't pick them. This interactive version of his menu (with similar treatment for all his other signature sandwiches) actually contains a lot more information than the one printed on the big board behind him, and thus has the potential to do a better job than the physical one of conveying not only his versatility but his expertise.  

It's also supposed to help him cope with the lunch-time crush with a little more anticipation and efficiency. His workspace out front is pretty small, and his kitchen in back is large, but if people are standing in front of you when they order, they like to see their sandwiches being made right there. Online orders not only can be turned around in 10 minutes instead of 1 minute, but they can be made back in the kitchen by Rudy's assistants, getting those out of his own way in both time and space.  

So, at the end of this long and arguably artificially obtuse case-study in design by myopic trial and error, we've come to the point where a life-improving software project should really begin: with the problem, and its aspects, and the context in which these aspects are aspects of a problem, and the characteristics that a solution in this context needs to have. Usually nobody (at least not the user, the customer, the programmer or the product manager) is going to know from the beginning exactly what the right solution should be, but everybody involved has some knowledge and some ideas and some dreams, and since those are ultimately both the cause and the goal of the project, they should be where the design process starts, and where it comes back to see how it's doing every time anybody wonders. Or, more often and even more critically, every time anybody notices that a little too much time has gone by without anybody wondering.
Site contents published by glenn mcdonald under a Creative Commons BY/NC/ND License except where otherwise noted.