I was listening to a retrospective talk about research projects from the past. The speaker was describing the work he had done a decade earlier on video-on-demand. At that time it had been considered to be the potential big new moneymaker for the telephone companies. The technology for video storage and transport being described seemed quaint and primitive by today’s standards. Moreover, the expectations for customer behavior were proven misguided. It would have been almost considered humorous if it hadn’t been embarrassingly sad. I had been there, and I remembered, but I couldn’t recreate the feeling of those circumstances. What had we been thinking?

Before video-on-demand the big thing had been home information systems, where everything that a consumer might want in the way of information would be provided by a central server – news, weather, sports, shopping, and even transportation schedules. All of the focus was from the inside of the intelligent network looking outwards. No one gave a thought to any alternative system in which control would be distributed to the periphery of the system. In view of what actually happened with the World Wide Web and peer-to-peer networking, it is hard to imagine the mindset of those days. What were we thinking?

Lest we think that this was just the incompetence of the telecommunications industry, let me mention a parallel elsewhere. In the computer industry the focus was on mainframes. The centralized paradigm was seen as the obvious answer to providing efficient and economical computing for the masses. Even when the minicomputer broke this mold, much of the industry rejected the idea of turning over the responsibility of provision of the computing environment to the users themselves with the personal computer. Alas, what were we thinking?

The tempting answer now is to say that engineers of those days were trying to preserve the legacy business models of their employers. If that had been the case, it could have been explained as the rational response of endangered corporations. However, I was there, and I can’t remember anyone ever even mentioning such considerations. Maybe it could have been an unconscious bias, but I don’t think so. I believe that the engineers were simply trying to design optimized systems and they were blinded to uncontrollable alternatives.

I imagine an engineering discussion at a mainframe manufacturer, and a junior engineer suggests that every user be asked to buy his own small computer and maintain his own complicated, buggy operating system. After the laughter had died down, other engineers would point out the gross inefficiencies of such an approach. Look at the wasted cycles, they would say. Look at the heterogeneity, the wasted time by people everywhere, and the terrible economics. And, of course, they would have been right!

At the telephone company an engineer suggesting that users themselves actually supply the information, rather than a central authority, would have been greeted with incredulity. Look at the duplication, the unreliability, and the necessity of having thousands of uncoordinated servers, they would have objected. Totally unworkable, they would have concluded. But I don’t think such a discussion ever happened, because to my knowledge no one ever even suggested an idea like this.

I’m wondering about the lessons we should draw for the future. Why did we engineers focus almost exclusively on centrally controlled and administered systems? Perhaps some of the answer lies in our training and culture. We’re trained to provide complete, optimized solutions, and the thought of partial solutions with which a lot of other people would be allowed to tinker is unappealing. Nowhere in our calculations do we value the potential for evolution in terms of the relative number of people empowered to create modifications and improvements. Yet history has shown that distributed, user-empowered solutions are remarkably effective in dimensions other than efficiency. Today we may be seeing the same phenomenon in the rise of user-maintained WiFi systems, as opposed to the centrally-planned 3G systems.

Eric Raymond in his insightful essay on “The Cathedral and the Bazaar” has discussed a similar situation in the open software movement. Engineers like to plan cathedrals, but the world often favors a bustling open bazaar. The fact that large-scale, evolvable systems can actually pull together instead of diverging into chaos is fascinating and conceptually important. We need to understand this phenomenon and incorporate it into our planning and thinking about future systems.

So I’m wondering. Twenty years from now when people look back on the systems of today I’m sure that they’ll shake their heads in bemusement and say, “What were they thinking?” I’m sure there will be something, and in retrospect we’ll all look ridiculous. I just don’t know what it will be.

Robert Lucky