The Top of the Heap

By David Ness
Tuesday, June 11, 2002

My brother pointed out to me the other day that `Piled Higher and Deeper' was getting to be long enough that it was a pain to download if you didn't have a fast connection. So I'm going to try splitting the `Top of the Heap' from where the rest of the stuff is piled. I'll keep the index here, though.

Direct links to the top of each paper are:

The Pile

11 June 2002 Organization: Nice Work if You Can Get It

23 May 2002 There Ain't No Streets Mid-Ocean

5 May 2002 ASCII Export and Import

21 April 2002 Manhattan and the WTC

14 April 2002 The Outlook for Outlook

7 April 2002 Instant Outlines

31 March 2002 Live Blogging

11 June 2002: Organization: Nice Work if You Can Get It

If we are not careful, getting things organized can become a goal in and of itself. Nice work, if you can get it, but generally not very valuable in and of itself.

Perhaps if we can come to grips with this, we will find a way that we can understand the rather dismal history of organizers and organizing technology.

Two Stages of History

There are at least two important sweeps in the history of organizational / planning models. The first can roughly be associated with the original devolving of computers into managerial processes that is associated with the advent of time-sharing in the 1960s and 1970s.

The second stage comes with the arrival and gradual sweep of PCs into organizations and on to everyone's desks.

The Dismal Past

Perhaps we should have read it as a sign of things to come, but it took a long time to understand the facts of the situation. And longer still to appreciate their significance.

As computers became accessible, they entered the `managerial process'. The very tentative and preliminary computer experiments of the 1950s gave way to much more substantial and direct use of the technology as broader availability and then time-sharing made computational technology available to the managers that ran corporations.

It was natural to begin to use this technology to help plan activities.

Some heavy investments were made. Westinghouse built the first decision support systems in the middle 1960s, and while these systems didn't produce any really startling or provable results, the outcomes were tantalizing enough that investment continued.

And so did publication of the results. Perhaps we should have been more wary. If the results were really all that startling, would people have been so anxious to publish them? After all, why give away something that had cost a lot to develop---unless the goal was to induce others to follow down the same expensive path.

We later found out some elements of the truth. Quite often the actual operating results of running the early planning models was so inconsequential that publishing those results did not expose anything of use. And at least the publication generated some buzz that made the companies involved look suitably avant guarde.

More Recently

And now we have about two decades of serious use of PCs in organizations where they are increasingly linked into the organizational and planning processes of those organizations. And I'm afraid that the prospects for the organizational technologies do not seem to be much improved.

The products, even the best of them, continue to seem to have a rather dismal history. Many products, good products have come, and then---much to the disappointment and chagrin of substantial bodies of users---have disappeared into oblivion.

Names like Agenda and Ecco rise to prominence, only to sink back. Zoot gains a very favorable press in the mid-1990s, then disappears, is reborn and is now struggling to survive.

And this is the best of the products. The worst of them hardly ever get out of the box or on to the shelves. They often seem to die within a few weeks or months of their original announcement.

What's Going On?

So this raises the question of What is Going On? How can a problem which is so seemingly suited to computers have such trouble succeeding in an era where computers continue to get better and cheaper every day.

Observations

Let's look for some of the possible trouble spots. There are some lessons we can learn both from direct observation and from consideration of the lessons taught by some of the more perceptive philosophers in this area.

By Grace Alone ...

Grace was an acquaintance, now many years ago. I still remember well her habit of getting up early in the morning to organize her day. To do this she made lists of all of the tasks that she had to accomplish, and reorganized them into a carefully structured program of activities. The lists were elaborate, nicely written (if there were cross-outs, for example, they'd get redrafted and cleaned up quickly), and well organized.

And over the time I knew her, Grace gradually got up earlier and earlier each day to engage in this activity. It got to quite a terrifying point as she began to start her day at 4 or 5 am in order to have time to perform this increasingly elaborate ritual.

And then it dawned on me. As time passed, Grace was also doing less and less of the things on her elaborate lists. That was at least one of the reasons that the activity of creating them was getting longer and longer.

The activity of organizing had become the goal. It was no longer a means to get things done. I was, itself, a thing that had to be done and that was no longer a means. It had become the end in itself.

As Simon Said ...

This wouldn't have surprised Herb Simon. In one of the best known of his many perceptive essays Simon pointed out that structured activity drives out unstructured activity. This notion has particular applicability here.

There are two kinds of planning processes. Reasonable names for them might be

ad hoc; and
regular.

Ad hoc planning takes place generally as the problems it attempts to solve present themselves. We decide we are going to take a trip, and so we plan for the trip. We decide we need a new car, so we develop a plan on where to save enough money to be able to afford one. This is an important kind of planning, but generally it is not particularly amenable to computer enhancement, so we won't discuss it further here.

Regular planning and organizing is, on the other hand, by its very nature, generally a structured activity for many people. We may all have somewhat different rituals for performing this task, but it is safe to say that it is a ritual activity for most of the people who regularly engage in it.

For some, the task can be quite detailed. Different levels of priority may be assigned to different tasks. And some give careful consideration to the likely time that at will take to perform a task, and make commitments (perhaps only to themselves) about the date on which the tasks should be started and/or finished.

Some even go to the elaborate extent of recognizing the interconnections between tasks in an almost PERT-like diagram that shows an order and structure to activities.

And the information may be assembled in more or less formal terms depending on the proclivities of the individuals involved. For some, a simple `to do' list on a 3x5 card may suffice. Others may want color drawings of boxes and lines that describe almost a `road map' to their day.

Getting back to Simon

Let's get back to Simon's observation, though. His point was essentially a psychological one. Structured activities have a regularity and authority associated with them. By comparison, unstructured activities seem like a hodge-podge.

To make the point visual, think of the activity of processing your (computer or real) `In-Box' in comparison with trying to come up with an effective strategy for dealing with a challenging competitor. When I take a stack of 5 inches of material from my In-Box and process it to my Out-Box, I have no question that I am doing work. By contrast, when I think about how to effectively tackle my strongest competitor I may spend a whole hour and not come up with anything, not even a doodle on a piece of paper. I may have actually accomplished something highly important and useful, for example, deciding not to take several approaches which might have been tempting had I not thought them through. But it doesn't feel like work, and I don't have anything much tangible to show for it.

This is the core of Simon's point. Structured activities feel like work, and lots of unstructured activities don't. The problem is that this may be just the opposite of the importance of the activities, and it leaves us---unless we are very careful---spending much too much time and effort on unimportant structured tasks while we have the excuse to avoid the more difficult, less tangible, but more important unstructured tasks that should be occupying our time.

During my Freshman year at MIT I found a newly invigorated interest in shining my shoes for ROTC (military training) class. It didn't matter much either to me, or---thank God---to the defense posture of the United States whether my shoes were shined or not. But with effort I knew I had done something when I converted them from scuffed to highly polished. When I studied my Chemistry, or Physics or Philosophy it was never quite so clear if I really was doing anything. The structured activity of doing often overwhelmed the less structured activities of understanding complex matters.

So I had well shined shoes.

And the danger of `Organizing'

So this points us to the danger that is inherent in spending our time and energy organizing things. We get so involved in the structuring and organizing of the very activity of planning and organizing that we start to forget all about what it is we are trying to plan and organize.

The Past

All of this has contributed to an amazingly short life for most planning tools. Many organization-level tools do not survive the change of personnel associated with hiring a new round of management. Often the projects end up being early victims of the change in managerial approach and style. And this is no surprise give their questionable value and the fact that they are so intimately linked into the structure of the organization.

Added to this is the complexity of the issue of another Microsoft Giveaway. In this case designed to seize control over personal information as a part of a strategy to become an element in the distribution channel between users and all forms of information stored on the computer both local and remote.

Microsoft Giveaway

Throw into the mix the fact that Microsoft decided to give away Outlook, or at least some major segment of this software, as a part of the delivery of the basic Operating System. Outlook joined Internet Explorer as a `free' piece of software. And `free' is a tough price to compete with and make money. Just ask Netscape and Sun.

Outlook deals with the domain of planning and organizing one's life and communications on the computer. It serves as an EMail client, calendaring system and crude contact management system---attempting to bring into some common focus the various parts of the problem domain represented by personal information management.

That Microsoft's decision to give away Outlook has had some undoubted consequences on the issues associated with organizing personal information. But it would be a mistake to think that the short life of other products was completely determined by these economic effects. The facilities in Outlook are quite crude and unsophisticated, and if there was much advantage in handling any of the problems in a more rich fashion, then the costs of other software alternatives were not sufficient to provide much of an obstacle to keeping an active user community alive.

Tries at Organizing Things

`Organization' applies to many domains in life. Among the important ones for me are

Documents and Computer Files;
E-Mail;
Diaries; and
GPS Data

Perhaps a quick look at some of the potential gains in each of these areas might prove instructive.

Documents / Computer Files

Documents are increasingly stored in computer form rather than as hard copy. This makes the problem of managing documents into a problem of managing computer files. So let's treat that problem.

First, the magnitude of documents and files that need to be organized has exploded in the past few years. As gigabyte storage became cheaply available, and as terabyte storage became affordable to anybody who needed it, the number of documents and files stored has risen precipitously. As an example, I now have nearly a million separate files on my home computer net.

Photos, movies and music add huge quantities to this store. Each photo might occupy on the order of one megabyte, each CD more like 500mb and each movie something like 5gb.

Second, copying information that is stored on the computer is very simple. When a piece of paper is copied by conventional means, both paper and ink are required. When a computer file is copied only a tiny fractional amount of disk storage is required, and beyond that the copy is an exact digital copy, verifiably identical to a previous image. Under normal circumstances this information will not age or decay.

Third, costs have seen a precipitous drop. Not charging off any duplication costs, the simple piece of paper needed to copy an image might cost two or three cents. Storing an equivalent amount of information on a modern disk might cost somewhere between one one-hundredth and one one-thousandth as much.

Fourth, indexing documents and files is becoming an increasingly difficult problem. When it required a shoe box to store 200,000 bytes (on punched cards), I couldn't store enough information without filling up my house to really lose track of things. Now that I can have a million files on my home computer net, organization becomes more and more important.

EMail

There's been a similar preciptous increase in mail. Back in the 1960s, it was a busy year if I wrote 100 letters. With computers available, but prior to the growth of the Internet, I would write more like 500 letters a year. In the last 4 months of this year I have written about 200 letters a month, Granted, the modern day letters are substantially shorter on a letter-by-letter basis than those of 30 years ago, but the can be no question that current output volumes represent a very substantial increase.

Filing and organizing old letters was always a problem. Now, although the technology has improved our ability to store and cross-index information, the rapid growth in volumes has made the current situation at least as complex, and probably more complex, than ever before.

Diaries

Diaries and calendars present attractive candidates for some form of computer based organization. But even here the situation is fraught with potential difficulties. In some of our earliest personal calendaring experiments, for example, back in the 1970s, we spent a lot of time and took a lot of trouble to discover that people often kept calendars that were full of inaccuracies and, occasionally, downright lies.

Even in cases---which I wouldn't comment on here anyway---where there was no fraudulent intent, lies proved to be an important part of the calendaring process. I don't like to have to hurry back from lunch to make appointments, so I may `schedule' events to cover those particular times just to avoid other things being scheduled for those periods.

Further, calendar events aren't absolute. If I have been trying to get together with someone for a month or two, I may be willing to adapt my calendar to their availability.

GPS Data

GPS devices offer other tantalizing prospects. With a GPS I not only can get a log of my `location', but I can get a hyper accurate time log as well. And all of this can be accomplished at a very low cost both in terms of money and effort.

In my case, I use my GPS to generate a hyper-accurate detailed diary of daily activity. At the present time this is done principally for experimental purposes, but it could all be organized and made very simple if it were to prove to be useful to do so.

With this kind of information I not only get records about where I am, I get very accurate time records about how long I spend there. I also generate a lot of information about the various routes that I take driving around as I accomplish my daily activities.

Coordinating the Data

Taken together, it becomes quite possible to build a very coordinated picture of a lot of data: GPS logs, diaries, Photographic records with digital timestamps etc. When coordinated this can produce a very accurate basis for some forms of planning activities.

A Modern Fable

Maybe it's time for an example. For me, the `organization' present in my life is pretty much in inverse proportion to the importance of the activities that I have on my plate. When I am busy, it's hard to find time to organize. When I don't have work to do, I can organize to a high degree.

For example, my PDA currently contains an episode-by-episode TV schedule for the past year or so. Why? Because it happens to be a natural consequence of data that is organized into my personal diary and scheduling system. Of course, I don't have to type this in by hand---it is a natural consequence of the flow of information through other parts of my scheduling system. But processing and maintaining the information does take at least a little time, and it is time that I wouldn't cheerfully or productively spend if there were dramatically higher demands on time that could be more productively invested.

The Obstacles

There are lots of obstacles that stand in the way of the effectiveness of organizing and planning support technologies. Let's take a look at some of them.

It Works for Me

One of the worst problems involved in evaluating this kind of technology is the fact that some people will have an idiosyncratic response to virtually any new capability. There will be some It works for me testimony for just about anything. And as medical researchers know all too well, it is very difficult to separate spontaneous random effects from real and significant differences---particularly without collecting a considerable quantity of data.

Idiopathic results are very hard on us. And while it does not seem to be appropriate to deny any particular result, or to challenge the authenticity of any testimony, it would be equally wrong to simply accept such testimony on face value.

It works for me may be a perfectly valid and true statement, but it's not much a basis for belief by others.

And Then There's Hawthorne

The fact that we are likely to have some spontaneous It works for me testimony to deal with is additionally complicated by the Hawthorne Effect. This not the place for a detailed description of the Hawthorne experiment. But everyone responsible for evaluating the impact of new technologies needs to be well aware of the existence of this problem.

The Hawthorne Effect, named after the Massachusetts Plant where it was first noticed, suggests that some important quantity of improvement in the behavior of new systems may well be due to the simple fact we are changing things and making observations, quite independent of the value and/or significance of the substance of what is going on. When we change things we are paying attention to people. This very fact may cause them to improve their performance even if the changes that we made actually have no particular impact in and of themselves.

Just to emphasize the point, translate it into the current political milieu. If we change the way we run our schools, we may well see an improvement in outcomes, temporarily, even if the changes we made actually don't matter in and of themselves. We are changing things, talking about it, measuring effects and generally stirring things up. This very fact may result in improved outcomes.

Maybe you Can't be Too Rich or Too Thin ...

... but you can be Too Good Looking.

And that's something that we sometimes forget in our wish to present beautifully organized plans. There is good reason to believe that good looking reports are often selected over well thought out reports. And when this happens we are in trouble. If we are not careful it is easy to impute problem intelligence to someone who is actually only expressing presentation intelligence. And while it would be foolish to suggest that we ought to choose documents that are worse looking just because they are worse looking, we must be equally careful not to choose better looking reports just because they are better looking. In the long run we will rise or fall based on the content, not the form.

This has taught me to be at least somewhat suspicious of the `unusually good looking report'. Like extra-slick presentations in other domains, things can look too good to be true. And, as they say, if it looks too good to be true, it probably is.

You Can't Get Blood Out of a Stone

And we must remember an all too often forgotten point. If it isn't worth getting there, then all of the effort we invest in organizing how we get there is likely to be unproductive at best. This may seem like an obvious point. Yet we make this mistake all the time.

Consider, for example, technology which might make it easier to organize meetings. Do you really want to make scheduling meetings easier? If you do, then we differ. I don't like (most) meetings. And I wouldn't be happy or, I think, productive if I were to spend more time in them.

Thus any technology which might make it easier to schedule meetings is unlikely to contribute much to my productivity, not matter how may well organized it might make me feel. We need to keep a firm grip on the overarching objectives, and not give ourselves credit for accomplishment, regardless of how seemingly efficient, of objectives which aren't in and of themselves worth accomplishing.

Organizing: Avoiding the `Gotcha'

And while organizing things may mean that we will be able to avoid the blame for failure that comes from unanticipated outcomes, one should be careful in overestimating this value. There are lots of circumstances which, even if we were to have planned for them, we might nevertheless not be able to do anything useful anyway, thus rendering the effect of planning moot.

We all live with the illusion that if we had `known about it in time' or `planned for it ahead of time' then we would have been able to `do something about it'. But how often is this really the case.

All too often, planning and organizing activities are performed simply to avoid having to say `we didn't plan'. Listen to the typical government response to a crisis situation if you need evidence.

The Idiot in the Closet

In academic circles we used to talk about The Idiot in the Closet. This was the colleague who was far enough `out of it' to have few students, no consulting jobs and no extra-mural career. As a result, he did have something that few of his colleagues had: time. And what did he do with his time? Plan and organize.

So when the time came that a `plan' was needed, guess who had one? The Idiot in the Closet that's who. And he not only had the plan, he had the time for the countless meetings and the other boring and tedious stuff that made up what passed for `progress' in solving the institution's problem.

So the next time someone complains about Universities failing to plan for the future, look for the Idiot in the Closet. He's lurking somewhere. And his very existence is a sign of one of the greatest dangers of planning and organizing activities: those who can, are busy doing, while those who can't are often organizing the future.

Sink and Swim

Another effective obstacle to the effectiveness of this technology is that its success is almost never evaluated. We use organizing and planning technology, and then carefully avoid evaluating it to see if its use proved to be effective or distractive. Thus we are condemned to repeat, with each successive problem, the same set of errors and problems that we had in previous rounds. Even though we may well sink, all we do is swim again.

And all too often, whatever evaluation we might make of the various technologies are completely dominated by outcomes even if the outcome was in no way determined or even affected by the technology. A planning process is deemed to be good if the outcome it produced was good. And yet, in most real economic circumstances outcomes are materially influenced and often even determined by things which are far outside of our control and which have no relationship to the planning and organizing processes.

What Has Succeeded?

Of all the tries at supporting the planning process, the only one that seems to have developed even a modicum of general acceptance is Excel and other spread-sheet modeling languages. Starting with Visi-Calc (which, incidentally, sold a whole lot of Apples back in the early days of personal computers) and Lotus, and continuing through all of the versions of Excel to the present incarnation in today's Microsoft Office.

It is interesting to speculate on why this particular modeling technology has been so successful while so many other, seemingly more sophisticated, technologies have failed. I don't know of any careful studies of the matter. But some straightforward guesses are quite sensibly possible:

Ease of Comprehension;
Simplicity of Use;
Simplicity of Formulation;
Ubiquity

What Does Excel do right?

Simple Excel models can be comprehended on first glance. Mostly they consist of straightforward sums of numbers very much like all of the standard accounting reports that one reads everyday.

And these models `communicate' with their users with particular effectiveness. They are easy to `operate' and most often quite intuitive. For most models, no elaborate instruction book is needed.

Sidebar: Of course any technology can be pushed to the extreme, and I have certainly seen some incredibly elaborate Excel models that required detailed instructions. For the most part, these were either used in very narrow circumstances, or they did not prove to have very long lives.

The models are also easy to generate. They can be used to capture ideas and world-views quite effectively. They are simple enough that they do not typically distract users from the pursuit of their real objectives.

Finally, the technology is ubiquitous. While it would be a stretch to say that everyone has Excel, the number of people that do have access to the technolgy is huge, and it would be hard to find people in the civilized corporate world who are more than one or two people away from access to this technology, even in the rare circumstance that they did not have direct access to the technology themselves.

Try it, You Won't Like It

Another way of seeing if we are likely to get anything productive out of any of this kind of technology is to play an adult version of the child's game Let's Pretend. Instead of developing and installing a system, pretend you have already done so. Simulate having done it and then see what happens to processes and procedures. Log and accumulate notions of what is actually accomplished.

For example, if you are contemplating developing some elaborate form of calendar management, pretend you've got it. Send someone out to collect, by hand, whatever information you might expect to later have available through the auspicies of the system. Then see what happens. Are the outcomes really dramatically better? If not, should we go forward? Even if outcomes are better, are they enough better to make a difference and to be worth the costs implicit in accomplishing them.

These questions generally can, and should, be answered before proceeding, not after decisions have already been made to move forward.

Try and Try Again*

The application of technology to the problems of organization and planning seems to me to have much of the same limitations that we have already discussed in some detail in http://mywebpages.comcast.net/dness/Current/PHD.html#23May2002.

But it is not exclusive to these areas. Purveyors of this new technology remind me mostly of the Snake Oil salesmen that used to work The Circuit. Like Diet Pills, Supply-Side Economics and other things that pander to our lazy side, they offer the promise of great help with little effort. Like the rest of these technologies, they are far more likely to deliver no help at considerable cost and with the prospect for substantial disappointment.

Conclusion

Watch out for things which claim to help you get organized. In large measure these things are like the proverbial Snake Oil. Organization is fine, but it is only a way station on the road to somewhere, and we must keep firmly in mind that our goal is to reach the destination, not to congratulate ourselves for how well organized we are for the trip.

And one important thing can be done. If we practice planning and organization in order to learn how to adapt ourselves to situations when and if they arise, then there is every reason to believe we can get better and better at the task of adapting to the circumstances when the are forthcoming. Notice that this is quite different from `trying to cover all bases' and anticipate every situation which might possibly obtain. Instead we are developing a robust process, able to perform the kind of ad hoc adaptation which we are almost sure to need in a world as complex as the one we face today.

But for me, I'll continue to treat the promises of planning and organizing support technologies as just that, namely, promises. And I'll recognize that they are unlikely to dramatically influence outcomes in and of themselves. Good technology won't get in my way, and may even help me take some small steps. But the more elaborate and complex that the technology is the more likely it is to be an obstacle to outcomes not a help to accomplishing them.

David Ness' summary of work can be found at http://mywebpages.comcast.net/dness
Spending time organizing things is often a sign that we have run out of real work. And it has lots of attractive aspects: high level; structured; deals with consequential matters. So it's no wonder it sometimes becomes a replacement for real efforts.

Home