-11583

195818

Commentary: The Use of TeX in a Commercial Environment

	Title	Ness Original	David Ness
	A Long View	The experiment presented here was terminated not due to any particular success or failure on its part, but rather because Ambassador Annenberg sold TVGuide to Rupert Murdoch. Murdoch's publications already had some computer systems and their own way of looking at things, and his staff decided that their pre-existing systems would form the basis of future developments. I ended my relationship with TVGuide around that time, and so I am not an expert in the details of how things really went on from that point forward. It is my impression that this change in direction proved to be both enormously costly and innefective, but even if that is true, it is unlikely that it had much to do with technical matters, as political matters proved to be much more important in determining the course of final outcomes.
	Introduction	Using TeX in a commercial environment involves both pain and pleasure. This paper describes some of both. It concludes that while the pleasure outweighs the pain as is, the pain could still be reduced significantly.	I stand by this statement. Although time did not allow us to complete our experiment, I remain convinced that this is true.
	Purpose	We have been using TeX in a `commercial' environment for nearly three years. We have also been experimenting with MetaFont for a little less than one year. The purpose of this note is to record some of both the rewards and frustrations that we have encountered during this experiment.	It should be noted, I suppose, that the primary audience for this paper was people who were already veryTeX.
	What Led to the Experiment	At TVG we render a lot of type. We print in the neighborhood of 1,000,000,000 (one billion) copies of our magazine each year. During the year we render about 1,000,000 (one million) different pages, and bind them into 5,000 (five thousand) different 200 page magazines.	These numbers are still quite staggering. While Reader's Digest---which people surprisingly often mention when they mean TVGuide---had a larger single issue sale, it TVGuide was published 52 times each year rather than 12, and so it presented more page-impressions.
	Producing Output	The job of producing this output requires a complex coordination of both human and automated processes. It is---needless to say---not a simple matter to obtain information from nearly 2,000 (two thousand) TV stations nationwide and get it laid out and printed in a timely fashion in a magazine which is delivered to nearly 20,000,000 (twenty million) American homes.	The systems involved in accomplishing this task were the result of a long evolution of processes over a substantial time period. Of course it should be remembered that this system long pre-dated the commercial utilization of small computers, so all data was processed centrally. The TeX system was developed on small computers, so the orgainzation was undergoing the changes necessary to adapt to this technology at the same time that we were developing the systems in other ways. We were fortunate to have recognized from the beginning that the typesetting business was likely to be one of those most impacted by the dramatic changes taking place in technology, and that proved---over time---to have been very much the case.
	Experimentation	In the course of experimenting with the look and readability of this magazine, we have many occasions to manipulate type. This involves setting and proofing lots and lots of different formats and styles of type. As those who have done this extensively know all too well, this can become quite expensive. Doubly so when we have to train people in particular typesetting technology only to find that either the people leave or that the typesetting technology becomes obsolete and replaced by newer and better stuff---which then requires a retraining of the staff.	It is perhaps worth noting that in 1987 printing was still a very expensive proposition. Much of this was to change in the late 1980s and early 1990s, but these changes had not taken place yet at the time of this work.
	Dimensions	Experimentation with type takes many dimensions. It is particularly useful to be able to develop page formats that test hypotheses about the legibility and readability of different page layouts and structures. Even seemingly subtle changes can have enormous consequence when you are printing a billion copies. In our situation saving an average of one line every 10 pages saves 300,000,000 pages/year.	Again, the numbers and the scale are quite staggering. While making changes was necessarily an expensive proposition, the economic consequences were often considerable.
	Adapting to an Unknown Future	It is also clear that typesetting technology is in a great state of flux. Within the last five years the cost of the equipment required to set type of the quality presented in this memo has dropped from more than $100,000 to well under $5,000. At the present time there is little sign that this dramatic rate of progress is slowing down. Both scanning and printing equipment seems to be improving at a great rate. Every day we seem to be able to do more, faster and cheaper.	Unless one lived through this period it is hard to really understand the difficulty of doing any long-range planning and design in an environment where costs were as volitile as they were in the printing business in the late 1980s.
	Future Guesses	This makes it difficult, if not impossible, to make much of a guess where things will be a few years from now. Major players in the typesetting business could well fall by the wayside as new companies come to the fore. Three years ago the `Canon Engine' was in a preliminary announcement from a ``copier company.'' Now Canon is a major player in the printing business, and have delivered hundreds of thousands of their engines under Canon, Apple, Corona, QMS, etc. trademarks.	And now there are lots of other players involved, and many of those mentioned here no longer exist.
	Going with the Flow	TeX affords us reasonable insulation from this innovation by virtually guaranteeing that we will be able to participate in the best of it. When we started our TeX project Allied wanted in excess of $10,000 for a screen preview device. Now we get higher quality from an $800 Wyse terminal (about 1000 by 1000 pixels) than we would have gotten from that older device. Today 300 dpi laser printers cost us about $1,200.	And the costs have continued to drop. The simples of micro computers that were purchased to support our initial systems development cost about $3,000. Today we could get similar equipment for well under $1,000. In addition, the systems work began before local area networks were practically feasible. While we served as beta test sites for some of this newly emerging equipment, most of our early systems depended on a `sneaker net' (i.e. people moving files from one computer to another by actually physically moving the floppies that contained the information.
	Community	Since the TeX community is so broad---and so competitive---innovation is (1) passed along quickly; and (2) major benefits of innovation are passed on to the consumers, rather than taken in profit improvements by the manufacturers.	TeX proved to be very cheap to use. On the other hand, a broad commercial user community never developed, and most innovations were pretty much restriced to academic environmments.
	Public Domain-Second Sources	One of the reasons that we decided to use TeX for our experimental environment was that the code for TeX is in the public domain. This meant that---worse comes to worse---we could always track down and innovate our own way around any particular problems which happened to arise.	This worked well, and never caused us a real problem.
	Public Availability	Further, wide public availability of TeX has produced both considerable innovation and alternative sources of supply in the TeX world. In the PC versions of TeX we are offered two widely available products, both of which appear to be very faithful to the letter and the spirit of the original version. We have made use of this, particularly in terms of product innovation, in many directions which were unanticipated at the time that we made our earliest TeX decisions.	Evolution in unanticipated directions is a basic theme of much of the innovation that took place in the typesetting business in the 1980s and 1990s. Being able to take advantage of it proved to be very valuable.
	Public Documentation	The publicly available documentation, The Texbook, The Joy of TeX, LaTeX, The PC-Tex Manual, Another Look at TeX and---of course---issues of TUGBoat provide lots of valuable information, examples and clues on how to use these powerful facilities. This is much more help than is available for most of the other typesetting systems. Beginner's Guide to TeX and the other helpful (and sometimes operating system specific) documents published by schools and manufacturers also provide a good secondary source of information.	The number of staff people that had to be trained in TeX never grew very large. Lots of people used systems components that had the central core of their work performed by TeX, but only a very limited number (less than a dozen) ever had to learn much TeX.
	Support	Finally, there is the sharing of information accomplished through TeX courses, local user support groups and---last, not least---TUG itself that help us survive some of the frustrations and equally (or more often) help us share the `fruits' of TeX labors.	On several occasions we imported some TeX experts to help us with systems development as well as training tasks. Having access to this pool of talented individuals proved to be quite valuable to us.
	Our Fonts	TVG requires some unique fonts. For example, we print the numbers of the TV channels with a little `TV screen-like' faces. Obviously we have these characters in our fonts on our traditional typesetting equipment. However, we are not very likely to find them on any proof equipment, particularly those with built-in fonts. While we could, no doubt, induce some interest on the part of some of the suppliers with a sufficiently large payment, it is much nicer to have the support of the broader TeX community to help us produce these characters. We have been able, for example, able to develop some general-purpose software to help us create some of the characters by hand. This allowed us to produce some fonts in standard resolutions at standard sizes. Now the arrival of MetaFont affords us the opportunity to develop our fonts with resolution and scale independence.	MetaFont was not anticipated in the earliest stages of our planning. And it proved to be sufficiently complicated to use that we never ended up making very much use of it, although if the project had run longer we might well have.
	Investment of Time	However, using MetaFont isn't easy, and it takes a substantial investment of time an energy before one can gain any but the simplest control over the MetaFont output. This piece of software has an unusual characteristic. One gets something important (all of the CM fonts at all reasonable scales and resolutions) for just about nothing. However, then to go beyond this seems to require a very large investment before further fruits are obtained.	I think this was a proper judgement.
	Producing Listings and Grids	We experiment a lot with different alternatives for listings and grids. There are a large number of variables associated with getting a reasonable display of type, and it sometimes takes many attempts before satisfactory results are obtained. TeX allows listings and grids to be produced very efficiently, at a low cost. More importantly, it reduces the turn-around time for producing type from what was often days to a few minutes.	Grids are now pretty much an everyday thing. However, in the 1980s, with lower resolution terminals and smaller presentation media, dealing with grids was by no means easy. We did lots of experiments, and in the end produced only very simple grids.
	Particular and Peculiar Problems	One of our toughest problems deals with the width of a column. As Knuth's beautifully typeset books show, good algorithms allow one to almost completely avoid hyphenation and ugly space expansion or shrinkage when setting material at 29 picas. However, when you are setting columns at less than 12 picas, then some ugly hyphenation or spacing is inevitable unless all of the text is actually re-written to fit the space. If we are not careful we regularly create text which is `intolerably ugly' by TeX's normal standards, and learn to become more and more tolerant of the intolerable.	TVGuide's staff was remarkably adept at `writing to fit'. There was a large file of terse movie descriptions, for example, and they had been well worked over to produce good looking type even when setting very narrow columns. It remained unclear if we were going to have to produce some innovations in TeX to allow us to control the setting of narrow columns.
	Layout	One of our major problems with TeX has to due with the difficulty of dealing with layout problems. This is not to say, of course, that TeX can't handle these problems, but rather only to observe that there are not any very direct mechanisms in TeX or any of the (so far released) macro packages AmSTeX or LaTeX to deal with layout problems.	TeX never did a particularly good job with layout. Indeed some layout problems were extremely frustrating (bad line breaks, widows, orphans, ...).
	Interesting Problems	There seem to be some intellectually interesting problems associated with layout, and I remain hopeful that some members of the TeX community will find themselves attracted to the problem. Since TeX has such substantial capability for typesetting, it is a shame not to be able to use it easily in the layout context where it might be so helpful.	The interests of the TeX User's Group took a different direction. A great deal of effort was invested in developing specifications for color---something not particularly important to us in the TVGuide context. And then there were some fractious (political, not technical) battles in the User's Group that caused many of us to depart as well as shrinking the User Community quite dramatically.
	Re-Encountering Text	We occasionally also need some relatively straightforward way to re-encounter some text after making some placement decisions. For example, in magazine typesetting it is not unusual to have pages that are set two-up followed by pages which are set three-up. When TeX encounters such a configuration, it has a tendency to complete the setting of a full paragraph at the width appropriate for two-up pages before being willing to adjust its hsize to the width appropriate for three-up.	This was part-and-parcel of TeX disinterest in dealing with layout in any substantial way. Layout in books, Knuth's principal interest, was generally a much more straightforward task than the kinds of complicated layouts that are often found in magazines.
	Badness	It is clear that imposing badness penalties across a break of this kind is conceptually quite a difficult matter. It would be difficult coding, indeed, to define---much less implement---a notion of `best' setting across such a two/three split. However, in general we are only interested in `good' solutions to this problem; we need not worry if the `best' eludes us. In most cases we would be quite happy to let TeX decide on the paragraph break with its conventional methods, if that could then be followed by re-setting the type after the break at the new width.	Shifting width layouts are quite common in magazines, and TeX didn't make handling them easy.
	Promising Suggestions	We have found it difficult to grapple with this problem, although a letter from Knuth contains some promising suggestions. We are trying to understand the full implications of his suggestions for this problem, and it isn't easy. However, with help we seem to be making some progress.	Knuth, as well as others, were extremely helpful and responsive when our problems were interesting.
	TeX, Programs and Documentation	The relationship between TeX and (computer) programming activities first occurred when we began to use TeX to prepare our program documentation. It proved to be very helpful when we could use typewriter-like fonts in the documents to highlight input and output displays. Once TeX became a normal part of the working environment it was natural to extend its capabilities to helping us with the actual display of programs themselves. We developed the CTEX and INDEX programs that produce very nice listings of our source code. This has allowed us to change our standards for code documentation without requiring a real `revolution' in the way that we write code---something that would be required if we were to convert to CWEB, or any similar environment.	We not only experimented with using TeX to set type in the body of the magazine, we also used it to produce programming documents. Had the project gone on further, I believe we would have done even more of this, perhaps eventaully converting our efforts to WEB.
	General Documentation	We also have incorporated the documentation associated with the programs into a more general documentation system. This system allows us to keep track of documents, their version, their status, etc. One of the nice things about TeX is that it is easy to incorporate as a part of a process which finds the document source, unarchives it, typesets it and then renders it on a typesetter or laser printer.	We also used TeX to help us manage our proggramming projects and other technical efforts that were in progress.
	Independence of Physical Size	We also make use of the fact that TeX allows us to easily describe and set documents that are independent of their final physical size. Sometimes 8 1/2in by 11in documents are convenient. However, often the new computer standard of 5 1/2in by 8 1/2in proves to be desirable. TeX conveniently allows us to largely separate the task of creating and maintaining the document from the tasks associated with rendering it.	When we use TeX for more than just producing output that is to appear in our magazine, then this aspect becomes particularly important. In typesetting documentation, for example, it can be quite useful to be able to adjust to other media sizes.
	Users and Code	We are also running an interesting experiment in having users read code. This is part of a general attempt to get our programmers to indicate their intent in their program documentation. Since the majority of our users are editorial professionals, we get lots of helpful (and painful) criticism of the English that we use in documenting things.	This particular part of our experiment had quite a successful `beginning', but we it did not last for long enough for us to really be able to evaluate the prospects for long-range success.
	De-Tuning TeX to Match Conventional Typesetters	We also have an interesting situation where the high quality of type set by TeX gets in our way. In some circumstances we want to set the equivalent of galleys that are to be cut to specific lengths later in the composition process. Conventional typesetting technology allows this in a rather simple way. We typeset the longest text and then simply observe whether shorter renderings of the text are legal i.e., they don't produce widows, for example) by pretending to `cut' the text to shorter fragments. When we go to final typesetting, we are able to tell the typesetter to set type to the appropriate length, and we know exactly what we get.	One of the great plusses of TeX is the high quality of the output that it produces. However, this brings along one of the minuses---in particular that we give up a considerable amount of the control over the typesetting process. The unique properties of integraing TeX typesetting into a system process causes this negative to be important in our rather peculiar circumstances.
	Mimic Typesetters	If we cut a TeX paragraph, however, TeX will re-set (perhaps all of) the lines of the text in a different fashion. TeX may well decide to move some words up or down from line to line to get a better looking paragraph now that the text is known to be shorter. When we want to mimic conventional (forward only) typesetters this is not desirable.	The broader circumstances under which we set type make it possible that this is important. We never developed effective means of controlling TeX in this regard.
	Control of Cuts	What we need to be able to handle this is an ability to de-tune the TeX algorithms so that it doesn't consider flowing the text back up into the paragraph after the cutting decisions are made.
	Democratization of Typesetting---Decentralization Issues	TeX allows us to both `democratize' and `distribute' typesetting tasks. This represents a substantial revolution in the process of getting ink committed to paper.	When the TeX project started at TVGuide all of the actual typesetting was performed by a Data General machine tucked away in an obscure part of the main office. The availability of TeX in combination with the availability of personal computers, brought the prospect of typesetting activities to every desktop. Similarly, the taks of producing one or two hundred different editions of TVGuide each week, while demanding in the aggregated, proved to be quite within the capabilities of the technology when divided and distributed down to separate machines.
	Wide Community of Users	First, TeX allows us to put typesetting capabilities into the hands of a wide community of possible users. These users can range in sophistication from people who know little or nothing of typesetting to old-line typesetters who understand the business quite well. There are many levels of TeX which correspond to this. One can know no more than to just put `funny looking' things (with backslashes and the like) into the text, and still find TeX quite useful.	While this did, indeed, prove to be quite possible, it never proved---at least for the the duration of our experiment---to be very important. Again, whether more time would have made it so remains an untested case.
	Individual Users	Second, and perhaps even more important, TeX allows us to distribute the typesetting capability down to individual users. We are economically able to put the ability to look at type into the hands of virtually every user who has a PC at their disposal. This allows us to distribute typesetting capability to each of our remote offices, printing sites and anywhere else that it might prove to be useful.
	Separation of Form and Content	If there is a big point to be made by adopting the TeX environment, it is this: Separation of form from content. Very unfortunately today's trends in typesetting are running precisely contrary to this. There is a naive pull toward WYSIWYG typesetting that completely integrates the content into the form. For our purposes---and indeed, I think most other purposes as well---this is clearly an error. We very definitely want the content of what we do to be quite independent of the form in which it will finally be rendered. Only in this way can we defer decisions about commitment to ink and actual type as long as is reasonably possible in the publishing process.
	Separation	One can only hope that it will not take too terribly long to realize that---for the most part---we should provide ourselves with an environment that allows us to separate worries about the form from worries about the content. TeX makes this not only possible---it makes it quite convenient.
	The Arguments Against TeX	In our environment TeX encounters a lot of resistance. As with many other great ideas this is no surprise. Let's treat the comments and their likely background and cause one-by-one.	TeX always had its `detractors'. Some of them were well informed people, but most weren't.
		For Mathematics: TeX is for mathematics. I guess TeX it is for mathematics. The other day I set six pages of beautiful notes from a calculus class that I took in my undergraduate days. However, it is simply poor reasoning to think that because TeX it is for mathematics, it isn't for something else. It works pretty well in lots of different environments.
		For Books: TeX is for books. TeX also it is for books. I guess it is a fair observation to suggest that Knuth has paid more attention, at least in print, to considerations relevant to producing books than for other varieties of typesetting. Perhaps if he had been principally interested in typesetting articles for the popular magazines, we might have an emphasis on other facilities in TeX. However, it is my feeling that the complexity of many of the problems faced in book production assure us that comfortable solutions associated with magazine production are likely to evolve.
		Complicated: TeX is too complicated. TeX is too complicated. But again, putting ink on paper is an extremely complicated process if one wants to be able to control it both exactly and with considerable generality. Each time that I learn more about how to use TeX in some complicated situation I become more convinced that the complications are necessary, and indeed desirable. The alternative would be to give up the control that is actually so important.
		Too Cheap: TeX is too cheap. It may seem, at first, that this is an absurd objection. Of course, it is never stated in these terms. Yet, quite often it is the case that people with a background in typesetting simply cannot believe that it is possible to set such good type so cheaply. I am reminded of the fact that in the 1940s Toni Home Permanents didn't sell when they were first test marketed at 25 cents. When the price was raised to $2, they sold like hot-cakes. People who have deep commitments to old-time typesetting technology don't want to believe that it is possible to set acceptable type, with great reliability, on 300 dpi laser printers generating your own fonts for a combined hardware/software cost of less than $5,000.
		Specialists: TeX requires specialists. Again, TeX requires some specialists, but actually it is less demanding on our environment than many of the conventional typesetting machines. What TeX allows, in direct contrast to more conventional typesetting technology, is for different levels of understanding to be effective for different users.
	Not Here	Thus, these first few objections to TeX do not seem to be very substantial in our environment. We continue to encounter these arguments, however, and it will probably be a very long time until they are no longer brought forward. However, there are a few additional arguments that should be at least discussed.
		Ugly: TeX source is too ugly. This does seem to me to be a valid criticism of TeX. For people who so obviously care so deeply about the appearance and quality of all output, it strikes me as odd to have such an ugly input language. While TeX is obviously quite logical in its structure, it is often difficult to make input read easily or look nice. It is mindful of APL or FORTH. Spaces count some places, and don't count other places, so one has to be a bit careful about breaking lines in macro definitions, or about spacing out definitions so that they look readable.
		Hard to Debug: TeX is too hard to debug. Again, this strikes me as largely a valid criticism. Exactly which space in a macro causes a particular (unwanted) space to appear in some output can occupy hours of tracing down. Further, when a macro definition blows up, a character gets misplaced in an alignment or, (horror of horrors) we overflow our font tables or macro definition space, it can take a considerable amount of work to (1) understand what is going on; and (2) to fix it. Here, again, I have the basic impression not that this is all wrong, but rather just that it is a good deal more complicated than it really needs to be.
		Hard Language: TeX is a hard language to program in. This, too, seems to be valid. One need only look at the complexity of the definitions in LaTeX, and at the fact that LaTeX requires nearly 1/4 million characters of text, to understand that `programming' in TeX is hard work. Of all of the tasks that need to be performed, detecting and handling error conditions seems to be one of the most difficult. Programming TeX to do things is generally difficult, but nevertheless rewarding.
		Macro Packages: Of course, some of the macro packages deal with this problem. However, I find that I rarely want all of the constraints of the non- plain macro packages. I would sometimes be very happy to be able to pull out the list processing capability or the token-processing parts of LaTeX, for example. The fact that TeX is hard to program in means that it is difficult to find and `lift' parts of these macro packages without absorbing them in toto. This actually makes the point about the `look' of TeX.	Again this is a difficulty that persisted, at least for the life of the TeX experiment at TVGuide.
		Need Features: Finally, TeX needs some `features'. We are about to fall into some difficulties along at least two fronts. First, and to us the most important of these areas, is that of layout. We somehow need to be able to exact more flexible control over the layout of documents than the very long \parshape commands will easily allow. We need to be able to mark flows of text, and then re-encounter the text.	Durin the duration of the TeX experiment we never really solved the layout problem to any significant degree. As far as I know, no `reencountering' facilities have ever been developed for TeX.
		Other Features: We also need to develop either some `features' or some philosophy which will help us properly define the role of .DVI vis-a-vis POSTSCRIPT, and how all of this should be related to `specials' in TeX. If we don't act reasonably quickly in this dimension, we may lose some of the considerable value gained by having had such an early and clear definition of problems and solutions by Knuth.	And not special relationship between .DVI ever developed.
	Where are we going?	We will continue to use TeX in our experimentation. Indeed, we are actually integrating some use of it into operational systems as a regular part of the process of producing some of our type. We are increasing the proportion of the type that we set via TeX, and are gradually obsoleting other technologies so that we can focus our support facilities on this particular way of producing type.	While this continued as long as the TeX project ran, some time after TVGuide was sold a decision was made to tackle the problem in a much different way, using some individually tailored equipment developed and sold by a former military documentaion supplier. This equipment proved to be very costly, and quite difficult to support. I believe that eventually the company selling the equipment went bankrupt.
	Needs	The TeX community is a robust one, but---at least so far---it has not proven to be particularly responsive to the needs of those of us who daily operate in a commercial environment where deadlines are the rule of the day. The TeX community is academic---in both the best and worst senses of that word. TeX help is truly priceless, often given freely and quite wonderfully, but sometimes not available at all. Academic concern, quite reasonably, is more often guided by interest than by the thought of reward. This sometimes means that we can get quite difficult problems solved for free if they are interesting, but often can't get simple problems solved for any price if they are boring.	As time passed, this never got much better.
	Academic	As a simple example, information that circulates quite freely in the academic community is not necessarily available to commercial users. A lot of TeX related information is shared in formal and informal networks that most commercial organizations are not a part of. Thus we are outside of the mainstream.	In addition, while many universities made use of TeX in their academic publications, few ever graduated many people with substantial TeX expertise, and thus no effective talent pool of hierable people developed. Whether this was due to lack of supply, lack of demand, or a combination of the two is difficult to say.
	Commercial	Commercial organizations often need to be able to `count on' things a little more. That's the bad part of using TeX in a commercial environment. Generally, we can also afford to pay for them. That's the other side of the coin.	A few small organizations developed that could support TeX efforts. There was also a group of about a dozen consultants that had reasonably active careers supporting TeX work. But no large community ever seemed to develop.
	Epilogue	It is interesting to speculate why TeX never got further in commercial environments. Perhaps it was effectively swept over by the arrival of HTML and the concamitant emphasis on screen presentation, as opposed to printing, technology. It may also be that the ubiquity of PostScript caused the problem of supporting technolgy on widely different platforms to be be solved in the basic hardware rather than through elaborate coordination of software systems. And the arrival of color screens with wide variation in size may have contributed to the eventual ebbing of interest. Suffice it to say, that the TeX experiment at TVGuide was of considerable interest for better than a decade.