Gathering People To Write An Operating System:
An Early Exercise in Software and Social Engineering

Jim Minton
Edited by Donna Mecozzi


There is an awful lot that one can say on the topic of writing an operating system, and, over the years, an awful lot has been said. However, it seems that most discussions on this topic are from a theoretical point of view and are written by managers who are looking down on a particular project from the 30,000-foot level. While the view from this altitude is often quite nice and is often quite interesting, I�d like to share my view, which was from a very different perspective�from the bottom, looking up.

When I was asked to write about The Development of an Operating System, I, of course, wondered where to start? There are many areas that I�d like to discuss, and, perhaps someday I�ll be able to discuss them all. However, in this note I decided to choose just one item from the list: I decided to discuss the attitudes of the programmers on the project and the attitudes of programmers, in general, at that time.

Work on NLTSS began in 1979 and I was one of the first programmers hired to work on the new project. I wrote the first line of code for NLTSS and when the project ended a decade later, I had written the last line of code. Meanwhile, several others made significant contributions to the code between these two lines.

To fully understand and appreciate the attitudes of LLNL programmers at that time, a brief description of the environment we lived in is in order. As has been mentioned many times and in many other papers, all software needed at LLNL was developed at LLNL. Now we are not just talking about the big application programs that were developed, we are talking about all software�every darn bit of it! Compilers, linkers, text editors, all utilities, and, of course, the operating systems. Much of the development of this software was done in isolation and over a period of years�someone somewhere would be frustrated by not having a tool that would make a copy of a file and so that person would write a program that would copy files. As you would expect, several other people by this time would have also written a program to copy files. As time passed, efforts were made to prevent this duplication of effort and the �best� of these efforts were collected into a �public� place where everyone could access them. And, because it was difficult to agree on which was best, it was often the case that all such programs were offered in the public place. There would be on-going debates about which ones were, indeed, the best. These debates would often hinge on minor details such as, �this version offers an option that will also set the permissions.� �Well, this version is smaller and will run 17 microseconds faster!� This assorted collection of utilities, of course, opened the door to a wide range of styles and quality. In fact, the options for at least one utility program had to be entered in German (I�m not making this up). I remember listening to this particular author brag about these options. He was quite proud of himself.

The level of competition among programmers was acute. There was a pecking order and a general acknowledgement about who was top gun. The size of a program was a huge competition issue�the smallest program was always the best! Remember, we ran on machines that, by today�s standards, were ridiculously small. The memory sizes were small and there wasn�t a heck of a lot of disk space. So, the smallest program was considered the best! People commonly wrote programs in assembly language so they would be smaller and quicker. There were legendary programmers who wrote assembly language programs that were self-modifying�as these programs ran; they changed their own instructions. This self-mutilation was done to take advantage of other instructions that were in that area that could thereby be reused. One programmer changed all of the error message text strings in a utility program to hexadecimal in order to make the program smaller. Not only did no one condemn this insane behavior, this behavior was encouraged and the programmer was given a pat on the back. �Good job! Your program is the smallest yet.�

You see, because of the limited computer resources available at that time, it was good to keep programs small. However, most (90%) of the size savings were easily accomplished using standard, simple practices. And this was almost always sufficient. Unfortunately, it was the last 10% that separated the men from the boys. At tremendous cost and effort, the remaining 10% was attacked. Months and years were spent squeezing programs down until they would �fit into a matchbox�. There was really no need to squeeze these programs any further. The only reason people continued to squeeze them was so they could announce that their program was squeezed to the limit and no one on the planet Earth could squeeze it any further! It was a totally macho thing.

Program speed was another competitive issue. �My program runs faster than your program.� �I can program this loop to run 7 microseconds faster than your loop.� Programmers studied the hardware and studied individual instruction sets to coax one more microsecond from a section of code. No one said anything about this. This is simply what programmers did. This is what was expected and encouraged. If any manager wanted to say anything, to my knowledge, they never did. In fact, it was sort of a feather in their cap to have a programmer that could whip another manager�s programmer. I�ve listened to groups of programmers argue over the fastest way to do a certain thing and then seen one of them show up triumphantly the next morning proclaiming that �to do it your way, you would have to insert a no-op after the third instruction! Doing that will cause another machine cycle, so my way is fastest!� The group would hoot and holler and congratulate the winner. The loser was consoled and jokingly chided, but it was usually apparent that the loser was pretty unhappy. And then there was the time a physicist wrote a loop that ran a bit faster than the programmer�s loop that had been proclaimed to be the best. This was a horrific insult and embarrassment of the first order. The damn physicists could order us around and treat us like dogs, but damn it, we were supposed to be the masters of writing code! The news of this major incident swept across the Laboratory and shook the faith and confidence of many programmers and, in addition, caused many programmers to vow to try even harder.

And because of this competition we naturally had to have �secret code.� Now, at the Laboratory, because of the nature our work, many of our codes are considered to be national secrets. However, during this particular time period, most programs were treated with even more secrecy than was required by the Department of Energy. Understand that code was kept secret because one programmer didn�t want another programmer to see their famous fast loop. If code was openly available, another programmer could copy it or improve it and gain the coveted status of king programmer. Also, no programmer wanted another programmer to see their coding style because this could expose them to criticism and, in addition, the looker could tell others about your crummy coding style. And, in addition, programs were a way to collect power�you need my program, therefore you need me. No one else knows how this program works and I�ll make sure that no one ever does. No one else has been allowed to even see it and even if they do, they�ll never be able to decipher it. And, after looking at some of these programs, I am convinced that some of them were purposefully made obscure to enhance the programmer�s job security. No one else can read it or make sense of it, so you will have to keep me and continue to need me.

There were informal, but well known, rules of behavior regarding the handling and viewing of program listings. To see a copy of your program, you had to send it to the big central printer. You could, of course, see a �portion� of your program by using your keyboard and television monitor, but to see the entire program, you sent it to the big central printer. By the way, the big central printer was a wonderful thing that deserves its own story.[1] It was a huge thing that looked like a small steam engine, only it was about 10 times more powerful. And when they shoveled enough coal into the boilers to get it up to speed, the floor would shake! It was a wonderful thing! The common practice was to send a copy of your program to the printer right before leaving for home. This would, hopefully, insure that a copy was ready the next morning. After arriving the next morning, each programmer trekked down to the printer room to pick up the latest and greatest copy of their �listing.� Just in case you don�t know, a �listing� is different from a copy of the source code. A listing is a piece of output produced by the compiler. In addition to the source code, it contains machine addresses, assembly code instructions, cross-references, and some other stuff. Each programmer picked up their listings and trekked back to their offices. Listings were often carried along under the programmer�s arm as they made their way throughout the day. Despite what the Laboratory might think about the ownership of these programs, each programmer thought of themselves as the program�s one and only owner and clutched these listings very tightly to their chests. If a programmer left a listing on their desk, it would be considered a major breach of etiquette to look at it or leaf through it. They would much rather you rifle through their underwear drawer or medicine cabinet than rifle through their listing! �Have you no decency? Were you raised by demons? How dare you!�

In this environment, almost everyone worked alone. This is sad, but true. If you were having problems, it was Ok to ask general questions about the problem, but it was very rare that anyone actually looked at your code. And this was by choice!

And what was sacrificed by all of this behavior? Code readability, that�s what! No one could read anyone else�s code. It was often literally impossible for anyone to make heads-or-tails of anyone else�s work! It was not at all unusual for someone to retire and have their 20 years of programming work be passed on to the new guy, who would work for two years trying to decipher it, and then throw the whole thing into the trash and start an effort to rewrite the entire program from scratch. And, years later, when the new guy left, the cycle would repeat.

However, because of the group I worked in, I, and the others in my group, were exceptions to this behavior.

I worked in the utility routines and libraries group. We were certainly aware of the Laboratory customs regarding programmers and their expected behavior. However, almost all, if not all, of our code was kept in the public area and was therefore available to everyone. I was aware of the power of being needed, but I wasn�t given much opportunity to hide anything and thereby gain power. This situation was unusual. On the positive side, I�ve had people phone me to tell me that at line number 456 in a certain file I should change the value from +1 to �1. On the negative side, I had people scrutinize my work and then give me grief.

For example, I maintained a little utility routine named �switch� that was used the change the names of files, and change the attributes of files. One day an important applications programmer, after taking a cursory look at the switch code, and noting its length, loudly asked, �What the hell is the matter with those people?� �Those people� being me. You see, any program that performed such a trivial task should be easily written in 100 lines of code or less, not the 1000+ that was required for the utility routine switch. In those days, about the only option left for me was to cringe.

So, this is the backdrop against which we began forming the new operating systems group. Interviews were held, various people were selected, and we began gathering together to discuss our charter and our mission. The original crew in January of 1979 were:



A couple of months later we were joined by:



Over the years people came and left. Others in the group at times included:


There are many stories to tell about these early days and about the contributions made by each of these people. Someday I hope to tell them. However, let�s continue our tale about the attitudes and the decisions made by the people from this era.

There was an early decision that hugely influenced our coding styles and our attitudes. This was the decision to use the Model programming language to write our new operating system. For those of you who may not know, the Model language was developed at Los Alamos, largely for use by the writers of the DEMOS operating system, which was being developed there. Model was an early adopter of object oriented concepts, and supported abstract data types, overloading, methods (not called that in Model), etc. People are often surprised to learn that these concepts were available and being used in the late 1970s.

At the time we were making our decision to use Model, we had no idea how much this decision would influence how we did our work. And like many of our other early decisions, this one caused blood to flow. There was a great deal of shouting and arguing and then each of us was asked to cast a vote�did we want to use Model or did we wish to stick with our old standby, LRLTran (an enhanced version of FORTRAN). Jed, Chuck, Bob, and I voted to use Model.

Pete immediately quit the project.

Pete said that he wouldn�t lead a project that did such a thing. He said that if we all disagreed with him that we should get another leader. He was very opposed to using Model because it was new, and it certainly suffered from its association with DEMOS[2] at LANL.

Pete�s feelings about Model were certainly well founded, but we needed to have Pete as the head of the NLTSS project. Pete led the previous OS group, and had led the OS development efforts on the newly departed CDC Star computers. Pete had credibility and was well respected. In addition, we needed Pete�s toughness and determination. So, we had to come to a compromise. Like most compromises made in software development, this one should never have been made. Nevertheless, we agreed that each of us would be free to program in the language of our choice�Pete would program in LRLTran and the rest of us would write in Model. This aberration was made possible by passing everything back-and-forth through common blocks between the two languages. This worked after a fashion, but as the years passed and attitudes and tensions eased, more and more of the LRLTran code was converted to Model. Pete himself ended up writing much Model code, and, although he mumbled and grumbled as he did so, most of us think that the mumbling and grumbling was just for show. He would never be seen to be admitting to any advantages to Model.

Later, when Pete left to accept a division leader position, and we began using threads to control our inter-process concurrency, we had to completely eliminate the remainder of this LRLTran code. To be sure, there were still sections of the OS (mostly the drivers) that were written in LRLTran and assembly language, however all of the �upper level� code was eventually written in Model.

However, I am getting ahead of myself. This is supposed to be a story about programmer attitudes during the development of an operating system.

At the beginning of the project, shortly after the great compromise, assignments were made and we all set out to work. I still have the hand-written sheet of paper showing each of our assignments. My assignments were



I had no idea what the Report Server was. I think that Pete was trying to come up with a name for the server that would manage and produce accounting data. At any rate we later produced an Account Server. (The story of the Account Server is, again, another very interesting tale in itself and I, again, have to resist the temptation to go off on another tangent.)

Up until this time, I had been fostered and schooled in the Lab�s closed, work-alone environment. However two events conspired to forever change all of my programming attitudes�I was put into the same office with Jed, and Donna moved into the office next door. If ever there was a person who was the opposite of �closed�, it was Jed. It�s not much of an exaggeration to say that Jed felt that everything should be shared. Why in the world would you not want someone else to see your code? Jed would pick up my or anyone else�s code and look at it as if he owned it. And then along comes Donna, fresh out of school and does not know �proper� Lab behavior. Donna came from an academic environment where attitudes are much more open and free. At DAS,[3] which heavily emphasized understandability, her classes were taught by Dick Watson, John Fletcher, and Danny Nessett, each of whom obviously polluted her thinking. She didn�t even know that she was supposed to work alone and jealously protect her code.

Donna and I ended up working together for years, and, with Jed, we helped each other write code, maintained each other�s code, modified each other�s code, and inspected each other�s code. And, because we were the original members of the project, our attitudes, styles, and ways rubbed off on the other members.

It is important for me to emphasize that there was something more going on here than just working on each other�s code. We probably didn�t realize it at the time, but we were becoming what others would later call a team! A closely knit team. A concept heretofore unknown in the laboratory�s computer department. A team that helped each other. We were a group that placed the success of our project above our personal objectives. Each of us helped our team members because it was the best thing for the project. (Actually, it wasn�t all this mushy and huggy, but you get the point). As I recall, the compiler group was also a very close-knit group.

Let me tell you about a completely unintended, but wonderful consequence of choosing the Model programming language. This is directly related to our attitudes and our openness. (No one could have seen this coming!)

Model was a completely new programming language here at LLNL. Donna had done some programming in Pascal in school but, otherwise, it was completely new to all of us. Consequently none of us had any preconceived ideas or notions about how to write in this language. We were able to look at some of the code that had been written at LANL, and we each had a copy of the Model Programming Language manual, but other than that, Model was completely new and it was up to us to develop some kind of programming style.

After Jeff Clark left, Donna and I were asked to try to pick up his work. We had asked Jeff to try to develop some sort of caching scheme that could be shared by multiple servers; we discovered that he hadn�t gotten very far on this project.

And here is where it all began.

Donna and I began working together to develop this library of routines that would later be called �Cachelib.� This work was urgently needed, so we dropped everything else and committed ourselves to this effort. We met with Jed to get ideas, opinions, and thoughts, and then we converted these ideas, opinions, and thoughts to code. Pam calls this �white paper programming.� Cachelib became the cornerstone of all the NLTSS servers. Donna and I both developed this code, and this code had to fit together. Not only did the indenting levels have to match, the �repeat�, �while�, and �for� loops also had to match. Everything had to match. We fought like cats and dogs during every inch of this process. I insisted on one indenting level and she insisted on another. I wanted to write �repeat� loops and she wanted to write �while� loops. I insisted on putting a semi-colon after the last statement before a closing brace. She wouldn�t do it. However we had to compromise for the good of the project, so we ended up doing everything her way. (This may be a bit of an exaggeration. Donna , in fact, says that it is an �extreme exaggeration�!) The bottom line is that we developed a Model language programming/coding style. And this style became the default NLTSS coding style. And as others joined the team, they adopted this style.

There was additional motivation for adopting this coding style�all servers had some sections of code that were similar. The token parser is an example of such a piece of code. Donna and I had, early on, written the canonical token parser and everyone who started a new server or utility would begin with this piece of code. And since this piece of code was written in a particular style, the programmer would be motivated to continue writing in that style.

Consequently, almost all of NLTSS was written in the same coding style.

This turned out to be a huge advantage. Everyone could pick up any piece of NLTSS code and it would look exactly like their code. The code looked as though it had been written by one person. No longer did people have to contend with foreign looking or strange code. It all looked the same. It always looked like your code. You were always right at home in anyone�s code. I hope people appreciate how important this was.

Because we all liked the feeling of being at home anywhere in the code. With few exceptions, we all perpetuated this by trying to adhere strictly to the coding style. This was truly openness on an unprecedented scale.

As newcomers joined our team, they were swept up and overwhelmed by our openness and our desire to have a really strong team. And we truly did mesh as a team. Some of this, of course, was partly just by luck and chance, but one thing is clear: We maximized the strengths of one another and compensated for one another�s weaknesses, and by doing so, we made a fantastic team! When someone came in unable or unwilling to let go of their ego or tried to cover their weaknesses, they just didn't fit in and, eventually, they left. Jed was our �idea man� and our General. He fought the diplomatic battles with other generals, and he fought hard to make sure that we soldiers weren�t doing unnecessary or dirty work. Donna and I were soldiers, down in the trenches, working with the code. A big part of what set the tone for the coding practices was the way we related and interacted with one another. There was no set formula.

So, you see, it is possible to teach an old dog new tricks. If you whack him enough times with a big enough bone, he eventually gets the idea. Many of the NLTSS programmers who were started in the closed Lab environment changed their attitudes. And, of course, there were some that didn�t. Whether they changed or not is not important. This is simply a story about the attitudes of a team of programmers who got together many years ago to write an operating system.



Editor's Notes:

[1] The keyboard was typically, an ASR 33 Teletype, and the big central printer, the Radiation Printer, is described in another essay in this collection.

[2] DEMOS was an operating system developed at the Los Alamos National Laboratory (LANL.) Initially it was deemed to be slow and because of strenuous user objections, was abandoned rather than being allowed to mature.

[3] DAS is the Livermore campus of the Department of Applied Science, at the University of California, Davis.