True skunkworks, the programmers work without pay

(written by lawrence krubner, however indented passages are often quotes). You can contact lawrence at: lawrence@krubner.com, or follow me on Twitter.

A fantastic story about a skunkworks project at Apple, back in the early 90s:

I asked my friend Greg Robbins to help me. His contract in another division at Apple had just ended, so he told his manager that he would start reporting to me. She didn’t ask who I was and let him keep his office and badge. In turn, I told people that I was reporting to him. Since that left no managers in the loop, we had no meetings and could be extremely productive. We worked twelve hours a day, seven days a week. Greg had unlimited energy and a perfectionist’s attention to detail. He usually stayed behind closed doors programming all day, while I spent much of my time talking with other engineers. Since I had asked him to help as a personal favor, I had to keep pace with him. Thanks to an uncurtained east-facing window in my bedroom, I woke with the dawn and usually arrived ten minutes before Greg did. He would think I had been working for hours and feel obliged to work late to stay on par. I in turn felt obliged to stay as late as he did. This feedback loop created an ever-increasing spiral of productivity.

People around the Apple campus saw us all the time and assumed we belonged. Few asked who we were or what we were doing.When someone did ask me, I never lied, but relied on the power of corporate apathy. The conversations usually went like this:

Q: Do you work here?
A: No.
Q: You mean you’re a contractor?
A: Actually, no.
Q: But then who’s paying you?
A: No one.
Q: How do you live?
A: I live simply.
Q: (Incredulously) What are you doing here?!

At that point I’d give a demo and explain that the project had been canceled but that I was staying to finish it anyway. Since I had neither a mortgage nor a family, I could afford to live off savings. Most engineers at Apple had been through many canceled projects and completely understood my motivation.

Apple at that time had a strong tradition of skunkworks projects, in which engineers continued to work on canceled projects in hopes of producing demos that would inspire management to revive them. On occasion, they succeeded. One project, appropriately code-named Spectre, was canceled and restarted no fewer than five times. Engineers worked after hours on their skunkworks, in addition to working full time on their assigned projects. Greg and I, as nonemployees who had no daytime responsibilities, were merely extending this tradition to the next level.

In September, Apple Facilities tried to move people into our officially empty offices. They noticed us. The Facilities woman assumed that I had merely changed projects and had not yet moved to my new group, something that happened all the time. She asked what group I worked in, since it would be that group’s responsibility to find me space. When I told her the truth, she was not amused. She called Security, had them cancel our badges, and told us in no uncertain terms to leave the premises.

We were saved by the layoffs that began that month. Twenty percent of Apple’s fifteen thousand workers lost their jobs, but Greg and I were safe because we weren’t on the books in the first place and didn’t officially exist. Afterwards, there were plenty of empty offices. We found two and started sneaking into the building every day, waiting out in front for real employees to arrive and casually tailgating them through the door. Lots of people knew us and no one asked questions, since we wore our old badges as decoys.

We were making great progress, but we couldn’t get it done alone. Creating sophisticated software requires a team effort. One person can use smoke and mirrors to make a demo that dazzles an audience. But shipping that to a million customers will expose its flaws and leave everyone looking bad. It is a cliche in our business that the first 90 percent of the work is easy, the second 90 percent wears you down, and the last 90 percent – the attention to detail – makes a good product. Making software that is simultaneously easy to learn, easy to use, friendly, useful, and powerful takes people with an incredible combination of skills, talent, and artistry working together with intensity and patience. Greg and I could do the core engineering, but that was a far cry from creating a finished product.

Among other things, we needed professional quality assurance (QA), the difficult and time-consuming testing that would show us the design flaws and implementation bugs we couldn’t see in our own work. Out of nowhere, two QA guys we had never met approached us, having heard about our venture through the rumor mill. (We had become a kind of underground cause célèbre.) Their day job, QA-ing system software, was mind-numbingly boring. They volunteered to help us, saying, “Let’s not tell our boss about this, OK?” One guy had a Ph.D. in mathematics; the other had previously written mathematical software himself. They were a godsend. They started right away.

Next, we needed help writing software to draw the three-dimensional images that our software produced. A friend with expertise in this area took a weekend off from his startup company to write all of this software. He did in two days what would have taken me a month.

My skunkworks project was beginning to look real with help from these professionals as well as others in graphic design, documentation, programming, mathematics, and user interface. The secret to programming is not intelligence, though of course that helps. It is not hard work or experience, though they help, too. The secret to programming is having smart friends.

There was one last pressing question: How could we get this thing included with the system software when the new machines shipped? The thought that we might fail to do this terrified me far more than the possibility of criminal prosecution for trespass. All the sweat that Greg and I had put in, all the clandestine aid from the friends, acquaintances, and strangers on whom I had shamelessly imposed, all the donations of time, expertise, hardware, soft drinks, and junk food would be wasted.

Once again, my sanity was saved by the kindness of a stranger. At 2:00 one morning, a visitor appeared in my office: the engineer responsible for making the PowerPC system disk master. He explained things this way: “Apple is a hardware company. There are factories far away building Apple computers. One of the final steps of their assembly line is to copy all of the system software from the ‘Golden Master’ hard disk onto each computer’s hard disk. I create the Golden Master and FedEx it to the manufacturing plant. In a very real and pragmatic sense, I decide what software does and does not ship.” He told me that if I gave him our software the day before the production run began, it could appear on the Golden Master disk. Then, before anyone realized it was there, thirty thousand units with our software on the disks would be boxed in a warehouse. (In retrospect, he may have been joking. But we didn’t know that, so it allowed us to move forward with confidence.)

Once we had a plausible way to ship, Apple became the ideal work environment. Every engineer we knew was willing to help us. We got resources that would never have been available to us had we been on the payroll. For example, at that time only about two hundred PowerPC chips existed in the world. Most of those at Apple were being used by the hardware design engineers. Only a few dozen coveted PowerPC machines were even available in System Software for people working on the operating system. We had two. Engineers would come to our offices at midnight and practically slip machines under the door. One said, “Officially, this machine doesn’t exist, you didn’t get it from me, and I don’t know you. Make sure it doesn’t leave the building.”

In October, when we thought we were almost finished, engineers who had been helping us had me demonstrate our software to their managers. A dozen people packed into my office. I didn’t expect their support, but I felt obliged to make a good-faith effort to go through their official channels. I gave a twenty-minute demonstration, eliciting “oohs” and “ahhs.” Afterward, they asked, “Who do you report to? What group are you in? Why haven’t we seen this earlier?” I explained that I had been sneaking into the building and that the project didn’t exist. They laughed, until they realized I was serious. Then they told me, “Don’t repeat this story.”

Post external references

  1. 1
    http://www.pacifict.com/Story/
Source