Woz is the Hero
Steve Wozniak was always my hero, compared to Steve Jobs. Here's an interview with Steve Wozniak after Steve Jobs' death. He just seems like such a totally decent person, and a great engineer. Here are some nice anecdotes.
Steve had a background working in computer stores buying stuff cheap and selling it for a lot more. I was shocked when he told me how you could buy something for 6 cents knowing he could sell it for 60 bucks. He felt that was normal and right, and I sort of didn’t. How could you do that? I was not for ripping people off. But then we started Apple and I went with the best advice which is that you should make good profit in order to grow.
I guess that typifies the difference between the two.
I was highly regarded for my engineering skills. But I never wanted money. I would have been a bad person to run a company. I wanted to be a nice guy. I wanted to make friends with everybody. Yes I came up with the idea for the personal computer but I don’t want to be known as a guy who changed the world. I want to be known as an engineer who connected chips in a really efficient way or wrote code that is unbelievable.
I think that's a nice ethos. Excellence without ruthlessness.
Lazy Minds
The subtitle for this blog is "There are Lazy Minds as Well as Lazy Bodies". This refers to a quote by Benjamin Franklin. It's a nice aphorism, but why is it important to me?
Well, laziness is a trait commonly confused with sloth. When someone refers to someone else as 'lazy', they usually mean that they aren't industrious, that they aren't willing to get up and go to work, or get a job, or 'be productive' (whatever that means). It can be dangerous, because sometimes you can squander your potential. If you're industrious, it can obscure a lot of other deficiencies. In other words, someone is physically lazy. It's obvious when someone is physically lazy, because they aren't doing anything.
But much more pernicious than having a lazy body, is having a lazy mind. It's a hidden problem. You can't tell when someone isn't doing the thinking they ought to be.
Sometimes, during the course of work, you come up against a problem. It's totally solvable, but it's also tricky. You have to mentally evaluate a set of possibilities and consider the situation from many perspectives. Imagine you're playing chess. You're thinking through some branch of possibilities, and you come up against a brick wall. You just don't want to think about it anymore, so you just make a move that doesn't have an obvious flaw... and lose a knight.
This happens a lot with race conditions when you're programming. You know basically what some of the common gotchas are. It's easy to put in a few simple checks and call it a day. And you'll be fine, 99.9% of the time (or more)! But when failure can be catastrophic, you can't just call it a day. It's easy to bang out a lot of crappy code, if you're not careful about what could go wrong.
That's one type of intellectual laziness: coming up against a barrier, knowing that it would take a lot of thought, and some more time to overcome it, throwing up your hands and saying 'fuck it!', and then papering over the problem.
Here's another type: suppose you've had a long day at work, and you get home, prepare some food, eat dinner. You're exhausted. You had a rough day. What do you do?
Well, I guess the natural thing to do is to turn on the TV, or put in a movie. It's what I frequently do, and judging by surveys, it's what the vast majority of Americans do. Your mind shuts down, and you hibernate. And then trundle off to bed. Or you could sit down, and read a book about philosophy, or do some programming, or try to figure out what's wrong with your car, or any number of things. Why don't you do that? If you think philosophy is boring, then don't read the book. Yet it could also be the case that you really enjoy philosophy, but you don't want to expend the mental energy actually thinking about it.
Have you ever watched someone watch TV? Have you ever pictured what you look like, laying there on the sofa, or sitting at the computer desk, doing absolutely nothing for hours at a time? It's repulsive. It's the temporal equivalent of soda: empty calories, empty time. Your life ticking away one second at a time.
I guess it feels as if there isn't a choice. "I just want to relax," you whine. But there's always a choice: to have a lazy mind, or to have an active mind.
Vim Modes
I've been using vim more regularly lately, and one of the most frustrating things about is that it can be difficult to visually tell what mode you're in (insert or command), leading to lots of wasted keystrokes Esc-ing or hitting 'i', backspace, cursing, etc. Adding the following to your .vimrc makes it much more distinctive:
set laststatus=2
if version >= 700
au VimEnter * hi StatusLine ctermfg=233 ctermbg=40
au InsertEnter * hi StatusLine ctermfg=253 ctermbg=196
au InsertLeave * hi StatusLine ctermfg=233 ctermbg=40
endif
Basically, make the statusbar always visible. On entry color the status bar black with green text. On switching to insert mode, color it gray with red text. Go back to black+green when in command mode. This assumes you have 255 colors available in your terminal. There are plenty of ways to enable this; if you just want a sense of what it looks like and you don't have 255 colors, 1-8 are available by default on most terminals. Now I can tell what mode I'm in purely in my peripheral vision.
Community
I've been (re)watching Community. It's a great show - the best comedy since Arrested Development. It's more self-conscious about itself than Arrested Development was (or maybe it's just more a little more strained).
I found this Four-Part Interview, in the form of 'episode commentary' (not bound too closely to the episodes themslves) with the creator, Dan Harmon, on The Onion AV Club, to be quite interesting. Apparently, he thinks that almost every episode is terrible after he's done shooting it, in a sort of David Foster Wallace, suffering artist way. Not what you'd expect from a comedy writer.
Puzzles and Programming
This is a quote from the great book "Surely You're joking, Mr. Feynman!"
Once I get on a puzzle I can't get away. If my mother's friend had said, "Never mind, it's too much work," I'd have blown my top, because I want to beat this damn thing, as long as I've gone this far. I can't just leave it after I've found out so much about it. I have to keep going to find out ultimately what is the matter with it in the end.
That's a puzzle drive. It's what accounts for my wanting to decipher Mayan hieroglyphics, for trying to open safes. I remember in high school, during first period a guy would come to me with a puzzle in geometry, or something which had been assigned in his advanced math class. I wouldn't stop until I figured the damn thing out - it would take me fifteen or twenty minutes. But during the day, other guys would come to me with the same problem, and I'd do it for them in a flash. So for one guy, to do it took me twenty minutes, while there were five guys who thought I was a super-genius.
This is the same way I feel about debugging (actually, it's how I feel about some types of debugging). Sometimes someone will come to me for advice about debugging this or that, or if I can take a look at this or that code. And if there's a bug, and something *shouldn't* be operating the way that it is, I really want to find out why it's so.
I actually enjoy this type of debugging, although in some cases it can be really frustrating. Nonetheless, most of the steps are interesting: you brainstorm what could possibly be going wrong; you devise tests to eliminate the possibilities, and then you implement them. Occasionally, you go around in circles and it takes you three hours to solve. Even what I call "single character errors" - a misplaced semicolon, a plus instead of a minus, are interesting, and you feel a great relief when you discover the solution (even greater because it can be wrapped up with a tidy bow).
On the other hand, there is a certain class of errors or debugging which has no enjoyment for me. I would call this "import errors" and it's a particular problem with java. Basically they boil down to the fact that you didn't import a package correctly in Eclipse, or have the wrong version of a jar, and you're reduced to googling the error and reading various unhelpful forums such as Experts Exchange (the worst feeling is googling the phrase and seeing no results). These errors are so painful because there's no process to resolving them. You aren't really required to think too much, you just read a bunch of websites and eventually discover the solution.
Ultimately, though, it's even worse than that: the problem is that the error is arbitrary: it's not something you can ever really work out logically according to the way the language operates. It's an artifact of some specific implementation, it doesn't really obey the rules.
I've found that fear of this type of error has been a major cause of static friction when starting a new project. Because I might need to import a few packages, or I think I might run into an intractable error of this type, I'll never even start some projects.
I suspect when people say they hate debugging, they either don't make this classification of bugs/errors, or they don't feel the puzzle drive to solve them.
Apprenticeships and Trade School
Something that most people seem to realize soon after becoming familiar with computer programming is that programming resembles a trade more than an academic pursuit. However, students almost always study 'Computer Science' in college, because it's the only avenue that's really presented to them. But, if you frequent Slashdot or other websites, you start to realize that most people agree that Computer Science would not be what most people would study if they had other options that made sense from a career or resume perspective.
I would have to group myself into this category. I enjoyed parts of my study, such as surveys of programming languages, finite automatons, graph theory, and so on. But there was a lot of other stuff mixed in that I didn't particularly care for.
Certainly, I didn't learn too much in college about the actual art of programming (by this, I mean topics such as what's covered in Code Complete and other books). And maybe that's ok - after all, Computer Science is an academic discipline and what I learned was academic.
But if we consider programming as a 'trade,' the same way that we consider carpentry, or car repair, or cooking, a trade, then there's precious little in the way of hands-on instruction. I haven't really seen any. And while I think that programming is more cerebral and more challenging in some respects than other trades, I do think that treating it as a trade makes more sense than treating it as an academic pursuit.
I just finished reading The Sorcerer's Apprentices which in part covers the system of training that professional chefs in high-end restaurants undergo. It's a rigorous process: usually the chef will be going to culinary school, but will also be interning at a series of restaurants from anywhere from 2-6 months, or more. Many of the internships eventually become jobs, and they serve as an entrance to a sort of fraternity of elite chefs. They also have extensive on-the-job training, as the intern is doing work that contributes to meals served to patrons of the restaurant they work at.
This sort of systematic journeyman training reflects the way people learned most skilled trades, from smithing to cooping to printing: you usually underwent training for 5-10 years, then did some journeying around, and finally were considered capable of opening your own shop. A good description of what this was like is "The Autobiography of Benjamin Franklin," which is a recommended read.
I can think of no such system for programming. The typical career path for programmers (at least, for American programmers) is: mess around programming a bit before college, study Computer Science in college, do one or two internships while at college, and then get a job where you get some minimal training before you start doing real work.
None of this prepares you in a way where you formally learn from experts who have real-life experience. To start with, the internship process is usually a sort of formality. Either you do something that isn't very important as a way to keep you from bothering the programmers, or you're dropped into doing production-ready work with no training beforehand. This has been my experience both in internships and helping to administer them.
Certainly no one went through debugging processes, common development models or patterns, etc. Sometimes this is discussed abstractly in software engineering courses, but there is rarely any hands-on instruction. I guess maybe it's too intensive from a personnel perspective. Maybe the closest thing I've heard of in commercial use is pairs programming, where two people program while thinking out loud. But I've never personally seen this done.
I think this process is detrimental to both software development as well as to individual programmers, and I wish there was a more rigorous way to learn exactly how other people approach development, and how to go about training in the essential details and nuances of software engineering.
Technical Debt and Frugality
There's a programming concept known as technical debt, which is exactly like real debt. It's something that I think is particular to programming and is not necessarily something that could be applied to other areas. Technical debt is a particular property of shoddy craftsmanship, where the craftsmanship must be (A) fixable, (B) incur exponential costs as it increases linearly, and (C) be revisited multiple times far into the future.
The idea is that all software has some sort of 'debt' associated with it. Debt is measured in terms of time: you can save yourself a little upfront time and write code very quickly - thus incurring debt - or you can be more careful and incur little or no debt. Each bit of debt that is incurred requires future maintainers of the software to 'pay interest' (this is usually due to bugs that shouldn't exist, designs that didn't take into consideration where future code expansion would occur, lack of documentation/tests, etc). The interest payments are in the form of time that these future developers have to spend in order to work with the codebase.
Developers can also pay down the 'principal' on the debt, which would involve refactoring, adding documentation, writing regtests, and so forth.
It's a great concept, and the analogy to financial debt is clear: if we have too much financial debt, we must declare bankruptcy; if we have too much technical debt, the project becomes too unwieldy and must be scrapped.
But we can take it one step further. Someone who is frugal is usually willing to stretch every dollar: they will not make superfluous purchases. And they could use debt wisely: for instance, to get training for a new job (college loans), or to make a wise investment beyond their current capital, or in an emergency (credit card debt). On the other hand, someone who isn't frugal could go into debt to buy a new car, or a boat, or a TV - or they just might not track where their money is going and waste $50/month on a cell phone plan they don't need.
This is also analogous to technical debt. That is, technical debt is usually presented as a bad thing, but it doesn't have to be. In emergencies, it's wise to go into debt to get the fix out the door. And it can be wise to run a debt to get the first iteration of a working product out the door. For instance, technical debt can act as a sort of leverage for start-ups to quickly roll out products, as if they were trading on margin. I would term this frugal use of technical debt: the developer (and management!) are consciously getting into technical debt for very specific, pre-defined reasons. But like all kinds of leverage, there is an inherent danger; there must also be some plan for paying off this technical debt - it can be very easy, once in debt, to continue incurring it with the best of intentions, without ever actually paying down the principal. This is a dangerous situation and the interest payments can eventually spiral out of control, causing bankruptcy.
On the other hand, in many cases developers will enter into technical debt because they're lazy and the first solution they think of seems like the best one, or they want to stop working early, or management keeps them under continual pressure, or they're simply not aware of it (lack of experience). These would all be forms of spendthrift technical debt that should be avoided at all costs.
I think this distinction makes intuitive sense, particularly for people who are naturally frugal. Technical debt is a tool that you can take advantage of, but it's very dangerous if you don't handle it properly, and it's even more dangerous if you don't know you're using it.
Shuttle Software
The conclusion of the space shuttle program happened to coincide with a recent re-read (for probably the third or fourth time) of Richard Feynman's What Do You Care What Other People Think. It's a great book, and includes an extended description of Feynman's experience investigating the crash of the space shuttle Challenger.
It's worth delving into Feynman's description of avionics software, which control the takeoff and landing of the shuttle (and many, many other things). In particular, Feynman's description of shuttle avionics reminds us that it is possible to write 'flawless' code, given enough time. It also serves as a warning of what can happen if management corrupts the software development (or other engineering) processes.
What he ends up describing is a dysfunctional bureaucracy that shows the real difference in thought processes of typical 'management' vs engineers/scientists.
After finding that management kept fudging the numbers provided by engineers (in the first case, engineers of the booster rockets, on which the O-rings were the cause of the crash), or ignoring their protests about safety risks, he starts to investigate the different teams working on the space shuttle. What he finds is that in most cases the culture had become corrupted by pressure to please the higher ups, and there was a clear disconnect between the engineers and management. He investigates the engines, boosters, and avionics.
It's worth noting that the system was 250,000 lines of code, written on antiquated hardware (Feynman was writing in 1988, but the computer systems hadn't been upgraded for 15 years, since about 1973). It controlled automatic takeoff and landing of the shuttle, among numerous other responsibilities.
Here is an extended description of their testing procedures.
I learned how the software engineers developed the avionics for the shuttle. One group would design the software programs, in pieces. After that, the parts would be put together into huge programs, and tested by an independent group.
After both groups thought all the bugs had been ironed out, they would have a simulation of an entire flight, in which every part of the shuttle system is tested. In such cases, they had a principle: this simulation is not just an exercise to check if the programs are all right; it is a real flight - if anything fails now, it's extremely serious, as if the astronauts were really on board and in trouble. Your reputation is on the line.
In the many years they had been doing this, they had had only six failures at the level of flight simulation, and not one in an actual flight.
So the computer people looked like they knew what they were doing: they knew the computer business was vital tot he shuttle but potentially dangerous, and they were being extremely careful. They were writing programs that operate a very complex machine in an environment where conditions are changing drastically - programs which measure those changes, are flexible in their responses, and maintain high safety and accuracy. I would say that in some ways they were once in the forefront of how to ensure quality in robotic or interactive computer systems, but because of the obsolete hardware, it's no longer true today.
I didn't investigate the avionics as extensively as I did the engines, so I might have been getting a little bit of a sales talk, but I don't think so. The engineers and the managers communicated well with each other, and they were all very careful not to change their criteria for safety.
I told the software engineers I thought their system and their attitude were very good.
One guy muttered something about higher-ups in NASA wanting to cut back on testing to save money: "They keep saying we always pass the tests, so what's the use of having so many?"
Later on, there's a further description of some of the testing procedures, after Feynman apparently did a little further digging on the last point.
The software is checked very carefully in a bottom-up fashion. First, each new line of code is checked; then sections of code (modules) with special functions are verified. The scope is increased step by step until the new changes are incorporated into a complete system and checked. This complete output is considered the final product, newly released. But working completely independently is a verification group that takes an adversary attitude to the software development group and tests the software as if it were a customer of the delivered product. There is additional verification in using the new programs in simulators, et cetera. An error during this stage of verification testing is considered very serious, and its origin is studied very carefully to avoid such mistakes in the future. Such inexperienced errors have been found only about six times in all the programming and program changing (for altered payloads) that has been done. The principle followed is: all this verification is not an aspect of program safety; it is a test of that safety in a non-catastrophic verification. Flight safety is to be judged solely on how well the programs do in verified tests. A failure here generates considerable concern.
To summarize, then, the computer software checking system is of highest quality. There appears to be no process of gradually fooling oneself while degrading standards, the process so characteristic of the solid rocket booster and space shuttle main engine safety systems. To be sure, there have been recent suggestions by management to curtail such elaborate and expensive tests as being unnecessary at this late date in shuttle history. Such suggestions must be resisted, for they do not appreciate the mutual subtle influences and sources of error generated by even small program changes in one part of a program on another. There are perpetual requests for program changes as new payloads and demands and modifications are suggested by users. Changes are expensive because they require extensive testing. The proper way to save money is to curtail the number of requested changes, not the quality of testing for each. [emphasis mine]
There are a couple of things to note: the first is that it does appear possible to remove essentially all bugs in a software program, given enough energy, diligence, and care. The question is: is that worth it, and that depends on what the software is. For many programs, it's just not worth the effort. Most commercial software falls into this category: sure, it's possible to prevent bugs from creeping into a video game, but the game would take much, much longer to be released.
It's also worth noting that, even with a very different environment than traditional engineers, the avionics systems were the only ones given a clean pass by Feynman. Traditionally people dump on software development, because bugs are quite frequent - everyone has experienced them - whereas there appear to be few or no bugs in, say, civil engineering or even mechanical engineering. But the error rates for the mechanical engineers were about once every 200 trials (and in some cases, much more frequent). Compare that to error rates for 250,000 lines of code (probably more 'moving parts', ie, changing variables, than the rest of the shuttle), tested over 15+ years.
This isn't to dump on other engineers. It's just to say that much of what is normally interpreted as inherent flaws in software development are more likely due to different workflow/development processes, different management environments, and different levels of complexity. For instance, I believe that there were probably fewer change requests for shuttle avionics than normal commercial software, which is be notoriously unconstrained and subject to feature creep.
It also tells us that we can aim high when developing software. That our goal, given time constraints, can be near perfection, and that bugs are not inevitable. We shouldn't set some artificial ceiling of 'good enough' for any important software, but rather 'perfect within the given development time.'
For more reading on shuttle avionics, here is a classic, quite detailed article, They Write the Right Stuff, which describes the development process and culture around coding for the space shuttle.
I don't want to be a product of my environment. I want my environment to be a product of me.