Return to the Source [programming, opinion, no-code]

If a system is to serve the creative spirit, it must be entirely comprehensible to a single individual. — Dan Ingalls

I saw Ellen Ullman speak last night, about her new book,1 and the topic turned to culpability for Y2K, systems that people never expected would run for decades, and systems that no one understands any more.

When a Peterborough nuclear facility reached out to retrocomputing enthusiasts looking for someone who knew PDP-11 assembler, I started thinking about the Foundation series (warning: possible spoilers follow). The idea that was most striking to me, in those books, was that eventually, societies who became comfortable with advanced technology could end up losing the knowledge of how that technology worked (cast in a very '50s nuclear vibe).2 I encounter a lot of people dismissive of the importance of systems programming (moreso online than IRL, thankfully), and it makes me wonder if we are rapidly heading in that direction.

Ullman talked about "returning to the source" — extracting the lost knowledge from code whose authors aren't around anymore. There couldn't have been a more serendipitous time for this, as I had just been discussing the merits (and pitfalls) of reading source with my fellow Recursers.

It's my sincere belief that code is the source of truth in computing (and by this, I also mean machine code, which is also worth reading; the success of Matt Godbolt's Compiler Explorer tells me I'm not alone in this). So I am writing this article to exhort you to read code written by someone you don't know, today, to save the future.

Why Read? Craftsmanship

Jon Bentley opens his first Programming Pearl on literate programming with the following:

When was the last time you spent a pleasant evening in a comfortable chair, reading a good program? I don't mean the slick subroutine you wrote last summer, nor even the big system you have to modify next week. I'm talking about cuddling up with a classic, and starting to read on page one. Sure, you may spend more time studying this elegant routine or worrying about that questionable decision, and everybody skims over a few parts they find boring. But let's get back to the question: When was the last time you read an excellent program?

(I like to ask this question in interviews, on both sides of the table; not to be a snob, but to open up a discussion about reading code.)

I always remember this better the way Steve McConnell paraphrases Jon Bentley in Code Complete:

One especially good way to learn about programming is to study the work of great programmers. Jon Bentley thinks that you should be able to sit down with a glass of brandy and a good cigar and read a program the way you would a good novel.

The intent is clear: you can improve as a craftsperson by reading masterpieces of software, the same as writers need to read other works3 and musicians need to listen to other performances. Empirically verifying whether this is true is unfortunately outside my abilities, but I believe I've benefitted greatly from "reading the greats".4

One thing to clarify from those quotes, though: these always made me picture reading the source from top to bottom, and it turns out this isn't particularly effective.

I've been passing Peter Seibel's Code is not Literature around a lot lately; this is an article I didn't understand when I first read it. I thought it was an attack on code reading, but in fact it's a suggestion of much better ways to approach reading code, especially in a group.

There's an extension to that: I think Pierre Bayard's How to Talk About Books You Haven't Read expresses this far better than I can, but I think it's self-defeating to believe that "having read the code" is a binary state: either you read (and understood) it all, or you haven't "read it". Diving into a codebase is just the start of a long relationship with that code; you can keep coming back, to familiar haunts and undiscovered territories every time.

Why Read? Personal mastery

I started this article with my favorite Dan Ingalls quote, from the design principles of Smalltalk. I think there's a deep truth about software in that. We don't seem to be able to build abstractions that aren't leaky, so you're always going to need to be able to go up and down in the layers of abstraction in a system just to fix the problems at the level you care about.

What will a system that lasts 10000 years look like? I don't think it will be one that no one can understand. Is it possible to build complete systems that can be understood by an individual? The work by Viewpoints Research Institute seems to suggest it's possible. Until the day when we all have our personal 10kLOC operating systems5 committed to memory, Fahrenheit 451-style, reading systems large and small helps one grapple with the nature of complexity, and find a personal relationship with it.

And, pragmatically, reading your dependencies helps you answer the question: will anyone be able to understand this when it breaks? (And it will break: because of bitrot; because the assumptions changed.)

Why Read? Procedural rhetoric

Ellen Ullman also talked about how algorithms have biases; software isn't neutral6. This reminded me of Ian Bogost's concept of procedural rhetoric, where interaction with systems can be persuasive, and can communicate ideas and opinions, in a way that is subtle.

This is a deep topic in its own right, but I think the first step in being the future masters of technology7 is to understand the workings of systems we interact with, and the purest form of that is reading their code. Even when we can't read the code of many systems around us, reading the code of similar systems is a part of understanding how they might work and the biases implicit within them.

What's worth reading?

You might be wondering, "where do I even start?". I think there are two classes of code especially worth reading: code that you use, and code that is great. The latter is incredibly subjective; I have been compiling a list of what I think are "masterpieces of software" and will post it at some point, to much criticism, I'm sure.8

However, the former is straightforward for everyone: read your dependencies. Now that we live in a Free Software utopia (hah)9, you probably have access to the source of the vast majority of libraries, servers, tools, and systems you use and depend on. This is a wealth that is often squandered.

Literate programming

Fans of literate programming are probably champing at the bit, waiting for me to unveil Knuth's perfect plan for programmer literacy. (If you're not familiar with literate programming at all, I think Knuth's book is still the best treatment, even if it is pretty dated at this point.)

I have done a bit of literate programming (and I still think literate assembly language is the most useful application of these techniques); I've read a lot of literate programs; and as it relates to the topic of this article, I feel it's mostly irrelevant. There will always be the need to read unadorned, unpresented programs. Literate programs are lovely, but they aren't a complete replacement for the kind of code reading I'm advocating here (even if it was a common enough practice that one could find a reasonable supply of them).

Reading about reading

Sadly, there haven't been a lot of books about reading code. There are many books that intersperse commentary with code, but these aren't really about reading code; they're more like literate programs.

The only book I know to exclusively treat this subject is Diomedis Spinellis's Code Reading. It's been a while since I read it, but I remember feeling that it was a good start, but not the complete picture. The author has also compiled a lot of arguments for why code reading is important on that site, if you find this article unconvincing.

Michael Feathers's Working Effectively with Legacy Code is a truly great book, but I don't remember it having much concrete advice about actually reading legacy code. (I might be misremembering, of course.) However, it is about testing, and one of the great ways to read a codebase is to try to write tests for it.

Bonus: the tension of comments

I mentioned that I feel that code is the only truth of the system. (Which is a terrible oversimplification as almost all systems are also data-driven to some extent.) So it's unsurprising that I agree with Kernighan and Pike's advice on commenting in The Practice of Programming, which is often misinterpreted as "don't write comments".

When I read code (especially when I review code), I actually skip the comments (sometimes I strip them out) on my first pass through the code. I find that, because I'm much faster at reading English text than source code, it is easy for the eye to get comfortable reading the comments and only skimming the code. This deceives me into thinking I've actually read the code, when I haven't.

However I have an egregious example from Darwin (macOS) osfmk/kern/thread_call.c:

if (cancel_all)
        result = _remove_from_pending_queue(func, param, cancel_all) |
                _remove_from_delayed_queue(func, param, cancel_all);
        result = _remove_from_pending_queue(func, param, cancel_all) ||
                _remove_from_delayed_queue(func, param, cancel_all);

I often trot this snippet out when I ask people where their threshold for "too clever" code is. To me, my first reaction on seeing this was "this is a typo", and only after looking over it again carefully did I realize what it was trying to do, and then a while longer thinking as to whether it actually did that correctly.

But the most recent time I went to show someone this snippet, it turned out it had been updated, including adding some crucial comments!

if (cancel_all) {
        /* exhaustively search every queue, and return true if any search found something */
        result = _cancel_func_from_queue(func, param, group, cancel_all, &group->pending_queue) |
                 _cancel_func_from_queue(func, param, group, cancel_all, &group->delayed_queues[TCF_ABSOLUTE])  |
                 _cancel_func_from_queue(func, param, group, cancel_all, &group->delayed_queues[TCF_CONTINUOUS]);
} else {
        /* early-exit as soon as we find something, don't search other queues */
        result = _cancel_func_from_queue(func, param, group, cancel_all, &group->pending_queue) ||
                 _cancel_func_from_queue(func, param, group, cancel_all, &group->delayed_queues[TCF_ABSOLUTE]) ||
                 _cancel_func_from_queue(func, param, group, cancel_all, &group->delayed_queues[TCF_CONTINUOUS]);

Reading this code gave me an appreciation for a kind of comment I would otherwise have tended to omit.

Bonus: How to read a C program

I would be remiss not to end this with some concrete advice. Reading tips will vary by language, but a lot of code I read is written in C; how do I approach reading a C program?

When in doubt, start from the bottom. Occasionally someone tries to fight the natural C order of definitions by forward-declaring static functions; this is unnatural and most code isn't written this way. Instead, you'll generally see that if you want a "top-down" view, you should go to the end and work backwards. (Incidentally, this is even more true for OCaml / Standard ML programs, where the order of declaration is very strict.)

Use unifdef to get rid of as much that is irrelevant to you as possible, at least for an initial reading. Get rid of those paths that are only taken on Acorns and Ataris.

Use ctags, cscope, GNU global, and whatever other support you can find to be able to quickly jump to and from the definitions of identifiers, ideally also seeing all the places that refer to those identifiers. Cross-reference tools like LXR (on the Linux kernel, on MRI) are sometimes nice for this, although I often find them more cumbersome than using my editor on my local machine.

Look at the header files included; what are the data structures that get used all the time? Sometimes there's tangly stuff with macros, like the queue.h macros for intrusive data structures; it can help to run the preprocessor over the file (cc -E) or write the structure out by hand on paper and annotate it.

Don't be afraid to mutilate the program to understand it. Cut things out and try to compile it. Make hypotheses and validate them. Is this struct field used by anything? Let's cut it out and see what breaks in the compile. (Newer statically-typed languages tend to be even more receptive to those kinds of experiments.) Attach a debugger and set breakpoints at functions you're reading.

If you're reading a library, consider starting with an example program, tracing through the API calls made, into the guts of the library.

Maybe you also have the version control history (it's wonderful that we can start almost taking this for granted). When you find something interesting, dig back with blame; what changed, and why? Also, if the code seems too complicated, it can be helpful to start from an early revision and then work forward in history. Seeing the code adapt as imagination encounters the real world paints a picture of evolution as vivid as any archaeological exhibit.

Serendipity can also be good. Seibel, cited above, describes "play[ing] the role of a 19th century naturalist returning from a trip to some exotic island to present to the local scientific society a discussion of the crazy beetles they found". Sometimes I like to just peek at different parts of the code at random, and see if there's something that catches my eye or delights me.

After all, reading code is not just good for you; it is fun.



I should probably wait until I've read her new book to write this, since I'm sure it has some great insights about this, but I can't wait.


Tangent: perhaps this will never happen in software because nothing runs reliably long enough for anyone to think we can get rid of the programmers.


"If you don't have time to read, you don't have the time (or the tools) to write. Simple as that." — Stephen King.


I feel this is advice commonly given to young mathematicians but I can't find any source for it, at the moment. I think it must derive from this Niels Abel quote.


I feel this is inherent, because software is made of decisions.


"The future masters of technology will have to be light-hearted and intelligent. The machine easily masters the grim and the dumb." — Marshall McLuhan


Ok, a friend convinced me to include a few places to start if you're really at a loss; since I talk about C at the end, how about some C code I've enjoyed reading recently: postgres, anything by cperciva, Illumos, sqlite. Some of these are pretty complicated, but typically stylistically good, and uncommonly well-commented.


Desperately absent in this article is an acknowledgement of how much free and open source software has changed the world, but I don't know how to write about it. The beginning of David MacIver's Programmer at Large makes me think, though.