When I was doing stuff on LiveJournal, I was thinking about things from JavaScript to how things were interacting in the kernel. I was reading Linux kernel code about epoll and I was like, “Well, what if we have all these long TCP connections and JavaScript is polling with these open TCP connections that are going to this load balancer?” I was trying to think of how much memory is in each structure here. That’s still somewhat high-level, but then we were thinking about things like, we’re getting so many interrupts on the Ethernet card—do we switch to this NAPI thing in the kernel where rather than the NIC sending an interrupt on every incoming packet it coalesces them to boundaries that were equivalent to 100 megabits speed even though it was a gigabit NIC. We were collecting numbers to see at what point this made sense and freed up the processor.
We were getting a lot of wins for really low-level stuff. I had somebody recently tell me about something: “Java takes care of that; we don’t have to deal with that.” I was like, “No, Java can’t take care of this because I know what kernel version you’re using and the kernel doesn’t support it. Your virtual machine may be hiding that from you and giving you some abstraction that makes it look like that’s efficient, but it’s only efficient when you’re running it on this kernel.” I get frustrated if people don’t understand at least the surface of the whole stack.
In practice, nothing works. There are all these beautiful abstractions that are backed by shit. The implementation of libraries that look like they could be beautiful are shit. And so if you’re the one responsible for the cost of buying servers, or reliability—if you’re on call for pages—it helps to actually know what’s going on under the covers and not trust everyone else’s libraries, and code, and interfaces.
I almost don’t think I would be a programmer today if I was starting off. It’s just too ugly. This is why I’m so excited about things like App Engine. Someone described Google’s App Engine as this generation’s BASIC. Because this generation, everything is networked. When I was programming, it was one language, and it was on my own machine, and the deploy was up enter, or RUN enter. Kids today don’t want to write something stupid like a “bounce a ball” app on their own machine. They want a web site to interact with.
I still have people mailing me who are like, “Hey, I have this idea—I want to make Wikipedia meets YouTube, meets—” Everyone wants to do a web site where their favorite four web sites aren’t quite right and they want to make one that looks kind of like that.
The fact that App Engine gives you one button, “Put this on the Web,” and you write in one language, arguably a pretty easy-to-learn one, Python, is perfect. It’s a great intro to programming—there are so many layers and layers of bullshit that it gets rid of.
Seibeclass="underline" How does that fit with your dismay at the Java guys who tell you, “Oh, Java takes care of that for you.” Isn’t that the same? “Well, App Engine will take care of that for you.”
Fitzpatrick: I don’t know. Maybe it’s because I know what’s going on. Actually the JVM isn’t that bad. I guess it’s when people have blind faith in their abstractions without understanding what’s going on.
Seibeclass="underline" You had a lot of programming experience by the time you got to college and studied computer science. How did that work out?
Fitzpatrick: I skipped a lot of my early C.S. classes, because they were just really boring. I would go and take the tests. Then towards the end they got kind of fun, once you get to the 300- and 400-level classes. But right when it got interesting, I graduated. And they wouldn’t let me take the fun grad-level classes, because I wasn’t a grad student.
I remember in the compiler class, the final project was we had to take this existing language that we had been playing with and add a whole bunch of features, including one feature of our own choosing as the bonus part of the project. So I chose to implement run-time array bounds checking. Anyway, the professor took our compiled binary and ran his test suite against it, and it failed a couple of his tests. He was like, “Sorry, you get a C because you failed my unit test,” When I went to look at it, I was like, “You have off-by ones in your test suite.” So he gave me the grade back and I got an A, but I never got the bonus points for adding a feature to the language. I was angry at school at that point.
And I remember our database class was taught by someone who, it seemed, had no real-world experience with databases. At this point I’d worked with Oracle, Microsoft Server, and tons of MySQL. So I was asking all these realworld questions I actually wanted answers to—things that were melting right now—they would just give me some textbook answer. I’m like, “No, no. That doesn’t work.”
Seibeclass="underline" You graduated in 2002. Do you have any greater appreciation now of what they were trying to teach you?
Fitzpatrick: Half the classes I totally loved, and either I learned something totally new that I wouldn’t have learned at the time, or I learned the proper background material and the proper terminology. Prior to that, I knew programming pretty well but I didn’t have the vocabulary to describe what it was I was doing. Or I would make up my own terminology for it and people would think I didn’t know what I was talking about. Formal C.S. education helped me be able to talk about it.
Seibeclass="underline" Do you have any regrets about combining running a business with school? Would you rather have just done one or the other?
Fitzpatrick: No, I think that was the best way. I had friends who went to college and just did college, but I knew so much of it already, I would’ve been bored. I had one friend who also knew a whole bunch of it but he was of this school of thought that he’s at college to learn, not for grades, so he was, on the side, studying Arabic and Chinese and Japanese. And all the crazy programming languages. Every week it was like, “I have a new favorite language. This week I’m only going to write in OCaml.” So he kept himself busy that way. I kept myself busy and not bored other ways.
Then I had friends who dropped out after their freshman year just to do web stuff. A couple were doing a porn web site or something. They were like, “Oh, we’re making all this money.” But they just worked a whole bunch; they were always in their basement working. College was awesome for meeting people and partying. If I just did LiveJournal, I would’ve killed myself stresswise.
Seibeclass="underline" Are you glad you studied computer science?
Fitzpatrick: I probably could have done without it. I did a lot of things I wouldn’t have done normally, so I guess it was good. I wish maybe I would have like done something else as well, maybe stayed another year and double-majored in something totally unrelated. Did linguistics more. I’m kind of sad I left college and I felt I only did half studying because so much of it I already knew. My early C.S. classes I barely attended and it was only towards the end where things just started to get interesting when it was like, “OK, you’re done.”
Seibeclass="underline" Did you ever think about going to grad school?
Fitzpatrick: Yeah. It would have been fun, but I was busy.
Seibeclass="underline" Do you try to keep up with the C.S. literature?
Fitzpatrick: Me and my friends still forward each other papers around, neat papers. I read something the other day about some new technique for resizing Bloom filters at runtime. It was pretty awesome. The papers that come out of the storage conferences, some out of industry and some from academics, about different cool systems—I try to read those. There are different reading groups at Google—systems reading groups or storage reading groups. I’ll see something on Reddit or a friend will forward a paper or something like that or link it on a blog.