Seibeclass="underline" When you were hiring programmers at LiveJournal, did you manage them?
Fitzpatrick: Well, I kind of assumed that none of them would need managing; that they would just all be self-driven like me. That was a learning experience in HR, that some people just do what they’re told and don’t really have a passion for excellence. They’re just like, “Done. Next assignment,” Or they don’t tell you and just browse the Web. So I had a couple of painful experiences. But I think after a year or two of that, I learned that people are different.
Some are purists. They would just do abstraction on abstraction on abstractions. They would go really slowly and are very religious about their style. They’re like, “I’m an artisan programmer.” And I was like, “Your code doesn’t run. It’s not efficient and it doesn’t look like any of the other code that you’re interacting with.”
Seibeclass="underline" Did you figure out how to make good use of people like that?
Fitzpatrick: One person, I tried dozens of different things. I think he might’ve been ten years older than me. I don’t know how much, because I never ask that—I was afraid of legal hiring questions. But I got the feeling that he didn’t want to work for some young punk. I was like 22. That one eventually didn’t work out. That was the only person I let go.
Other people I eventually figured out what motivated them. One guy was really good at tinkering and getting a prototype working. He wrote sysadmin Perl. He could wire stuff together, write shell scripts, and write really bad Perl and really bad C, but kind of get it working. Then we would be like, “Holy crap, you researched all this stuff, you got all these components talking to each other?”
We were setting up a voice bridge to LiveJournal so you record something and post it to LiveJournal. There were just so many moving parts involved. I thought it was painful as hell. He loved it. He figured it all out and got it working. Then we just rewrote it all. And we figured out that was the way it worked with him. He figured out the interface and we would fix it all up. Once I figured out that was his role, we got along great.
Seibeclass="underline" So you’ve hired for your own company, and I assume you’ve been involved in hiring at Google. How do you recognize a great programmer?
Fitzpatrick: I often look for people that have done, like, a lot of stuff on their own that wasn’t asked of them. Not just their school project or just what their previous employer had them do. Somebody who was passionate about something and had some side project. How did they maintain it and how serious did they get with it? Or do they do a lot of quick hacks and abandon them?
Seibeclass="underline" Do you have favorite interview questions?
Fitzpatrick: One of the ones I’ve given a few times because it was on my AP programming test is given two decimal numbers as strings of arbitrary length, multiply them. There are a lot of different ways that they could do it. If they’re really good at math—like I’m not—they can find some clever ways to do it really efficiently. Worst case, they can make a class that does just addition repeatedly.
I tell them from the beginning, “Don’t stress out. You don’t have to do it efficiently. Just get it done somehow.” Some people stress out and have no clue where to begin. That’s kind of a bad sign. The worst case, you just implement the algorithm you do in grade school.
I actually wrote a program in grade school to do my long division and multiplication and show the work. Including all the steps and where to cross out. So then we would get these problems, like ten per page or something, and I would type it into the computer and then just reproduce the problems in scribbles. I did the same thing in chemistry to find the orbitals of electrons. But the thing I find is by writing a program to cheat, you learn because you have to learn it really in depth to write that program.
Seibeclass="underline" Do you think that would work for anyone? Instead of teaching kids long division, should we teach them how to program and then say, “OK, now your task is to write a program to do this long-division procedure”? By the time they’ve actually written that program, they’ll understand division. Or does that only work if you have some natural inclination that way?
Fitzpatrick: It worked for me. A lot of times, someone could teach you something and you’re like, “Yeah, yeah, sure. I understand.” You delude yourself but once you actually have to get down and do it at a real level and understand all the corner cases, it forces you to actually learn the thing. But I don’t know if that would work for everyone.
Seibeclass="underline" Google has a bit of a reputation, as Microsoft also does, of interviewers using puzzle questions.
Fitzpatrick: I think those are kind of banned. Or strongly discouraged. Maybe some people still do them, but I think, in general, they’re discouraged.
Seibeclass="underline" What did they ask you in your interview?
Fitzpatrick: One question was, imagine you have a bunch of computers on a switch and they turn on the whole rack; come up with an algorithm so every machine on the rack knows the status of all the other ones about whether they’re on or off. So basically a presence thing. That was pretty much the constraint. Basically, they described Ethernet: you could send a broadcast to everyone, or you could send it to a specific MAC address. So I just kind of walked through all the different strategies to minimize bandwidth and to minimize latency discovering when something’s dead. That was a fun one.
Seibeclass="underline" What’s the worst bug you ever had to track down?
Fitzpatrick: I try not to remember them. I hate it when it’s something where your assumptions are so far off. The other day—this is definitely not an example of the worst one ever—I spent 90 minutes debugging something because I was writing to one output file and reading another file named the same thing but with one path component missing. I kept rerunning this huge MapReduce and seeing the output and putting it in GDB and stepping through it. “What the fuck? It’s not changing!” Finally I looked at the paths and I was like, “Holy crap.” I don’t know why I spent 90 minutes on it; I was so obsessed that I didn’t step back and check, is my command line correct?
There’s a lot of that. We always had some good stuff with Perl like the $_ isn’t lexically scoped. So if you fuck with $_ in a sort, you can mess with somebody else’s far away. So we had this bug that took us forever and we had a bunch of corruption going on. We finally figured that out. Then I audited all our code we had a new policy of “never do this.”
Seibeclass="underline" What are your debugging tools? Debuggers? Printlns? Something else?
Fitzpatrick: Println if I’m in an environment where I can do that. Debugger, if I’m in an environment that has good debuggers. GDB is really well maintained at Google and is kind of irreplaceable when you need it. I try not to need it too often. I’m not that great at it, but I can look around and kind of figure things out generally. If I have to go in there, I generally can find my way out. I love strace. Strace, I don’t think I could live without. If I don’t know what some program is doing, or what my program is doing, I run it under strace and see exactly what’s happening. If I could only have one tool, it would probably be that. All the Valgrind tools, Callgrind and all that, those are good.
But a lot of times lately, if there’s something weird going on, I’m like, “OK, that function is too big; let’s break that up into smaller parts and unit-test each one of them separately to figure out where my assumptions are wrong, rather than just sticking in random printlns.”
Then maybe in the process of refactoring, I have to think about the code more, and then it becomes obvious. I could, at that point, go back to the big, ugly state where it was one big function and fix it but I’m already halfway there; I might as well continue making it simpler for the next maintainer.