Code review interview guide

The goal of the code review interview is to explore the kind of code, thought processes, and choices that the candidate will make at 18F by examining a sample of their work.

Watch out for unconscious bias!

Remember that we all have unconscious bias, and that hiring is especially susceptible to bias. However, when we recognize and accept bias we can be on the lookout and it’ll be less likely to unconsciously guide our decisions.

Learn how.

Research shows that the effect of unconscious bias can be profound. For example, in one study, “applicants with white names needed to send about 10 resumes to get one callback; those with African-American names needed to send around 15 resumes to get one callback.”

This bias prevents us from achieving the results we want, which is to select the best possible candidates regardless of background.

Humans are not only biased, but we almost never realize that we’re being biased. However, when we realize and accept bias, and recognize it, we can be on the lookout for bias and it’ll be less likely to unconsciously guide our decisions.

In that spirit, here are a few ways you can look out and correct for bias during the hiring process.

Review these guidelines before every interview or round of resume review. Studies show that we’re less biased when we’re conscious of our own thinking, so continually reminding yourself to be aware will help.
Remember that we’re especially susceptible to assume that underrepresented minorities — women, people of color, etc. — are less qualified than their white male counterparts. When considering candidates from underrepresented backgrounds, check your thinking about qualification. Ask yourself: am I reading this person’s qualifications the same as if they were white, male, etc?
Continually re-check the guides and scoring rubrics to make sure you’re reviewing fairly. After a while, you’ll start to feel like you’ve memorized the guides and rubrics. This is good since it’ll help you be more efficient, but our memories are fickle things. The more you remind yourself of the concrete, established metrics, the less likely you’ll be to make “gut” decisions that could be colored by bias.
Watch out for assessments of candidates that may be colored by age, gender, race, etc. For example, we tend to be more likely to use words like “aggressive” or “competitive” when describing men, vs “supportive”, “nurturing” when describing women. Are you surprised that an older person has cutting-edge technical skills? Ask yourself if an assessment might be colored by an applicant’s demography.
Don’t check out the candidate on social media, or Google them. A person’s public profile almost certainly won’t have anything relevant to work, and might instead reveal all sorts of irrelevant information (age, gender, political affiliation, race, etc.). If the candidate application/resume links to a personal website, LinkedIn, or GitHub you can check those out — these are more work-focused, and we can assume a candidate has put what they want an employer to see there. If you use Chrome, the Unbias Me extension (written by Fureigh) can help by removing avatars and names on some websites such as LinkedIn.
If you come to a conclusion about a candidate very quickly, before you’ve read the whole resume or finished the interview, spend the rest of the session trying to disprove that conclusion. Snap judgments are much more likely to be prone to bias than considered ones. We tend to jump to conclusions and then look for evidence to support our hypothesis. To compensate, if you find you’ve reached a Yes/No conclusion very quickly, spend the rest of the session trying to disprove that hypothesis. Explicitly look for evidence that you’re wrong. If you’ve decided immediately that a candidate is not qualified, spend the rest of your time trying as hard as you can to find evidence that they are qualified.

Purpose

The candidate is instructed to complete a coding exercise, or bring an existing code sample. As an interviewer, you'll receive that code at least a day before your interview with the candidate.

The focus of the interview should be on their code, how it works, why they made the choices they did, and so forth. Before the interview, you should plan to spend an hour or two reading the candidate's code, running it, and preparing follow-up questions to ask when you interview them.

Remember to be as pleasant and friendly as you can be! You can deliver a demanding interview while being kind and empathetic.
Whenever possible, ask these questions exactly as they’re worded to try to get consistency between multiple candidates.

Before the interview

Before the interview, prepare by reviewing the code, trying to run it (and the tests), and review the list of what to look for and the questions.

As you review, remember that candidate only spent about three hours on this code. It doesn't need to be perfect. It should work, and solve the problem assigned, but minor issues, rough edges, and less-than-perfect parts are totally fine. A good way to think about this: you're looking for code that's about the same level as you might expect on an initial pull request. It might not be perfect yet, but that's what code review is for.

For the list of things to look for during code review, see 18F only, what to look for during code review.

18F only, Make a copy for your notes

Conversation outline

Take behavioral notes.

When taking notes, note what the candidate says, rather than your impressions. This helps combat unconscious bias and will help you share the reasons for your conclusions and decisions.

See an example.

Try to note what the candidate says, rather than your impressions — that will help you share behavioral reasons for your conclusions and decisions. That is, try to write down what the candidate said or did, rather than how you felt about it. For example:

✅ Good	❌ Bad
“candidate ‘forced engineers to do \$X’” (captures what a candidate said)	“candidate has an adversarial relationship with engineers” (interpretation of what they said)
“candidate has maintained an \$X as an open source module for 2.5 years” (captures what a candidate did)	“candidate is a good open source maintainer” (your feelings/interpretations of what they’ve done)

Begin by introducing yourself by saying this or something similar to it:

Hello! My name is ___, my pronouns are ___, and my role is ___.

Thanks for interviewing with me today. The purpose of this interview is to explore the code you submitted to get a sense for how you think about coding problems. I’ll ask a series of questions about the code sample you’ve submitted, how it works, why you made the choices you did, and so on.

My questions will take about 40 to 45 minutes or so, and then we’ll have the remaining time to answer any questions that you may have. Feel free to think for a moment before answering if that’s your style - you won’t be judged for it.

There will be times when I ask for more information, or want to dig deeper into your answers. That’s normal: I want to make sure I understand what you did and why. I’ll be taking notes, please don’t let that distract you.

I’m excited you’re here - do you have any questions for me before we get started?

Questions and follow-ups

In all of your questions, you should be trying to see that the candidate has a deep understanding of the code and the problem, and can explain it to you coherently. You don’t need to agree with all of the candidate’s choices, but they should be able to explain them clearly.

"Walk me through how your code works."
"How does this part (point to a tricky bit) work?"
"What would you do differently if you got to do this over again? What would you keep?"
"Where did you get stuck? How'd you get past it?"
"What was your testing strategy for this code?"
- "How effective do you think your test suite is? Are there any gaps?"
- "When did you write tests in relation to when you wrote your code? (Before/During/After)"
- "What would you do differently?"
"I see you used Module X here. How would you solve this problem without Module X?"
"What other programming languages would be appropriate to solve this problem with?"
- "(Based on their answer) What would be different about the solution in Language X?"
"(Where applicable) If your data file was several orders of magnitude larger (say, 1 TB), would this code still work? Why or why not? What would you do differently?"
"What would you do differently if you had a whole week to work on this problem?"
"I’ve asked all of the questions on my list, and the rest of the time is yours. What questions can I answer for you?"

For the list of things to look for, see 18F only, what to look for during the code review interview.

After the interview

Thank you for your focus! Use your notes to fill out the feedback form linked in the calendar invitation, which will prepare you for the debrief meeting.