Debrief meeting guide

The goal of the debrief meeting is to discuss the evidence and data the interview panel was able to learn about a candidate during the interview process.

Watch out for unconscious bias!

Remember that we all have unconscious bias, and that hiring is especially susceptible to bias. However, when we recognize and accept bias we can be on the lookout and it’ll be less likely to unconsciously guide our decisions.

Learn how.

Research shows that the effect of unconscious bias can be profound. For example, in one study, “applicants with white names needed to send about 10 resumes to get one callback; those with African-American names needed to send around 15 resumes to get one callback.”

This bias prevents us from achieving the results we want, which is to select the best possible candidates regardless of background.

Humans are not only biased, but we almost never realize that we’re being biased. However, when we realize and accept bias, and recognize it, we can be on the lookout for bias and it’ll be less likely to unconsciously guide our decisions.

In that spirit, here are a few ways you can look out and correct for bias during the hiring process.

Review these guidelines before every interview or round of resume review. Studies show that we’re less biased when we’re conscious of our own thinking, so continually reminding yourself to be aware will help.
Remember that we’re especially susceptible to assume that underrepresented minorities — women, people of color, etc. — are less qualified than their white male counterparts. When considering candidates from underrepresented backgrounds, check your thinking about qualification. Ask yourself: am I reading this person’s qualifications the same as if they were white, male, etc?
Continually re-check the guides and scoring rubrics to make sure you’re reviewing fairly. After a while, you’ll start to feel like you’ve memorized the guides and rubrics. This is good since it’ll help you be more efficient, but our memories are fickle things. The more you remind yourself of the concrete, established metrics, the less likely you’ll be to make “gut” decisions that could be colored by bias.
Watch out for assessments of candidates that may be colored by age, gender, race, etc. For example, we tend to be more likely to use words like “aggressive” or “competitive” when describing men, vs “supportive”, “nurturing” when describing women. Are you surprised that an older person has cutting-edge technical skills? Ask yourself if an assessment might be colored by an applicant’s demography.
Don’t check out the candidate on social media, or Google them. A person’s public profile almost certainly won’t have anything relevant to work, and might instead reveal all sorts of irrelevant information (age, gender, political affiliation, race, etc.). If the candidate application/resume links to a personal website, LinkedIn, or GitHub you can check those out — these are more work-focused, and we can assume a candidate has put what they want an employer to see there. If you use Chrome, the Unbias Me extension (written by Fureigh) can help by removing avatars and names on some websites such as LinkedIn.
If you come to a conclusion about a candidate very quickly, before you’ve read the whole resume or finished the interview, spend the rest of the session trying to disprove that conclusion. Snap judgments are much more likely to be prone to bias than considered ones. We tend to jump to conclusions and then look for evidence to support our hypothesis. To compensate, if you find you’ve reached a Yes/No conclusion very quickly, spend the rest of the session trying to disprove that hypothesis. Explicitly look for evidence that you’re wrong. If you’ve decided immediately that a candidate is not qualified, spend the rest of your time trying as hard as you can to find evidence that they are qualified.

Purpose

The purpose of the debrief meeting is to:

Share and discuss interview evidence and data. Each interview has a different purpose and goals, and collects data about different aspects of the candidate. When this data is combined, it creates a more full view of the candidate and allows us all to spot discrepancies in our understanding. Do not share your interpretations, thoughts, or judgments about the candidate’s actions until after you have presented the data and you are discussing your scoring.
Create a single, unified summary of a candidate, resolving any discrepancies in interview data. This summary will be used by the 18F Engineering Hiring team to make a final selection.

Before the meeting

If you haven’t already done so, fill out the feedback form from your interview calendar event. This gives you the opportunity to reflect on the candidate individually, given only the data you were able to gather in your interview.
Review your interview notes, and be prepared to present a 10 minute run-through of the interview using the candidate’s own words. Include in your summary any information that you used to determine your scoring. You’ll have an opportunity to add more information after hearing from other interviewers in case you determine something important was left out.

Review the goals and purpose of the interview you preformed, and be prepared to rate the candidate on a scale of 1 – 5:

❌ 1	…	✅ 5
The candidate showed large skill gaps or lacked some crucial ability to do the job.	…	The candidate showed great skills and ability to do the job.

For candidates you did not rate a 5, what were you looking for that would have increased the candidate’s score?

When considering your score, you will not have other information from other interviews. That’s okay! The purpose of the score is to get a rough sense of alignment on each aspect of the candidate’s experience — for example, a candidate who scores 5’s and 4’s tells one story; a candidate who scores all 5’s with one interview scoring a 1 tells another story.

In our debrief meeting, we’ll discuss the scores and any discrepancies between them — the scoring system is designed to spark discussion, and the scores are not used mathmatically to make hiring decisions.

If you find yourself wanting to adjust your score during the debrief meeting, please bring it up and say why! Resist the urge to adjust your score because you are an outlier ("Everyone else said 5! I probably should increase my score…" — we trust your judgement, and you may have seen something others did not!) but it is more than okay to adjust your score if you have a concern which data from another interview addresses.

Conversation outline

Begin by introducing the meeting:

Thank you for using your valuable time to help interview candidates! You are directly impacting the future of our organization, and we can’t say thank you enough for your assistance. Thank you, thank you, thank you!

Please share your interview evidence, in order of the interview loop with the candidate. Please focus on evidence and data — what the candidate said and did — and not your interpretations, thoughts, or judgments about the candidate’s actions just yet. Try to stick to 10 minutes or less.

Once you’ve shared, please rate the candidate on a scale of 1-5 with 5s being “the candidate showed great skills and ability to do the job” and 1s being “the candidate showed large skill gaps or lacked some crucial ability to do the job.”

For candidates you didn’t score a 5, what would you have liked to see to increase their score?

Recap each interview, in interview order

For each interview, the interviewer reviews their notes and presents a summary of the interview, using the candidate’s own words whenever possible, for approximately 10 minutes each. Then, they present their ranking (1 – 5) and what could have been done to improve the score.

As other interviewers are presenting their summary, pay attention to any discrepancies in what you believe to be true about the candidate, or anything that catches you off guard.

Open the floor for discussion

After all interviews are complete, open the floor for any additional discussion:

Now that you’ve heard everyone’s interview debrief, would anybody like to change their scores, or provide additional comments and thoughts?

Wrap-up the meeting

If you are unclear of anyone’s stance on the candidate, you may ask:

Given what you’ve heard, would you personally advocate for having this person on your team?

After the meeting

Thanks for your help in the interview process! It’s critically important to the future of our organization, and your help is very much appreciated.

Please retain your notes until the final selection meeting meeting has finished, in case there are discrepancies or further questions that come up.