Hiring is Like Getting Hitched
Here’s how we do it in the Human Centered Design group at NASA JPL.
I’ve often said that the traditional hiring process in the US is like going on one date and then getting married. It works out sometimes. But it’s fraught with risk and uncertainty.
Each new person you add to your team is like adding another partner to your marriage. They bring with them skills to contribute. They bring values and priorities of their own. And these skills, values and priorities have the power to change your culture — for better or for worse. In hiring, you need to think about all the people on your team, and make sure you are adding partners who will uplift and support the people you have, not drag them down.
I’ve been hiring designers and developers into companies for *mumbles about being old* 23 years now. I have seen those candidates who could talk the good talk but then when you hired them to do the work, couldn’t deliver. I’ve seen the people who claimed to be experts in stuff they turned out to have only googled. That’s why companies give code tests and sample projects.
But code tests and sample projects seem too much like asking people to work for free, to me. Once a company asked me to do a sample project and on my first day working for them, I learned that my sample design was live on their site. So offensive. Such a way of devaluing the very valuable work that designers and developers do.
Additionally, there’s not always rigor in the traditional hiring process. How do you objectively compare one candidate to another, unless you’ve asked them all the same questions and posed all the same hypotheticals to them? And what’s more: even if you ask them the same questions, the stories people tell don’t tell you a lot about what they’re actually capable of in the day-to-day.
Don’t get me wrong — those questions are important and helpful. They do indicate how the person thinks, and they show you how the person is able to communicate about their work, which is key for any designer/developer who has to work with anyone outside their team.
But at NASA JPL, we wanted to do this differently.
The team of designers I work with is one of the best in the world, in my not-at-all humble opinion. We talked over the best way to bring the same rigor and collaboration that we use in our design process, to our hiring process. We asked ourselves: what if we treated hiring like it was one of our projects, with a research plan and interview protocols and tasks?
We came up with a plan, and tried it on our next candidate. At the end of the day we did a retrospective — including the candidate in the discussion about what worked and what didn’t. We iterated on the process, and did it again with the next candidate. Another retro, another iteration. In fact, we’ve now used this process at least 20 times and we continue to improve it every time. Considering how amazing the people are that we have hired (and helped other teams hire using this process), I feel pretty great about it.
The plan
For each role, we write a phone screen interview protocol. Every candidate gets asked the same questions which are unique to the role. This weeds out people who don’t have the right communication maturity or experience, if that’s what the role needs, and it lets us start to get to know the candidate.
We also use the phone screen to explain our unusual hiring process to the candidate, and give them a chance to ask questions about the role. Our work is hard, and we never want someone to be surprised when they start.
If they pass the phone screen and they haven’t been scared away, we bring them in for an onsite interview. Or, in Pandemic Times, a virtual all-day interview.
In this full day, we work a problem, together, from end to end. We don’t ask the candidate to do the work by themselves, we ask them to work with us. It’s not a real problem, and we’re not going to use their work later. It’s a “toy problem”, but we’ve designed it to be representative of the type of design challenges you run into at JPL, simplified enough for someone to understand quickly. It’s also designed and scoped to be just small enough to rush through, but just big enough to test the full gamut of skills we need.
Every candidate gets the same toy problem. This way, we can compare each candidate’s approach, strengths and weaknesses against each other. We also send the toy problem materials to the candidate ahead of time, in case they’re the kind of person who likes to think things over in advance.
One day is a very short period of time to work a crazy Space Problem from end-to-end, but we’ve created shortcuts and we help the candidate out in every step. We want them to get a sense of what it’s like to work with us, as much as we want to get a sense of what it’s like to work with them.
We ask the candidate to lead us through the creation of a research plan, if senior, or the writing of a single interview protocol if junior. Then we role-play the stakeholders and SMEs that the candidate says they’d like to talk to as they do “research” — sometimes including real experts in the mix. Afterward, we switch roles to “fellow researchers” and pretend that we also did research on the topic. We synthesize with the candidate, filling in any gaps they might have missed so that the rest of the day goes well for them. Next, we work with the candidate to design three solution concepts, followed by the creation of a paper prototype. And finally, we ask the candidate to do some evaluative testing with their paper prototype, and ask them what they’d change about the design.
With each role and each level of candidate (Junior, Intermediate, Senior, Very Senior) we have a set of evaluation criteria, and as we work with the candidate through the problems each interview team member is looking critically to understand where the candidate fits in terms of these criteria.
One person spends the entire day with the candidate, so that they can provide continuity for both the candidate and for other team members who come and go as their schedule allows. Full disclosure: That’s usually me. We use a private slack channel to check in with each other so that no one is asking the same questions someone else asked, and so that if there’s something to be investigated further we can coordinate.
At the end of the day, we do the retrospective with the candidate, and send out a survey to everyone who met with them. This is important to capture gut reactions. The survey asks a standard set of questions based on the needs of the role.
The next day, we convene as a group to discuss the candidate. Sometimes one person’s opinion about something they saw in the candidate can be changed when they hear about what someone else saw. This gives us the chance to share with each other across a broad set of perspectives and experience. In this round table discussion, every person’s opinion has the same weight as every other person’s, and we take feedback from people in all disciplines — not just designers.
We then make the decision whether to hire that person, or interview the next one.
The result
Let’s be clear: this is a grueling, exhausting day for the candidate and the interviewers. However, we’ve found it’s ultimately worth it. The candidate leaves having spent a whole day noodling a fun problem like no other they’ll ever experience, and they know exactly what it’s like to work with us. We get a definitive sense of what level they can operate at in terms of research planning, research methods, design and prototyping. We know exactly what kind of projects we can put them on, and whether they can work alone or need a mentor to help them.
We also get a sense of what the person’s strengths and areas of improvement are. We don’t need them to be perfect at anything — one of our best hires did pretty terribly in the time management aspect of the day, and that was fine. The approach and the thought process they were using was solid, and it didn’t matter that they didn’t finish everything. We hired them and they performed like a rock star.
We also have the opportunity to see how the candidate takes feedback and works with constraints. We have a couple of carefully timed and planned “monkey wrenches” that we throw in, to see how the candidate adapts. This is fairly realistic for a JPL project — we’re working with such a large and complex domain that it’s impossible to plan perfectly. Anyone who succeeds here has to be nimble and flexible.
The drawbacks
Our hiring process by nature selects for people who think well on their feet, are willing to show messy or unfinished work, and can handle time pressure. This means that a designer who is perfectly great, but works slowly on their own, will not generally make it through the gauntlet. We’ve discussed this, and for the moment it’s ok. In the current UX maturity of JPL, designers who don’t fit that mold generally are miserable and leave. So, for the moment, but not forever, it’s ok.
Our hiring process might not work for extremely shy designers. I think about brand new, unsure-of-themselves kids, who are terrified and intimidated by being thrown in the deep end with a bunch of experienced pros. We have a culture of reassurance and mentorship which may help alleviate a little of this, but putting myself in those new grads’ shoes, I don’t know that we go far enough. We’re still talking amongst ourselves about how to do better in this arena.
In the end
Hiring is the most important thing a team does, besides their work. In the design world at JPL we feel it’s important to involve the whole team in evaluating and selecting new hires, so that we get a diversity of perspectives. We also feel it’s critical to see firsthand how a person approaches a problem, rather than just hearing about hypotheticals or past successes.
Whether you’re reading this because you are thinking of interviewing with us, or because you are looking to do a better job of hiring at your company, I would love to hear your feedback. And if you try this technique, please let me know how it goes!