Most companies don’t have a serious, repeatable interviewing process for hiring.

Instead they wing it, bringing people in for interviews, asking a few questions, turning it into a fireside chat, and then hoping for the best. In my experience that’s not good enough.

At Standout Jobs (2007-2010) we hired some great people. I still consider them friends today, but more importantly they’ve all gone on to bigger and better things. And many of them still say that their experience at Standout Jobs was a fantastic one, even though the company wasn’t a success. They learned a lot, grew as people, and found teammates that they genuinely respect and appreciate. A few of the guys went on to work together in other projects and startups, and I’m sure some of them will re-connect in the future.

That feedback is a testament -in part- to the effort we put into hiring and interviewing people. I’ve seen a lot of different hiring strategies, from the nonchalant to the super intense (even more than our approach, which I’ve shared below.) I can’t say there’s one approach that works for every company, or that our approach is even “the right one” for you, but I do know that everyone we hired that went through the process was successful and stayed with us; those that we bent the rules for (often to expedite the hiring process) didn’t pan out quite as well. So for us, the interview process we implemented was a very good filter.

Note: Most of the hiring we did was in 2007-2008. That’s a few years ago, and my memory of every specific detail is a touch fuzzy. I wish I had kept more notes on the specifics, questions we asked, etc. There are those out there that may be able to fill in some details…

1. Reviewing Resumes

The process typically started with a resume. The resume isn’t dead, but the format should change, especially for developers that have plenty of resources (like Github) for sharing their work and demonstrating their capabilities. Resumes still contain some nuggets of information that are helpful, and we went through them all. Here are some of the things we looked for:

• Spelling errors (almost an automatic “thanks but no thanks”)
• Lots of short hops between companies (a potentially bad sign)
• Open source contributions, side projects or other work shared in a portfolio
• Twitter account and/or blog (so we could take a look at what people were interested in, talking about, etc.
• Previous startup work and the quality of those startups

Both my co-founder (Fred Ngo) and I reviewed resumes. In some cases we shared resumes with other team members to get their opinions as well. If we were rejecting someone, I’d email them. Every single one. I did my best to make sure everyone that applied heard back from us at least once. This is a tough process, but I encourage every company to do it. After reviewing and filtering resumes, we’d screen the remaining candidates over the phone.

2. Screening Candidates

We put a lot of effort into screening candidates. Given that the next step (in-person interviews) lasted almost a day, we didn’t want to go through that with too many people.

Screening candidates was mostly about:

1. Verifying the validity of their resume (i.e. how much did they exaggerate?)
2. Testing for role fit (i.e. do they understand the role, and is that a good fit?)
3. Testing for cultural fit (i.e. do we think we’ll get along well?)

I tried keeping screening calls to 30 minutes, but they’d often go longer. It was always interesting to see what people talked about (specific work they’d done, accomplishments, interests.) I also gave people an opportunity to ask questions, which was also instructive on their personality and what they cared about.

We had a standard set of fairly open-ended questions, but I wasn’t too concerned if we got through all of them. The goal was to have a conversation and see how it went. Some of the questions I asked included:

• Tell me about [pick something on the resume]…
• What did you learn at [pick company name]?
• What are you looking for in a new company?
• Tell me about the [pick a side project s/he’s worked on]… (or: Have you ever thought of doing a side project of some kind? If so, why haven’t you? What would you do?)
• What’s an ideal work environment for you?
• Do you read Hacker News? (i.e. Are you genuinely interested in startups and new technology?)

Some of the gotchas I came across:

• Lying. A lot of people lie on their resume. They drastically overstate their contributions to things that were successful. You expect some exaggeration (or even lack of clarity because it’s hard to measure how valuable a specific person’s code was to a project’s success, for example) but lying is a no-no.
• Dissing past / existing employers. There are a lot of crappy employers out there, and a lot of crappy jobs, but spending 30 minutes complaining about them isn’t the right approach. It’s better to be more politically correct, tease out the positives (you must have learned something, met someone, had your eyes opened up to different ways of doing things) and move on. Badmouthing people isn’t going to win you points with potential new employers.
• Not having any questions. A candidate without any questions is most likely a disinterested candidate.
• Not having done any research on the company. Unfortunately a lot of candidates barely looked at our website when we screened them, so they really didn’t know what we were about. When you’d ask, “What interests you about Standout Jobs?” and they didn’t have an answer, it was definitely a bad sign.

Once screening was complete, we’d decide if the candidate should come in for the in-person interviews. If not, we’d let them know. If so, we’d ask them to block off half a day or more and we’d explain the process to them (verbally or via email) so they’d be somewhat prepared.

3. First In-Person Interview (with me)

I typically did the first in-person interview. Similar to the screening process, I’d prepare a list of fairly standard questions (with some deviation based on the specific person I was speaking to) and we’d spend 1-1.5 hours talking. Admittedly, these conversations got too casual at times, and I caught myself talking more than listening, but generally I found them constructive. You could probably be more rigorous here than I was, and make this interview shorter.

Ultimately, I was really looking for a personal connection. Did I want to bring this person onto my startup roller coaster and ride with them? It wasn’t about being friends (while I’m friendly with many ex-employees, we didn’t spend tons of time outside of work socializing), it was about finding a mutual connection and respect. You are going into battle with them, and a lot of it will get ugly, so there better be a real sense of camaraderie and understanding.

4. Second In-Person Interview (with Fred)

Next, Fred took over and did a more technical interview with candidates (assuming we were hiring programmers). If we were hiring for other positions, Fred still did an interview but didn’t ask technical questions.

Fred had his own style for interviews. He asked theoretical questions, math questions, and worked through different problems with candidates. He also spent a good chunk of time getting to know the candidates, how they communicated, what interested them technically, etc. So it was a combination of practical (does this person know what they’re talking about) and personal. These interviews typically lasted 1 hour.

5. Third In-Person Interview (with the team)

The next interview for a candidate was with a few members of our team. We wanted to give our existing employees a chance to meet prospective employees. This usually lasted 30-60 minutes. The discussions were broad ranging, because the team loved to talk technology and they often found candidates eager to discuss technology as well. I didn’t participate in these meetings (neither did my co-founder, Fred) so I can’t really go into a lot of detail, but I do like the idea of having candidates meet the team. That’s who they’ll be working with every single day. And the truth is that a lot of employees stay at a particular job because of their co-workers, even if they don’t like the job, the company or their boss. That’s how important it is for people to really bond with their co-workers.

6. The Practical

After three in-person interviews (now 2.5-3.5 hours in), we’d move to the practical portion of the interview. We used to call this a “technical test” but it really wasn’t about testing technical capabilities. Candidates coming in for an interview knew there’d be a practical component, but prior to the interview we didn’t provide any additional detail, except perhaps, “we’ll get you to code something.”

Note: This is for developers and technical people, but we did run others (like customer support people) through a practical exercise as well. I’ll share some details on that later.

Here’s how we designed the practical “test”:

(i) The Setup. Fred and I would tell the candidate that we were going to roleplay a scenario, and ask them to participate. At some points during the practical we’d be roleplaying, at other times we’d step out of that and review things. We’d explain the general steps to them – roleplay the initial scenario (where we’re a client asking the candidate to build an app), they’d design the app on paper and present it (partially roleplayed), they’d go build the app (not roleplayed), and they’d present the app after (partially roleplayed.)

(ii) The Scenario. Now it was time for a bit of roleplay. Fred and I would pretend we want the candidate to build a photo sharing application for our university dorm. We picked this application for a number of reasons:

1. Everyone understood what a photo sharing app was and generally how it would work (we weren’t trying to invent something radically new here)
2. It was a simple enough application, but it had a bunch of important “web” elements (login, user accounts, content management, etc.)
3. People felt like they could build it very easily, which proved interesting later on

Fred and I would tell the candidate, “We want a photo sharing app for our friends in our university dorm,” but wouldn’t provide a lot of details. It was up to the candidate to ask questions, get information from us and propose ideas. Imagine a scenario where you’re doing work-for-hire with a “less than brilliant or clear” customer. That was the scenario.

Ultimately we were testing communication skills and critical thinking. What questions did the candidate ask? What assumptions did they make? How did they approach communication with us as a client? Did they embrace the experience or not?

We learned a lot during this phase.

It was amazing to see how some people dove into the scenario and started designing UIs on paper with us, while others didn’t ask very much at all and said, “OK, I’ll go build a photo sharing app.” In some cases we had to prompt candidates with specific cues: “Um, you didn’t ask us about privacy.” Or Fred and I would start talking to each other, “Do we want everyone to have access? Should people be able to delete photos? What happens if someone starts posting too much junk?” Good point, good point…then we’d wait to see if the candidate jumped in.

We were also looking to see how people prioritized the steps and things they could do, and how they communicated that to us. Some people were great at negotiating: “So is A or B more important? A is easier, but B might do more of what you want.” Some were great at brainstorming new ideas and proposing stuff, which we loved (as “clients” and employers!)

Once we were finished with this process (~30 minutes), we’d break out of roleplaying and move to the next step.

(iii) The Design. At this point we’d ask the candidate to design the system on paper. They’d often ask, “What do you want to see?” And we’d respond, “It’s up to you. What do you think would be helpful for explaining how you’re going to build this for us [as the roleplayed customer], and what do you think would be useful for you as the developer?”

We saw all kinds of things here, from UML to UI mockups. Some people had reams of paper and they’d walk us through the entire application they were proposing to build (soliciting feedback the entire way), others had one or two sheets. It was very interesting to see how people think, structure things, and again, how well they communicate.

Candidates had 30 minutes. At the end we’d jump back into roleplaying so they could present their work; to help us (as “clients”) understand what they were going to do. We’d come to a conclusion with them on what the deliverable should be. Then we’d move out of our roleplayed roles and ask them about their approach, why they did certain things, etc.

Before they could jump into coding, we’d ask the following question:

“OK, so given the scope of the project, how much of it can you get done in 2 hours?”

(Sometimes we only did 1.5 hours depending on time constraints.)

This was one of the most interesting parts of the interview.

Universally, candidates would say that they could get all of it, or nearly all of it done. We’d dive into specifics and they’d give us a list of deliverables they were prepared to commit to. We knew they couldn’t complete all the work, but it was a “just small enough project” to give people confidence that they could. It set the tone: This is going to be hard, stressful and intense, but go for it! Most candidates were eager for the challenge.

Three points:

• Initially we were giving candidates a clean computer. It was almost completely empty, and they could set it up as they saw fit. The idea was to see how well they could stay focused on the task at hand versus setting up the machine to their perfect parameters. It made the situation a bit more challenging. After a couple interviews we stopped doing this. I don’t think it really provided us with a lot of valuable information about candidates, and the task was hard enough as it is.
• We always offered people more time if they wanted it, usually 30-60 minutes. Most people took us up on that offer.
• We always allowed people to ask us questions (either as the clients in the roleplay scenario) or not. If they had technical questions, or got stuck somewhere, they could come to us and talk. Very few did.

No one ever finished.

We knew this would be the case; candidates did not. And what came out of this experience with candidates was always interesting. Occasionally, a candidate would get almost nothing done, spending all their time setting up the clean computer, or just staring at the screen paralyzed. Others quit halfway through. In one particular case, the candidate admitted to us that he just wasn’t ready for a job at an early stage startup (there were personal things going on his life that were just too distracting) and so he bowed out. He was a very good developer, but the timing just wasn’t right. And that’s OK. He’s gone on to be very successful.

7. The Post-Practical

Once the practical was finished, Fred and I would get back together with the candidate and review his or her work. They’d show us a demo of what they’d built (if anything) and talk through their approach, decisions they made, and the places where they got stuck. It was always fascinating to see where people put their energy and limited time: some made sure it looked good even if the functionality was shallow, some wanted to get all the plumbing in place, and others got caught on specific (often minor) details and couldn’t adapt.

We always asked them, “Now that you’ve gone through the process, where did you underestimate the effort / work required?” Again, it was fascinating and instructive to hear people’s answers. We didn’t go back into roleplaying mode at this point, but it was important to see how candidates reflected on the conversations we had in those scenarios. How were they trying to address client needs? How did they shift on-the-fly, while keeping in mind how they would explain their decisions to the client? What assumptions did they make that they could then properly communicate?

We were always looking for clarity and adaptability. How they worked under pressure was certainly part of it, but not the only thing that mattered to us; after all, this was a fake scenario. Their ability to work in this scenario wasn’t completely indicative of their ability to work on a deadline, although I believe it was a decent proxy. You also got a sense for who had that special extra gear; the few people with the determination to plough through, take chances and be capable of clearly and confidently explain what they did after the fact.

Once the post-practical was done, the interview was over. All told, the candidate had now spent 5.5-7.5 hours with us. It was a gruelling, but effective process.

Conclusions

Hiring the right people is extremely hard. If you don’t take it seriously and put the effort into it, you’ll end up hiring sub-par people that don’t have the necessary talent or cultural fit. And as we discovered, timing is important as well; if someone isn’t ready to make the commitment needed, it’s better to find out beforehand.

The fact is you will never be 100% successful at hiring – some hires just won’t work out. There could be any number of reasons (most of which are out of your control), but you can reduce your risk significantly with a thorough interview process.

Here are some final thoughts about our process:

• Very few candidates disliked the process. I remember a few of them thought the roleplaying was silly, and some of them couldn’t understand why we were bothering with the whole thing, but most of them genuinely appreciated the effort. Even those that fell apart during the process understood why we were going through it in the first place. And many of the candidates were more interested in working for us because we put in so much effort.
• The interview process was the best way of presenting our company’s culture. Candidates got to meet most of the team, talk to them, learn about them, and then they were put through the gauntlet, and came out the other side with an appreciation for the intensity of the company.
• Technical ability is fairly easy to measure. Communication skills, adaptiveness, eagerness, heart, intensity and culture are much harder to measure and understand. We already knew (or were fairly certain) about a candidate’s technical abilities before going through the practical, but we didn’t know if they’d be the right coder for us.
• As I’ve said above, our process is just one way of doing things. I think it was effective, but it won’t work for every company. Those that performed the best throughout the interview were definitely some of our best employees. I suppose that could be a fluke, but I don’t think so. We filtered a lot of people out, and I know (now looking back with the power of hindsight) that we made the right decisions.

We also used practical tests when hiring for other positions other than developers. For example, we ran customer support candidates through a test (although a simpler one than above.) Candidates applying for a customer support role went through the same in-person interviews (with me, Fred and other employees). They would then spend 5-10 minutes with one of our customer support people. Our employee would walk candidates through our website and talk through our core benefits, as well as some of the frequently asked questions we received from customers. Candidates then had 5-10 minutes to prepare and were asked to present what they’d learned back to us. Watching a candidate present on the benefits of Standout Jobs was very helpful in assessing their communication skills, personality and ability to adapt quickly.

Some candidates had done some research in advance, and they often performed better because they were prepared. That was just fine with me. Customer support candidates were extremely surprised when we told them we’d go through a practical assessment, but it was definitely helpful for us.

Recruiting successfully is crazy hard.

The pool of talent is always smaller than you’d like, there are always competing interests, and it takes a significant time commitment to recruit consistently and well. When interesting candidates do come knocking, there’s a temptation to hire them quickly and see how it goes. I’ve done that before, and it usually didn’t work out. You get lucky sometimes, but you’re taking a big risk; not just with the one individual you’re hiring, but with the effectiveness and culture of the entire company. There’s merit in the saying, “hire slow, fire fast.” Don’t be too slow about it (or you’ll miss people), but you do need to be extremely deliberate and thoughtful about the process. Put the time into a recruitment and interview process as early as possible in your company’s existence, measure the results, learn and iterate as you go along.