How do you assess engineers?

Engineers complete various adaptive online quizzes (most quizzes are multiple-choice and some require actually writing code). Trained on over 300,000 engineers, the Triplebyte evaluation uses machine learning to identify talented candidates.

We've developed a set of questions and a well-defined rubric to ensure standardization across candidate scoring. The quizzes ask each candidate different questions from our large pool of potential questions, based on where they have shown strength and weakness. By doing this we are able to get more accurate results in a shorter amount of time. The statistical model we use for our quiz (item response theory) is similar to what is behind the GRE and GMAT tests.

Most of the quiz questions involve looking at blocks of code and answering questions about them (finding bugs and predicting behavior). Some questions test specific knowledge.

VIDEO Q&A (2:46 min)

Rachel Wolford is the Product Manager for the Triplebyte Assessments team. Eric Bakan leads the Triplebyte Machine Learning team.

Video Transcription:

What is your approach to assessing engineers?

RACHEL: We are rooted in this empirical approach. We have downstream data for some candidates, not for all of them, and we build models that generalize that downstream data to candidates where we don't have downstream data. So, we know that this past candidate passed an interview, we know roughly how they did on our individual questions, and we can calibrate our expectations about whether [...] another candidate [with similar scores] would pass an interview at a different company.

Any interviewer has a notion that the questions they're asking are proxies for certain skills that they're interested in. And our approach is no different, it's just based on a much larger dataset, where fundamentally what we're trying to do is extract the strength of each proxy for skills that we're interested in, and calibrate that in a single, meshed-together way.

We don't just say, "Well, they did well on this question, and poorly on this question, well on this question, and poorly on this question... I guess it's a wash." We don't say that. We have a sense of how much should we adjust our expectations upward or downward from each question based on this downstream data. We know that some questions are more predictable than others, and some questions are harder than others.

ERIC: And I think the thing that Ammon [Triplebyte CEO] likes to throw in there as well, is that we're aggregating results from hundreds of companies. We're not just looking at how well do candidates do at Google's interview or Facebook's interview, but looking at all sorts of companies of all different sizes and stages. You find that companies interview very differently and candidates who are successful at one org are not going to be successful at another. You can start to get a sense of the scope of things that companies care about, and then you can present that information. So when companies know that they care a lot more about system design than algorithms, they have access to that data, and if a company cares a lot more about coding ability or low-level systems, they have measures on those fronts as well.

RACHEL: Success in an interview is dependent both on candidate-specific factors and company-specific factors. And we don't, and are not trying, to get everything from the company-specific side. We assume that companies know what they're hiring for; we assume that their interviews will cover the content that they care about. Our job is to cover the content that everybody cares about: the common factors across all different organizations, or most different organizations, that allow us to extract information that's relevant to everybody. We expect that companies will extract different signal that's relevant to their specific industry, and we think that's good. We think that having our approach not completely match every internal approach is actually beneficial, because you get these different sources of signal.

Please Note: Current Triplebyte company users must create an account and log in to see the complete Company Help Center. Some general content is visible in logged-out mode, with additional content available for logged-in users. Learn how to set up your Company Help Center account.

If you have additional questions, please let us know.
Screen-only users: please contact Screen Support.
Triplebyte Hire users (Professional & Team): please contact your Account Manager or Company Support.