How are the results of technical assessments validated?

VIDEO Q&A (2:07 min)

Eric Bakan leads the Triplebyte Machine Learning team.

Video Transcription:

How are the results of technical assessments validated?

ERIC: I'd say that there are a couple of approaches to this notion of validity. So one is what I think of as "internal validity" in, sort of, psychometrics parlance. So that's: how internally consistent is your test? If you have five questions that are all about algorithms, do candidates tend to get a lot of them right if they're really good at algorithms, or none of them right if they're bad at algorithms? And so you can start to take a look at how good are you at replicating your results if you apply a bunch of questions to different candidates?

Second is this notion of face validity. So if we say questions about algorithms or loads on the algorithms factor, is it actually the case there? And the way we've done that is by having subject matter experts and senior engineers, whether that's people like me or people that we're working with externally, come in and say "yeah, we're writing questions about algorithms and we're labeling it that for these reasons." We can point to some evidence that we're testing this specific area.

But then there's this notion of external validity. So even if you label all these things as "algorithms," and you're pretty good at measuring what you call "algorithms," maybe that's not what Google says. Maybe Google actually says "oh, we care about this specific flavor that's maybe slightly different." And the way we test that is by seeing how well the candidates that we interview do at onsites. So we say "ok, all these companies passed or failed this candidate, and they gave this feedback on where they thought they did well, whether that's coding or algorithms or architecture." Then we can go back and calibrate our interview process, and from that our quiz questions to say "ok, are we actually measuring what the industry thinks is algorithms, properly?" So it's sort of a different sequence of steps that we think about in assessment development.

Our assessment process today is not designed to, and not going to, replace your onsite end-to-end. Even in the long-term, if we're perfectly good at understanding a candidate's technical ability, there's still a lot more that goes into fit for a role or fit for a team, than just someone's raw technical strength. And so we think about how we can replace other surrogate measures of skill, like where you went to school or where you've worked, or we try to replace other parts of the interview process, like the technical interview. We can do a pretty good job of tech screening; better than a lot of companies. 

Please Note: Current Triplebyte company users must create an account and log in to see the complete Company Help Center. Some general content is visible in logged-out mode, with additional content available for logged-in users. Learn how to set up your Company Help Center account.

If you have additional questions, please let us know.
Screen-only users: please contact Screen Support.
Triplebyte Hire users (Professional & Team): please contact your Account Manager or Company Support.