Drawbacks of Technical Interviews

Posted by

Risks When Recruiting Software Engineers

After engineering systems in Human Resources Analytics at the world’s largest hedge fund and more recently in Talent Analytics at another >$1B company, I’ve learned a lot about assessing productivity and how to ensure that a team’s recruiting strategy best aligns with the broader goals of the company and does not succumb to the all-too-common pitfalls. 

(Of course, every company is different, and so I’m curious to hear your own perspective regarding your organization.)

Technical interviews (especially live coding on tools such as CoderPad) can be fraught with potential flaws that could make them an inaccurate predictor of success in the role (and even timed quizzes—such as hosted by HackerRank—suffer from many of these flaws):

  • Technical interviews block candidates from the habits that great engineers cultivate so that they are productive even when they get stuck. Great engineers habitually go for a walk or meditate or do whatever leads them to the insight they had been missing (which is impossible during a quick live interview).
    • Similarly, engineers are rarely working on just one problem, and so the best ones know how to productively work asynchronously in teams, crafting a targeted question to an available resource and then working in parallel on a separate project even before coming to an insight (or receiving advice) on the first.
  • They are usually biased towards candidates with certain personality types (e.g. extraverts who think out loud).
    • Filtering out candidates who don’t thrive in a live coding test can reduce valuable diversity of personalities (and styles of thinking) within a team.
  • Live coding tests tend to be quite unrepresentative of what the actual work in the role will be.
    • For example, if the test or interviewer forbids a candidate from referring to code she has written in previous projects or from Googling, using StackOverflow, etc, then engineers are being measured against a standard that is not relevant to the real work.
      • Actually, even if the interviewer encourages using other sources, it still won’t feel the same as “real life” and will feel discouraged implicitly.
    • Similarly, the focus of live coding tests (such as ability to determine an efficient algorithm) is often a mismatch to what the majority of the responsibilities and opportunities of the role will be (and sometimes a 0% match).
  • They pose challenges that won’t be part of normal daily life:
    • Most engineers are acclimated to using the full width of their screen (usually across multiple monitors) and working in an IDE (powerful text editor) that is familiar and is geared to enhance their productivity. They aren’t used to working in text boxes that are only a few inches wide and are intentionally locked down to handcuff/stifle their productivity.
    • Most engineers set up their own environments with hotkeys, aliases, and other scripts specifically so that they don’t need to remember commands and syntax.
  • A candidate’s success in such a test depends in large part on whether she is subject to performance anxiety (instead of peak insightfulness and productivity) in an environment where someone else is scrutinizing in real-time (and a clock is ticking down), even though very few real-world roles present such conditions often, if ever—even those with fast-paced deadlines.
  • The trend of recruiters relying on these types of esoteric challenges creates perverse incentives.
    • Many candidates spend lots of time and money on study guides and programs teaching about obscure concepts rather than gaining experience in real-world scenarios. (Often they even purchase company-specific interview guides!)
    • Unfortunately, then even more candidates feel pressure to act similarly just to “keep up with” candidates making those choices.
  • Blank (or nearly empty) screens are not similar to real-world experience.
    • Coding multiple different languages in the same day is a skill that is useful or even required in many jobs, but in those real-world situations, engineers have the benefit of context.
    • For example, imagine a senior engineer who knows many different languages. If he/she spent recent hours or days working in certain languages, those syntaxes are the ones most front-of-mind. Normally, in a real project, he/she could easily switch to working on another language, given that many existing functions of that language would be visible in the files of its project. But in a coding test where the screen is nearly blank, none of the typical reference points are available, which poses a silly, artificial challenge.

So, these points can be helpful for hiring managers to keep in mind since technical interviews are subject to producing “false negatives”.

I wonder what your thoughts are.

What others are saying:

https://www.zamiang.com/post/why-i-don-t-do-live-coding-interviews

The trouble with technical interviews? They aren’t like the job you’re interviewing for. Forget the coding exercise. Here’s how to create realistic scenarios for engineering candidates in technical interviews.

https://about.gitlab.com/blog/2020/03/19/the-trouble-with-technical-interviews/

Candidates who spend more time honing their skills in the domain of solving traditional coding problems, and less time building real things in the real world might have an advantage.

https://medium.com/hackernoon/why-coding-interviews-still-suck-aa189e3b6c93

I’m a senior software engineer with 6 years of experience in React & node.js. I have built products that are used by millions and have generated $billions in revenue. I’ve worked at successful startups. But you won’t hire me because I can’t solve an algorithm challenge. lol.

@farazamiruddin https://www.linkedin.com/posts/shashank219_hiring-interviews-activity-6843052009696878592-VLht
https://twitter.com/farazamiruddin/status/1436837939842007044

P.S. A couple more thoughts.

Even so-called “collaborative exercise” interviews are flawed. Candidates cannot truly act as if they are working on a team collaboratively and are learning and contributing and producing something great collectively; they are being tested the entire time. Imagine, for example, a romantic relationship where one person “tests” the other—it’s a recipe for disaster.

A recruiting process should be a two-way conversation, where the candidate is learning about the company and gathering insights that help him/her make a decision about whether it feels like the right fit. Technical screens are almost entirely one-sided. It’s a turn-off when companies assume that every candidate ought to want to work there (and therefore skew the recruiting process to be unbalanced, where the candidate needs to jump through more and more hoops). It likely reveals beliefs held by the leadership of the company that will permeate employees’ experiences down the line too (e.g. that the employees ought to feel lucky to be there rather than it being a mutually beneficial relationship).

Leave a Reply

Your email address will not be published. Required fields are marked *