Drawbacks of Technical Interviews

Risks When Recruiting Software Engineers

After engineering systems in Human Resources Analytics at the world’s largest hedge fund and more recently in Talent Analytics at another >$1B company, I’ve learned a lot about assessing productivity and how to ensure that a team’s recruiting strategy best aligns with the broader goals of the company and does not succumb to the all-too-common pitfalls. 

(Of course, every company is different, and so I’m curious to hear your own perspective regarding your organization.)

Technical interviews (especially live coding, and especially when on in-browser apps such as CoderPad, but even in an IDE) can be fraught with potential flaws that could make them an inaccurate predictor of success in the role (and even timed quizzes—such as hosted by HackerRank—suffer from many of these flaws):

• Technical interviews block candidates from the habits that great engineers cultivate so that they are productive even when they get stuck. Great engineers habitually go for a walk or meditate or do whatever leads them to the insight they had been missing (which is impossible during a quick live interview).
  • Similarly, engineers are rarely working on just one problem, and so the best ones know how to productively work asynchronously in teams, crafting a targeted question to an available resource and then working in parallel on a separate project even before coming to an insight (or receiving advice) on the first.
• They are usually biased towards candidates with certain personality types (e.g. extraverts who think out loud).
  • Filtering out candidates who don’t thrive in a live coding test can reduce valuable diversity of personalities (and styles of thinking) within a team.
• Live coding tests tend to be quite unrepresentative of what the actual work in the role will be.
  • For example, if the test or interviewer forbids a candidate from referring to code she has written in previous projects or from Googling, using StackOverflow, etc, then engineers are being measured against a standard that is not relevant to the real work.
    • Actually, even if the interviewer encourages using other sources, it still won’t feel the same as “real life” and will feel discouraged implicitly.
  • Similarly, the focus of live coding tests (such as ability to determine an efficient algorithm) is often a mismatch to what the majority of the responsibilities and opportunities of the role will be (and sometimes a 0% match).
• They pose challenges that won’t be part of normal daily life:
  • Most engineers are acclimated to using the full width of their screen (usually across multiple monitors) and working in an IDE (powerful text editor) that is familiar and is geared to enhance their productivity. They aren’t used to working in text boxes that are only a few inches wide and are intentionally locked down to handcuff/stifle their productivity.
    • (Copying and pasting is disabled? Really? And tests really take this long to run?)
  • Most engineers set up their own environments with autocompletes, hotkeys, aliases, and other scripts specifically so that they don’t need to remember commands and syntax.
• A candidate’s success in such a test depends in large part on whether she is subject to performance anxiety (instead of peak insightfulness and productivity) in an environment where someone else is scrutinizing in real-time (and a clock is ticking down), even though very few real-world roles present such conditions often, if ever—even those with fast-paced deadlines.
• The trend of recruiters relying on these types of esoteric challenges creates perverse incentives.
  • Many candidates spend lots of time and money on study guides and programs teaching about obscure concepts rather than gaining experience in real-world scenarios. (Often they even purchase company-specific interview guides!)
  • Unfortunately, then even more candidates feel pressure to act similarly just to “keep up with” candidates making those choices.
• Blank (or nearly empty) screens are not similar to real-world experience.
  • Coding multiple different languages in the same day is a skill that is useful or even required in many jobs, but in those real-world situations, engineers have the benefit of context.
  • For example, imagine a senior engineer who knows many different languages. If he/she spent recent hours or days working in certain languages, those syntaxes are the ones most front-of-mind. Normally, in a real project, he/she could easily switch to working on another language, given that many existing functions of that language would be visible in the files of its project. But in a coding test where the screen is nearly blank, none of the typical reference points are available, which poses a silly, artificial challenge.

So, these points can be helpful for hiring managers to keep in mind since technical interviews are subject to producing “false negatives”.

I wonder what your thoughts are.

What others are saying:

Our industry has gone off rails when it comes to hiring engineers.
I was a hiring manager at Facebook and Amazon and interviewed hundreds of engineers for big tech companies.
I hated the way interviews were conducted there because the interview process did not assess real experience.

https://sagivo.com/hiring/engineers/faang/interviews/2022/02/09/there-must-be-a-better-way.html

The trouble with technical interviews? They aren’t like the job you’re interviewing for. Forget the coding exercise. Here’s how to create realistic scenarios for engineering candidates in technical interviews.

https://about.gitlab.com/blog/2020/03/19/the-trouble-with-technical-interviews/

Candidates who spend more time honing their skills in the domain of solving traditional coding problems, and less time building real things in the real world might have an advantage.

https://medium.com/hackernoon/why-coding-interviews-still-suck-aa189e3b6c93

I’m a senior software engineer with 6 years of experience in React & node.js. I have built products that are used by millions and have generated $billions in revenue. I’ve worked at successful startups. But you won’t hire me because I can’t solve an algorithm challenge. lol.

@farazamiruddin https://www.linkedin.com/posts/shashank219_hiring-interviews-activity-6843052009696878592-VLht
https://twitter.com/farazamiruddin/status/1436837939842007044

Live coding isn’t the meritocratic space that it pretends to be. Live coding interviews weed out the people who are good at live coding interviews.

https://www.zamiang.com/post/why-i-don-t-do-live-coding-interviews

I’m a professional engineer with 40 years’ experience writing (or managing or supporting) software – 12 of those years at Microsoft.

I have a recent masters degree in Software engineering (2016) and 14 software patents.

I’ve written two compilers for proprietary languages (both in the 80s and 90s).

But, if you get me in an interview and want me to spin off code for some algorithm I’ve never used before (but, that your folks can do in 20 min), I’ll fail miserably. I’ve interviewed devs for decades – that’s a terrible way to decide.

https://workplace.stackexchange.com/users/93107/flydog57

P.S. A couple more thoughts.

1. Even so-called “collaborative exercise” interviews are flawed. Candidates cannot truly act as if they are working on a team collaboratively and are learning and contributing and producing something great collectively; they are being tested the entire time.
   
   Imagine, for example, a romantic relationship where one person “tests” the other—it’s a recipe for disaster.
   
2. A recruiting process should be a two-way conversation, where the candidate is learning about the company and gathering insights that help him/her make a decision about whether it feels like the right fit.
   
   Technical screens are almost entirely one-sided. It’s a turn-off when companies assume that every candidate ought to want to work there (and therefore skew the recruiting process to be unbalanced, where the candidate needs to jump through more and more hoops).
   
   It likely reveals beliefs held by the leadership of the company that will permeate employees’ experiences down the line too (e.g. that the employees ought to feel lucky to be there rather than it being a mutually beneficial relationship).

P.P.S. Alternative approaches

Especially nowadays with work being so global and asynchronous, communication (especially written communication, such as emails, Slack messages, comments on documents, etc) is even more important.

It’s a substantial part of the job.

Very few organizations do enough to assess which candidates are excellent communicators.

I recommend that hiring managers (and their recruiting processes) don’t just ask themselves “Which languages can this candidate code in and how well?” (which is actually a smaller portion of an engineer’s job than many people realize, especially for a manager).

I try to assess more holistically the different parts of what it means to contribute to an organization as an engineer.

To gauge ability to write thoughtful, clean, testable, maintainable, accessible code, I’ve had success with:

• interviews where I ask about their coding practices and how they think (but without live coding and without whiteboards)
  
• candidates walking me through code that they’ve already written (as long as it’s not proprietary)
  
• reviewing their open-source work
  
• short take-home exercises (and these can even have a follow-up interview where I ask about decision-making, how to handle risks and new feature ideas, etc)
  
• short paid projects
  
• trial periods (e.g. 1 week of moonlighting after work)
  
• asking others

(The above are also helpful in general for assessing other ways that an engineer or manager would make an impact, such as zooming out and architecting systems well and also thinking in a principled way and viewing the organization as a “machine” that ought to be tuned to improve its outputs in a repeatable way.)

I see the ideal situation as when a company:

• creates a following of engineers (and other roles) who are interested in working there, and stays on their minds via tech blog posts, newsletters, social media, personal connections, etc.
  
• doesn’t view recruiting as “here are a bunch of hoops for candidates to try to jump through to impress us” but more as “we’re looking for the best possible partners for our future journey”
  
• operates with full transparency and good will
  
• has a long-term view (which means they want to create such an inspiring experience for all candidates that they all want to apply again even if they don’t end up here this time)

I love talking about management and recruiting but will stop myself here!

I’m curious to hear your reflections.