An experiment in online technical screening for hiring software developers

 
exams.jpeg

With our latest Junior Developer role, we had 150 applicants so we were faced with a problem, how can we filter down all these applicants to a volume where telephone interviews or coffee shops chats become viable for a single interviewer while being true to our company values. 

I’ve heard some horror stories like running a lottery automatically rejecting half of applications, I don’t know if they are true, but they are certainly a long way away from our company values. It had been suggested that we use an online technical screening tool to filter the applicant list down, but we’ve always avoided using a technical screen tool for biases they introduce. Here are just 2 examples:

  • Time the candidate has available to complete the online assessment. People with family commitments, volunteering, and other commitments all have issues finding the time to complete an assessment. Some companies screening challenges can take between 2 to 4 hours, and some are even open ended on time.

  • Some people don’t perform well in time pressure or examination type environments, this has a particular impact on people who aren’t Neuro typical (Autism spectrum disorder, dyslexia, dyspraxia, mental health issues and many others) and suffer from imposters syndrome.  

We decide to experiment with online screening but looking at how we could adapt it to minimise the bias introduced. We settled on the following guidelines:

  • This won’t be a full technical assessment but just a very basic skills test as we have a technical interview (It’s not a takeaway task) later on in the process.

  • Include a short coding task and a set of multiple-choice questions so people had different ways to demonstrate their knowledge.

  • Should be able to be completed in a lunch break. We settled on a 30-minute limit for the coding task and 15-minutes for multiple choice questions.

  • They have one week to find time to do the assessment with extensions available upon request

  • Don’t use the automatic grading of the online tool or set a pass mark. We reviewed the code of every person to complete the challenge. We wanted to see what they know and give credit for working out, process and tiny errors that would been picked up in pair programming (This is a junior level role).

This screening assessment allowed us to filter the applicants down to 11 for video call or coffee with me, and this in turn filtered the applicants down to 4 for the final round interview that includes our full technical assessment, and 1 job offer was made.  

Just to demonstrate the issues with online technical assessment and their grading, the lady we offered the role to was ranked quite low by the tool in its default configuration, but our own process ranked her quite high during the final round. 

Conclusion

The comparison of the results between the tools grade and our own technical assessment demonstrates these tools can have bias issues and need to be adapted for use. This assessment was for a Junior role so the results and adaptation maybe different for a senior role. 

Our hiring process is constantly evolving, and this is one experiment is changing it to deal with a high volume of candidates while trying to create a level playing field all. I don’t think we have a solution yet and it’s something we are going to continue experiment with new ideas and improve. One change we will be making is to a create a document to explain how we conduct the assessment to help reassure people.  

I would be interested in hearing other people’s stories and ideas for improving the initial phases of the tech recruitment process.