The Tech-Talk Balance: What Technical Interviewers Expect from Technical Candidates

Ever wonder what the top companies expect from their software engineering applicants?

Advertisements

The increased demand for software engineers in the upcoming years and their impressive salary makes the job a popular one that is highly coveted among those in STEM. With so many resources such as interviewing.io, codassium, and countless blogs about how to prepare you would think that most software engineers would have the interview process nailed down, right? WRONG!

In fact, software engineer job candidates are not exactly succeeding at technical interviews according to interviewers. Although candidates are able to answer technical questions, there is a mismatch of what candidates think interviewers assess versus what criteria is actually used in practice. Unfortunately, this mismatch in expectations can cost candidates the job of their dreams. To connect these engineers to the job of their dreams and confirm what type of mismatches may exist we sought to find out what their interviewers are looking for.

Ultimately, we were curious if there are differences across companies for how software engineer candidates are evaluated. But we were also interested in how interviewers interpret criteria for software engineer job candidates.

To soothe our curiosity, as well as other job-seeking developers, we studied interviewers in their natural habitat.

How do we really know what interviewers are expecting?

Short answer is, they told us. With interviewers from 9 major software companies, we conducted 70 mock technical interviews with software engineer candidates at our local university to better understand what companies expect from them. We anonymize the names of the company, but we give you details on their industry in the table below.

Statistical and Qualitative Analysis of Evaluations. We also collected interviewer evaluations of candidates in order to understand the expectations of interviewers and clarify how candidates should prepare for a technical interview.

After the interviews, we analyzed interviewer expectations from their evaluations and outlined how candidates can prepare for future technical interviews beyond being technically sound. We compared the Likert scores (1 through 4) interviewers gave to candidates using a Fisher’s exact test across companies followed by a Post-hoc Steel Dwass analysis on statistically significant results (α< 0:05).

What did we find in our analysis?

Evaluation Scores. In our statistical analysis, we found that all post-hoc pairs identified C1_WEB, a large internet Search company, as being significantly different than any of the other 8 companies. We did not identify any other pairs of companies with significant differences in scores for each of the evaluation criteria. We show the correspondence analysis of company ratings of candidates in the chart below. The most outstanding difference shown in this diagram demonstrates how C1_WEB scored how well candidate gave clear and concrete examples.

Criteria Interpretation. We were also able to determine how interviewers interpreted criteria used to evaluate candidates.

Example: [Original Criteria]-> [Interpretation from Interviewers]

Problem Solving-> Algorithms: When hiring candidates for a job, the top concern is whether candidates have sufficient technical skills to handle problem solving.

Nonverbal-> Interest: Interviewers noticed when there was poor communication during the interview.

Oral/Verbal Clarity->Fluent Speech: When making a first impression, the first words a candidate speaks are often the most important.

Clear, Concrete Examples-> Connected Experiences: Another way of demonstrating a candidate’s fit is their ability to communicate clear and concrete examples.

Enthusiasm-> Visible Excitement: How a candidate displayed enthusiasm is one measure of interest and engagement in the interview.

What should you share about this work?

Interviewers care about technical soundness and the ability for candidates to communicate that skill. Surprisingly, technical interviewers place an emphasis on interpersonal skills and effective communication in the interview. Interviewers wanted to hire a person, not just a candidate who can solve problems.

Most companies have consistent expectations for candidates across industry and size. According to interviewers, candidates are prepared technically, but encounter challenges translating their technical knowledge. Interviewers identified that candidates were not able make connections between previous work experiences.

Interviewers notice candidates who made the investment to prepare. It is important that candidates come prepared for a company-specific interview. We found that some companies emphasized specific technical skills.

Interested in the details?

The formal research paper,The Tech-Talk Balance: What Technical Interviewers expect from Technical Candidates by Denae Ford, Titus Barik, Leslie Rand-Pickett, and Chris Parnin was recently accepted at CHASE, IEEE 10th International Workshop on Cooperative and Human Aspects of Software Engineering. This year’s workshop is co-located with ICSE and will take place in Buenos Aires, Argentina on May 23, 2017. A pre-print of the paper is now available online.

The FUTURE of Technical Phone Interviews

THE Problem.

For a software developer technical interviews can often feel like the most annoying process ever.  Candidates apply for a job and are called back for an interview where they are asked random intricate details about programming that are not necessary for the job. This process often feels like an intense formality, especially if you are an awesome programmer. But really, what is the gold standard for technical interviews? There are many kinds of technical interviews that candidates go through. Some of them are hackathon style (where you get into groups of 4 with strangers and code all day), riddles (where you’re asked to solve a brain teaser with pseudocode), or whiteboard coding (where you’re asked to go up to the whiteboard and write runnable code with no IDE). What can often seem overwhelming is when you are working on coding question given to you and you are instructed to ‘think-aloud’ as you work. However, everyone doesn’t work well this way. This is a skill in itself that has to be worked on strenuously especially if the interviewer is judging you based on this skill. In addition to all this, the anxiety you as the candidate experience knowing that the interviewer is studying your every word can definitely throw you off your game.

In fact, it’s the worst when you’re trying to walk through the coding example and you get interrupted by the interviewer during your thought process. This interruption at the wrong time can often result in losing your train of thought. Often times these interruptions happen on the phone when the interviewer cannot see you or cannot see the code  you are writing to solve the programming problem.

What if I told you we had the answer to your problems and I can answer them all with eye tracking?

THE Answer!

Remote Focus Lights. When driving in a car, passengers are better able to sense when a driver is busy than someone having a phone conversation with the driver. In remote interviews, a candidate may be in a mental state of high cognitive load but is not easily observable by an interviewer. For example, when writing code, an interviewer may use typing as a cue not to interrupt. But if a candidate is reflecting on a problem or reading code while deep in thought, that information is not easily accessible when on the phone. The remote focus lighjob-interviewt is an intervention that indicates when a candidate is currently involved with a high mental workload (red-circle-hi | red light) or is accessible for questions (alex-green-circle-hi |green light).

 

Remote Blackouts. A good interviewer might allow some time for a candidate to reflect on a problem in isolation, without worrying about the presence of an interviewer pressuring the candidate. For example, the interviewer might say, “now that I’ve explained the problem, I’ll put the phone down and walk out for about 4 minutes to allow you to digest the problem.” For coding activities, having your live state exposed to the interviewer can cause constant anxiety about making mistakes in front of others. A proposed intervention is the capture of the benefits of a “walk-out” during remote interviews by only refreshing the candidate’s screen in 2 minute intervals. This allows the candidate to have moments to reflect and perform simple tasks in a time-boxed manner, without constant fear of interruption.

BUT How?

I’m glad you asked. By studying task-evoked pupillary response, we can determine variations in cognitive load. Large pupil dilation is associated with high cognitive load and small pupil dilation is associated with low cognitive load. Obviously, it’s not ideal to interrupt a person when they have a high cognitive load as they are attempting to make sense of a lot of information. Using eye tracking tools we can detect when that moment occurs. The remote focus light intervention will reflect the variations in pupil dilation through a red or green light.

The nerves may feel inevitable since you unconsciously know that the interviewer is constantly rating your performance. However, you may be able to perform better if the interviewer cannot use the nonverbal cues that show your nerves. A few of these nonverbal cues include your rate of blinks, how you blink, or even where you look unconsciously through saccades. The remote blackouts provides control over your visibility to the interviewer. This control can also give you, as the candidate, confidence to continue the interview without letting your nerves show.

Future Experiment

To put these interventions to the test, we’ll recreate a technical interview environment and have candidates go through combinations of each. Further work would be to add another dynamic of this study with the different types of interview styles mentioned earlier. We can then study which types of interviews work well for different types of programmers and develop a true gold standard for technical interviews. The reduced cost in eye-tracking tools make this functionality now more available than ever and technical interviews provide a realistic platform for studies of this nature to be conducted.

Want to Know More?

The paper, Studying Sustained Attention and Cognitive States in Remote Technical Interviews, (Authored by myself, Titus Barik and Chris Parnin) has been accepted for presentation at the 2015 International Workshop Eye Movements in Programming Workshop, in Joensuu, Finland. I presented this work on November 2015 but will still be responding actively to inquiries about the project on twitter!

 

Images source: http://www.nadinemuller.org.uk/wp-content/uploads/2013/03/job-interview.jpg