The FUTURE of Technical Phone Interviews

THE Problem.

For a software developer technical interviews can often feel like the most annoying process ever.  Candidates apply for a job and are called back for an interview where they are asked random intricate details about programming that are not necessary for the job. This process often feels like an intense formality, especially if you are an awesome programmer. But really, what is the gold standard for technical interviews? There are many kinds of technical interviews that candidates go through. Some of them are hackathon style (where you get into groups of 4 with strangers and code all day), riddles (where you’re asked to solve a brain teaser with pseudocode), or whiteboard coding (where you’re asked to go up to the whiteboard and write runnable code with no IDE). What can often seem overwhelming is when you are working on coding question given to you and you are instructed to ‘think-aloud’ as you work. However, everyone doesn’t work well this way. This is a skill in itself that has to be worked on strenuously especially if the interviewer is judging you based on this skill. In addition to all this, the anxiety you as the candidate experience knowing that the interviewer is studying your every word can definitely throw you off your game.

In fact, it’s the worst when you’re trying to walk through the coding example and you get interrupted by the interviewer during your thought process. This interruption at the wrong time can often result in losing your train of thought. Often times these interruptions happen on the phone when the interviewer cannot see you or cannot see the code  you are writing to solve the programming problem.

What if I told you we had the answer to your problems and I can answer them all with eye tracking?

THE Answer!

Remote Focus Lights. When driving in a car, passengers are better able to sense when a driver is busy than someone having a phone conversation with the driver. In remote interviews, a candidate may be in a mental state of high cognitive load but is not easily observable by an interviewer. For example, when writing code, an interviewer may use typing as a cue not to interrupt. But if a candidate is reflecting on a problem or reading code while deep in thought, that information is not easily accessible when on the phone. The remote focus lighjob-interviewt is an intervention that indicates when a candidate is currently involved with a high mental workload (red-circle-hi | red light) or is accessible for questions (alex-green-circle-hi |green light).


Remote Blackouts. A good interviewer might allow some time for a candidate to reflect on a problem in isolation, without worrying about the presence of an interviewer pressuring the candidate. For example, the interviewer might say, “now that I’ve explained the problem, I’ll put the phone down and walk out for about 4 minutes to allow you to digest the problem.” For coding activities, having your live state exposed to the interviewer can cause constant anxiety about making mistakes in front of others. A proposed intervention is the capture of the benefits of a “walk-out” during remote interviews by only refreshing the candidate’s screen in 2 minute intervals. This allows the candidate to have moments to reflect and perform simple tasks in a time-boxed manner, without constant fear of interruption.

BUT How?

I’m glad you asked. By studying task-evoked pupillary response, we can determine variations in cognitive load. Large pupil dilation is associated with high cognitive load and small pupil dilation is associated with low cognitive load. Obviously, it’s not ideal to interrupt a person when they have a high cognitive load as they are attempting to make sense of a lot of information. Using eye tracking tools we can detect when that moment occurs. The remote focus light intervention will reflect the variations in pupil dilation through a red or green light.

The nerves may feel inevitable since you unconsciously know that the interviewer is constantly rating your performance. However, you may be able to perform better if the interviewer cannot use the nonverbal cues that show your nerves. A few of these nonverbal cues include your rate of blinks, how you blink, or even where you look unconsciously through saccades. The remote blackouts provides control over your visibility to the interviewer. This control can also give you, as the candidate, confidence to continue the interview without letting your nerves show.

Future Experiment

To put these interventions to the test, we’ll recreate a technical interview environment and have candidates go through combinations of each. Further work would be to add another dynamic of this study with the different types of interview styles mentioned earlier. We can then study which types of interviews work well for different types of programmers and develop a true gold standard for technical interviews. The reduced cost in eye-tracking tools make this functionality now more available than ever and technical interviews provide a realistic platform for studies of this nature to be conducted.

Want to Know More?

The paper, Studying Sustained Attention and Cognitive States in Remote Technical Interviews, (Authored by myself, Titus Barik and Chris Parnin) has been accepted for presentation at the 2015 International Workshop Eye Movements in Programming Workshop, in Joensuu, Finland. I presented this work on November 2015 but will still be responding actively to inquiries about the project on twitter!


Images source:

Exploring Causes of Frustration for Software Developers

The lifestyle of a developer includes constantly learn new technologies, adapting to new environments, and overcoming challenges when learning and practicing their craft. Unfortunately, failure to overcome obstacles in these situations can introduce a sense of mounting frustration in developers that can negatively impact learning outcomes and influence retention in a field. Aside from heated venting session on Reddit, Twitter, and other blogging sites there has been a lack of a concerted effort in cataloging and categorizing the frustrating experiences developers face. Understanding these frustrating experiences is the first step to alleviating them. To gain better grasp, we sent a survey to software developers asking them to describe their frustrating experiences.  From the survey, 67% of respondents acknowledged that frustration is a severe problem for them. With the 45 responses we received, we were able to define a taxonomy of the causes.

P.I.L.E. of Trouble

These frustrations were grouped into 4 categories Perception- how the individual views the issue, Internal- sprouting from some internal conflict, Lack Of Experience- situations that require generally more familiarity which comes with time, and External- sprouting from something out of the individual’s control.

Under the hood of these categories the groups of frustration were defined  as follows:


  • Size:
    • (1) large goal spaces, high cognitive complexity: “Where do I begin? “
    • (2) gulf of completion: “I’m almost done but there is a little thing in my way”
    • (3) large artifacts: “There’s so much to understand”


  • Fear of Failure: The obsession over the fear of not succeeding setting the individual back
  • Internal Hurdles: Individuals placing pressure on themselves
  • Simple Problem: Management and peers downplaying the intensity of tasks

Lack Of Experience

  • Mapping Behavior to Cause: Difficulties identifying the connections between portions of code; having issues creating a mental model
  • Programming Experience: Learning a new language warrants learning a new syntax
  • New Project Adjustment: Getting acclimated to a new project
  • Programming Tools : The learning curve of new tools includes learning new features and shortcuts; broken tools add to the burden


  • Unavailability of Resources: Undocumented code creates confusion
  • Peers: Sometimes it’s hard to resist the urge to help your neighbor, what about when your neighbor needs too much from you?
  • Limited Time: Not enough time allotted to complete a task


In comparing the causes, we found instances where one causes contributed to another. We called these instances multipliers. Multipliers acknowledged what supplemental frustrations could have increased the frustration of the participant. However, every response does not have a multiplier. Multipliers were not mentioned in the single response of every participant but were conceived from further questions asked during the survey. This discovery opened the opportunity to explore this taxonomy further.

Next Steps

After this study we found that a deep understanding of frustration in software development is a field that has a lot of questions that remain unanswered. There are a couple questions that we plan on exploring in the future:

  • How often do these causes show up?
  • Are any of them more severe than others?
  • Do these situations vary across genders?
  • How do certain multipliers affect other causes?

Read More

The paper, Exploring Causes of Frustration for Software Developers, presenting these results (co-authored by Denae Ford and Chris Parnin) has been accepted for presentation at the 2015 ICSE Cooperative and Human Aspects of Software Engineering Workshop, in Florence, Italy, in May 2015. In addition, a buzzfeed post has been published with these findings.

11 Signs of A Frustrated Programmer

Check out this BuzzFeed Post I created about my research. It’s titled 11 Signs of a Frustrated Programmer. Read it and let me know what you think.