Featured

What is Open Source for Social Good and Why People Contribute?

When you first hear “open source software” your mind will likely go directly to developers building more tools that only developers will use. While projects like TensorFlow or React are popular for open source contributors, there are other galvanizing projects with direct societal impacts that inspire people to contribute to open source.

A gif from the Little Window Project of engaging with a chatbot
A screenshot from the Restroom Refuge Tool

Specifically, one project to consider is Little Window which supports victims of intimate partner violence and provides resources to leave abusive relationships. Another is Restroom Refuge, which identifies and highlights safe restrooms for trans and gender nonconforming individuals. These are only a few of the open source (OSS) for social good projects we identified. But although they may seem to be fewer in quantity, they are definitely powerful in influence.

What we asked

To understand the scope of these unique types of open source projects, we ask the following questions…

RQ1: How do contributors define OSS for Social Good? 

RQ2: What motivations do contributors have to contribute to OSS for Social Good? 

RQ3: What factors do contributors consider to select an OSS for Social Good project? 

RQ4: What are the current challenges to work in OSS for Social Good?

To answer these questions, we conducted a mixed-methods study of semi-structured interviews with Open Source for Social Good (OSS4SG) contributors and distributed a large scale survey to OSS and OSS4SG contributors. We identified OSS4SG projects from Ovio and the Digital Public Goods and strategically selected project topics from those platforms. Additional details on our sampling approach can be found in the paper.

In total, we interviewed 21 OSS4SG contributors and received 517 valid survey responses (8.97% response rate) from OSS and OSS4SG contributors.

What we found

For RQ1, “How do contributors define OSS for Social Good?”, we found that there are core properties that influenced participants perspective of OSS4SG:

  • Whether the project targets people or communities that need help
  • Whether the project aims to solve some societal issues or provide social benefits

Another priority that was mentioned but was not as prevalently featured in our survey was whether the project is for profit or not.

For RQ2, “What motivations do contributors have to contribute to OSS for Social Good?”, we found that OSS4SG contributors were slightly more motivated by the desire to solve a societal issue rather than building something for their own career or school project (more data on this can be found in the figure below). This result encourages us to think about how people decide that an OSS4SG project aligns with these motivations.

For RQ3, “What factors do contributors consider to select an OSS for Social Good project?”, we found that there are several factors that influence the selection process. One consideration that stood out was who the owners of the project are:

“If it’s even a charity organization I go and look at who their sponsors are. And if it’s a government I’m already like, no, it’s not gonna happen. A political party maybe. But government is too far for me.” (P14) 

From our survey responses, we found the OSS4SG group cared significantly more about whether they trusted the owner of the project they were contributing to. Other strategies for identifying projects included having a personal connection to the projects through a friend and also resonating with the issue this project is trying to solve.

To better understand differences in the scale of impact people desired to have on these projects, we designed a set of trolley problem style questions across 3 levels of psychological distance: spatial, temporal, and social proximity. In this survey we found that contributors in our survey overall preferred to work on a project with a global need, would have long term benefits, and that would contribute to a need someone they know personally has (see Table 5 for more).

For RQ4, “What are the current challenges to work in OSS for Social Good?”, we found that there are several challenges including finding a project to contribute to and securing consistent funding:

“It is difficult to know where the projects are. Where the communities are. And getting involved in it. There are many, many, many developers that might want to contribute, but they never get, you know, an announce or publication, a post, something.” (P1) 

“I honestly think the hardest thing about working on social good is very frequently they’re funded by charities, so it’s very hard to get people’s full focus on it. Like, paid full focus on it.” (P4) 

Although we explicitly sought to find challenges that exist for the OSS4SG projects, these issues resonate for the broader open source community as well. However, for OSS4SG projects in particular, these challenges present higher risks for the sustainability as global citizens are more likely to be relying on these projects. 

What this means

From this work, we’ve been able to capture a more confident understanding of Open Source Software for social good in hopes of being able to better support these projects. We’ve outlined a couple opportunities to help the OSS4SG community grow such as highlighting the societal impacts of the project more explicitly through semi-automated badging processes. We also suggest more safety and privacy guidelines to protect contributors working on these high impact (and sometimes polarizing) projects.

Read more

The formal research paper, Leaving My Fingerprints: Motivations and Challenges of Contributing to OSS for Social Good by Yu Huang,  Denae Ford, and Thomas Zimmermann is published at ICSE‘ 21, 43rd ACM/IEEE International Conference on Software Engineering. 

The presentation from my former intern Yu Huang’s talk, paper, and study materials are available locally online. This work was also featured in the 2020 GitHub Octoverse report and highlighted on GitHub’s Social Impact website.

Since this work has been published we have also conducted additional research investigating how to support developers growing their open source skills, matching developers who want to contribute to social good projects, and offering community tracking tools to the maintainers and organizers that keep these projects afloat. In fact, the follow ups from this work have also inspired how GitHub measures and sustains newcomer activity through GitHub Discussions Insights Dashboard. I am very proud of the work my interns Jenny T. Liang, Nischal Shrestha, and Mariam Guizani completed that summer.

To find the latest project updates be sure to check out the project website!

The GitHub Social Impact, Tech for Social Good Team partnered with OBI Digital to conduct a deeper investigation into open source work as digital public goods in low- and middle-income countries. This teams included Director Mala Kumar, Gina Assaf, Priyanka Patha, Yacoub Saad, and a host of others. Their in-depth study focused on the countries on India, Kenya, Egypt, and Mexico highlighting opportunities for sustainability and inclusive design. You can check out their summaries on their project website (which also includes the report available in English, Arabic, and Spanish)!

Advertisement

How Programmers REALLY Look at Pull Requests

Social signals are in fact a factor when reviewing a pull request, even when programmers don’t think they are.

Simply put, pull requests are the decoupled process of submitting code to a repository and having it reviewed by others. There’s a bit of unclarity on the decision process from the pull request being opened to the ending outcome of the pull request being merged or closed. Often times this process seems unlawful, bias, and even like a game of pull request roulette is happening to the submitter (or even amongst the project maintainers).

This research unveils this mysterious process within the context of GitHub—an online programming community which embodies this process while also highlighting a detailed profile of users and their activity.

To understand more of the process of signals that are used to determine we build on prior work and conduct an eye-tracking experiment where we go into depth about this process.

What we asked

RQ1: How do programmers review pull requests?

RQ2: Where do programmers think they look vs. where they really look?

RQ3: What strategies do people use to manage signals for their personal identity?

In order to gain insight into the process, we conducted an eye-tracking experiment in which we presented participants with a profile page and pull request. Then we asked them the likelihood of either accepting (merging) or declining (closing) the pull requests.

In our analysis, we classified different Areas of Interests (AOIs) to fit into three categories:

Code signals → areas where the primary content is code

Technical signals → areas where content provides evidence of technical skills or experience

Social signals → areas that communicate unique identifying information about the user

Findings

For RQ1, we find that although programmers focused on code signals the most (mean = 57.15%, median = 64.23%), they also considered technical (mean = 32.42%, median = 28.45%) and social signals (mean = 10.43%, median = 7.38%) as well. The figure below demonstrates how programmers spent their time during the experiment.

For RQ2, we find that programmers review more social signals than reported. As the ranged dot plot below shows, we compared the AOIs they fixated upon and the areas they reported reviewing in the post-survey. One particularly interesting result to point out here is that only 1 developer reported using the submitter’s avatar image(labeled AI in the figure), however, all participants reviewed that element of the profile.

blog-fig-2

For RQ3, we find that programmers use different strategies in traditionally technical online communities (e.g, GitHub) than in traditionally social ones (e.g., Facebook). Our multi-cycle qualitative analysis resulted in 5 themes centered around approaches to protecting their identity and their contributions.

One theme, in particular, emphasized that ’Nameless code should stand alone’. One participant described how they segment their identity across platforms in hopes to encourage the merit-based evaluation of their technical contributions:

“I prefer using my real name on social media platforms but I use other names when it comes to technical communities, I’ve different accounts for the different kinds of work. [...] It may be because my code has nothing to do with my name or my image, the code needs to talk for itself.” (S26)

This strategy can be helpful for implicitly encouraging pull request reviewers. But is it enough to reduce the bias to social signals that reviewers already have? Who’s to say this doesn’t further insight reviewers to rummage online searching for a submitter’s technical clout that warranted an audacious pull request?

Discussion

From a broader perspective, these signals can be seen as representing the submitters’ social and human capital: human capital refers to an individual’s ability while social capital is derived from interactions with others. Such capital can be made explicit using reputation scores as, e.g., customary at Stack Overflow, or visualized using badges akin to those used on GitHub to represent the status of a project. In the future, platform designers must be more mindful in balancing the power of signals that can amplify bias or harm against users, while still providing mechanisms for users to freely evaluate the merits of potential code contributions.

Conclusion

With the data we have presented in this paper, we can say that our findings are in chorus with previous studies:

Social signals are in fact a factor when reviewing a pull request, even when programmers don’t think they are.

We can also say that software developers use strategies for sharing an identity that may also not match up the strategies they expect from others. We will have to conduct further analysis to understand how the types of signals available can influence decisions.

Read more

The formal research paper, Beyond the Code Itself: How Programmers Really Look at Pull Requests by Denae Ford, Mahnaz Behroozi, Alexander Serebrenik, and Chris Parnin is published at ICSE‘19, 41st ACM/IEEE International Conference on Software Engineering, in the Software Engineering in Society track (ICSE SEIS).

The slides from my talk, paper, and study materials are available locally on my website.

Just-in-Time Mentoring: How We Improved the Novice Experience with Private and Timely Collaborative Editing

As helpful as Stack Overflow can be with over 14 million programming questions, it can also be just as toxic due to the malfunctioning community mechanics that cause users to suffer and feel un-welcomed in the community. Our previous work demonstrated that barriers such as jumping through onboarding hoops and the fear of negative feedback affect who feels they can participate on the site. To dismantle these barriers, we designed a just-in-time mentoring experience.

Just-in-time mentoring

We created this just-in-time mentoring experience to enhance the feedback process for new question askers on Stack Overflow.  Previously, users would gather feedback via slow comment sections taking hours or even days.  In this new format, novices(users with 15 points or less) were able to get instant feedback on how to structure a well-received question in this community. Drawing on insights from communities of practice, we found that guidance from experts would help encourage new users to engage. For example, guiding novices through onboarding hoops with the help of a mentor or reducing the feeling of an intimidating community size with a private space where users can make improve their questions, can help users feel more comfortable participating in this community and others like it.

How?

To take advantage of the existing chat room feature in the community, we built designated chat Room to serve as 4 Help Rooms and 1 Private Mentor Room. In the Private Mentor Room, mentors are notified of novices that need help and announce who will help each one. In the Help Room, novices interact with mentors about their question draft.

Once novices entered the help room, a mentor greeted them and suggested improvements(often based on mentor and author generated guidelines) for their question. From here, novices iteratively edited their question with feedback from mentors and then chose to post their question at any point. Mentors did not edit novice questions directly, we wanted to keep the ball in the novice’s court. One of our design goals was to understand how experts teach novice users to fish, not necessarily fishing for them.

By the end of our 33-day pilot experiment, we presented 71,068 novices with the option to join the help room, 520 enter the help room, and 271 who interacted with a mentor and post a question.

Findings

Novices have higher rated questions. Following Stack Overflow’s question characterization framework, we had more GOOD(positive score) questions and less BAD(negative score) questions. We also observed a 50% increase in the mean question score for mentored questions.

Mentors offer high-fidelity improvements. Overall, mentors suggested many very explicit changes for what novices could do to make their questions more likely to be answered. This included adding more details about what they have already tried, adapting to the community’s culture of asking by removing greetings, and finding the right home for a question that may belong in another stack exchange community.

Both novices and mentors were highly satisfied. It was important for us to not only conduct the experiment but also gather feedback on the design from novices and mentors. From our follow up surveys, we found that novices found the suggestion from mentors very helpful. Even in interviews, mentors were genuinely excited to make sure novices had a great experience:

“If we can get the [original poster] through the first question with a positive experience and they can see how this site really works, then we should get more good questions which feeds in to having more good answers.”

What does this mean?

Human-human guidance definitely paid off in this experiment. Having human mentors allowed novices to engage in an in-depth clarifying dialogue that welcomed user across language barriers and varying levels of programming experience.

The greatest takeaway from this project is we can now enhance the quality of experience for new users. But even better, it leads to a new feature design beyond renaming the code of conduct! In fact, one direct resolution that has come for this study as described by the EVP of Culture and Experience described is a beginner ask page where we can break down the suggestions mentors offered into an automated prompt to guide their questions a bit better.

Read more

The formal research paper, “We Don’t Do That Here”: How Collaborative Editing with Mentors Improves Engagement in Social Q&A Communities by Denae Ford, Kristina Lustig, Jeremy Banks, and Chris Parnin is published at CHI‘18, ACM Conference on Human Factors in Computing Systems.

The slides from my talk and the paper are online at the ACM digital library and locally on my website. Also, Kristina, the first Stack Overflow Researcher and now Design Manager, was featured on the Stack Overflow podcast to talk about this work so check it out!

Questions? Comment below!

Selfies for Science: How #ILookLikeAnEngineer Broke the Internet and What Future Hashtags Can Learn

Selfies and hashtags have transformed how we follow trends and receive news on the internet. The beauty of these hashtags is that they provide an additional vessel to shed light on the stories they share on social media. One software engineer did just that with a selfie and a hashtag to break down stereotypes of what an engineer looks like. We investigated what made participants so drawn to share their personal-professional stories online in the popular #ILookLikeAnEngineer identity hashtag movement and what future hashtag crafters can do to have a similar impact .

What are identity hashtag movements?

Identity hashtag movements such as #ProfessionalLocs, #BlackLivesMatter, and #YesToAllWomen have united marginalized groups to reclaim their truth and dismantle stereotypes that may be tied to the group. Posting on social media with these hashtags, amplified with a selfie, has the potential to give a face to the social issues experienced by these marginalized groups. In a way, these movements amplify voices of those who have gone unheard. To understand who participates, why they do, and their perceived impact, we conducted a qualitative study of the #ILookLikeAnEngineer identity hashtag movement.

Image source: https://www.indiegogo.com/projects/let-s-put-up-an-ilooklikeanengineer-billboard#/

What did we do?We identified both what influenced participation in the movement and what the perceived impact according to participants and non-participants. Based on these findings we determine recommendations for future identity hashtag movements.

How?

First we interviewed Isis Anchalee, the founder of the #ILookLikeAnEngineer hashtag, to discuss her inspiration behind the hashtag. We then conducted remote semi-structured interviews with 32 participants and observers of the hashtag from around the world. We asked them about their engagements and perceptions of the movement.

What did we find?

We found cross cutting themes that defined how people decided to participate including: (1) identifying with the movement as a “true form of an engineer” or marginalized group, (2) recognizing public image consequences associated with sharing your personal and professional identity, and (3) determining the audience seeing the post and how it may affect their relationship moving forward.

The other aspect we found is how the movement impacted participants, observers, and the engineering community as a whole. Interviewees reported that they felt empowered by the movement and felt like it raised awareness by maintaining the ongoing conversation about inclusion. Yet, many participants still felt that the same people who heard about the movement are the ones who were already interested; not the ones who actually need to see the movement. In an attempt to connect people who were unaware of the movement, participants coordinated offline activities to engage their local community through meetups and company-wide discussions.

What now?

We know it is too late to give guidelines for previous hashtags that came and went, however, it is not too late for the upcoming ones. If there are other who are interested in a successful (relatively measured through participation and impact) movement we have a couple recommendations in the table below.

Recommendation Description Support from study
Movement exemplars Multiple exemplars showing who should participate and modeling ideal participation behavior Ambiguity around collective identity associated with movement, who is allowed to participate and how
Diverse forms of participation Stories, anonymous posting, concrete actions as alternative forms of participation Discomfort with intersection of multiple identities, desire for tangible change
Audience awareness Indicators of audience presence and viewing behavior on social media platforms, targeting posts for specific subgroups Ambiguity around who would see posts and impact of the movement

Read more

The formal research paper, Selfies as Social Movements: Influences on Participation and Perceived Impact on Stereotypes by Fannie Liu, Denae Ford, Laura Dabbish, and Chris Parnin was recently published at CSCW ‘18, 21st ACM Conference on Computer-Supported Cooperative Work and Social Computing. This year’s conference will be held in Jersey City, New Jersey from November 3-7th, 2018.The paper is now online at the digital library and locally.

Question? Comments? Email us or find me on Twitter @DenaeFordRobin! Don’t forget to use the hashtag #ILookLikeAnEngineer.

Someone Like Me: How does Peer Parity Influence Participation of Women on Stack Overflow?

Women who are answered by other women on Stack Overflow reengage sooner.

However, they do not have higher reputation.

Motivation

It’s pretty challenging to post online when you don’t see many people like you. When multiplied with the fact that the community size can seem intimidatingly large, users can be even further discouraged from participating. In a recent study we conducted about Stack Overflow participation of women, subjects mentioned that one reason they not post on Stack Overflow is that, “They are just not even on the same race track.” In this work, we investigate how exposure to peers can affect the likelihood of further activity. We call this notion of observing people on the same “race track” or having similar individuals to compare oneself peer parity.

What is Peer Parity?

When an individual can identify with at least one other peer when interacting in a community.

With this definition, parity could be in the instance of men, but since there exist many more occurrences to where men find parity than women, we decided to serve the underrepresented community of women for this study.

How we did it?

Data set. Using Stack Exchange’s data archive, we extracted user and post data. Then we generated the genders of the user extracted by modifying an existing Gender Computing Tool. Our modified tool uses the first name of user display names on Stack Overflow and achieves higher precision among identifiable women. Of the 5,987,284 users we extracted, we identified 363,133 women. Of the women we identified, only 32% of have ever posted a question.

For this analysis, we had a strict definition of peer parity:

Parity = more than one distinct woman on a thread based on Q&A

Non-Parity = threads that only have one distinct woman

For example the figure above demonstrates a thread that would classify as parity.

Approach.

Step 1. Selected 1000 women who posted more than 1 question

Step 2. Record their reputation points and badges

Step 3. Record date of their first and second activity

Step 4. Identify if first and second activity was parity or non-parity

Step 5. Statistical comparison!

Findings

We found a significant difference in types of second activity after participating on a parity or non-parity thread (p = 2.799e- 06, α =.05), which was either posting a question(N = 833) or posting an answer(N = 167). We found a significant difference in the time between posts for women who asked a question on parity threads in comparison to non-parity threads (p = 1.83e-05, α =.05). The cumulative time differences by posts are demonstrated in the figure below. However, we did not find a significant difference in reputation points or number of badges.

Simply put:

Women who are answered by other women reengage sooner.

However, they do not have higher reputation.

Implications

Sharing success. The very idea of being transparent about a community problem may have played a factor in the increased interest in the site. Perhaps one way to inspire and increase others to participate is to showcase top-rated questions asked by women. This will not only demonstrate how to post successful questions on Stack Overflow, but also shows the diverse set of users contributing.

Paired guidance. As we found peer parity can influence participation, we hypothesize that building mentorship programs around shared identity can be a strong way to build communities and encourage participation for a broad audience.Mentorship is a bidirectional relationship—both parties have something to gain. Encouraging users to seek guidance can benefit both the mentor, providing guidance, and the mentee, seeking guidance.

Revealing user identity. It can be difficult for users to separate from their identity in public spaces, furthermore, they should not have to. We should embrace and support users who wish to disclose this information. Allowing users the opportunity to bring their whole self into a community where they seek help may just be the encouragement they need to be active contributors

More?

Our research paper, Someone Like Me: How Does Peer Parity Influence Participation of Women on Stack Overflow? by Denae Ford, Alisse Harkins, and Chris Parnin was recently accepted this year at VL/HCC, IEEE Symposium on Visual Languages and Human-Centric Computing. This year’s conference will be taking place in Raleigh, North Carolina from October 11 -14, 2016. The paper is now available online.

Questions? Comments? Concerns? Let us know below!

Personas from the Golden Collars of Software Engineers

7 personas to describe how software engineers spend their time

Many organizations may mistake the versatility of software engineers as universal skills they all have. However, there is no one-size-fits all solution for software engineers. Basically, let’s not match software engineers with descriptions but use practical knowledge of the work they are doing to describe the many roles software engineers can take on.

To clarify how software engineers vary, we adapted Reinhardt’s knowledge worker actions to what software engineers do.

What is a knowledge worker anyway?

According to Reinhardt….

Knowledge workers have high degrees of expertise, education, or experience, and the primary purpose of their jobs involves the creation, distribution, or application of knowledge.

But for the purposes of our work let’s just simplify it to mean people who think for a living, also known as golden-collar workers. In addition, software engineering in one of the many careers that require workers to use their creative abilities to solve a problem.

Conducted Interviews + Surveys

We wanted to make sure our questions about how knowledge workers use their time was appropriately fitted for software engineers. To ensure this, we rephrased knowledge worker actions to be in the context of software development, such as the Feedback knowledge worker action now specifically referring to examples such as code review and architecture reviews.

We then interviewed 21 software engineers and received survey responses from another 868 software engineers to figure out a day in the life would look like. More specifically, we asked them what a typical day would look like for them in terms of hours, who they collaborate with, to what extent, their inputs/ outputs, and perceptions of autonomy on tasks.

Analysis

We then coded interview responses about task execution and analyzed survey responses starting with hours spent. Specifically, we performed k-means clustering on how many hours they spent performing Software Engineering knowledge worker actions in a day. Using the resulting 7 clusters, shown in the figure below, we identified significant differences between clusters based on how the software engineers described their work.

Member checking. We performed member checking by following up with interview participants about their specific time spent to determine how they align with the defined personas. We were able to group all software engineers into one of our determined personas.

clusteranalysis_larger_font_withkey.png

Interesting Findings

Collaboration Styles. In brief, we found an interesting connection between how software engineers spent their time collaborating both within and outside their teams with other roles, performing code review, and being active participants in meetings.

Personas. We reviewed our cluster and cluster descriptions and inferred 7 personas of software engineers. You can find the described persona and a brief quote from a corresponding participant.

Acantha, the Autonomist Acumen ← Cluster 1

Acantha is known for her autonomy and keen abilities to make good judgments when writing code. On average, more than 20% of her time is spent in the Authoring knowledge worker action.

You know, I’m very much a risk-taker and I’d much rather ask for forgiveness. I’d much rather cross the line and wait for somebody to catch me. I’m willing to cross the line, and I do, and occasionally I get my hand slapped. I’m in this wonderful position.

Lilo, the Continuous Learner ← Cluster 2

Lilo is known for her interest in learning and getting adapted to a new team early in her career. She spends, on average, more than 23% of her time in the Learning knowledge worker action.

I mean it’s usually scoped but we don’t know the A, B, C’s of it, like where to start, how to start and any such things. But it usually starts with talking to different people. We know at least one contact who would then redirect us to multiple other contacts or they redirect us to some other people.

Isabelle, the Investigator ← Cluster 3

Isabelle is best known for being the top investigator on her team. She spends the most time, over 30% on average, in the Analyzing knowledge worker action.

It depends on what kind of task. For example, to fix bugs in theory you need to reproduce the problem and to just do a debug step-by-step, going into the very developed source code to figure it out. That takes a long, long time sometimes.

Cameron, the Communicator ← Cluster 4

Cameron is considered to be a communicator because of her ability to distribute her time evenly and early in her career while securing the logistics of her work environment. She is able to spread her time across all knowledge worker actions well.

The concentrated work time usually involves coding. Right now it involves learning, collating documentation, arranging resources for the people who are gonna come after me.

Iman, the Interactive ← Cluster 5

Iman is known for her activity with collaborative code writing style but also for her dynamic level of interaction that goes on within her team in terms of the Co-authoring knowledge worker action.

So we do all the code review ourselves – that’s a very important part. [Another] is the stand-ups, which have to be done – not every day, but twice in a week, we have meetings. And that’s very important because as we are still in a preview tool, so we have a lot of discussions

Ava, the Advisor ← Cluster 6

Ava is known for the high value she places on being helpful to others and giving feedback. On average, she spends over 20% of her time offering feedback.

That’s where people have an actual coding problem and they don’t know what to do to proceed, and you’re trying to give them advice or “here’s an example of code you can follow,” or “go read this article.” You’re kind of trying to hand them enough information so they can go solve the problem.

Ciara, the Team Coder ← Cluster 7

Ciara embodies this idea of being a team coder for her immense amount of time in the Co-authoring knowledge worker action (sometimes over 40%). She spends a significant amount of time producing code for her large company with a team by her side.

My role, in particular, has three things that we focus on. One is basically our automated test runs. This helps us keep the business running. Basically, when the service team deploys changes or updates stuff or new content comes in, we have a suite of tests that run four times a day. So a lot of times something might roll out and because they’re the service team, they don’t really do as much robust testing from the client-side.

What do we gain from this?

Inclusion through persona use. With our determined personas, software practitioners and researchers now have a resource to reference when encouraging a broader thinking of who users are. This can, in turn, encourage practitioners to be more inclusive of the many types of users.

Transient Modes. Software engineers may in fact transition through personas. We now have support for how to document that transition through hours spent. In the future, it may be interesting to conduct a longitudinal study that captures these transient modes based on types of projects developers work on.

Interested in the details?

I did this work as part of a summer internship in 2016! The research paper, Characterizing Software Engineering Work with Personas based on Knowledge Worker Actions by Denae Ford, Thomas Zimmermann, Christian Bird, and Nachi Nagappan was recently accepted at ESEM, ACM/IEEE 11th International Symposium on Empirical Software Engineering and Measurement. This year’s symposium will take place in Toronto, Canada on November 9-10, 2017. A preprint of the paper is now available online.

As always, comment (or email us) to let us know what you think about the work!

The Tech-Talk Balance: What Technical Interviewers Expect from Technical Candidates

Ever wonder what the top companies expect from their software engineering applicants?

The increased demand for software engineers in the upcoming years and their impressive salary makes the job a popular one that is highly coveted among those in STEM. With so many resources such as interviewing.io, codassium, and countless blogs about how to prepare you would think that most software engineers would have the interview process nailed down, right? WRONG!

In fact, software engineer job candidates are not exactly succeeding at technical interviews according to interviewers. Although candidates are able to answer technical questions, there is a mismatch of what candidates think interviewers assess versus what criteria is actually used in practice. Unfortunately, this mismatch in expectations can cost candidates the job of their dreams. To connect these engineers to the job of their dreams and confirm what type of mismatches may exist we sought to find out what their interviewers are looking for.

Ultimately, we were curious if there are differences across companies for how software engineer candidates are evaluated. But we were also interested in how interviewers interpret criteria for software engineer job candidates.

To soothe our curiosity, as well as other job-seeking developers, we studied interviewers in their natural habitat.

How do we really know what interviewers are expecting?

Short answer is, they told us. With interviewers from 9 major software companies, we conducted 70 mock technical interviews with software engineer candidates at our local university to better understand what companies expect from them. We anonymize the names of the company, but we give you details on their industry in the table below.

Statistical and Qualitative Analysis of Evaluations. We also collected interviewer evaluations of candidates in order to understand the expectations of interviewers and clarify how candidates should prepare for a technical interview.

After the interviews, we analyzed interviewer expectations from their evaluations and outlined how candidates can prepare for future technical interviews beyond being technically sound. We compared the Likert scores (1 through 4) interviewers gave to candidates using a Fisher’s exact test across companies followed by a Post-hoc Steel Dwass analysis on statistically significant results (α< 0:05).

What did we find in our analysis?

Evaluation Scores. In our statistical analysis, we found that all post-hoc pairs identified C1_WEB, a large internet Search company, as being significantly different than any of the other 8 companies. We did not identify any other pairs of companies with significant differences in scores for each of the evaluation criteria. We show the correspondence analysis of company ratings of candidates in the chart below. The most outstanding difference shown in this diagram demonstrates how C1_WEB scored how well candidate gave clear and concrete examples.

Criteria Interpretation. We were also able to determine how interviewers interpreted criteria used to evaluate candidates.

Example: [Original Criteria]-> [Interpretation from Interviewers]

Problem Solving-> Algorithms: When hiring candidates for a job, the top concern is whether candidates have sufficient technical skills to handle problem solving.

Nonverbal-> Interest: Interviewers noticed when there was poor communication during the interview.

Oral/Verbal Clarity->Fluent Speech: When making a first impression, the first words a candidate speaks are often the most important.

Clear, Concrete Examples-> Connected Experiences: Another way of demonstrating a candidate’s fit is their ability to communicate clear and concrete examples.

Enthusiasm-> Visible Excitement: How a candidate displayed enthusiasm is one measure of interest and engagement in the interview.

What should you share about this work?

Interviewers care about technical soundness and the ability for candidates to communicate that skill. Surprisingly, technical interviewers place an emphasis on interpersonal skills and effective communication in the interview. Interviewers wanted to hire a person, not just a candidate who can solve problems.

Most companies have consistent expectations for candidates across industry and size. According to interviewers, candidates are prepared technically, but encounter challenges translating their technical knowledge. Interviewers identified that candidates were not able make connections between previous work experiences.

Interviewers notice candidates who made the investment to prepare. It is important that candidates come prepared for a company-specific interview. We found that some companies emphasized specific technical skills.

Interested in the details?

The formal research paper,The Tech-Talk Balance: What Technical Interviewers expect from Technical Candidates by Denae Ford, Titus Barik, Leslie Rand-Pickett, and Chris Parnin was recently accepted at CHASE, IEEE 10th International Workshop on Cooperative and Human Aspects of Software Engineering. This year’s workshop is co-located with ICSE and will take place in Buenos Aires, Argentina on May 23, 2017. A pre-print of the paper is now available online.

Paradise Unplugged: Barriers to Stack Overflow Use for Females

Only 5.8% of users identified as females. We identified barriers for female contributions on Stack Overflow using semi structured interviews and surveys to validate these barriers. In this approach, we also found barriers that affected the entire community. As the big picture here is making sure all users feel comfortable using this resource, we also supplement these findings with an outline of potential site design modifications.

What is Stack Overflow and why do I care?

Stack Overflow is a popular question and answer site that many programmers go to for quick solutions to problems they encounter. Many have referred to it as “a programmer’s paradise”. Though this site is considered “a heaven sent” it is far from a true utopia. There has been much criticism of the lack of female participation on the site. In Stack Overflow’s 2015 and 2016 developer survey only 5.8% of users identified as females. The sad part about this is how often the conversation about this disparity often gets shut down with negative votes and closed as off topic discussions by moderators.

Stack Overflow does acknowledge the disappointing number saying: 

“Software development has a gender balance problem. Our internal stats 
suggest the imbalance isn't quite as severe as the survey results would
make it seem, but there's no doubt everyone who codes needs to be more 
proactive welcoming women into the field.”

There are many movements to get women into programming, but what about keeping them there? If they don’t feel comfortable using the resources that are available for all programmers then that is a big problem for retention in the field. To do our part in being more proactive in welcoming women into the field, we sought to uncover some reasons for this low participation.

What did we do? 

We identified barriers for female contributions on Stack Overflow using semi-structured interviews and surveys to validate these barriers. In this approach, we also found barriers that affected the entire community. As the big picture here is making sure all users feel comfortable using this resource, we also supplement these findings with an outline of potential site design modifications.

How did we do it?

Interviews. In order to find out to figure out what is hindering the community we went straight to the users. We interviewed 22 female users and asked them about their experiences when they posted a comment, edited and times when their activity was not as high. Other questions asked included how people communicate on the site, personal incentives, potential scenarios that may occur when using the site and potential modifications to the site that could increase usage. From our semi-structured interviews we transcribed audio recordings, performed a card sort, and 14 distinct barriers emerged. The 14 barriers are described in the table below.

barriers_table

General Survey. After identifying the barriers we wanted to find out which barriers varied across genders. We distributed the survey to the general developer population. We sent targeted emails, posted to programming forums, contacted large corporations, and posted in computer science Facebook groups. We received responses from 1470 females and males by the time we closed the survey.

With this survey we were able to identify 5 statistically significant barriers that females rated higher than males. These barriers include being aware of site features, not feeling qualified enough to chime in, the large intimidating community size, the discomfort of online strangers, and the perception of slacking on the job. All statistically significant barriers have been starred below in the diverging stacked bar chart of all barriers identified.

OpenRating_ranked_starred
Barriers to Stack Overflow usage for females and males. Red star indicates a statistically significant difference (p<.0012)

What Now?

As mentioned before, the ultimate goal is to make sure all users can feel comfortable relying on this resource. Using the results from our interviews and survey, we propose some design choices to consider as Stack Overflow grows to be more popular and inclusive.

5 Min V.S. 30 Min Questions. It is a bit of a task trying to identify which questions will take 5 minutes to answer versus 30 minutes manually. Research is needed to create and sustain a ranking algorithm of questions response time per user’s skill and question difficulty. This features would encourage a wider range of users with varied availability.

Quality Questions. Instead of discouraging users from posting questions, enhance the posting process by automatically providing feedback on the quality of the question in terms of how fast and how likely it will be answered.

Sub-communities. Sub-communities should be an available option to users who are interested in creating a more approachable group to interact with.

Mentorship. Retention of one-day flies and other contributors may increase if  a per user mentorship program is incorporated.

Many of these design choices have been shared with Stack Overflow affiliates and are now in the implementation phase. Keep your eyes peeled to see what Stack Overflow has up their sleeves in the future.

Read More

The formal research paper, Paradise Unplugged: Identifying Barriers for Female Participation on Stack Overflow by Denae Ford, Justin Smith, Philip Guo, and Chris Parnin was recently accepted at FSE, ACM SigSoft International Symposium on the Foundations of Software Engineering. This year’s conference will be taking place in Seattle, Washington from November 13 -19, 2016. A version of the paper is now available online.

Just plain curious about the work? We welcome you to check out the paper,message us with inquiries, or comment below.

 

The FUTURE of Technical Phone Interviews

THE Problem.

For a software developer technical interviews can often feel like the most annoying process ever.  Candidates apply for a job and are called back for an interview where they are asked random intricate details about programming that are not necessary for the job. This process often feels like an intense formality, especially if you are an awesome programmer. But really, what is the gold standard for technical interviews? There are many kinds of technical interviews that candidates go through. Some of them are hackathon style (where you get into groups of 4 with strangers and code all day), riddles (where you’re asked to solve a brain teaser with pseudocode), or whiteboard coding (where you’re asked to go up to the whiteboard and write runnable code with no IDE). What can often seem overwhelming is when you are working on coding question given to you and you are instructed to ‘think-aloud’ as you work. However, everyone doesn’t work well this way. This is a skill in itself that has to be worked on strenuously especially if the interviewer is judging you based on this skill. In addition to all this, the anxiety you as the candidate experience knowing that the interviewer is studying your every word can definitely throw you off your game.

In fact, it’s the worst when you’re trying to walk through the coding example and you get interrupted by the interviewer during your thought process. This interruption at the wrong time can often result in losing your train of thought. Often times these interruptions happen on the phone when the interviewer cannot see you or cannot see the code  you are writing to solve the programming problem.

What if I told you we had the answer to your problems and I can answer them all with eye tracking?

THE Answer!

Remote Focus Lights. When driving in a car, passengers are better able to sense when a driver is busy than someone having a phone conversation with the driver. In remote interviews, a candidate may be in a mental state of high cognitive load but is not easily observable by an interviewer. For example, when writing code, an interviewer may use typing as a cue not to interrupt. But if a candidate is reflecting on a problem or reading code while deep in thought, that information is not easily accessible when on the phone. The remote focus lighjob-interviewt is an intervention that indicates when a candidate is currently involved with a high mental workload (red-circle-hi | red light) or is accessible for questions (alex-green-circle-hi |green light).

 

Remote Blackouts. A good interviewer might allow some time for a candidate to reflect on a problem in isolation, without worrying about the presence of an interviewer pressuring the candidate. For example, the interviewer might say, “now that I’ve explained the problem, I’ll put the phone down and walk out for about 4 minutes to allow you to digest the problem.” For coding activities, having your live state exposed to the interviewer can cause constant anxiety about making mistakes in front of others. A proposed intervention is the capture of the benefits of a “walk-out” during remote interviews by only refreshing the candidate’s screen in 2 minute intervals. This allows the candidate to have moments to reflect and perform simple tasks in a time-boxed manner, without constant fear of interruption.

BUT How?

I’m glad you asked. By studying task-evoked pupillary response, we can determine variations in cognitive load. Large pupil dilation is associated with high cognitive load and small pupil dilation is associated with low cognitive load. Obviously, it’s not ideal to interrupt a person when they have a high cognitive load as they are attempting to make sense of a lot of information. Using eye tracking tools we can detect when that moment occurs. The remote focus light intervention will reflect the variations in pupil dilation through a red or green light.

The nerves may feel inevitable since you unconsciously know that the interviewer is constantly rating your performance. However, you may be able to perform better if the interviewer cannot use the nonverbal cues that show your nerves. A few of these nonverbal cues include your rate of blinks, how you blink, or even where you look unconsciously through saccades. The remote blackouts provides control over your visibility to the interviewer. This control can also give you, as the candidate, confidence to continue the interview without letting your nerves show.

Future Experiment

To put these interventions to the test, we’ll recreate a technical interview environment and have candidates go through combinations of each. Further work would be to add another dynamic of this study with the different types of interview styles mentioned earlier. We can then study which types of interviews work well for different types of programmers and develop a true gold standard for technical interviews. The reduced cost in eye-tracking tools make this functionality now more available than ever and technical interviews provide a realistic platform for studies of this nature to be conducted.

Want to Know More?

The paper, Studying Sustained Attention and Cognitive States in Remote Technical Interviews, (Authored by myself, Titus Barik and Chris Parnin) has been accepted for presentation at the 2015 International Workshop Eye Movements in Programming Workshop, in Joensuu, Finland. I presented this work on November 2015 but will still be responding actively to inquiries about the project on twitter!

 

Images source: http://www.nadinemuller.org.uk/wp-content/uploads/2013/03/job-interview.jpg

Exploring Causes of Frustration for Software Developers

The lifestyle of a developer includes constantly learn new technologies, adapting to new environments, and overcoming challenges when learning and practicing their craft. Unfortunately, failure to overcome obstacles in these situations can introduce a sense of mounting frustration in developers that can negatively impact learning outcomes and influence retention in a field. Aside from heated venting session on Reddit, Twitter, and other blogging sites there has been a lack of a concerted effort in cataloging and categorizing the frustrating experiences developers face. Understanding these frustrating experiences is the first step to alleviating them. To gain better grasp, we sent a survey to software developers asking them to describe their frustrating experiences.  From the survey, 67% of respondents acknowledged that frustration is a severe problem for them. With the 45 responses we received, we were able to define a taxonomy of the causes.

P.I.L.E. of Trouble

These frustrations were grouped into 4 categories Perception- how the individual views the issue, Internal- sprouting from some internal conflict, Lack Of Experience- situations that require generally more familiarity which comes with time, and External- sprouting from something out of the individual’s control.

Under the hood of these categories the groups of frustration were defined  as follows:

Perception

  • Size:
    • (1) large goal spaces, high cognitive complexity: “Where do I begin? “
    • (2) gulf of completion: “I’m almost done but there is a little thing in my way”
    • (3) large artifacts: “There’s so much to understand”

Internal

  • Fear of Failure: The obsession over the fear of not succeeding setting the individual back
  • Internal Hurdles: Individuals placing pressure on themselves
  • Simple Problem: Management and peers downplaying the intensity of tasks

Lack Of Experience

  • Mapping Behavior to Cause: Difficulties identifying the connections between portions of code; having issues creating a mental model
  • Programming Experience: Learning a new language warrants learning a new syntax
  • New Project Adjustment: Getting acclimated to a new project
  • Programming Tools : The learning curve of new tools includes learning new features and shortcuts; broken tools add to the burden

External

  • Unavailability of Resources: Undocumented code creates confusion
  • Peers: Sometimes it’s hard to resist the urge to help your neighbor, what about when your neighbor needs too much from you?
  • Limited Time: Not enough time allotted to complete a task

Multipliers

In comparing the causes, we found instances where one causes contributed to another. We called these instances multipliers. Multipliers acknowledged what supplemental frustrations could have increased the frustration of the participant. However, every response does not have a multiplier. Multipliers were not mentioned in the single response of every participant but were conceived from further questions asked during the survey. This discovery opened the opportunity to explore this taxonomy further.

Next Steps

After this study we found that a deep understanding of frustration in software development is a field that has a lot of questions that remain unanswered. There are a couple questions that we plan on exploring in the future:

  • How often do these causes show up?
  • Are any of them more severe than others?
  • Do these situations vary across genders?
  • How do certain multipliers affect other causes?

Read More

The paper, Exploring Causes of Frustration for Software Developers, presenting these results (co-authored by Denae Ford and Chris Parnin) has been accepted for presentation at the 2015 ICSE Cooperative and Human Aspects of Software Engineering Workshop, in Florence, Italy, in May 2015. In addition, a buzzfeed post has been published with these findings.

%d bloggers like this: