How Programmers REALLY Look at Pull Requests

Social signals are in fact a factor when reviewing a pull request, even when programmers don’t think they are.

Simply put, pull requests are the decoupled process of submitting code to a repository and having it reviewed by others. There’s a bit of unclarity on the decision process from the pull request being opened to the ending outcome of the pull request being merged or closed. Often times this process seems unlawful, bias, and even like a game of pull request roulette is happening to the submitter (or even amongst the project maintainers).

This research unveils this mysterious process within the context of GitHub—an online programming community which embodies this process while also highlighting a detailed profile of users and their activity.

To understand more of the process of signals that are used to determine we build on prior work and conduct an eye-tracking experiment where we go into depth about this process.

What we asked

RQ1: How do programmers review pull requests?

RQ2: Where do programmers think they look vs. where they really look?

RQ3: What strategies do people use to manage signals for their personal identity?

In order to gain insight into the process, we conducted an eye-tracking experiment in which we presented participants with a profile page and pull request. Then we asked them the likelihood of either accepting (merging) or declining (closing) the pull requests.

In our analysis, we classified different Areas of Interests (AOIs) to fit into three categories:

Code signals → areas where the primary content is code

Technical signals → areas where content provides evidence of technical skills or experience

Social signals → areas that communicate unique identifying information about the user


For RQ1, we find that although programmers focused on code signals the most (mean = 57.15%, median = 64.23%), they also considered technical (mean = 32.42%, median = 28.45%) and social signals (mean = 10.43%, median = 7.38%) as well. The figure below demonstrates how programmers spent their time during the experiment.

For RQ2, we find that programmers review more social signals than reported. As the ranged dot plot below shows, we compared the AOIs they fixated upon and the areas they reported reviewing in the post-survey. One particularly interesting result to point out here is that only 1 developer reported using the submitter’s avatar image(labeled AI in the figure), however, all participants reviewed that element of the profile.


For RQ3, we find that programmers use different strategies in traditionally technical online communities (e.g, GitHub) than in traditionally social ones (e.g., Facebook). Our multi-cycle qualitative analysis resulted in 5 themes centered around approaches to protecting their identity and their contributions.

One theme, in particular, emphasized that ’Nameless code should stand alone’. One participant described how they segment their identity across platforms in hopes to encourage the merit-based evaluation of their technical contributions:

“I prefer using my real name on social media platforms but I use other names when it comes to technical communities, I’ve different accounts for the different kinds of work. [...] It may be because my code has nothing to do with my name or my image, the code needs to talk for itself.” (S26)

This strategy can be helpful for implicitly encouraging pull request reviewers. But is it enough to reduce the bias to social signals that reviewers already have? Who’s to say this doesn’t further insight reviewers to rummage online searching for a submitter’s technical clout that warranted an audacious pull request?


From a broader perspective, these signals can be seen as representing the submitters’ social and human capital: human capital refers to an individual’s ability while social capital is derived from interactions with others. Such capital can be made explicit using reputation scores as, e.g., customary at Stack Overflow, or visualized using badges akin to those used on GitHub to represent the status of a project. In the future, platform designers must be more mindful in balancing the power of signals that can amplify bias or harm against users, while still providing mechanisms for users to freely evaluate the merits of potential code contributions.


With the data we have presented in this paper, we can say that our findings are in chorus with previous studies:

Social signals are in fact a factor when reviewing a pull request, even when programmers don’t think they are.

We can also say that software developers use strategies for sharing an identity that may also not match up the strategies they expect from others. We will have to conduct further analysis to understand how the types of signals available can influence decisions.

Read more

The formal research paper, Beyond the Code Itself: How Programmers Really Look at Pull Requests by Denae Ford, Mahnaz Behroozi, Alexander Serebrenik, and Chris Parnin is published at ICSE‘19, 41st ACM/IEEE International Conference on Software Engineering, in the Software Engineering in Society track (ICSE SEIS).

The slides from my talk, paper, and study materials are available locally on my website.


Selfies for Science: How #ILookLikeAnEngineer Broke the Internet and What Future Hashtags Can Learn

Selfies and hashtags have transformed how we follow trends and receive news on the internet. The beauty of these hashtags is that they provide an additional vessel to shed light on the stories they share on social media. One software engineer did just that with a selfie and a hashtag to break down stereotypes of what an engineer looks like. We investigated what made participants so drawn to share their personal-professional stories online in the popular #ILookLikeAnEngineer identity hashtag movement and what future hashtag crafters can do to have a similar impact .

What are identity hashtag movements?

Identity hashtag movements such as #ProfessionalLocs, #BlackLivesMatter, and #YesToAllWomen have united marginalized groups to reclaim their truth and dismantle stereotypes that may be tied to the group. Posting on social media with these hashtags, amplified with a selfie, has the potential to give a face to the social issues experienced by these marginalized groups. In a way, these movements amplify voices of those who have gone unheard. To understand who participates, why they do, and their perceived impact, we conducted a qualitative study of the #ILookLikeAnEngineer identity hashtag movement.

Image source:

What did we do?We identified both what influenced participation in the movement and what the perceived impact according to participants and non-participants. Based on these findings we determine recommendations for future identity hashtag movements.


First we interviewed Isis Anchalee, the founder of the #ILookLikeAnEngineer hashtag, to discuss her inspiration behind the hashtag. We then conducted remote semi-structured interviews with 32 participants and observers of the hashtag from around the world. We asked them about their engagements and perceptions of the movement.

What did we find?

We found cross cutting themes that defined how people decided to participate including: (1) identifying with the movement as a “true form of an engineer” or marginalized group, (2) recognizing public image consequences associated with sharing your personal and professional identity, and (3) determining the audience seeing the post and how it may affect their relationship moving forward.

The other aspect we found is how the movement impacted participants, observers, and the engineering community as a whole. Interviewees reported that they felt empowered by the movement and felt like it raised awareness by maintaining the ongoing conversation about inclusion. Yet, many participants still felt that the same people who heard about the movement are the ones who were already interested; not the ones who actually need to see the movement. In an attempt to connect people who were unaware of the movement, participants coordinated offline activities to engage their local community through meetups and company-wide discussions.

What now?

We know it is too late to give guidelines for previous hashtags that came and went, however, it is not too late for the upcoming ones. If there are other who are interested in a successful (relatively measured through participation and impact) movement we have a couple recommendations in the table below.

Recommendation Description Support from study
Movement exemplars Multiple exemplars showing who should participate and modeling ideal participation behavior Ambiguity around collective identity associated with movement, who is allowed to participate and how
Diverse forms of participation Stories, anonymous posting, concrete actions as alternative forms of participation Discomfort with intersection of multiple identities, desire for tangible change
Audience awareness Indicators of audience presence and viewing behavior on social media platforms, targeting posts for specific subgroups Ambiguity around who would see posts and impact of the movement

Read more

The formal research paper, Selfies as Social Movements: Influences on Participation and Perceived Impact on Stereotypes by Fannie Liu, Denae Ford, Laura Dabbish, and Chris Parnin was recently published at CSCW ‘18, 21st ACM Conference on Computer-Supported Cooperative Work and Social Computing. This year’s conference will be held in Jersey City, New Jersey from November 3-7th, 2018.The paper is now online at the digital library and locally.

Question? Comments? Email us or find me on Twitter @DenaeFordRobin! Don’t forget to use the hashtag #ILookLikeAnEngineer.

%d bloggers like this: