Simply put, pull requests are the decoupled process of submitting code to a repository and having it reviewed by others. There’s a bit of unclarity on the decision process from the pull request being opened to the ending outcome of the pull request being merged or closed. Often times this process seems unlawful, bias, and even like a game of pull request roulette is happening to the submitter (or even amongst the project maintainers).
This research unveils this mysterious process within the context of GitHub—an online programming community which embodies this process while also highlighting a detailed profile of users and their activity.
To understand more of the process of signals that are used to determine we build on prior work and conduct an eye-tracking experiment where we go into depth about this process.
What we asked
RQ1: How do programmers review pull requests?
RQ2: Where do programmers think they look vs. where they really look?
RQ3: What strategies do people use to manage signals for their personal identity?
In order to gain insight into the process, we conducted an eye-tracking experiment in which we presented participants with a profile page and pull request. Then we asked them the likelihood of either accepting (merging) or declining (closing) the pull requests.
In our analysis, we classified different Areas of Interests (AOIs) to fit into three categories:
Code signals → areas where the primary content is code
Technical signals → areas where content provides evidence of technical skills or experience
Social signals → areas that communicate unique identifying information about the user
Findings
For RQ1, we find that although programmers focused on code signals the most (mean = 57.15%, median = 64.23%), they also considered technical (mean = 32.42%, median = 28.45%) and social signals (mean = 10.43%, median = 7.38%) as well. The figure below demonstrates how programmers spent their time during the experiment.
For RQ2, we find that programmers review more social signals than reported. As the ranged dot plot below shows, we compared the AOIs they fixated upon and the areas they reported reviewing in the post-survey. One particularly interesting result to point out here is that only 1 developer reported using the submitter’s avatar image(labeled AI in the figure), however, all participants reviewed that element of the profile.
For RQ3, we find that programmers use different strategies in traditionally technical online communities (e.g, GitHub) than in traditionally social ones (e.g., Facebook). Our multi-cycle qualitative analysis resulted in 5 themes centered around approaches to protecting their identity and their contributions.
One theme, in particular, emphasized that ’Nameless code should stand alone’. One participant described how they segment their identity across platforms in hopes to encourage the merit-based evaluation of their technical contributions:
“I prefer using my real name on social media platforms but I use other names when it comes to technical communities, I’ve different accounts for the different kinds of work. [...] It may be because my code has nothing to do with my name or my image, the code needs to talk for itself.” (S26)
This strategy can be helpful for implicitly encouraging pull request reviewers. But is it enough to reduce the bias to social signals that reviewers already have? Who’s to say this doesn’t further insight reviewers to rummage online searching for a submitter’s technical clout that warranted an audacious pull request?
Discussion
From a broader perspective, these signals can be seen as representing the submitters’ social and human capital: human capital refers to an individual’s ability while social capital is derived from interactions with others. Such capital can be made explicit using reputation scores as, e.g., customary at Stack Overflow, or visualized using badges akin to those used on GitHub to represent the status of a project. In the future, platform designers must be more mindful in balancing the power of signals that can amplify bias or harm against users, while still providing mechanisms for users to freely evaluate the merits of potential code contributions.
Conclusion
With the data we have presented in this paper, we can say that our findings are in chorus with previous studies:
Social signals are in fact a factor when reviewing a pull request, even when programmers don’t think they are.
We can also say that software developers use strategies for sharing an identity that may also not match up the strategies they expect from others. We will have to conduct further analysis to understand how the types of signals available can influence decisions.
Read more
The formal research paper, Beyond the Code Itself: How Programmers Really Look at Pull Requests by Denae Ford, Mahnaz Behroozi, Alexander Serebrenik, and Chris Parnin is published at ICSE‘19, 41st ACM/IEEE International Conference on Software Engineering, in the Software Engineering in Society track (ICSE SEIS).
The slides from my talk, paper, and study materials are available locally on my website.
fordable 