Stack Overflow – fordable

Just-in-Time Mentoring: How We Improved the Novice Experience with Private and Timely Collaborative Editing

As helpful as Stack Overflow can be with over 14 million programming questions, it can also be just as toxic due to the malfunctioning community mechanics that cause users to suffer and feel un-welcomed in the community. Our previous work demonstrated that barriers such as jumping through onboarding hoops and the fear of negative feedback affect who feels they can participate on the site. To dismantle these barriers, we designed a just-in-time mentoring experience.

Just-in-time mentoring

We created this just-in-time mentoring experience to enhance the feedback process for new question askers on Stack Overflow. Previously, users would gather feedback via slow comment sections taking hours or even days. In this new format, novices(users with 15 points or less) were able to get instant feedback on how to structure a well-received question in this community. Drawing on insights from communities of practice, we found that guidance from experts would help encourage new users to engage. For example, guiding novices through onboarding hoops with the help of a mentor or reducing the feeling of an intimidating community size with a private space where users can make improve their questions, can help users feel more comfortable participating in this community and others like it.

How?

To take advantage of the existing chat room feature in the community, we built designated chat Room to serve as 4 Help Rooms and 1 Private Mentor Room. In the Private Mentor Room, mentors are notified of novices that need help and announce who will help each one. In the Help Room, novices interact with mentors about their question draft.

Once novices entered the help room, a mentor greeted them and suggested improvements(often based on mentor and author generated guidelines) for their question. From here, novices iteratively edited their question with feedback from mentors and then chose to post their question at any point. Mentors did not edit novice questions directly, we wanted to keep the ball in the novice’s court. One of our design goals was to understand how experts teach novice users to fish, not necessarily fishing for them.

By the end of our 33-day pilot experiment, we presented 71,068 novices with the option to join the help room, 520 enter the help room, and 271 who interacted with a mentor and post a question.

Findings

Novices have higher rated questions. Following Stack Overflow’s question characterization framework, we had more GOOD(positive score) questions and less BAD(negative score) questions. We also observed a 50% increase in the mean question score for mentored questions.

Mentors offer high-fidelity improvements. Overall, mentors suggested many very explicit changes for what novices could do to make their questions more likely to be answered. This included adding more details about what they have already tried, adapting to the community’s culture of asking by removing greetings, and finding the right home for a question that may belong in another stack exchange community.

Both novices and mentors were highly satisfied. It was important for us to not only conduct the experiment but also gather feedback on the design from novices and mentors. From our follow up surveys, we found that novices found the suggestion from mentors very helpful. Even in interviews, mentors were genuinely excited to make sure novices had a great experience:

“If we can get the [original poster] through the first question with a positive experience and they can see how this site really works, then we should get more good questions which feeds in to having more good answers.”

What does this mean?

Human-human guidance definitely paid off in this experiment. Having human mentors allowed novices to engage in an in-depth clarifying dialogue that welcomed user across language barriers and varying levels of programming experience.

The greatest takeaway from this project is we can now enhance the quality of experience for new users. But even better, it leads to a new feature design beyond renaming the code of conduct! In fact, one direct resolution that has come for this study as described by the EVP of Culture and Experience described is a beginner ask page where we can break down the suggestions mentors offered into an automated prompt to guide their questions a bit better.

The slides from my talk and the paper are online at the ACM digital library and locally on my website. Also, Kristina, the first Stack Overflow Researcher and now Design Manager, was featured on the Stack Overflow podcast to talk about this work so check it out!

Questions? Comment below!

Someone Like Me: How does Peer Parity Influence Participation of Women on Stack Overflow?

Women who are answered by other women on Stack Overflow reengage sooner.

However, they do not have higher reputation.

Motivation

It’s pretty challenging to post online when you don’t see many people like you. When multiplied with the fact that the community size can seem intimidatingly large, users can be even further discouraged from participating. In a recent study we conducted about Stack Overflow participation of women, subjects mentioned that one reason they not post on Stack Overflow is that, “They are just not even on the same race track.” In this work, we investigate how exposure to peers can affect the likelihood of further activity. We call this notion of observing people on the same “race track” or having similar individuals to compare oneself peer parity.

What is Peer Parity?

When an individual can identify with at least one other peer when interacting in a community.

With this definition, parity could be in the instance of men, but since there exist many more occurrences to where men find parity than women, we decided to serve the underrepresented community of women for this study.

How we did it?

Data set. Using Stack Exchange’s data archive, we extracted user and post data. Then we generated the genders of the user extracted by modifying an existing Gender Computing Tool. Our modified tool uses the first name of user display names on Stack Overflow and achieves higher precision among identifiable women. Of the 5,987,284 users we extracted, we identified 363,133 women. Of the women we identified, only 32% of have ever posted a question.

For this analysis, we had a strict definition of peer parity:

Parity = more than one distinct woman on a thread based on Q&A

Non-Parity = threads that only have one distinct woman

For example the figure above demonstrates a thread that would classify as parity.

Approach.

Step 1. Selected 1000 women who posted more than 1 question

Step 2. Record their reputation points and badges

Step 3. Record date of their first and second activity

Step 4. Identify if first and second activity was parity or non-parity

Step 5. Statistical comparison!

Findings

We found a significant difference in types of second activity after participating on a parity or non-parity thread (p = 2.799e- 06, α =.05), which was either posting a question(N = 833) or posting an answer(N = 167). We found a significant difference in the time between posts for women who asked a question on parity threads in comparison to non-parity threads (p = 1.83e-05, α =.05). The cumulative time differences by posts are demonstrated in the figure below. However, we did not find a significant difference in reputation points or number of badges.

Simply put:

Women who are answered by other women reengage sooner.

However, they do not have higher reputation.

Implications

Sharing success. The very idea of being transparent about a community problem may have played a factor in the increased interest in the site. Perhaps one way to inspire and increase others to participate is to showcase top-rated questions asked by women. This will not only demonstrate how to post successful questions on Stack Overflow, but also shows the diverse set of users contributing.

Paired guidance. As we found peer parity can influence participation, we hypothesize that building mentorship programs around shared identity can be a strong way to build communities and encourage participation for a broad audience.Mentorship is a bidirectional relationship—both parties have something to gain. Encouraging users to seek guidance can benefit both the mentor, providing guidance, and the mentee, seeking guidance.

Revealing user identity. It can be difficult for users to separate from their identity in public spaces, furthermore, they should not have to. We should embrace and support users who wish to disclose this information. Allowing users the opportunity to bring their whole self into a community where they seek help may just be the encouragement they need to be active contributors

More?

Our research paper, Someone Like Me: How Does Peer Parity Influence Participation of Women on Stack Overflow? by Denae Ford, Alisse Harkins, and Chris Parnin was recently accepted this year at VL/HCC, IEEE Symposium on Visual Languages and Human-Centric Computing. This year’s conference will be taking place in Raleigh, North Carolina from October 11 -14, 2016. The paper is now available online.

Questions? Comments? Concerns? Let us know below!

Paradise Unplugged: Barriers to Stack Overflow Use for Females

Only 5.8% of users identified as females. We identified barriers for female contributions on Stack Overflow using semi structured interviews and surveys to validate these barriers. In this approach, we also found barriers that affected the entire community. As the big picture here is making sure all users feel comfortable using this resource, we also supplement these findings with an outline of potential site design modifications.

What is Stack Overflow and why do I care?

Stack Overflow is a popular question and answer site that many programmers go to for quick solutions to problems they encounter. Many have referred to it as “a programmer’s paradise”. Though this site is considered “a heaven sent” it is far from a true utopia. There has been much criticism of the lack of female participation on the site. In Stack Overflow’s 2015 and 2016 developer survey only 5.8% of users identified as females. The sad part about this is how often the conversation about this disparity often gets shut down with negative votes and closed as off topic discussions by moderators.

Stack Overflow does acknowledge the disappointing number saying:

“Software development has a gender balance problem. Our internal stats 
suggest the imbalance isn't quite as severe as the survey results would
make it seem, but there's no doubt everyone who codes needs to be more 
proactive welcoming women into the field.”

There are many movements to get women into programming, but what about keeping them there? If they don’t feel comfortable using the resources that are available for all programmers then that is a big problem for retention in the field. To do our part in being more proactive in welcoming women into the field, we sought to uncover some reasons for this low participation.

What did we do?

We identified barriers for female contributions on Stack Overflow using semi-structured interviews and surveys to validate these barriers. In this approach, we also found barriers that affected the entire community. As the big picture here is making sure all users feel comfortable using this resource, we also supplement these findings with an outline of potential site design modifications.

How did we do it?

Interviews. In order to find out to figure out what is hindering the community we went straight to the users. We interviewed 22 female users and asked them about their experiences when they posted a comment, edited and times when their activity was not as high. Other questions asked included how people communicate on the site, personal incentives, potential scenarios that may occur when using the site and potential modifications to the site that could increase usage. From our semi-structured interviews we transcribed audio recordings, performed a card sort, and 14 distinct barriers emerged. The 14 barriers are described in the table below.

General Survey. After identifying the barriers we wanted to find out which barriers varied across genders. We distributed the survey to the general developer population. We sent targeted emails, posted to programming forums, contacted large corporations, and posted in computer science Facebook groups. We received responses from 1470 females and males by the time we closed the survey.

With this survey we were able to identify 5 statistically significant barriers that females rated higher than males. These barriers include being aware of site features, not feeling qualified enough to chime in, the large intimidating community size, the discomfort of online strangers, and the perception of slacking on the job. All statistically significant barriers have been starred below in the diverging stacked bar chart of all barriers identified.

OpenRating_ranked_starred — Barriers to Stack Overflow usage for females and males. Red star indicates a statistically significant difference (p<.0012)

What Now?

As mentioned before, the ultimate goal is to make sure all users can feel comfortable relying on this resource. Using the results from our interviews and survey, we propose some design choices to consider as Stack Overflow grows to be more popular and inclusive.

5 Min V.S. 30 Min Questions. It is a bit of a task trying to identify which questions will take 5 minutes to answer versus 30 minutes manually. Research is needed to create and sustain a ranking algorithm of questions response time per user’s skill and question difficulty. This features would encourage a wider range of users with varied availability.

Quality Questions. Instead of discouraging users from posting questions, enhance the posting process by automatically providing feedback on the quality of the question in terms of how fast and how likely it will be answered.

Sub-communities. Sub-communities should be an available option to users who are interested in creating a more approachable group to interact with.

Mentorship. Retention of one-day flies and other contributors may increase if a per user mentorship program is incorporated.

Many of these design choices have been shared with Stack Overflow affiliates and are now in the implementation phase. Keep your eyes peeled to see what Stack Overflow has up their sleeves in the future.

Read More

The formal research paper, Paradise Unplugged: Identifying Barriers for Female Participation on Stack Overflow by Denae Ford, Justin Smith, Philip Guo, and Chris Parnin was recently accepted at FSE, ACM SigSoft International Symposium on the Foundations of Software Engineering. This year’s conference will be taking place in Seattle, Washington from November 13 -19, 2016. A version of the paper is now available online.

Just plain curious about the work? We welcome you to check out the paper,message us with inquiries, or comment below.

Share with the world!

Like this:

Share with the world!

Like this:

Share with the world!

Like this: