Receiving thanks increases retention, but not the time contributed to Wikipedia

Recent research

Receiving thanks increases retention, but not the time contributed to Wikipedia

A monthly overview of recent academic research about Wikipedia and other Wikimedia projects, also published as the Wikimedia Research Newsletter.

Receiving thanks increases retention of new editors, but not the time contributed to Wikipedia

The Citizens and Technology (CAT) Lab at Cornell University recently published two preprints studying the effects of Wikipedia's inbuilt "thanks button" on participation. These experiments were conducted in collaboration with community members from the Arabic, German, Persian and Polish language Wikipedias.

In the first study^[1], all editors who had made a "thankable edit" (defined e.g. as receiving a "good-faith" and "non-damaging" rating from the automated ORES classifier) in the last 90 days were eligible to be randomly assigned into a test and a control group. Edits from the test group were then presented to experienced volunteers (334 overall) who decided whether to send a thanks for them. The researchers measured the effect on three preregistered main outcomes:

No significant changes were found in the daily labor hours contributed (comparing the six weeks before and after receiving the thank). The authors write that "this finding may result from an effect that is inconsequentially small, from the high variance in labor hours between participants, or from measurement error in the method for inferring time contributed" (which was based on grouping edits into "sessions" as proposed by Halfaker and Geiger, cf. our earlier coverage).
The retention of editors increased by 2% on average (an editor was defined as retained if they made "at least one edit to Wikipedia in a five week period starting at the beginning of the second week" after receiving the thank).
As the biggest effect, the study found that "receiving an expression of gratitude caused contributors to send 1.6 times more thanks" (according to the preprint; in an accompanying blog post, the researchers give a slightly different figure of a 43% increase).

These results are somewhat in contrast with a study by Goel, Anderson and Zia (conducted in collaboration with the Wikimedia Foundation's research team and presented at last year's WWW conference) which found "that receiving a thank has a strong positive effect on short-term editor activity", but had to base that conclusion on correlations as it was a merely observational study, instead of a randomized controlled experiment.

CAT's second study, titled "Expressing Gratitude and Feeling Emotionally Drained on Wikipedia"^[2] randomly assigned experienced editors from the German, Polish and Persian Wikipedias (recruited via banners to participate in the study) to either "review [edits by other contributors] and send personal expressions of gratitude to four Wikipedia contributors", or, in the control group, "to carry out common, routine activities on Wikipedia". Before and after this, they were asked to fill out a survey that asked whether they "feel positive about" their Wikipedia contributions, and whether they found contributing is "emotionally draining." Contrary to the researchers' preregistered hypothesis, the experiment "failed to find an effect from expressing gratitude on differences in socially supportive activity (p=0.74) [measured in article edits, article talk page edits, and thanks sent] or positive feelings about one’s contributions to Wikipedia (p=0.065)".

In general, the researchers found that (as summarized in an accompanying blog post) "People who spend more time mentoring and people who do more to monitor Wikipedia for vandalism report feeling more emotionally drained than others. Yet people who do more monitoring also feel more positive about their contributions".

The CAT team emphasizes that both studies were conducted in collaboration with community members from the involved Wikipedias, not unlike an earlier study that found a positive effect of barnstar-like awards on the German Wikipedia. In contrast, a quite similar project where researchers from Carnegie Mellon University had planned to study "How role-specific rewards influence Wikipedia editors’ contribution" was withdrawn in early 2019 after being met with resistance from editors on the English Wikipedia.

See also this Facebook discussion with two of the CAT researchers

Briefly

See the page of the monthly Wikimedia Research Showcase for videos and slides of past presentations.

Other recent publications

Other recent publications that could not be covered in time for this issue include the items listed below. Contributions, whether reviewing or summarizing newly published research, are always welcome.

"Openness, inclusion and self-affirmation: Indigenous knowledge in open knowledge projects"

From the abstract:^[3]

"This paper is based on an action research project [...] conducted in 2016-2017 in partnership with the Atikamekw Nehirowisiw Nation and Wikimedia Canada. Built into the educational curriculum of a secondary school on the Manawan reserve, the project led to the launch of a Wikipedia encyclopaedia in the Atikamekw Nehirowisiw language. We discuss the results of the project by examining the challenges and opportunities raised in the collaborative process of creating Wikimedia content in the Atikamekw Nehirowisiw language ... "

See also by one of the authors: "Ethics and responsibilities of open access. Lessons learned from the Wikipedia project of the Atikamekw First Nation" (abstract and 15min video)

Librarians "Reverting Hegemonic Ideology" by editing Wikipedia

From the abstract:^[4]

"... Using both Antonio Gramsci and LIS theorist Michael Harris as starting points, this paper argues that Wikipedia is predicated on a philosophy of pluralism that serves as a transmitter of hegemonic ideology, thereby upholding the oppressive status quo. To counter this issue, the paper encourages librarians to embrace 'critical editing'—an approach to Wikipedia editing built around an awareness of power, a penchant for critical literacy, a focus on desocialization, and an emphasis on self-education. The paper concludes with an example of critical editing praxis (dubbed the "Library Repository-to-Wikipedia" method) that research librarians and information professionals can replicate to counteract aspects of Wikipedia that inherently support the status quo and thus, hegemonic ideology."

"From MDMA to Lady Gaga: Expertise and Contribution Behavior of Editing Communities on Wikipedia"

From the abstract:^[5]

"we present a methodology for gaining a better understanding of the contribution behavior, interests and expertise of communities of Wikipedia users. Starting from a list of core articles and their main editors, we identify which other articles (outside of the initial list) they contributed to ‘significantly’. The ordering is based on (empirical) Bayesian estimates of the contribution probabilities for each of the articles. [...We ] use the editors that contributed to the articles on designer drugs as a case study. We find that the users in this community contribute significantly to articles on pharmaceuticals, popular party drugs, chemistry, mental illnesses, diseases, medicine and cell biology."

"Does Copyright Affect Reuse? Evidence from Google Books and Wikipedia"

From the abstract:^[6]

"I use the digitization of in-copyright and out-of-copyright issues of Baseball Digest magazine by Google Books to measure the impact of copyright on knowledge reuse in Wikipedia. I exploit a feature of the 1909 Copyright Act whereby material published before 1964 has lapsed into the public domain, allowing for the causal estimation of the impact of copyright across this sharp cutoff. I find that, while digitization encourages knowledge reuse, copyright restrictions reduce citations to copyrighted issues of Baseball Digest by up to 135% and affect readership by reducing traffic to affected pages by 20%. These impacts are highly uneven: copyright hurts the reuse of images rather than text and affects Wikipedia pages for less-popular players greater than more-popular ones."

See also university press release: "Wikipedia Readers Get Shortchanged by Copyrighted Material" and our coverage of an early presentation of the results (five years before the publication of this paper): "How Wikipedia articles benefit from the availability of public domain resources"

"Knowledge Graphs on the Web -- an Overview"

From the abstract:^[7]

"While companies such as Google, Microsoft, and Facebook have their own, non-public knowledge graphs, there is also a larger body of publicly available knowledge graphs, such as DBpedia or Wikidata. In this chapter, we provide an overview and comparison of those publicly available knowledge graphs, and give insights into their contents, size, coverage, and overlap."

How Wikipedia illustrates coup d’états in Soviet Russia and Egypt

From the abstract:^[8]

"we analyze variation in image usage across Wikipedia language editions to understand if, like text, visual narratives reflect distinct perspectives in articles about culturally-tethered events. We focus on articles about coup d’états as an example of highly contextual sociopolitical events likely to display such variation. [...] we use an iterative inductive coding process to arrive at a 46-item typology for categorizing the content of images relating to contested sociopolitical events, and a typology of network motifs that characterizes structural patterns of image use. We apply these typologies in a large-scale quantitative analysis that establishes clusters of image themes, two detailed qualitative case studies comparing Wikipedia articles on coup d’états in Soviet Russia and Egypt, and four quantitative analyses clustering image themes by language usage at the article level. [...] We find substantial cultural variation in both content and network structure."

"Wikipedia's Network Bias on Controversial Topics"

From the abstract and paper:^[9]

"Due to the high influence that links' disposition has on users' navigation sessions, one needs to verify that, given a controversial topic, the hyperlinks' network does not expose users to only one side of the subject. [...] In this work, we define the static structural bias, which indicates if the strength of connections between pages of contrasting inclinations is the same, and the dynamic structural bias, which quantifies the network's level bias that users face over the course of their navigation sessions [based on clickstream data ]. Our measurements of structural bias on several controversial topics demonstrate its existence, revealing that users have low likelihood of reaching pages of opposing inclination from where they start, and that they navigate Wikipedia showing a behaviour much more biased than the expected from the baselines. [...] For instance, we consider the topic abortion. The anti-abortion movement looks more represented than the abortion-rights one. It means that a user who randomly picks a Wikipedia’s page, has double the probability of reading an article related to anti-abortion than abortion rights."

References

^ Matias, J. Nathan; Kamin, Julia; Al-Kashif, Reem; Klein, Maximilian; Pennington, Eric (2020-07-07). "The Diffusion and Influence of Gratitude Expressions in Large-Scale Cooperation: A Field Experiment in Four Knowledge Networks". OSF.
^ Matias, J. Nathan; Kamin, Julia; Al-Kashif, Reem; Klein, Maximilian; Pennington, Eric (2020-07-07). "Expressing Gratitude and Feeling Emotionally Drained on Wikipedia". OSF.
^ Casemajor, Nathalie; Coocoo, Christian; Gentelet, Karine (2018). "Openness, inclusion and self-affirmation: Indigenous knowledge in open knowledge projects". The Journal of Peer Production (13).
^ To appear as Thomas, Paul A. (2020). "Reverting ideological hegemony: Librarians as 'critical editors' of Wikipedia". College & Research Libraries. 82 (2). hdl:1808/30577.
^ Dijkstra, Louis J.; Krieg, Lisa J. (2016). "From MDMA to Lady Gaga: Expertise and Contribution Behavior of Editing Communities on Wikipedia". Procedia Computer Science. 5th International Young Scientist Conference on Computational Science, YSC 2016, 26-28 October 2016, Krakow, Poland. 101: 96–106. doi:10.1016/j.procs.2016.11.013. ISSN 1877-0509.
^ Nagaraj, Abhishek (2017-07-26). "Does Copyright Affect Reuse? Evidence from Google Books and Wikipedia" (PDF). Management Science. 64 (7): 3091–3107. doi:10.1287/mnsc.2017.2767. ISSN 0025-1909.
^ Heist, Nicolas; Hertling, Sven; Ringler, Daniel; Paulheim, Heiko (2020-03-12). "Knowledge Graphs on the Web -- an Overview". arXiv:2003.00719 [cs].
^ Porter, Emily; Krafft, P. M.; Keegan, Brian (2020-02-26), Visual Narratives and Collective Memory across Peer-Produced Accounts of Contested Sociopolitical Events, Association for Computing Machinery, doi:10.1145/3373147
^ Menghini, Cristina; Anagnostopoulos, Aris; Upfal, Eli (2020-07-20). "Wikipedia's Network Bias on Controversial Topics". arXiv:2007.08197 [cs].

← Previous "Recent research"

Next "Recent research" →

In this issue

2 August 2020 (all comments)

Discuss this story

These comments are automatically transcluded from this article's talk page. To follow comments, add the page to your watchlist. If your comment has not appeared here, you can try purging the cache.

Maybe I'm missing something but I don't quite understand the conclusion of this study. ("Receiving thanks increases retention, but not the time contributed to Wikipedia.") Give and editor thanks if one feels like doing so but it's really not going to make any difference? Maybe giving thanks and barnstars should be just considered a form of civility for its own good. Blue Riband► 23:03, 2 August 2020 (UTC)[reply]
- From reading the summary about it, I think the idea is that the editor doesn't spend longer on the site per day, but does stay for a longer number of days. But this title does somewhat conflict with that. There are caveats in that retention was defined as making at least one edit to Wikipedia in a five week period starting at the beginning of the second week.
  My intuition and personal reaction would tell me that receiving "thanks" makes me more productive and increases the proportion of my future interactions which are of positive sentiment rather than negative sentiment (i.e. I find it easier to be cordial and my writing style may change from something that would have provoked tiresome argument into one that might lead to productive discussion), but doesn't increase the amount of time I spend or how long I will edit for. But I gather that the researchers focused on new editors rather than already long-term ones, and it would be (almost) impossible for them to systematically determine how sentiment is affected in an accurate manner. — Bilorv (talk) 22:25, 3 August 2020 (UTC)[reply]
Re: "In contrast, a quite similar project where researchers from Carnegie Mellon University had planned to study "How role-specific rewards influence Wikipedia editors’ contribution" was withdrawn in early 2019 after being met with resistance from editors on the English Wikipedia." To a researcher, a project on awarding Thanks may be quite similar to one on awarding Barnstars. To Wikipedians the two are very different, I must get dozens of times as many thanks as barnstars. Ϣere SpielChequers 15:46, 3 August 2020 (UTC)[reply]
- You're right about hinting on the absurd equation of 'Thanks' and barnstars. Researchers embarrass themselves if they don't understand differences. -- Just N. (talk) 21:31, 3 August 2020 (UTC)[reply]

I wrote this sentence and am very familiar with the similarities and differences, thank you - I am a Wikipedian as well. And nobody proposed an "equation". Also note that the German Wikipedia study that the preceding sentence refers to was about barnstar-like awards too.

Regards, HaeB (talk) 10:43, 4 August 2020 (UTC)[reply]

Confusing commentary of yours, HaeB. My remarks above weren't aimed at attacking anyone. Why you are complaining to be hit I have no idea. At what location do you think you were part of the scene ("I wrote this sentence")? Playing with ridfdles? I definitely just wanted to consent to Were's notes. Regards, Just N. (talk) 18:20, 4 August 2020 (UTC)[reply]

What is so hard to understand about the fact that I, as the author of this review, wrote the sentence that WereSpielChequers quoted above, including the comparison (not "equation") that you called "absurd"? Especially after I already told you above? Experienced Wikipedia editors are usually able to read a wiki page's revision history, or, failing that, should at least be capable of inspecting the byline on top. Regards, HaeB (talk) 08:28, 5 August 2020 (UTC)[reply]

"People who spend more time mentoring and people who do more to monitor Wikipedia for vandalism report feeling more emotionally drained than others. Yet people who do more monitoring also feel more positive about their contributions". I believe that mentoring is demanding: you're unavoidably focussed on someone else instead of your own preferences. That it is counted as just the same (as a stressing factor) as patrolling against vandalism is indeed irritating me. I'd guess that the above short summary is failing to give a better understanding of researching these quite different experience fields. If not so, the academic research designers would have delivered an embarassing approach due to not being prepared to go deep enough into differenciation of special Wp working fields. -- Just N. (talk) 21:20, 3 August 2020 (UTC)[reply]

Let me get this straight: If the Wikipedians who responded to this survey feel (on average) differently that you yourself feel or would expect to feel, then that must mean that the researchers have made "embarrassing" mistakes? Regards, HaeB (talk) 10:43, 4 August 2020 (UTC)[reply]

Hey HaeB, please read carefully! I said that it remains unclear which conclusion has to be drawn. It is unfair and not at all correct to ascribe (citation) *must mean* to me instead of correct 'could mean'. Regards, -- Just N. (talk) 18:07, 4 August 2020 (UTC)[reply]

That's not what you said. You were criticizing the researchers of either failing to summarize their own results correctly (as mentioned in the review, the quoted sentences were taken from their blog post), or else of "deliver[ing] an embarassing approach due to not being prepared to go deep enough". And you were basing this criticism on the argument that the survey results did not match what you personally "believe" about mentoring and patrolling.

I think that at this point it might be useful to read that blog post and/or the preprint itself, before launching into even further speculations and judgments about the study.

Regards, HaeB (talk) 08:28, 5 August 2020 (UTC)[reply]

For instance, we consider the topic abortion. The anti-abortion movement looks more represented than the abortion-rights one. It means that a user who randomly picks a Wikipedia’s page, has double the probability of reading an article related to anti-abortion than abortion rights. I was a little surprised to see this given the common wisdom that Wikipedia has—relative to America—a liberal bias. Perhaps that wisdom isn't so wise. Perhaps it's a case of "know your enemy". Or perhaps there is simply more to write about in terms of notable anti-abortion subjects. Answers on a postcard, please. — Bilorv (talk) 22:29, 3 August 2020 (UTC)[reply]
- The claim that Wikipedia has a "liberal bias" is an absurd over-simplification. Wikipedia has an educated white male American/European bias, per its demographics. When it comes to things like science and religion, that means it's "liberal", but when it comes to things like feminism or racial justice, it has a more conservative bias. Kaldari (talk) 20:59, 4 August 2020 (UTC)[reply]

The Signpost needs your help putting together the next issue.

Home

About