"Evaluating the Impact of the Wikipedia Teahouse on Newcomer Retention"^[4]

Aaron Halfaker is a WMF employee and edits Wikipedia as EpochFail. Jonathan Morgan is a WMF employee and, while editing as Jtmorgan, is a Teahouse Host.

From the abstract: "[F]ew interventions employed to increase newcomer retention over the long term by improving aspects of the onboarding experience have demonstrated success. This study presents an evaluation of the impact of one such intervention, the Wikipedia Teahouse, on new editor survival. In a controlled experiment, we find that new editors invited to the Teahouse are retained at a higher rate than editors who do not receive an invite. The effect is observed for both low- and high-activity newcomers, and for both short- and long-term survival."

Can there ever be a solution to the dwindling number of new and active users? In their paper, Halfaker and Morgan explain that "[s]o far, neither purely social or purely technical efforts have been shown to be effective at providing effective socialization at a scale that leads to a substantial increase in the number of new editors who go on to become Wikipedians." They examine the Teahouse which they describe as "one of the most potentially impactful retention mechanisms that has been attempted". Their research demonstrates that "new editors who are invited to the Teahouse are significantly more likely to continue contributing after three weeks, two months, and six months than a similar cohort who were not invited."

Comparing it with the Wikipedia Adventure new editor training system developed in 2013, which—although often used—did not show any long-term impact, the Teahouse (created in 2012) "combined social and technical components to provide comprehensive socialization on a large scale, and was designed to promote long-term retention." Their study evaluates "the effect of invitation to the Teahouse, rather than participation in the Teahouse." Both the 24-hour and 5-edit threshold before issuing a Teahouse invitation, while avoiding vandalism-only accounts, may result in "many good-faith newcomers being denied the opportunity for positive socialization", they say.

"ORES", they explain, "provides powerful predictive models that can accurately discriminate between damaging and a non-damaging edits, and between malicious edits and edits that were made in good faith, even if they introduce errors or fail to comply with policies." The authors consider that "most new editors" receive "overwhelmingly negative and alienating experience [...] when they first join Wikipedia," but they do not appear to draw on any data for this assumption.

Summarized by Morten Warncke-Wang and Tilman Bayer

One of the presentations at the recent Wikimania 2018 conference was on the "State of Wikimedia Research 2017–2018". An almost yearly occurrence since 2009, this presentation gives a quick look into the overarching themes in research published about Wikimedia projects over the previous year. This year's presentation (slides) is now available on YouTube, and covers five main themes: images & media, talk pages, multilingual comparisons, non-participation (who is not contributing?), and Wikipedia as a source of data. The first of these highlights the "Tower_of_Babel.jpg" paper also covered above.

A keynote titled "Creating Knowledge Equity and Spatial Justice on Wikipedia" (summary, slides) by Martin Dittus, a data scientist at the Oxford Internet Institute, featured various results about the geographical distribution of geotagged Wikipedia articles and IP edits, partly from earlier research by Dittus' colleague Mark Graham and others (cf. earlier coverage). Other presentations included:

• "The State of Research in Knowledge Gaps" (video), showcasing various research results and technology projects by the Wikimedia Foundation's research team and its academic collaborators;
• "Research on gender gap in Wikipedia: What do we know so far?" (video, slides); and
• A lightning talk (video, slides, poster) about the "Wikipedia Cultural Diversity Observatory" (WCDO) project, which likewise uses geolocation coordinates to associate articles with cultural contexts, but also draws on other data such as article categories and Wikidata properties to overcome some of the shortcomings of the geotagging data.

Other recent publications that could not be covered in time for this issue include the items listed below. Contributions are always welcome for reviewing or summarizing newly published research.

Compiled by Kudpung and Tilman Bayer

"Vandalism on Collaborative Web Communities: An Exploration of Editorial Behaviour in Wikipedia"^[5]

The study found that "most vandalisms [on Wikipedia] were reverted within five minutes" on average and that "the majority of the articles targeted [with vandalism] are related to Politics (29.4%), followed by Culture (26.4%), Music (23.5%), Animals (11.7%) and History (8.8%)."

"Evaluating lexical coverage in the Simple English Wikipedia articles: a corpus-driven study"^[6]

From the paper: "[Simple English Wikipedia] articles require surprisingly large vocabularies to comprehend, comparable to that required to read standard Wikipedia articles."

From the abstract:^[7] "This paper reports from a multi-year ethnographic study of automated software agents in Wikipedia, where bots play key roles in moderation and gatekeeping. Automated software agents are playing increasingly important roles in how networked publics are governed and gatekept, with internet researchers increasingly focusing on the politics of algorithms. [...] In most platforms, algorithmic systems are developed in-house, where there are few measures for public accountability or auditing, much less the ability for publics to shape the design or operation of such systems. However, Wikipedia's model presents a compelling alternative, where members of the editing community heavily participate in the design and development of such algorithmic systems."

"Cultural diversity of quality of information on Wikipedias"^[8]

From the abstract: "This article explores the relationship between linguistic culture and the preferred standards of presenting information based on article representation in major Wikipedias. Using primary research analysis of the number of images, references, internal links, external links, words, and characters, as well as their proportions in Good and Featured articles on the eight largest Wikipedias, we discover a high diversity of approaches and format preferences, correlating with culture. We demonstrate that high-quality standards in information presentation are not globally shared and that in many aspects, the language culture's influence determines what is perceived to be proper, desirable, and exemplary for encyclopedic entries."

"Case study in political user behavior on Wikipedia"^[9]

From the paper:

"Combinatorics of the suffix -itis on talk pages of Wikipedia: A word formation pattern for the discursive regulation in the collaborative knowledge production" (in German)^[10]

From the English abstract: "The study reveals that -itis is a highly productive suffix in meta(-linguistic) discourses of the online-encyclopaedia: Wikipedia authors using word formation products with the suffix -itis (e.g. Newstickeritis or WhatsAppitis) try to standardise the collaborative knowledge production with the help of these linguistic innovations. The corpus analysis delivers evidence for the fact that certain linguistic innovations and special types of word formation characterise the community of Wikipedia authors and their discourse traditions."

From the abstract:^[11] "The linkage of ImageNet WordNet synsets to Wikidata items will leverage deep learning algorithm with access to a rich multilingual knowledge graph. [...] I show an example on how the linkage can be used in a deep learning setting with real-time image classification and labeling in a non-English language and discuss what opportunities lies ahead."

From the abstract:^[12] "[We] show that meaningful results on the influence of country ties can be extracted from the hyperlinked structure of Wikipedia. We leverage a novel stochastic matrix representation of Markov chains of complex directed networks called the reduced Google matrix theory. [...] We apply this analysis to two chosen sets of countries (i.e. the set of 27 European Union countries and a set of 40 top worldwide countries). We [...] can exhibit easily very meaningful information on geopolitics from five different Wikipedia editions (English, Arabic, Russian, French and German)." (See also earlier by some of the same authors: "Multi-cultural Wikipedia mining of geopolitics interactions leveraging reduced Google matrix analysis".)

"A Hybrid Model for Quality Assessment of Wikipedia Articles"^[13]

From the abstract: "We explore the task [of document quality assessment] in the context of a Wikipedia article assessment task, and propose a hybrid approach combining deep learning with features proposed in the literature. Our method achieves 6.5% higher accuracy than the state of the art in predicting the quality classes of English Wikipedia articles over a novel dataset of around 60k Wikipedia articles." (See also earlier coverage in August 2017 of related research by a different team: "Improved article quality predictions with deep learning".)

"Using big data and network analysis to understand Wikipedia article quality"^[14]

From the abstract: "The research reported in this paper focuses on the question of why Wikipedia articles are different in quality. [...] We focus on three major types of social capital with respect to teams of contributors working on Wikipedia articles: internal bonding, external bridging and functional diversity. Through a social network analysis of these articles based on a dataset extracted from its edit history, our research finds that all three types of social capital have a significant impact on their quality. In addition, we found that internal bonding interacts positively with external bridging resulting in a multiplier effect on article quality."