Cross-language editors, election predictions, vandalism experiments

A monthly overview of recent academic research about Wikipedia and other Wikimedia projects, also published as the Wikimedia Research Newsletter.

Analyzing edits to the then 46 largest Wikipedias between July 9 and August 8, 2013, a study^[1] identified a set of about 8,000 contributors (labeled multilingual) with a global user account who have edited more than one of these language versions (excluding Simple English, which was treated separately) in that time frame. It tested five hypotheses about cross-language editing and editors and looked, for instance, at the proportion of contributions that any of these Wikipedias receives from multilingual editors versus contributions from those only editing one language version. The research found that Esperanto and Malay stick out with a high proportion of contributions from multilinguals, and on the other end, that Japanese has few contributions from multilinguals. Overall, in terms of edits per user, multilingual users made more than twice the number of contributions to the study corpus than monolinguals did; they often work on the same topics across language; and in any given language, they are frequently editing articles not edited by monolinguals during the one-month period analyzed here. They thus serve a bridging function between languages.

Two existing write-ups are good starting points to putting the study in context.^{[supp 1][supp 2]} In the long run, it would be interesting to extend the research to (a) cover a longer time span, (b) include contributions from non-registered users, despite technical difficulties, (c) include smaller Wikipedias, and (d) explore the effects of that bridging function in more detail, perhaps in search for ways to support its beneficial effects while minimizing the non-beneficial ones. It would also be interesting to focus on some aspects of those multilingual users (e.g. how do the languages they edit in match with the languages they display on their user pages) or their contributions (e.g. how do their contributions to text, illustrations, references, links, templates, categories or talk page discussions differ across languages, or how contributions from multilinguals differ across topics or between pages with high and low traffic – or to entertain ideas for a multilingual version of editing tools like User:SuggestBot. The paper is one of the first to make use of Wikidata; comparing such cross-lingual Wikipedia contributions with contributions to multi-lingual projects like Wikidata and Commons may also be a fruitful avenue for further research. (See also earlier coverage of a CSCW paper about a similar topic: "Activity of content translators on Wikipedia examined")

A new paper on arXiv asks the question "Can electoral popularity be predicted using socially generated big data?"^[2] Operating on the assumption that "sentiment data is implied in information seeking behaviour," the authors Yasseri and Bright compare Wikipedia page views and Google search trends to election outcomes in Iran, Germany and the UK. In Iran and the UK, where the researchers were able to use the articles of individual politicians, the page view and search trend data correctly pick the winners of the elections. In the UK, the data polled even correctly picks the orders of the runners-up, but the same is not true for Iran. In the German case, no correlation is found between search data and election results. Yasseri and Bright defer to the argument from previous studies on Twitter prediction that conclude that the sample data is too self-selecting. Overall, it is shown that "people do not simply search in the same proportions that they vote." Still the researchers note that these techniques react "quickly to the emergence of new 'insurgent' candidates."

A book titled Confidentiality and Integrity in Crowdsourcing Systems contains a chapter on the integrity of the English Wikipedia as a case study of integrity management in crowdsourcing systems.^[3] To test the integrity of Wikipedia, they first tried to start a new article with "invalid content" (it got deleted) and then turned to vandalizing pages systematically, both of which violates Wikipedia policies (cf. Wikipedia:Vandalism). They noted that simple cases were caught by automated counter-vandalism tools (ClueBot and XLinkBot, whose user pages – one of them with a typo – are the only references cited in the chapter), whereas more subtle cases ("incorrect information containing words related to the page’s topic" or adding external links present in related Wikipedia articles) were not. No indication was given as to whether these inappropriate edits had later been removed (by the authors themselves or by other users), nor what the affected pages were or what IP address(es) they had used to make those edits.

In a next step, the authors went through dumps of the English Wikipedia from 2001 to 2011 and analyzed revision histories for "100 good and featured articles" (which refers to Wikipedia:Good articles and Wikipedia:Featured articles – later, they call this set "high-quality articles") and "100 non-featured articles" (by which they mean neither good nor featured – later, they refer to this set as "low-quality articles"). In this sample (of which no further details are given), they observed that the number of contributions to high-quality articles is about one order of magnitude higher than that of low-quality articles and "that there is a highly active group of contributors involved from the creation of high quality articles until present", while most editors to low-quality articles never contributed to those pages again. They then looked at revert rates, at the overlap between sets of top contributors to a given article across years, and at the range of topics edited by top contributors to an article, observing that "the top contributors have become the owners of high quality articles and their engagement has increased" (which runs contrary to WP:OWN), "[T]his results in higher quality for a small portion of articles in Wikipedia" and "[T]op contributors of high quality articles are more like- minded than the top contributors of low quality articles", concluding "that the main difference between low quality and featured articles is the number of contributions."

From that, they venture into extrapolating to crowdsourcing systems more generally: "[w]e observe that to have higher integrity in crowdsourcing systems, we need to have a permanent set of contributors who are dedicated for maintaining the quality of the contributions to the articles. For systems with open access such as Wikipedia, this can be a huge burden for the permanent editors. Therefore, we need new mechanisms for coordinating the activities in a crowdsourcing information system." No discussion of these new mechanisms is offered.

The chapter has a few simple tables and plots but no link to the underlying data nor the code used for the analysis, nor links to relevant literature or Wikipedia policies, but it is paywalled behind a price tag of $29.95 / €24.95 / £19.95. Given that the experimental edits to Wikipedia actually damaged the project, it is hard to imagine that an ethical review panel involving Wikipedians might have approved the study in that form. In fact, such a panel does exist in the form of the Research Committee, which had not been contacted about the project. Considering further that the conclusions of the study are not new, their possibly interesting implications for crowdsourcing more generally are not discussed and neither the paper nor its materials are available to those concerned about the integrity of Wikipedia, it is hard to see any benefit of this study that would outweigh the damage it caused (cf. earlier coverage: "Link spam research with controversial genesis but useful results", "Traffic analysis report and research ethics").

• "How we found a million style and grammar errors in the English Wikipedia": The abstract^[4] of a keynote talk with this title at the upcoming FOSDEM conference reports on the application of "Language Tool", an open source grammar-and-style error checker for multiple languages – including English, German and Polish – that has been in development since 2003, to Wikipedia articles (example). Its method is like regular expressions except it takes into account parts-of-speech. When it was applied to 20,000 English Wikipedia articles, 37,000 errors were found. However, the false positive rate was high, as only 29 of 200 manual inspections were human validated. This raises a question: does Wikipedia need to be grammar and style checked differently from other texts?

• "Evaluation of gastroenterology and hepatology articles on Wikipedia" – In this paper,^[5] the author was concerned about "the tendency of medical students to use Wikipedia as their preferred learning resource", so he and his assistants assessed 39 articles on topics related to gastroenterology and hepatology for a number of quality indicators (e.g. length, depth, accuracy, readability, number of tables or illustrations, topicality of listed references, missing important references). He concludes that "Wikipedia is not suitable for medical students as a learning resource for gastroenterology and hepatology topics" and suggests they turn to the peer-reviewed literature instead. The study is hidden behind a paywall, which reflects current practice in the field and is one of the reasons why some of the articles in the study do not contain suitable illustrations. The article has been discussed at WikiProject Medicine but taking concrete action is difficult, as the details of the assessment are hidden further away in an appendix not available "ahead of print".
• Overview of research on FLOSS and Wikipedia: A draft paper titled "Peer Production: A Modality of Collective Intelligence"^[6] co-authored by one of the pioneers of peer production/collaborative intelligence research, Yochai Benkler, discusses some of the developments in this field, using Wikipedia as one of several case studies, alongside free software (FLOSS). The paper, in essence, is a literature review of the fields of "organization, motivation, and quality" of peer production.
• In battle over Walt Whitman's sexuality, Wikipedia policies "tamed the mass into producing a good encyclopedia entry": The revision history and talk page of the Wikipedia article on American poet Walt Whitman are examined closely in a paper^[7] by three scholars, two of whom have previously published research on Whitman themselves, and another who was the main contributor to the Wikipedia article on Whitman. The authors focus on how Wikipedians covered Whitman's sexual orientation, which "has been settled in a particularly ambiguous way" in the academic literature. The paper examines the talk page discussions on this aspect and how they were influenced by Wikipedia policies on e.g. original research, neutrality and verifiability, and concludes: "Significantly, and perhaps ironically, Wikipedia’s evolving editorial codes have resulted in a Whitman page that, while curtailing open accessibility, or at least free-form contributions [the article has been semi-protected since 2008], generally reflects the nuance and complexity of the scholarly consensus regarding Whitman's sexuality. In this regard, Wikipedia has instituted something like the professional standards to which all scholars are subject: peer review, citations, and at least an affected tone of neutrality. Contrary to its democratic origins, Wikipedia has, at least in this case, tamed the mass into producing a good encyclopedia entry".
• Elinor Ostrom's theories applied to Wikipedia: A French-language paper^[8] in Management et Avenir (Management and Future) by two researchers from the University of Montpellier applies Nobelist Elinor Ostrom's analysis of self-organizing systems to the governance of Wikipedia.
• New dissertation on Wiktionary: Wiktionary, "currently the largest collaboratively constructed dictionary project", is the subject of a dissertation at the TU Darmstadt^[9] (cf. earlier coverage of publications by the same author and his doctoral adviser: "Can Wiktionary rival traditional lexicons?", "Wiktionary and OmegaWiki compared")

Supplementary references:

← Previous "Recent research"

Next "Recent research" →

In this issue

25 December 2013 (all comments)

Recent research

Featured content

Discussion report

WikiProject report

News and notes

Technology report

+ Add a comment

Discuss this story

These comments are automatically transcluded from this article's talk page. To follow comments, add the page to your watchlist. If your comment has not appeared here, you can try purging the cache.

=== Multilingual contributors ===

The fact that Esperanto has a large number of multilingual contributors is not surprising. It's no one's native language (so every Esperanto speaker is fluent in something else), and it's usually the third, fourth, or fifth (or more) language. Jut updating articles about Esperanto can take you to many wikis.

I'd like to know whether these contributions were text, or if adding the same set of images to many Wikipedias counted the same as being able to write a sentence. WhatamIdoing (talk) 22:53, 28 December 2013 (UTC)[reply]

Agreed. I occasionally contribute to other languages' Wikipedias, but almost always either to add images or to perform housekeeping or vandal-catching, and pretty much never without help either from other users or from Google Translate. Nyttend (talk) 12:54, 30 December 2013 (UTC)[reply]

Well, George Soros is a native speaker! --NaBUru38 (talk) 14:37, 29 December 2013 (UTC)[reply]

LanguageTool

I'd like to hear anything people have to say about open-source grammar checkers. The one mentioned here, LanguageTool, isn't likely to be useful. (When I asked it to check World War II, it told me: "The noun 'all' seems to be countable, so consider using: alls.") - Dank (push to talk) 19:51, 28 December 2013 (UTC)[reply]

LanguageTool looks like something interesting to play with, although you need to review your changes carefully before saving to ensure you're not changing text inside quotes. If you want to add it to your Tools menu, you can add this to your Custom JavaScript file:

// Add LanguageTool launcher in the toolbox on left
addOnloadHook(function () {
 addPortletLink(
  "p-tb",
  "http://community.languagetool.org/wikiCheck/index?url=" + wgPageName,
  "LanguageTool"
)});

GoingBatty (talk) 17:10, 29 December 2013 (UTC)[reply]

LanguageTools is a waste of time, as you'll see by running it on just about any article; try World War II. Anything that's trying to be a grammar checker ought to at least check the words it recommends ("alls") against a dictionary. But it's true that there are now serviceable open-source parsers that label parts of speech, and these ought to help us crowdsource a grammar checker. - Dank (push to talk) 18:06, 29 December 2013 (UTC)[reply]

Thank you for the user script! I think the LanguageTool WikiCheck looks very promising. I would really like to see it mark long complicated sentences that should be split to be easier to understand / more readable. For general grammar/typo checks, browser tools are OK.

Another tool, a bit similar, is "wikilint" or "autoreviewer" by user:Tim.landscheidt. I use it on German Wikipedia, but it also works on english WP. See wikilint about World War II and Zweiter Weltkrieg. How can I add this to the "dropdown action menu" by user script, any ideas? --Atlasowa (talk) 17:58, 30 December 2013 (UTC)[reply]

EC @ Dank, I suspect that any spellchecker applied to the English language Wikipedia would struggle somewhat with the way our spelling is consistent at the article level rather than the project level - so loads of typos will be of the kind "if Wikipedia's spelling follows American English conventions then all these are wrong, if it is supposed to follow English conventions then these are wrong". Another problem with such tools is that our articles contain a lot of names, including song names, some articles about popstars and their work are festooned with them. But the potential for this sort of thing is great, maybe we should offer some sort of a prize for a tool that finds as yet undiscovered errors on the site. But to be useful I would add the conditions that the errors must have persisted for a month undiscovered, and the level of false positives needs to be no more than fifty percent, otherwise there are better ways of improving the pedia. ϢereSpielChequers 18:40, 30 December 2013 (UTC)[reply]

The linked abstract states "If you want less errors in your Wikipedia: ..." If they cannot get their grammar right in their intro then LanguageTool is a long way from being a serious analytical or proofreading tool. I also agree with ϢereSpielChequers in that the inevitable conflict between American and British English usage is not addressed. Apart from conflict between Chicago and Folwer's, other distinctive forms such as Hiberno-English and Indian English will tend to be marginalised (NZ English spelling) and inappropriately "corrected" if LanguageTool were to be taken seriously. FanN | talk | 04:51, 1 January 2014 (UTC)[reply]

I can forgive them a few mistakes in their blurb, but to extrapolate 1.1 million errors from a sample where they concede that 171 of 200 were false positives is a bit tabloidy. 171 false positives out of a sample of 200 probable errors is too small to come up with a percentage or anything that one could meaningfully extrapolate like that. It would be wrong to say that it had a 85.5% false positive rate on the strength of such a small sample. But one can confidently say that their own figures indicate a very high proportion of false positives. looking at what their tool makes of Leonard Cohen gives an idea why. However if they can refine their tool to avoid song titles and so forth it might have potential. ϢereSpielChequers 08:44, 1 January 2014 (UTC)[reply]

Integrity of Wikipedia and Wikipedia research

"... it is hard to see any benefit of this study ..." or of summarising it or otherwise reporting on it. -- Michael Bednarek (talk) 08:16, 31 December 2013 (UTC)[reply]

re: Evaluation of gastroenterology and hepatology articles on Wikipedia

And indeed all the reasons mentioned there are why Wikipedia articles, imperfect as they are, are still much much better than most peer reviewed literature: they are accessible. Unlike the said study, which, paywalled or not, will not be read by not only most students, but even by most instructors and practitioners. Rather than wasting time trying to warn the students way from Wikipedia, the study should do the more constructive thing, which is to encourage the instructors and students to improve Wikipedia articles, as some others, commendable initiatives have done. --_{Piotr Konieczny aka Prokonsul Piotrus| reply here} 12:13, 1 January 2014 (UTC)[reply]

This issue was the genesis of the Association of Psychological Science's Wikipedia Initiative. Both students and the lay public are going to read Wikipedia for medical guidance regardless of warnings from the professional community. To that end, bringing the professionals (or at least their students) to Wikipedia makes more sense. Chris Troutman (talk) 18:16, 2 January 2014 (UTC)[reply]