Foundation presents results of harassment research, plans for automated identification; Wikiconference submissions open

News and notes

Foundation presents results of harassment research, plans for automated identification; Wikiconference submissions open

Foundation presents results of harassment research, plans for automated identification

The figures are deeply unsettling: 28% of editors say they have experienced criticism of their work, 24% believe that other editors are "not fun to work with", and 11% have experienced harassment. This was the introduction of a presentation by Maggie Dennis—the WMF’s director of support and safety and interim senior director of community engagement—about onwiki harassment to the WMF's monthly Metrics and Activities Meeting on 28 July. In the Foundation's consultation late last year, its workshop in February 2016, and in the results of its survey in 16 languages, stories have emerged that highlight the "frustration, shame, and helplessness" of harassed editors, she said. Among the key findings were that:

females are more likely to experience harassment than males;
more than half of those who reported harassment subsequently decreased their onwiki participation;
42% found that their attempts to deal with harassment were ineffective;
half of those who witnessed harassment avoid becoming involved;
two-thirds of those who did intervene experienced retaliation from the protagonist

Maggie Dennis's presentation (13:35–23:20) and Ellery Wulczyn's presentation (23:30–34.25)

Among the most common forms of harassment reported in the slide at 17:31 were content vandalism (27%), trolling or flaming (24%), name-calling (17%), discrimination (14%), and stalking (13%). Less prevalent but more concerning were threats of violence (6%), outing (6%), impersonation (5%), hacking (3%), and revenge porn (2%). Unspecified experiences were rated at 15%.

What becomes clear from viewing Maggie Dennis's presentation is that harassment is a highly prevalent behaviour at the interface of three problematic phenomena that continue to plague the WMF's sites: the gender gap, the flatlining of editor numbers, and the maintenance of the quality of the sites for readers. The Foundation is investigating measures to address the harassment problem in the communities; proposals for impending action include the default protection of user pages, the creation of a help page on all Wikipedias, and research into current mechanisms for dealing with harassment.

Dennis then introduced Ellery Wulczyn, from Wikimedia Research, who explained the progress of a program to develop an algorithmic approach to detecting personal attacks on the English Wikipedia—a collaborative project between the WMF and Jigsaw, a division of Alphabet, a holding company for Google. The project has created a data "pipeline" of examples of personal comments on the site, used this to develop a model for automated detection of harassment, and analysed the data to try to develop a system with the same level of accuracy as humans. Samples of comments were judged by 10 humans and a scale was derived of how likely each comment was to be harassing. From this a model was developed, and the claim was made that this is a 95% match with a later pooled human assessment of whether examples constituted harassment. A demonstration is at wikidetox.appspot.com, which readers are invited to visit and test for themselves. The algorithm determined that there is an 82% likelihood that this statement of mixed but ultimately insulting intention was harassment:

"Congratulations. I don't know whether you are aware of this fact or not, but you have shown your qualified stupidity."

The algorithm determined a 69% probability that "F#@$ you, a$$h0l3" was a personal attack; and the different grammatical contexts of "I will punch your lights out" and "Let's drink punch" were rated at 59% and 17% likelihoods of harassment, respectively. However, Wulczyn pointed out that the system is only as good as the depth of the corpus of personal attack patterns to which it has been exposed, with human rankings; for example, "Your intellect is lacking" was determined as having only a 10% probability of being an attack.

The intention now is to continue the program of "training" the system to achieve scores approaching zero false positives. The immediate goal is to explore the prevalence, dynamics, and impact of personal attacks on the English Wikipedia, and to create a complete historical dataset of talkpage comments with probability scores (which will be released publicly) for input to the "training" process.

The program is still at an early stage. Among the next goals is to integrate the algorithm with the ORES API system to enable extensions and tools to be built on top of the model. Readers with questions or suggestions are welcome to visit the dedicated page on Meta. T

Wikiconference submissions open

2016 Wikiconference North America, which will occur in San Diego, California, from October 7–10. Shown here: the San Diego Central Library, envisaged to be one of the venues for the event.

The 2016 Wikiconference North America, which will occur in San Diego, California, from October 7–10, invites interested editors to submit proposals to host a workshop, seminar, panel, tutorial, or other program during the event. Submissions can be made here. GP

Brief notes

Revamped app unveiled: A revamped Android app was released last week for Wikipedia. Its new homepage has been designed to help users access information more quickly and efficiently, and the app is now available worldwide in the Google Play store. More information was released in a Foundation blog post. GP
Language detection added to Wikimedia search engine: Wikipedia co-founder Jimmy Wales's vision to provide everyone in the world with an encyclopedia "in their own language" took another step last week when a feature was deployed on the English, French, German, Italian, and Spanish-language Wikipedias that will detect unsuccessful searches that may have been intended in another language. The blog post that announced the feature did not define a timeline for deploying the feature globally (it did state a few other language Wikipedias that will get it next), but did encourage users to test the feature in an online demo. GP

← Previous "News and notes"

Next "News and notes" →

In this issue

4 August 2016 (all comments)

News and notes

Discuss this story

These comments are automatically transcluded from this article's talk page. To follow comments, add the page to your watchlist. If your comment has not appeared here, you can try purging the cache.

I hope that some of the ideas from June's Inspire Campaign against harassment are adopted. I'm currently working on an RfC for my idea to protect userspace by default, which got a fair amount of support (and was also mentioned in Maggie Dennis' presentation). Funcrunch (talk) 00:26, 5 August 2016 (UTC)[reply]

It's interesting that only one target seems to be identifiable in your montage. Did you consider contacting that target? Have you consulted experts in social media and natural language? The statistics cited above are very unimpressive, and fail to give confidence intervals A fairly good research literature exists: do you know it well? Did you ignore then use of blackmail on Wikipedia for a reason, or were you simply unaware of the recent episode? MarkBernstein (talk) 01:58, 5 August 2016 (UTC)[reply]

I found two of the WMF harassment study's findings surprising:
- "28% of editors say they have experienced criticism of their work"... so the remaining 72% never got any negative criticism? Does that mean they simply ploughed their little patches and never collaborated in any contentious area?
- "Revenge porn (2%)"... That's a nasty surprise! Deryck C. 23:37, 7 August 2016 (UTC)[reply]

"Congratulations. I don't know whether you are aware of this fact or not, but you have shown your qualified stupidity."
- To be fair I'd be confused as well if I were an algorithm interpreting this logically/grammatically. The adjective "qualified" has two definitions, the first "competent or knowledgeable to do something" implies sentience which "stupidity" does not possess, this leaves the second definition of qualified "not complete or absolute; limited" which leaves us with an unusual sentence congratulating someone for their limited stupidity. M. A. Bruhn (talk) 04:59, 8 August 2016 (UTC)[reply]
  User:M. A. Bruhn—Indeed. I'm intrigued that the system appears to be so successful at this early stage, given the difficulty of programming grammar into computers. If you try it out at that link, I'll be interested to hear what you make of its interpretation of your examples. User:Deryck Chan, I agree, it's dreadful that one in 50 ticked that box; but I wonder what people mean by it, and how "revenge porn" could be sourced to attack an editor. Tony (talk) 09:54, 8 August 2016 (UTC)[reply]
- Ack! Please revise this article; at least some parts are simplistic to the point of being counterproductive. "Criticism of your work" is absolutely essential to maintain the encyclopedia's qulaity. I assume you don't believe Ta-Nehisi Coates was advocating harassment when he wrote: "Good fact-checkers have a preternatural inclination toward pedantry, and sometimes will address you in a prosecutorial tone. That is their job and the adversarial tone is even more important than the actual facts they correct." His remarks Lingzhi ♦ (talk) 05:22, 15 August 2016 (UTC)[reply]
  - Perhaps it should say "non-constructive criticism of your work"? Praemonitus (talk) 18:01, 15 August 2016 (UTC)[reply]
    - I agree that "criticism of your work" should not generally be interpreted as a negative thing, and high-level studies of Wikipedia interaction often make that mistake. Still, we at the Signpost are not doing the study, merely reporting on the study. So rephrasing things to reflect the way we might wish the study had been done is not really an option. -Pete (talk) 19:22, 15 August 2016 (UTC)[reply]

WikiCon North America

I see the announcement in English and Spanish but no Francophone wikilove. Any Signpost people ask why not or if French is supported? Les pauvres gens du Québec. Ô Canada!

--DHeyward (talk) 11:11, 18 August 2016 (UTC)[reply]

The Signpost needs your help putting together the next issue.

Home

About