The Huggle Experiment: interview with the research team

As part of the 2011 Summer of Research, the Wikimedia Foundation's Community Department has announced an experiment to investigate potential improvements to the new editors' experience of their first contact with patrollers, using the Huggle anti-vandalism tool. The Summer of Research is a three-month intensive project to study aspects of participation in Wikipedia that may have a significant effect on editor retention. It brings together a group of researchers, mostly PhD candidates, who have experience in both computer science and the social sciences, to give us a more well-rounded understanding of participation in the projects. (See also earlier Signpost coverage: "Wikimedia Summer of Research: Three topics covered so far", "WMF Community Department announces 'Summer of Research' participants")

The Signpost interviewed researchers R Stuart Geiger (who uses the Staeiou account for non-research editing), Aaron Halfaker, and Wikimedia Foundation Fellow Steven Walling to find out more. Steven has been a volunteer editor on the English Wikipedia since 2006, and before taking up the Foundation Fellowship was a professional writer and blogger, mostly for technology publications and companies. Stuart has been a Wikipedia editor since late 2004, and has been studying the project as an academic since his undergraduate senior thesis in 2006. Since then, he's been gathering from a number of fields the conceptual, theoretical, and methodological tools necessary to study something as complex as Wikipedia. "At present, I'm a doctoral student from the School of Information at the University of California, Berkeley, and I have a keen interest in both the digital humanities and social statistics movements." Aaron is a computer science graduate student from the University of Minnesota. He's been an editor since 2008 and has published academic research on Wikipedia since WikiSym 2009. He specializes in statistical data mining and designs user-scripts for Wikipedia to understand/improve editor interactions.

How did the Summer of Research project come about, and what questions will it investigate? According to Steven, the experiment aims to test "warning templates that are explicitly more personalized and set out to teach new editors more directly, rather than simply pointing them to policy and asking them not to do something". Steven says he personally got involved because, as a Fellow at the Foundation, research has been part of his job. "I currently share the responsibility for leading the project team with Diederik van Liere and Maryana Pinchuk. Diederik has experience with the technical side of this project, Maryana is a qualitative researcher with an academic background, and I lend community experience to round out the leadership team. We built an enormous, multi-part question list publicly on Meta. But it turns out that was just a beginning guide. We've been structuring the summer as a series of weekly sprints, and to get a feel for the research topics that have been and are currently being explored, I'd check out the public list on our Meta page. Because the team has a wide variety of skills, we've looked at many different aspects of Wikipedia as a community so far."

Aaron said they decided to experiment with Huggle's standardised warning system because the project goal is to understand the decline in new editors, so it seemed logical to focus on new editors' experience in the community. "Team-member Dr. Melanie Kill suspected that welcome messages might have an effect on how new editors perceive the community. So because Hugglers send out the most messages to new editors, we wanted to see if we could improve conversion (from damage) and other retention rates by just changing the wording of the message."

We wondered how the Foundation's sometimes lofty strategic goals, like "Support the recruitment and acculturation of newer contributors", are translated into practical initiatives such as this. Steven points to the Board resolution on Openness and the Foundation's Annual Plan for 2011–12. "Recruiting and retaining editors for Wikipedia is now one of our top priorities, and Zack Exley, our Chief Community Officer, designed the summer to really dig deeper into the exact areas of English Wikipedia and other projects that have the largest effect on new editors, and whether those editors stick around. The Editor trends study gave us a high-level understanding of the trends in participation, but it didn't tell us with certainty what internal community factors most have an impact. We need to have more data we're confident in if we're going to make good decisions, thus the Huggle experiment, which is clarifying that automated editing tools have a huge impact on new editors. The project was in the sweet spot of being able to gather a statistically significant sample quickly and with minimal impact on the normal functioning of the community."

Stuart's background seems ideally matched to an experiment that seeks to understand social phenomena using technical methodologies. "I'm an adherent of the sociotechnical systems approach, which thinks in terms of how social and technical phenomena are inherently intertwined, especially when we study processes in communities as technologically mediated as Wikipedia. Our motto, 'the free encyclopedia that anyone can edit' speaks to this principle that the Wikipedia community can't be fully understood without taking into account the code on which it runs – and vice versa. Huggle is a great example of this: scripts, tools, and bots like Huggle, Twinkle, and User:ClueBot have become the predominant way in which new users are introduced into Wikipedia. In fact, here's a statistic that is hot off the research press: almost 75% of newbies have their first talk page message sent to them from one of those semi- or full-automated software systems."

How were the parameters of the experiment decided on – for example, the number of warnings delivered, the proportion of changed warnings? Aaron says they settled on three variables for testing in the experiment: personalized, teaching-oriented, and image. "Dr. Kill, a professor of rhetoric, produced personalized and teaching-oriented versions of the default warning template for Huggle; Stuart and Aaron then expanded these templates with image/no-image versions and prepared a random template generator. Our requirement for the number of experimental welcomes/warnings is based on a bit of statistical algebra that allows us predict how many observations we'll need to find statistically significant differences between the variables."

The Huggle experiment is not the first to investigate the interactions of patrollers and new page creators. In the 2009 community-led Newbie treatment at Criteria for speedy deletion experiment (Signpost coverage), experienced editors (one Signpost interviewer included) posed as inexperienced article creators to look into how new contributors are treated in the patrolling process. The experiment attracted significant controversy due to ethical concerns surrounding the lack of informed consent of the participants. Steven says that before the experiment they posted a public notice at the Village Pump. "We also spoke directly with the main Huggle developers over email, IRC, and on-wiki (Addshore, Gurch and other volunteer developers deserve a lot of credit here; we couldn't have done this without their help and consent beforehand). I should probably point out that we felt pretty confident about this experiment because Stuart is a prolific Huggler himself. Even if we had no volunteer editing experienced as a team, I think the key difference between this and the treatment experiment you referred to is that we've been transparent about our actions before going forward with it."

Aaron points out that Huggle users come across hundreds of potential editors every day, and a surprising proportion of these editors are testing whether they can, in fact, edit Wikipedia by damaging an article. "We suspect that the reaction these potential editors receive affects whether they'll register an account and try contributing productively. We hypothesized that the tone of the welcome/warning message could be an important factor in this decision. We have Hugglers testing a few variations of the 1st-level warning message to find out if we're right."

So can we expect more experiments like this in the future? Steven says he could probably do an entire Signpost report just on this topic. "But let me give it a quick shot by saying that the recently released Annual Plan (see link above) will give you a very good idea of what direction we're focusing on, as well as activity on mediawiki.org, the tech portion of blog.wikimedia.org, and the impending software deployments page. We try to make sure to push a message locally here when new experiments with features or anything else is happening, but those three places are where to look if you're interested in these topics in the future."

Reader comments

2011-08-01

Little Project, Big Heart — WikiProject Croatia

Contribute  —  

Share this

•  E-mail
•  Facebook
•  Twitter
•  LinkedIn
•  Reddit
•  Digg

By Mabeenot

WikiProject news

News in brief

• WikiProject Abandoned Drafts is hosting an Abandoned Drafts Drive to improve user-space drafts in preparation for their move to main-space. Join the fun and earn a barnstar for your efforts.
• Motto of the Day's "Motto Shop" can provide you with words to live by... or a pithy comeback for that snide IP editor you've been reverting all week. Drop by the shop for their free services and consider offering your own words of wisdom to the other editors there.
• Stephen King, Wes Craven, Alfred Hitchcock, Edgar Allen Poe, H. P. Lovecraft, M. Night Shyamalan, and R. L. Stine could use your help at WikiProject Horror.
• The August Wikify Drive starts today. Get busy and earn some barnstars!
• Check out the new Trees Portal.

Submit your project's news and announcements for next week's WikiProject Report at the Signpost's WikiProject Desk.

In honor of Croatia's Victory Day on Friday, we interviewed WikiProject Croatia. The project focuses on the history, geography, politics and culture of Croatia as well as biographies of notable Croats. The central-southeastern European country is represented on the English Wikipedia by 33 editors maintaining over 10,000 pages, including 5 pieces of featured content and 16 Good Articles. In addition, WikiProject Croatia maintains a portal, popular pages list, intertranswiki page, and a task force focusing on the capital city Zagreb. WikiProject Croatia is a child of WikiProject Europe and, as a former component of several countries, shares articles with WikiProject Military History's Ottoman Empire Task Force, WikiProject Former Countries' Austria-Hungary Task Force, and WikiProject Yugoslavia. We interviewed project member GregorB.

What motivated you to join WikiProject Croatia?

GregorB: What motivated me to join the project was the following question: if we, Croatian editors, won't create and improve Croatia-related articles, who will? This means: if we - who "have it easy", and who are directly interested in this content - don't do it, how can we expect other editors to fill in? In a way, this question still motivates me.

Do you speak Croatian? What are some of the challenges to writing about a non-anglophone country on the English Wikipedia? Have you contributed to the Croatian Wikipedia?

GregorB: In practice, it is difficult, if not virtually impossible, to write a high-quality Croatia-related article without any reliance on Croatian-language sources. That's why writing about Croatia can be difficult for editors without at least a basic command of Croatian. This is perhaps a greater obstacle than the lack of "local", "contextual" knowledge.

I haven't contributed to the Croatian Wikipedia at all. Reasons for this are hard to explain in a few words, but it is sufficient to say that I find the environment and general feel of Croatian Wikipedia to be significantly different from the English Wikipedia. Apart from that, writing for a larger audience seems to me to be a better way to use one's limited free time.

Why has WikiProject Croatia remained more active than the projects of many larger European countries like WikiProject Spain or WikiProject Turkey? Does WikiProject Croatia collaborate with the projects of any former-Yugoslav countries?

GregorB: WikiProject Croatia has rather scarce resources - measured by the number of active editors - but it indeed has a disproportionally high output. I'm not quite sure why. It seems that small number of highly dedicated editors can be more efficient than a large number of casual contributors. WikiProject Slovenia is a good example: User:Kaktus999 is the sole reason why WP Slovenia is ahead of us in article count, because he created thousands of articles on virtually every settlement in Slovenia. A single editor can make a difference.

Unfortunately, no cross-project collaboration of note happened thus far. People who write about former Yugoslav countries mostly understand each other's native languages, so the use of sources should not be a barrier. These projects are rather light on resources too, and collaboration would really make sense. I'd like to see organized efforts of this kind in the future.

Croatia recently released the results of the country's 2011 census. Were there any surprises in the census results? Was there any controversy? How much work was it to update the data on Wikipedia's articles?

GregorB: No major surprises or controversies. From the encyclopedic point of view, one minor point of contention is the fact that the 2011 census uses a slightly different methodology, so 2001 and 2011 population figures are not directly comparable. The biggest problem is, of course, the physical volume of work to update all the articles - more than 2,000 of them. The work started on the day the first results were published, and it will probably continue into 2012. Final, detailed results will be published some time in early 2012 - yet more work for us... But looking on the bright side: there's a chance to introduce more consistent referencing and - time permitting - more consistent article layout. (Again, WP Slovenia is ahead of us in this department.)

WikiProject Croatia is home to 5 pieces of Featured material and 16 Good Articles. Have you contributed to any of these articles? Are there any tips you could give for improving articles about Croatia to FA or GA status?

GregorB: I've written or contributed to several GA articles and - most recently - a featured list. It is difficult to give useful tips here as I feel that there is no royal road to GA/FA content. It is hard work. But here are three tips anyway: 1) referencing is crucial, 2) don't do it alone if you don't have to, collaboration really makes things easier, and 3) skilled copyeditors are a great help, drop them a note.

Have you contributed to the Croatia Portal or the Zagreb Task Force?

GregorB: I've never contributed to our portal. In fact, I'm barely aware of its existence, and this question got me thinking. Sadly, almost the same story with the Zagreb Task Force - I'm formally a member, but it is de facto inactive now. Both require extra time and resources, and these are lacking.

What are the project's greatest needs? How can a new editor help today?

GregorB: More editors. I really wish we had more. Quite literally any kind of work would be of help.

Anything else you'd like to add?

GregorB: On behalf of my fellow editors, thanks for the opportunity to say a few things about our project. I hope this discussion will give us some new ideas.

Next week, we'll take aim at a high caliber project. While we're busy reloading the Report, we hope you'll be blown away by our previous issues in the armory archive.

Reader comments

2011-08-01

Featured pictures is back in town

Contribute  —  

Share this

•  E-mail
•  Facebook
•  Twitter
•  LinkedIn
•  Reddit
•  Digg

By Tony1 and Dabomb87

This week's "Featured content" covers Sunday 24 – Saturday 30 July

Featured articles

Five articles were promoted to featured status:

• Joppenbergh Mountain (nom) in Ulster County, New York, a topographical feature with an interesting past; nominator Gyrobo says Joppenbergh "was blasted apart in the 19th century, skied on in the 20th, and could very well become public property in the 21st." picture at right
• Kennet and Avon Canal (nom), a historic British industrial waterway that fell into disuse and has now been restored. The article covers not just the history and engineering of the canal but social and environmental factors. (Rodw) picture at right
• James B. McCreary (nom) (1838–1918), 37th governor of Kentucky, who campaigned on a platform of progressive reforms and convinced the legislature to make women eligible to vote in school board elections, to mandate direct primary elections, and to create a state public utilities commission (Nominated by Acdixon)
• Valston Hancock (nom) (1907–98), another of the remarkable series of men who headed the Royal Australian Air Force in the mid-20th century, whose common attribute was their status as former cadets of the Royal Military College, Duntroon—that is, they studied as Army officers before joining the Air Force. (Ian Rose)
• Homicide: Life on the Street (season 2) (nom), from the American police procedural drama television series that originally aired in the US in 1994 and involved several demands for production changes by NBC. (Hunter Kahn)

Three featured articles were delisted:

• Swedish allotment system (nom: referencing)
• Gremlins 2: The New Batch (nom: referencing)
• Daniel Webster (nom: prose, referencing, style)

Featured lists

Four lists were promoted:

• List of accolades received by The Young Victoria (nom) (Nominated by Ruby2010.)
• List of accolades received by 127 Hours (nom) (Crystal Clear x3.)
• List of Atlanta Braves team records (nom) (Albacore.)
• List of Giro d'Italia general classification winners (nom) (NapHit.)

One featured list was delisted:

• FC Barcelona in Europe (nom: considered an article rather than a list)

Featured pictures

Twelve images were promoted. Medium-sized images can be viewed by clicking on "nom":

• Flat-Earth Map (nom; related article), a rare map drawn by an enthusiast in 1893 and now in the Library of Congress. Reviewer Jujutacular noted "the depiction of the round earth to the right, with those poor guys hanging on for dear life" (Headbomb (uploader), Fallschirmjäger (restoration)). (picture at top)
• Pale-yellow Robin (nom; related article), with grey head and olive upperparts, white throat and yellow underparts, endemic to eastern Australia (created by User:JJ Harrison). picture at right
• White-lipped Tree Frog (nom; related article), the world's largest tree frog, whose dorsal surface is usually bright green, although the colour changes depending on the temperature and background, and can be brown. (created by User:JJ Harrison). picture at right
• Radjah Shelduck (nom; related article), Both the male and female of the species are mostly white, with dark wing-tips and a distinctive "collar" of dark feathers. Seen from above in flight the birds have green bands on the tops of their wings. The female has a harsh rattle and the male has a breathy, sore-throat whistle. (created by User:JJ Harrison). picture at right
• Coreopsis (nom; related article), a genus of herbs with striking yellow and orange flowers; eaten by caterpillars (created by User:Alvesgaspar).
• Hurricane Bob (nom; related article), approaching New England on 19 August 1991 (created by the US National Oceanic and Atmospheric Administration).
• Kiril Lazarov (nom; related article), an action-shot of the Macedonian handball player during the Schlecker Cup, Germany, 2009 (created by User:Kuebi). picture at right
• Comma (butterfly) (nom; related article), a species of butterfly with white markings on its underwings resembling commas (created by User:Quartl).
• Dusky Honeyeater (nom; related article), a small, brown bird that is a common resident of New Guinea, the Moluccas, the islands of Torres Strait, and northern Australia. They are active and fast moving (created by User:JJ Harrison). picture at right
• Straw-necked Ibis (nom; related article), taken at Centenary Lakes, Cairns, Queensland, Australia. These large birds are around 60–75 centimetres (24–30 in) long, with dark wings that show an iridescent, multicoloured sheen in sunlight. They have a long, black, downcurved bill, and their legs are usually red near the top and dark grey toward the feet. Straw-like feathers on the neck of adults give the bird its common name. (created by User:JJ Harrison).
• Bush Stone-curlew (nom; related article), a large, ground-dwelling bird endemic to Australia—a terrestrial predator filling a similar ecological niche to that of the roadrunners of North America. The bird is officially listed as endangered. (created by User:JJ Harrison).
• North walls of Ani (nom; related article), a panorama of the north walls of Ani, the medieval Armenian "City of 1001 Churches". Ani stood on trade routes and its many religious buildings, palaces, and fortifications were among the most technically and artistically advanced structures in the world (created by User:Ggia). picture below

A new featured picture taken by User:Ggia in April 2011, this panorama shows the north walls of the ruined and uninhabited medieval Armenian city-site of Ani in a visually dramatic and naturally defensive landscape. At its height, Ani had a population of 100,000–200,000 people and was the rival of Constantinople, Baghdad and Cairo. To view the image, please use the horizontal scroller or the interactive large-image-viewer (non-Flash).

Reader comments

2011-08-01

Proposed decision submitted for one case

Contribute  —  

Share this

•  E-mail
•  Facebook
•  Twitter
•  LinkedIn
•  Reddit
•  Digg

By Ncmvocalist

The Arbitration Committee opened no new cases this week. Two cases are currently open.

Open cases

Cirt and Jayen466 (Week 2)

See previous Signpost coverage for background and the specific scope of this case. During the week, several editors submitted on-wiki evidence.

MickMacNee (Week 7)

See earlier Signpost coverage for background. During the week, drafter Kirill Lokshin submitted a proposed decision for arbitrators to vote on, including 5 principles, 5 findings of fact, and 4 remedies. In total, rulings concerning three editors and one administrator are being considered.

Reader comments

2011-08-01

Developers descend on Haifa; wikitech-l discussions; brief news

Contribute  —  

Share this

•  E-mail
•  Facebook
•  Twitter
•  LinkedIn
•  Reddit
•  Digg

By Jarry1250

Developers descend on Haifa

With Wikimania 2011 set to begin on 4 August, a number of developers are getting ready to make the trip to Haifa in Israel. Indeed, even before the annual Wikimedia conference begins, two events are likely to catch the eye of the technically aware traveller (for those not making the trips, The Signpost will present a round-up of events in forthcoming issues). The first is a special "Developer Days" event, held on 2 and 3 August, to which 39 Wikimedians are already subscribed. Its schedule includes a workshop period for those interested in getting involved in MediaWiki development as well as a series of "lightning talks" about MediaWiki development. The second preliminary event is a meeting for "everyone interested in technical details around Wikipedia Offline... and get[ting] involved with openZIM"; the OpenZIM Project has sponsored several developers' attendance at the meetup.

When Wikimania proper begins, the schedule is likely to be as intense, although in general organisers have tried to avoid running sessions developers might be interested in concurrently (instead, one of the five "tracks" – analogous to stages at music festivals – has more or less been allocated to technical issues). Planned sessions include a number on topics with which Signpost readers will be very familiar: one on the ResourceLoader, another on Wikimedia Offline (including OpenZIM); two on Wikimedia's support for mobile devices (WMF Mobile Research and Wikimedia Mobile Panel) and a fifth on Testing for MediaWiki, which is set to include both an overview of existing functionality and practical advice for developers on how to take advantage of it. There is also a talk by Brion Vibber planned entitled Editing 2.0: MediaWiki's upcoming visual editor and the future of templates.

Other talks over the three day conference focus on other common issues, including interwiki links (Interlanguage links in Wikipedia and Discussion and Improvement Proposals for the Current Interwiki Linking System on Wikipedia), cross-wiki transclusions (CoSyne: Multilingual Content Synchronization with Wikis and A framework to visualizing wiki-based transclusion) and combating vandalism (Autonomous Detection of Collaborative Link Spam). Four more talks focus on very specific issues: A Qt library for MediaWiki, and what you can do with it (demonstrating a new tool to edit MediaWiki built using the cross-platform Qt framework), Opening up Wikipedia's data, The Site Architecture You Can Edit (focussing on Ryan Lane's effort to devolve basic sysadmin tasks to local administrators) and a proposal for Collaborative Watchlists.

Among less specific talks due to be held in Haifa next week is Ask the Developers (a question-and-answer panel) and a talk entitled Wikimedia Operations Overview. The Foundation's Guillaume Paumier will attempt to acknowledge and ease the potential for WMF staff to "appear to the community as ignoring requests, lacking transparency, and thus lacking accountability" (Wikimedia technical staff vs. the World) while Volunteer Development Coordinator Sumana Harihareswara will talk on How to get what you want from MediaWiki developers. There will also be a talk for those wanting A brief introduction to MediaWiki extension development.

In brief

Not all fixes may have gone live to WMF sites at the time of writing; some may not be scheduled to go live for many weeks.

How you can help

Translate MediaWiki

Know English and another language? Help with translating MediaWiki and other worthwhile open source projects at translatewiki.net.

• The Toolserver experienced difficulties for 90 minutes on 31 July following a hardware failure affecting its Cassia server (toolserver-l mailing list).
• On 25 July an update was made to the Code Review extension, which enables developers to check all revisions made to the MediaWiki software that power Wikimedia wikis (more software deployments).
• Resetting your email address now requires confirmation of your password, preventing session hijacking attempts that look to get a password recovery email sent to the hijacker's email account (bug #20185).
• There was renewed discussion about various outstanding proposals to modify the standard diff colours for MediaWiki (some wikis, including the French Wikipedia, have already chosen to move away from the green and yellow that ships by default). An additional proposal was to offer a high contrast version as an optional preference (wikitech-l mailing list).
• Two wide-ranging threads on the wikitech-l mailing list focussed on updating the current advice for contributors to the MediaWiki codebase. "On refactorings and huge changes" focussed on an appeal by Chad Horohoe to developers to work on their own branches, whilst a separate appeal made by Lead Software Architect Brion Vibber was entitled "Please don't commit broken code".
• With the resolution of bug #29402, fundraising banners are unlikely to be displayed on mobile devices this year in their now standard image-heavy form.
• Deputy Director Erik Möller led calls on 29 July to honour the spirit on sysadmin appreciation day and thank those at the Foundation responsible for keeping the servers running (wikitech-l mailing list).
  
  Reader comments
  

If articles have been updated, you may need to refresh the single-page edition.

The Signpost is written by editors like you – join in!

Home

About

Archives

Newsroom

Subscribe

Suggestions