The Signpost

Technology report

WMF and the German chapter face up to Toolserver uncertainty

Wikimedia Deutschland's CEO Pavel Richter attributes the organisation's compromise over Toolserver funding to the establishment of the multi-million dollar Wikimedia Labs project.

The Toolserver is an external service hosting the hundreds of webpages and scripts (collectively known as "tools") that assist Wikimedia communities in dozens of mostly menial tasks. Few people think that it has been operating well recently; the problems, which include high database replication lag and periods of total downtime, have caused considerable disruption to the Toolserver's usual functions. Those functions are highly valued by many Wikimedia communities, comprising data reports on the relationships between pages, categories, images, and external links; support for Wiki Loves Monuments, OpenStreetMap and GLAM projects; talk-page archiving services; edit counters; and tools aimed at easing many automated administrative processes such as the account and unblock request processes on several major wikis, as well as cross-wiki abuse detection.

How did the Toolserver start?

It was originally set up in 2005 through the donation by Sun Microsystems of servers to Wikimedia Deutschland (WMDE); so it was almost by coincidence that the German chapter was prompted to take on responsibility for the project. WMDE has since invested heavily in Toolserver infrastructure and its operations—an unusually global role for a chapter, resulting from the particular nature of its revenue streams and German charity laws. There has been in-kind support from the Wikimedia Foundation, mostly in the form of database replication and space in its Amsterdam data centre (valued at US$65k a year), as well as financial grants to expand the hardware (example). Nevertheless, WMDE still makes up the bulk of the general budget of about €100k (US$130k); other chapters, such as Wikimedia UK, have also made smaller contributions.

Wikimedia Labs vs Toolserver: a comedy of errors?

In 2011, the Foundation announced the creation of Wikimedia Labs, a much better funded project that among other things aimed to mimic the Toolserver's functionality by mid-2013. At the same time, Erik Möller, the WMF's director of engineering, announced that the Foundation would no longer be supporting the Toolserver financially, but would continue to provide the same in-kind support as it had done previously.

DaB is the volunteer who administers the Toolserver, and who in the process has acquired unique expertise for running the system. (WMDE has also contracted Marlen Caemmerer to assist in Toolserver administration since October 2011.) DaB told the Signpost that there is a simple reason for the recent degradation in performance: the Toolserver's hardware was not added to in 2012, while more tools have been written and more people are using the tools. The German chapter, he says, has refused his request to extend the hardware infrastructure, giving only a vague commitment of support. But its September forward planning allocates just a fraction of last year's funding.

DaB's comments are a reference to a message from WMDE's CEO, Pavel Richter, who publicly reassured Toolserver developers this week that "Wikimedia Deutschland will make all necessary investments [including new hardware] to keep the Toolserver up and running", but said that the chapter could not ignore the existence and growth of Labs. The movement now faces a complex challenge in working out how to maintain continuous support of the tools, a complexity that is obvious from recent debates (conducted in German) on Meta and the German Wikipedia; moreover, DaB has threatened to resign if WMDE does not allocate funds for hardware purchase.

What the WMF didn't anticipate, and what it now seems as though they're naively ignoring despite the outcry, is that WMDE doesn't have anything like the foundation's eight-figure budget, and apparently the WMF has decided the Toolserver is going to get the short end of the stick when it comes to funding.

 — Hersfold

Richter's reference to Wikimedia Labs' rapid growth prompted WMF deputy director Erik Möller to express the Foundation's thinking (full version, including rationale) in response to questions raised about the scenario:

It is true that we (the WMF) have ... asked WMDE to work with us in transitioning from Toolserver to Labs. ... Chapters are autonomous organizations, and it's WMDE's call how much / whether it wants to continue to invest in [the Toolserver] ... However, for our part, we will not continue to support the current arrangement ... indefinitely. The timeline we've discussed with Wikimedia Germany is roughly as follows:
  • Wind down new account creation on Toolserver by Q2 of 2013 calendar year
  • Decommission Toolserver by December 2013
Möller accepted that Labs, while well-resourced both in terms of processing capability and storage space, is not yet suitable for Toolserver migrants, lacking (among other things) both database replication and a "Quick Start" mode for users uninterested in Labs' capability for custom server setups. While funding has been put aside for developing such features, Möller would not commit to targeted WMF funding for tool transition, and therein lies the cause of concern among volunteer Toolserver developers: that they could be left facing a switchover deadline without being in a position (lacking either the time, the capabilities, or both) to migrate their tools themselves. They are concerned, then, that only time will tell what will happen to these popular but difficult to migrate tools, to whose continued existence both WMDE and the WMF seem unwilling to commit.

English Wikipedia arbitrator Hersfold was closely involved in writing the "unblock ticket request system" (UTRS), which allows blocked users—including innocent parties caught up in range-blocks—to appeal their blocks. UTRS, created only recently and now officially mandated by the Foundation, is written for the Toolserver, not the Labs environment. Hersfold told the Signpost:

How Labs functions seems to be almost completely different from how the Toolserver functions. We've been told multiple times that Labs will provide lots of "beefy" infrastructure for tools development; ... users will be able to set up virtual machines, or "instances" ... to handle their development, and submit new programming code to a shared location. As one may expect from the Foundation, it's a very collaborative setup. Once inside their instance, a user can more-or-less do whatever they want; install MediaWiki, run a bot, set up web pages for tools, whatever. But most people on the Toolserver don't need "beefy"; we just need a web server that will let us run our tools and access the databases holding information about Wikipedia and the other projects. If someone needed "beefy," they'd have set up their own server ages ago. While Labs is all swishy and fancy (and presumably has less downtime than the Toolserver), it's an environment we're all completely unused to, and perhaps worst of all, it provides no access to the Wikimedia databases, which will prevent most tools and bots from working at all. Supposedly this functionality will be available at some point in the future [editor's note: planned for the first quarter of 2013] ... I don't think either organization fully realizes how much Wikipedia, the Commons, and all the other projects rely on the tools provided by the Toolserver ... [if it goes,] most of the tools and bots we take for granted will suddenly cease to function.

Carl, another developer, agreed, "labs will be useful for some projects, particularly for developing MediaWiki extensions. [But] the current plans seem to be intentionally preventing [other] Toolserver users from simply migrating their tools to Labs; the result will be a great leap backwards when/if the toolserver is taken offline."

The Signpost understands that a further sticking-point is licensing: while recommended to, some tool operators have not released their code under a free license, which is a requirement for using Labs (one operator has stated he legally cannot do so, since he created the tool using his company's computer systems, so the company holds the copyright).

An earlier version of this article incorrectly asserted that access to the Wikimedia databases would occur in December 2013. It is actually planned for the first quarter of next year.

In brief

Signpost poll
Code review priorities
In my view, the WMF's code review priority should be...: reducing the disparity between volunteers and staff: 29%; widening the pool of reviewers: 18%; reducing the average/maximum wait time: 12%; other / unsure / impossible to pick: 41%.
You can now give your opinion on next week's poll: How often do you use the Toolserver?

Not all fixes may have gone live to WMF sites at the time of writing; some may not be scheduled to go live for several weeks.

At the time of writing, 14 BRFAs are active. As usual, community input is encouraged.

















Wikipedia:Wikipedia Signpost/2012-10-01/Technology_report