Bugs, Repairs, and Internal Operational News
MediaWiki 1.17 deployment failed, postponed
The planned update of MediaWiki as the underlying software which forms the basis of WMF wikis to version 1.17 failed last week (Wikimedia Techblog). The original deployment was expected to begin 07:00 UTC on February 8 (see previous Signpost coverage), but preparations took longer than anticipated and actual deployment began at around 13:00 UTC.
Several issues became apparent almost immediately. The parser cache miss rate almost doubled with the new deployment, at which point the Apache servers, which are responsible for delivering content to users, became overloaded and started behaving unpredictably. The increased load culminated with multiple issues across the project from increased lag to even outage for some users. At this point, the deployment was rolled back to the previous 1.16 release. The tech team investigated and prepared for another attempt after resolving some technical issues. A second attempt was made at 16:27 UTC, but this ran into similar performance issues and had to be called off 90 minutes later. Further attempts were put on hold.
Danese Cooper, Wikimedia's Chief Technical Officer, blogged about the failed deployment and explained what the Foundation had attempted to deploy:
| “ | The 1.17 release process has been longer than we would have liked, which has meant more code to review, and more likelihood for accumulating a critical mass of problems that would cause us to abort a deployment.... [it] was an omnibus collection of fixes, including a large number of patches which had been waiting for review for a long time. The Foundation’s big contribution to the release was the ResourceLoader, a piece of MediaWiki infrastructure that allows for on-demand loading of JavaScript. Many other incremental improvements were made in how MediaWiki parses and caches pages and page fragments. | ” |
After further investigation and several fixes to the release, Rob Lanphier, a developer with the WMF, added that "some of the unsolved issues are complicated enough that the only timely and reasonable way to investigate them is to deploy and react". As a result of this, he said, a new plan had been drawn up in which 1.17 will be deployed on "just a few wikis at a time". The tech team believes the problem was located in the configuration of the $wgCacheEpoch variable, which caused a more aggressive culling of the cache than the servers could handle (Wikimedia Techblog).
The team decided on a two-stage deployment for their next attempt (reviving some old code for project-wise upgrading). The first phase took place 6:00–12:00 UTC on Friday, February 11. This was limited to the Simple English Wikipedia and Wiktionary; the Usability and Strategy Wikis; Meta; the Hebrew Wikisource; the English Wikiquote, Wikinews and Wikibooks; the Beta Wikiversity; and the Esperanto and Dutch Wikipedias.
At the time of writing, the deployment had been completed on all but the last two projects. The Hebrew Wikisource, included after a request from a community member, gave a chance to observe the deployment on a right-to-left language wiki. The team also reported some localization issues which triggered ParserFunction bugs on both nl.wikipedia.org and eo.wikipedia.org. The traffic from nl.wikipedia.org was enough at the time to cause a noticeable spike in CPU usage on the web servers, including some time-out errors; thus, deployment onto nl.wikipedia.org had to be delayed. After these issues are resolved , the second wave of deployment is expected to start on Wednesday, February 16 (see the current list of WMF wikis that are already running 1.17).
An IRC office hour Q&A was held on matters related to the ResourceLoader, which is expected to cause compatibility issues with some existing Javascript code. Trevor Parscal and Roan Kattouw, the main developers of the ResourceLoader, were available on IRC on February 14 at 18:00 (UTC) to answer queries related to the new feature.
In brief
Not all fixes may have gone live to WMF sites at the time of writing; some may not be scheduled to go live for many weeks.
- RefTools default: Through a poll at the Village Pump, it was decided by near unanimous consensus to make the refTools gadget a default feature on the English Wikipedia. Implementation discussion is proceeding at MediaWiki talk:Common.js, and the switch is currently on hold until MediaWiki 1.17 is deployed (see also last September's Signpost dispatch on RefTools/RefToolbar and other references-related tools).
- New wikitext parser? After the last attempt at creating a wikitext parser generated excitement and lengthy discussion on the wikitext-l mailing list, a new attempt, by AboutUs.org developer Karl Matthias, has been brought to light under the name of "Kiwi". Discussion is ongoing in the post's comments and on wikitext-l.
- Distributed wikis: One of the applications of a defined parser grammar would be the ability to have alternative, non-MediaWiki interfaces to interact with the content. The full video was released of "Distributed wikis", a talk at linux.conf.au by Brianna Laugher (User:pfctdayelise; abstract) that touched on the subject. She proposes the use of the write function of the API and custom interfaces to create WikiProject-sized "forks" as the key to the long-term survival of the project, arguing that community decentralization would create smaller, more manageable and more welcoming groups of editors. Laugher compares these to the editing groups that existed at the beginning of Wikipedia, and those that can still be found in various specialized wikis (most notably wikias). Similar ideas have been floating around for a while; see meta:Versioning and distributed data.
The talk also examined the possible application of the concept of distributed revision control to wikis, to allow easy forking and merging (a concept explored in various earlier proposals), and compared Flagged revisions to release versions in software development. - Call for skins: A call for skins was made on the wikitech-l mailing list.
- Public status-monitoring page: The WMF announced that a public status monitoring page has been set up at http://status.wikimedia.org to allow staff, community members and the general public to keep an eye on the servers' uptime and status (Wikimedia blog). The service, powered by Watchmouse, has been in the works for a while, having been mentioned in the last three WMF Engineering Updates. Its relationship with previous monitoring tools Nagios and Ganglia is still unclear.
- Bug fixes: Several long-standing bugs have been fixed:
- The successful close of bug #17160 (opened January 2009) allows for the display of namespace aliases in userpages to be gender-specific (revision #82029). As The Signpost reported last week ("Widespread discussions about the low participation of women in Wikipedia"), the blanket use of masculine namespace forms—such as Benutzer (male user) rather than Benutzerin (female user) on the German Wikipedia—had come up in the recent discussions about Wikipedia's "gender gap", where it was criticized as "hardcoded discrimination" and Sue Gardner had called it "awful".
- Bug #16013 (opened October 2008) was resolved in revision 81573, allowing the direct linking to sections (tabs) of the preferences page.
- Bug #8680 (opened January 2007) was closed successfully, after the introduction of $wgFooterIcons allowed easy customisation of MediaWiki sites' footers.
Discuss this story