The Signpost

Technology report

Search gets faster, GSoC gets more detail and 1.20wmf2 gets deployed

April engineering report published

In April 2012:
  • 67[updated 1] unique committers contributed code to MediaWiki.
  • The total number of unreviewed commits went from about 100 to 138.
  • About 34 shell requests were processed.
  • 63 developers got developer access to Git and Wikimedia Labs, of which 60 are volunteers.
  • Wikimedia Labs now hosts 81 projects, 136 instances and 305 users.

Engineering metrics, Wikimedia blog

The Wikimedia Foundation's engineering report for April 2012 was published this week on the Wikimedia Techblog and on the MediaWiki wiki, giving an overview of all Foundation-sponsored technical operations in that month (as well as brief coverage of Wikimedia Deutschland's Wikidata project). Three of the headlines for the month have already received coverage in previous issues of the Signpost: the selection of nine Google Summer of Code students (explored in more detail below), the shift to a rapid deployment cycle with the deployment of 1.20wmf1 and a new version of the Wikimedia iOS app. Of the two others, one relates to work on a document detailing Wikimedia engineering’s goals for the next fiscal year, which will be featured in the "Technology report" as soon as it becomes official, and the deployment of a new mobile skin (example), which occurred after the publication of last week's issue. The skin update has since received broadly positive commentary among users; it provides a rival to the more native experience of an Android or iOS app.

Elsewhere, the roundup contained details of a massive improvement in the amount of time taken to use Wikimedia's Lucene-based internal search engine after "months of preparation and refactoring work". The difference reported was "quite amazing": the actual search component of 99% of search requests now takes under a second, down from nine seconds before; and the average search time is now 100 ms, down from 700 ms. Among the interesting updates included in the report was the news that the localisation team had started coding for a universal language selector, over 18 months after the feature was first proposed.

Coding has begun on a full system of Lua scripting, while April also saw over 20x improvements in the processing speed of the new parser on template heavy pages, suggesting that preparations for its rollout (a prerequisite for deployment of the new Visual Editor) will begin shortly. Among the relative failures of the month was the deployment of a new media caching layer ("Varnish"), which, while it has the potential to improve performance and scalability, seems to be preventing users from downloading large files successfully (bug #36577).

Corrections:

  1. ^ Originally reported as 53, later revised upwards.

Google Summer of Code students and their project

Google Summer of Code participants are officially still in the time set aside by Google for pre-coding work, such as deciding on the data flows their project will focus on (example diagram pictured).

As announced a fortnight ago, nine students have now been selected to work on MediaWiki this year, supported by Google stipends and WMF mentors (Wikimedia blog). The projects they represent fall across a broad spectrum: some, like Aaron Pramana's project to rethink the display and functionality of Wikimedians' watchlists, involve highly visible changes; others will have a more indirect effect on the average Wikimedian user experience (such as Suhas HS's project to improve the OpenStackManager extension that underpins the virtualisation functionality of Wikimedia Labs and Robin Pepermans' attempts to improve the usability, performance, and coverage of Wikimedia's Incubator for nascent language editions). It will be the first time many of the students have undertaken such ambitious projects in the name of open-source development; they join hundreds of other students worldwide, each working on different projects for different open-source initiatives (several countries are represented even among the WMF's nine students).

In terms of focus, three of the nine projects selected for WMF mentorship relate to media handling: Ankur Anand will work on integrating Flickr upload and geolocation into Wikimedia Commons' UploadWizard, Platonides on a new cross-platform mass media uploader with a broader function set than existing tools, particularly with regard to image upload "campaigns" such as Wiki Loves Monuments, and Harry Burt (the author of this Signpost report) on an upgrade to the Translate extension to allow it to translate Scalable Vector Graphic (SVG) files and with a view to eventual deployment to Wikimedia Commons.

Other projects defy such easy categorisation by topic; these include Akshay Chugh's work on a convention/conference extension for MediaWiki that will ease the job of meetup organisers, Ashish Dubey's attempts to get real-time collaboration integrated into the upcoming visual editor (a topic that hit mailing lists again this week), and last but not least Nischay Nahata's attempts to optimise the performance of the Semantic MediaWiki (SMW) extension. If particularly successful, the last project could clear the way for its use on a test Wikimedia wiki (see previous Signpost coverage).

Students will officially start coding later in the month, although they may begin when they wish. They must present their final work in late August for evaluation.

In brief

Signpost poll
Diff colours revisited
You can now give your opinion on next week's poll: Which of the following best describes your thoughts on Wikimedia's internal search engine?

Not all fixes may have gone live to WMF sites at the time of writing; some may not be scheduled to go live for many weeks.

At the time of writing, 14 BRFAs are active. As usual, community input is encouraged.

















Wikipedia:Wikipedia Signpost/2012-05-07/Technology_report