The Signpost

Technology report

Signpost investigation: code review times

Code review figures mixed, improving

The chart that accompanied last month's story about code review times, the vagaries of which prompted the present investigation.

Late last month, the "Technology report" included a story using code review backlog figures – the only code review figures then available – to construct a rough narrative about the average experience of code contributors. This week, we hope to go one better, by looking directly at code review wait times, and, in particular, median code review times

To this end, the Signpost independently analysed data from the first 23,900 changesets as they stood on September 17, incorporating some 66,500 reviews across 32,100 patchsets. From this base, changes targeted at branches other than the default "master" branch were discarded, as were changesets submitted and reviews performed by bots. Self-reviews were also discarded, but reviews made by a different user in the form of a superseding patch were retained. Finally, users were categorised by hand according to whether they would be best regarded as staff or volunteers.[nb 1] Although this week's article focuses mainly on so-called "core" MediaWiki code, future issues will probe extension-related statistics.

WMF bosses will, on the whole, be pleased with the final figures. 50% of revisions to core MediaWiki code submitted during August was reviewed for the first time in just 3 hours 30 minutes, with 25% being reviewed in 20 minutes and 75% within 27 hours. These figures were similar across both first patchsets and later amendments, and similar with regard to slight changes in what qualified as "a review".[nb 2] The relevant trend over time is considered in the following tables. On the left is review time across all patchsets submitted to core; with the right hand table, just the first patchset in any given changeset is included.

Month 25% Median 75% Current mean
May 42 minutes 4 hours and 25 minutes 1 day, 11 hours and 27 minutes 3 days, 3 hours and 38 minutes
June 47 minutes 19 hours and 10 minutes 3 days, 16 hours and 45 minutes 5 days, 8 hours and 29 minutes
July[nb 3] 39–40 minutes 7 hours and 4–8 minutes 2 days, 5–9 hours 2 days, 16 hours and 38 minutes
August[nb 3] 20–21 minutes 3 hours and 11–29 minutes 21–24 hours 1 day, 11 hours and 52 minutes
Month 25% Median 75% Current mean
May 38 minutes 3 hours and 27 minutes 1 day, 5 hours and 4 minutes 2 days, 1 hour and 58 minutes
June 45 minutes 12 hours and 34 minutes 2 days, 13 hours and 31 minutes 3 days, 7 hours and 39 minutes
July 22 minutes 3 hours and 16 minutes 1 day, 7 hours and 21 minutes 1 day, 17 hours and 18 minutes
August 19 minutes 3 hours and 33 minutes 19 hours and 50 minutes 1 day, 1 hour and 26 minutes

The data show, then, that there has been a marked improvement in getting followup patchsets reviewed quicker, while review times for "first attempt" patchsets have improved less dramatically. Other analyses are more concerning. For example, a volunteer-written patchset waits, on average (either median or mean) twice as long as a staff-written one for its first review, although the gap has closed from three times as long in June and July. Staff provide 86% of first reviews for core, with just five staff members collectively accounting for some 55% of the total.[nb 4] Moreover, even in August, more than 5% of patchsets targeted at core waited a week for their first review.

As with all large datasets, it is difficult to rule out subtle methodological issues and in any case unideal to pinpoint trends over as short a period as four months. The full data set is available upon request.

Notes

  1. ^ One notable side effect of the methodology employed was the exclusion from the final analysis of patches that were abandoned or amended without review, even if the user had intended for them to be so reviewed and/or the amendments were minimal. Future analyses may wish to refine their methodology to take this into account.
  2. ^ Specifically, the in/exclusion of reviews not assigning numerical scores and the in/exclusion of force abandonments and reversions.
  3. ^ a b Ranges indicate the possible impact of patchsets still awaiting a review.
  4. ^ The equivalent figures for core plus WMF-deployed extensions are 95% and 43% respectively.

In brief

Signpost poll
External sites
External users deserve: consideration and resources (9%); consideration (49%); neither, but they should be able to piggyback (26%); caution (14%); other (3%)
You can now give your opinion on next week's poll: In my view, the WMF's code review priority should be...

Not all fixes may have gone live to WMF sites at the time of writing; some may not be scheduled to go live for several weeks.


















Wikipedia:Wikipedia Signpost/2012-09-24/Technology_report