Peer review – a history and call for reviewers

Special report

Peer review – a history and call for reviewers

Tom (LT910001) has been a Wikipedia editor since 2013. He is an active anatomy editor who also dabbles in good article reviewing, template discussions, and occasional edits to medical articles. He discovered peer review in 2013 and has become very active in the process since.

English Wikipedia's peer review process is a relatively unknown yet fairly active venue for feedback. Editors can request a review of an article, regardless of its stage of development, in general form or highlighting specific areas they are looking to improve. One or more editors may weigh in.

Here, I outline the history of peer review and my contributions to it, and detail future plans and a request for more reviewers.

Introduction

Peer review runs similarly to other processes such as good articles and featured articles: an editor may post a request, and one or more editors answer that request, critiquing the article. Editors who submit their work for review range from novices with their first article to those seeking feedback before nomination to the good article or featured article forums.

The peer-review venue was created in 2003 by Wapcaplet (now inactive); the first article reviewed was States of the United States. The process quickly gained popularity. Reviews were manually added to a list, until a 2005 redesign based on transclusion. Editors had to manually add and remove articles until 2008, when the processes for adding articles to the list and for closing old reviews were automated by VeblenBot (inactive), created by CBM (inactive), and reviews were split into categories.

Processes and procedures

After creation, peer review rapidly developed a set of instructions and advice. A "policy" page formalising review closure criteria had developed by 2005, created first by Bishonen. Ta bu shi da yu (inactive) began to archive old reviews from 2005. A list of volunteers by topic area was created in 2007 by Marskell (inactive). From 2008, Ruhrfisch began a backlog of unanswered reviews. In 2012 an "alert" template cautioning editors from directly posting reviews on the main peer review page was created by Ruhrfisch.

I first encountered peer review in 2013. In comparison to processes such as "good articles" and "featured articles" I found it messy and disorganised. These had developed friendlier interfaces, clearer instructions, had split into several pages for ease of use and load times (a crippling issue for me, using wireless internet in a rural area at the time), had a greater degree of automation, and more active users. These had changed with time; peer review appeared mostly unaltered for many years. Three things struck me:

Repetitive, mind-numbing processes existed which could be eased or automated
Processes had been intermittently stalled because of a bus factor of 1 (Emphasised by the poetic interludes of Ruhrfisch, styled on Geometry guy (inactive): "Still getting darker in the north, lighter in the south, such is the time of year" [1] [2] [3]
There was significant duplication among pages. This made it confusing to work out the "definitive" version of any set of instructions and also made it harder to improve and discuss change.

Finding the actual peer reviews very useful, I have since striven to gradually improve the process, with discussion and consensus. My goal has been to make it easier to use, and therefore more widely used, as part of Wikipedia's growing infrastructure for supporting new editors, and another way of helping users non-confrontationally gather feedback about their articles.

I started by creating a tabbed header of commonly used pages, similar to good and featured articles; and by branding the peer review pages with a colour scheme. I created a sidebar with other main pages (identified via "what links here" and also "subpages") and put links to as many relevant templates into the "tools" subpage, where I tried to document the byzantine template- and bot-based technical underpinnings of the peer review process.

A sample:

The Category Format (CF) templates are a suite of templates to be used in conjunction with automatic category listing. An automatically listed category is a category which has an associated category list page where the items in the category are listed ... for maximum flexibility, the entries on the category list page are formatted using a subtemplate of CF named after the category. For instance each page in an automatically listed Category:foo has a corresponding entry.

I have centralised information, truncated text, simplified instructions, and made small changes to ease reviewing (such as including the date of creation and last edit on all displayed reviews; slipping in a request for reciprocity in instructions). I have simplified the technical side of things for future editors (this has included creating simpler and comprehensive documentation; centralising the 11 identical templates; simplifying the archive process.) We now link to categories of monthly reviews; previously, an editor had to copy and parse reviews each month, a time-consuming process that understandably was often months behind.

Problems

Reviews would not transclude to the main page during mid-2015, due to a technical cap on the amount of text that can be transcluded. When this is exceeded, nothing transcludes and, with great foresight, peer review receives a message from VeblenBot. With too many reviews (and delays in automatic closure), this limit had been exceeded. We quickly implemented a fix – reviews of a certain length now display their text on the main page. After this limit, a link is provided to the main review. I hope in the future that we continue to have this problem, because it indicates that the process is being used more and more.

We've had long-running issues with on-again–off-again bots over the last two to three years, which has impacted the displaying of new reviews, and the closing of old reviews. I hope this has been finally solved by AnomieBOT (thank you!!). The bot was designed in response to requests at the technical village pump and bot requests, and even a failed attempt I made to learn bot coding. This again illustrates the frustrating aspect of a small bus factor, particularly for such a critical bot as VeblenBot.

The "unanswered reviews" list was automated in 2016. This replaced a "backlog" template that was infrequently edited, with months often lapsing between edits due to the complexity and time-consuming nature of the task. An editor has to manually scan the list of reviews (daily if the template were to be kept up to date) and add reviews older than a certain duration to the template; and check all reviews on the template to see if they had been reviews. This required a lot of effort and months often passed without reviews. The current automatic method is imperfect and not entirely satisfactory, but at least provides a more up-to-date list. It displays only reviews that have never been edited. I should mention and thank on behalf of the community Cirt (inactive) and Brianboulton (active), who maintained the backlog list for some time.

Call for reviewers

Peer review is busier and busier than ever, and is attracting more and more reviews. This is great news and I'm pleased to see the diverse nature of requests posted. But we're still plagued by a lack of reviewers; there's nothing special needed to become a reviewer, and frankly not that much time needed for a review.

Most editors, in my experience, are seeking some general tips about the articles they are editing – general help about sourcing, structure, titles, images and so forth. Interested editors have the potential to help ease newer editors' introduction to Wikipedia – this may appeal particularly to those civic-minded people who help out at the teahouse.

Editors seeking review often do so for no other reason than a desire to improve articles and better write articles in the future. By helping these editors, who are often new, you can make them feel welcome, and make a positive contribution to their future edits and to the encyclopedia as a whole. I hope I've given you an idea of the evolving nature of peer review, and that you'll consider participating! Please drop by and have a look at peer review every so often, and pick up one or two reviews.

← Previous "Special report"

Next "Special report" →

In this issue

27 February 2017 (all comments)

Special report

Discuss this story

These comments are automatically transcluded from this article's talk page. To follow comments, add the page to your watchlist. If your comment has not appeared here, you can try purging the cache.

Peer review is something I need to do more of. As a comment, what would be really good would be setting up some kind of "expert peer review" (maybe as part of review for an FA submission?) in which actual, credentialed experts are invited in to give a public assessment of articles. I've heard many times from experts that they grumble about how Wikipedia is bad - not just factual mistakes, that an article overemphasises the wrong things, or spends too much time talking about a theory most experts on the topic think is on the way out, or the hot new controversial idea that isn't very plausible. (See, for example, Emily Gould's comments on her article.)

It would be nice if we could actually invite people who know about a topic to publicly say this stuff and use it as a guideline for improving an article. I often find that experts on a topic don't automatically make the best writers about it - they may be too embedded in a point of view to give due weight to different sides of a debate - but having this kind of opinion on the talk page would be nice. Finding actual qualified experts like academics - or publicly known people like journalists, political consultants or lawyers - willing to donate their time to this kind of thing is the kind of thing the WMF sounds like it would be well-placed to negotiate (it would probably be a more efficient use of their time than actually editing the article directly and figuring out how the formatting works). How about it, guys? Blythwood (talk) 03:53, 28 February 2017 (UTC)[reply]

This has been tried a number of times, in various areas, and has produced some good results, but is very difficult to sustain. I've done it with both medical and museum experts, and WikiProject Medicine has taken an interest in the question for a long time. Part of the problem is that proper experts usually need to be given an article that is already at a pretty decent standard to get good detailed reviews. If it is full of issues they tend just to say so briefly, perhaps understandably; they are used to peer-reviewing work that is thought by the authors, professionals in the field, to be ready for publication. They are busy people and WP reviews tend to stay at the bottom of the in-tray unless they have a particular motivation to push them through. Also, their real top-level expertise may be limited to very specific areas within their subject and they can be reluctant to pronounce outside that, as they don't keep up as fully with the research. We have a number of regular editors closer to the pre/post doctoral level, and in practice they tend to give better sustained results. Johnbod (talk) 15:37, 2 March 2017 (UTC)[reply]

It's interesting to note that not much actually changed to the Emily Gould article, despite her comments.. 47.188.47.96 (talk) 16:40, 7 March 2017 (UTC)[reply]

Many thanks to Tom for his work improving Peer Review and for this article. For several years we had an agreement that no peer review request went unreviewed (which initially came about from how bad it felt to manually archive PR requests that were still unanswered after weeks or months at PR), and many editors chipped in to help with the task, including several already mentioned. The four who I recall doing the most reviews were Finetooth, Brianboulton, The Rambling Man and myself (apologies to all those I did not mention by name). The problem has always been getting enough reviewers.

Reviewers at PR have theoretically always been required to look at at least three things: they need to be able to 1) make suggestions on how well the article is written, 2) give advice on how well the article meets the Manual of Style requirements, as well as keeping an eye on what was needed to pass GAN or FAC or FLC, and 3) say how well the article covers the subject matter. The first two were relatively easy for experienced reviewers, while subject matter is often the most difficult part of a review (though even there some articles have obvious omissions, or do not do a good enough job making the topic clear to an interested layperson).

In an effort to meet the subject area requirement, for a while reviewers listed themselves by subject areas in which they felt they had some ability to review articles. One problem with that was a mismatch between articles at PR and the reviewers available (so there might be a large number of video game or pop music or sport articles per reviewer in those areas, while there were reviewers in more arcane areas (covered bridges, say) that rarely had articles nominated at PR. Another problem was that the reviewers would leave WP or lose interest in reviewing, which frustrated editors who would ask them for a PR and get no repsonse. Another problem was that many times an article would be nominated at PR in good faith by an editor thinking it was ready for FAC, when it had major issues with the writing and/or MOS that needed to be addressed first.

If there is an expert reviewer system, then it might make sense to have a "director" or "editor in chief" who assigns articles - this person would presumably be able to filter out the poorly written and non-MOS compliant articles, and would know who the expert reviewers were who were available and what their recent PR workloads had been. I am still not sure how expert reviewers would be recruited, but it is always good to give it a try and see what happens. Good luck, Ruhrfisch _><>°^° 16:36, 16 March 2017 (UTC)[reply]

PS I did not realize my edit summaries to Geometry guy were poetry. It's always good to / see ones' self through the eyes of / another - surprise!

Did you mean for this to be in Category:X1?

@LT910001: in an early edit, you put [[Category:foo]] in an example. A bot changed it from foo to X1. Did you mean to put this page into that category, or did you mean to type [[:Category:foo]], which would have rendered as Category:foo without adding the page to any category? davidwr/(talk)/(contribs) 22:40, 18 March 2017 (UTC)[reply]

Thanks davidwr for pointing this out. This was originally a quote... Fixed. --Tom (LT) (talk) 01:29, 19 March 2017 (UTC)[reply]

Make sure we cover what matters to you – leave a suggestion.

Home

About