Taking Jimmy Wales up on his stated goal of "Britannica-or-better quality" for Wikipedia articles, the scientific journal Nature has conducted a study pitting Wikipedia against the Encyclopædia Britannica. In an article by Jim Giles, "Internet encyclopaedias go head to head", it reported that Wikipedia approaches the quality of Britannica for articles on scientific topics, but hasn't fully matched this standard yet. It was one of the first formal peer reviews that used experts to compare Wikipedia articles with those of another publication.

The results of the study found that in the sample of 42 articles, Encyclopædia Britannica had 123 errors while Wikipedia had 162 (for averages of 2.9 and 3.9 errors per article, respectively). The study was widely discussed in the media, usually interpreted as reflecting favorably on Wikipedia in the aftermath of the Seigenthaler incident, although some coverage emphasized that it showed Wikipedia to be more error-prone.

In an accompanying editorial, Nature encouraged readers to review and help improve articles on scientific topics related to their work. Based on a survey the journal conducted, it noted that relatively few scientists edit Wikipedia (less than 10% of Nature authors). However, the editorial argued that these contributions would be critical "to increase the quality of the mushrooming number of entries." A separate item profiled the efforts of two such contributors, Vaughan Bell and William M. Connolley. Meanwhile, Nobel Prize winner Barry Marshall has reportedly endorsed the Wikipedia article on Helicobacter pylori, the discovery for which he and Robin Warren received the award. (Helicobacter pylori is, in fact, a featured article.)

Although the source of each article was not identified to reviewers in the Nature study, several people commented that a reader likely would have been able to tell the difference based on the content. In media coverage about the study, Britannica officials pointed out that the focus on factual errors neglected considerations of style and writing quality, while spokesman Tom Panelas gave Wikipedia an apparently backhanded compliment for its coverage of such topics as extreme ironing.

A blog entry by a Nature staffer expressed it as a "final score" of 22-10 in favor of Britannica, based on the head-to-head results for each article (the reviews on ten articles resulted in a "draw"). Wikipedia did come out ahead in one sense, 4-2, in the number of articles deemed to have zero errors (these were Subrahmanyan Chandrasekhar, Lipid, Punctuated equilibrium, and Quark).

The study indicated that the articles were chosen to be of similar length, occasionally helped by trimming material like reference lists. Attempts to determine exactly which revisions of Wikipedia articles had been studied proved difficult, however. An initial reconstruction suggested that Wikipedia's articles were generally longer and therefore actually had fewer errors relative to the amount of content. But this conclusion seems to have been erroneous, as the reconstruction failed to consider all of the material on Britannica's site that is considered a single article. Nevertheless, after attempting to correct for this problem, Wikipedia editor Steven G. Johnson said he remained "mystified" by some of the apparent differences in length.

Nature indicated that they were looking into the possibility of publishing the full list of errors with an explanation of how they were counted. Reviewers may also be identified if they give permission. Preparing this material is expected to take a little time, however. In the meantime, Wikipedia articles that were found to have errors have been flagged with a new "NatureDispute" template.

Tied in with this exploration of Wikipedia from an academic perspective is a long-running debate over the practice of citing Wikipedia (see archived story). While some condone this, it is not generally accepted as good academic practice, and even the press has become more careful about citing Wikipedia since the Seigenthaler controversy. As the Nature editorial put it:

"Researchers should read Wikipedia cautiously and amend it enthusiastically."

The return to this issue last week was prompted by an interview with Jimmy Wales, conducted by BusinessWeek's Burt Helm, in which Wales was quoted saying that students and researchers shouldn't cite Wikipedia in their work. This quote was picked up and repeated in a number of other places as Wikipedia's founder telling people not to cite it. Less noted was his general comment that "People shouldn't be citing encyclopedias in the first place", advice that predates the internet itself, as Jason Fry of the Wall Street Journal observed in an online column.