Please click here to download Improving science through online commentary: Nature (2003), 423:15.

 

The idea:

Progress in science depends on public debate and criticism of ideas. Unfortunately, there is a poverty of forums for immediate public debate in science. Debate does occur, but is largely restricted to three domains: conferences, private conversations, and peer-reviewed publications. The first two channels tend to be too private and ephemeral to help the community at large, and the third channel - publication in peer-reviewed journals - runs on a time scale of years. Debate through journal publications becomes one in which people can speak only a few times a year, and, unless they bring new data, they are lucky to be allowed to speak at all.

This deficiency of fast communication leads to a problem that slows science: many people in the community do not know other colleagues’ views on published manuscripts. Often, when a manuscript is published, researchers around the world spot logical flaws, experimental weaknesses, questionable assumptions, or alternative interpretations. However, these insights rarely disseminate quickly, especially since individual criticisms may not be considered important enough to warrant their own publication in a peer-reviewed journal. Before a major criticism appears in publication, the slow publication cycle ensures that months or years go by. The end result is that other researchers inevitably fall prey to some of the same mistakes as the paper they cite. A further, related problem, is that scientists often miss official retractions, continuing to accidentally cite withdrawn data [1].

We propose a simple solution to these problems. Each record in publication databases - specifically, we are thinking of the National Library of Medicine’s PubMed database - should have an associated link for the posting of scientific commentary: essentially an electronic Post-it® note. The submitter of a comment would be required to provide his/her name, email address, a short communication, and a password (for later editing or deletion). Insights, criticism, replications, and non-replications could be posted by readers of journal articles.

Some of the benefits of implementing this mechanism would be:
1. Immediate free and open debate of scientific ideas and results.
2. Easy dissemination of non-replications and negative results.
3. A reduction in wild-goose chases by researchers unaware of their colleagues’ insights into a published paper.
4. Posting of links to subsequent papers that may provide important verifications or contradictions.

The accrual of commentary on a database already enjoys many successful existence proofs. Rootsweb.com, a website for genealogists, has added ‘Post-em’ links to a public-domain database, the Social Security Death Index. Upon finding a relative in the database (or someone else about whom you have information to share), one can click the ‘Add a Post-em’ link. When others pull up that record in the future, associated ‘Post-em’s are attached. (See http://ssdi.genealogy.rootsweb.com/cgi-bin/ssdi.cgi). A similar idea has been implemented for book reviews on Amazon.com: readers post their thoughts on books they have read.

A potential problem with this method is abuse through the posting of unhelpful or libellous comments. While posting will be unmoderated (the only financially feasible option), notes that are slanderous or otherwise obstructive can be reported to a removal moderator, who will arbitrate a decision about taking the comment out of the database. On Amazon.com, the mechanism for preventing milder forms of abuse is simply a question, “Was this review helpful to you?” Answers to this question are tallied to give a sense of how the community values the comment. Further, to ensure that web-crawling bots cannot automatically post advertisements, common solutions can be employed, such as required keying-in of a text code readable only by humans.

Given the importance of peer-reviewed publications to scientists' careers [2], will researchers freely provide their insights when they receive no publication credit [3]? While this is an empirical question, the vibrancy and success of other online forums [4-6] gives hope.

A distantly related strategy to our proposal can be seen in a handful of journals that have begun experiments in ‘open’ peer review [7]. Such journals post papers online before they go out for peer review, and scientists then debate, discuss, and suggest online. However, while a pre-publication review process necessarily works on a journal-by-journal basis, we propose that a post-publication commentary process should be centralized and independent of the journal of publication. That is, the commentaries on all papers should be stored in a single database, under the open, free-access, disinterested aegis of the National Library of Medicine.

We believe this simple, technologically feasible, practical idea could go a long way toward improving the practice of science.

David M. Eagleman is at the Computational Neurobiology Laboratory at The Salk Institute for Biological Studies, La Jolla, CA 92037. Alex O. Holcombe is at the Department of Psychology at the University of California, San Diego, La Jolla, CA 92093

1. Budd, J.M., Sievert, M.E. & Schultz, T.R., JAMA 280, 296-297 (1998).
2. Lawrence, P. Nature 422, 259 - 261 (2003).
3. Raban, D. & Rafaeli, S. Proc. 2003 IADIS Conf. (2003).
4. Rafaeli, S. & LaRose R. J. Communication Research 20(2): 277-297 (1993).
5. Jarvenpaa, S. L. & Staples D. S. Journal of Strategic Information Systems 9(2-3): 129-154 (2000).
6. Jones Q., & Rafaeli. S., (2000) EM - Electronic Markets, 10(4) (2000).
7. Gura, T. Nature 416, 258 - 260 (2002).

 

FEEDBACK

 

We have received a great deal of positive feedback from the community about this proposal; please continue to contact us with your ideas.

Further, this proposal has generated much online discussion. See, for example:

From Corante Tech News: "There's a neat letter in the latest issue of Nature. David Eagleman of the Salk Institute and Alex Holcombe of UCSD are proposing something for the scientific literature that every blog reader is familiar with: an online comments section. (Their examples aren't blogs, though, but rather things like Amazon's book reviews.) The comments would be accessible from the paper's record in the PubMed database...." more

Follow up in Corante from noted Internet thinker Clay Shirky: "There's a great proposal by David M. Eagleman and Alex O. Holcombe of the Salk Institute, suggesting making a link from articles in public medical databases to a comments section, essentially making every article a latent blog entry which can capture public debate and criticism...." more

Please continue to show your support for this idea by emailing us with your comments, questions, and debating points. Thank you.

 

Related ideas

HubMed, an alternative interface to PubMed

BioMedCentral allows commentary on their journal articles.

Brainstorm builds a visual map of any web-addressable items deemed to be relevant by a given user community. Users are encouraged to make notes on the correspondence of items to each other, which are also viewable from the graphical map.

Nodalpoint.org is a bioinformatics site that includes discussion of pubmed articles.

Etc.

 

The accrual of commentary around original ideas is an ancient one. Shown here is a page from the Jewish Talmud. The central column of text contains Rabbinic teachings as formulated from the 1st - 3rd century of the Common Era. That text is surrounded by commentary (composed over several generations, from the 3rd - 6th centuries), which is itself fringed by layers of further commentary and cross-referencing. Technology now makes this scheme possible for the burgeoning numbers of records in scientific publication databases.