Open Evaluation: about the future of scientific publications
A scientific publication system needs to provide two basic services: access and evaluation.
The traditional publication system restricts the access to papers by requiring payment, and it restricts the evaluation of papers by relying on just 2-4 pre-publication peer reviews and by keeping the reviews secret. As a result, the current system suffers from a lack of quality and transparency of the peer-review evaluation process, and the only immediately available indication of a new paper’s quality is the prestige of the journal it appeared in states Nikolaus Kriegeskorte, a brain scientist from the Univeristy of Cambridge.
What are the options?
While open access (OA) is becoming a reality, Nikolaus Kriegeskorte promotes the establishment of open evaluations (OE). Evaluation steers the attention of the scientific community and thus the very course of science. It also influences the use of scientific findings in public policy.
An open evaluation (OE) system is pursued, in which papers are evaluated post-publication in an ongoing fashion by means of open peer review and rating.
Through signed ratings and reviews, scientists steer the attention of their field and build their reputation. Reviewers are motivated to be objective, because low-quality or self-serving signed evaluations will negatively impact their reputation. A core feature of this proposal is a division of powers between the accumulation of evaluative evidence and the analysis of this evidence by paper evaluation functions (PEFs). PEFs can be freely defined by individuals or groups (e.g., scientific societies) and provide a plurality of perspectives on the scientific literature. Simple PEFs will use averages of ratings, weighting reviewers (e.g., by H-index), and rating scales (e.g., by relevance to a decision process) in different ways. Complex PEFs will use advanced statistical techniques to infer the quality of a paper. Papers with initially promising ratings will be more deeply evaluated. The continual refinement of PEFs in response to attempts by individuals to influence evaluations in their own favor will make the system ungameable. OA and OE together have the power to revolutionize scientific publishing and usher in a new culture of transparency, constructive criticism, and collaboration.
In the area of pharmaceutical publishing the open access journal Results in Pharma Sciences (Elsevier) is going a first step having established PeerChoice. A select group of reviewers choose actively the manuscripts they want to review from the abstracts of submitted manuscripts. However it is not open to everyone and ,as with traditional systems, the information is kept secure throughout the peer review process.
If You want to learn more about open evaluations visit the following Blog: “the future of scientific publishing” (http://futureofscipub.wordpress.com/).
