Peer review is an essential component of scientific learning and progress, and yet researchers who are just starting out on their career are rarely schooled in the skills needed to provide a useful and constructive review. Academic publishers have sought to fill the gap with workshops and training materials that offer guidance to newcomers on how to critique research work that has been submitted to their journals. But IOP Publishing has gone one step further by launching a free online training programme that enables early-career researchers anywhere in the world to gain the skills and confidence they need to achieve excellence in peer review.

“I believe that peer-review training is crucial for experienced and non-experienced reviewers to improve the quality of reports, and eventually, the quality of publications,” says Marijan Beg, a recent graduate from the programme and a teaching fellow in computational data science at Imperial College London, UK. “Most often, we begin peer reviewing in the early stages of our careers without any training. Our understanding of the peer-review process is based solely on the reviews we received for the manuscripts we wrote as PhD students.”

To anyone who may be jaded by the prospect of yet another online training course, the Peer Review eLearning Hub exploits the latest innovations in training software to offer participants an engaging and interactive learning experience. The interface is also fully responsive, allowing trainees to access the training platform on any device and from any part of the world. “It is easy for anyone to sign up and register for the programme,” comments Laura Feetham, reviewer engagement manager at IOP Publishing. “The training portal is free for anyone to use, and registrants gain immediate access to the learning resources.”

The impetus for the project came from an extensive survey of IOP Publishing’s existing pool of peer reviewers, conducted in 2020, as well as in-depth interviews with researchers at all stages of their career. “We know that peer review is not perfect, and we wanted to find out how we could improve it,” comments Feetham.

One of the main themes to emerge from the research was the need for more training in peer review. While that focus was particularly evident in the responses from early-career researchers, Feetham says that even senior academics still remember the stress of writing their first review. “It’s just bizarre that you might typically be invited to write a review towards the end of your PhD, and are just expected to know how to do it,” she says.

To address that need, in September 2020 IOP Publishing launched a series of online workshops under the banner “Peer review excellence: IOP training and certification”. Building on previous experience of delivering live presentations at key conferences and research institutions, Feetham developed an interactive curriculum that engages participants through discussions and practical exercises. “We only include about 50 people in each session, and nearly all of the sessions have been oversubscribed,” she comments. “It has shown us that there is a real appetite for this type of training.”

Based on that success, Feetham sought to recreate the same learning experience in a more scalable and accessible format. To deliver a similar level of interaction and engagement through a purely online portal, she worked closely with Laura Lee Gibbs, a learning consultant at Learn Fox Consultancy, who was able to translate the curriculum into interactive elements that enable participants to learn specific skills. “Laura is an expert in cognition and learning software, and she was brilliant at telling us the most engaging way to get each point across,” explains Feetham.

The comprehensive training programme includes three modules that together take around two hours to complete. The first module focuses on the fundamentals of peer review, including its evolution since it was first introduced in 1665. The second teaches participants how to write a meaningful and constructive review, with clear guidance on how to structure the report and what information to include. And the third one focuses on peer-review ethics, ensuring that new reviewers can spot and report any signs of research misconduct.

“Before this training I didn’t even know what peer review reports should include, or the most important criteria for evaluating a work,” comments Yuchun Sun of the California Institute of Technology. “I have learned that when I review a manuscript, I need to critically read the article and think about many aspects of the work.”

Each module is taught through a series of elements, which might consist of a few lines of text followed by an interactive exercise such as a flip-card game. “It’s a way of learning that isn’t just about absorbing information and regurgitating it,” explains Feetham. Beg agrees: “I found the experience of using the online training platform concise, enjoyable, and well-structured,” he says.

Trainees must engage with all the elements and pass an assessment at the end of each module to progress to the next one, with a final assessment determining whether they pass the course. Since launching the hub in March, some 900 people have registered for the course and 300 have graduated. Feetham has also been encouraged by the geographic diversity of the participants, with plenty of sign-ups from under-represented communities in Asia, South America and Africa.

But it’s not just about delivering the training. Another important theme to emerge from the initial research is the need for more recognition for peer reviewers. “In physics, at least, researchers see peer review as a way of giving something back to their community,” explains Feetham. “But they do want to be recognized for their work.”

As a result, last year IOP Publishing introduced the concept of a Trusted Reviewer. To kick-start the scheme, this special status was given to anyone who had submitted an outstanding review – as scored by the journal editors – within the past two years. Less than 15% of all reports achieve this highest possible grade, lending an air of prestige to the Trusted Reviewer title, and later this year IOP Publishing plans to provide more public recognition through an external database and badges that Trusted Reviewers can use on their professional profiles.

With the launch of the online training hub, Feetham was keen that graduates from the scheme should also have the opportunity to gain Trusted Reviewer status. Participants who have completed the training have the option of a fast-tracked process, in which the IOP Publishing team helps them to develop a searchable selection of research interests that enables them to be matched to a manuscript in their area of expertise. If the graduate submits a review that is graded as being outstanding or excellent, they are awarded Trusted Reviewer status.

“What’s really impressive is that more than half of those who opt for fast-tracking immediately become an IOP Trusted Reviewer, compared with less than 15% for our general reviewer pool,” comments Feetham. “That suggests we are doing something right. The scientific quality of their reviews is really high, and we are giving them the skills and confidence they need to critique a manuscript in a meaningful way.”

Which brings us to the third element of the ecosystem that Feetham is aiming to create. Graduates who don’t quite make the grade for Trusted Reviewer status are given clear and honest feedback on how to improve, and are then given the opportunity to try again. While that constructive criticism is currently offered on an informal basis, in future there will be an automated system that provides graduates with more structured feedback on the quality of their review and, if necessary, how to improve.

Feetham believes that this powerful combination of training, recognition and feedback will engage a new and more diverse generation of researchers in the peer-review process. “One of the reasons that it’s getting harder to source high-quality reviews in a timely way is that publishers are overburdening a relatively small group of people,” she says. “We want to find early-career researchers who have the time and energy to engage with the process, and give them the skills and expertise they need to produce good reviews.”

And, judging by the response from recent graduates from the programme, that approach appears to be working. “Through the whole training process, I learnt a lot about peer review and how to write a high-quality assessment,” says Nergui Nanding of Sun Yat-sen University in China, who has recently become an IOP Trusted Reviewer. “It will bring long-term benefits to my career, and the Trusted Reviewer status will provide me with more opportunities to review papers in high-quality journals and to build better relationships with the editors and the wider community.”