Dissemination Centre Show menu

Guidance on rating research abstracts for NIHR Signals

The role of raters

Raters play a crucial role in the work of the NIHR Dissemination Centre by helping us to decide which research abstracts are important enough to turn into Signals. A wide range of health and social care professionals, patients and carers make up our pool of raters. We have a substantial number of raters, so please don’t worry if you are invited to rate very infrequently.

When an abstract is ready to be rated, our system automatically assigns the task to five raters, of whom one is a patient or member of the public. The system finds raters based on their roles, experience and areas of interest, according to the information that people give us when they register.  

When you are assigned a rating task, we will ask you to read a selected research abstract and assess its likely importance to professionals or patients with similar interests. You do not need to have a detailed technical knowledge of a clinical or research area but you do need to be familiar with, and have an interest in, the area or issue covered by the abstract. 

Receiving an abstract for rating

An e-mail will be sent to each rater with a link to the abstract and a deadline date. At this point, you can decide whether or not you are willing to rate the abstract. If you are not able to complete a rating for any reason, simply click the ‘Decline’ button. You will still be taken to the rating form as above, but at the bottom of that page you can click the ‘Sorry I’m unable to rate this’ button.

If you are willing to provide a rating for the abstract that you have received, click the ‘View Abstract’ button. You will then be taken to the full abstract to review and rate. We expect your rating task, based on a quick judgement of likely importance and interest, to take around 10-20 minutes.  You do not need to link to the full article; your rating is based on how you judge the abstract we have sent you.

When reading the abstract, you may find it useful to keep the following questions in mind. Please note, these are just a guide and you do not need to answer them specifically:

  • Is the information in this abstract new?
  • Is the evidence in this abstract important?
  • Is the information in the abstract relevant to current NHS practice?
  • Would a Signal based on this abstract be useful for people like you, who have an interest in this issue? Would you share it with colleagues?
  • Would a Signal based on this abstract be useful for the people living with the condition/issue or those who care for them?
  • Does the abstract show findings that are likely to lead to a change in practice?

After considering the above, please score the abstract between 1 and 6 (rating scale below):

6 Definitely worth turning into a Signal
5 Worth turning into a Signal
4 Probably worth turning into a Signal
3 Probably not worth turning into a Signal
2 Not worth turning into a Signal
1 Definitely not worth turning into a Signal

We also need your comments as they are really useful in helping us to decide which abstracts to take forward as Signals. Once you have entered a score and your comments, press the ‘Submit your rating’ button. Your feedback will then be saved. You will receive an automated e-mail thanking you for your assistance.

Your scores and comments will be taken into account by the Editorial Board who will decide which abstracts to take forward. You can see the Signals as they are published each week on the Discover portal.

What happens next?

The abstracts that you have rated are discussed at the Dissemination Centre’s next editorial team meeting.  Rating scores and comments are valuable parts of the decision whether or not to develop a Signal from any given abstract. Abstracts that are selected are then developed as Signals by the writing staff and edited by the Centre’s senior team. Finally the Signals are published on the ‘Discover’ portal.

Payments to public ratersPublic (non-professional) raters are offered a small honorarium payment for each abstract rated.  If you are a public rater, keep the acknowledgement e-mails sent after you have rated an abstract and use them as reference when claiming your payments. Payments are made once every three months. See our payment guidance for further information.

National Institute for Health Research, Room 132, Richmond House, 79 Whitehall, London, SW1A 2NS

© 2016 NIHR, all rights reserved