Alissa Gulin, The Daily Record Newswire
Earlier this month, I wrote a story about increased competition among medical researchers for grants from the National Institutes of Health at a time of declining federal funding of the Bethesda-based agency.
Many of the numbers speak for themselves, but other statistics aren’t so clear. For example, according to NIH data, 18 percent of grant applications were funded in 2012, the same percentage as in 2011. Prior to that, the success rate hovered around 20 percent to 22 percent for a few years, after dropping from above 30 percent a decade ago.
While those are not encouraging numbers, they don’t seem catastrophic. That is, they don’t seem to match the doomsday scenario described by many in the research community. But I kept hearing the same references to “less than 10 percent” of research projects being funded, including at a symposium where 200 state government, business and academic leaders discussed, among other things, the problem of fewer NIH grants and the need to protect our research institutions).
So I’m digging a little deeper. It turns out success rates aren’t the whole story. In fact, many scientists say they’re deceptive.
The success rate is calculated at the end of a fiscal year by dividing the number of grant applications that did get funded that year by the total number of peer-reviewed grant applications. That seems rational, but scientists say it doesn’t illustrate a researcher’s true probability of success.
That’s where three other measurements — scores, paylines and percentiles — come in.
When an application is submitted to one of the NIH’s 27 institutes and centers (which focus on particular research areas), it is peer-reviewed and given a score based on its merit.
The percentile is a ranking that shows the relative position of each application’s score among all scores recently assigned by that peer review group. An application ranked in the 5th percentile, for example, is considered more meritorious than 95 percent of the applications reviewed by that group. Using percentiles to compare applications helps control for differences in scoring tendencies from one peer group to another, the NIH says.
A payline is essentially a cutoff at a certain percentile. Applications scored below that cutoff don’t, or are very unlikely to, get funded. For instance, a payline at the 10th percentile means about 10 percent of applications will be funded.
Paylines vary from institute to institute. The National Institute of Allergy and Infectious Diseases, for example sets the payline for a certain type of grant in fiscal 2013 at the 8th percentile. In 2012, it was at the 10th percentile. In 2003, it was at the 22th percentile.
There are several reasons why paylines are different, and usually lower, than success rates.
Not all applications are assigned a percentile, such as those in response to an institute’s request for applications. When those applications are funded, they are calculated into the success rate but don’t increase the payline.
Also, applications that are revised and resubmitted during the same fiscal year are only counted once in the success rate calculations (to better reflect the funding of “unique” research applications, the NIH says), whereas all applications, including resubmissions, are included when the percentiles are calculated.
