Critiquing systematic review search strategies on PubMed

More than 1.1 million publications were indexed in PubMed in 2016, bringing the total number of PubMed records to more than 27 million. It’s no wonder that systematic reviews have become popular (currently there are more than 40,000 systematic reviews in PubMed Health alone). Systematic reviews and related methods aim to pull together all relevant studies on a defined topic and synthesize the evidence to evaluate what’s known. The approach has been used to inform clinical research and practice for decades, and its use is spreading.

As with any research, systematic reviews are only as good as their methods. A critical method here is literature searching. Some librarians and information specialists have taken to PubMed Commons to tackle issues surrounding the quality and efficacy of search strategies and their reporting. They also hope to raise awareness of librarians’ expertise in this area. We interviewed 5 librarians to learn more about their perspectives and how they’re using PubMed Commons.


Designing and reporting for reproducibility

Melissa Rethlefsen2

Melissa Rethlefsen

Melissa Rethlefsen is deputy director of the Eccles Health Sciences Library at University of Utah and section director of the Systematic Review Core, which is integrated with the Center for Clinical and Translational Science. She has been investigating the quality of reported systematic review search strategies. She and colleagues at Mayo Clinic found that systematic reviews that included librarians as co-authors were more likely to meet standards such as those recommended by the Institute of Medicine. “It really does benefit you to have an information specialist or librarian on your team,” Rethlefsen says.

“Just like any other type of research, your method should be described clearly enough that it can be reproduced. We see so many systematic reviews that are published without this really critical information, and then it’s really hard to assess their quality,” Rethlefsen notes.

A number of journals have endorsed the use of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Michelle Fiander, a systematic review librarian at the University of Utah, says, “PRISMA tells you what to say, what to report, the types of information that should be there. But they don’t tell you how.”

Mary Klem, a research and instruction librarian in the Health Sciences Library System at the University of Pittsburgh, has noticed the disconnect between statements concerning PRISMA and actual implementation. “In the article I commented on, the authors’ primary rationale for completing their review was that a prior review on the topic had not used a systematic or well-defined search strategy,” she shares. “I thought it was awesome that someone had critiqued a review like that! So I was disappointed to see that the documentation and searches in this new improved review weren’t thorough or comprehensive, and felt like I needed to note that.”


Putting expertise forward

Some librarians have used PubMed Commons because it’s visible, it’s fast, and they see potential for the impact to extend beyond a single publication.


Donna Berryman

“In a practical sense, using PubMed Commons seemed to be a better choice than writing a letter to the editor because of its immediacy and visibility,” Donna Berryman, the director of the Miner Libraries in the Institute for Innovative Education at the University of Rochester Medical Center, says. “Many, many people will find the article I commented on by doing a search in PubMed. If they look at the record for the article, my comment will be there. I’m not sure how many people would even look at a letter to the editor. In addition, there’s always a chance the letter won’t get published, and, if it does, there’s generally a long gap between when an article appears in a journal and when the letter to the editor might appear. All of those things argue against visibility. So, PubMed Commons gives my words visibility and immediacy.”

Wichor Bramer also favors the transparency and timeliness of PubMed Commons. He is a biomedical information specialist at Erasmus University Medical Center in the Netherlands, where he’s also currently working on his PhD on search methodologies for systematic reviews. He shares, “My last comment was on the details of a search strategy. Julie Glanville, who’s a famous searcher for reviews, responded to that, so you can communicate publicly with the authors.” Author responses can create a “vivid discussion that’s available for anyone to see.”


Michelle Fiander

Bramer is also finding value as an author himself. He notes that he’s used comments on his first article comparing PubMed and Google Scholar to change the way he did some things for his second article. “The comments that we get help me create better articles in the future.”

For Fiander, PubMed Commons offers an opportunity to “get my voice out there and point out things. Maybe it will end up stirring some better standards among journal editors. If you have a paper and you’re indexed in there, you can comment. It’s easier than writing a letter to the editor. I think the freedom of it is good.”


Commenting with care

But freedom doesn’t mean off the cuff for these commenters. “I tend to read my comment, be careful that I’m being accurate, that I’m not overstating or saying something that’s inaccurate,” Fiander notes.

Berryman has commented once but suspects she will comment again. “PubMed Commons strikes me as a place to have reasoned, deliberate comments. It’s not like commenting on Facebook or Twitter. So, one thing I always think about is whether I can write my comment in a way that is constructive and will add to the body of knowledge – and that takes both thought and time.”

I see it as post-publication peer review,” Bramer says. “I first create it. I don’t post it immediately. I put it away for maybe a day and look at it the next day and see different things, see if I can improve some things.”

Rethlefsen understands that commenting on PubMed, especially the first time, isn’t necessarily easy. “It’s not really a space where librarians had actively engaged before. Irreproducible search strategies were always a thing that librarians talked to each other about.” But she had concerns about what looked like, on the face of it, an excellent search strategy that she couldn’t reproduce. So she decided to go to PubMed Commons. “I worked on it, I deleted it, I re-wrote it, and I deleted it. And finally I pushed the publish button. It was intimidating because I’d never done it before. But once I got the reaction that I did [from colleagues and the librarian community], it became really clear to me that this was a more important space than I’d thought before.”


Joining in

Feeling inspired? If you have a publication indexed in PubMed, then you’re eligible to join PubMed Commons and start commenting! Learn more about getting started with PubMed Commons.

PubMed Commons stats through March 31, 2017. Members: 10962 joined, 1637 commented. Comments: 6372 comments posted to 5078 publications.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s