Critiquing systematic review search strategies on PubMed

More than 1.1 million publications were indexed in PubMed in 2016, bringing the total number of PubMed records to more than 27 million. It’s no wonder that systematic reviews have become popular (currently there are more than 40,000 systematic reviews in PubMed Health alone). Systematic reviews and related methods aim to pull together all relevant studies on a defined topic and synthesize the evidence to evaluate what’s known. The approach has been used to inform clinical research and practice for decades, and its use is spreading.

As with any research, systematic reviews are only as good as their methods. A critical method here is literature searching. Some librarians and information specialists have taken to PubMed Commons to tackle issues surrounding the quality and efficacy of search strategies and their reporting. They also hope to raise awareness of librarians’ expertise in this area. We interviewed 5 librarians to learn more about their perspectives and how they’re using PubMed Commons.

 

Designing and reporting for reproducibility

Melissa Rethlefsen2

Melissa Rethlefsen

Melissa Rethlefsen is deputy director of the Eccles Health Sciences Library at University of Utah and section director of the Systematic Review Core, which is integrated with the Center for Clinical and Translational Science. She has been investigating the quality of reported systematic review search strategies. She and colleagues at Mayo Clinic found that systematic reviews that included librarians as co-authors were more likely to meet standards such as those recommended by the Institute of Medicine. “It really does benefit you to have an information specialist or librarian on your team,” Rethlefsen says.

“Just like any other type of research, your method should be described clearly enough that it can be reproduced. We see so many systematic reviews that are published without this really critical information, and then it’s really hard to assess their quality,” Rethlefsen notes.

A number of journals have endorsed the use of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Michelle Fiander, a systematic review librarian at the University of Utah, says, “PRISMA tells you what to say, what to report, the types of information that should be there. But they don’t tell you how.”

Mary Klem, a research and instruction librarian in the Health Sciences Library System at the University of Pittsburgh, has noticed the disconnect between statements concerning PRISMA and actual implementation. “In the article I commented on, the authors’ primary rationale for completing their review was that a prior review on the topic had not used a systematic or well-defined search strategy,” she shares. “I thought it was awesome that someone had critiqued a review like that! So I was disappointed to see that the documentation and searches in this new improved review weren’t thorough or comprehensive, and felt like I needed to note that.”

 

Putting expertise forward

Some librarians have used PubMed Commons because it’s visible, it’s fast, and they see potential for the impact to extend beyond a single publication.

Berryman_headshot_small

Donna Berryman

“In a practical sense, using PubMed Commons seemed to be a better choice than writing a letter to the editor because of its immediacy and visibility,” Donna Berryman, the director of the Miner Libraries in the Institute for Innovative Education at the University of Rochester Medical Center, says. “Many, many people will find the article I commented on by doing a search in PubMed. If they look at the record for the article, my comment will be there. I’m not sure how many people would even look at a letter to the editor. In addition, there’s always a chance the letter won’t get published, and, if it does, there’s generally a long gap between when an article appears in a journal and when the letter to the editor might appear. All of those things argue against visibility. So, PubMed Commons gives my words visibility and immediacy.”

Wichor Bramer also favors the transparency and timeliness of PubMed Commons. He is a biomedical information specialist at Erasmus University Medical Center in the Netherlands, where he’s also currently working on his PhD on search methodologies for systematic reviews. He shares, “My last comment was on the details of a search strategy. Julie Glanville, who’s a famous searcher for reviews, responded to that, so you can communicate publicly with the authors.” Author responses can create a “vivid discussion that’s available for anyone to see.”

Fiander

Michelle Fiander

Bramer is also finding value as an author himself. He notes that he’s used comments on his first article comparing PubMed and Google Scholar to change the way he did some things for his second article. “The comments that we get help me create better articles in the future.”

For Fiander, PubMed Commons offers an opportunity to “get my voice out there and point out things. Maybe it will end up stirring some better standards among journal editors. If you have a paper and you’re indexed in there, you can comment. It’s easier than writing a letter to the editor. I think the freedom of it is good.”

 

Commenting with care

But freedom doesn’t mean off the cuff for these commenters. “I tend to read my comment, be careful that I’m being accurate, that I’m not overstating or saying something that’s inaccurate,” Fiander notes.

Berryman has commented once but suspects she will comment again. “PubMed Commons strikes me as a place to have reasoned, deliberate comments. It’s not like commenting on Facebook or Twitter. So, one thing I always think about is whether I can write my comment in a way that is constructive and will add to the body of knowledge – and that takes both thought and time.”

I see it as post-publication peer review,” Bramer says. “I first create it. I don’t post it immediately. I put it away for maybe a day and look at it the next day and see different things, see if I can improve some things.”

Rethlefsen understands that commenting on PubMed, especially the first time, isn’t necessarily easy. “It’s not really a space where librarians had actively engaged before. Irreproducible search strategies were always a thing that librarians talked to each other about.” But she had concerns about what looked like, on the face of it, an excellent search strategy that she couldn’t reproduce. So she decided to go to PubMed Commons. “I worked on it, I deleted it, I re-wrote it, and I deleted it. And finally I pushed the publish button. It was intimidating because I’d never done it before. But once I got the reaction that I did [from colleagues and the librarian community], it became really clear to me that this was a more important space than I’d thought before.”

 

Joining in

Feeling inspired? If you have a publication indexed in PubMed, then you’re eligible to join PubMed Commons and start commenting! Learn more about getting started with PubMed Commons.

PubMed Commons stats through March 31, 2017. Members: 10962 joined, 1637 commented. Comments: 6372 comments posted to 5078 publications.

Spotlight on… Amanda Capes-Davis, setting the cell line record straight

450755747You think you’ve found what you need – a cell model for a specific type of cancer you’re studying. But the label on those cells may have you fooled. In labs around the world, many cell lines are mistaken for characters they are not. Dr. Amanda Capes-Davis has seen it happen, time and again. She received her medical training and PhD in cancer genetics from the University of Sydney. After six years as a research officer at the Children’s Medical Research Institute (CMRI), she helped establish CellBank Australia, a non-profit cell line repository. Now working as a cell culture consultant and chair of the International Cell Line Authentication Committee (ICLAC), Capes-Davis is on a mission to bring the long-standing problem of cell line misidentification to light – and she’s using PubMed Commons as a tool in this fight.

Mistaken identities

Cell culture is a part of everyday life for many life science and biomedical researchers. To address biological questions, scientists often work with populations of cells grown in incubators. Typically, cells isolated from a human or other animal will only survive and multiply for a few days or weeks, even under optimal conditions. “Immortalized cell lines,” on the other hand, can be grown in flasks or dishes for months, even years. Often these lines are derived from cancers that allow cells to bypass checkpoints that would normally stop them from dividing.

The advent of immortalized cell culture 60 years ago opened doors for new studies, but there are also persistent problems. Many cell lines bear mistaken identities. So far, ICLAC has identified 472 cross-contaminated or misidentified cell lines, based on 89 publications. The cell type or tissue origin of cells grown is sometimes mislabeled at the outset. Other times, a cell line cross-contaminates another, overtaking the original line. “Perhaps 10-15% of all cell lines are cross-contaminated,” Capes-Davis reports.

HeLa cells exemplify the potential and the pitfalls of cell culture. Henrietta Lacks was a woman who unknowingly pushed biomedical science forward. While she was under treatment for aggressive cervical cancer, a sample of her tumor was taken. Shortly before her death in 1951, those cells were used to establish the first reported immortalized human cell line. Within a few years, the cells were widely used, including in development of the polio vaccine. However, HeLa cells grow so robustly that, if a few cells mistakenly get mixed in with another cell type, HeLa can quickly overtake the others. In their survey of the literature, ICLAC has noticed an abundance of HeLa contamination. Capes-Davis notes, “There are 135 different cell line contaminants, but HeLa is by far the commonest. We list 113 misidentified cell lines where HeLa is the contaminant.”

HeLa cross-contamination of cell lines was first reported in 1967. Yet many new publications continue to misidentify cross-contaminated cell lines. “It’s very understandable, I think, for a scientist,” Capes-Davis says. “You’re doing research in that field. You’re seeing everyone else using this cell line. You think this must be appropriate because all of your colleagues use it.” Reports of cross-contamination disappear under the mountain of other publications using the cell line. In doing as they were trained – building on work published by others – many scientists are actually perpetuating errors. Capes-Davis notes, “They say, research is meant to be self-correcting, but with these cell lines, that doesn’t appear to be the case.”

It’s impossible to fully assess the impact that cell line contamination has had – and continues to have – on research and development. “You only really get hints here or there,” Capes-Davis says. “I first learned about cell line contamination as a PhD student… The lab I was working in started to require testing as I was writing up my PhD.  That was quite a stressful experience for me – my PhD was entirely cell culture-based, and it was frightening to think that all my hard work might be wasted if those cell lines were contaminated.” In some cases, experimental results from misidentified cell lines supported patenting and testing of compounds in people. “These days, I think, with the level of regulation that comes in, problems with cross-contaminated cell lines will be picked up before an agent is trialed, but you have to wonder about the waste of time and waste of money required to get to that point.Amanda Capes-Davis

For Capes-Davis, though, solving cell line misidentification is about more than scientists’ time and research funding. “I am also a medical doctor and would go in to operating theatres to collect tissue samples from individuals who consented to their tissue being used in research.  Many cell lines represent a legacy from donors who have died as a result of that disease.  To me, testing of cell lines is part of our responsibility to the donor, and makes sure that we put the donor’s gift to the best use we can in finding future treatments.”

Correcting the record and the future course

In 2009, Capes-Davis joined two initiatives to address cell line misidentification. The American Type Culture Collection (ATCC) Standards Development Organization established a workgroup (ASN-0002) to develop standards for human cell line authentication. Capes-Davis recalls, “Some members wanted to have an ongoing group on cross-contamination.” During this time, she also collaborated with Ian Freshney to develop a database of cross-contaminated cell lines. “We could really see the value of having involvement of a larger group of people, the advantages of people being able to contribute data or having additional expert opinions on whether a cell line was cross-contaminated, whether or not it’s possible to find authentic stocks.” In 2012, she and other scientists  with a shared interest in the problem of cell line misidentification founded ICLAC.

“I think publicity and awareness is always a challenge. It’s always a bit of a surprise to people, even people who’ve been in the field a number of years,” Capes-Davis notes. “A lot of what we’re involved with is making people aware of the need for quality, even when they’re doing preclinical research. It’s not a series of steps that are going to be difficult to do, but it’s something that should be part of good lab practice. It’s part of good research.” ICLAC is also seeking to establish infrastructure for researchers, such as maintaining the database she and Freshney created, developing guidelines for best practices, and making resources available online.

Capes-Davis has taken to PubMed Commons to annotate publications that misreport identities of known cross-contaminated cell lines. “Ideally we want these things picked up as part of peer review, but it’s not as easy sometimes as you might think,” she notes. “Correcting the scientific record is a really important thing because even if you take the assumption that a signaling pathway isn’t going to be affected, that’s not necessarily going to be the case for another paper that cites that work. The assumption is that if you describe a cell line as oral carcinoma, that’s what it is. It might not matter for one person’s work, but it could well matter for the person who reads that paper.”

“I use PubMed pretty much everyday… it’s an area where a lot of people come to look at abstracts and hunt down papers,” Capes-Davis notes. “So [PubMed Commons] seemed like a great opportunity to raise awareness of our database… We’re hoping that if we do the best we can to bring all the known publications that relate to cross-contaminated cell lines into our database, that will be a useful resource for people. It’s not enough…but it’s certainly a step in the right direction.”

Learn more about PubMed Commons. If you’re an author in PubMed, join and make your own contribution to scientific discourse.