PubMed comments & their continuing conversations

bubbles_3aWe have many options for communication. We can choose platforms that fit our style, approach, and time constraints. From pop culture to current events, information and opinions are shared and discussed across multiple channels. And scientific publications are no exception.

PubMed Commons was established to enable commenting in PubMed, the largest biomedical literature database. In the past year, commenters posted to more than 1,400 publications. Of those publications, 80% have a single comment today, and 12% have comments from multiple members. The conversation carries forward in other venues.

Sometimes comments pull in discussion from other locations or spark exchanges elsewhere.Here are a few examples where social media prompted PubMed Commons posts or continued the commentary on publications.

Debating disease association

On June 3, 2016, Daniel MacArthur took to Twitter to express his skepticism of a report describing an association between a gene mutation and familial multiple sclerosis published in the journal Neuron. His critique stirred a bit of interest. A few days later, he posted a comment, co-written with Eric Minikel, to PubMed Commons. MacArthur and Minikel highlighted, “Enrichment in cases over controls is one important criterion for establishing pathogenicity of sequence variants.” The comment prompted more discussion on Twitter.

Over the following days, author Carles Vilariño-Güell responded, and MacArthur and Minikel replied. Shortly Chris Cotsapas posted a comment on behalf of the International Multiple Sclerosis Genetics Consortium. His comment summarized an attempt to validate the findings, linking to results posted in a bioRχiv preprint. With Simon Heath, Daniel Weeks noted further concerns in an August comment on the journal’s website, which he linked from PubMed Commons.

As the critiques unfolded, some readers commented on blog posts highlighting the results, (such as here and here) to point to the comments on PubMed. In September, STAT published a story reviewing the concerns that had been raised on PubMed Commons and elsewhere. In October, Neuron published letters from the International Multiple Sclerosis Genetics Consortium and Minikel and MacArthur, as well as a response from the authors. In an accompanying editorial note, the editors remark that the peer-reviewed letters offer “an important complement to other forms of commentary” including social media, PubMed Commons, and the journal’s online comments section.

Comments also sparked discussion of topics beyond the specific gene variant in question.

Self-correcting statements

In July 2016, a publication co-authored by Thomas Nichols reported on an artifact that might give rise to high false-positive rates in functional magnetic resonance imaging (fMRI) analyses. Across blogs and professional publications, there was consideration of what the findings meant for neuroscience research. But some in the community thought results were being too broadly extrapolated to all fMRI studies, not just the specific issues examined.

As the publication was discussed online, the authors recognized that some wording was being interpreted in ways they had not anticipated. So they asked to publish an erratum. That was initially rejected by the journal, since there was no change to the results or conclusions. Nichols published the note on his blog. Following an exchange on Twitter, he subsequently posted a comment on PubMed Commons to make a more circumspect significance statement.

At least one blogger updated a post to reflect the authors’ statement.

Although the journal ultimately published a correction a month later, PubMed Commons enabled authors to rapidly communicate a reframed interpretation of their work.

Replicating and reviewing search strategies

Comments can initiate discussion of specific results and interpretations. But they can also serve as a jumping off point to evaluate approaches and highlight practices.

Literature search strategies lie at the core of systematic reviews and meta-analyses. Melissa Rethlefsen posted a comment describing an attempt to replicate the search strategy reported in a meta-analysis. She noted key missing information such as date ranges. She concluded: “This study highlights the need for more accurate and comprehensive reporting needed for search strategies in systematic reviews and other literature search-based research syntheses, and the need for better peer review of search strategies by information specialists/medical librarians.”

One library used this example to encourage the use of structured reporting guidelines for systematic reviews and meta-analyses.

Another library used the comment to illustrate the importance of reviewing search strategies. And medical librarians and researchers chimed in on Twitter.

Extending the reach of scientific discourse

As you browse the web, you might just run across a mention of a comment on PubMed. Blog authors and readers might mention comments, as they have about a genetic variant associated with body mass index , ‘bad luck’ and cancer, or the occurrence of amphetamines in water systems. They might even appear in the references list, such as a roundup of publications on cancer risk or a look at psychological debriefing after traumatic events. Perhaps the most talked-about comments were those from Jennifer Doudna and Emmanuelle Charpentier on a perspective of the history of CRISPR. The comments were shared and discussed on social media. They were also mentioned in several blog posts and articles, including ones from news outlets such as the Washington Post.

PubMed Commons offers a key place to quickly anchor critical points for future readers to see­­—in one of biomedical science’s most heavily used resources. On a typical day in 2016, 1.6 million users ran 2.5 million web searches on PubMed and viewed more 8 million records.

Through October 31, 2016, PubMed Commons had 10,632 members. They’ve posted 5,739 comments to 4,595 publications. Want to join in? Check out our Get Started page for more information!

The PubMed Commons Team

Signposts from research to resources

Woman working at computerFrom repositories to blogs, the web has expanded means to share information and resources widely. Access to data and code enables other researchers to check published analyses and undertake new ones. Having another way to look at results can help people connect with them and deepen understanding. PubMed Commons members are tying these pieces back to publications by adding external links to PubMed records.

Tagging inputs 

High-throughput assays generate heaps of data, which can require custom software tools to process and analyze. Some authors are annotating current locations and updates for data and code via PubMed Commons.

Proteomics studies approach a wide range of questions about proteins and pathways, often with mass spectrometry data at the core. Author David Simpson provides the identifier and URL to access the dataset for a recent publication. Attila Csordas has also connected several proteomics articles to deposited data.

Patrick Schloss and colleagues published an approach for characterizing microbiomes using a particular high-throughput sequencing platform. He links to “a fully executable version” of his paper. The repository includes the R code, as well as raw and processed data, so that users can reproduce results in the publication.

With the end of Google Code on the horizon, researchers are moving projects to new locations. Pedro Mendes has migrated code for a tool used in modeling of biochemical networks to GitHub. He’s added a comment to point to the code’s new home.

Sometimes authors will update code and append new options. Ross Lazarus summarizes features added to a toolkit for high-throughput biology workflow software. He also includes a link to the new version.

Adding dimensions

Three-dimensional structures of biological molecules can offer useful insight into how proteins function. But as figures in papers, structures can fall flat. Some are using PubMed Commons to restore depth.

Michael Cianfrocco and colleagues solved the structure of a transcription factor complex bound to DNA. He provides a link to FigShare where users can download files for a visualization program. They can then dive into the structure and even create their own figures.

Sandra Porter links to a blog post, where she writes, “One of the most amazing things, to me at least, is how spider silk changes from a liquid form, inside the spider, to a solid, strong material that we see in their webs and other constructions.” She shows readers how to use their tablets to explore the structure and properties of a protein in spider silk that permit this change.

Mary Mangan offers a resource for a literal hands-on approach. She used data from an X-ray crystal structure to create a 3D-printable model of γ-hemolysin, a pore forming protein from Staphylococcus aureus. She points readers to the model on the NIH 3D Print Exchange.

Have something you want to add to a publication? Any author of a PubMed-indexed publication is eligible to join PubMed Commons. Learn how! And check out more examples of how PubMed Commons is being put to use.

The PubMed Commons Team

Refining & revising research on the public record

179024690The life of a research project often doesn’t end when a publication appears in a journal. Experiments continue. Errors are found. Supporting or refuting data is published. Alternative explanations emerge… So PubMed Commons members are posting updated and complementary information to keep work current.

Honing interpretations

Consider, for example, Daniel Haft’s reflection on his publication from 2009. He identified some peptides in two Bacillus strains that he referred to as putative “bacteriocins”. The term describes bacterial peptides that are toxic to related strains. Use of the term was “overconfident”, he notes, “…given the lack of evidence then that these heterocycle-containing natural products were toxins rather than, say, peptide pheromones.” But now results from another group suggest he was on the right track after all.

Andrew Sharp’s group studies how genome structure relates to human disease including X-chromosome inactivation. He considers how a new publication describing DNA methylation in development might be relevant to their own work.

Meta-analysis can show patterns and discrepancies between results from different studies. The data can be calculated in different ways. Valeria Fadda provides alternative data visualizations by posting forest plots for risk differences from published meta-analyses.

Dopamine is a central player in addiction in rodents. Many factors, including where it acts, influence the effects of this neurotransmitter. Roy Wise notes how his group’s work supports recent findings on the consequences of location and what this might mean for the field.

“The mechanisms for establishing, maintaining, and reinstatement of cocaine self-administration have been studied extensively, but have not yet led to a proven medication for cocaine addiction,” Wise remarks. “Perhaps it is time to turn attention to the endogenous mechanisms for what appears to be a state of drug satiety.”

Amending prior work

Some authors highlight work that might strengthen or refine interpretations. Others note how new or existing data might change conclusions.

With co-author Raphael Silberzahn, Eric Uhlmann documents their new work on names and career outcomes. They had previously reported, “Germans with noble-sounding surnames, such as Kaiser (“emperor”), König (“king”), and Fürst (“prince”), more frequently hold managerial positions…” Their new analysis seems to overturn their previous conclusions.

Like many researchers trying to identify new antitumor agents, Miguel Lopez-Lazaro and his colleagues began by testing compounds in a cancer cell line. Since publishing their work, they followed up in other cell lines. Now he considers whether these in vitro effects will really translate to therapeutic potential.

Teasing apart the functions of specific proteins in cells or animals can get complicated. Using inhibitors or genetic manipulation doesn’t always provide a clear picture.

For example, Jens Staal explored the interplay of the protease MALT1 and the deubiquitnase CYLD in immune cells. After further experiments, he updates his interpretation.

Meanwhile, Jim Woodgett notes that care should be taken with assigning specific functions to a single protein isoform. He highlights the trouble with glycogen synthase kinase-3 inhibitors. After a few comments on the subject, he remarks, “…no small molecule inhibitors are isoform selective – I am a scratched record.”

Bridging published updates

When substantive errors are found, updates are not always clearly connected to original publications. PubMed Commons gives authors a chance to link pieces together in PubMed.

Sometimes errors are reported in follow-up studies. Consider a mutation in an HIV protein that was correlated with survival. Stuart Ray notes that a subsequent report of an alignment error invalidates the findings.

Other times, flaws in methods or results lead to correction or retraction of a report.

Randy Blakely finds that a paper about the human dopamine transporter continues to be improperly cited, despite a correction to the results. He says, “I am hoping the PubMed Commons forum will provide a suitable opportunity to redirect readers attention to the, as yet, unknown properties of the E602G mutation…”

Ivan Oransky uses PubMed Commons to annotate recent retractions. He includes links with more about the history of the publication and why it was withdrawn. For instance, here he points to where co-blogger Adam Marcus notes, “Failure to cite leads to ignoble end for xenon paper, and a correction.”

If you’re an author of a PubMed-indexed publication and ready to get involved, check out this page to learn how you can get started.

The PubMed Commons Team


Spotlight on… Amanda Capes-Davis, setting the cell line record straight

450755747You think you’ve found what you need – a cell model for a specific type of cancer you’re studying. But the label on those cells may have you fooled. In labs around the world, many cell lines are mistaken for characters they are not. Dr. Amanda Capes-Davis has seen it happen, time and again. She received her medical training and PhD in cancer genetics from the University of Sydney. After six years as a research officer at the Children’s Medical Research Institute (CMRI), she helped establish CellBank Australia, a non-profit cell line repository. Now working as a cell culture consultant and chair of the International Cell Line Authentication Committee (ICLAC), Capes-Davis is on a mission to bring the long-standing problem of cell line misidentification to light – and she’s using PubMed Commons as a tool in this fight.

Mistaken identities

Cell culture is a part of everyday life for many life science and biomedical researchers. To address biological questions, scientists often work with populations of cells grown in incubators. Typically, cells isolated from a human or other animal will only survive and multiply for a few days or weeks, even under optimal conditions. “Immortalized cell lines,” on the other hand, can be grown in flasks or dishes for months, even years. Often these lines are derived from cancers that allow cells to bypass checkpoints that would normally stop them from dividing.

The advent of immortalized cell culture 60 years ago opened doors for new studies, but there are also persistent problems. Many cell lines bear mistaken identities. So far, ICLAC has identified 472 cross-contaminated or misidentified cell lines, based on 89 publications. The cell type or tissue origin of cells grown is sometimes mislabeled at the outset. Other times, a cell line cross-contaminates another, overtaking the original line. “Perhaps 10-15% of all cell lines are cross-contaminated,” Capes-Davis reports.

HeLa cells exemplify the potential and the pitfalls of cell culture. Henrietta Lacks was a woman who unknowingly pushed biomedical science forward. While she was under treatment for aggressive cervical cancer, a sample of her tumor was taken. Shortly before her death in 1951, those cells were used to establish the first reported immortalized human cell line. Within a few years, the cells were widely used, including in development of the polio vaccine. However, HeLa cells grow so robustly that, if a few cells mistakenly get mixed in with another cell type, HeLa can quickly overtake the others. In their survey of the literature, ICLAC has noticed an abundance of HeLa contamination. Capes-Davis notes, “There are 135 different cell line contaminants, but HeLa is by far the commonest. We list 113 misidentified cell lines where HeLa is the contaminant.”

HeLa cross-contamination of cell lines was first reported in 1967. Yet many new publications continue to misidentify cross-contaminated cell lines. “It’s very understandable, I think, for a scientist,” Capes-Davis says. “You’re doing research in that field. You’re seeing everyone else using this cell line. You think this must be appropriate because all of your colleagues use it.” Reports of cross-contamination disappear under the mountain of other publications using the cell line. In doing as they were trained – building on work published by others – many scientists are actually perpetuating errors. Capes-Davis notes, “They say, research is meant to be self-correcting, but with these cell lines, that doesn’t appear to be the case.”

It’s impossible to fully assess the impact that cell line contamination has had – and continues to have – on research and development. “You only really get hints here or there,” Capes-Davis says. “I first learned about cell line contamination as a PhD student… The lab I was working in started to require testing as I was writing up my PhD.  That was quite a stressful experience for me – my PhD was entirely cell culture-based, and it was frightening to think that all my hard work might be wasted if those cell lines were contaminated.” In some cases, experimental results from misidentified cell lines supported patenting and testing of compounds in people. “These days, I think, with the level of regulation that comes in, problems with cross-contaminated cell lines will be picked up before an agent is trialed, but you have to wonder about the waste of time and waste of money required to get to that point.Amanda Capes-Davis

For Capes-Davis, though, solving cell line misidentification is about more than scientists’ time and research funding. “I am also a medical doctor and would go in to operating theatres to collect tissue samples from individuals who consented to their tissue being used in research.  Many cell lines represent a legacy from donors who have died as a result of that disease.  To me, testing of cell lines is part of our responsibility to the donor, and makes sure that we put the donor’s gift to the best use we can in finding future treatments.”

Correcting the record and the future course

In 2009, Capes-Davis joined two initiatives to address cell line misidentification. The American Type Culture Collection (ATCC) Standards Development Organization established a workgroup (ASN-0002) to develop standards for human cell line authentication. Capes-Davis recalls, “Some members wanted to have an ongoing group on cross-contamination.” During this time, she also collaborated with Ian Freshney to develop a database of cross-contaminated cell lines. “We could really see the value of having involvement of a larger group of people, the advantages of people being able to contribute data or having additional expert opinions on whether a cell line was cross-contaminated, whether or not it’s possible to find authentic stocks.” In 2012, she and other scientists  with a shared interest in the problem of cell line misidentification founded ICLAC.

“I think publicity and awareness is always a challenge. It’s always a bit of a surprise to people, even people who’ve been in the field a number of years,” Capes-Davis notes. “A lot of what we’re involved with is making people aware of the need for quality, even when they’re doing preclinical research. It’s not a series of steps that are going to be difficult to do, but it’s something that should be part of good lab practice. It’s part of good research.” ICLAC is also seeking to establish infrastructure for researchers, such as maintaining the database she and Freshney created, developing guidelines for best practices, and making resources available online.

Capes-Davis has taken to PubMed Commons to annotate publications that misreport identities of known cross-contaminated cell lines. “Ideally we want these things picked up as part of peer review, but it’s not as easy sometimes as you might think,” she notes. “Correcting the scientific record is a really important thing because even if you take the assumption that a signaling pathway isn’t going to be affected, that’s not necessarily going to be the case for another paper that cites that work. The assumption is that if you describe a cell line as oral carcinoma, that’s what it is. It might not matter for one person’s work, but it could well matter for the person who reads that paper.”

“I use PubMed pretty much everyday… it’s an area where a lot of people come to look at abstracts and hunt down papers,” Capes-Davis notes. “So [PubMed Commons] seemed like a great opportunity to raise awareness of our database… We’re hoping that if we do the best we can to bring all the known publications that relate to cross-contaminated cell lines into our database, that will be a useful resource for people. It’s not enough…but it’s certainly a step in the right direction.”

Learn more about PubMed Commons. If you’re an author in PubMed, join and make your own contribution to scientific discourse.



Blogs and their links with PubMed Commons

Photo of blog entry in a dictionary

Blogs grew quickly into a dynamic and substantial element in the way we discuss, criticize and share information about scientific publications. Science blogs are one of the filters many now use for the science literature – and one of the ways members of the public learn about research results and efforts.

Anchoring these blog posts to the publications they discuss is one of the ways members are putting PubMed Commons to good use. Here are examples from the spectrum covered by dozens of links to blog posts so far in the Commons.

Dorothy Bishop responded to Michael Farthing’s lecture on the challenge of research conduct, including links to her blog posts on perverse incentives for academics and registering proposed research in psychology.

Daniel Simons writes post-publication peer reviews on his blog. He’s posted one on Joaquin Anguera and colleagues’ study of video game training for cognitive control,  and one on a recent study of inattentional blindness to African Americans by Jazmin Brown-Iannuzzi at the Commons.

Coming across a media report of a study draws the attention of many academic bloggers, spurring both posts on their blogs and links to them on PubMed Commons. This happened when Andrew Kniss saw a newspaper article on Carsten Bruehl’s conclusions about pesticide use and frog population decline.

In other areas where there’s a lot of debate about a publication, or they read a post they found helpful, Commons members draw readers’ attention to detailed coverage in another person’s blog. George McNamara, for example, points to a post by Michael Eisen at the PubMed record for John Bohannon’s publication about peer review at open-access journals.

Important blog posts also unwrap the history of misleading use of studies in public debates. An example now in the Commons is about an autism prevalence study by Simon Baron-Cohen and colleagues. It was added by Jamie Horder, pointing to Ben Goldacre’s post on “The story behind this paper, and the remarkable media coverage it received.” It’s a Whodunit? about an MMR vaccine scare in the media.

Retractions of articles are being noted with links to blog posts about the background, for example by Allison Stelling and Ivan Oransky.

Attacks on research lead defenders to their blogs, too – and now to the Commons afterwards. James Coyne responded to critics of a randomized trial by William Hollingworth and colleagues of screening for distress in cancer patients.

Authors of publications who discuss their work on their own blogs draw readers’ attention to these posts at PubMed Commons. For example, Anne-Marie Cunningham has posted links to blog posts on her work with colleagues incorporating a pre-publication version with one article and an audio-visual presentation with another.

Seth Borderstein links to a post explaining the background to a publication on maternal microbe transmission. And Jonathan Eisen links to his post telling the story behind his team’s work on phylogenetic trees.

Discussing implications for future research of a study’s findings are another part of blog posts brought into the Commons. Graham Coop posted a link to his blog post on Sriram Sankararaman and colleagues’ work on genetic variants on the possible offspring of Neandertal and modern human parents.

The Commons is growing: from 2,000 members in January to over 3,000 today, who have made just over 1,000 comments so far. Want to keep up with comments that are getting good ranking scores in PubMed Commons? We tweet them @PubMedCommons.

That’s one more important reason for PubMed Commons members to add their “helpful? yes or no” clicks to the system: when you read it, please rate it. Help shape the Commons into a valuable place for discussions about biomedical publications – and drawing together the valuable discussions from outside. If you write or read a blog with a valuable discussion about a publication, think about sharing it at PubMed Commons.

The PubMed Commons team

More information

Membership of PubMed Commons is open to all authors in PubMed.

Join PubMed Commons

“You found this helpful:” What people like in comments on PubMed

Comment search and alert: A PubMed Commons guide


“You found this helpful:” What people like in comments on PubMed

Commented rated "You found this helpful"

Rating is a critical function for a commenting system. It’s a key way a forum’s members can encourage the kind of discussion they want to see – and discourage what they don’t find helpful.

Increasingly, ratings are influencing PubMed Commons, the new commenting system in PubMed. There’s a selected comment stream on the PubMed Commons landing page. We’ve also started sending out tweets when new comments start ranking well on several factors.

The rating options are only available for comments – and you can’t rate your own comment. PubMed authors who have joined PubMed Commons can click on “yes” or “no” to the question, “Was this helpful?”

Although it’s early days, ratings are already sending some messages. One of those is what people don’t find helpful: the kind of comment that praises an article without saying why – or that’s only communicating something that is already clear in the abstract.

The message seems to be: use the comment function when you have something specific to say about, or add to, the publication. Making a similar comment repeatedly also tends to be unpopular, as is self-promotion.

Which comment has been the most popular so far? It’s a comment by Gonzalo Otazu in December. Otazu calls into question the validity of statistics in a paper that got a lot of media attention: a study suggesting that mice can inherit fear of smells associated with traumatization of their parents.

As well as good critiques, people have rated discussions positively – especially when the publication’s authors respond or encourage discussion about their paper. Some recent examples:

Integrating outside discussions in journal clubs and blog posts with PubMed Commons comments gets positive ratings, even when the authors don’t participate.

You may be able to invite yourself with our “Get started” wizard. If your email isn’t there, one of your colleagues should be able to join and then invite you. Any member of PubMed Commons can invite others – including an author of a paper that’s been commented on.

You can check out the selected comments at the PubMed Commons landing page – one click away from PubMed’s home page – or follow us on Twitter. And you can find out how to set up alerts on topics, articles or authors you’re interested in via our “Comment search and alert” guide.

From now on, the quality of PubMed Commons will rely on both the comments and the ratings. You need to be a member of PubMed Commons to rate articles. If you’re an author of a publication in PubMed, you’re eligible.

So join today – to discuss the biomedical literature or, just as importantly, to rate the comments.

The PubMed Commons team

More information

Meet PubMed Commons: The new comments forum in PubMed

Join PubMed Commons

Comment search and alert: A PubMed Commons guide


Meet PubMed Commons: The new comments forum in PubMed

Image of comments icon

If you are one of the millions of people who visit PubMed today, be on the look-out for something different. On each abstract page, there’s now a section called PubMed Commons. It’s a forum for scientific discussion on publications open to any authors in the world’s largest biomedical literature database.

Several hundred comments have been made during a closed pilot in the last few months. But there are over 23 million articles in PubMed, with thousands more pouring in every day, from Tuesday to Saturday. So the chance of coming across an article with comments is still very low.

We’ll show you some interesting ones shortly, though – and you can learn how to look for articles with comments and set up alerts in one of last week’s blog posts. Or you can check out the stream of selected new comments – as well as articles that are trending in PubMed – at the PubMed Commons home page.

If you happen onto an article that has comments, the first sign will be in your search results. There will be a little icon letting you know an article has comments, and how many there are – like this:

Comment icon with number 7

Anyone can read the comments. Members may also have rated their helpfulness, which looks like this on the comment:

8 out of 8 found this helpful

Each comment comes with a “Permalink.” That’s a permanent link you can use to share the comment or cite it. You can find more about how to cite a comment in the PubMed Commons FAQs.

If a comment has cited another article in PubMed, and used particular PubMed Commons formatting to do it, you will see a note letting you know it is mentioned in a PubMed comment.

If you are the author of any publication in PubMed, you are eligible to join PubMed Commons now. PubMed Commons is a pilot, beginning an evaluation phase. Through that time, we’ll be exploring possible ways to widen participation.

Registered members of the Commons can invite others to join. We have developed a wizard to help you and your colleagues work out who can get the ball rolling. Check out the new instructions and get started today.

Then you could rate whether a comment was helpful or not, join in a discussion, or make a new comment on an article. For members of PubMed Commons who are also signed into their NCBI account, the PubMed Commons section looks like this:

Screenshot of member view of PubMed Commons showing name and commenting box

Members have already used PubMed Commons in many ways:

There are more examples in our recent post on how authors are using PubMed Commons. We will be blogging in detail about other ways people are using the Commons, as well as talking about new features under development and offering tips about making comments and using the system.

We’re looking forward to our first Twitter chat soon, too, so keep your eye out on @PubMedCommons and this blog for the announcement.

If you’re an author we hope you will join and contribute by rating comments – and adding your thoughts about scientific publications, too. Thanks again to the hundreds of people who contributed to the development of PubMed Commons this year, and the discussion about it. And welcome to everyone who is new to the Commons: we hope you will visit the home page and find out more!

The PubMed Commons team

More information:

How to join PubMed Commons

PubMed Commons FAQs

PubMed Commons Guidelines (Updated version on 19 December 2013)