Altmetrics – A Collated Adjunct Beyond Citations for Scholarly Impact: A Systematic Review

The present day medicine is based on contemporary best evidence which is used in conscientious, clear and sensible manner for making judgement regarding the individual health problems. Scholars may conventionally have obtained pertinent articles by looking into journals, being present at meetings and communicating with peers but in the epoch of digital sources online browsing or hunting keyword are mainly used. Scholars require design to help them to pinpoint the most related articles amid the large sets while using digital libraries. Digital libraries typically provide an alternative for finding the results by date or to limit the results to a certain year [1].

The elemental roles in medicine were writing and publishing. There is no existence of science until the article is published. It is very essential that scientific publications should be continued to keep up with the advances of the professional news. The Surgeon General’s Office library of the US army has published medical scientific journal articles from 1879 to 2004 and has maintained a complete bibliographic index known as Index Medicus which later developed into the US National Library of Medicine (NLM). MEDLARS, a bibliographic database which became MEDLINE was created by NLM in 1960s [2]. Scholarly publishing is infiltrating into an era where the paper journal is slowly becoming out dated, and contemporary publication types have transpired from open science communities on the internet [3]. The exposure of new internet-based-technologies unfolds new outlook to assess the effect of research. Now a days scholars have moved their regular work to the web; biomedical researchers, healthcare professionals and patients are applying social media and new scholarly e-tools in a broader way to facilitate and improve their knowledge and communication [4,5]. When a large-scale analysis was done which included the complete range of medical disciplines, it was revealed that twitter coverage has grown substantially over time in the biomedical literature [6]. The rising of new online scholarly tools has created a new metrics for the scholarly publications impact, particularly for the public. To this end, Jason Priem in 2010 introduced ‘altmetrics’ [7]. Despite the fact that altmetrics is a new term, inquisitiveness in altmetrics is increasing rapidly in comparison with bibliometrics [8].

The Altmetric score reveals the instantaneous attentiveness of a research paper as indicated by articles in news outlets, commentary on blogs, number of tweets and other digital communications and is manifested as a number within a colored wreath at the finish of each full text article online. The Impact factor expresses the citations by other scientific journal for all the articles in a journal together. In contrast, the Almetrics score yields prompt response about an individual article and reflects the public (as well as scientific) interest in the research findings. It is also helpful in knowing the impact of the research out in today’s increasingly online world by being precisely represented and elucidated as well as getting to the correct people at the proper time thus having a immense influence [8].

Altmetrics is computed by numerous websites and publishers like Impact Story, Plum Analytic, Altmetric, Elsevier, Wiley, BioMed Central, Nature Publishing Group, Public Library Of Science (PLOS) and Frontiers [6,8].

Altmetrics is a young discipline of metric which is based on the social web for interpreting scholarship. Altmetrics act as supplement to the citation based filters on which we are dependant for the past 50 years and making an effort to conquer upon some of their flaws: citations are slow to assemble, and frequently lack new forms of scholarly content such as datasets, software, and research blogs whereas altmetrics are fast and data appears in days or weeks [9,10].

Altmetric analysis include data sources involving policy documents, news (available via: http://www.altmetric.com/sources-news.php), more than 8000 academic and non academic blogs, reference managers available online, comprising of Mendeley and CiteULike, online journal clubs which include peer-review forums after publication, comprising of PubPeer and Publons, social media, involving twitter which include public comments and retweets only, facebook having public posts only, weibo, google+, pinterest and reddit having original posts only, further online references like wikipedia, reviews on YouTube [11].

The purpose of the method is to give an alternative, multidimensional as well as broader view on the impact of research [12]. This broader aspect comprises different sources of impact, as described above, but also different objects of estimation, such as data sets or software [13].

Traditional systems to assess the impact of a publication are based on the track of citations to a journal. The SCImago Journal and Country Rank has devised the Scimago Journal Rank (SJR) an indicator for journals included in the Scopus database. Journal’s impact factor, as defined and calculated by Thomson’s Journal Citation Reports (JCR; thomsonreuters.com/journalcitation-reports), is the mean number of citations for all the articles published in a particular journal in the preceding two years and is deduced to indicate the quality of the research published by that journal in subsequent years [14,15]. Both indicators are based on citations, but their calculation differs. But the difficulty with the application of the impact factor about making the opinion regarding research impact is that it takes a lot of time for the citations of an article to appear with the occurrence of most number of citations per year after about 3-7 years of publication [16]. Another conventional method was the Eigenfactor Score and the Article Influence Score which have been developed by the Eigenfactor Project as substitute (and openly available) measures to the JCR impact factors [17]. Although based on JCR data, the Article Influence score for a journal is a normalised measure of the average influence of each of its articles over the first five years after publication. Due to the limited efficacy of the impact factor, researcher-centric metrics have made an effort to provide more direct measures of the quality of an individual researcher’s output. The h-index was developed to measure both the productivity and impact of an individual, based on the number of articles that they have published and the number of citations those publications have received [18]. It is still unpredictable whether the h-index is superior or inferior to the journal impact factors for forecasting a researcher’s future productivity [19].

Altmetrics measures the impact of a research product by investigating the number of times that it is gazed at, downloaded, put aside for reading in future, debated by the scientific community, and suggested to others for reading. For regulating the online usage, some journals may cling to standards such as COUNTER (www.projectcounter.org) when evaluating the number of views and downloads in order to rule out the possibility of ‘gaming’ activity due to robots. Additional altmetrics encompasses the number of times an article has been shared via e-mail or social media platforms such as Twitter, mentioned in blogs or added to citation management tools such as CiteULike (www.citeulike.org) or Mendeley (www.mendeley.com) which have been implemented by many publishers and can be issued by companies such as Altmetric (www.altmetric.com) or generated in-house. The foremost benefit of altmetrics is that they begin to collect data regarding an article as soon as it is published, and are independent of the altmetrics being induced for any other article published in the same journal at the same time. Despite that, the utilization of article-level altmetrics for scholarly assessment and authorization is not widespread [19].

Conventional techniques, such as the impact factor of the journal that a researcher publishes in, may not be proper or precise means of estimating the overall productivity of an individual. The emergence of altmetrics gives the prospect for a fuller evaluation of a researcher’s productivity based on both their traditional and non traditional scholarly outputs. New aids should make it easier to embrace non-traditional outputs such as data, software and contributions to peer review in the estimation of early- and mid-career researchers. There have been no reviews conducted based on medical and dental literature to correlate the altmetrics with the traditional method of citations. Hence, the present study was conducted with the aim to assess the correlation of conventional method of the citation with the altmetrics based on medical literature.

Materials and Methods

Research Question

Is there any correlation between conventional citation and altmetrics related to research output in the medical field?

Eligibility Criteria

The review was conducted in November, 2016 The review included those articles which were published in English from the year 2010 to 2016. The primary search terms used were altmetrics, medical and dental or in various combinations.

Inclusion Criteria

Full text original research articles.

The articles emphasising on the correlation of altmetrics with the traditional citation.

Exclusion Criteria

Review articles, case series, case reports, letters to the editor, unpublished articles in the press.

Search Method for Identification of Studies

For the recognition of the studies to be included in this review, we devised the search strategy for each database. The search strategy used a combination of controlled vocabulary and free text terms. The main database including PubMed, PubMed Central, Cochrane Review, Embase and Google Scholar were searched for the articles from the year 2010 to 2016.

Other Sources

The search also included the hand search of the journals fulfilling the inclusion criterion for the review. No article was acquired through hand search as well as through e-mails.

Search Criteria

A literature review was performed using MeSH terms altmetrics, medical and dental. A total of 78 articles appeared in which seven titles/abstracts fulfilled the inclusion criteria for the review. The search criteria was represented in the form of Figures and tables for the current review [Table/Fig-1].

[Table/Fig-1]:

Search strategy

Reasons for Exclusion of Articles

A total of 71 articles were excluded. Out of 71 articles, 43 were review articles, the review objectives were not met for 25 articles, one was message to the editor and two were executive summary.

Results

The results of the studies have been summarised in [Table/Fig-2] [1,2,20-24].

[Table/Fig-2]:

Summary of the studies.

Sl No.	Author	Objective	Methodology	Results	Conclusion
1.	Thelwall M et al., [1]	To compare eleven different altmetric sources with citation data for 182 to 135,331(depending on the metric) PubMed documents published between 2010 and 2012. Specifically, this study seeks to answer the following research question: To what extent do the altmetric indicators associate with citation counts?	This study compares 11 altmetrics with Web of Science citations for 76 to 208,739 PubMed articles with at least one altmetric mention in each case and upto 1,891 journals per metric. It also introduces a simple sign test to overcome biases caused by different citation and usage windows.	Statistically significant associations were found between higher metric scores and higher citations for articles with positive altmetric scores in all cases with sufficient evidence (Twitter, Facebook wall posts, research highlights, blogs, mainstream media and forums) except perhaps for Google+ posts (p<0.05).	The results provide strong evidence that six of the eleven altmetrics (tweets, Facebook wall posts, research highlights, blog mentions, main stream media mentions and forum posts) associate with citation counts, at least in medical and biological sciences and for articles with at least one altmetric mention.
2.	Scarlat MM et al., [2]	To compare the traditional tools of calculation for a journal’s efficacy and visibility with the new tools that have arrived from the Internet, social media and search engines. The examples concern publications of orthopaedic surgery and in particular International Orthopaedics.	The prestige of publications, authors or journals was evaluated by the number of citations using the traditional citation metrics, most commonly the impact factor. Over the last few years, scientific medical literature has developed exponentially. The Internet has dramatically changed the way of sharing and the speed of flow of medical information. New tools have allowed readers from all over the world to access information and record their experience. Web platforms such as Facebook® and Twitter® have allowed for inputs from the general public. Professional sites such as Linkedln® and more specialised sites such as ResearchGate®, BioMed Central® and OrthoEvidence® have provided specific information on defined fields of science.	The number of papers published in different orthopaedic journals may vary; the strongest performance is however dependent on a critical volume. Strong journals have a high volume of papers and a high volume of citations. With alternative metrics and media, this reality may probably remain because high-volume publications also have high-volume social media impact	Social media may provide alternative metrics for a journal’s impact. An increasing number of researchers view articles and visit journals from social media, search engines and social platforms. That is why social media should be used by journals.
3.	Barblc D et al., [20]	To describe the traditional metrics and Altmetric scores of the 50 most frequently cited articles published in emergency medicine (EM) journals.	A structured search of the Institute for Scientific Information Web of Science version of the Science Citation Index Expanded was conducted. The 200 most frequently cited articles in the top 10 EM journals (2011 Journal Citation Report) were identified. The 200 most frequently cited articles from the rest of the medical literature, matching a predefined list of keywords relevant to the specialty of EM, were identified. Two authors reviewed the lists of citations for relevance to EM and a consensus approach was used to arrive at the final lists of the top 50 cited articles. The Altmetric scores for the top 50 cited articles in EM and other journals were determined. Descriptive statistics and Spearman correlation were performed.	The highest Altmetric score for EM articles was 25.0; the mean (±SD) was 1.9 (±5.0). The EM journal with the highest mean article Altmetric score was Resuscitation. The highest Altmetric score for other journals was 176.0 (mean ± SD = 23.3 ± 40.8). The other journal with the highest mean article Altmetric score was the New England Journal of Medicine. The main clinical areas shared for articles were critical care sepsis; cardiology and infectious diseases Spearman correlation demonstrated weakly positive correlation between citation counts and Altmetric scores for EM articles and other journals.	There is a mild correlation between citation counts and Altmetric scores for the top papers in EM and other biomedical journals (p<0.05) Future research to explore this relationship and its temporal trends will benefit the understanding of the reach and dissemination of EM research within the scientific community and society in general
4.	Thoma B et al., [21]	To develop the Social Media index (SMi), to assess their impact or quality and correlation with journal impact metrics	Data from social media platforms (Google PageRanks, Alexa ranks, Facebook likes, Twitter followers, and Google+ followers) for Emergency Medicine and Critical Care (EMCC) blogs and podcasts to derive three normalised (ordinal, logarithmic, and raw) formulas. The most statistically robust formula was assessed for 1) temporal stability using repeated measures and website age, and 2) correlation with impact by applying it to EMCC journals and measuring the correlation with known journal impact metrics	The logarithmic version of the SMi containing four metrics is Alexa, Page Rank, Twitter and Facebook. The strongest correlations were seen with the Immediacy Index (r=0.609; p<0.001) and Article Influence Score (r=0.608; p<0.001). Five-year Journal Impact Factor (r=0.526, p-value=0.001), Journal Impact Factor (r=0.526, p-value=0.003), and the Eigenfactor score (r=0.425, p-value=0.02) correlated less strongly.	The SMi has the potential to be a stable and accessible indicator of their impact. If the results of this study can be replicated, it would benefit medical professionals by identifying resources for learners and assessing the scholarly impact of educators that are using these media.
5.	Baatoli Z et al., [22]	To assess the scientific output of scholars at Kashan University of Medical Sciences by the end of March 2014 based on scientometric measures of Scopus, ResearchGate, and Mendeley.	A survey method was used to study the articles published in Scopus journals by scholars at Kashan University of Medical Sciences by the end of March 2014. The required data were collected from Scopus, ResearchGate, and Mendeley. The data were analysed with descriptive statistics. Also, the Spearman correlation was used between the number of views of articles in ResearchGate with citation number of the articles in Scopus and reading frequency of the articles in Mendeley with citation number in Scopus were examined using the Spearman correlation in SPSS 16.	Five-hundred and thirty-three articles were indexed in the Scopus Citation Database by the end of March 2014. Collectively, those articles were cited 1,315 times. The articles were covered by ResearchGate (74%) more than Mendeley (44%). In addition, 98% of the articles indexed in ResearchGate and 92% of the articles indexed in Mendeley were viewed at least once. The results showed that there was a positive correlation between the number of views of the articles in ResearchGate and Mendeley and the number of citations of the articles in Scopus (r=0.310 and r=0.247 respectively (p<0.01).	Coverage and the number of visitors were higher in ResearchGate than in Mendeley. The increase in the number of views of articles in ResearchGate and Mendeley also increased the number of citations of the papers. However, the correlation between the numbers of views of articles in ResearchGate was associated with higher citations of reading frequency of the articles in Mendeley with the number of citations to the articles
6.	De Gregori MD et al., [23]	To investigate the impact of scientific publications of the Italian SIMPAR (Study in Multidisciplinary Pain Research) group by using altmetrics, defined as nontraditional metrics constituting an alternative to more traditional citation-impact metrics, such as impact factor and h-index.	For all the 12 members of the group analysed (pain therapists, biologists, and pharmacologists), Open researcher and Contributor ID and Impact story accounts were created, and synchronized these data. Manually, we calculated the level metrics for each article by dividing the data obtained from the research community by those obtained from the public community. We analysed 759 articles, 18 of which were published by the SIMPAR Group.	It was found that the alternative metrics were generally correlated low for facebook post, impact story views and tweets, (p>0.05) with the exception of those for Mendeley readers (r=0.47, P<0.0001)	We found significant correlations between the SIMPAR Group collective publications and their impact on the indicator linked to research activity (Mendeley readers), although not to public discussion (such as Facebook and tweets).
7.	Allen HG et al., [24]	To quantify the impact of social media release on views and downloads of articles in the clinical pain sciences.	Sixteen PLOS ONE articles were blogged and released via Facebook, Twitter, LinkedIn and ResearchBlogging. org on one of two randomly selected dates. The other date served as a control. The primary outcomes were the rate of HTML views and PDF downloads of the article, over a seven-day period. Scopus citation count was taken almost nine months after the completion of the experimental period, and 1-2 years after the publication date of the target articles, as a conventional measure of impact.	The critical result was an increase in both outcome variables in the week after the blog post and social media release. The final result, that citation count did not relate to any social media measures, casts doubt over the intuitively sensible idea that social media impact reflects future citation-related impact. Citations at 03/09/2012 related to Total PDF downloads (Pearson r = 0.51; p= 0.045)but not to Total HTML views (Pearson r = 0.06; p = 0.826).	The size of the effect is not related to conventional social media metrics, such as reach, engagement and virality. Our results highlight the difference between social media reach and social media impact and suggest that the latter is not a simple function of the former.

Discussion

In this paper, we have carried out a broad investigation about the existence of altmetric data across medical fields and correlations with traditional citations. Batooli Z et al., in 2016 found that the correlation coefficient between the two variables of the perspective number of the articles in ResearchGate with citations to the article in Scopus and the correlation coefficient between two variables of scrutinizing the articles in Mendeley with the number of citations to articles in Scopus were positive and significant. It can be established that rise in a number of viewed articles in ResearchGate and Mendeley increased the number of citations to articles. A positive correlation indicates that the two are not entirely different from each other and are familiar with each other [22].

De Gregori M et al., in 2016 found that the correlation between Scopus citations and Mendeley readers was high with the exception of those for facebook posts, impact story views and tweets [23].

Thoma B et al., correlated the two elements of the social media index: Twitter followers and Facebook likes with traditional journal, impact factors was low in comparison to the correlation with Alexa rank and Google PageRank [21]. Barbic D et al., found that the correlation between citation counts and Altmetric scores for the articles in emergency medicine and other biomedical journals was mild [20]. Allen HG et al., observed that the citation count did not recount to any social media measures, project uncertainty over the intuitively rational idea that social media is related to future citation-related impact [24]. Hence, the apparent popularity of an article on social media does not essentially forsee its short-term citation count. To our understanding, an important setback in dentistry is the slow acknowledgment of modern technologies by dental scholars and practitioners. Reluctance to the impact of these documents in social media, news outlets, scientific blogs, policy documents, post publication peer review resources etc., appears irrational. On the other hand, according to Kolahi J et al., among Altmetric top 50 dental articles, Twitter (67.88%) was much more trendy than

Facebook (2.69%) [11]. However, in spite of the importance of this new theory, we could not come across any post publication peer reviews among Altmetric’s top 50 dental articles. Altmetrics is considered appendage to bibliometrics, but not a substitute. In view of this point, altmetrics only assess online interests surrounding journal articles and not assessing scientific eminence.

We consider that clinical practitioners, research scientists, and journal editors must give additional consideration to altmetrics as a fresh diverse and speedy means to assess the researcher social impact.

Challenges

Altmetrics come across several challenges as they put forward prospects. These take account of the heterogeneity, data quality issues and specific dependencies of altmetrics.

Heterogeneity

The main prospect suggested by altmetrics—their diversity or heterogeneity is also characterised as one of their key challenges.

Altmetrics involve many different kind of metrics, which has increased difficulty in ascertaining a precise meaning of what they signify. The fact that they have been measured as an integrated, monolithic substitute to certification has held up discussions, descriptions, and understanding of what they actually assess: the miscellany as a state on Twitter, an expert recommendation on F1000Prime, a reader count on Mendeley, a like on Facebook, a citation in a blog post and the reuse of a dataset does not allocate a common definition. The challenges associated with their heterogeneity and lack of sense are argued by addressing the nonexistence of a common definition, the diversity of social media acts, users and their enthusiasm, as well as the lack of a theoretical framework or abstract.

Data Quality

In altmetrics, data quality is a major challenge and surpasses the known inaccuracies and biases for citation facts. While bibliometrics sources are static documents, most information source in the framework of altmetrics are dynamic, which can be changed or deleted wholly. Precision, consistency and reproducibility can be recognized as the key issues of altmetrics data quality. It may not be out of place to mention that many providers (e.g., Twitter, Facebook, Reddit) are not focused on academia and altmetrics data quality is thus not their precedence.

Dependancy

The dynamic nature of most of the procedures of altmetrics faces a particular challenge in their precision, consistency and reproducibility. Ensuring high data quality and its pro longevity is further hindered by the strong reliance on single data providers and aggregators. Above all, the bulk of data is in the hands of for-profit companies, which restricts the transparency that has provoked the suggestion of altmetrics and it needs to be alleviated by impeding misuse and avoiding too much importance on one indicator [25].

Limitation

In this paper, we have performed a broad analysis of the presence of altmetric data across medical fields and emphasis was their correlations with citations. Altmetric data has diversification among different fields and in social media. Altmetrics is arriving nearer to specify quality when compared to the citations as they currently do. It is basically judged as an attention measure rather measuring quality. Altmetrics act as an adjunct to citations, not substituting them. Citations are still considered to grasp a position of high value in academics.

Recommendations

Altmetrics is still a relatively young field, and research is still needed into the motivations for others to use and discuss the research work online. Institutions should start to use altmetrics with better assurance in their progress and data collection in order to make altmetrics more accessible than earlier. Authors are required to maintain their existence on author profile platforms, able to use current tactics to augment their presence and competently follow research output and activities.

Conclusion

From this study, it is also likely to conclude that the existence and concreteness of social media altmetrics is yet low among scientific publications, thus testing the dependability of indicators based on them. Besides the fact that they reveal only feeble correlations with citations, proposes that the prospective of altmetrics as a substitute to the more conventional citation analysis is not very well-built but they can be complement to the conventional citations. Altmetrics could proffer an awfully fast vision about the social impact of science. They suggest broad spectra of the direct visibility of publications in social networks exemplified by very speedy diffusion in the web. The fact that an article is discussed passionately does not signify the importance of the article, but the curiosity among readers. In this sense, more study is required to facilitate conclusion and certify these newer form of metric system. Thus newer dimensions, such as altmetrics and article-level metrics are an effort to scrutinize the authority of research across the worldwide population.

[1]. Thelwall M, Haustein S, Larivière V, Sugimoto CR, Do altmetrics work? Twitter and ten other social web services PloS one 2013 8(5):e64841  [Google Scholar]

[2]. Scarlat MM, Mavrogenis AF, Pećina M, Niculescu M, Impact and alternative metrics for medical publishing:Our experience with International Orthopaedics Int Orthop 2015 39(8):1459-64.  [Google Scholar]

[3]. Rasmussen PG, Andersen JP, Altmetrics:An alternate perspective on research evaluation Sciecom info 2013 9(2)  [Google Scholar]

[4]. Hamm MP, Klassen TP, Scott SD, Moher D, Hartling L, Education in health research methodology:Use of a wiki for knowledge translation PLoS One 2013 8:e64922  [Google Scholar]

[5]. Hamm MP, Chisholm A, Shulhan J, Milne A, Scott SD, Klassen TP, Social media use by health care professionals and trainees:A scoping review Acad Med 2013 88:1376-83.  [Google Scholar]

[6]. Haustein S, Peters I, Sugimoto CR, Thelwall M, Larivière V, Tweeting biomedicine:An analysis of tweets and citations in the biomedical literature J Assoc Inf Sci Technol 2014 65:656-69.  [Google Scholar]

[7]. Priem J, Taraborelli D, Groth P, Neylon C, Altmetrics:A manifesto 2010 cited 2017 Apr 3Available from: http://altmetrics.org/manifesto  [Google Scholar]

[8]. Kwok R, Research impact:Altmetrics make their mark Nature 2013 500:491-93.  [Google Scholar]

[9]. Priem J, Groth P, Taraborelli D, The altmetrics collection PloS one 2012 7(11):e48753  [Google Scholar]

[10]. Fenner M, Bartling S, Friesike Sascha Altmetrics and other novel measures for scientific impact Opening science. The evolving guide on how the Internet is changing research, collaboration and scholarly publishing 2014 ChamSpringer:179-189.  [Google Scholar]

[11]. Kolahi J, Khazaei S, Altmetric:Top 50 dental articles in 2014 Br Dent J 2016 220:569-74.  [Google Scholar]

[12]. Priem J, Piwowar HA, Hemminger BM, Altmetrics in the wild:Using social media to explore scholarly impact 2012 Last Accessed 2016 October 21Available: http://arxiv.org/abs/1203.4745  [Google Scholar]

[13]. Lapinski S, Piwowar H, Priem J, Riding the crest of the altmetrics wave How librarians can help prepare faculty for the next generation of research impact metrics Coll Res Libr News 2013 74(6):292-300.  [Google Scholar]

[14]. Scimago Journal RankLast Accessed 2016 October 28Available at: http://www.scimagojr.com/journalrank.php  [Google Scholar]

[15]. Journal Impact factorLast Accessed 2016 October 28Available at: http://admin-apps.webofknowledge.com/JCR/help/h_impfact.htm  [Google Scholar]

[16]. Melero R, Altmetrics–A complement to conventional metrics Biochem Med 2015 25(2):152-60.  [Google Scholar]

[17]. Bergstrom CT, West JD, Wiseman MA, The Eigenfactor metrics J Neurosci 2008 28:11433-34.  [Google Scholar]

[18]. Hirsch JE, An index to quantify an individual’s scientific research output Proc Natl Acad Sci 2005 102:16569-72.  [Google Scholar]

[19]. Khodiyar VK, Rowlett KA, Lawrence RN, Altmetrics as a means of assessing scholarly output Learn Publ 2014 27(5):25-32.  [Google Scholar]

[20]. Barbic D, Tubman M, Lam H, Barbic S, An analysis of altmetrics in Emergency Medicine Acad Emerg Med 2016 23(3):251-68.  [Google Scholar]

[21]. Thoma B, Sanders JL, Lin M, Paterson QS, Steeg J, Chan TM, The social media index:Measuring the impact of emergency medicine and critical care websites West J Emerg Med 2015 16(2):242-49.  [Google Scholar]

[22]. Batooli Z, Ravandi SN, Bidgoli MS, Evaluation of scientific outputs of Kashan university of medical sciences in scopus citation database based on scopus, researchgate, and mendeley scientometric measures Electron Physician 2016 8(2):2048-56.  [Google Scholar]

[23]. De Gregori M, Scotti V, De Silvestri A, Curti M, Fanelli G, Allegri M, Does a research group increase impact on the scientific community or general public discussion? Alternative metric-based evaluation J Pain Res 2016 9:391-95.  [Google Scholar]

[24]. Allen HG, Stanton TR, Di Pietro F, Moseley GL, Social media release increases dissemination of original articles in the clinical pain sciences PLoS One 2013 8(7):e68914  [Google Scholar]

[25]. Haustein S, Grand challenges in altmetrics:Heterogeneity, data quality and dependencies Scientometrics 2016 108(1):413-23.  [Google Scholar]