Citation Analysis & Impact Factors

Introduction

When thinking about citation analysis and impact factors, you should try to consider the perspective from your own field of study. Some possible questions include:

  • How do you measure success of an article or book in your discipline?
  • How do you measure the success of a scholar in your discipline?
  • Is there discussion of citation analysis/impact factor particular to your discipline?
  • Are impact factors used in promotion and tenure decisions regarding professors in your department? In evaluating job applicants?

Purpose of Citations

Attribution of ideas/research

  • for confirming/illustrating a point
  • for disputing/correcting/questioning
  • for use of methods, tools, design, definition or data

Provide proof that position is well-researched by providing

  • holistic view of research
  • literature review
  • primary and historical sources

Help disseminate useful knowledge

  • demonstrating other points of view
  • supplementary information

Give formal credit for research as a normative research practice

  • for purpose of association and community
  • to meet funding, legal, and governmental requirements

What is Citation Analysis

Citation Analysis is the examination of the frequency and patterns of citations to books, journal articles, individual journals, conference proceedings, etc. Citation analysis began in the sciences in the 1960s and spread to the social sciences and humanities in the 1970s. Citation analysis includes but is not limited to citation counts, statistical analysis (e.g. h-factor), and data visualization.

Citation Metrics are used to evaluate

the impact of an individual researcher

  • Citation metrics are commonly used as one measure in academic job applications as well as tenure and promotion applications
  • Must not be conflated with quality of research

the impact of an individual journal article, book, etc.

  • Can help with selecting key works in a field.

the impact of an individual journal

  • May be helpful in choosing a journal in which to publish

the impact of a department or university

  • Included in the calculation of widely-published university ranking lists.
  • Can be a factor in funding, for example, the UK's Research Excellence Framework.

Citation Metrics Shortcomings

  • Can be susceptible to self- or endogenous-citation practices. Examples
  • May not favour interdisciplinary work, new journals, or Canadian journals
  • Comparisons across disciplines are not valid. Disciplinary publishing and citation practices vary, so the top journal in once discipline may have an impact factor of 36 while the top journal in different discipline may have an impact factor of 8.

Metrics

Traditional scholarly metrics count publications and citations in journals, books, etc. Altmetrics are new metrics that count numbers of downloads, views, comments on scholarly websites and blogs, etc.

Scholarly Metrics

H-index from a plot of decreasing citations for numbered papers

Individual author/article citation counts Citation Tracking (CT) counts the number of times an individual article or a particular author has been cited by other scholars. Large numbers are associated with greater impact and influence. Article and author level citation counts are available on Web of Science, Google Scholar, PLoS, BioMed Central, plus numerous discipline-specific databases.

H-index A measure of author influence, an h-index is the number where the number of articles published by an author intersects on a graph with the number of citations for each article. For instance, an author with h-index of 10 has published 10 papers that have been cited at least 10 times each. The h-index is the first (and most well-known) of many author metrics. Available on Web of Science and Google Scholar if the scholar has created a user profile.

Journal Impact Factor (used by ISI Web of Science) A Journal Impact Factor (JIF) is measured by dividing the number of current citations a journal has in a given year, by the number of articles published in the two previous years. The JIF is used to indicate the relative importance of a journal within a given field and a higher JIF is seen as providing more “authority” or “weight” to a researchers’ work.

Eigenfactor A non-commercial alternative to ISI Web of Science's JIF. The Eigenfactor algorithm also takes into account the significance of the citations coming into a journal so that citations from important journals are weighted more highly.

Acceptance Rate Acceptance rate compares the number of articles selected for publication to the number of articles submitted. For example, the very prestigious journal Nature publishes fewer than 10% of the articles submitted. In general, a low acceptance rate is associated with high prestige. Not all journals publish their acceptance rates, but in many cases it is included in the "instructions to authors" section of the journal.


Concerns Use of the Journal Impact Factor as "the primary parameter with which to compare the scientific output of individuals and institutions" has been criticized by a number of scholars and organizations, most notably here.

Most scholarly metric tools include only a small fraction of citations that appear in books or book chapters: the work of Humanities and Social Sciences scholars is not accurately reflected.

Alternative Metrics

Download counts and View Counts These article level metrics count the number of times an article has been downloaded and the number of times that it has been viewed. Download counts can be used to demonstrate use of articles for purposes other than citation such as education, background research, presentation, or use in grey literature such as white papers or reports. Download and view counts are available for many, but certainly not all journals.

Counting Citations in non-scholarly media Altmetric.com tracks tweets, blog posts, news stories and other content that mention scholarly articles.

Counting Citations to Additional Publication Types For example, impactstory.org includes datasets, software, slide decks, figures and posters.

Authority scores (e.g. Klout) Authority scores attempt to measure how many people you reach through your own contacts, their amplified reach (e.g. how many people are followed by people who follow you on Twitter) and how often your content elicits a response (i.e. a comment or reply).

Tools

While many publishers and scholarly databases support searching for cited references, the feature is generally limited to articles within their own database. These academic and altmetric tools are commonly used for more thorough citation analysis.

Academic tools

ISI Web of Science Core Collection

ISI Journal Citation Reports

  • One of the most standard commercial tools for finding journal Impact factors, article citation counts and other author metrics
  • Favours sciences over social sciences and humanities
  • Journal-based (i.e. Uses journal articles as a source for citations. Books, patents, etc. are included only if they have been cited by a journal article.)

Video example of creating citation reports for journals, articles, authors, etc. via the Web of Science Core Collection

Video example of creating citation reports for journals via the separate Journal Citation Reports database.

SciVal

  • Large database of bibliographic citations, plus tools for comparing institutions, departments, research groups and individual scholars.

Google Scholar Citations; Sample Profile UBC Math Professor Nassif Ghoussoub

  • Video Tutorial of Google Scholar Citations
  • Allows authors to curate their own list of publications
  • Calculates citation counts, h-index, and i10-index
  • Tracks conference proceedings, chapters in edited volumes, and books, but not consistently
  • Better for humanities and many social sciences than ISI Web of Science or Scopus/SciVal

Google Books

  • Tracks references in books to books, book chapters (and journal articles)
  • Finds unique citations, i.e. not included in Google Scholar, Web of Science or Scopus/SciVal
  • Books to journal citation ratio, roughly: Philosophy 4:1 |Sociology, Fine Arts 3:1 | Psychology 1.5:1 | Physics .001:1

Publish or Perish

Altmetrics

ImpactStory

  • Set up your own researcher profile and share all the "diverse impacts" your research has made, from journal articles to data sets and blog posts. (Free 30-day subscription; 60 USD/Year.)

Altmetric.com/

  • "Captures hundreds of thousands of tweets, blog posts, news stories and other content that mention scholarly articles." Add the free Altmetric Bookmark to your bookmarks toolbar and see the metrics for any article with a DOI and items in digital repositories.

Other Tools

Top Ten (10) indices measuring scholarly impact in the digital age from The Search Principle, April 2012

Readings

Fang, F. C., & Casadevall, A. (2011). Retracted science and the retraction index. Infection and Immunity, 79(10), 3855-3859. doi: 10.1128/IAI.05661-11

García-Pérez, M. A. (2010), Accuracy and completeness of publication and citation records in the Web of Science, PsycINFO, and Google Scholar: A case study for the computation of h indices in Psychology. Journal of the American Society for Information Science and Technology, 61: 2070–2085. doi: 10.1002/asi.21372

Kousha, K., Thelwall, M., & Rezaie, S. (2011). Assessing the citation impact of books: The role of Google Books, Google Scholar, and Scopus. Journal of the American Society for Information Science and Technology, 62(11), 2147-2164. doi:10.1002/asi.21608

McDonald, John (2003). The Book is Dead: Citation Rates in the Humanities and Social Sciences. In Strauch, Katina (ed.) Charleston Conference Proceedings (21st, Charleston, South Carolina, November 1-3, 2001) 143-152. [Google Books Link, not all pages viewable]

Moed, Henk F. (2005). Citation Analysis in Research Evaluation. Information Science and Knowledge Management 9. doi:10.1007/1-4020-3714-7

source: http://wiki.ubc.ca/Library:Citation_Analysis_&_Impact_Factors