Citing in science

A common way to measure the quality of academic research is counting how frequently it is cited. Citation counts are used in making decisions, including hiring and promotion, awarding grants, evaluating academic departments, and purchasing journals. To understand why authors cite specific references, we asked the lead authors of 99 research articles in psychology and biology to rate every reference in their article for its importance to the research paper using a scale of 1 (slightly important) to 7 (absolutely important). We also asked them to indicate the primary reason for that reference, as well as their relationship, if any, to the cited author.

So what determines importance?

Frequency, Length, and Citation Reasons
The average rating for importance was approximately 5, indicating that most authors thought the citations were important for their paper. On average, 83% of citations were listed only in parentheses or by a reference number, with no specific mention in the text, and another 10% were barely mentioned in the text. A more important reference was cited more frequently. Importance was also related to the number of words (citation length) used to describe the citation.

The three major reasons for citation were general background (37%), conceptual reasons (31%), methods and data (24%). References cited for general background were rated as less important than those cited for conceptual or methodological reasons.

Location, Location, Location
Combining articles from both disciplines, on average, 58% of references were cited in the Introduction, 19% in the Methods, 8% in the Results, and 36% in the Discussion. References cited in either the Methods, the Results, the Discussion, or in multiple locations, particularly in both the Introduction and Discussion, all tended to be considered as more important than references that were cited only in the Introduction.

Other Factors Affecting Importance
Two factors qualify conclusions about importance based only on indicators such as frequency, length, citation reasons, and location.

Pond Effects
Like the proverbial big fish in a small pond, a frequently-cited reference was regarded as particularly important for an article if there were relatively few other references cited multiple times. Similarly, a reference cited for reasons of conceptual ideas was judged as being particularly important if there were relatively few other references cited for conceptual ideas. In other words, the importance of a cited reference depended in part on citation patterns of the article as a whole.

Almost 10% of citations were self-citations, and another 10% were citations to authors with whom the citing author had collaborated. Location and quantitative measures of a citation predicted the author’s assessment of importance better for non-self-citations than for self-citations. Participants often judged their own prior research as very important for the current article even if they cited it just once, in the introduction, as general background without any elaboration. The following graph illustrates this pattern.

Self versus Non-Self citations

Self- versus Non-Self citations

In searching the literature in a field, citation-based retrieval supplements term-based retrieval by finding an additional 25% of relevant documents. We suggest information retrieval systems and bibliographic databases include more citation metadata information such as where the citation occurs and how frequently it was cited within a given article.

Rong Tang1 and Martin A. Safer2
1School of Library and Information Science, Simmons College, Boston, Massachusetts
2Department of Psychology, Catholic University of America, Washington, DC



The Psychology of Referencing in Psychology Journal Articles.
Safer MA, Tang R
Perspect Psychol Sci. 2009 Jan


Leave a Reply