- Original Contribution
- Open Access
Temporal trend of research related to gun violence from 1981 to 2018 in the United States: a bibliometric analysis
Injury Epidemiology volume 7, Article number: 9 (2020)
- The Correction to this article has been published in Injury Epidemiology 2020 7:43
We aimed to evaluate the variation in gun violence-related research in the US over time to determine if there are meaningful changes in frequency of research at certain time points. Related publications were searched from the Web of Science.
We searched articles from Web of Science to collect publication data of gun violence research in three disciplines (clinical sciences, life sciences, and social behavior sciences) from 1981 to 2018. The joinpoint regression approach was applied to evaluate the trend of publication ratio. We also adopted the generalized additive mixed model to compare the publication ratio among the three research disciplines.
During the study period, each research discipline had a significant decrease in publication ratios, especially social behavioral sciences from 2001 to 2011, with an annual percentage change = − 9.77% (95% CI = − 13.45, − 5.93; p-value < .0001). After combining the three research disciplines, the average change of the publication ratio was significantly increased 9.18% (95% CI = 6.42, 12.01; p-value < .0001) per year from 1981 to 2018. Compared to social behavioral sciences, both clinical sciences and life sciences had a significantly smaller publication ratio.
Gun violence research exhibited a significant downward trend in publications in the early 2000s, which may be attributed at least in part to limited federal funding, but the publication ratio increased since the 2010s. To enhance the amount of peer-reviewed gun violence research so that research-informed gun violence interventions are more likely to succeed, decision-makers should keep supporting quality research.
Gun violence is a critical public health issue in the US and contributes significantly to premature morbidity and mortality. In 2017 there were nearly 40,000 firearm related deaths (Pilkington 2018). Firearm death rates in the US far surpass those of all other populous high income OECD countries; US overall firearm death rates are 10 times higher, firearm suicide rates are 8 times higher, and firearm homicide rates are 25 times higher (Grinshteyn and Hemenway 2016). Furthermore, gun violence has substantial economic consequences, with medical and work-loss costs alone for fatal and non-fatal firearm injuries estimated at over $45 billion annually (Centers for Disease Control and Prevention 2010), and these costs do not even take into account lost quality of life and any physical, social, or emotional toll on families and loved ones.
Firearm ownership has been associated with increased rates of intentional and unintentional firearm deaths. State-level gun ownership rates have been found to be a strong correlate of firearm homicide rates when using various national datasets (Siegel et al. 2013; Siegel et al. 2014). Additionally, research posits that states that have more permissive concealed-carry gun laws have higher homicide rates (Siegel et al. 2017). Firearm ownership, access and storage practices are a risk factor for suicide (Brent et al. 1993; Brent et al. 1991; Cummings et al. 1997; Bukstein et al. 1993; Kellermann et al. 1992; Grossman et al. 2005; Miller et al. 2002). A meta-analysis of 16 observational studies concluded that access to firearms is associated with risk for completed suicide and being the victim of homicide (Anglemyer et al. 2014). Various studies have also linked firearm access to unintentional firearm deaths, particularly among youth (Miller et al. 2002; Grossman et al. 1999; Levine and McKnight 2017).
Despite what we know about gun violence, many related topics remain only partially understood and therefore not fully actionable. For example, a Community Guide systematic review focused specifically on law-based interventions to gun violence concluded in 2005 that there was insufficient evidence to determine whether firearm laws were effective interventions (Hahn et al. 2005). More recently, an expert-crafted research agenda around gun violence calls for a need to better understand, among other issues, gun violence characteristics, ways to prevent gun-related injuries, and risks and protective factors of gun violence (Council NR 2013).
The persistence of such important questions in this area may not be surprising in light of patterns of gun violence research. It may be explained, at least in part, by a chilling effect created by Congressional efforts that began in 1996 to limit funding for gun-violence research at and supported by federal agencies, and in particular the Centers for Disease Control and Prevention (CDC) that especially affected health-related disciplines (Stark and Shah 2017; Rubin 2016; Kellermann and Rivara 2013; Ladapo et al. 2013). In the context of health-related research, studies have shown under-funding of gun violence research relative to its health consequences. For example, Stark and Shah demonstrated that the expected rates of publication and federal funding for gun violence research appears low compared to other causes of death (2017). As Kellermann and Rivara articulate: “The nation might be in a better position to act if medical and public health researchers had continued to study these issues” (2013).
Bibliometric analysis, frequently used to assess journal article output, is a sub-field of infometrics, used to measure publication outputs and includes assessing the amount of research in a particular field (Ismail et al. 2012). Bibliometric research output information can be a proxy for how much research is being conducted on a particular topic and can also be used to help propel progress in a research field (Ismail et al. 2012). In the area of gun violence research, existing bibliometric research indicates that gun violence research article publication has varied by year since the 1970s, with an increase between 1985 and 1999 and then a level period through 2012 (Alcorn 2017). Relative to the overall growth of scientific literature, however, there appears to be a decline in gun violence publications between 1998 and 2012 (Alcorn 2017).
While these general patterns emerge, what is less clear is whether there are points in time in which there are meaningful changes in frequency of publication. These points are important to identify because they can help pinpoint what periods of time should be probed further to determine why the incidence of publication changed. Determining changes in publication incidence can help understand research around gun violence, the findings of which can help identify evidence-informed interventions to reduce gun violence and thereby suicide and homicide. As a result, we aimed to evaluate the variation in gun violence-related publication in the US over time to identify whether there are meaningful changes in frequency of publications at certain time points and to consider whether covariates of gun-violence are associated with changes in publication ratios.
We retrieved all publication information from the Web of Science Core Collection database (Clarivate Analytics). We found these publications through a series of key terms searches. First, we set up 10 primary keywords to capture gun concepts (i.e., gun, handgun, firearm, rifle, shotgun, pistol, revolver, semi-automatic, bullet, and ammunition). Then, we selected five secondary keyword sets to ensure that the articles collected through primary keyword searches actually focused on researching elements of gun violence that related to its health risks, health consequences, and policy interventions: violence (10 keywords), perpetrator (6 keywords), health outcome (10 keywords), risk factor (11 keywords), and law (39 keywords) concepts. Researchers generated these keywords with the help of online databases, such as Medical Subject Headings, which is a vocabulary thesaurus used for indexing articles in PubMed, and PubReMiner, which links to the PubMed literature database for text mining. Detailed keywords for each category are provided Table 1. Finally, a total of 770 (10 × (10 + 6 + 10 + 11 + 39)) keyword combinations from a primary keyword and a secondary keyword were searched in all titles, abstracts, keywords, and KeyWords Plus® fields. We limited our analysis to research articles or other forms of communication that presented new knowledge and were published between 1981 and 2018.
Subject category selection and review process
We examined trends in research related to gun violence by relying on Web of Sciences’ 254 subject categories (e.g., anthropology, biomedical engineering, ecology, and emergency medicine). Two researchers of this study independently determined whether journal articles within each subject category could potentially contain gun violence-related research. They then independently assigned “screened in” subject categories into the following disciplines: clinical sciences, life sciences, and social behavior sciences. In the event of discordance, a third reviewer broke the tie. We adopted the highest principle to assign a subject category in a group by reaching an agreement from the two reviewers. Thus, a total of 68 subject categories (26.77%) were assigned, where clinical sciences have 15, life sciences have 14, and social behavior sciences have 39 subject matter categories. The corresponding subject categories in each research discipline are contained in Table S1 (Additional file 1).
The original keywords query in the Web of Sciences returned 12,675 publications, including 3815 in clinical sciences, 3816 in life sciences, and 5044 in social behavior sciences. Because we found many duplicate publications in the original query, we de-duplicated to filter out replicated publications in each research discipline. We then excluded articles without keywords in the titles. These steps resulted in 1037 publications in clinical sciences, 914 publications in life sciences, and 1373 publications in social behavior sciences. Then, two researchers of this study independently screened article titles to determine if the publication was about gun violence and resolved any discordances through discussion. If the two researchers determined that an article was not gun violence research, a third researcher reviewed the article’s contents to verify that the article was indeed unrelated to gun violence research. After screening, 513 publications remained in clinical sciences, 544 publications remained in life sciences, and 843 publications remained in social behavioral sciences. We further excluded publications for which the study areas were outside the US after reviewing the contents of the articles. This exclusion resulted in 448 publications in clinical sciences, 537 in life sciences, and 812 in social behavior sciences.
This study adopted two statistical modeling approaches. First, we applied the joinpoint regression approach (Kim et al. 2000) to test whether an apparent change in publication trend was statistically significant from 1981 to 2018 in each research discipline and the disciplines combined. The approach can evaluate the number of joinpoints to partition the trend into several segments, and the change in each segment can be estimated by the following model equation:
, where NPt is the number of publications in gun violence research at calendar time t, and log(Totalt) is the offset from the annual total publications. A model selection was adopted to choose a joinpoint regression from 0 to 4 joinpoints, and the best model was determined by the smallest Bayesian information criterion (BIC). Because the whole study period is across four decades, we limited the maximum number of joinpoints by four to prevent short segments. The annual percent change (APC) of each segment and the average annual percent change (AAPC) of the entire study period were also calculated based on the best model.
Second, we built a generalized additive mixed model to compare the differences of publication ratios among the three research disciplines, and evaluated the influence of gun ownership and firearm homicide and suicide death on publication ratios. Suppose the number of publication of research discipline i (1 = clinical sciences; 2 = life sciences; 3 = social behavior sciences) at time t is denoted by NPit, which follows a Poisson distribution POI(μit), where μit is the mean of NPit, the model equation can be built as:
, where α is a fixed intercept, and αi is a random intercept specified by research disciplines nested with time, which can account for unobserved predictors affecting the number of publications. The symbols β1 and β2 are the coefficient parameters of two indicator functions for clinical sciences (i.e., I(i = 1)) and life sciences (i.e., I(i = 2)), respectively, as social behavior research is defined as the reference level. The autocorrelation is controlled by a time smoothing spline f(t), where its smoothing parameter is estimated by generalized cross validation. The estimated β1 and β2 were transformed into rate ratios to compare the publication ratio of clinical sciences and life sciences versus social behavior sciences. The last term log(Totalit) is an offset calculated from the logarithm of the total number of publications in each research discipline at time t.
Statistical analyses were implemented by Joinpoint Regression Program V.188.8.131.52 (National Cancer Institute, USA) and SAS v9.4 (SAS Institute Inc., Cary). The significance level was set to 5%.
Figure 1 shows certain variations in both the number of publications and the crude publication ratio of gun violence research in each discipline. Social behavioral sciences had more related publications and a higher publication ratio in most study years. As Table 2 indicates, in the social behavioral sciences discipline, on average, 21.37 papers about gun violence were published per year (Standard deviation = 16.46), which is more than the annual average of clinical sciences and life sciences. Moreover, social behavioral sciences had the highest publication ratio (51.68 per 100,000 publications) compared to the other two research disciplines. Furthermore, even though life sciences had more publications than clinical sciences per year (14.13 vs. 11.79), its annual publication ratio was smaller (26.14 vs. 30.97 per 100,000 publications).
Table 3 shows the BIC among different joinpoints in terms of individual research disciplines and the combination of disciplines. In individual research disciplines, both clinical sciences and social behavioral sciences had the best model with three joinpoints, where their smallest BIC was 0.40 and 1.05, respectively. The best model for life sciences only had two joinpoints with the smallest BIC = 1.01. Considering the combination of the three research disciplines, the model with two joinpoints was determined the best model with the smallest BIC = 1.45.
The selected joinpoints, shown in Fig. 2, explicitly reveal an increasing trend before 1998 in each research discipline, except for social behavior sciences from 1981 to 1984. Moreover, after 1998 each specific research discipline had at least one decreased APC in the publication ratio. Table 4 shows that those decreased trends significantly appeared from 1998 to 2012 in life sciences (APC = − 4.99%; 95% confidence interval (CI) = − 8.23, − 1.64; p-value = 0.0052) and from 2001 to 2011 in social behavior sciences (APC = − 9.77%; 95% CI = − 13.45, − 5.93; p-value < .0001). In clinical sciences, the decreased trend was shorter (2004–2008) and not significant. The decreased trend was first bounced back in clinical sciences since 2008 (APC = 17.65%; 95% CI = 13.96, 21.46; p-value < .0001), and then in social behavior sciences since 2011 (APC = 16.96%; 95% CI = 11.35, 22.85; p-value < .0001). Even though the APC in life sciences did not increase until 2012, it was 26.10% (95% CI = 16.77, 36.17; p-value < .0001), the largest significant increase, compared to the other two research disciplines. Combining the three research disciplines,. the total publication ratio had two significant increases from 1984 to 1996 (APC = 17.76%; 95% CI = 11.14, 24.76; p-value < .0001) and from 2011 to 2018 (APC = 20.44%; 95% CI = 15.06, 26.07; p-value < .0001). However, a significant decrease was observed from 2002 to 2011 (APC = − 8.37%; 95% CI = − 12.84, − 3.68; p-value < .0001).
From 1981 to 2018, the average change of the publication ratio in the combined research disciplines per year was significantly increased with an AAPC = 9.18% (95% CI = 6.42, 12.01; p-value < .0001). In the individual research disciplines, the AAPC was the highest in clinical sciences, where the publication ratio significantly increased 9.92% per year (95% CI = 5.52, 14.50; p-value < .0001). Life sciences had a lower but still significant AAPC = 8.49% (95% CI = 5.60, 11.46; p-value < .0001). No significant AAPC was found in social behavioral sciences.
The publication ratio had significant differences across research disciplines after controlling for gun ownership, firearm homicide, and firearm suicide rates. Compared to social behavioral sciences, clinical sciences had a significantly smaller publication ratio by 0.64 times (95% CI = 0.55, 0.73; p-value < .0001). In addition, the publication ratio in life sciences was 0.54 times (95% CI = 0.47, 0.61; p-value < .0001), significantly fewer than that in social behavioral sciences.
This study bolsters the existing literature about gun violence research. We used bibliometric analysis to analyze variation in gun violence-related research in three discipline areas between 1981 and 2018. We found that all disciplines followed a similar pattern, with two increases bisected by a decrease. The late 1990’s/early 2000’s began a decade long decrease in gun violence related research publications. Previous studies have tracked the prevalence of gun violence research over time (Stark and Shah 2017; Ladapo et al. 2013; Alcorn 2017) and have demonstrated that gun violence research has been relatively limited both in quantity and relative the impact of gun violence on health (Stark and Shah 2017; Ladapo et al. 2013). This study adds to the literature by identifying points of significant increase and decrease in gun violence research over time across disciplines. It reaches these findings through systematic searching and screening of published literature in multiple databases across disciplines over four decades and applying a statistical model with sound methodology to evaluate changes over time.
The findings of this study are important for understanding past and contemporary policy choices and their effects. This study lends analytical support for links made in the literature suggesting a direct relationship between the availability of federal funding for gun violence research and peer-reviewed publications (Stark and Shah 2017; Ladapo et al. 2013; Alcorn 2017). This study found meaningful increases in research in 1984 and 2011 and a meaningful decrease in 2002. The observed decrease is consistent with certain federal funding patterns for health-related gun violence research. The Dickey Amendment, which if not in substance at least in effect halted CDC-funded research in this area, was originally adopted in 1996 and subsequently re-authorized. Similar limitations also applied to the National Institute of Health (NIH) in 2011 (Rostron 2018). The decrease we observed in overall publications in 2002 may be explained by the cumulative effect of the Dickey Amendment given, at least in part, the typical lag between when federal research funding is awarded and when resulting peer-reviewed research is ultimately published. In our study, the significant decrease in the disciplines of life sciences and social and behavioral health are especially consistent with this explanation since research in these disciplines can be funded by the US Department of Health and Human Services (HHS). Interestingly, clinical sciences, also often funded by HHS, saw less of a decrease at this time. This may be explained in part by the adoption of alternate funding stipulations on the NIH, which funds much more clinical research than the CDC.
This study is not without limitations. First, we used only one database, Web of Science, which may have missed relevant articles. We also only analyzed research in three discipline areas, though these three areas did have the highest amount of publications related to firearm violence. While a strength of our study is that we screened each article title to ensure it was firearm-related, it is possible that we may have missed some relevant articles that did not make apparent their connection to firearm violence through their titles. Therefore, it is possible that relevant articles were missed. We also only considered articles that were examining topics specific to the US. While the US has the highest firearm mortality rates in the world, and very unique policy and funding related issues, a more global picture of firearm research may be beneficial. Furthermore, two researchers assigned subject categories to research. It is possible that outside researchers would assign some subject categories to different research disciplines. To minimize this bias, however, the researchers assigned subject categories to research disciplines independently and in the event of discordance, a third reviewer broke the tie. In addition, if an article was categorized into multiple subject categories by Web of Science and these subject categories were assigned to different research disciplines, the article would appear multiple times in our results. Counting them twice means some duplicate articles across research categories; but it also means that each research category received “credit” for such an article.
Our findings are consistent with the notion that limited funding tracks with gun violence publication across research disciplines. Gun violence has remained a problem in the U.S. during the time period of study but publication rates seemed to shift. Our findings of specific upward and downward trends in publication sets the stage for future inferential work that examine the exact determinants of these changes. Monitoring both funding and publication of gun violence research is especially important if federal funding truly becomes more readily available for this type of work. Such monitoring could include systematically gathering all kinds of research resources (e.g., data, funding opportunities, publications) in an open platform to facilitate accessing necessary information to develop gun violence research. A recent good news is that federal agencies will receive $25 million from Congress to study gun violence, including $12.5 million each for the CDC and NIH (Hellmann 2019). This is the first time after 1996 when Congress stopped funding gun violence research according to the Dickey Amendment. We hope the upward trend in the past 10 years revealed by this study along with the future’s federal funding sources will motivate more researchers, no matter experienced or new blood, to bring an optimistic prospect in gun violence research area.
Research and the search for knowledge may be intrinsically worthwhile. In public health, however, research is also critical to understand the scope of a problem, its risk factors, and the effectiveness of potential interventions - including programs, policies, and laws. Approaching gun violence as a public health problem therefore requires producing quality research that identifies the scope of the problem and potential solutions and then utilizing this research in practice. Without a robust body of gun violence research, the effectiveness of gun violence interventions is a function of trial and error. This study supports the notion that the aggregate amount of gun violence research is sensitive to the context in which it is completed. Such context might be research funding and gun violence trends. Thus, to enhance the amount of peer-reviewed gun violence research so that research-informed gun violence interventions are more likely to succeed, decision-makers should keep supporting quality research.
Availability of data and materials
The datasets generated during and analyzed during the current study are available from the corresponding author on reasonable request.
Average annual percent change
Annual percent change
Bayesian information criterion
Centers for disease control and prevention
Alcorn T. Trends in research publications about gun violence in the United States, 1960 to 2014. JAMA Intern Med. 2017;177:124–6.
Anglemyer A, Horvath T, Rutherford G. The accessibility of firearms and risk for suicide and homicide victimization among household members: a systematic review and meta-analysis. Ann Intern Med. 2014;160:101–10.
Brent DA, Perper JA, Allman CJ, Moritz GM, Wartella ME, Zelenak JP. The presence and accessibility of firearms in the homes of adolescent suicides: a case-control study. JAMA. 1991;266:2989–95.
Brent DA, Perper JA, Moritz G, Baugher M, Schweers J, Roth C. Firearms and adolescent suicide: a community case-control study. Am J Dis Child. 1993;147:1066–71.
Bukstein OG, Brent OA, Perper JA, Moritz G, Baugher M, Schweers J, et al. Risk factors for completed suicide among adolescents with a lifetime history of substance abuse: a case-control study. Acta Psychiatr Scand. 1993;88:403–8.
Centers for Disease Control and Prevention. Cost of Injury Reports. Centers for Disease Control and Prevention. 2010. https://wisqars.cdc.gov:8443/costT/. Accessed 15 Oct 2019.
Council NR. Priorities for research to reduce the threat of firearm-related violence. The National Academies of Sciences Engineering Medicine 2013. http://www.nationalacademies.org/hmd/Activities/%20PublicHealth/FirearmViolenceReduction.aspx. Accessed 15 Oct 2019.
Cummings P, Koepsell TD, Grossman DC, Savarino J, Thompson RS. The association between the purchase of a handgun and homicide or suicide. Am J Public Health. 1997;87:974–8.
Grinshteyn E, Hemenway D. Violent death rates: the US compared with other high-income OECD countries, 2010. Am J Med. 2016;129:266–73.
Grossman DC, Mueller BA, Riedy C, Dowd MD, Villaveces A, Prodzinski J, et al. Gun storage practices and risk of youth suicide and unintentional firearm injuries. JAMA. 2005;293:707–14.
Grossman DC, Reay DT, Baker SA. Self-inflicted and unintentional firearm injuries among children and adolescents: the source of the firearm. Arch Pediatr Adolesc Med. 1999;153:875–8.
Hahn RA, Bilukha O, Crosby A, Fullilove MT, Liberman A, Moscicki E, et al. Firearms laws and the reduction of violence: a systematic review. Am J Prev Med. 2005;28:40–71.
Hellmann J. Congress reaches deal to fund gun violence research for first time in decades. The Hill 2019. https://thehill.com/policy/healthcare/474740-25m-set-aside-for-gun-violence-research-in-spending-agreement-in-win-for. Accessed 24 Jan 2020.
Ismail S, Nason E, Marjanovic S, Grant J. Bibliometrics as a tool for supporting prospective R&D decision-making in the health sciences: strengths, weaknesses and options for future development. Rand Health Q. 2012;1(4):11.
Kellermann AL, Rivara FP. Silencing the science on gun research. JAMA. 2013;309:549–50.
Kellermann AL, Rivara FP, Somes G, Reay DT, Francisco J, Banton JG, et al. Suicide in the home in relation to gun ownership. N Engl J Med. 1992;327:467–72.
Kim HJ, Fay MP, Feuer EJ, Midthune DN. Permutation tests for joinpoint regression with applications to cancer rates. Stat Med. 2000;19:335–51.
Ladapo JA, Rodwin BA, Ryan AM, Trasande L, Blustein J. Scientific publications on firearms in youth before and after congressional action prohibiting Federal Research Funding. JAMA. 2013;310:532–4.
Levine PB, McKnight R. Firearms and accidental deaths: evidence from the aftermath of the Sandy hook school shooting. Science. 2017;358:1324–8.
Miller M, Azrael D, Hemenway D. Firearm availability and unintentional firearm deaths, suicide, and homicide among 5–14 year olds. J Trauma Acute Care Surg. 2002;52:267–75.
Pilkington E. Gun deaths in US rise to highest level in 20 years, data shows. Support the Guardian 2018. https://www.theguardian.com/us-news/2018/dec/13/us-gun-deaths-levels-cdc-2017. Accessed 15 Oct 2019.
Rostron A. The dickey amendment on Federal Funding for research on gun violence: a legal dissection. Am J Public Health. 2018;108:865–7.
Rubin R. Tale of 2 agencies: CDC avoids gun violence research but NIH funds it. JAMA. 2016;315:1689–92.
Siegel M, Negussie Y, Vanture S, Pleskunas J, Ross CS, King C III. The relationship between gun ownership and stranger and nonstranger firearm homicide rates in the United States, 1981-2010. Am J Public Health. 2014;104:1912–9.
Siegel M, Ross CS, King C III. The relationship between gun ownership and firearm homicide rates in the United States, 1981–2010. Am J Public Health. 2013;103:2098–105.
Siegel M, Xuan Z, Ross CS, Galea S, Kalesan B, Fleegler E, et al. Easiness of legal access to concealed firearm permits and homicide rates in the United States. Am J Public Health. 2017;107:1923–9.
Stark DE, Shah NH. Funding and publication of research on gun violence and other leading causes of death. JAMA. 2017;317:84–5.
This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.
Ethics approval and consent to participate
Consent for publication
The authors declare that they no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Following publication of the original article, the authors identified two errors: 1) The equations were not adequately presented since they all Greek symbols disappeared in the PDF version of the article. 2) The grey horizontal lines were missing in Fig. 1. The original article has been updated.
About this article
Cite this article
Chien, L., Gakh, M., Coughenour, C. et al. Temporal trend of research related to gun violence from 1981 to 2018 in the United States: a bibliometric analysis. Inj. Epidemiol. 7, 9 (2020). https://doi.org/10.1186/s40621-020-0235-6
- Gun violence
- Bibliometric analysis