A comparison and analysis of seven gun law permissiveness scales
Injury Epidemiology volume 8, Article number: 2 (2021)
Due to the differences in the way gun law permissiveness scales were created and speculation about the politically motivated underpinnings of the various scales, there have been questions about their reliability.
We compared seven gun law permissiveness scales, varying by type and sources, for an enhanced understanding of the extent to which choice of a gun law permissiveness scale could affect studies related to gun violence outcomes in the United States. Specifically, we evaluated seven different scales: two rankings, two counts, and three scores, arising from a range of sources. We calculated Spearman correlation coefficients for each pair of scales compared. Cronbach’s standardized alpha and Guttman’s lambda were calculated to evaluate the relative reliability of the scales, and we re-calculated Cronbach’s alpha after systematically omitting each scale to assess whether the omitted scale contributed to lower internal consistency between scales. Factor analysis was used to determine single factor loadings and estimates. We also assessed associations between permissiveness of gun laws and total firearm deaths and suicides in multivariable regression analyses.
All pairs of scales were highly correlated (average Spearman’s correlation coefficient r = 0.77) and had high relative reliability (Cronbach’s alpha = 0.968, Guttman’s lambda = 0.975). All scales load onto a single factor. The choice of scale did not meaningfully change the parameter estimates for the associations between permissiveness of gun laws and gun deaths and suicides.
Gun law permissiveness scales are highly correlated despite any perceived political agenda, and the choice of gun law permissiveness scale has little effect on study conclusions related to gun violence outcomes.
More than 32,000 people die and another 67,000 people are injured by firearms each year in the United States (Fowler et al., 2015). Rates of firearm death and injury dramatically vary state by state. For instance, according to the Centers for Disease Control and Prevention (CDC), Mississippi, Alabama, and Alaska have an age adjusted firearms death rate approximately seven times that in Rhode Island, Hawaii, and Massachusetts (Firearm Mortality Per State, 2020). Understanding the impact of state firearm legislation on health outcomes has therefore been a topic of interest for many health policy researchers. These studies commonly employ the use of scales that rate or rank “restrictiveness” or “permissiveness” of firearm legislation at the state level. These scales are useful for several reasons: to generally understand how the landscape of laws affects gun outcomes, to adjust for confounding from permissiveness of laws when it is impractical to include all state gun laws in a single analysis, and to avoid statistical complications due to collinearity of laws. This is especially relevant when a study’s statistical power is low due to small numbers, which is often an issue when units of analyses are US states over a short period of time or when the outcome of interest is rare, like mass shootings.
There are three primary ways in which the restrictiveness of state laws is rated—rankings, law counts, and scores. Rankings involve sequential lists of states from 1 to 50 based on restrictiveness based on predetermined subjective or objective assessment. Rankings assume the equivalence of each 1-unit increase. For example, Arizona might be ranked 1 in permissiveness, and Louisiana 17, (Wood, 2019) even though there might be little discernible difference between the state laws. Similarly, scores assign numeric values or letter grades based upon on predetermined criteria but allow ties. The gradation for scores can vary greatly—for example, a numeric value between 1 and 100 with decimal places—meaning that the interpretation of the results using these scales must be performed with care. Finally, law counts are the sum of active laws in each state, which allows for ties and skewed distributions, but has the disadvantage that laws are assumed to have equal effect sizes.
Permissiveness scales also arise from a variety of sources. One of the most commonly used scales (Siegel & Boine, 2019) is from the Giffords Law Center to Prevent Gun Violence’s Annual Scorecard/Legal Community Against Violence, an organization that is inherently gun-safety oriented (Gun Violence Statistics, 2019). Contrastingly, one study has recently used The Traveler’s Guide to the Firearm Laws of the Fifty States, a reference guide written for gun owners traveling across the United States that scores states on their permissiveness of gun laws (Reeping et al., 2019). Due to the differences in the ways each scale was created and the authors’ apparent political agenda, there have been questions from both the scientific and lay community about their reliability. For example, the study that used that Traveler’s Guide to the Firearm Laws of the United States was criticized by other gun violence researchers because “the gun law permissiveness scale used in the study has not been fully described, evaluated, or validated”(Webster et al., 2020) and was also criticized by the media for similar reasons (Hawkins, 2019). Furthermore, criticisms of Giffords Scorecard include articles written by pro-gun media such as, “Latest Gun Death Scorecard From Giffords Is Grossly Misleading” (Adelmann, 2019) and “Debunking Anti-Gun Giffords Law Center’s ‘Gun Law Scorecard.’“ (Tuohy, 2018).
The aim of this paper is to compare seven commonly available and prominent gun law permissiveness scales, of varying types and sources, to better understand the extent to which the choice of gun law permissiveness scale could affect the results of analyses on gun violence related outcomes. Two rankings, two counts, and three scoring scales are evaluated in this paper, arising from three gun-safety leaning sources and three gun-rights leaning sources.
Data sources and variables
We included the scales from the following sources for this evaluation: The Cato Institute (score), (Freedom in the 50 States, 2020) Everytown for Gun Research (count), (Gun Law Navigator, 2020) Giffords Law Center (rank and score), (Annual Gun Law Scorecard, 2020) Guns and Ammo Magazine (rank), (Wood, 2019) Siegel count of provisions (count), (McClenathan & Pahn, 2016) and the Traveler’s Guide to the Firearm Laws of the Fifty States (score) (Kappas, 2019). Giffords, Everytown, and the Siegel count are gun-safety oriented, while Cato, Guns and Ammo, and the Traveler’s Guide are gun-rights oriented. The years in which the restrictiveness scales are publicly available for use, the methods and criteria that were used to create the scale, and the location where the scales can be accessed are provided in Table 1. Because 2016 was the last time that data were available across all of these scales (late-July 2015 for Guns and Ammo, as 2016 is missing), we conducted our comparisons of the scales using 2016 as a reference year.
In order to visualize the relationship between each of the sources, we produced scatterplots of each pair of scales. We then calculated Spearman correlation coefficients between each of the scales.
To examine internal consistency, we calculated the Cronbach’s standardized alpha and Guttman’s lambda for the set of seven scales (Santos, 1999). We re-calculated Cronbach’s alpha after systematically omitting each scale to assess whether the omitted scale was contributing to lower internal consistency between scales. If this “dropped” Cronbach’s alpha appeared to be meaningfully closer to one than when all the scales are included, then that specific scale could be considered to be less reliable than the others. This method has previously been used to assess reliability (Rao et al., 2005).
We then used factor analysis (Rummel, 1988) to evaluate if all of the scales load onto a single factor. To do this, we produced a scree plot of the eigenvalues for each additional component with an oblique rotation (D’agostino Sr & Russell, 2005; Ledesma et al., 2015). Scales measuring the same construct should load onto a single factor (i.e. with an eigenvalue above one for the first factor). We also calculated the rotated factor loadings (eigenvectors) for each assessed scale, wherein a loading approaching one indicates the underlying scale strongly influences the factor (Dien et al., 2005).
We replicated the methods used in a paper published in 2013 (Fleegler et al., 2013) that examined the association between legislative strength of firearm laws and both total firearm related deaths and firearm suicides between 2007 and 2010, obtained from the Web-Based Injury Statistics Query (WISQUARS) (Underlying Cause of Death, 1999-2017) . This paper, by Fleegler et al., used a Poisson regression to estimate the association, and adjusted for state-level statistics that they received from the US Census, (American FactFinder, 2016) including race, sex, percent below poverty level, percent unemployment, percent with a college education, and state population density. An offset of population was utilized so that a relative rate could be calculated. We standardized each of the seven permissiveness scales, then repeated the analysis with each scale included as the exposure of interest.
Fleegler et al. (2013) assessed state firearm law permissiveness using data from the Law Center to Prevent Gun Violence and the Brady Center to Prevent Gun Violence. States were grouped as quartiles. Because this data served as a pre-cursor for the Giffords Scorecard, (Gun Violence Statistics, 2019) we used quartiles of state permissiveness using the Giffords Rank scale to estimate the association between permissiveness and total firearm deaths. We then compared this analysis to the original analysis to confirm that the results and conclusions were accurately replicated.
Although there is little variation in the scales from year to year (as states typically do not make major changes to their legislation that would impact their score, rank, or count of laws), we also examined the results from years 2015 and 2017.
A scatterplot illustrating the relationships between each pair of scores is shown in Fig. 1. These plots demonstrate the differences between scale types. For example, the scales that are scores are more likely to have clusters of states, while rankings do not.
The Spearman correlation coefficients between scales are shown in Fig. 2. Overall, the scores were strongly correlated with one another; the highest correlation (excluding the Giffords score vs. Gifford rank) was seen between the Everytown Count and the Siegel Count, and the Siegel Count and Giffords Rank (both at r = 0.89, p < 0.001). The scales that were the least correlated were the Cato Score and the Everytown Count (r = 0.65, p < 0.001); however, these scores would still be considered moderately to strongly correlated (Akoglu, 2018). The average correlation coefficient between scales was r = 0.77. Gun-rights and gun safety-oriented scores were no more or less correlated with one another. Gun-safety scores had an on average correlation coefficient of r = 0.86 with one another; gun-advocating sources had an average correlation coefficient of r = 0.75; and scores from opposite sources had an on average correlation coefficient of r = 0.74.
The Cronbach’s standardized alpha measuring the reliability of these scales is 0.968, and the Guttman’s lambda was 0.975, indicating that these scales were highly internally consistent. Table 2 presents the values for when a single scale is excluded from the calculation of the Cronbach’s alpha. There was no meaningful change in any of the Cronbach’s alphas, indicating again that the scores are consistently reliable.
A scree plot is presented in Fig. 3. The eigenvalue for the first factor was 5.72, and the second was 0.27, indicating that all scales load onto a single factor. The specific rotated factor loadings for each scale are presented in Table 2. These values ranged from 0.82 to 0.96, also indicating that these scales load heavily on a single, underlying factor.
The regression results using the methods from the Fleegler et al. paper, for both total firearm deaths and firearm suicides, are displayed in Table 3. All of the estimates for total firearm suicides are significant with similar magnitude. The lowest coefficient for the scale is 1.15 [95%CI: (1.04, 1.27)] for Giffords Rank, and the highest is 1.28 [95%CI: (1.16, 1.43)] for the Cato Score. Similar to the previous regression, all estimates for firearm suicides are significant with similar magnitude. Giffords Rank again has the lowest coefficient at 1.15 [95%CI: (1.04, 1.27)], and the Cato score is the highest at 1.32 [95%CI: (1.18, 1.46)]. The confidence intervals between all analyses are predominantly overlapping, and the conclusions would be the same for both outcomes regardless of which scale is used.
The replicated results from the regression using quartiles of the Giffords Rank scale, which are most similar to the scale used in the Fleegler et al. paper, are displayed with the results found in the aforementioned paper in Appendix Table 1. Although different years are examined, the coefficients and confidence intervals are nearly identical, indicating that we faithfully reproduced their methods.
All analyses produced similar findings for 2015 and 2017.
Through an examination of seven gun law permissiveness scales, each created using a different method (i.e. Score, Rank, and Count of Laws), we found that all of the scales are highly similar, even when comparing those more traditionally considered gun-rights vs. gun-safety scales.
Specifically, the scales are all highly correlated. We calculated measures of reliability (Cronbach’s alpha and Guttman’s lambda), and found that the scales are very reliable, which implies that one of the scales can be substituted for another. We also conducted a factor analysis, which showed that the scales all load onto a single factor. Finally, in our replication of a previous study, the choice of scale did not change the ultimate conclusion of the analysis.
There has been speculation that the different operationalizations of the scales, the different sources that the scales arise from, or the different laws that are included in the scales can result in meaningfully different conclusions. Our findings provide no evidence to support this criticism. This may be because states typically pass a series of laws that make their gun laws more restrictive or push legislation of all types that make their gun laws more permissive. Although a single law may not be included in the construction of the scales, the presence of other laws of similar type are highly indicative that the law would have been included. For example, states with stricter conceal carry requirements are also more likely to have stricter background check requires and child access prevention (CAP) laws (Gun Law Navigator, 2020).
The choice of scale for use in a given study as a composite measure of gun law strength may be immaterial. Nevertheless, practical considerations may necessitate selection of one scale over another, in part because the scales assessed here are available for different years. Ideally, a single scale would be chosen for all gun related research, as it would make it easier to compare results; otherwise, to generalize the findings to other research, the estimates would need to be standardized similarly to the way that was done here.
These scales also do not inform researchers what policies should be implemented, as they still represent a conglomerate of permissiveness, nor are they useful for regression discontinuity analysis, as there is little change from year to year in each state. Instead, these scales should be used to control for confounding by laws that are typically highly collinear. For example, an investigator might want to use the scale as a confounder in their analysis to adjust for the extent to which gun laws affect a specific relationship between an exposure and outcome. This is especially useful as there can be more than one hundred laws or policies enacted in a single state that would make individual control for each impossible due to issues of collinearity. Additionally, determining the reliability of these scales is important to address due to the criticism of past research by the academic and lay community.
A limitation of this work is that we did not include all possible scales in the analysis. We did, however, use scales that are calculated in different ways and from different sources, and have covered the most commonly used scales in firearm research to date. We also only primarily examined 2016, the most proximal year for which data from all scores were available. Regardless, we do not believe that the use of a different year would result in significantly different findings. A sensitivity analysis for 2015 and 2017 was performed (but excluded the Siegel Count and Cato Score as it was unavailable in 2017), and the results were the same across sources. Overall, the scales are relatively constant over across these years, with states like California and Massachusetts on one end of the spectrum, and Arizona and Alaska on the other end. Finally, we did not examine the extent to which the prevalence of gun ownership is correlated with the gun law permissiveness scales; however, previous research has found the two to be highly correlated,(Reeping et al., 2019) and gun ownership is likely contributing to the extent to which these scales load on a single factor.
In a comparison of seven scales, created with different methods (score, rank, and count of laws), and across gun-advocating and gun-safety organizations, all of the scales are highly correlated, reliable, and load onto a single factor. We do not find evidence that conclusions from analyses on gun violence related outcomes would differ meaningfully based on choice of scale.
Availability of data and materials
The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.
Adelmann B. Latest gun death Scorecard from Giffords is grossly misleading. The New American. Published 2019. https://www.thenewamerican.com/usnews/constitution/item/31452-latest-gun-death-scorecard-from-giffords-is-grossly-misleading.
Akoglu H. User’s guide to correlation coefficients. Turkish J Emerg Med. 2018;18(3):91–3.
American FactFinder. n.d.United States Census Bureau American Community Survey Office. 2016. https://factfinder.census.gov/faces/nav/jsf/pages/index.xhtml.
Annual Gun Law Scorecard. Giffords law center. Published 2020. https://lawcenter.giffords.org/scorecard/#rankings.
D’agostino RB Sr, Russell HK. Scree test. Encycl Biostat. 2005;7.
Dien J, Beal DJ, Berg P. Optimizing principal components analysis of event-related potentials: matrix type, factor loading weighting, extraction, and rotations. Clin Neurophysiol. 2005;116(8):1808–25.
Firearm Mortality Per State. CDC/National Center for Health Statistics. Published 2020. https://www.cdc.gov/nchs/pressroom/sosmap/firearm_mortality/firearm.htm.
Fleegler EW, Lee LK, Monuteaux MC, Hemenway D, Mannix R. Firearm legislation and firearm-related fatalities in the United States. JAMA Intern Med. 2013;173(9):732–40. https://doi.org/10.1001/jamainternmed.2013.1286.
Fowler KA, Dahlberg LL, Haileyesus T, Annest JL. Firearm injuries in the United States. Prev Med (Baltim). 2015;79:5–14.
Freedom in the 50 States: Gun Rights Rankings. The Cato Institute. Published 2020. https://www.freedominthe50states.org/guns.
Gun Law Navigator. Everytown for Gun Safety. Published 2020. https://everytownresearch.org/navigator/country.html.
Gun Violence Statistics. Published online 2019. https://lawcenter.giffords.org/facts/gun-violence-statistics/.
Hawkins A. Oops: Hypothesis for Latest Gun Control Study ‘Has Not Been Validated.’ Breitbart. Published 2019. https://www.breitbart.com/politics/2019/03/09/hypothesis-for-latest-gun-control-study-has-not-been-validated/.
Kappas J. Traveler’s guide to the firearm Laws of the fifty states (1997–2018): Traveler’s Guide, Inc; 2019.
Ledesma RD, Valero-Mora P, Macbeth G. The scree test and the number of factors: a dynamic graphics approach. Span J Psychol. 2015;18.
McClenathan J, Pahn M, Siegel M. The Changing Landscape of U.S. Gun Policy: State Firearm Laws, 1991–2016. Published online 2016. http://www.statefirearmlaws.org/.
Rao D, Zajecka J, Skubiak T. The modified rush sexual inventory: preliminary psychometric findings. Psychiatry Res. 2005;137(3):175–81.
Reeping PM, Cerdá M, Kalesan B, Wiebe DJ, Galea S, Branas CC. State gun laws, gun ownership, and mass shootings in the US: cross sectional time series. BMJ. 2019;364. https://doi.org/10.1136/bmj.l542.
Rummel RJ. Applied factor analysis: Northwestern University Press; 1988.
Santos JRA. Cronbach’s alpha: a tool for assessing the reliability of scales. J Ext. 1999;37(2):1–5.
Siegel M, Boine C. What are the most effective policies for reducing firearm homicide? Rockefeller Institute of Government. Published 2019. https://rockinst.org/wp-content/uploads/2019/03/3-28-19-Firearm-Laws-Homicide-Deaths-Brief.pdf.
Tuohy A. Debunking Anti-Gun Giffords Law Center’s “Gun Law Scorecard.” Omaha Outdoors. Published 2018. https://www.omahaoutdoors.com/blog/debunking-anti-gun-giffords-law-centers-gun-law-scorecard/.
Underlying Cause of Death 1999-2017 On CDC WONDER online database. Centers for Disease Control and Prevention National Center for Health Statistics. http://wonder.cdc.gov/ucd-icd10.html.
Webster DW, McCourt AD, Crifasi CK, Booty MD, Stuart EA. Evidence concerning the regulation of firearms design, sale, and carrying on fatal mass shootings in the United States. Criminol Public Policy. 2020;19(1):171–212.
Wood K. Best states for gun owners (2019). Guns and Ammo. Published 2019. https://www.gunsandammo.com/editorial/best-states-for-gun-owners/369075.
We would like to acknowledge Sarah McKetta for her input on this manuscript.
NCIPC 5R49CE003094–02 and NICHD 5R24HD087149–04.
Ethics approval and consent to participate
Consent for publication
The authors declare they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Reeping, P.M., Morrison, C.N., Rudolph, K.E. et al. A comparison and analysis of seven gun law permissiveness scales. Inj. Epidemiol. 8, 2 (2021). https://doi.org/10.1186/s40621-020-00296-5
- Gun violence
- Policy analysis
- Gun policy