-
The disruption index suffers from citation inflation and is confounded by shifts in scholarly citation practice
Authors:
Alexander M. Petersen,
Felber Arroyave,
Fabio Pammolli
Abstract:
Measuring the rate of innovation in academia and industry is fundamental to monitoring the efficiency and competitiveness of the knowledge economy. To this end, a disruption index (CD) was recently developed and applied to publication and patent citation networks (Wu et al., Nature 2019; Park et al., Nature 2023). Here we show that CD systematically decreases over time due to secular growth in res…
▽ More
Measuring the rate of innovation in academia and industry is fundamental to monitoring the efficiency and competitiveness of the knowledge economy. To this end, a disruption index (CD) was recently developed and applied to publication and patent citation networks (Wu et al., Nature 2019; Park et al., Nature 2023). Here we show that CD systematically decreases over time due to secular growth in research and patent production, following two distinct mechanisms unrelated to innovation -- one behavioral and the other structural. Whereas the behavioral explanation reflects shifts associated with techno-social factors (e.g. self-citation practices), the structural explanation follows from `citation inflation' (CI), an inextricable feature of real citation networks attributable to increasing reference list lengths, which causes CD to systematically decrease. We demonstrate this causal link by way of mathematical deduction, computational simulation, multi-variate regression, and quasi-experimental comparison of the disruptiveness of PNAS versus PNAS Plus articles, which differ only in their lengths. Accordingly, we analyze CD data available in the SciSciNet database and find that disruptiveness incrementally increased from 2005-2015, and that the negative relationship between disruption and team-size is remarkably small in overall magnitude effect size, and shifts from negative to positive for team size $\geq$ 8 coauthors.
△ Less
Submitted 21 June, 2024;
originally announced June 2024.
-
The disruption index is biased by citation inflation
Authors:
Alexander M. Petersen,
Felber Arroyave,
Fabio Pammolli
Abstract:
A recent analysis of scientific publication and patent citation networks by Park et al. (Nature, 2023) suggests that publications and patents are becoming less disruptive over time. Here we show that the reported decrease in disruptiveness is an artifact of systematic shifts in the structure of citation networks unrelated to innovation system capacity. Instead, the decline is attributable to 'cita…
▽ More
A recent analysis of scientific publication and patent citation networks by Park et al. (Nature, 2023) suggests that publications and patents are becoming less disruptive over time. Here we show that the reported decrease in disruptiveness is an artifact of systematic shifts in the structure of citation networks unrelated to innovation system capacity. Instead, the decline is attributable to 'citation inflation', an unavoidable characteristic of real citation networks that manifests as a systematic time-dependent bias and renders cross-temporal analysis challenging. One driver of citation inflation is the ever-increasing lengths of reference lists over time, which in turn increases the density of links in citation networks, and causes the disruption index to converge to 0. A second driver is attributable to shifts in the construction of reference lists, which is increasingly impacted by self-citations that increase in the rate of triadic closure in citation networks, and thus confounds efforts to measure disruption, which is itself a measure of triadic closure. Combined, these two systematic shifts render the disruption index temporally biased, and unsuitable for cross-temporal analysis. The impact of this systematic bias further stymies efforts to correlate disruption to other measures that are also time-dependent, such as team size and citation counts. In order to demonstrate this fundamental measurement problem, we present three complementary lines of critique (deductive, empirical and computational modeling), and also make available an ensemble of synthetic citation networks that can be used to test alternative citation-based indices for systematic bias.
△ Less
Submitted 2 June, 2023;
originally announced June 2023.
-
On the social and cognitive dimensions of wicked environmental problems characterized by conceptual and solution uncertainty
Authors:
Felber Arroyave,
Oscar Yandy Romero Goyeneche,
Meredith Gore,
Gaston Heimeriks,
Jeffrey Jenkins,
Alexander Petersen
Abstract:
We develop a quantitative framework for understanding the class of wicked problems that emerge at the intersections of natural, social, and technological complex systems. Wicked problems reflect our incomplete understanding of interdependent global systems and the systemic risk they pose; such problems escape solutions because they are often ill-defined, and thus mis-identified and under-appreciat…
▽ More
We develop a quantitative framework for understanding the class of wicked problems that emerge at the intersections of natural, social, and technological complex systems. Wicked problems reflect our incomplete understanding of interdependent global systems and the systemic risk they pose; such problems escape solutions because they are often ill-defined, and thus mis-identified and under-appreciated by communities of problem-solvers. While there are well-documented benefits to tackling boundary-crossing problems from various viewpoints, the integration of diverse approaches can nevertheless contribute confusion around the collective understanding of the core concepts and feasible solutions. We explore this paradox by analyzing the development of both scholarly (social) and topical (cognitive) communities -- two facets of knowledge production studies here that contribute towards the evolution of knowledge in and around a problem, termed a knowledge trajectory -- associated with three wicked problems: deforestation, invasive species, and wildlife trade. We posit that saturation in the dynamics of social and cognitive diversity growth is an indicator of reduced uncertainty in the evolution of the comprehensive knowledge trajectory emerging around each wicked problem. Informed by comprehensive bibliometric data capturing both social and cognitive dimensions of each problem domain, we thereby develop a framework that assesses the stability of knowledge trajectory dynamics as an indicator of wickedness associated with conceptual and solution uncertainty. As such, our results identify wildlife trade as a wicked problem that may be difficult to address given recent instability in its knowledge trajectory.
△ Less
Submitted 17 August, 2021; v1 submitted 20 April, 2021;
originally announced April 2021.
