-
Interpretation and inference for altmetric indicators arising from sparse data statistics
Authors:
Lawrence Smolinsky,
Bernhard Klingenberg,
Brian D. Marx
Abstract:
In 2018 Bornmann and Haunschild (2018a) introduced a new indicator called the Mantel-Haenszel quotient (MHq) to measure alternative metrics (or altmetrics) of scientometric data. In this article we review the Mantel-Haenszel statistics, point out two errors in the literature, and introduce a new indicator. First, we correct the interpretation of MHq and mention that it is still a meaningful indica…
▽ More
In 2018 Bornmann and Haunschild (2018a) introduced a new indicator called the Mantel-Haenszel quotient (MHq) to measure alternative metrics (or altmetrics) of scientometric data. In this article we review the Mantel-Haenszel statistics, point out two errors in the literature, and introduce a new indicator. First, we correct the interpretation of MHq and mention that it is still a meaningful indicator. Second, we correct the variance formula for MHq, which leads to narrower confidence intervals. A simulation study shows the superior performance of our variance estimator and confidence intervals. Since MHq does not match its original description in the literature, we propose a new indicator, the Mantel-Haenszel row risk ratio (MHRR), to meet that need. Interpretation and statistical inference for MHRR are discussed. For both MHRR and MHq, a value greater (less) than one means performance is better (worse) than in the reference set called the world.
△ Less
Submitted 26 January, 2022; v1 submitted 10 January, 2022;
originally announced January 2022.
-
Co-author weighting in bibliometric methodology and subfields of a scientific discipline
Authors:
Lawrence Smolinsky,
Aaron J. Lercher
Abstract:
Collaborative work and co-authorship are fundamental to the advancement of modern science. However, it is not clear how collaboration should be measured in achievement-based metrics. Co-author weighted credit introduces distortions into the bibliometric description of a discipline. It puts great weight on collaboration - not based on the results of collaboration - but purely because of the existen…
▽ More
Collaborative work and co-authorship are fundamental to the advancement of modern science. However, it is not clear how collaboration should be measured in achievement-based metrics. Co-author weighted credit introduces distortions into the bibliometric description of a discipline. It puts great weight on collaboration - not based on the results of collaboration - but purely because of the existence of collaborations. In terms of publication and citation impact, it artificially favors some subdisciplines. In order to understand how credit is given in a co-author weighted system (like the NRC's method), we introduced credit spaces. We include a study of the discipline of physics to illustrate the method. Indicators are introduced to measure the proportion of a credit space awarded to a subfield or a set of authors.
△ Less
Submitted 11 May, 2020;
originally announced May 2020.
-
Citations versus expert opinions: Citation analysis of Featured Reviews of the American Mathematical Society
Authors:
Lawrence Smolinsky,
Daniel S. Sage,
Aaron J. Lercher,
Aaron Cao
Abstract:
Peer review and citation metrics are two means of gauging the value of scientific research, but the lack of publicly available peer review data makes the comparison of these methods difficult. Mathematics can serve as a useful laboratory for considering these questions because as an exact science, there is a narrow range of reasons for citations. In mathematics, virtually all published articles ar…
▽ More
Peer review and citation metrics are two means of gauging the value of scientific research, but the lack of publicly available peer review data makes the comparison of these methods difficult. Mathematics can serve as a useful laboratory for considering these questions because as an exact science, there is a narrow range of reasons for citations. In mathematics, virtually all published articles are post-publication reviewed by mathematicians in Mathematical Reviews (MathSciNet) and so the data set was essentially the Web of Science mathematics publications from 1993 to 2004. For a decade, especially important articles were singled out in Mathematical Reviews for featured reviews. In this study, we analyze the bibliometrics of elite articles selected by peer review and by citation count. We conclude that the two notions of significance described by being a featured review article and being highly cited are distinct. This indicates that peer review and citation counts give largely independent determinations of highly distinguished articles. We also consider whether hiring patterns of subfields and mathematicians' interest in subfields reflect subfields of featured review or highly cited articles. We reexamine data from two earlier studies in light of our methods for implications on the peer review/citation count relationship to a diversity of disciplines.
△ Less
Submitted 16 December, 2020; v1 submitted 11 May, 2020;
originally announced May 2020.
-
Arbitrage opportunities in publication and ghost authors
Authors:
Lawrence Smolinsky
Abstract:
In some research evaluation systems, credit awarded to an article depends on the number of co-authors on the article with total credit to the article increasing with the number of co-authors. There are many examples of such evaluation systems (e.g., the United States National Research Council evaluation of graduate programs gave full credit to each co-author). Such credit systems run the risk of e…
▽ More
In some research evaluation systems, credit awarded to an article depends on the number of co-authors on the article with total credit to the article increasing with the number of co-authors. There are many examples of such evaluation systems (e.g., the United States National Research Council evaluation of graduate programs gave full credit to each co-author). Such credit systems run the risk of encouraging ghost or honorary authorships. In a recent article, Antonio Osorio and Lutz Bornmann (2019) propose a scheme to discourage ghost authorships but increase the total credit to a paper when co-authorships increase. It is shown that if articles are valued more highly as the number of co-authorships increases, then there are opportunities to increase credit by mutually agreeing to add each other as authors. Unrelated authors of unrelated papers may all benefit by expanding their co-author list. I call this phenomena arbitrage--a term borrowed from economics and finance--since the content of the articles do not change, but the value increases by moving to a "market" of more co-authors where articles are valued differently.
△ Less
Submitted 12 November, 2019;
originally announced November 2019.
-
Lotka's Inverse Square Law of Scientific Productivity: Its Methods and Statistics
Authors:
Stephen J. Bensman,
Lawrence J. Smolinsky
Abstract:
This brief communication analyzes the statistics and methods Lotka used to derive his inverse square law of scientific productivity from the standpoint of modern theory. It finds that he violated the norms of this theory by extremely truncating his data on the right. It also proves that Lotka himself played an important role in establishing the commonly used method of identifying power-law behavio…
▽ More
This brief communication analyzes the statistics and methods Lotka used to derive his inverse square law of scientific productivity from the standpoint of modern theory. It finds that he violated the norms of this theory by extremely truncating his data on the right. It also proves that Lotka himself played an important role in establishing the commonly used method of identifying power-law behavior by the R^2 fit to a regression line on a log-log plot that modern theory considers unreliable by basing the derivation of his law on this very method.
△ Less
Submitted 19 January, 2016;
originally announced January 2016.
-
Discrete power law with exponential cutoff and Lotka's Law
Authors:
Lawrence Smolinsky
Abstract:
The first bibliometric law appeared in Alfred J. Lotka's 1926 examination of author productivity in chemistry and physics. The result is that the productivity distribution is thought to be described by a power law. In this paper, Lotka's original data on author productivity in chemistry is reconsidered by comparing the fit of the data to both a discrete power law and a discrete power law with expo…
▽ More
The first bibliometric law appeared in Alfred J. Lotka's 1926 examination of author productivity in chemistry and physics. The result is that the productivity distribution is thought to be described by a power law. In this paper, Lotka's original data on author productivity in chemistry is reconsidered by comparing the fit of the data to both a discrete power law and a discrete power law with exponential cutoff.
△ Less
Submitted 29 December, 2015;
originally announced December 2015.
-
Power-law distributions, the h-index, and Google Scholar (GS) citations: a test of their relationship with economics Nobelists
Authors:
Stephen J. Bensman,
Alice Daugherty,
Lawrence J. Smolinsky,
Daniel S. Sage,
J. Sylvan Katz
Abstract:
This paper presents proof that Google Scholar (GS) can construct documentary sets relevant for evaluating researchers' works. Nobelists in economics were the researchers under analysis, and two types of tests of the GS cites to their works were performed: distributional and semantic. Distributional tests found that the GS cites to the laureates' works conformed to the power-law model with an asymp…
▽ More
This paper presents proof that Google Scholar (GS) can construct documentary sets relevant for evaluating researchers' works. Nobelists in economics were the researchers under analysis, and two types of tests of the GS cites to their works were performed: distributional and semantic. Distributional tests found that the GS cites to the laureates' works conformed to the power-law model with an asymptote or "tail" conterminous with their h-index demarcating their core oeuvre, validating both GS and the h-index. Semantic tests revealed that their works highest in GS cites were on topics for which they were awarded the prize.
△ Less
Submitted 4 November, 2014;
originally announced November 2014.
-
Comparison of the Research Effectiveness of Chemistry Nobelists and Fields Medalist Mathematicians with Google Scholar: the Yule-Simon Model
Authors:
Stephen J. Bensman,
Lawrence J. Smolinsky,
Daniel S. Sage
Abstract:
This paper uses the Yule-Simon model to estimate to what extent the work of chemistry Nobelists and Fields medalist mathematicians is incorporated into the knowledge corpus of their disciplines as measured by Google Scholar inlinks. Due to differences in the disciplines and prizes, it finds that the work of chemistry Nobelists is better incorporated than that of Fields medalists.
This paper uses the Yule-Simon model to estimate to what extent the work of chemistry Nobelists and Fields medalist mathematicians is incorporated into the knowledge corpus of their disciplines as measured by Google Scholar inlinks. Due to differences in the disciplines and prizes, it finds that the work of chemistry Nobelists is better incorporated than that of Fields medalists.
△ Less
Submitted 18 April, 2014;
originally announced April 2014.
