Showing 1–50 of 102 results for author: Kaneko, M

Social Bias Evaluation for Large Language Models Requires Prompt Variations

Authors: Rem Hida, Masahiro Kaneko, Naoaki Okazaki

Abstract: Warning: This paper contains examples of stereotypes and biases. Large Language Models (LLMs) exhibit considerable social biases, and various studies have tried to evaluate and mitigate these biases accurately. Previous studies use downstream tasks as prompts to examine the degree of social biases for evaluation and mitigation. While LLMs' output highly depends on prompts, previous studies evaluat… ▽ More Warning: This paper contains examples of stereotypes and biases. Large Language Models (LLMs) exhibit considerable social biases, and various studies have tried to evaluate and mitigate these biases accurately. Previous studies use downstream tasks as prompts to examine the degree of social biases for evaluation and mitigation. While LLMs' output highly depends on prompts, previous studies evaluating and mitigating bias have often relied on a limited variety of prompts. In this paper, we investigate the sensitivity of LLMs when changing prompt variations (task instruction and prompt, few-shot examples, debias-prompt) by analyzing task performance and social bias of LLMs. Our experimental results reveal that LLMs are highly sensitive to prompts to the extent that the ranking of LLMs fluctuates when comparing models for task performance and social bias. Additionally, we show that LLMs have tradeoffs between performance and social bias caused by the prompts. Less bias from prompt setting may result in reduced performance. Moreover, the ambiguity of instances is one of the reasons for this sensitivity to prompts in advanced LLMs, leading to various outputs. We recommend using diverse prompts, as in this study, to compare the effects of prompts on social bias in LLMs. △ Less

Submitted 3 July, 2024; originally announced July 2024.

Sampling-based Pseudo-Likelihood for Membership Inference Attacks

Authors: Masahiro Kaneko, Youmi Ma, Yuki Wata, Naoaki Okazaki

Abstract: Large Language Models (LLMs) are trained on large-scale web data, which makes it difficult to grasp the contribution of each text. This poses the risk of leaking inappropriate data such as benchmarks, personal information, and copyrighted texts in the training data. Membership Inference Attacks (MIA), which determine whether a given text is included in the model's training data, have been attracti… ▽ More Large Language Models (LLMs) are trained on large-scale web data, which makes it difficult to grasp the contribution of each text. This poses the risk of leaking inappropriate data such as benchmarks, personal information, and copyrighted texts in the training data. Membership Inference Attacks (MIA), which determine whether a given text is included in the model's training data, have been attracting attention. Previous studies of MIAs revealed that likelihood-based classification is effective for detecting leaks in LLMs. However, the existing methods cannot be applied to some proprietary models like ChatGPT or Claude 3 because the likelihood is unavailable to the user. In this study, we propose a Sampling-based Pseudo-Likelihood (\textbf{SPL}) method for MIA (\textbf{SaMIA}) that calculates SPL using only the text generated by an LLM to detect leaks. The SaMIA treats the target text as the reference text and multiple outputs from the LLM as text samples, calculates the degree of $n$-gram match as SPL, and determines the membership of the text in the training data. Even without likelihoods, SaMIA performed on par with existing likelihood-based methods. △ Less

Submitted 17 April, 2024; originally announced April 2024.

Millimeter-wave CO and SiO Observations toward the Broad-velocity-width Molecular Feature CO 16.134-0.553: a Smith cloud scenario?

Authors: Hiroki Yokozuka, Tomoharu Oka, Shiho Tsujimoto, Yuto Watanabe, Miyuki Kaneko

Abstract: We report the results of the CO $\textit J$=1-0 and SiO $\textit J$=2-1 map** observations towards the broad-velocity-width molecular feature CO 16.134-0.553 with the Nobeyama Radio Observatory 45 m telescope. The high quality CO map shows that the 5-pc size broad-velocity-width feature bridges two separate velocity components at $\textit V_{\rm{LSR}}$$\quad$$\simeq$ 40 km s$^{-1}$ and 65 km s… ▽ More We report the results of the CO $\textit J$=1-0 and SiO $\textit J$=2-1 map** observations towards the broad-velocity-width molecular feature CO 16.134-0.553 with the Nobeyama Radio Observatory 45 m telescope. The high quality CO map shows that the 5-pc size broad-velocity-width feature bridges two separate velocity components at $\textit V_{\rm{LSR}}$$\quad$$\simeq$ 40 km s$^{-1}$ and 65 km s$^{-1}$ in the position-velocity space. The kinetic power of CO 16.134-0.553 amounts to $7.8\times10^2$ $\textit L$$_\odot$ whereas no apparent driving sources were identified. Prominent SiO emission was detected from the broad-velocity-width feature and its root in the $\textit V_{\rm{LSR}}$$\quad$$\simeq$ 40 km s$^{-1}$ component. In the CO Galactic plane survey data, CO 16.134-0.553 appears to correspond to the Galactic eastern rim of a 15-pc diameter expanding CO shell. An $1°$-diameter H I emission void and $4°$-long vertical H I filament were also found above and below the CO shell, respectively. We propose that the high-velocity plunge of a dark matter subhalo with a clump of baryonic matter was responsible for the formation of the H I void, CO 16.134-0.553/CO shell, and the H I filament. △ Less

Submitted 15 April, 2024; originally announced April 2024.

Comments: 7 pages, 6 figures, 2 table, accepted for publication in ApJ

Journal ref: 2024ApJ...964...52Y

A Little Leak Will Sink a Great Ship: Survey of Transparency for Large Language Models from Start to Finish

Authors: Masahiro Kaneko, Timothy Baldwin

Abstract: Large Language Models (LLMs) are trained on massive web-crawled corpora. This poses risks of leakage, including personal information, copyrighted texts, and benchmark datasets. Such leakage leads to undermining human trust in AI due to potential unauthorized generation of content or overestimation of performance. We establish the following three criteria concerning the leakage issues: (1) leakage… ▽ More Large Language Models (LLMs) are trained on massive web-crawled corpora. This poses risks of leakage, including personal information, copyrighted texts, and benchmark datasets. Such leakage leads to undermining human trust in AI due to potential unauthorized generation of content or overestimation of performance. We establish the following three criteria concerning the leakage issues: (1) leakage rate: the proportion of leaked data in training data, (2) output rate: the ease of generating leaked data, and (3) detection rate: the detection performance of leaked versus non-leaked data. Despite the leakage rate being the origin of data leakage issues, it is not understood how it affects the output rate and detection rate. In this paper, we conduct an experimental survey to elucidate the relationship between the leakage rate and both the output rate and detection rate for personal information, copyrighted texts, and benchmark data. Additionally, we propose a self-detection approach that uses few-shot learning in which LLMs detect whether instances are present or absent in their training data, in contrast to previous methods that do not employ explicit learning. To explore the ease of generating leaked information, we create a dataset of prompts designed to elicit personal information, copyrighted text, and benchmarks from LLMs. Our experiments reveal that LLMs produce leaked information in most cases despite less such data in their training set. This indicates even small amounts of leaked data can greatly affect outputs. Our self-detection method showed superior performance compared to existing detection methods. △ Less

Submitted 24 March, 2024; originally announced March 2024.

PSHop: A Lightweight Feed-Forward Method for 3D Prostate Gland Segmentation

Authors: Yi**g Yang, Vasileios Magoulianitis, Jiaxin Yang, **tang Xue, Masatomo Kaneko, Giovanni Cacciamani, Andre Abreu, Vinay Duddalwar, C. -C. Jay Kuo, Inderbir S. Gill, Chrysostomos Nikias

Abstract: Automatic prostate segmentation is an important step in computer-aided diagnosis of prostate cancer and treatment planning. Existing methods of prostate segmentation are based on deep learning models which have a large size and lack of transparency which is essential for physicians. In this paper, a new data-driven 3D prostate segmentation method on MRI is proposed, named PSHop. Different from dee… ▽ More Automatic prostate segmentation is an important step in computer-aided diagnosis of prostate cancer and treatment planning. Existing methods of prostate segmentation are based on deep learning models which have a large size and lack of transparency which is essential for physicians. In this paper, a new data-driven 3D prostate segmentation method on MRI is proposed, named PSHop. Different from deep learning based methods, the core methodology of PSHop is a feed-forward encoder-decoder system based on successive subspace learning (SSL). It consists of two modules: 1) encoder: fine to coarse unsupervised representation learning with cascaded VoxelHop units, 2) decoder: coarse to fine segmentation prediction with voxel-wise classification and local refinement. Experiments are conducted on the publicly available ISBI-2013 dataset, as well as on a larger private one. Experimental analysis shows that our proposed PSHop is effective, robust and lightweight in the tasks of prostate gland and zonal segmentation, achieving a Dice Similarity Coefficient (DSC) of 0.873 for the gland segmentation task. PSHop achieves a competitive performance comparatively to other deep learning methods, while kee** the model size and inference complexity an order of magnitude smaller. △ Less

Submitted 23 March, 2024; originally announced March 2024.

Comments: 11 pages, 5 figures, 5 tables

PCa-RadHop: A Transparent and Lightweight Feed-forward Method for Clinically Significant Prostate Cancer Segmentation

Authors: Vasileios Magoulianitis, Jiaxin Yang, Yi**g Yang, **tang Xue, Masatomo Kaneko, Giovanni Cacciamani, Andre Abreu, Vinay Duddalwar, C. -C. Jay Kuo, Inderbir S. Gill, Chrysostomos Nikias

Abstract: Prostate Cancer is one of the most frequently occurring cancers in men, with a low survival rate if not early diagnosed. PI-RADS reading has a high false positive rate, thus increasing the diagnostic incurred costs and patient discomfort. Deep learning (DL) models achieve a high segmentation performance, although require a large model size and complexity. Also, DL models lack of feature interpreta… ▽ More Prostate Cancer is one of the most frequently occurring cancers in men, with a low survival rate if not early diagnosed. PI-RADS reading has a high false positive rate, thus increasing the diagnostic incurred costs and patient discomfort. Deep learning (DL) models achieve a high segmentation performance, although require a large model size and complexity. Also, DL models lack of feature interpretability and are perceived as ``black-boxes" in the medical field. PCa-RadHop pipeline is proposed in this work, aiming to provide a more transparent feature extraction process using a linear model. It adopts the recently introduced Green Learning (GL) paradigm, which offers a small model size and low complexity. PCa-RadHop consists of two stages: Stage-1 extracts data-driven radiomics features from the bi-parametric Magnetic Resonance Imaging (bp-MRI) input and predicts an initial heatmap. To reduce the false positive rate, a subsequent stage-2 is introduced to refine the predictions by including more contextual information and radiomics features from each already detected Region of Interest (ROI). Experiments on the largest publicly available dataset, PI-CAI, show a competitive performance standing of the proposed method among other deep DL models, achieving an area under the curve (AUC) of 0.807 among a cohort of 1,000 patients. Moreover, PCa-RadHop maintains orders of magnitude smaller model size and complexity. △ Less

Submitted 23 March, 2024; originally announced March 2024.

Comments: 13 pages, 7 figures, 5 tables

Robust Locomotion via Zero-order Stochastic Nonlinear Model Predictive Control with Guard Saltation Matrix

Authors: Sotaro Katayama, Noriaki Takasugi, Mitsuhisa Kaneko, Norio Nagatsuka, and Masaya Kinoshita

Abstract: This paper presents a stochastic/robust nonlinear model predictive control (NMPC) to enhance the robustness of legged locomotion against contact uncertainties. We integrate the contact uncertainties into the covariance propagation of stochastic/robust NMPC framework by leveraging the guard saltation matrix and an extended Kalman filter-like covariance update. We achieve fast stochastic/robust NMPC… ▽ More This paper presents a stochastic/robust nonlinear model predictive control (NMPC) to enhance the robustness of legged locomotion against contact uncertainties. We integrate the contact uncertainties into the covariance propagation of stochastic/robust NMPC framework by leveraging the guard saltation matrix and an extended Kalman filter-like covariance update. We achieve fast stochastic/robust NMPC computation by utilizing the zero-order stochastic/robust NMPC algorithm with additional improvements in computational efficiency concerning the feedback gains. We conducted numerical experiments and demonstrate that the proposed method can accurately forecast future state covariance and generate trajectories that satisfies constraints even in the presence of the contact uncertainties. Hardware experiments on the perceptive locomotion of a wheeled-legged robot were also carried out, validating the feasibility of the proposed method in a real-world system with limited on-board computation. △ Less

Submitted 21 March, 2024; originally announced March 2024.

Comments: 8 pages, 8 figures

Likelihood-based Mitigation of Evaluation Bias in Large Language Models

Authors: Masanari Ohi, Masahiro Kaneko, Ryuto Koike, Mengsay Loem, Naoaki Okazaki

Abstract: Large Language Models (LLMs) are widely used to evaluate natural language generation tasks as automated metrics. However, the likelihood, a measure of LLM's plausibility for a sentence, can vary due to superficial differences in sentences, such as word order and sentence structure. It is therefore possible that there might be a likelihood bias if LLMs are used for evaluation: they might overrate s… ▽ More Large Language Models (LLMs) are widely used to evaluate natural language generation tasks as automated metrics. However, the likelihood, a measure of LLM's plausibility for a sentence, can vary due to superficial differences in sentences, such as word order and sentence structure. It is therefore possible that there might be a likelihood bias if LLMs are used for evaluation: they might overrate sentences with higher likelihoods while underrating those with lower likelihoods. In this paper, we investigate the presence and impact of likelihood bias in LLM-based evaluators. We also propose a method to mitigate the likelihood bias. Our method utilizes highly biased instances as few-shot examples for in-context learning. Our experiments in evaluating the data-to-text and grammatical error correction tasks reveal that several LLMs we test display a likelihood bias. Furthermore, our proposed method successfully mitigates this bias, also improving evaluation performance (in terms of correlation of models with human scores) significantly. △ Less

Submitted 1 March, 2024; v1 submitted 24 February, 2024; originally announced February 2024.

Comments: 4 main pages

Eagle: Ethical Dataset Given from Real Interactions

Authors: Masahiro Kaneko, Danushka Bollegala, Timothy Baldwin

Abstract: Recent studies have demonstrated that large language models (LLMs) have ethical-related problems such as social biases, lack of moral reasoning, and generation of offensive content. The existing evaluation metrics and methods to address these ethical challenges use datasets intentionally created by instructing humans to create instances including ethical problems. Therefore, the data does not refl… ▽ More Recent studies have demonstrated that large language models (LLMs) have ethical-related problems such as social biases, lack of moral reasoning, and generation of offensive content. The existing evaluation metrics and methods to address these ethical challenges use datasets intentionally created by instructing humans to create instances including ethical problems. Therefore, the data does not reflect prompts that users actually provide when utilizing LLM services in everyday contexts. This may not lead to the development of safe LLMs that can address ethical challenges arising in real-world applications. In this paper, we create Eagle datasets extracted from real interactions between ChatGPT and users that exhibit social biases, toxicity, and immoral problems. Our experiments show that Eagle captures complementary aspects, not covered by existing datasets proposed for evaluation and mitigation of such ethical challenges. Our code is publicly available at https://huggingface.co/datasets/MasahiroKaneko/eagle. △ Less

Submitted 21 February, 2024; originally announced February 2024.

Evaluating Gender Bias in Large Language Models via Chain-of-Thought Prompting

Authors: Masahiro Kaneko, Danushka Bollegala, Naoaki Okazaki, Timothy Baldwin

Abstract: There exist both scalable tasks, like reading comprehension and fact-checking, where model performance improves with model size, and unscalable tasks, like arithmetic reasoning and symbolic reasoning, where model performance does not necessarily improve with model size. Large language models (LLMs) equipped with Chain-of-Thought (CoT) prompting are able to make accurate incremental predictions eve… ▽ More There exist both scalable tasks, like reading comprehension and fact-checking, where model performance improves with model size, and unscalable tasks, like arithmetic reasoning and symbolic reasoning, where model performance does not necessarily improve with model size. Large language models (LLMs) equipped with Chain-of-Thought (CoT) prompting are able to make accurate incremental predictions even on unscalable tasks. Unfortunately, despite their exceptional reasoning abilities, LLMs tend to internalize and reproduce discriminatory societal biases. Whether CoT can provide discriminatory or egalitarian rationalizations for the implicit information in unscalable tasks remains an open question. In this study, we examine the impact of LLMs' step-by-step predictions on gender bias in unscalable tasks. For this purpose, we construct a benchmark for an unscalable task where the LLM is given a list of words comprising feminine, masculine, and gendered occupational words, and is required to count the number of feminine and masculine words. In our CoT prompts, we require the LLM to explicitly indicate whether each word in the word list is a feminine or masculine before making the final predictions. With counting and handling the meaning of words, this benchmark has characteristics of both arithmetic reasoning and symbolic reasoning. Experimental results in English show that without step-by-step prediction, most LLMs make socially biased predictions, despite the task being as simple as counting words. Interestingly, CoT prompting reduces this unconscious social bias in LLMs and encourages fair predictions. △ Less

Submitted 28 January, 2024; originally announced January 2024.

On finite analogues of Euler's constant

Authors: Masanobu Kaneko, Toshiki Matsusaka, Shin-ichiro Seki

Abstract: We introduce and study finite analogues of Euler's constant in the same setting as finite multiple zeta values. We define a couple of candidate values from the perspectives of a ``regularized value of $ζ(1)$'' and of Mascheroni's and Kluyver--Nörlund's series expressions of Euler's constant using Gregory coefficients. Moreover, we reveal that the differences between them always lie in the… ▽ More We introduce and study finite analogues of Euler's constant in the same setting as finite multiple zeta values. We define a couple of candidate values from the perspectives of a ``regularized value of $ζ(1)$'' and of Mascheroni's and Kluyver--Nörlund's series expressions of Euler's constant using Gregory coefficients. Moreover, we reveal that the differences between them always lie in the $\mathbb{Q}$-vector space spanned by 1 and values of a finite analogue of logarithm at positive integers. △ Less

Submitted 18 January, 2024; originally announced January 2024.

Comments: 14 pages

MSC Class: 11M32; 11B68; 11A07

The Gaps between Pre-train and Downstream Settings in Bias Evaluation and Debiasing

Authors: Masahiro Kaneko, Danushka Bollegala, Timothy Baldwin

Abstract: The output tendencies of Pre-trained Language Models (PLM) vary markedly before and after Fine-Tuning (FT) due to the updates to the model parameters. These divergences in output tendencies result in a gap in the social biases of PLMs. For example, there exits a low correlation between intrinsic bias scores of a PLM and its extrinsic bias scores under FT-based debiasing methods. Additionally, appl… ▽ More The output tendencies of Pre-trained Language Models (PLM) vary markedly before and after Fine-Tuning (FT) due to the updates to the model parameters. These divergences in output tendencies result in a gap in the social biases of PLMs. For example, there exits a low correlation between intrinsic bias scores of a PLM and its extrinsic bias scores under FT-based debiasing methods. Additionally, applying FT-based debiasing methods to a PLM leads to a decline in performance in downstream tasks. On the other hand, PLMs trained on large datasets can learn without parameter updates via In-Context Learning (ICL) using prompts. ICL induces smaller changes to PLMs compared to FT-based debiasing methods. Therefore, we hypothesize that the gap observed in pre-trained and FT models does not hold true for debiasing methods that use ICL. In this study, we demonstrate that ICL-based debiasing methods show a higher correlation between intrinsic and extrinsic bias scores compared to FT-based methods. Moreover, the performance degradation due to debiasing is also lower in the ICL case compared to that in the FT case. △ Less

Submitted 16 January, 2024; originally announced January 2024.

Two formulas for certain double and multiple polylogarithms in two variables

Authors: Masanobu Kaneko, Hirofumi Tsumura

Abstract: We give a weighted sum formula for the double polylogarithm in two variables, from which we can recover the classical weighted sum formulas for double zeta values, double $T$-values, and some double $L$-values. Also presented is a connection-type formula for a two-variable multiple polylogarithm, which specializes to previously known single-variable formulas. This identity can also be regarded as… ▽ More We give a weighted sum formula for the double polylogarithm in two variables, from which we can recover the classical weighted sum formulas for double zeta values, double $T$-values, and some double $L$-values. Also presented is a connection-type formula for a two-variable multiple polylogarithm, which specializes to previously known single-variable formulas. This identity can also be regarded as a generalization of the renowned five-term relation for the dilogarithm. △ Less

Submitted 6 January, 2024; originally announced January 2024.

Comments: 11 pages

MSC Class: 11M32; 11M99

Versatile Telescopic-Wheeled-Legged Locomotion of Tachyon 3 via Full-Centroidal Nonlinear Model Predictive Control

Authors: Sotaro Katayama, Noriaki Takasugi, Mitsuhisa Kaneko, Masaya Kinoshita

Abstract: This paper presents a nonlinear model predictive control (NMPC) toward versatile motion generation for the telescopic-wheeled-legged robot Tachyon 3, the unique hardware structure of which poses challenges in control and motion planning. We apply the full-centroidal NMPC formulation with dedicated constraints that can capture the accurate kinematics and dynamics of Tachyon 3. We have developed a c… ▽ More This paper presents a nonlinear model predictive control (NMPC) toward versatile motion generation for the telescopic-wheeled-legged robot Tachyon 3, the unique hardware structure of which poses challenges in control and motion planning. We apply the full-centroidal NMPC formulation with dedicated constraints that can capture the accurate kinematics and dynamics of Tachyon 3. We have developed a control pipeline that includes an internal state integrator to apply NMPC to Tachyon 3, the actuators of which employ high-gain position-controllers. We conducted simulation and hardware experiments on the perceptive locomotion of Tachyon 3 over structured terrains and demonstrated that the proposed method can achieve smooth and dynamic motion generation under harsh physical and environmental constraints. △ Less

Submitted 14 December, 2023; originally announced December 2023.

Comments: 8 pages, 9 figures

Constraining nucleon effective masses with flow and stop** observables from the S$π$RIT experiment

Authors: C. Y. Tsang, M. Kurata-Nishimura, M. B. Tsang, W. G. Lynch, Y. X. Zhang, J. Barney, J. Estee, G. Jhang, R. Wang, M. Kaneko, J. W. Lee, T. Isobe, T. Murakami, D. S. Ahn, L. Atar, T. Aumann, H. Baba, K. Boretzky, J. Brzychczyk, G. Cerizza, N. Chiga, N. Fukuda, I. Gasparic, B. Hong, A. Horvat , et al. (30 additional authors not shown)

Abstract: Properties of the nuclear equation of state (EoS) can be probed by measuring the dynamical properties of nucleus-nucleus collisions. In this study, we present the directed flow ($v_1$), elliptic flow ($v_2$) and stop** (VarXZ) measured in fixed target Sn + Sn collisions at 270 AMeV with the S$π$RIT Time Projection Chamber. We perform Bayesian analyses in which EoS parameters are varied simultane… ▽ More Properties of the nuclear equation of state (EoS) can be probed by measuring the dynamical properties of nucleus-nucleus collisions. In this study, we present the directed flow ($v_1$), elliptic flow ($v_2$) and stop** (VarXZ) measured in fixed target Sn + Sn collisions at 270 AMeV with the S$π$RIT Time Projection Chamber. We perform Bayesian analyses in which EoS parameters are varied simultaneously within the Improved Quantum Molecular Dynamics-Skyrme (ImQMD-Sky) transport code to obtain a multivariate correlated constraint. The varied parameters include symmetry energy, $S_0$, and slope of the symmetry energy, $L$, at saturation density, isoscalar effective mass, $m_{s}^*/m_{N}$, isovector effective mass, $m_{v}^{*}/m_{N}$ and the in-medium cross-section enhancement factor $η$. We find that the flow and VarXZ observables are sensitive to the splitting of proton and neutron effective masses and the in-medium cross-section. Comparisons of ImQMD-Sky predictions to the S$π$RIT data suggest a narrow range of preferred values for $m_{s}^*/m_{N}$, $m_{v}^{*}/m_{N}$ and $η$. △ Less

Submitted 8 December, 2023; originally announced December 2023.

How You Prompt Matters! Even Task-Oriented Constraints in Instructions Affect LLM-Generated Text Detection

Authors: Ryuto Koike, Masahiro Kaneko, Naoaki Okazaki

Abstract: To combat the misuse of Large Language Models (LLMs), many recent studies have presented LLM-generated-text detectors with promising performance. When users instruct LLMs to generate texts, the instruction can include different constraints depending on the user's need. However, most recent studies do not cover such diverse instruction patterns when creating datasets for LLM detection. In this pape… ▽ More To combat the misuse of Large Language Models (LLMs), many recent studies have presented LLM-generated-text detectors with promising performance. When users instruct LLMs to generate texts, the instruction can include different constraints depending on the user's need. However, most recent studies do not cover such diverse instruction patterns when creating datasets for LLM detection. In this paper, we reveal that even task-oriented constraints -- constraints that would naturally be included in an instruction and are not related to detection-evasion -- cause existing powerful detectors to have a large variance in detection performance. We focus on student essay writing as a realistic domain and manually create task-oriented constraints based on several factors for essay quality. Our experiments show that the standard deviation (SD) of current detector performance on texts generated by an instruction with such a constraint is significantly larger (up to an SD of 14.4 F1-score) than that by generating texts multiple times or paraphrasing the instruction. We also observe an overall trend where the constraints can make LLM detection more challenging than without them. Finally, our analysis indicates that the high instruction-following ability of LLMs fosters the large impact of such constraints on detection performance. △ Less

Submitted 12 June, 2024; v1 submitted 14 November, 2023; originally announced November 2023.

Comments: under review

SAIE Framework: Support Alone Isn't Enough -- Advancing LLM Training with Adversarial Remarks

Authors: Mengsay Loem, Masahiro Kaneko, Naoaki Okazaki

Abstract: Large Language Models (LLMs) can justify or critique their predictions through discussions with other models or humans, thereby enriching their intrinsic understanding of instances. While proactive discussions in the inference phase have been shown to boost performance, such interactions have not been extensively explored during the training phase. We hypothesize that incorporating interactive dis… ▽ More Large Language Models (LLMs) can justify or critique their predictions through discussions with other models or humans, thereby enriching their intrinsic understanding of instances. While proactive discussions in the inference phase have been shown to boost performance, such interactions have not been extensively explored during the training phase. We hypothesize that incorporating interactive discussions into the training process can enhance the models' understanding and improve their reasoning and verbal expression abilities during inference. This work introduces the SAIE framework, which facilitates supportive and adversarial discussions between learner and partner models. The learner model receives responses from the partner, and its parameters are then updated based on this discussion. This dynamic adjustment process continues throughout the training phase, responding to the evolving outputs of the learner model. Our empirical evaluation across various tasks, including math problems, commonsense reasoning, and multi-domain knowledge, demonstrates that models fine-tuned with the SAIE framework outperform those trained with conventional fine-tuning approaches. Furthermore, our method enhances the models' reasoning capabilities, improving both individual and multi-agent inference performance. △ Less

Submitted 29 February, 2024; v1 submitted 14 November, 2023; originally announced November 2023.

Controlled Generation with Prompt Insertion for Natural Language Explanations in Grammatical Error Correction

Authors: Masahiro Kaneko, Naoaki Okazaki

Abstract: In Grammatical Error Correction (GEC), it is crucial to ensure the user's comprehension of a reason for correction. Existing studies present tokens, examples, and hints as to the basis for correction but do not directly explain the reasons for corrections. Although methods that use Large Language Models (LLMs) to provide direct explanations in natural language have been proposed for various tasks,… ▽ More In Grammatical Error Correction (GEC), it is crucial to ensure the user's comprehension of a reason for correction. Existing studies present tokens, examples, and hints as to the basis for correction but do not directly explain the reasons for corrections. Although methods that use Large Language Models (LLMs) to provide direct explanations in natural language have been proposed for various tasks, no such method exists for GEC. Generating explanations for GEC corrections involves aligning input and output tokens, identifying correction points, and presenting corresponding explanations consistently. However, it is not straightforward to specify a complex format to generate explanations, because explicit control of generation is difficult with prompts. This study introduces a method called controlled generation with Prompt Insertion (PI) so that LLMs can explain the reasons for corrections in natural language. In PI, LLMs first correct the input text, and then we automatically extract the correction points based on the rules. The extracted correction points are sequentially inserted into the LLM's explanation output as prompts, guiding the LLMs to generate explanations for the correction points. We also create an Explainable GEC (XGEC) dataset of correction reasons by annotating NUCLE, CoNLL2013, and CoNLL2014. Although generations from GPT-3 and ChatGPT using original prompts miss some correction points, the generation control using PI can explicitly guide to describe explanations for all correction points, contributing to improved performance in generating correction reasons. △ Less

Submitted 20 September, 2023; originally announced September 2023.

Comments: Work in progress

Evaluating Gender Bias of Pre-trained Language Models in Natural Language Inference by Considering All Labels

Authors: Panatchakorn Anantaprayoon, Masahiro Kaneko, Naoaki Okazaki

Abstract: Discriminatory gender biases have been found in Pre-trained Language Models (PLMs) for multiple languages. In Natural Language Inference (NLI), existing bias evaluation methods have focused on the prediction results of one specific label out of three labels, such as neutral. However, such evaluation methods can be inaccurate since unique biased inferences are associated with unique prediction labe… ▽ More Discriminatory gender biases have been found in Pre-trained Language Models (PLMs) for multiple languages. In Natural Language Inference (NLI), existing bias evaluation methods have focused on the prediction results of one specific label out of three labels, such as neutral. However, such evaluation methods can be inaccurate since unique biased inferences are associated with unique prediction labels. Addressing this limitation, we propose a bias evaluation method for PLMs, called NLI-CoAL, which considers all the three labels of NLI task. First, we create three evaluation data groups that represent different types of biases. Then, we define a bias measure based on the corresponding label output of each data group. In the experiments, we introduce a meta-evaluation technique for NLI bias measures and use it to confirm that our bias measure can distinguish biased, incorrect inferences from non-biased incorrect inferences better than the baseline, resulting in a more accurate bias evaluation. We create the datasets in English, Japanese, and Chinese, and successfully validate the compatibility of our bias measure across multiple languages. Lastly, we observe the bias tendencies in PLMs of different languages. To our knowledge, we are the first to construct evaluation datasets and measure PLMs' bias from NLI in Japanese and Chinese. △ Less

Submitted 18 May, 2024; v1 submitted 18 September, 2023; originally announced September 2023.

Comments: LREC-COLING 2024

The Impact of Debiasing on the Performance of Language Models in Downstream Tasks is Underestimated

Authors: Masahiro Kaneko, Danushka Bollegala, Naoaki Okazaki

Abstract: Pre-trained language models trained on large-scale data have learned serious levels of social biases. Consequently, various methods have been proposed to debias pre-trained models. Debiasing methods need to mitigate only discriminatory bias information from the pre-trained models, while retaining information that is useful for the downstream tasks. In previous research, whether useful information… ▽ More Pre-trained language models trained on large-scale data have learned serious levels of social biases. Consequently, various methods have been proposed to debias pre-trained models. Debiasing methods need to mitigate only discriminatory bias information from the pre-trained models, while retaining information that is useful for the downstream tasks. In previous research, whether useful information is retained has been confirmed by the performance of downstream tasks in debiased pre-trained models. On the other hand, it is not clear whether these benchmarks consist of data pertaining to social biases and are appropriate for investigating the impact of debiasing. For example in gender-related social biases, data containing female words (e.g. ``she, female, woman''), male words (e.g. ``he, male, man''), and stereotypical words (e.g. ``nurse, doctor, professor'') are considered to be the most affected by debiasing. If there is not much data containing these words in a benchmark dataset for a target task, there is the possibility of erroneously evaluating the effects of debiasing. In this study, we compare the impact of debiasing on performance across multiple downstream tasks using a wide-range of benchmark datasets that containing female, male, and stereotypical words. Experiments show that the effects of debiasing are consistently \emph{underestimated} across all tasks. Moreover, the effects of debiasing could be reliably evaluated by separately considering instances containing female, male, and stereotypical words than all of the instances in a benchmark dataset. △ Less

Submitted 16 September, 2023; originally announced September 2023.

Comments: IJCNLP-AACL 2023

In-Contextual Gender Bias Suppression for Large Language Models

Authors: Daisuke Oba, Masahiro Kaneko, Danushka Bollegala

Abstract: Despite their impressive performance in a wide range of NLP tasks, Large Language Models (LLMs) have been reported to encode worrying-levels of gender biases. Prior work has proposed debiasing methods that require human labelled examples, data augmentation and fine-tuning of LLMs, which are computationally costly. Moreover, one might not even have access to the model parameters for performing debi… ▽ More Despite their impressive performance in a wide range of NLP tasks, Large Language Models (LLMs) have been reported to encode worrying-levels of gender biases. Prior work has proposed debiasing methods that require human labelled examples, data augmentation and fine-tuning of LLMs, which are computationally costly. Moreover, one might not even have access to the model parameters for performing debiasing such as in the case of closed LLMs such as GPT-4. To address this challenge, we propose bias suppression that prevents biased generations of LLMs by simply providing textual preambles constructed from manually designed templates and real-world statistics, without accessing to model parameters. We show that, using CrowsPairs dataset, our textual preambles covering counterfactual statements can suppress gender biases in English LLMs such as LLaMA2. Moreover, we find that gender-neutral descriptions of gender-biased objects can also suppress their gender biases. Moreover, we show that bias suppression has acceptable adverse effect on downstream task performance with HellaSwag and COPA. △ Less

Submitted 20 February, 2024; v1 submitted 13 September, 2023; originally announced September 2023.

Comments: EACL 2024 Findings - Long Paper

OUTFOX: LLM-Generated Essay Detection Through In-Context Learning with Adversarially Generated Examples

Authors: Ryuto Koike, Masahiro Kaneko, Naoaki Okazaki

Abstract: Large Language Models (LLMs) have achieved human-level fluency in text generation, making it difficult to distinguish between human-written and LLM-generated texts. This poses a growing risk of misuse of LLMs and demands the development of detectors to identify LLM-generated texts. However, existing detectors lack robustness against attacks: they degrade detection accuracy by simply paraphrasing L… ▽ More Large Language Models (LLMs) have achieved human-level fluency in text generation, making it difficult to distinguish between human-written and LLM-generated texts. This poses a growing risk of misuse of LLMs and demands the development of detectors to identify LLM-generated texts. However, existing detectors lack robustness against attacks: they degrade detection accuracy by simply paraphrasing LLM-generated texts. Furthermore, a malicious user might attempt to deliberately evade the detectors based on detection results, but this has not been assumed in previous studies. In this paper, we propose OUTFOX, a framework that improves the robustness of LLM-generated-text detectors by allowing both the detector and the attacker to consider each other's output. In this framework, the attacker uses the detector's prediction labels as examples for in-context learning and adversarially generates essays that are harder to detect, while the detector uses the adversarially generated essays as examples for in-context learning to learn to detect essays from a strong attacker. Experiments in the domain of student essays show that the proposed detector improves the detection performance on the attacker-generated texts by up to +41.3 points F1-score. Furthermore, the proposed detector shows a state-of-the-art detection performance: up to 96.9 points F1-score, beating existing detectors on non-attacked texts. Finally, the proposed attacker drastically degrades the performance of detectors by up to -57.0 points F1-score, massively outperforming the baseline paraphrasing method for evading detection. △ Less

Submitted 18 February, 2024; v1 submitted 21 July, 2023; originally announced July 2023.

Comments: AAAI 2024 camera ready. Code and dataset available at https://github.com/ryuryukke/OUTFOX

Exploring Effectiveness of GPT-3 in Grammatical Error Correction: A Study on Performance and Controllability in Prompt-Based Methods

Authors: Mengsay Loem, Masahiro Kaneko, Sho Takase, Naoaki Okazaki

Abstract: Large-scale pre-trained language models such as GPT-3 have shown remarkable performance across various natural language processing tasks. However, applying prompt-based methods with GPT-3 for Grammatical Error Correction (GEC) tasks and their controllability remains underexplored. Controllability in GEC is crucial for real-world applications, particularly in educational settings, where the ability… ▽ More Large-scale pre-trained language models such as GPT-3 have shown remarkable performance across various natural language processing tasks. However, applying prompt-based methods with GPT-3 for Grammatical Error Correction (GEC) tasks and their controllability remains underexplored. Controllability in GEC is crucial for real-world applications, particularly in educational settings, where the ability to tailor feedback according to learner levels and specific error types can significantly enhance the learning process. This paper investigates the performance and controllability of prompt-based methods with GPT-3 for GEC tasks using zero-shot and few-shot setting. We explore the impact of task instructions and examples on GPT-3's output, focusing on controlling aspects such as minimal edits, fluency edits, and learner levels. Our findings demonstrate that GPT-3 could effectively perform GEC tasks, outperforming existing supervised and unsupervised approaches. We also showed that GPT-3 could achieve controllability when appropriate task instructions and examples are given. △ Less

Submitted 29 May, 2023; originally announced May 2023.

Comments: Accepted in BEA 2023

Reducing Sequence Length by Predicting Edit Operations with Large Language Models

Authors: Masahiro Kaneko, Naoaki Okazaki

Abstract: Large Language Models (LLMs) have demonstrated remarkable performance in various tasks and gained significant attention. LLMs are also used for local sequence transduction tasks, including grammatical error correction (GEC) and formality style transfer, where most tokens in a source text are kept unchanged. However, the models that generate all target tokens in such tasks have a tendency to simply… ▽ More Large Language Models (LLMs) have demonstrated remarkable performance in various tasks and gained significant attention. LLMs are also used for local sequence transduction tasks, including grammatical error correction (GEC) and formality style transfer, where most tokens in a source text are kept unchanged. However, the models that generate all target tokens in such tasks have a tendency to simply copy the input text as is, without making needed changes, because the difference between input and output texts is minimal in the training data. This is also inefficient because the computational cost grows quadratically with the target sequence length with Transformer. This paper proposes predicting edit spans for the source text for local sequence transduction tasks. Representing an edit span with a position of the source text and corrected tokens, we can reduce the length of the target sequence and the computational cost for inference. We apply instruction tuning for LLMs on the supervision data of edit spans. Experiments show that the proposed method achieves comparable performance to the baseline in four tasks, paraphrasing, formality style transfer, GEC, and text simplification, despite reducing the length of the target text by as small as 21%. Furthermore, we report that the task-specific fine-tuning with the proposed method achieved state-of-the-art performance in the four tasks. △ Less

Submitted 20 October, 2023; v1 submitted 19 May, 2023; originally announced May 2023.

Comments: EMNLP2023

Solving NLP Problems through Human-System Collaboration: A Discussion-based Approach

Authors: Masahiro Kaneko, Graham Neubig, Naoaki Okazaki

Abstract: Humans work together to solve common problems by having discussions, explaining, and agreeing or disagreeing with each other. Similarly, if a system can have discussions with humans when solving tasks, it can improve the system's performance and reliability. In previous research on explainability, it has only been possible for the system to make predictions and for humans to ask questions about th… ▽ More Humans work together to solve common problems by having discussions, explaining, and agreeing or disagreeing with each other. Similarly, if a system can have discussions with humans when solving tasks, it can improve the system's performance and reliability. In previous research on explainability, it has only been possible for the system to make predictions and for humans to ask questions about them rather than having a mutual exchange of opinions. This research aims to create a dataset and computational framework for systems that discuss and refine their predictions through dialogue. Through experiments, we show that the proposed system can have beneficial discussions with humans improving the accuracy by up to 25 points in the natural language inference task. △ Less

Submitted 30 January, 2024; v1 submitted 19 May, 2023; originally announced May 2023.

Comments: EACL2024 Findings

Comparing Intrinsic Gender Bias Evaluation Measures without using Human Annotated Examples

Authors: Masahiro Kaneko, Danushka Bollegala, Naoaki Okazaki

Abstract: Numerous types of social biases have been identified in pre-trained language models (PLMs), and various intrinsic bias evaluation measures have been proposed for quantifying those social biases. Prior works have relied on human annotated examples to compare existing intrinsic bias evaluation measures. However, this approach is not easily adaptable to different languages nor amenable to large scale… ▽ More Numerous types of social biases have been identified in pre-trained language models (PLMs), and various intrinsic bias evaluation measures have been proposed for quantifying those social biases. Prior works have relied on human annotated examples to compare existing intrinsic bias evaluation measures. However, this approach is not easily adaptable to different languages nor amenable to large scale evaluations due to the costs and difficulties when recruiting human annotators. To overcome this limitation, we propose a method to compare intrinsic gender bias evaluation measures without relying on human-annotated examples. Specifically, we create multiple bias-controlled versions of PLMs using varying amounts of male vs. female gendered sentences, mined automatically from an unannotated corpus using gender-related word lists. Next, each bias-controlled PLM is evaluated using an intrinsic bias evaluation measure, and the rank correlation between the computed bias scores and the gender proportions used to fine-tune the PLMs is computed. Experiments on multiple corpora and PLMs repeatedly show that the correlations reported by our proposed method that does not require human annotated examples are comparable to those computed using human annotated examples in prior work. △ Less

Submitted 27 January, 2023; originally announced January 2023.

Comments: EACL 2023

Discovery of the Tadpole Molecular Cloud near the Galactic Nucleus

Authors: Miyuki Kaneko, Tomoharu Oka, Hiroki Yokozuka, Rei Enokiya, Shunya Takekawa, Yuhei Iwata, Shiho Tsujimoto

Abstract: In this paper, we report the discovery of an isolated, peculiar compact cloud with a steep velocity gradient at $2\farcm 6$ northwest of Sgr A*. This ``Tadpole'' molecular cloud is unique owing to its characteristic head-tail structure in the position-velocity space. By tracing the CO {\it J}=3--2 intensity peak in each velocity channel, we noticed that the kinematics of the Tadpole can be well re… ▽ More In this paper, we report the discovery of an isolated, peculiar compact cloud with a steep velocity gradient at $2\farcm 6$ northwest of Sgr A*. This ``Tadpole'' molecular cloud is unique owing to its characteristic head-tail structure in the position-velocity space. By tracing the CO {\it J}=3--2 intensity peak in each velocity channel, we noticed that the kinematics of the Tadpole can be well reproduced by a Keplerian motion around a point-like object with a mass of $1\!\times\! 10^{5}\,M_{\odot}$. Changes in line intensity ratios along the orbit are consistent with the Keplerian orbit model. The spatial compactness of the Tadpole and absence of bright counterparts in other wavelengths indicate that the object could be an intermediate-mass black hole. △ Less

Submitted 12 January, 2023; originally announced January 2023.

Comments: 10 pages, 9 figures, 2 tables, accepted for publication on the Astrophysical Journal

Journal ref: The Astrophysical Journal, January 10, 2023, 942, 46

Design guidelines for the SPICE parameters of waveform-selective metasurfaces varying with the incident pulse width at a constant oscillation frequency

Authors: Shiori Imai, Haruki Homma, Kairi Takimoto, Mizuki Tanikawa, ** Nakamura, Masaya Kaneko, Yuya Osaki, Kiichi Niitsu, Cheng Yongzhi, Ashif Aminulloh Fathnan, Hiroki Wakatsuchi

Abstract: In this study, we numerically demonstrate how the response of recently reported circuit-based metasurfaces is characterized by their circuit parameters. These metasurfaces, which include a set of four diodes as a full wave rectifier, are capable of sensing different waves even at the same frequency in response to the incident waveform, or more specifically the pulse width. This study reveals the r… ▽ More In this study, we numerically demonstrate how the response of recently reported circuit-based metasurfaces is characterized by their circuit parameters. These metasurfaces, which include a set of four diodes as a full wave rectifier, are capable of sensing different waves even at the same frequency in response to the incident waveform, or more specifically the pulse width. This study reveals the relationship between the electromagnetic response of such waveform-selective metasurfaces and the SPICE parameters of the diodes used. First, we show that reducing a parasitic capacitive component of the diodes is important for realization of waveform-selective metasurfaces in a higher frequency regime. Second, we report that the operating power level is closely related to the saturation current and the breakdown voltage of the diodes. Moreover, the operating power range is found to be broadened by introducing an additional resistor into the inside of the diode bridge. Our study is expected to provide design guidelines for circuit-based waveform-selective metasurfaces to select/fabricate optimal diodes and enhance the waveform-selective performance at the target frequency and power level. △ Less

Submitted 25 December, 2022; originally announced December 2022.

Comments: 9 pages, 9 figures

Isoscaling in central Sn+Sn collisions at 270 MeV/u

Authors: J. W. Lee, M. B. Tsang, C. Y. Tsang, R. Wang, J. Barney, J. Estee, T. Isobe, M. Kaneko, M. Kurata-Nishimura, W. G. Lynch, T. Murakami, A. Ono, S. R. Souza, D. S. Ahn, L. Atar, T. Aumann, H. Baba, K. Boretzky, J. Brzychczyk, G. Cerizza, N. Chiga, N. Fukuda, I. Gasparic, B. Hong, A. Horvat , et al. (39 additional authors not shown)

Abstract: Experimental information on fragment emissions is important in understanding the dynamics of nuclear collisions and in the development of transport model simulating heavy-ion collisions. The composition of complex fragments emitted in the heavy-ion collisions can be explained by statistical models, which assume that thermal equilibrium is achieved at collision energies below 100 MeV/u. Our new exp… ▽ More Experimental information on fragment emissions is important in understanding the dynamics of nuclear collisions and in the development of transport model simulating heavy-ion collisions. The composition of complex fragments emitted in the heavy-ion collisions can be explained by statistical models, which assume that thermal equilibrium is achieved at collision energies below 100 MeV/u. Our new experimental data together with theoretical analyses for light particles from Sn+Sn collisions at 270 MeV/u, suggest that the hypothesis of thermal equilibrium breaks down for particles emitted with high transfer momentum. To inspect the system's properties in such limit, the scaling features of the yield ratios of particles from two systems, a neutron-rich system of ${}^{132}\mathrm{Sn}+{}^{124}\mathrm{Sn}$ and a nearly symmetric system of ${}^{108}\mathrm{Sn}+{}^{112}\mathrm{Sn}$, are examined in the framework of the statistical multifragmentation model and the antisymmetrized molecular dynamics model. The isoscaling from low energy particles agree with both models. However the observed breakdown of isoscaling for particles with high transverse momentum cannot be explained by the antisymmetrized molecular dynamics model. △ Less

Submitted 5 November, 2022; originally announced November 2022.

Journal ref: The European Physical Journal A volume 58, Article number: 201 (2022)

Debiasing isn't enough! -- On the Effectiveness of Debiasing MLMs and their Social Biases in Downstream Tasks

Authors: Masahiro Kaneko, Danushka Bollegala, Naoaki Okazaki

Abstract: We study the relationship between task-agnostic intrinsic and task-specific extrinsic social bias evaluation measures for Masked Language Models (MLMs), and find that there exists only a weak correlation between these two types of evaluation measures. Moreover, we find that MLMs debiased using different methods still re-learn social biases during fine-tuning on downstream tasks. We identify the so… ▽ More We study the relationship between task-agnostic intrinsic and task-specific extrinsic social bias evaluation measures for Masked Language Models (MLMs), and find that there exists only a weak correlation between these two types of evaluation measures. Moreover, we find that MLMs debiased using different methods still re-learn social biases during fine-tuning on downstream tasks. We identify the social biases in both training instances as well as their assigned labels as reasons for the discrepancy between intrinsic and extrinsic bias evaluation measurements. Overall, our findings highlight the limitations of existing MLM bias evaluation measures and raise concerns on the deployment of MLMs in downstream applications using those measures. △ Less

Submitted 6 October, 2022; originally announced October 2022.

Comments: COLING 2022

Parametric Apéry-type Series and Hurwitz-type Multiple Zeta Values

Authors: Masanobu Kaneko, Wei** Wang, Ce Xu, Jianqiang Zhao

Abstract: In this paper, we will establish many explicit relations between parametric Apéry-type series involving one or two parametric binomial coefficients and Hurwitz-type multiple zeta values (with $r$-variables) by using the method of iterated integral. Furthermore, we also establish some new identities of integrals involving multiple polylogarithm functions and Kaneko--Tsumura {\rm A}-functions in ter… ▽ More In this paper, we will establish many explicit relations between parametric Apéry-type series involving one or two parametric binomial coefficients and Hurwitz-type multiple zeta values (with $r$-variables) by using the method of iterated integral. Furthermore, we also establish some new identities of integrals involving multiple polylogarithm functions and Kaneko--Tsumura {\rm A}-functions in terms of Hurwitz-type multiple zeta values. Then, using these formulas obtained, we obtain the explicit formulas of Arakawa-Kaneko zeta values and Kaneko-Tsumura $ψ$-values. Moreover, we can find a symmetric formula of Kaneko-Tsumura's conjecture about double $T$-values. Finally, we define and discuss the Hurwitz-type multiple zeta values of parametric binomial coefficient. △ Less

Submitted 18 September, 2022; v1 submitted 12 September, 2022; originally announced September 2022.

Comments: 33 pages

Multiple $L$-values of level four, poly-Euler numbers, and related zeta functions

Authors: Masanobu Kaneko, Hirofumi Tsumura

Abstract: We present several formulas for some specific multiple $L$-values of conductor four. This grew out from the study of zeta functions of level four of Arakawa-Kaneko type. Closely related is a new version of multiple poly-Euler numbers and we briefly discuss this too. We present several formulas for some specific multiple $L$-values of conductor four. This grew out from the study of zeta functions of level four of Arakawa-Kaneko type. Closely related is a new version of multiple poly-Euler numbers and we briefly discuss this too. △ Less

Submitted 10 August, 2022; originally announced August 2022.

Comments: 28pages

MSC Class: Primary 11M32; Secondary 11B68; 11M41

Are Neighbors Enough? Multi-Head Neural n-gram can be Alternative to Self-attention

Authors: Mengsay Loem, Sho Takase, Masahiro Kaneko, Naoaki Okazaki

Abstract: Impressive performance of Transformer has been attributed to self-attention, where dependencies between entire input in a sequence are considered at every position. In this work, we reform the neural $n$-gram model, which focuses on only several surrounding representations of each position, with the multi-head mechanism as in Vaswani et al.(2017). Through experiments on sequence-to-sequence tasks,… ▽ More Impressive performance of Transformer has been attributed to self-attention, where dependencies between entire input in a sequence are considered at every position. In this work, we reform the neural $n$-gram model, which focuses on only several surrounding representations of each position, with the multi-head mechanism as in Vaswani et al.(2017). Through experiments on sequence-to-sequence tasks, we show that replacing self-attention in Transformer with multi-head neural $n$-gram can achieve comparable or better performance than Transformer. From various analyses on our proposed method, we find that multi-head neural $n$-gram is complementary to self-attention, and their combinations can further improve performance of vanilla Transformer. △ Less

Submitted 27 July, 2022; originally announced July 2022.

Gender Bias in Meta-Embeddings

Authors: Masahiro Kaneko, Danushka Bollegala, Naoaki Okazaki

Abstract: Different methods have been proposed to develop meta-embeddings from a given set of source embeddings. However, the source embeddings can contain unfair gender-related biases, and how these influence the meta-embeddings has not been studied yet. We study the gender bias in meta-embeddings created under three different settings: (1) meta-embedding multiple sources without performing any debiasing (… ▽ More Different methods have been proposed to develop meta-embeddings from a given set of source embeddings. However, the source embeddings can contain unfair gender-related biases, and how these influence the meta-embeddings has not been studied yet. We study the gender bias in meta-embeddings created under three different settings: (1) meta-embedding multiple sources without performing any debiasing (Multi-Source No-Debiasing), (2) meta-embedding multiple sources debiased by a single method (Multi-Source Single-Debiasing), and (3) meta-embedding a single source debiased by different methods (Single-Source Multi-Debiasing). Our experimental results show that meta-embedding amplifies the gender biases compared to input source embeddings. We find that debiasing not only the sources but also their meta-embedding is needed to mitigate those biases. Moreover, we propose a novel debiasing method based on meta-embedding learning where we use multiple debiasing methods on a single source embedding and then create a single unbiased meta-embedding. △ Less

Submitted 6 October, 2022; v1 submitted 19 May, 2022; originally announced May 2022.

Comments: Findings of EMNLP 2022

Gender Bias in Masked Language Models for Multiple Languages

Authors: Masahiro Kaneko, Aizhan Imankulova, Danushka Bollegala, Naoaki Okazaki

Abstract: Masked Language Models (MLMs) pre-trained by predicting masked tokens on large corpora have been used successfully in natural language processing tasks for a variety of languages. Unfortunately, it was reported that MLMs also learn discriminative biases regarding attributes such as gender and race. Because most studies have focused on MLMs in English, the bias of MLMs in other languages has rarely… ▽ More Masked Language Models (MLMs) pre-trained by predicting masked tokens on large corpora have been used successfully in natural language processing tasks for a variety of languages. Unfortunately, it was reported that MLMs also learn discriminative biases regarding attributes such as gender and race. Because most studies have focused on MLMs in English, the bias of MLMs in other languages has rarely been investigated. Manual annotation of evaluation data for languages other than English has been challenging due to the cost and difficulty in recruiting annotators. Moreover, the existing bias evaluation methods require the stereotypical sentence pairs consisting of the same context with attribute words (e.g. He/She is a nurse). We propose Multilingual Bias Evaluation (MBE) score, to evaluate bias in various languages using only English attribute word lists and parallel corpora between the target language and English without requiring manually annotated data. We evaluated MLMs in eight languages using the MBE and confirmed that gender-related biases are encoded in MLMs for all those languages. We manually created datasets for gender bias in Japanese and Russian to evaluate the validity of the MBE. The results show that the bias scores reported by the MBE significantly correlates with that computed from the above manually created datasets and the existing English datasets for gender bias. △ Less

Submitted 4 May, 2022; v1 submitted 1 May, 2022; originally announced May 2022.

Comments: NAACL 2022

Sense Embeddings are also Biased--Evaluating Social Biases in Static and Contextualised Sense Embeddings

Authors: Yi Zhou, Masahiro Kaneko, Danushka Bollegala

Abstract: Sense embedding learning methods learn different embeddings for the different senses of an ambiguous word. One sense of an ambiguous word might be socially biased while its other senses remain unbiased. In comparison to the numerous prior work evaluating the social biases in pretrained word embeddings, the biases in sense embeddings have been relatively understudied. We create a benchmark dataset… ▽ More Sense embedding learning methods learn different embeddings for the different senses of an ambiguous word. One sense of an ambiguous word might be socially biased while its other senses remain unbiased. In comparison to the numerous prior work evaluating the social biases in pretrained word embeddings, the biases in sense embeddings have been relatively understudied. We create a benchmark dataset for evaluating the social biases in sense embeddings and propose novel sense-specific bias evaluation measures. We conduct an extensive evaluation of multiple static and contextualised sense embeddings for various types of social biases using the proposed measures. Our experimental results show that even in cases where no biases are found at word-level, there still exist worrying levels of social biases at sense-level, which are often ignored by the word-level bias evaluation measures. △ Less

Submitted 16 March, 2022; v1 submitted 14 March, 2022; originally announced March 2022.

Comments: Accepted to ACL 2022

Interpretability for Language Learners Using Example-Based Grammatical Error Correction

Authors: Masahiro Kaneko, Sho Takase, Ayana Niwa, Naoaki Okazaki

Abstract: Grammatical Error Correction (GEC) should not focus only on high accuracy of corrections but also on interpretability for language learning. However, existing neural-based GEC models mainly aim at improving accuracy, and their interpretability has not been explored. A promising approach for improving interpretability is an example-based method, which uses similar retrieved examples to generate cor… ▽ More Grammatical Error Correction (GEC) should not focus only on high accuracy of corrections but also on interpretability for language learning. However, existing neural-based GEC models mainly aim at improving accuracy, and their interpretability has not been explored. A promising approach for improving interpretability is an example-based method, which uses similar retrieved examples to generate corrections. In addition, examples are beneficial in language learning, hel** learners understand the basis of grammatically incorrect/correct texts and improve their confidence in writing. Therefore, we hypothesize that incorporating an example-based method into GEC can improve interpretability as well as support language learners. In this study, we introduce an Example-Based GEC (EB-GEC) that presents examples to language learners as a basis for a correction result. The examples consist of pairs of correct and incorrect sentences similar to a given input and its predicted correction. Experiments demonstrate that the examples presented by EB-GEC help language learners decide to accept or refuse suggestions from the GEC output. Furthermore, the experiments also show that retrieved examples improve the accuracy of corrections. △ Less

Submitted 14 March, 2022; originally announced March 2022.

Comments: ACL 2022

Proficiency Matters Quality Estimation in Grammatical Error Correction

Authors: Yu** Takahashi, Masahiro Kaneko, Masato Mita, Mamoru Komachi

Abstract: This study investigates how supervised quality estimation (QE) models of grammatical error correction (GEC) are affected by the learners' proficiency with the data. QE models for GEC evaluations in prior work have obtained a high correlation with manual evaluations. However, when functioning in a real-world context, the data used for the reported results have limitations because prior works were b… ▽ More This study investigates how supervised quality estimation (QE) models of grammatical error correction (GEC) are affected by the learners' proficiency with the data. QE models for GEC evaluations in prior work have obtained a high correlation with manual evaluations. However, when functioning in a real-world context, the data used for the reported results have limitations because prior works were biased toward data by learners with relatively high proficiency levels. To address this issue, we created a QE dataset that includes multiple proficiency levels and explored the necessity of performing proficiency-wise evaluation for QE of GEC. Our experiments demonstrated that differences in evaluation dataset proficiency affect the performance of QE models, and proficiency-wise evaluation helps create more robust models. △ Less

Submitted 16 January, 2022; originally announced January 2022.

Comments: 6 pages (4 pages + references)

ExtraPhrase: Efficient Data Augmentation for Abstractive Summarization

Authors: Mengsay Loem, Sho Takase, Masahiro Kaneko, Naoaki Okazaki

Abstract: Neural models trained with large amount of parallel data have achieved impressive performance in abstractive summarization tasks. However, large-scale parallel corpora are expensive and challenging to construct. In this work, we introduce a low-cost and effective strategy, ExtraPhrase, to augment training data for abstractive summarization tasks. ExtraPhrase constructs pseudo training data in two… ▽ More Neural models trained with large amount of parallel data have achieved impressive performance in abstractive summarization tasks. However, large-scale parallel corpora are expensive and challenging to construct. In this work, we introduce a low-cost and effective strategy, ExtraPhrase, to augment training data for abstractive summarization tasks. ExtraPhrase constructs pseudo training data in two steps: extractive summarization and paraphrasing. We extract major parts of an input text in the extractive summarization step, and obtain its diverse expressions with the paraphrasing step. Through experiments, we show that ExtraPhrase improves the performance of abstractive summarization tasks by more than 0.50 points in ROUGE scores compared to the setting without data augmentation. ExtraPhrase also outperforms existing methods such as back-translation and self-training. We also show that ExtraPhrase is significantly effective when the amount of genuine training data is remarkably small, i.e., a low-resource setting. Moreover, ExtraPhrase is more cost-efficient than the existing approaches. △ Less

Submitted 14 January, 2022; originally announced January 2022.

On finite multiple zeta values of level two

Authors: Masanobu Kaneko, Takuya Murakami, Amane Yoshihara

Abstract: We introduce and study a ``level two'' analogue of finite multiple zeta values. We give conjectural bases of the space of finite Euler sums as well as that of usual finite multiple zeta values in terms of these newly defined elements. A kind of ``parity result'' and certain sum formulas are also presented. We introduce and study a ``level two'' analogue of finite multiple zeta values. We give conjectural bases of the space of finite Euler sums as well as that of usual finite multiple zeta values in terms of these newly defined elements. A kind of ``parity result'' and certain sum formulas are also presented. △ Less

Submitted 26 September, 2021; originally announced September 2021.

Comments: 10 pages

MSC Class: 11M32; 11A07

Fluctuation in background synaptic activity controls synaptic plasticity

Authors: Yuto Takeda, Katsuhiko Hata, Tokio Yamasaki, Masaki Kaneko, Osamu Yokoi, Chengta Tsai, Kazuo Umemura, Tetsuro Nikuni

Abstract: Synaptic plasticity is vital for learning and memory in the brain. It consists of long-term potentiation (LTP) and long-term depression (LTD). Spike frequency is one of the major components of synaptic plasticity in the brain, a noisy environment. Recently, we mathematically analysed the frequency-dependent synaptic plasticity (FDP) in vivo and found that LTP is more likely to occur with an increa… ▽ More Synaptic plasticity is vital for learning and memory in the brain. It consists of long-term potentiation (LTP) and long-term depression (LTD). Spike frequency is one of the major components of synaptic plasticity in the brain, a noisy environment. Recently, we mathematically analysed the frequency-dependent synaptic plasticity (FDP) in vivo and found that LTP is more likely to occur with an increase in the frequency of background synaptic activity. Previous studies suggest fluctuation in the amplitude of background synaptic activity. However, little is understood about the relationship between synaptic plasticity and the fluctuation in the background synaptic activity. To address this issue, we performed numerical simulations of a calcium-based synapse model. Then, we found attenuation of the tendency to become LTD due to an increase in the fluctuation of background synaptic activity, leading to an enhancement of synaptic weight. Our result suggests that the fluctuation affect synaptic plasticity in the brain. △ Less

Submitted 12 August, 2021; originally announced August 2021.

Comments: 9 pages, 4 figures

Applying machine learning to determine impact parameter in nuclear physics experiments

Authors: C. Y. Tsang, Yongjia Wang, M. B. Tsang, J. Estee, T. Isobe, M. Kaneko, M. Kurata-Nishimura, J. W. Lee, Fupeng Li, Qingfeng Li, W. G. Lynch, T. Murakami, R. Wang, Dan Cozma, Rohit Kumar, Akira Ono, Ying-Xun Zhang

Abstract: Machine Learning (ML) algorithms have been demonstrated to be capable of predicting impact parameter in heavy-ion collisions from transport model simulation events with perfect detector response. We extend the scope of ML application to experimental data by incorporating realistic detector response of the S$π$RIT Time Projection Chamber into the heavy-ion simulation events generated from the UrQMD… ▽ More Machine Learning (ML) algorithms have been demonstrated to be capable of predicting impact parameter in heavy-ion collisions from transport model simulation events with perfect detector response. We extend the scope of ML application to experimental data by incorporating realistic detector response of the S$π$RIT Time Projection Chamber into the heavy-ion simulation events generated from the UrQMD model to resemble experimental data. At 3 fm, the predicted impact parameter is 2.8 fm if simulation events with perfect detector is used for training and testing; 2.4 fm if detector response is included in the training and testing, and 5.8 fm if ML algorithms trained with perfect detector is applied to testing data that has included detector response. The last result is not acceptable illustrating the importance of including the detector response in develo** the ML training algorithm. We also test the model dependence by applying the algorithms trained on UrQMD model to simulated events from four different transport models as well as using different input parameters on UrQMD model. Using data from Sn+Sn collisions at E/A=270 MeV, the ML determined impact parameters agree well with the experimentally determined impact parameter using multiplicities, except in the very central and very peripheral regions. ML selects central collision events better and allows impact parameters determination beyond the sharp cutoff limit imposed by experimental methods. △ Less

Submitted 26 July, 2021; originally announced July 2021.

Complementary junction field-effect transistor logic gate operational at 300$^\circ$C with 1.4 V supply voltage

Authors: Mitsuaki Kaneko, Masashi Nakajima, Qimin **, Tsunenobu Kimoto

Abstract: Integrated circuits (ICs) that can operate at high temperature have a wide variety of applications in the fields of automotive, aerospace, space exploration, and deep-well drilling. Conventional silicon-based complementary metal-oxide-semiconductor (CMOS) circuits cannot work at higher than 200 $^\circ$C, leading to the use of wide bandgap semiconductor, especially silicon carbide (SiC). However,… ▽ More Integrated circuits (ICs) that can operate at high temperature have a wide variety of applications in the fields of automotive, aerospace, space exploration, and deep-well drilling. Conventional silicon-based complementary metal-oxide-semiconductor (CMOS) circuits cannot work at higher than 200 $^\circ$C, leading to the use of wide bandgap semiconductor, especially silicon carbide (SiC). However, high-density defects at an oxide-SiC interface make it impossible to predict electrical characteristics of SiC CMOS logic gates in a wide temperature range and high supply voltage (typically ${\geqq 15}$ V) is required to compensate their large logic threshold voltage shift. Here, we show that SiC complementary logic gates composed of p- and n-channel junction field-effect transistors (JFETs) operate at 300 $^\circ$C with a supply voltage as low as 1.4 V. The logic threshold voltage shift of the complementary JFET (CJFET) inverter is 0.2 V from room temperature to 300 $^\circ$C. Furthermore, temperature dependencies of the static and dynamic characteristics of the CJFET inverter are well explained by a simple analytical model of SiC JFETs. This allows us to perform electronic circuit simulation, leading to superior designability of complex circuits or memories based on SiC CJFET technology, which operate within a wide temperature range. △ Less

Submitted 17 June, 2021; originally announced June 2021.

Comments: 30 pages, 4 figures, 2 supplementary figures

Sentence Concatenation Approach to Data Augmentation for Neural Machine Translation

Authors: Seiichiro Kondo, Kengo Hotate, Masahiro Kaneko, Mamoru Komachi

Abstract: Neural machine translation (NMT) has recently gained widespread attention because of its high translation accuracy. However, it shows poor performance in the translation of long sentences, which is a major issue in low-resource languages. It is assumed that this issue is caused by insufficient number of long sentences in the training data. Therefore, this study proposes a simple data augmentation… ▽ More Neural machine translation (NMT) has recently gained widespread attention because of its high translation accuracy. However, it shows poor performance in the translation of long sentences, which is a major issue in low-resource languages. It is assumed that this issue is caused by insufficient number of long sentences in the training data. Therefore, this study proposes a simple data augmentation method to handle long sentences. In this method, we use only the given parallel corpora as the training data and generate long sentences by concatenating two sentences. Based on the experimental results, we confirm improvements in long sentence translation by the proposed data augmentation method, despite its simplicity. Moreover, the translation quality is further improved by the proposed method, when combined with back-translation. △ Less

Submitted 17 April, 2021; originally announced April 2021.

Comments: 7 pages; camera-ready for NAACL Student Research Workshop 2021

Comparison of Grammatical Error Correction Using Back-Translation Models

Authors: Aomi Koyama, Kengo Hotate, Masahiro Kaneko, Mamoru Komachi

Abstract: Grammatical error correction (GEC) suffers from a lack of sufficient parallel data. Therefore, GEC studies have developed various methods to generate pseudo data, which comprise pairs of grammatical and artificially produced ungrammatical sentences. Currently, a mainstream approach to generate pseudo data is back-translation (BT). Most previous GEC studies using BT have employed the same architect… ▽ More Grammatical error correction (GEC) suffers from a lack of sufficient parallel data. Therefore, GEC studies have developed various methods to generate pseudo data, which comprise pairs of grammatical and artificially produced ungrammatical sentences. Currently, a mainstream approach to generate pseudo data is back-translation (BT). Most previous GEC studies using BT have employed the same architecture for both GEC and BT models. However, GEC models have different correction tendencies depending on their architectures. Thus, in this study, we compare the correction tendencies of the GEC models trained on pseudo data generated by different BT models, namely, Transformer, CNN, and LSTM. The results confirm that the correction tendencies for each error type are different for every BT model. Additionally, we examine the correction tendencies when using a combination of pseudo data generated by different BT models. As a result, we find that the combination of different BT models improves or interpolates the F_0.5 scores of each error type compared with that of single BT models with different seeds. △ Less

Submitted 15 April, 2021; originally announced April 2021.

Comments: 10 pages; camera-ready for NAACL Student Research Workshop 2021

Unmasking the Mask -- Evaluating Social Biases in Masked Language Models

Authors: Masahiro Kaneko, Danushka Bollegala

Abstract: Masked Language Models (MLMs) have shown superior performances in numerous downstream NLP tasks when used as text encoders. Unfortunately, MLMs also demonstrate significantly worrying levels of social biases. We show that the previously proposed evaluation metrics for quantifying the social biases in MLMs are problematic due to following reasons: (1) prediction accuracy of the masked tokens itself… ▽ More Masked Language Models (MLMs) have shown superior performances in numerous downstream NLP tasks when used as text encoders. Unfortunately, MLMs also demonstrate significantly worrying levels of social biases. We show that the previously proposed evaluation metrics for quantifying the social biases in MLMs are problematic due to following reasons: (1) prediction accuracy of the masked tokens itself tend to be low in some MLMs, which raises questions regarding the reliability of the evaluation metrics that use the (pseudo) likelihood of the predicted tokens, and (2) the correlation between the prediction accuracy of the mask and the performance in downstream NLP tasks is not taken into consideration, and (3) high frequency words in the training data are masked more often, introducing noise due to this selection bias in the test cases. To overcome the above-mentioned disfluencies, we propose All Unmasked Likelihood (AUL), a bias evaluation measure that predicts all tokens in a test case given the MLM embedding of the unmasked input. We find that AUL accurately detects different types of biases in MLMs. We also propose AUL with attention weights (AULA) to evaluate tokens based on their importance in a sentence. However, unlike AUL and AULA, previously proposed bias evaluation measures for MLMs systematically overestimate the measured biases, and are heavily influenced by the unmasked tokens in the context. △ Less

Submitted 15 April, 2021; originally announced April 2021.

Simultaneous Multi-Pivot Neural Machine Translation

Authors: Raj Dabre, Aizhan Imankulova, Masahiro Kaneko, Abhisek Chakrabarty

Abstract: Parallel corpora are indispensable for training neural machine translation (NMT) models, and parallel corpora for most language pairs do not exist or are scarce. In such cases, pivot language NMT can be helpful where a pivot language is used such that there exist parallel corpora between the source and pivot and pivot and target languages. Naturally, the quality of pivot language translation is mo… ▽ More Parallel corpora are indispensable for training neural machine translation (NMT) models, and parallel corpora for most language pairs do not exist or are scarce. In such cases, pivot language NMT can be helpful where a pivot language is used such that there exist parallel corpora between the source and pivot and pivot and target languages. Naturally, the quality of pivot language translation is more inferior to what could be achieved with a direct parallel corpus of a reasonable size for that pair. In a real-time simultaneous translation setting, the quality of pivot language translation deteriorates even further given that the model has to output translations the moment a few source words become available. To solve this issue, we propose multi-pivot translation and apply it to a simultaneous translation setting involving pivot languages. Our approach involves simultaneously translating a source language into multiple pivots, which are then simultaneously translated together into the target language by leveraging multi-source NMT. Our experiments in a low-resource setting using the N-way parallel UN corpus for Arabic to English NMT via French and Spanish as pivots reveals that in a simultaneous pivot NMT setting, using two pivot languages can lead to an improvement of up to 5.8 BLEU. △ Less

Submitted 15 April, 2021; originally announced April 2021.

Comments: preliminary work. pardon the messy writing and mistakes. will be submitted to emnlp after major overhaul

New Look at the Molecular Superbubble Candidate in the Galactic Center

Authors: Shiho Tsujimoto, Tomoharu Oka, Shunya Takekawa, Yuhei Iwata, Asaka Uruno, Hiroki Yokozuka, Ryosuke Nakagawara, Yuto Watanabe, Akira Kawakami, Sonomi Nishiyama, Miyuki Kaneko, Shoko Kanno, Takuma Ogawa

Abstract: The $l\!=\!+1.\!\!^\circ3$ region in the Galactic center is characterized by multiple shell-like structures and their extremely broad velocity widths. We revisit the molecular superbubble hypothesis for this region, based on high resolution maps of CO {\it J}=1--0, $^{13}$CO {\it J}=1--0, H$^{13}$CN {\it J}=1--0, H$^{13}$CO$^{+}$ {\it J}=1--0, SiO {\it J}=2--1, and CS {\it J}=2--1 lines obtained f… ▽ More The $l\!=\!+1.\!\!^\circ3$ region in the Galactic center is characterized by multiple shell-like structures and their extremely broad velocity widths. We revisit the molecular superbubble hypothesis for this region, based on high resolution maps of CO {\it J}=1--0, $^{13}$CO {\it J}=1--0, H$^{13}$CN {\it J}=1--0, H$^{13}$CO$^{+}$ {\it J}=1--0, SiO {\it J}=2--1, and CS {\it J}=2--1 lines obtained from the Nobeyama radio observatory 45-m telescope, as well as CO {\it J}=3--2 maps obtained from the James Clerk Maxwell telescope. We identified eleven expanding shells with total kinetic energy and typical expansion time $E_{\rm kin}\!\sim\! 10^{51.9}$ erg and $t_{\rm exp}\!\sim\! 10^{4.9}$ yr, respectively. In addition, the $l\!=\!+1.\!\!^\circ3$ region exhibited high SiO {\it J}=2--1/H$^{13}$CN {\it J}=1--0 and SiO {\it J}=2--1/H$^{13}$CO$^{+}$ {\it J}=1--0 intensity ratios, indicating that the region has experienced dissociative shocks in the past. These new findings confirm the molecular superbubble hypothesis for the $l\!=\!+1.\!\!^\circ3$ region. The nature of the embedded star cluster, which may have supplied 20--70 supernova explosions within 10$^5$ yr, is discussed. This work also show the importance of compact broad-velocity-width features in searching for localized energy sources hidden behind severe interstellar extinction and stellar contamination. △ Less

Submitted 18 March, 2021; originally announced March 2021.

Probing the Symmetry Energy with the Spectral Pion Ratio

Authors: J. Estee, W. G. Lynch, C. Y. Tsang, J. Barney, G. Jhang, M. B. Tsang, R. Wang, M. Kaneko, J. W. Lee, T. Isobe, M. Kurata-Nishimura, T. Murakami, D. S. Ahn, L. Atar, T. Aumann, H. Baba, K. Boretzky, J. Brzychczyk, G. Cerizza, N. Chiga, N. Fukuda, I. Gasparic, B. Hong, A. Horvat, K. Ieki , et al. (38 additional authors not shown)

Abstract: Many neutron star (NS) properties, such as the proton fraction within a NS, reflect the symmetry energy contributions to the Equation of State that dominate when neutron and proton densities differ strongly. To constrain these contributions at supra-saturation densities, we measure the spectra of charged pions produced by colliding rare isotope tin (Sn) beams with isotopically enriched Sn targets.… ▽ More Many neutron star (NS) properties, such as the proton fraction within a NS, reflect the symmetry energy contributions to the Equation of State that dominate when neutron and proton densities differ strongly. To constrain these contributions at supra-saturation densities, we measure the spectra of charged pions produced by colliding rare isotope tin (Sn) beams with isotopically enriched Sn targets. Using ratios of the charged pion spectra measured at high transverse momenta, we deduce the slope of the symmetry energy to be $42 < L < 117$ MeV. This value is slightly lower but consistent with the $L$ values deduced from a recent measurement of the neutron skin thickness of $^{208}$Pb. △ Less

Submitted 11 March, 2021; originally announced March 2021.

Journal ref: Phys. Rev. Lett. 126, 162701 (2021)

Dictionary-based Debiasing of Pre-trained Word Embeddings

Authors: Masahiro Kaneko, Danushka Bollegala

Abstract: Word embeddings trained on large corpora have shown to encode high levels of unfair discriminatory gender, racial, religious and ethnic biases. In contrast, human-written dictionaries describe the meanings of words in a concise, objective and an unbiased manner. We propose a method for debiasing pre-trained word embeddings using dictionaries, without requiring access to the original training r… ▽ More Word embeddings trained on large corpora have shown to encode high levels of unfair discriminatory gender, racial, religious and ethnic biases. In contrast, human-written dictionaries describe the meanings of words in a concise, objective and an unbiased manner. We propose a method for debiasing pre-trained word embeddings using dictionaries, without requiring access to the original training resources or any knowledge regarding the word embedding algorithms used. Unlike prior work, our proposed method does not require the types of biases to be pre-defined in the form of word lists, and learns the constraints that must be satisfied by unbiased word embeddings automatically from dictionary definitions of the words. Specifically, we learn an encoder to generate a debiased version of an input word embedding such that it (a) retains the semantics of the pre-trained word embeddings, (b) agrees with the unbiased definition of the word according to the dictionary, and (c) remains orthogonal to the vector space spanned by any biased basis vectors in the pre-trained word embedding space. Experimental results on standard benchmark datasets show that the proposed method can accurately remove unfair biases encoded in pre-trained word embeddings, while preserving useful semantics. △ Less

Submitted 23 January, 2021; originally announced January 2021.

Comments: EACL 2021