Understanding and Mitigating Language Confusion in LLMs
Authors:
Kelly Marchisio,
Wei-Yin Ko,
Alexandre Bérard,
Théo Dehaze,
Sebastian Ruder
Abstract:
We investigate a surprising limitation of LLMs: their inability to consistently generate text in a user's desired language. We create the Language Confusion Benchmark (LCB) to evaluate such failures, covering 15 typologically diverse languages with existing and newly-created English and multilingual prompts. We evaluate a range of LLMs on monolingual and cross-lingual generation reflecting practic…
▽ More
We investigate a surprising limitation of LLMs: their inability to consistently generate text in a user's desired language. We create the Language Confusion Benchmark (LCB) to evaluate such failures, covering 15 typologically diverse languages with existing and newly-created English and multilingual prompts. We evaluate a range of LLMs on monolingual and cross-lingual generation reflecting practical use cases, finding that Llama Instruct and Mistral models exhibit high degrees of language confusion and even the strongest models fail to consistently respond in the correct language. We observe that base and English-centric instruct models are more prone to language confusion, which is aggravated by complex prompts and high sampling temperatures. We find that language confusion can be partially mitigated via few-shot prompting, multilingual SFT and preference tuning. We release our language confusion benchmark, which serves as a first layer of efficient, scalable multilingual evaluation at https://github.com/for-ai/language-confusion.
△ Less
Submitted 28 June, 2024;
originally announced June 2024.
Optimal Sensor Fusion Method for Active Vibration Isolation Systems in Ground-Based Gravitational-Wave Detectors
Authors:
T. T. L. Tsang,
T. G. F. Li,
T. Dehaeze,
C. Collette
Abstract:
Sensor fusion is a technique used to combine sensors with different noise characteristics into a super sensor that has superior noise performance. To achieve sensor fusion, complementary filters are used in current gravitational-wave detectors to combine relative displacement sensors and inertial sensors for active seismic isolation. Complementary filters are a set of digital filters, which have t…
▽ More
Sensor fusion is a technique used to combine sensors with different noise characteristics into a super sensor that has superior noise performance. To achieve sensor fusion, complementary filters are used in current gravitational-wave detectors to combine relative displacement sensors and inertial sensors for active seismic isolation. Complementary filters are a set of digital filters, which have transfer functions that are summed to unity. Currently, complementary filters are shaped and tuned manually rather than optimized, which can be suboptimal and hard to reproduce for future detectors. In this paper, an optimization-based method called $\mathcal{H}_\infty$ synthesis is proposed for synthesizing optimal complementary filters according to the sensor noises themselves. The complementary filter design problem is converted into an optimization problem that seeks minimization of an objective function equivalent to the maximum difference between the super sensor noise and the lower bound in logarithmic scale. The method is exemplified by synthesizing complementary filters for sensor fusion of 1) a relative displacement sensor and an inertial sensor, 2) a relative displacement sensor coupled with seismic noise and an inertial sensor, and 3) hypothetical displacement sensor and inertial sensor, which have slightly different noise characteristics compared to the typical ones. In all cases, the method produces complementary filters that suppress the super sensor noise equally close to the lower bound at all frequencies in logarithmic scale. The synthesized filters contain features that better suppress the sensor noises compared to the pre-designed complementary filters. Overall, the proposed method allows the synthesis of optimal complementary filters according to the sensor noises themselves and is a better and versatile method for solving sensor fusion problems.
△ Less
Submitted 5 September, 2022; v1 submitted 29 November, 2021;
originally announced November 2021.
