Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code
Authors:
Rangeet Pan,
Ali Reza Ibrahimzada,
Rahul Krishna,
Divya Sankar,
Lambert Pouguem Wassi,
Michele Merler,
Boris Sobolev,
Raju Pavuluri,
Saurabh Sinha,
Reyhaneh Jabbarvand
Abstract:
Code translation aims to convert source code from one programming language (PL) to another. Given the promising abilities of large language models (LLMs) in code synthesis, researchers are exploring their potential to automate code translation. The prerequisite for advancing the state of LLM-based code translation is to understand their promises and limitations over existing techniques. To that en…
▽ More
Code translation aims to convert source code from one programming language (PL) to another. Given the promising abilities of large language models (LLMs) in code synthesis, researchers are exploring their potential to automate code translation. The prerequisite for advancing the state of LLM-based code translation is to understand their promises and limitations over existing techniques. To that end, we present a large-scale empirical study to investigate the ability of general LLMs and code LLMs for code translation across pairs of different languages, including C, C++, Go, Java, and Python. Our study, which involves the translation of 1,700 code samples from three benchmarks and two real-world projects, reveals that LLMs are yet to be reliably used to automate code translation -- with correct translations ranging from 2.1% to 47.3% for the studied LLMs. Further manual investigation of unsuccessful translations identifies 15 categories of translation bugs. We also compare LLM-based code translation with traditional non-LLM-based approaches. Our analysis shows that these two classes of techniques have their own strengths and weaknesses. Finally, insights from our study suggest that providing more context to LLMs during translation can help them produce better results. To that end, we propose a prompt-crafting approach based on the symptoms of erroneous translations; this improves the performance of LLM-based code translation by 5.5% on average. Our study is the first of its kind, in terms of scale and breadth, that provides insights into the current limitations of LLMs in code translation and opportunities for improving them. Our dataset -- consisting of 1,700 code samples in five PLs with 10K+ tests, 43K+ translated code, 1,748 manually labeled bugs, and 1,365 bug-fix pairs -- can help drive research in this area.
△ Less
Submitted 16 January, 2024; v1 submitted 6 August, 2023;
originally announced August 2023.
Double-hit separation and dE/dx resolution of a time projection chamber with GEM readout
Authors:
Yumi Aoki,
David Attié,
Ties Behnke,
Alain Bellerive,
Oleg Bezshyyko,
Deb Bhattacharya Sankar,
Purba Bhattacharya,
Sudeb Bhattacharya,
Yue Chang,
Paul Colas,
Gilles De Lentdecker,
Klaus Dehmelt,
Klaus Desch,
Ralf Diener,
Madhu Dixit,
Ulrich Einhaus,
Oleksiy Fedorchuk,
Ivor Fleck,
Keisuke Fujii,
Takahiro Fusayasu,
Serguei Ganjour,
Philippe Gros,
Peter Hayman,
Katsumasa Ikematsu,
Leif Jönsson
, et al. (46 additional authors not shown)
Abstract:
A time projection chamber (TPC) with micropattern gaseous detector (MPGD) readout is investigated as main tracking device of the International Large Detector (ILD) concept at the planned International Linear Collider (ILC). A prototype TPC equipped with a triple gas electron multiplier (GEM) readout has been built and operated in an electron test beam. The TPC was placed in a 1 T solenoidal field…
▽ More
A time projection chamber (TPC) with micropattern gaseous detector (MPGD) readout is investigated as main tracking device of the International Large Detector (ILD) concept at the planned International Linear Collider (ILC). A prototype TPC equipped with a triple gas electron multiplier (GEM) readout has been built and operated in an electron test beam. The TPC was placed in a 1 T solenoidal field at the DESY II Test Beam Facility, which provides an electron beam up to 6 GeV/c. The performance of the readout modules, in particular the spatial point resolution, is determined and compared to earlier tests. New studies are presented with first results on the separation of close-by tracks and the capability of the system to measure the specific energy loss dE/dx. This is complemented by a simulation study on the optimization of the readout granularity to improve particle identification by dE/dx.
△ Less
Submitted 25 November, 2022; v1 submitted 24 May, 2022;
originally announced May 2022.
