-
LLM-based Test-driven Interactive Code Generation: User Study and Empirical Evaluation
Authors:
Sarah Fakhoury,
Aaditya Naik,
Georgios Sakkas,
Saikat Chakraborty,
Shuvendu K. Lahiri
Abstract:
Large language models (LLMs) have shown great potential in automating significant aspects of coding by producing natural code from informal natural language (NL) intent. However, given NL is informal, it does not lend easily to checking that the generated code correctly satisfies the user intent. In this paper, we propose a novel interactive workflow TiCoder for guided intent clarification (i.e.,…
▽ More
Large language models (LLMs) have shown great potential in automating significant aspects of coding by producing natural code from informal natural language (NL) intent. However, given NL is informal, it does not lend easily to checking that the generated code correctly satisfies the user intent. In this paper, we propose a novel interactive workflow TiCoder for guided intent clarification (i.e., partial formalization) through tests to support the generation of more accurate code suggestions. Through a mixed methods user study with 15 programmers, we present an empirical evaluation of the effectiveness of the workflow to improve code generation accuracy. We find that participants using the proposed workflow are significantly more likely to correctly evaluate AI generated code, and report significantly less task-induced cognitive load. Furthermore, we test the potential of the workflow at scale with four different state-of-the-art LLMs on two python datasets, using an idealized proxy for a user feedback. We observe an average absolute improvement of 38.43% in the pass@1 code generation accuracy for both datasets and across all LLMs within 5 user interactions, in addition to the automatic generation of accompanying unit tests.
△ Less
Submitted 15 April, 2024;
originally announced April 2024.
-
Inversion and Integral Identities in dCFTs
Authors:
Georgios Sakkas
Abstract:
This work derives an application from the identities of arXiv:hep-th/0602028 in order to invert four point functions in defect conformal field theories. For this, a recursion relation is established and the O(N) model with a line defect is considered as a testing ground of this application. Specifically, the CFT data are calculated from inversion of tilt and displacement four point functions. The…
▽ More
This work derives an application from the identities of arXiv:hep-th/0602028 in order to invert four point functions in defect conformal field theories. For this, a recursion relation is established and the O(N) model with a line defect is considered as a testing ground of this application. Specifically, the CFT data are calculated from inversion of tilt and displacement four point functions. The recursion relation enables efficient computation of hypergeometrics at order $ε$ in the $ε$-expansion, leading to the inversion of four point functions and the derivation of CFT data. The inversion method presented offers a faster alternative to traditional approaches using arXiv:hep-ph/0507094v2, arXiv:0708.2443v2. The study also explores a general ansatz approach, assessing the algorithm's restrictiveness, and concludes by examining implications for the integral identity constraint of arXiv:2203.17157v2, predicting corrections to OPE coefficients.
△ Less
Submitted 8 March, 2024;
originally announced March 2024.
-
Interactive Code Generation via Test-Driven User-Intent Formalization
Authors:
Shuvendu K. Lahiri,
Sarah Fakhoury,
Aaditya Naik,
Georgios Sakkas,
Saikat Chakraborty,
Madanlal Musuvathi,
Piali Choudhury,
Curtis von Veh,
Jeevana Priya Inala,
Chenglong Wang,
Jianfeng Gao
Abstract:
Large language models (LLMs) have shown great potential in automating significant aspects of coding by producing natural code from informal natural language (NL) intent. However, when interacting with LLMs, users have no guarantees that the code suggestions produced correctly satisfy the intent they provided. In fact, it is hard to define a notion of correctness since natural language can be ambig…
▽ More
Large language models (LLMs) have shown great potential in automating significant aspects of coding by producing natural code from informal natural language (NL) intent. However, when interacting with LLMs, users have no guarantees that the code suggestions produced correctly satisfy the intent they provided. In fact, it is hard to define a notion of correctness since natural language can be ambiguous and lacks a formal semantics.
In this paper, we propose the workflow of {\it interactive test-driven code generation}, which leverages lightweight user feedback to (a) formalize the user intent using generated tests that can be useful for debugging, and (b) produce an improved set of code suggestions by pruning and ranking candidate code suggestions. We describe a language-agnostic abstract algorithm and a concrete implementation TiCoder. We perform an automated evaluation of TiCoder on the \emph{MBPP} and \emph{HumanEval} code generation benchmarks. Our results are promising with using the OpenAI Codex LLM: our best algorithm improves the \passk{1} code generation accuracy (in absolute percentages) between $22.49\%$ to $37.71\%$ for MBPP and between $24.79\%$ to $53.98\%$ for HumanEval using between 1 to 5 simulated user queries.
△ Less
Submitted 3 October, 2023; v1 submitted 11 August, 2022;
originally announced August 2022.
-
Broken global symmetries and defect conformal manifolds
Authors:
Nadav Drukker,
Ziwen Kong,
Georgios Sakkas
Abstract:
Just as exactly marginal operators allow to deform a conformal field theory along the space of theories known as the conformal manifold, appropriate operators on conformal defects allow for deformations of the defects. When a defect breaks a global symmetry, there is a contact term in the conservation equation with an exactly marginal defect operator. The resulting defect conformal manifold is the…
▽ More
Just as exactly marginal operators allow to deform a conformal field theory along the space of theories known as the conformal manifold, appropriate operators on conformal defects allow for deformations of the defects. When a defect breaks a global symmetry, there is a contact term in the conservation equation with an exactly marginal defect operator. The resulting defect conformal manifold is the symmetry breaking coset and its Zamolodchikov metric is expressed as the 2-point function of the exactly marginal operator. As the Riemann tensor on the conformal manifold can be expressed as an integrated 4-point function of the marginal operators, we find an exact relation to the curvature of the coset space. We confirm this relation against previously obtained 4-point functions for insertions into the 1/2 BPS Wilson loop in ${\cal N} = 4$ SYM and 3d ${\cal N} = 6$ theory and the 1/2 BPS surface operator of the 6d ${\cal N} = (2, 0)$ theory.
△ Less
Submitted 18 October, 2022; v1 submitted 31 March, 2022;
originally announced March 2022.
