-
Evaluating the Decency and Consistency of Data Validation Tests Generated by LLMs
Authors:
Rohan Alexander,
Lindsay Katz,
Callandra Moore,
Michael Wing-Cheung Wong,
Zane Schwartz
Abstract:
We investigated whether large language models (LLMs) can develop data validation tests. We considered 96 conditions each for both GPT-3.5 and GPT-4, examining different prompt scenarios, learning modes, temperature settings, and roles. The prompt scenarios were: 1) Asking for expectations, 2) Asking for expectations with a given context, 3) Asking for expectations after requesting a data simulatio…
▽ More
We investigated whether large language models (LLMs) can develop data validation tests. We considered 96 conditions each for both GPT-3.5 and GPT-4, examining different prompt scenarios, learning modes, temperature settings, and roles. The prompt scenarios were: 1) Asking for expectations, 2) Asking for expectations with a given context, 3) Asking for expectations after requesting a data simulation, and 4) Asking for expectations with a provided data sample. The learning modes were: 1) zero-shot, 2) one-shot, and 3) few-shot learning. We also tested four temperature settings: 0, 0.4, 0.6, and 1. And the two distinct roles were: 1) helpful assistant, 2) expert data scientist. To gauge consistency, every setup was tested five times. The LLM-generated responses were benchmarked against a gold standard data validation suite, created by an experienced data scientist knowledgeable about the data in question. We find there are considerable returns to the use of few-shot learning, and that the more explicit the data setting can be the better, to a point. The best LLM configurations complement, rather than substitute, the gold standard results. This study underscores the value LLMs can bring to the data cleaning and preparation stages of the data science workflow, but highlights that they need considerable evaluation by experienced analysts.
△ Less
Submitted 1 April, 2024; v1 submitted 2 October, 2023;
originally announced October 2023.
-
User Guided Abductive Proof Generation for Answer Set Programming Queries (Extended Version)
Authors:
Avishkar Mahajan,
Martin Strecker,
Meng Weng Wong
Abstract:
We present a method for generating possible proofs of a query with respect to a given Answer Set Programming (ASP) rule set using an abductive process where the space of abducibles is automatically constructed just from the input rules alone. Given a (possibly empty) set of user provided facts, our method infers any additional facts that may be needed for the entailment of a query and then outputs…
▽ More
We present a method for generating possible proofs of a query with respect to a given Answer Set Programming (ASP) rule set using an abductive process where the space of abducibles is automatically constructed just from the input rules alone. Given a (possibly empty) set of user provided facts, our method infers any additional facts that may be needed for the entailment of a query and then outputs these extra facts, without the user needing to explicitly specify the space of all abducibles. We also present a method to generate a set of directed edges corresponding to the justification graph for the query. Furthermore, through different forms of implicit term substitution, our method can take user provided facts into account and suitably modify the abductive solutions. Past work on abduction has been primarily based on goal directed methods. However these methods can result in solvers that are not truly declarative. Much less work has been done on realizing abduction in a bottom up solver like the Clingo ASP solver. We describe novel ASP programs which can be run directly in Clingo to yield the abductive solutions and directed edge sets without needing to modify the underlying solving engine.
△ Less
Submitted 16 September, 2022;
originally announced September 2022.
-
Automating Defeasible Reasoning in Law
Authors:
How Khang Lim,
Avishkar Mahajan,
Martin Strecker,
Meng Weng Wong
Abstract:
The paper studies defeasible reasoning in rule-based systems, in particular about legal norms and contracts. We identify rule modifiers that specify how rules interact and how they can be overridden. We then define rule transformations that eliminate these modifiers, leading in the end to a translation of rules to formulas. For reasoning with and about rules, we contrast two approaches, one in a c…
▽ More
The paper studies defeasible reasoning in rule-based systems, in particular about legal norms and contracts. We identify rule modifiers that specify how rules interact and how they can be overridden. We then define rule transformations that eliminate these modifiers, leading in the end to a translation of rules to formulas. For reasoning with and about rules, we contrast two approaches, one in a classical logic with SMT solvers as proof engines, one in a non-monotonic logic with Answer Set Programming solvers.
△ Less
Submitted 15 May, 2022;
originally announced May 2022.
-
Implementation of Smart Contracts Using Hybrid Architectures with On- and Off-Blockchain Components
Authors:
Carlos Molina-Jimenez,
Ioannis Sfyrakis,
Ellis Solaiman,
Irene Ng,
Meng Weng Wong,
Alexis Chun,
Jon Crowcroft
Abstract:
Recently, decentralised (on-blockchain) platforms have emerged to complement centralised (off-blockchain) platforms for the implementation of automated, digital (smart) contracts. However, neither alternative can individually satisfy the requirements of a large class of applications. On-blockchain platforms suffer from scalability, performance, transaction costs and other limitations. Off-blockcha…
▽ More
Recently, decentralised (on-blockchain) platforms have emerged to complement centralised (off-blockchain) platforms for the implementation of automated, digital (smart) contracts. However, neither alternative can individually satisfy the requirements of a large class of applications. On-blockchain platforms suffer from scalability, performance, transaction costs and other limitations. Off-blockchain platforms are afflicted by drawbacks due to their dependence on single trusted third parties. We argue that in several application areas, hybrid platforms composed from the integration of on- and off-blockchain platforms are more able to support smart contracts that deliver the desired quality of service (QoS). Hybrid architectures are largely unexplored. To help cover the gap, in this paper we discuss the implementation of smart contracts on hybrid architectures. As a proof of concept, we show how a smart contract can be split and executed partially on an off-blockchain contract compliance checker and partially on the Rinkeby Ethereum network. To test the solution, we expose it to sequences of contractual operations generated mechanically by a contract validator tool.
△ Less
Submitted 31 July, 2018;
originally announced August 2018.
-
On Dissipative Nonlinear Evolutional Pseudo-Differential Equations
Authors:
Mingjuan Chen,
Baoxiang Wang,
Shuxia Wang,
M. W. Wong
Abstract:
First, using the uniform decomposition in both physical and frequency spaces, we obtain an equivalent norm on modulation spaces. Secondly, we consider the Cauchy problem for the dissipative evolutionary pseudo-differential equation \partial_t u + A(x,D) u = F\big((\partial^α_x u)_{|α|\leq κ}\big), \ \ u(0,x)= u_0(x), where $A(x,D)$ is a dissipative pseudo-differential operator and $F(z)$ is a mult…
▽ More
First, using the uniform decomposition in both physical and frequency spaces, we obtain an equivalent norm on modulation spaces. Secondly, we consider the Cauchy problem for the dissipative evolutionary pseudo-differential equation \partial_t u + A(x,D) u = F\big((\partial^α_x u)_{|α|\leq κ}\big), \ \ u(0,x)= u_0(x), where $A(x,D)$ is a dissipative pseudo-differential operator and $F(z)$ is a multi-polynomial. We will develop the uniform decomposition techniques in both physical and frequency spaces to study its local well posedness in modulation spaces $M^s_{p,q}$ and in Sobolev spaces $H^s$. Moreover, the local solution can be extended to a global one in $L^2$ and in $H^s$ ($s>κ+d/2$) for certain nonlinearities.
△ Less
Submitted 30 August, 2017;
originally announced August 2017.
