-
Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks
Authors:
Wenyue Hua,
Jiang Guo,
Mingwen Dong,
Henghui Zhu,
Patrick Ng,
Zhiguo Wang
Abstract:
Current approaches of knowledge editing struggle to effectively propagate updates to interconnected facts. In this work, we delve into the barriers that hinder the appropriate propagation of updated knowledge within these models for accurate reasoning. To support our analysis, we introduce a novel reasoning-based benchmark -- ReCoE (Reasoning-based Counterfactual Editing dataset) -- which covers s…
▽ More
Current approaches of knowledge editing struggle to effectively propagate updates to interconnected facts. In this work, we delve into the barriers that hinder the appropriate propagation of updated knowledge within these models for accurate reasoning. To support our analysis, we introduce a novel reasoning-based benchmark -- ReCoE (Reasoning-based Counterfactual Editing dataset) -- which covers six common reasoning schemes in real world. We conduct a thorough analysis of existing knowledge editing techniques, including input augmentation, finetuning, and locate-and-edit. We found that all model editing methods show notably low performance on this dataset, especially in certain reasoning schemes. Our analysis over the chain-of-thought generation of edited models further uncover key reasons behind the inadequacy of existing knowledge editing methods from a reasoning standpoint, involving aspects on fact-wise editing, fact recall ability, and coherence in generation. We will make our benchmark publicly available.
△ Less
Submitted 30 January, 2024;
originally announced January 2024.
-
Design and analysis of a microplate assay in the presence of multiple restrictions on the randomization
Authors:
Alexandre Bohyn,
Eric D. Schoen,
Chee ** Ng,
Kristina Bishard,
Manon Haarmans,
Sebastian J. Trietsch,
Peter Goos
Abstract:
Experiments using multi-step protocols often involve several restrictions on the randomization. For a specific application to in vitro testing on microplates, a design was required with both a split-plot and a strip-plot structure. On top of two-level treatment factors and the factors that define the randomization restrictions, a multi-level fixed blocking factor not involving further restrictions…
▽ More
Experiments using multi-step protocols often involve several restrictions on the randomization. For a specific application to in vitro testing on microplates, a design was required with both a split-plot and a strip-plot structure. On top of two-level treatment factors and the factors that define the randomization restrictions, a multi-level fixed blocking factor not involving further restrictions on the randomization had to be added. We develop a step-by-step approach to construct a design for the microplate experiment and analyze a response. To consolidate the approach, we study various alternative scenarios for the experiment.
△ Less
Submitted 10 March, 2023;
originally announced March 2023.
-
Testing Multiple Linear Regression Systems with Metamorphic Testing
Authors:
Quang-Hung Luu,
Man F. Lau,
Sebastian P. H. Ng,
Tsong Yueh Chen
Abstract:
Regression is one of the most commonly used statistical techniques. However, testing regression systems is a great challenge because of the absence of test oracle in general. In this paper, we show that Metamorphic Testing is an effective approach to test multiple linear regression systems. In doing so, we identify intrinsic mathematical properties of linear regression, and then propose 11 Metamor…
▽ More
Regression is one of the most commonly used statistical techniques. However, testing regression systems is a great challenge because of the absence of test oracle in general. In this paper, we show that Metamorphic Testing is an effective approach to test multiple linear regression systems. In doing so, we identify intrinsic mathematical properties of linear regression, and then propose 11 Metamorphic Relations to be used for testing. Their effectiveness is examined using mutation analysis with a range of different regression programs. We further look at how the testing could be adopted in a more effective way. Our work is applicable to examine the reliability of predictive systems based on regression that has been widely used in economics, engineering and science, as well as of the regression calculation manipulated by statistical users.
△ Less
Submitted 17 August, 2021;
originally announced August 2021.
-
dna2vec: Consistent vector representations of variable-length k-mers
Authors:
Patrick Ng
Abstract:
One of the ubiquitous representation of long DNA sequence is dividing it into shorter k-mer components. Unfortunately, the straightforward vector encoding of k-mer as a one-hot vector is vulnerable to the curse of dimensionality. Worse yet, the distance between any pair of one-hot vectors is equidistant. This is particularly problematic when applying the latest machine learning algorithms to solve…
▽ More
One of the ubiquitous representation of long DNA sequence is dividing it into shorter k-mer components. Unfortunately, the straightforward vector encoding of k-mer as a one-hot vector is vulnerable to the curse of dimensionality. Worse yet, the distance between any pair of one-hot vectors is equidistant. This is particularly problematic when applying the latest machine learning algorithms to solve problems in biological sequence analysis. In this paper, we propose a novel method to train distributed representations of variable-length k-mers. Our method is based on the popular word embedding model word2vec, which is trained on a shallow two-layer neural network. Our experiments provide evidence that the summing of dna2vec vectors is akin to nucleotides concatenation. We also demonstrate that there is correlation between Needleman-Wunsch similarity score and cosine similarity of dna2vec vectors.
△ Less
Submitted 23 January, 2017;
originally announced January 2017.
