Applying Ensemble Methods to Model-Agnostic Machine-Generated Text Detection
Authors:
Ivan Ong,
Boon King Quek
Abstract:
In this paper, we study the problem of detecting machine-generated text when the large language model (LLM) it is possibly derived from is unknown. We do so by apply ensembling methods to the outputs from DetectGPT classifiers (Mitchell et al. 2023), a zero-shot model for machine-generated text detection which is highly accurate when the generative (or base) language model is the same as the discr…
▽ More
In this paper, we study the problem of detecting machine-generated text when the large language model (LLM) it is possibly derived from is unknown. We do so by apply ensembling methods to the outputs from DetectGPT classifiers (Mitchell et al. 2023), a zero-shot model for machine-generated text detection which is highly accurate when the generative (or base) language model is the same as the discriminative (or scoring) language model. We find that simple summary statistics of DetectGPT sub-model outputs yield an AUROC of 0.73 (relative to 0.61) while retaining its zero-shot nature, and that supervised learning methods sharply boost the accuracy to an AUROC of 0.94 but require a training dataset. This suggests the possibility of further generalisation to create a highly-accurate, model-agnostic machine-generated text detector.
△ Less
Submitted 18 June, 2024;
originally announced June 2024.
Knowledge Representation for Conceptual, Motivational, and Affective Processes in Natural Language Communication
Authors:
Seng-Beng Ho,
Zhaoxia Wang,
Boon-Kiat Quek,
Erik Cambria
Abstract:
Natural language communication is an intricate and complex process. The speaker usually begins with an intention and motivation of what is to be communicated, and what effects are expected from the communication, while taking into consideration the listener's mental model to concoct an appropriate sentence. The listener likewise has to interpret what the speaker means, and respond accordingly, als…
▽ More
Natural language communication is an intricate and complex process. The speaker usually begins with an intention and motivation of what is to be communicated, and what effects are expected from the communication, while taking into consideration the listener's mental model to concoct an appropriate sentence. The listener likewise has to interpret what the speaker means, and respond accordingly, also with the speaker's mental state in mind. To do this successfully, conceptual, motivational, and affective processes have to be represented appropriately to drive the language generation and understanding processes. Language processing has succeeded well with the big data approach in applications such as chatbots and machine translation. However, in human-robot collaborative social communication and in using natural language for delivering precise instructions to robots, a deeper representation of the conceptual, motivational, and affective processes is needed. This paper capitalizes on the UGALRS (Unified General Autonomous and Language Reasoning System) framework and the CD+ (Conceptual Representation Plus) representational scheme to illustrate how social communication through language is supported by a knowledge representational scheme that handles conceptual, motivational, and affective processes in a deep and general way. Though a small set of concepts, motivations, and emotions is treated in this paper, its main contribution is in articulating a general framework of knowledge representation and processing to link these aspects together in serving the purpose of natural language communication for an intelligent system.
△ Less
Submitted 20 October, 2022; v1 submitted 25 September, 2022;
originally announced October 2022.
