-
Learning to Explore and Select for Coverage-Conditioned Retrieval-Augmented Generation
Authors:
Takyoung Kim,
Kyungjae Lee,
Young Rok Jang,
Ji Yong Cho,
Gangwoo Kim,
Minseok Cho,
Moontae Lee
Abstract:
Interactions with billion-scale large language models typically yield long-form responses due to their extensive parametric capacities, along with retrieval-augmented features. While detailed responses provide insightful viewpoint of a specific subject, they frequently generate redundant and less engaging content that does not meet user interests. In this work, we focus on the role of query outlin…
▽ More
Interactions with billion-scale large language models typically yield long-form responses due to their extensive parametric capacities, along with retrieval-augmented features. While detailed responses provide insightful viewpoint of a specific subject, they frequently generate redundant and less engaging content that does not meet user interests. In this work, we focus on the role of query outlining (i.e., selected sequence of queries) in scenarios that users request a specific range of information, namely coverage-conditioned ($C^2$) scenarios. For simulating $C^2$ scenarios, we construct QTree, 10K sets of information-seeking queries decomposed with various perspectives on certain topics. By utilizing QTree, we train QPlanner, a 7B language model generating customized query outlines that follow coverage-conditioned queries. We analyze the effectiveness of generated outlines through automatic and human evaluation, targeting on retrieval-augmented generation (RAG). Moreover, the experimental results demonstrate that QPlanner with alignment training can further provide outlines satisfying diverse user interests. Our resources are available at https://github.com/youngerous/qtree.
△ Less
Submitted 1 July, 2024;
originally announced July 2024.
-
The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models
Authors:
Seungone Kim,
Juyoung Suk,
Ji Yong Cho,
Shayne Longpre,
Chaeeun Kim,
Dongkeun Yoon,
Gui** Son,
Ye** Cho,
Sheikh Shafayat,
**heon Baek,
Sue Hyun Park,
Hyeonbin Hwang,
**kyung Jo,
Hyowon Cho,
Haebin Shin,
Seongyun Lee,
Hanseok Oh,
Noah Lee,
Namgyu Ho,
Se June Joo,
Miyoung Ko,
Yoonjoo Lee,
Hyungjoo Chae,
Jamin Shin,
Joel Jang
, et al. (7 additional authors not shown)
Abstract:
As language models (LMs) become capable of handling a wide range of tasks, their evaluation is becoming as challenging as their development. Most generation benchmarks currently assess LMs using abstract evaluation criteria like helpfulness and harmlessness, which often lack the flexibility and granularity of human assessment. Additionally, these benchmarks tend to focus disproportionately on spec…
▽ More
As language models (LMs) become capable of handling a wide range of tasks, their evaluation is becoming as challenging as their development. Most generation benchmarks currently assess LMs using abstract evaluation criteria like helpfulness and harmlessness, which often lack the flexibility and granularity of human assessment. Additionally, these benchmarks tend to focus disproportionately on specific capabilities such as instruction following, leading to coverage bias. To overcome these limitations, we introduce the BiGGen Bench, a principled generation benchmark designed to thoroughly evaluate nine distinct capabilities of LMs across 77 diverse tasks. A key feature of the BiGGen Bench is its use of instance-specific evaluation criteria, closely mirroring the nuanced discernment of human evaluation. We apply this benchmark to assess 103 frontier LMs using five evaluator LMs. Our code, data, and evaluation results are all publicly available at https://github.com/prometheus-eval/prometheus-eval/tree/main/BiGGen-Bench.
△ Less
Submitted 9 June, 2024;
originally announced June 2024.
-
Discursive objection strategies in online comments: Develo** a classification schema and validating its training
Authors:
Ashley L. Shea,
Aspen K. B. Omapang,
Ji Yong Cho,
Miryam Y. Ginsparg,
Natalie Bazarova,
Winice Hui,
René F. Kizilcec,
Chau Tong,
Drew Margolin
Abstract:
Most Americans agree that misinformation, hate speech and harassment are harmful and inadequately curbed on social media through current moderation practices. In this paper, we aim to understand the discursive strategies employed by people in response to harmful speech in news comments. We conducted a content analysis of more than 6500 comment replies to trending news videos on YouTube and Twitter…
▽ More
Most Americans agree that misinformation, hate speech and harassment are harmful and inadequately curbed on social media through current moderation practices. In this paper, we aim to understand the discursive strategies employed by people in response to harmful speech in news comments. We conducted a content analysis of more than 6500 comment replies to trending news videos on YouTube and Twitter and identified seven distinct discursive objection strategies (Study 1). We examined the frequency of each strategy's occurrence from the 6500 comment replies, as well as from a second sample of 2004 replies (Study 2). Together, these studies show that people deploy a diversity of discursive strategies when objecting to speech, and reputational attacks are the most common. The resulting classification scheme accounts for different theoretical approaches for expressing objections and offers a comprehensive perspective on grassroots efforts aimed at stop** offensive or problematic speech on campus.
△ Less
Submitted 13 May, 2024;
originally announced May 2024.
-
A Benchmark Dataset for Tornado Detection and Prediction using Full-Resolution Polarimetric Weather Radar Data
Authors:
Mark S. Veillette,
James M. Kurdzo,
Phillip M. Stepanian,
John Y. N. Cho,
Siddharth Samsi,
Joseph McDonald
Abstract:
Weather radar is the primary tool used by forecasters to detect and warn for tornadoes in near-real time. In order to assist forecasters in warning the public, several algorithms have been developed to automatically detect tornadic signatures in weather radar observations. Recently, Machine Learning (ML) algorithms, which learn directly from large amounts of labeled data, have been shown to be hig…
▽ More
Weather radar is the primary tool used by forecasters to detect and warn for tornadoes in near-real time. In order to assist forecasters in warning the public, several algorithms have been developed to automatically detect tornadic signatures in weather radar observations. Recently, Machine Learning (ML) algorithms, which learn directly from large amounts of labeled data, have been shown to be highly effective for this purpose. Since tornadoes are extremely rare events within the corpus of all available radar observations, the selection and design of training datasets for ML applications is critical for the performance, robustness, and ultimate acceptance of ML algorithms. This study introduces a new benchmark dataset, TorNet to support development of ML algorithms in tornado detection and prediction. TorNet contains full-resolution, polarimetric, Level-II WSR-88D data sampled from 10 years of reported storm events. A number of ML baselines for tornado detection are developed and compared, including a novel deep learning (DL) architecture capable of processing raw radar imagery without the need for manual feature extraction required for existing ML algorithms. Despite not benefiting from manual feature engineering or other preprocessing, the DL model shows increased detection performance compared to non-DL and operational baselines. The TorNet dataset, as well as source code and model weights of the DL baseline trained in this work, are made freely available.
△ Less
Submitted 26 January, 2024;
originally announced January 2024.
-
Machine Learning-based Anomaly Detection in Optical Fiber Monitoring
Authors:
Khouloud Abdelli,
Joo Yeon Cho,
Florian Azendorf,
Helmut Griesser,
Carsten Tropschug,
Stephan Pachnicke
Abstract:
Secure and reliable data communication in optical networks is critical for high-speed Internet. However, optical fibers, serving as the data transmission medium providing connectivity to billons of users worldwide, are prone to a variety of anomalies resulting from hard failures (e.g., fiber cuts) and malicious physical attacks (e.g., optical eavesdrop** (fiber tap**)) etc. Such anomalies may…
▽ More
Secure and reliable data communication in optical networks is critical for high-speed Internet. However, optical fibers, serving as the data transmission medium providing connectivity to billons of users worldwide, are prone to a variety of anomalies resulting from hard failures (e.g., fiber cuts) and malicious physical attacks (e.g., optical eavesdrop** (fiber tap**)) etc. Such anomalies may cause network disruption and thereby inducing huge financial and data losses, or compromise the confidentiality of optical networks by gaining unauthorized access to the carried data, or gradually degrade the network operations. Therefore, it is highly required to implement efficient anomaly detection, diagnosis, and localization schemes for enhancing the availability and reliability of optical networks. In this paper, we propose a data driven approach to accurately and quickly detect, diagnose, and localize fiber anomalies including fiber cuts, and optical eavesdrop** attacks. The proposed method combines an autoencoder-based anomaly detection and an attention-based bidirectional gated recurrent unit algorithm, whereby the former is used for fault detection and the latter is adopted for fault diagnosis and localization once an anomaly is detected by the autoencoder. We verify the efficiency of our proposed approach by experiments under various anomaly scenarios using real operational data. The experimental results demonstrate that: (i) the autoencoder detects any fiber fault or anomaly with an F1 score of 96.86%; and (ii) the attention-based bidirectional gated recurrent unit algorithm identifies the the detected anomalies with an average accuracy of 98.2%, and localizes the faults with an average root mean square error of 0.19 m.
△ Less
Submitted 19 March, 2022;
originally announced April 2022.
-
DeepALM: Holistic Optical Network Monitoring based on Machine Learning
Authors:
Joo Yeon Cho,
Jose-Juan Pedreno-Manresa,
Sai Kireet Patri,
Khouloud Abdelli,
Carsten Tropschug,
Jim Zou,
Piotr Rydlichowski
Abstract:
We demonstrate a machine learning-based optical network monitoring system which can integrate fiber monitoring, predictive maintenance of optical hardware, and security information management in a single solution.
We demonstrate a machine learning-based optical network monitoring system which can integrate fiber monitoring, predictive maintenance of optical hardware, and security information management in a single solution.
△ Less
Submitted 25 March, 2022;
originally announced March 2022.
-
ML-based Anomaly Detection in Optical Fiber Monitoring
Authors:
Khouloud Abdelli,
Joo Yeon Cho,
Carsten Tropschug
Abstract:
Secure and reliable data communication in optical networks is critical for high-speed internet. We propose a data driven approach for the anomaly detection and faults identification in optical networks to diagnose physical attacks such as fiber breaks and optical tap**. The proposed methods include an autoencoder-based anomaly detection and an attention-based bidirectional gated recurrent unit a…
▽ More
Secure and reliable data communication in optical networks is critical for high-speed internet. We propose a data driven approach for the anomaly detection and faults identification in optical networks to diagnose physical attacks such as fiber breaks and optical tap**. The proposed methods include an autoencoder-based anomaly detection and an attention-based bidirectional gated recurrent unit algorithm for the fiber fault identification and localization. We verify the efficiency of our methods by experiments under various attack scenarios using real operational data.
△ Less
Submitted 23 February, 2022;
originally announced February 2022.
