-
ACQUIRED: A Dataset for Answering Counterfactual Questions In Real-Life Videos
Authors:
Te-Lin Wu,
Zi-Yi Dou,
Qingyuan Hu,
Yu Hou,
Nischal Reddy Chandra,
Marjorie Freedman,
Ralph M. Weischedel,
Nanyun Peng
Abstract:
Multimodal counterfactual reasoning is a vital yet challenging ability for AI systems. It involves predicting the outcomes of hypothetical circumstances based on vision and language inputs, which enables AI models to learn from failures and explore hypothetical scenarios. Despite its importance, there are only a few datasets targeting the counterfactual reasoning abilities of multimodal models. Am…
▽ More
Multimodal counterfactual reasoning is a vital yet challenging ability for AI systems. It involves predicting the outcomes of hypothetical circumstances based on vision and language inputs, which enables AI models to learn from failures and explore hypothetical scenarios. Despite its importance, there are only a few datasets targeting the counterfactual reasoning abilities of multimodal models. Among them, they only cover reasoning over synthetic environments or specific types of events (e.g. traffic collisions), making them hard to reliably benchmark the model generalization ability in diverse real-world scenarios and reasoning dimensions. To overcome these limitations, we develop a video question answering dataset, ACQUIRED: it consists of 3.9K annotated videos, encompassing a wide range of event types and incorporating both first and third-person viewpoints, which ensures a focus on real-world diversity. In addition, each video is annotated with questions that span three distinct dimensions of reasoning, including physical, social, and temporal, which can comprehensively evaluate the model counterfactual abilities along multiple aspects. We benchmark our dataset against several state-of-the-art language-only and multimodal models and experimental results demonstrate a significant performance gap (>13%) between models and humans. The findings suggest that multimodal counterfactual reasoning remains an open challenge and ACQUIRED is a comprehensive and reliable benchmark for inspiring future research in this direction.
△ Less
Submitted 2 November, 2023;
originally announced November 2023.
-
EnDex: Evaluation of Dialogue Engagingness at Scale
Authors:
Guangxuan Xu,
Ruibo Liu,
Fabrice Harel-Canada,
Nischal Reddy Chandra,
Nanyun Peng
Abstract:
We propose EnDex, the first human-reaction based model to evaluate dialogue engagingness. EnDex is trained on 80k Reddit-based Engagement Dataset (RED) curated using a novel distant-supervision framework. Engagingness is a key measure that captures high-level quality of AI dialogue systems and closely reflects actual user experience. However, data shortage, plus the abstract and extensive definiti…
▽ More
We propose EnDex, the first human-reaction based model to evaluate dialogue engagingness. EnDex is trained on 80k Reddit-based Engagement Dataset (RED) curated using a novel distant-supervision framework. Engagingness is a key measure that captures high-level quality of AI dialogue systems and closely reflects actual user experience. However, data shortage, plus the abstract and extensive definition of engagingness makes it challenging to develop an automatic metric. Our work departs from mainstream approaches that use synthetic negative examples to train binary classifiers, and instead, proposes a solution using distant-supervision from human-reaction feedback. To support the soundness of our EnDex metric, we offer a theoretical foundation for engagement, an extensive ablation study, and empirical evidence of high correlation on five engagingness related datasets. We will release code, off-the-shelf EnDex model, and a large-scale dataset upon paper publication to facilitate future research.
△ Less
Submitted 22 October, 2022;
originally announced October 2022.
-
Investigation of ephaptic interactions in peripheral nerve of sheep using 6 kHz subthreshold currents
Authors:
James Hope,
Narrendar Ravi Chandra,
Frederique Vanholsbeeck,
Andrew McDaid
Abstract:
The objective of this work was to determine whether application of subthreshold currents to the peripheral nerve increases the excitability of the underlying nerve fibres, and how this increased excitability would alter neural activity as it propagates through the subthreshold currents. Experiments were performed on two Romney cross-breed sheep in vivo, by applying subthreshold currents either at…
▽ More
The objective of this work was to determine whether application of subthreshold currents to the peripheral nerve increases the excitability of the underlying nerve fibres, and how this increased excitability would alter neural activity as it propagates through the subthreshold currents. Experiments were performed on two Romney cross-breed sheep in vivo, by applying subthreshold currents either at the stimulus site or between the stimulus and recording sites. Neural recordings were obtained from nerve cuff implanted on the peroneal or sciatic nerve branches, while stimulus was applied to either the peroneal nerve or pins placed through the lower hindshank. Results showed that subthreshold currents applied to the same site as stimulus increased excitation of underlying nerve fibres (p < 0.0001). With stimulus and subthreshold currents applied to different sites on the peroneal nerve, the primary CAP in the sciatic displayed a temporal shift of -2.5 to -3 us which agreed with statistically significant changes in the CAP waveform (p<0.02). These findings contribute to the understanding of mechanisms in myelinated fibres of subthreshold current neuromodulation therapies.
△ Less
Submitted 4 December, 2019;
originally announced December 2019.
