-
Leveraging Machine-Generated Rationales to Facilitate Social Meaning Detection in Conversations
Authors:
Ritam Dutt,
Zhen Wu,
Kelly Shi,
Divyanshu Sheth,
Prakhar Gupta,
Carolyn Penstein Rose
Abstract:
We present a generalizable classification approach that leverages Large Language Models (LLMs) to facilitate the detection of implicitly encoded social meaning in conversations. We design a multi-faceted prompt to extract a textual explanation of the reasoning that connects visible cues to underlying social meanings. These extracted explanations or rationales serve as augmentations to the conversa…
▽ More
We present a generalizable classification approach that leverages Large Language Models (LLMs) to facilitate the detection of implicitly encoded social meaning in conversations. We design a multi-faceted prompt to extract a textual explanation of the reasoning that connects visible cues to underlying social meanings. These extracted explanations or rationales serve as augmentations to the conversational text to facilitate dialogue understanding and transfer. Our empirical results over 2,340 experimental settings demonstrate the significant positive impact of adding these rationales. Our findings hold true for in-domain classification, zero-shot, and few-shot domain transfer for two different social meaning detection tasks, each spanning two different corpora.
△ Less
Submitted 27 June, 2024;
originally announced June 2024.
-
CodeBenchGen: Creating Scalable Execution-based Code Generation Benchmarks
Authors:
Yiqing Xie,
Alex Xie,
Divyanshu Sheth,
Pengfei Liu,
Daniel Fried,
Carolyn Rose
Abstract:
To facilitate evaluation of code generation systems across diverse scenarios, we present CodeBenchGen, a framework to create scalable execution-based benchmarks that only requires light guidance from humans. Specifically, we leverage a large language model (LLM) to convert an arbitrary piece of code into an evaluation example, including test cases for execution-based evaluation. We illustrate the…
▽ More
To facilitate evaluation of code generation systems across diverse scenarios, we present CodeBenchGen, a framework to create scalable execution-based benchmarks that only requires light guidance from humans. Specifically, we leverage a large language model (LLM) to convert an arbitrary piece of code into an evaluation example, including test cases for execution-based evaluation. We illustrate the usefulness of our framework by creating a dataset, Exec-CSN, which includes 1,931 examples involving 293 libraries revised from code in 367 GitHub repositories taken from the CodeSearchNet dataset. To demonstrate the complexity and solvability of examples in Exec-CSN, we present a human study demonstrating that 81.3% of the examples can be solved by humans and 61% are rated as "requires effort to solve". We conduct code generation experiments on open-source and proprietary models and analyze the performance of both humans and models. We provide the code at https://github.com/Veronicium/CodeBenchGen.
△ Less
Submitted 7 May, 2024; v1 submitted 31 March, 2024;
originally announced April 2024.
-
Reconstructing Materials Tetrahedron: Challenges in Materials Information Extraction
Authors:
Kausik Hira,
Mohd Zaki,
Dhruvil Sheth,
Mausam,
N M Anoop Krishnan
Abstract:
The discovery of new materials has a documented history of propelling human progress for centuries and more. The behaviour of a material is a function of its composition, structure, and properties, which further depend on its processing and testing conditions. Recent developments in deep learning and natural language processing have enabled information extraction at scale from published literature…
▽ More
The discovery of new materials has a documented history of propelling human progress for centuries and more. The behaviour of a material is a function of its composition, structure, and properties, which further depend on its processing and testing conditions. Recent developments in deep learning and natural language processing have enabled information extraction at scale from published literature such as peer-reviewed publications, books, and patents. However, this information is spread in multiple formats, such as tables, text, and images, and with little or no uniformity in reporting style giving rise to several machine learning challenges. Here, we discuss, quantify, and document these challenges in automated information extraction (IE) from materials science literature towards the creation of a large materials science knowledge base. Specifically, we focus on IE from text and tables and outline several challenges with examples. We hope the present work inspires researchers to address the challenges in a coherent fashion, providing a fillip to IE towards develo** a materials knowledge base.
△ Less
Submitted 26 April, 2024; v1 submitted 12 October, 2023;
originally announced October 2023.
-
Rationale-Guided Few-Shot Classification to Detect Abusive Language
Authors:
Punyajoy Saha,
Divyanshu Sheth,
Kushal Kedia,
Binny Mathew,
Animesh Mukherjee
Abstract:
Abusive language is a concerning problem in online social media. Past research on detecting abusive language covers different platforms, languages, demographies, etc. However, models trained using these datasets do not perform well in cross-domain evaluation settings. To overcome this, a common strategy is to use a few samples from the target domain to train models to get better performance in tha…
▽ More
Abusive language is a concerning problem in online social media. Past research on detecting abusive language covers different platforms, languages, demographies, etc. However, models trained using these datasets do not perform well in cross-domain evaluation settings. To overcome this, a common strategy is to use a few samples from the target domain to train models to get better performance in that domain (cross-domain few-shot training). However, this might cause the models to overfit the artefacts of those samples. A compelling solution could be to guide the models toward rationales, i.e., spans of text that justify the text's label. This method has been found to improve model performance in the in-domain setting across various NLP tasks. In this paper, we propose RGFS (Rationale-Guided Few-Shot Classification) for abusive language detection. We first build a multitask learning setup to jointly learn rationales, targets, and labels, and find a significant improvement of 6% macro F1 on the rationale detection task over training solely rationale classifiers. We introduce two rationale-integrated BERT-based architectures (the RGFS models) and evaluate our systems over five different abusive language datasets, finding that in the few-shot classification setting, RGFS-based models outperform baseline models by about 7% in macro F1 scores and perform competitively to models finetuned on other source domains. Furthermore, RGFS-based models outperform LIME/SHAP-based approaches in terms of plausibility and are close in performance in terms of faithfulness.
△ Less
Submitted 27 July, 2023; v1 submitted 30 November, 2022;
originally announced November 2022.
-
A Unified Framework for Pun Generation with Humor Principles
Authors:
Yufei Tian,
Divyanshu Sheth,
Nanyun Peng
Abstract:
We propose a unified framework to generate both homophonic and homographic puns to resolve the split-up in existing works. Specifically, we incorporate three linguistic attributes of puns to the language models: ambiguity, distinctiveness, and surprise. Our framework consists of three parts: 1) a context words/phrases selector to promote the aforementioned attributes, 2) a generation model trained…
▽ More
We propose a unified framework to generate both homophonic and homographic puns to resolve the split-up in existing works. Specifically, we incorporate three linguistic attributes of puns to the language models: ambiguity, distinctiveness, and surprise. Our framework consists of three parts: 1) a context words/phrases selector to promote the aforementioned attributes, 2) a generation model trained on non-pun sentences to incorporate the context words/phrases into the generation output, and 3) a label predictor that learns the structure of puns which is used to steer the generation model at inference time. Evaluation results on both pun types demonstrate the efficacy of our model over strong baselines.
△ Less
Submitted 24 October, 2022;
originally announced October 2022.
-
CABACE: Injecting Character Sequence Information and Domain Knowledge for Enhanced Acronym and Long-Form Extraction
Authors:
Nithish Kannen,
Divyanshu Sheth,
Abhranil Chandra,
Shubhraneel Pal
Abstract:
Acronyms and long-forms are commonly found in research documents, more so in documents from scientific and legal domains. Many acronyms used in such documents are domain-specific and are very rarely found in normal text corpora. Owing to this, transformer-based NLP models often detect OOV (Out of Vocabulary) for acronym tokens, especially for non-English languages, and their performance suffers wh…
▽ More
Acronyms and long-forms are commonly found in research documents, more so in documents from scientific and legal domains. Many acronyms used in such documents are domain-specific and are very rarely found in normal text corpora. Owing to this, transformer-based NLP models often detect OOV (Out of Vocabulary) for acronym tokens, especially for non-English languages, and their performance suffers while linking acronyms to their long forms during extraction. Moreover, pretrained transformer models like BERT are not specialized to handle scientific and legal documents. With these points being the overarching motivation behind this work, we propose a novel framework CABACE: Character-Aware BERT for ACronym Extraction, which takes into account character sequences in text and is adapted to scientific and legal domains by masked language modelling. We further use an objective with an augmented loss function, adding the max loss and mask loss terms to the standard cross-entropy loss for training CABACE. We further leverage pseudo labelling and adversarial data generation to improve the generalizability of the framework. Experimental results prove the superiority of the proposed framework in comparison to various baselines. Additionally, we show that the proposed framework is better suited than baseline models for zero-shot generalization to non-English languages, thus reinforcing the effectiveness of our approach. Our team BacKGProp secured the highest scores on the French dataset, second-highest on Danish and Vietnamese, and third-highest in the English-Legal dataset on the global leaderboard for the acronym extraction (AE) shared task at SDU AAAI-22.
△ Less
Submitted 25 December, 2021;
originally announced December 2021.
-
Perturbation CheckLists for Evaluating NLG Evaluation Metrics
Authors:
Ananya B. Sai,
Tanay Dixit,
Dev Yashpal Sheth,
Sreyas Mohan,
Mitesh M. Khapra
Abstract:
Natural Language Generation (NLG) evaluation is a multifaceted task requiring assessment of multiple desirable criteria, e.g., fluency, coherency, coverage, relevance, adequacy, overall quality, etc. Across existing datasets for 6 NLG tasks, we observe that the human evaluation scores on these multiple criteria are often not correlated. For example, there is a very low correlation between human sc…
▽ More
Natural Language Generation (NLG) evaluation is a multifaceted task requiring assessment of multiple desirable criteria, e.g., fluency, coherency, coverage, relevance, adequacy, overall quality, etc. Across existing datasets for 6 NLG tasks, we observe that the human evaluation scores on these multiple criteria are often not correlated. For example, there is a very low correlation between human scores on fluency and data coverage for the task of structured data to text generation. This suggests that the current recipe of proposing new automatic evaluation metrics for NLG by showing that they correlate well with scores assigned by humans for a single criteria (overall quality) alone is inadequate. Indeed, our extensive study involving 25 automatic evaluation metrics across 6 different tasks and 18 different evaluation criteria shows that there is no single metric which correlates well with human scores on all desirable criteria, for most NLG tasks. Given this situation, we propose CheckLists for better design and evaluation of automatic metrics. We design templates which target a specific criteria (e.g., coverage) and perturb the output such that the quality gets affected only along this specific criteria (e.g., the coverage drops). We show that existing evaluation metrics are not robust against even such simple perturbations and disagree with scores assigned by humans to the perturbed output. The proposed templates thus allow for a fine-grained assessment of automatic evaluation metrics exposing their limitations and will facilitate better design, analysis and evaluation of such metrics.
△ Less
Submitted 13 September, 2021;
originally announced September 2021.
-
Develo** and Evaluating Deep Neural Network-based Denoising for Nanoparticle TEM Images with Ultra-low Signal-to-Noise
Authors:
Joshua L. Vincent,
Ramon Manzorro,
Sreyas Mohan,
Binh Tang,
Dev Y. Sheth,
Eero P. Simoncelli,
David S. Matteson,
Carlos Fernandez-Granda,
Peter A. Crozier
Abstract:
A deep convolutional neural network has been developed to denoise atomic-resolution TEM image datasets of nanoparticles acquired using direct electron counting detectors, for applications where the image signal is severely limited by shot noise. The network was applied to a model system of CeO2-supported Pt nanoparticles. We leverage multislice image simulations to generate a large and flexible da…
▽ More
A deep convolutional neural network has been developed to denoise atomic-resolution TEM image datasets of nanoparticles acquired using direct electron counting detectors, for applications where the image signal is severely limited by shot noise. The network was applied to a model system of CeO2-supported Pt nanoparticles. We leverage multislice image simulations to generate a large and flexible dataset for training and testing the network. The proposed network outperforms state-of-the-art denoising methods by a significant margin both on simulated and experimental test data. Factors contributing to the performance are identified, including most importantly (a) the geometry of the images used during training and (b) the size of the network's receptive field. Through a gradient-based analysis, we investigate the mechanisms learned by the network to denoise experimental images. This shows that the network exploits global and local information in the noisy measurements, for example, by adapting its filtering approach when it encounters atomic-level defects at the nanoparticle surface. Extensive analysis has been done to characterize the network's ability to correctly predict the exact atomic structure at the nanoparticle surface. Finally, we develop an approach based on the log-likelihood ratio test that provides a quantitative measure of the agreement between the noisy observation and the atomic-level structure in the network-denoised image.
△ Less
Submitted 17 March, 2021; v1 submitted 19 January, 2021;
originally announced January 2021.
-
Unsupervised Deep Video Denoising
Authors:
Dev Yashpal Sheth,
Sreyas Mohan,
Joshua L. Vincent,
Ramon Manzorro,
Peter A. Crozier,
Mitesh M. Khapra,
Eero P. Simoncelli,
Carlos Fernandez-Granda
Abstract:
Deep convolutional neural networks (CNNs) for video denoising are typically trained with supervision, assuming the availability of clean videos. However, in many applications, such as microscopy, noiseless videos are not available. To address this, we propose an Unsupervised Deep Video Denoiser (UDVD), a CNN architecture designed to be trained exclusively with noisy data. The performance of UDVD i…
▽ More
Deep convolutional neural networks (CNNs) for video denoising are typically trained with supervision, assuming the availability of clean videos. However, in many applications, such as microscopy, noiseless videos are not available. To address this, we propose an Unsupervised Deep Video Denoiser (UDVD), a CNN architecture designed to be trained exclusively with noisy data. The performance of UDVD is comparable to the supervised state-of-the-art, even when trained only on a single short noisy video. We demonstrate the promise of our approach in real-world imaging applications by denoising raw video, fluorescence-microscopy and electron-microscopy data. In contrast to many current approaches to video denoising, UDVD does not require explicit motion compensation. This is advantageous because motion compensation is computationally expensive, and can be unreliable when the input data are noisy. A gradient-based analysis reveals that UDVD automatically adapts to local motion in the input noisy videos. Thus, the network learns to perform implicit motion compensation, even though it is only trained for denoising.
△ Less
Submitted 19 August, 2021; v1 submitted 30 November, 2020;
originally announced November 2020.
-
Deep Denoising For Scientific Discovery: A Case Study In Electron Microscopy
Authors:
Sreyas Mohan,
Ramon Manzorro,
Joshua L. Vincent,
Binh Tang,
Dev Yashpal Sheth,
Eero P. Simoncelli,
David S. Matteson,
Peter A. Crozier,
Carlos Fernandez-Granda
Abstract:
Denoising is a fundamental challenge in scientific imaging. Deep convolutional neural networks (CNNs) provide the current state of the art in denoising natural images, where they produce impressive results. However, their potential has barely been explored in the context of scientific imaging. Denoising CNNs are typically trained on real natural images artificially corrupted with simulated noise.…
▽ More
Denoising is a fundamental challenge in scientific imaging. Deep convolutional neural networks (CNNs) provide the current state of the art in denoising natural images, where they produce impressive results. However, their potential has barely been explored in the context of scientific imaging. Denoising CNNs are typically trained on real natural images artificially corrupted with simulated noise. In contrast, in scientific applications, noiseless ground-truth images are usually not available. To address this issue, we propose a simulation-based denoising (SBD) framework, in which CNNs are trained on simulated images. We test the framework on data obtained from transmission electron microscopy (TEM), an imaging technique with widespread applications in material science, biology, and medicine. SBD outperforms existing techniques by a wide margin on a simulated benchmark dataset, as well as on real data. Apart from the denoised images, SBD generates likelihood maps to visualize the agreement between the structure of the denoised image and the observed data. Our results reveal shortcomings of state-of-the-art denoising architectures, such as their small field-of-view: substantially increasing the field-of-view of the CNNs allows them to exploit non-local periodic patterns in the data, which is crucial at high noise levels. In addition, we analyze the generalization capability of SBD, demonstrating that the trained networks are robust to variations of imaging parameters and of the underlying signal structure. Finally, we release the first publicly available benchmark dataset of TEM images, containing 18,000 examples.
△ Less
Submitted 13 July, 2021; v1 submitted 24 October, 2020;
originally announced October 2020.
-
Halo Orbits around $L_1$ and $L_2$ in the Photogravitational Sun-Earth System with Oblateness
Authors:
Dhwani Sheth,
Thomas V. O
Abstract:
The Photogravitational Restricted Three Body Problem with oblateness has been studied to obtain halo orbits around the Lagrangian points $L_1$ and $L_2$ of the Sun-Earth system in which the Sun is taken as radiating and the Earth as an oblate spheroid. The halo orbits corresponding to fourth and fifth order approximations around $L_1$ and $L_2$ for actual oblateness of the Earth and for different…
▽ More
The Photogravitational Restricted Three Body Problem with oblateness has been studied to obtain halo orbits around the Lagrangian points $L_1$ and $L_2$ of the Sun-Earth system in which the Sun is taken as radiating and the Earth as an oblate spheroid. The halo orbits corresponding to fourth and fifth order approximations around $L_1$ and $L_2$ for actual oblateness of the Earth and for different radiation pressures for the Sun are displayed graphically. The time period of halo orbits around $L_1$ decreases with increase in oblateness and increases with increase in radiation pressure. A reverse effect is observed due to increase in oblateness and radiation pressure on time period of orbits around $L_2$. It is also observed that halo orbits around $L_1$ shifts towards the source of radiation due to increase in both radiation pressure and oblateness. However, halo orbits around $L_2$ shifts towards the source of radiation due to increase in radiation but recedes with increase in oblateness.
△ Less
Submitted 21 March, 2020;
originally announced March 2020.
-
Smart Laptop Bag with Machine Learning for Activity Recognition
Authors:
Dwij Sukeshkumar Sheth,
Shantanu Singh,
Prakhar S Mathur,
Vydeki D
Abstract:
In todays world of smart living, the smart laptop bag, presented in this paper, provides a better solution to keep track of our precious possessions and monitoring them in real time. As the world moves towards a much tech-savvy direction, the novel laptop bag discussed here facilitates the user to perform location tracking, ambiance monitoring, user-state monitoring etc. in one device. The innovat…
▽ More
In todays world of smart living, the smart laptop bag, presented in this paper, provides a better solution to keep track of our precious possessions and monitoring them in real time. As the world moves towards a much tech-savvy direction, the novel laptop bag discussed here facilitates the user to perform location tracking, ambiance monitoring, user-state monitoring etc. in one device. The innovative design uses cloud computing and machine learning algorithms to monitor the health of the user and many parameters of the bag. The emergency alert system in this bag could be trained to send appropriate notifications to emergency contacts of the user, in case of abnormal health conditions or theft of the bag. The experimental smart laptop bag uses deep neural network, which was trained and tested over the various parameters from the bag and produces above 95% accurate results.
△ Less
Submitted 14 April, 2019;
originally announced April 2019.
-
Galois coverings of Schreier graphs of groups generated by bounded automata
Authors:
Asif Shaikh,
Daniele D'Angeli,
Hemant Bhate,
Dilip Sheth
Abstract:
We give a characterization of the covering Schreier graphs of groups generated by bounded automata to be Galois. We also investigate the zeta and $L$ functions of Schreier graphs of few groups namely the Grigorchuk group, Gupta-Sidki $p$ group, Gupta-Fabrykowski group and BSV torsion-free group.
We give a characterization of the covering Schreier graphs of groups generated by bounded automata to be Galois. We also investigate the zeta and $L$ functions of Schreier graphs of few groups namely the Grigorchuk group, Gupta-Sidki $p$ group, Gupta-Fabrykowski group and BSV torsion-free group.
△ Less
Submitted 9 November, 2017;
originally announced November 2017.
-
What Are People Tweeting about Zika? An Exploratory Study Concerning Symptoms, Treatment, Transmission, and Prevention
Authors:
Michele Miller,
Dr. Tanvi Banerjee,
RoopTeja Muppalla,
Dr. William Romine,
Dr. Amit Sheth
Abstract:
The purpose of this study was to do a dataset distribution analysis, a classification performance analysis, and a topical analysis concerning what people are tweeting about four disease characteristics: symptoms, transmission, prevention, and treatment. A combination of natural language processing and machine learning techniques were used to determine what people are tweeting about Zika. Specifica…
▽ More
The purpose of this study was to do a dataset distribution analysis, a classification performance analysis, and a topical analysis concerning what people are tweeting about four disease characteristics: symptoms, transmission, prevention, and treatment. A combination of natural language processing and machine learning techniques were used to determine what people are tweeting about Zika. Specifically, a two-stage classifier system was built to find relevant tweets on Zika, and then categorize these into the four disease categories. Tweets in each disease category were then examined using latent dirichlet allocation (LDA) to determine the five main tweet topics for each disease characteristic. Results 1,234,605 tweets were collected. Tweets by males and females were similar (28% and 23% respectively). The classifier performed well on the training and test data for relevancy (F=0.87 and 0.99 respectively) and disease characteristics (F=0.79 and 0.90 respectively). Five topics for each category were found and discussed with a focus on the symptoms category. Through this process, we demonstrate how misinformation can be discovered so that public health officials can respond to the tweets with misinformation.
△ Less
Submitted 17 January, 2017;
originally announced January 2017.
