-
Crossing Linguistic Horizons: Finetuning and Comprehensive Evaluation of Vietnamese Large Language Models
Authors:
Sang T. Truong,
Duc Q. Nguyen,
Toan Nguyen,
Dong D. Le,
Nhi N. Truong,
Tho Quan,
Sanmi Koyejo
Abstract:
Recent advancements in large language models (LLMs) have underscored their importance in the evolution of artificial intelligence. However, despite extensive pretraining on multilingual datasets, available open-sourced LLMs exhibit limited effectiveness in processing Vietnamese. The challenge is exacerbated by the absence of systematic benchmark datasets and metrics tailored for Vietnamese LLM eva…
▽ More
Recent advancements in large language models (LLMs) have underscored their importance in the evolution of artificial intelligence. However, despite extensive pretraining on multilingual datasets, available open-sourced LLMs exhibit limited effectiveness in processing Vietnamese. The challenge is exacerbated by the absence of systematic benchmark datasets and metrics tailored for Vietnamese LLM evaluation. To mitigate these issues, we have finetuned LLMs specifically for Vietnamese and developed a comprehensive evaluation framework encompassing 10 common tasks and 31 metrics. Our evaluation results reveal that the fine-tuned LLMs exhibit enhanced comprehension and generative capabilities in Vietnamese. Moreover, our analysis indicates that models with more parameters can introduce more biases and uncalibrated outputs and the key factor influencing LLM performance is the quality of the training or fine-tuning datasets. These insights underscore the significance of meticulous fine-tuning with high-quality datasets in enhancing LLM performance.
△ Less
Submitted 26 May, 2024; v1 submitted 5 March, 2024;
originally announced March 2024.
-
An Experimental Design Framework for Label-Efficient Supervised Finetuning of Large Language Models
Authors:
Gantavya Bhatt,
Yifang Chen,
Arnav M. Das,
Jifan Zhang,
Sang T. Truong,
Stephen Mussmann,
Yinglun Zhu,
Jeffrey Bilmes,
Simon S. Du,
Kevin Jamieson,
Jordan T. Ash,
Robert D. Nowak
Abstract:
Supervised finetuning (SFT) on instruction datasets has played a crucial role in achieving the remarkable zero-shot generalization capabilities observed in modern large language models (LLMs). However, the annotation efforts required to produce high quality responses for instructions are becoming prohibitively expensive, especially as the number of tasks spanned by instruction datasets continues t…
▽ More
Supervised finetuning (SFT) on instruction datasets has played a crucial role in achieving the remarkable zero-shot generalization capabilities observed in modern large language models (LLMs). However, the annotation efforts required to produce high quality responses for instructions are becoming prohibitively expensive, especially as the number of tasks spanned by instruction datasets continues to increase. Active learning is effective in identifying useful subsets of samples to annotate from an unlabeled pool, but its high computational cost remains a barrier to its widespread applicability in the context of LLMs. To mitigate the annotation cost of SFT and circumvent the computational bottlenecks of active learning, we propose using experimental design. Experimental design techniques select the most informative samples to label, and typically maximize some notion of uncertainty and/or diversity. In our work, we implement a framework that evaluates several existing and novel experimental design techniques and find that these methods consistently yield significant gains in label efficiency with little computational overhead. On generative tasks, our methods achieve the same generalization performance with only $50\%$ of annotation cost required by random sampling.
△ Less
Submitted 7 July, 2024; v1 submitted 12 January, 2024;
originally announced January 2024.
-
DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models
Authors:
Boxin Wang,
Weixin Chen,
Hengzhi Pei,
Chulin Xie,
Mintong Kang,
Chenhui Zhang,
Chejian Xu,
Zidi Xiong,
Ritik Dutta,
Rylan Schaeffer,
Sang T. Truong,
Simran Arora,
Mantas Mazeika,
Dan Hendrycks,
Zinan Lin,
Yu Cheng,
Sanmi Koyejo,
Dawn Song,
Bo Li
Abstract:
Generative Pre-trained Transformer (GPT) models have exhibited exciting progress in their capabilities, capturing the interest of practitioners and the public alike. Yet, while the literature on the trustworthiness of GPT models remains limited, practitioners have proposed employing capable GPT models for sensitive applications such as healthcare and finance -- where mistakes can be costly. To thi…
▽ More
Generative Pre-trained Transformer (GPT) models have exhibited exciting progress in their capabilities, capturing the interest of practitioners and the public alike. Yet, while the literature on the trustworthiness of GPT models remains limited, practitioners have proposed employing capable GPT models for sensitive applications such as healthcare and finance -- where mistakes can be costly. To this end, this work proposes a comprehensive trustworthiness evaluation for large language models with a focus on GPT-4 and GPT-3.5, considering diverse perspectives -- including toxicity, stereotype bias, adversarial robustness, out-of-distribution robustness, robustness on adversarial demonstrations, privacy, machine ethics, and fairness. Based on our evaluations, we discover previously unpublished vulnerabilities to trustworthiness threats. For instance, we find that GPT models can be easily misled to generate toxic and biased outputs and leak private information in both training data and conversation history. We also find that although GPT-4 is usually more trustworthy than GPT-3.5 on standard benchmarks, GPT-4 is more vulnerable given jailbreaking system or user prompts, potentially because GPT-4 follows (misleading) instructions more precisely. Our work illustrates a comprehensive trustworthiness evaluation of GPT models and sheds light on the trustworthiness gaps. Our benchmark is publicly available at https://decodingtrust.github.io/ ; our dataset can be previewed at https://huggingface.co/datasets/AI-Secure/DecodingTrust ; a concise version of this work is at https://openreview.net/pdf?id=kaHpo8OZw2 .
△ Less
Submitted 26 February, 2024; v1 submitted 20 June, 2023;
originally announced June 2023.
-
Systematic Analysis of Biomolecular Conformational Ensembles with PENSA
Authors:
Martin Vögele,
Neil J. Thomson,
Sang T. Truong,
Jasper McAvity,
Ulrich Zachariae,
Ron O. Dror
Abstract:
Molecular simulations enable the study of biomolecules and their dynamics on an atomistic scale. A common task is to compare several simulation conditions - like mutations or different ligands - to find significant differences and interrelations between them. However, the large amount of data produced for ever larger and more complex systems often renders it difficult to identify the structural fe…
▽ More
Molecular simulations enable the study of biomolecules and their dynamics on an atomistic scale. A common task is to compare several simulation conditions - like mutations or different ligands - to find significant differences and interrelations between them. However, the large amount of data produced for ever larger and more complex systems often renders it difficult to identify the structural features that are relevant for a particular phenomenon. We present a flexible software package named PENSA that enables a comprehensive and thorough investigation into biomolecular conformational ensembles. It provides a wide variety of featurizations and feature transformations that allow for a complete representation of biomolecules like proteins and nucleic acids, including water and ion cavities within the biomolecular structure, thus avoiding bias that would come with manual selection of features. PENSA implements various methods to systematically compare the distributions of these features across ensembles to find the significant differences between them and identify regions of interest. It also includes a novel approach to quantify the state-specific information between two regions of a biomolecule which allows, e.g., the tracing of information flow to identify signaling pathways. PENSA also comes with convenient tools for loading data and visualizing results in ways that make them quick to process and easy to interpret. PENSA is an open-source Python library maintained at https://github.com/drorlab/pensa along with an example workflow and a tutorial. Here we demonstrate its usefulness in real-world examples by showing how it helps to determine molecular mechanisms efficiently.
△ Less
Submitted 5 December, 2022;
originally announced December 2022.
-
Quantum Neural Architecture Search with Quantum Circuits Metric and Bayesian Optimization
Authors:
Trong Duong,
Sang T. Truong,
Minh Tam,
Bao Bach,
Ju-Young Ryu,
June-Koo Kevin Rhee
Abstract:
Quantum neural networks are promising for a wide range of applications in the Noisy Intermediate-Scale Quantum era. As such, there is an increasing demand for automatic quantum neural architecture search. We tackle this challenge by designing a quantum circuits metric for Bayesian optimization with Gaussian process. To this goal, we propose a new quantum gates distance that characterizes the gates…
▽ More
Quantum neural networks are promising for a wide range of applications in the Noisy Intermediate-Scale Quantum era. As such, there is an increasing demand for automatic quantum neural architecture search. We tackle this challenge by designing a quantum circuits metric for Bayesian optimization with Gaussian process. To this goal, we propose a new quantum gates distance that characterizes the gates' action over every quantum state and provide a theoretical perspective on its geometrical properties. Our approach significantly outperforms the benchmark on three empirical quantum machine learning problems including training a quantum generative adversarial network, solving combinatorial optimization in the MaxCut problem, and simulating quantum Fourier transform. Our method can be extended to characterize behaviors of various quantum machine learning models.
△ Less
Submitted 28 June, 2022;
originally announced June 2022.
-
A Simple and Scalable Tensor Completion Algorithm via Latent Invariant Constraint for Recommendation System
Authors:
Tung Nguyen,
Sang T. Truong,
Jeffrey Uhlmann
Abstract:
In this paper we provide a latent-variable formulation and solution to the recommender system (RS) problem in terms of a fundamental property that any reasonable solution should be expected to satisfy. Specifically, we examine a novel tensor completion method to efficiently and accurately learn parameters of a model for the unobservable personal preferences that underly user ratings. By regularizi…
▽ More
In this paper we provide a latent-variable formulation and solution to the recommender system (RS) problem in terms of a fundamental property that any reasonable solution should be expected to satisfy. Specifically, we examine a novel tensor completion method to efficiently and accurately learn parameters of a model for the unobservable personal preferences that underly user ratings. By regularizing the tensor decomposition with a single latent invariant, we achieve three properties for a reliable recommender system: (1) uniqueness of the tensor completion result with minimal assumptions, (2) unit consistency that is independent of arbitrary preferences of users, and (3) a consensus ordering guarantee that provides consistent ranking between observed and unobserved rating scores. Our algorithm leads to a simple and elegant recommendation framework that has linear computational complexity and with no hyperparameter tuning. We provide empirical results demonstrating that the approach significantly outperforms current state-of-the-art methods.
△ Less
Submitted 3 July, 2022; v1 submitted 27 June, 2022;
originally announced June 2022.
-
The Effect of Education on Smoking Decisions in the United States
Authors:
Sang T. Truong
Abstract:
This paper explores the link between education and the decision to start smoking as well as the decision to quit smoking. Data is gathered from IPUMS CPS and Centers for Disease Control and Prevention. Probit analysis (with the use of probability weight and robust standard error) indicates that every additional year of education will reduce the 2.3 percentage point of the smoking probability and w…
▽ More
This paper explores the link between education and the decision to start smoking as well as the decision to quit smoking. Data is gathered from IPUMS CPS and Centers for Disease Control and Prevention. Probit analysis (with the use of probability weight and robust standard error) indicates that every additional year of education will reduce the 2.3 percentage point of the smoking probability and will add 3.53 percentage point in quitting likelihood, holding home restriction, public restriction, cigarette price, family income, age, gender, race, and ethnicity constant. I believe that tobacco epidemic is a serious global issue that may be mitigated by using careful regulations on smoking restriction and education.
△ Less
Submitted 14 November, 2020;
originally announced November 2020.
