-
OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web
Authors:
Raghav Kapoor,
Yash Parag Butala,
Melisa Russak,
**g Yu Koh,
Kiran Kamble,
Waseem Alshikh,
Ruslan Salakhutdinov
Abstract:
For decades, human-computer interaction has fundamentally been manual. Even today, almost all productive work done on the computer necessitates human input at every step. Autonomous virtual agents represent an exciting step in automating many of these menial tasks. Virtual agents would empower users with limited technical proficiency to harness the full possibilities of computer systems. They coul…
▽ More
For decades, human-computer interaction has fundamentally been manual. Even today, almost all productive work done on the computer necessitates human input at every step. Autonomous virtual agents represent an exciting step in automating many of these menial tasks. Virtual agents would empower users with limited technical proficiency to harness the full possibilities of computer systems. They could also enable the efficient streamlining of numerous computer tasks, ranging from calendar management to complex travel bookings, with minimal human intervention. In this paper, we introduce OmniACT, the first-of-a-kind dataset and benchmark for assessing an agent's capability to generate executable programs to accomplish computer tasks. Our scope extends beyond traditional web automation, covering a diverse range of desktop applications. The dataset consists of fundamental tasks such as "Play the next song", as well as longer horizon tasks such as "Send an email to John Doe mentioning the time and place to meet". Specifically, given a pair of screen image and a visually-grounded natural language task, the goal is to generate a script capable of fully executing the task. We run several strong baseline language model agents on our benchmark. The strongest baseline, GPT-4, performs the best on our benchmark However, its performance level still reaches only 15% of the human proficiency in generating executable scripts capable of completing the task, demonstrating the challenge of our task for conventional web agents. Our benchmark provides a platform to measure and evaluate the progress of language model agents in automating computer tasks and motivates future work towards building multimodal models that bridge large language models and the visual grounding of computer screens.
△ Less
Submitted 28 February, 2024; v1 submitted 27 February, 2024;
originally announced February 2024.
-
Becoming self-instruct: introducing early stop** criteria for minimal instruct tuning
Authors:
Waseem AlShikh,
Manhal Daaboul,
Kirk Goddard,
Brock Imel,
Kiran Kamble,
Parikshith Kulkarni,
Melisa Russak
Abstract:
In this paper, we introduce the Instruction Following Score (IFS), a metric that detects language models' ability to follow instructions. The metric has a dual purpose. First, IFS can be used to distinguish between base and instruct models. We benchmark publicly available base and instruct models, and show that the ratio of well formatted responses to partial and full sentences can be an effective…
▽ More
In this paper, we introduce the Instruction Following Score (IFS), a metric that detects language models' ability to follow instructions. The metric has a dual purpose. First, IFS can be used to distinguish between base and instruct models. We benchmark publicly available base and instruct models, and show that the ratio of well formatted responses to partial and full sentences can be an effective measure between those two model classes. Secondly, the metric can be used as an early stop** criteria for instruct tuning. We compute IFS for Supervised Fine-Tuning (SFT) of 7B and 13B LLaMA models, showing that models learn to follow instructions relatively early in the training process, and the further finetuning can result in changes in the underlying base model semantics. As an example of semantics change we show the objectivity of model predictions, as defined by an auxiliary metric ObjecQA. We show that in this particular case, semantic changes are the steepest when the IFS tends to plateau. We hope that decomposing instruct tuning into IFS and semantic factors starts a new trend in better controllable instruct tuning and opens possibilities for designing minimal instruct interfaces querying foundation models.
△ Less
Submitted 5 July, 2023;
originally announced July 2023.
-
Performance Evaluation of Vanilla, Residual, and Dense 2D U-Net Architectures for Skull Strip** of Augmented 3D T1-weighted MRI Head Scans
Authors:
Anway S. Pimpalkar,
Rashmika K. Patole,
Ketaki D. Kamble,
Mahesh H. Shindikar
Abstract:
Skull Strip** is a requisite preliminary step in most diagnostic neuroimaging applications. Manual Skull Strip** methods define the gold standard for the domain but are time-consuming and challenging to integrate into processing pipelines with a high number of data samples. Automated methods are an active area of research for head MRI segmentation, especially deep learning methods such as U-Ne…
▽ More
Skull Strip** is a requisite preliminary step in most diagnostic neuroimaging applications. Manual Skull Strip** methods define the gold standard for the domain but are time-consuming and challenging to integrate into processing pipelines with a high number of data samples. Automated methods are an active area of research for head MRI segmentation, especially deep learning methods such as U-Net architecture implementations. This study compares Vanilla, Residual, and Dense 2D U-Net architectures for Skull Strip**. The Dense 2D U-Net architecture outperforms the Vanilla and Residual counterparts by achieving an accuracy of 99.75% on a test dataset. It is observed that dense interconnections in a U-Net encourage feature reuse across layers of the architecture and allow for shallower models with the strengths of a deeper network.
△ Less
Submitted 21 January, 2023; v1 submitted 29 November, 2022;
originally announced November 2022.
