-
IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representations
Authors:
Deqing Fu,
Ghazal Khalighinejad,
Ollie Liu,
Bhuwan Dhingra,
Dani Yogatama,
Robin Jia,
Willie Neiswanger
Abstract:
Current foundation models exhibit impressive capabilities when prompted either with text only or with both image and text inputs. But do their capabilities change depending on the input modality? In this work, we propose $\textbf{IsoBench}$, a benchmark dataset containing problems from four major areas: math, science, algorithms, and games. Each example is presented with multiple…
▽ More
Current foundation models exhibit impressive capabilities when prompted either with text only or with both image and text inputs. But do their capabilities change depending on the input modality? In this work, we propose $\textbf{IsoBench}$, a benchmark dataset containing problems from four major areas: math, science, algorithms, and games. Each example is presented with multiple $\textbf{isomorphic representations}$ of inputs, such as visual, textual, and mathematical presentations. IsoBench provides fine-grained feedback to diagnose performance gaps caused by the form of the representation. Across various foundation models, we observe that on the same problem, models have a consistent preference towards textual representations. Most prominently, when evaluated on all IsoBench problems, Claude-3 Opus performs 28.7 points worse when provided with images instead of text; similarly, GPT-4 Turbo is 18.7 points worse and Gemini Pro is 14.9 points worse. Finally, we present two prompting techniques, $\textit{IsoCombination}$ and $\textit{IsoScratchPad}$, which improve model performance by considering combinations of, and translations between, different input representations.
△ Less
Submitted 2 April, 2024; v1 submitted 1 April, 2024;
originally announced April 2024.
-
Extracting Polymer Nanocomposite Samples from Full-Length Documents
Authors:
Ghazal Khalighinejad,
Defne Circi,
L. C. Brinson,
Bhuwan Dhingra
Abstract:
This paper investigates the use of large language models (LLMs) for extracting sample lists of polymer nanocomposites (PNCs) from full-length materials science research papers. The challenge lies in the complex nature of PNC samples, which have numerous attributes scattered throughout the text. The complexity of annotating detailed information on PNCs limits the availability of data, making conven…
▽ More
This paper investigates the use of large language models (LLMs) for extracting sample lists of polymer nanocomposites (PNCs) from full-length materials science research papers. The challenge lies in the complex nature of PNC samples, which have numerous attributes scattered throughout the text. The complexity of annotating detailed information on PNCs limits the availability of data, making conventional document-level relation extraction techniques impractical due to the challenge in creating comprehensive named entity span annotations. To address this, we introduce a new benchmark and an evaluation technique for this task and explore different prompting strategies in a zero-shot manner. We also incorporate self-consistency to improve the performance. Our findings show that even advanced LLMs struggle to extract all of the samples from an article. Finally, we analyze the errors encountered in this process, categorizing them into three main challenges, and discuss potential strategies for future research to overcome them.
△ Less
Submitted 29 February, 2024;
originally announced March 2024.
-
Approximating CKY with Transformers
Authors:
Ghazal Khalighinejad,
Ollie Liu,
Sam Wiseman
Abstract:
We investigate the ability of transformer models to approximate the CKY algorithm, using them to directly predict a sentence's parse and thus avoid the CKY algorithm's cubic dependence on sentence length. We find that on standard constituency parsing benchmarks this approach achieves competitive or better performance than comparable parsers that make use of CKY, while being faster. We also evaluat…
▽ More
We investigate the ability of transformer models to approximate the CKY algorithm, using them to directly predict a sentence's parse and thus avoid the CKY algorithm's cubic dependence on sentence length. We find that on standard constituency parsing benchmarks this approach achieves competitive or better performance than comparable parsers that make use of CKY, while being faster. We also evaluate the viability of this approach for parsing under \textit{random} PCFGs. Here we find that performance declines as the grammar becomes more ambiguous, suggesting that the transformer is not fully capturing the CKY computation. However, we also find that incorporating additional inductive bias is helpful, and we propose a novel approach that makes use of gradients with respect to chart representations in predicting the parse, in analogy with the CKY algorithm being a subgradient of a partition function variant with respect to the chart.
△ Less
Submitted 4 November, 2023; v1 submitted 3 May, 2023;
originally announced May 2023.
-
Gallo** in fast-growth natural merge sorts
Authors:
Elahe Ghasemi,
Vincent Jugé,
Ghazal Khalighinejad,
Helia Yazdanyar
Abstract:
We study the impact of merging routines in merge-based sorting algorithms. More precisely, we focus on the gallo** routine that TimSort uses to merge monotonic sub-arrays, hereafter called runs, and on the impact on the number of element comparisons performed if one uses this routine instead of a naïve merging routine.
This routine was introduced in order to make TimSort more efficient on arra…
▽ More
We study the impact of merging routines in merge-based sorting algorithms. More precisely, we focus on the gallo** routine that TimSort uses to merge monotonic sub-arrays, hereafter called runs, and on the impact on the number of element comparisons performed if one uses this routine instead of a naïve merging routine.
This routine was introduced in order to make TimSort more efficient on arrays with few distinct values. Alas, we prove that, although it makes TimSort sort array with two values in linear time, it does not prevent TimSort from requiring up to $Θ(n \log(n))$ element comparisons to sort arrays of length~$n$ with three distinct values. However, we also prove that slightly modifying TimSort's gallo** routine results in requiring only $\mathcal{O}(n + n \log(σ))$ element comparisons in the worst case, when sorting arrays of length $n$ with $σ$ distinct values.
We do so by focusing on the notion of dual runs, which was introduced in the 1990s, and on the associated dual run-length entropy. This notion is both related to the number of distinct values and to the number of runs in an array, which came with its own run-length entropy that was used to explain TimSort's otherwise "supernatural" efficiency. We also introduce new notions of fast- and middle-growth for natural merge sorts (i.e., algorithms based on merging runs), which are found in several merge sorting algorithms similar to TimSort.
We prove that algorithms with the fast- or middle-growth property, provided that they use our variant of TimSort's gallo** routine for merging runs, are as efficient as possible at sorting arrays with low run-induced or dual-run-induced complexities.
△ Less
Submitted 1 December, 2023; v1 submitted 7 December, 2020;
originally announced December 2020.
