-
InCA: Rethinking In-Car Conversational System Assessment Leveraging Large Language Models
Authors:
Ken E. Friedl,
Abbas Goher Khan,
Soumya Ranjan Sahoo,
Md Rashad Al Hasan Rony,
Jana Germies,
Christian Süß
Abstract:
The assessment of advanced generative large language models (LLMs) poses a significant challenge, given their heightened complexity in recent developments. Furthermore, evaluating the performance of LLM-based applications in various industries, as indicated by Key Performance Indicators (KPIs), is a complex undertaking. This task necessitates a profound understanding of industry use cases and the…
▽ More
The assessment of advanced generative large language models (LLMs) poses a significant challenge, given their heightened complexity in recent developments. Furthermore, evaluating the performance of LLM-based applications in various industries, as indicated by Key Performance Indicators (KPIs), is a complex undertaking. This task necessitates a profound understanding of industry use cases and the anticipated system behavior. Within the context of the automotive industry, existing evaluation metrics prove inadequate for assessing in-car conversational question answering (ConvQA) systems. The unique demands of these systems, where answers may relate to driver or car safety and are confined within the car domain, highlight the limitations of current metrics. To address these challenges, this paper introduces a set of KPIs tailored for evaluating the performance of in-car ConvQA systems, along with datasets specifically designed for these KPIs. A preliminary and comprehensive empirical evaluation substantiates the efficacy of our proposed approach. Furthermore, we investigate the impact of employing varied personas in prompts and found that it enhances the model's capacity to simulate diverse viewpoints in assessments, mirroring how individuals with different backgrounds perceive a topic.
△ Less
Submitted 15 November, 2023; v1 submitted 13 November, 2023;
originally announced November 2023.
-
CarExpert: Leveraging Large Language Models for In-Car Conversational Question Answering
Authors:
Md Rashad Al Hasan Rony,
Christian Suess,
Sinchana Ramakanth Bhat,
Viju Sudhi,
Julia Schneider,
Maximilian Vogel,
Roman Teucher,
Ken E. Friedl,
Soumya Sahoo
Abstract:
Large language models (LLMs) have demonstrated remarkable performance by following natural language instructions without fine-tuning them on domain-specific tasks and data. However, leveraging LLMs for domain-specific question answering suffers from severe limitations. The generated answer tends to hallucinate due to the training data collection time (when using off-the-shelf), complex user uttera…
▽ More
Large language models (LLMs) have demonstrated remarkable performance by following natural language instructions without fine-tuning them on domain-specific tasks and data. However, leveraging LLMs for domain-specific question answering suffers from severe limitations. The generated answer tends to hallucinate due to the training data collection time (when using off-the-shelf), complex user utterance and wrong retrieval (in retrieval-augmented generation). Furthermore, due to the lack of awareness about the domain and expected output, such LLMs may generate unexpected and unsafe answers that are not tailored to the target domain. In this paper, we propose CarExpert, an in-car retrieval-augmented conversational question-answering system leveraging LLMs for different tasks. Specifically, CarExpert employs LLMs to control the input, provide domain-specific documents to the extractive and generative answering components, and controls the output to ensure safe and domain-specific answers. A comprehensive empirical evaluation exhibits that CarExpert outperforms state-of-the-art LLMs in generating natural, safe and car-specific answers.
△ Less
Submitted 14 October, 2023;
originally announced October 2023.
-
Quantum Inspired Adaptive Boosting
Authors:
Bálint Daróczy,
Katalin Friedl,
László Kabódi,
Attila Pereszlényi,
Dániel Szabó
Abstract:
Building on the quantum ensemble based classifier algorithm of Schuld and Petruccione [arXiv:1704.02146v1], we devise equivalent classical algorithms which show that this quantum ensemble method does not have advantage over classical algorithms. Essentially, we simplify their algorithm until it is intuitive to come up with an equivalent classical version. One of the classical algorithms is extreme…
▽ More
Building on the quantum ensemble based classifier algorithm of Schuld and Petruccione [arXiv:1704.02146v1], we devise equivalent classical algorithms which show that this quantum ensemble method does not have advantage over classical algorithms. Essentially, we simplify their algorithm until it is intuitive to come up with an equivalent classical version. One of the classical algorithms is extremely simple and runs in constant time for each input to be classified. We further develop the idea and, as the main contribution of the paper, we propose methods inspired by combining the quantum ensemble method with adaptive boosting. The algorithms were tested and found to be comparable to the AdaBoost algorithm on publicly available data sets.
△ Less
Submitted 1 February, 2021;
originally announced February 2021.
-
Some Improvements to Total Degree Tests
Authors:
Katalin Friedl,
Madhu Sudan
Abstract:
A low-degree test is a collection of simple, local rules for checking the proximity of an arbitrary function to a low-degree polynomial. Each rule depends on the function's values at a small number of places. If a function satisfies many rules then it is close to a low-degree polynomial. Low-degree tests play an important role in the development of probabilistically checkable proofs.
In this pap…
▽ More
A low-degree test is a collection of simple, local rules for checking the proximity of an arbitrary function to a low-degree polynomial. Each rule depends on the function's values at a small number of places. If a function satisfies many rules then it is close to a low-degree polynomial. Low-degree tests play an important role in the development of probabilistically checkable proofs.
In this paper we present two improvements to the efficiency of low-degree tests. Our first improvement concerns the smallest field size over which a low-degree test can work. We show how to test that a function is a degree $d$ polynomial over prime fields of size only $d+2$.
Our second improvement shows a better efficiency of the low-degree test of Rubinfeld and Sudan (Proc. SODA 1992) than previously known. We show concrete applications of this improvement via the notion of "locally checkable codes". This improvement translates into better tradeoffs on the size versus probe complexity of probabilistically checkable proofs than previously known.
△ Less
Submitted 15 July, 2013;
originally announced July 2013.
-
When the largest eigenvalue of the modularity and normalized modularity matrix is zero
Authors:
Marianna Bolla,
Brian Bullins,
Sorathan Chaturapruek,
Shiwen Chen,
Katalin Friedl
Abstract:
In July 2012, at the Conference on Applications of Graph Spectra in Computer Science, Barcelona, D. Stevanovic posed the following open problem: which graphs have the zero as the largest eigenvalue of their modularity matrix? The conjecture was that only the complete and complete multipartite graphs. They indeed have this property, but are they the only ones? In this paper, we will give an affirma…
▽ More
In July 2012, at the Conference on Applications of Graph Spectra in Computer Science, Barcelona, D. Stevanovic posed the following open problem: which graphs have the zero as the largest eigenvalue of their modularity matrix? The conjecture was that only the complete and complete multipartite graphs. They indeed have this property, but are they the only ones? In this paper, we will give an affirmative answer to this question and prove a bit more: both the modularity and the normalized modularity matrix of a graph is negative semidefinite if and only if the graph is complete or complete multipartite.
△ Less
Submitted 9 May, 2013;
originally announced May 2013.
-
Singular value decomposition of large random matrices (for two-way classification of microarrays)
Authors:
Marianna Bolla,
Katalin Friedl,
Andras Kramli
Abstract:
Asymptotic behavior of the singular value decomposition (SVD) of blown up matrices and normalized blown up contingency tables exposed to Wigner-noise is investigated.It is proved that such an m\times n matrix almost surely has a constant number of large singular values (of order \sqrt{mn}), while the rest of the singular values are of order \sqrt{m+n} as m,n\to\infty. Concentration results of Al…
▽ More
Asymptotic behavior of the singular value decomposition (SVD) of blown up matrices and normalized blown up contingency tables exposed to Wigner-noise is investigated.It is proved that such an m\times n matrix almost surely has a constant number of large singular values (of order \sqrt{mn}), while the rest of the singular values are of order \sqrt{m+n} as m,n\to\infty. Concentration results of Alon et al. for the eigenvalues of large symmetric random matrices are adapted to the rectangular case, and on this basis, almost sure results for the singular values as well as for the corresponding isotropic subspaces are proved. An algorithm, applicable to two-way classification of microarrays, is also given that finds the underlying block structure.
△ Less
Submitted 22 May, 2008;
originally announced May 2008.
-
On the black-box complexity of Sperner's Lemma
Authors:
Katalin Friedl,
Gabor Ivanyos,
Miklos Santha,
Yves F. Verhoeven
Abstract:
We present several results on the complexity of various forms of Sperner's Lemma in the black-box model of computing. We give a deterministic algorithm for Sperner problems over pseudo-manifolds of arbitrary dimension. The query complexity of our algorithm is linear in the separation number of the skeleton graph of the manifold and the size of its boundary. As a corollary we get an…
▽ More
We present several results on the complexity of various forms of Sperner's Lemma in the black-box model of computing. We give a deterministic algorithm for Sperner problems over pseudo-manifolds of arbitrary dimension. The query complexity of our algorithm is linear in the separation number of the skeleton graph of the manifold and the size of its boundary. As a corollary we get an $O(\sqrt{n})$ deterministic query algorithm for the black-box version of the problem {\bf 2D-SPERNER}, a well studied member of Papadimitriou's complexity class PPAD. This upper bound matches the $Ω(\sqrt{n})$ deterministic lower bound of Crescenzi and Silvestri. The tightness of this bound was not known before. In another result we prove for the same problem an $Ω(\sqrt[4]{n})$ lower bound for its probabilistic, and an $Ω(\sqrt[8]{n})$ lower bound for its quantum query complexity, showing that all these measures are polynomially related.
△ Less
Submitted 24 May, 2005;
originally announced May 2005.
-
Hidden Translation and Translating Coset in Quantum Computing
Authors:
K. Friedl,
G. Ivanyos,
F. Magniez,
M. Santha,
P. Sen
Abstract:
We give efficient quantum algorithms for the problems of Hidden Translation and Hidden Subgroup in a large class of non-abelian solvable groups including solvable groups of constant exponent and of constant length derived series. Our algorithms are recursive. For the base case, we solve efficiently Hidden Translation in $\Z_{p}^{n}$, whenever $p$ is a fixed prime. For the induction step, we introd…
▽ More
We give efficient quantum algorithms for the problems of Hidden Translation and Hidden Subgroup in a large class of non-abelian solvable groups including solvable groups of constant exponent and of constant length derived series. Our algorithms are recursive. For the base case, we solve efficiently Hidden Translation in $\Z_{p}^{n}$, whenever $p$ is a fixed prime. For the induction step, we introduce the problem Translating Coset generalizing both Hidden Translation and Hidden Subgroup, and prove a powerful self-reducibility result: Translating Coset in a finite solvable group $G$ is reducible to instances of Translating Coset in $G/N$ and $N$, for appropriate normal subgroups $N$ of $G$. Our self-reducibility framework combined with Kuperberg's subexponential quantum algorithm for solving Hidden Translation in any abelian group, leads to subexponential quantum algorithms for Hidden Translation and Hidden Subgroup in any solvable group.
△ Less
Submitted 16 September, 2013; v1 submitted 15 November, 2002;
originally announced November 2002.
-
Quantum testers for hidden group properties
Authors:
Katalin Friedl,
Frederic Magniez,
Miklos Santha,
Pranab Sen
Abstract:
We construct efficient or query efficient quantum property testers for two existential group properties which have exponential query complexity both for their decision problem in the quantum and for their testing problem in the classical model of computing. These are periodicity in groups and the common coset range property of two functions having identical ranges within each coset of some norma…
▽ More
We construct efficient or query efficient quantum property testers for two existential group properties which have exponential query complexity both for their decision problem in the quantum and for their testing problem in the classical model of computing. These are periodicity in groups and the common coset range property of two functions having identical ranges within each coset of some normal subgroup. Our periodicity tester is efficient in Abelian groups and generalizes, in several aspects, previous periodicity testers. This is achieved by introducing a technique refining the majority correction process widely used for proving robustness of algebraic properties. The periodicity tester in non-Abelian groups and the common coset range tester are query efficient.
△ Less
Submitted 29 August, 2002;
originally announced August 2002.
