-
Increasing the LLM Accuracy for Question Answering: Ontologies to the Rescue!
Authors:
Dean Allemang,
Juan Sequeda
Abstract:
There is increasing evidence that question-answering (QA) systems with Large Language Models (LLMs), which employ a knowledge graph/semantic representation of an enterprise SQL database (i.e. Text-to-SPARQL), achieve higher accuracy compared to systems that answer questions directly on SQL databases (i.e. Text-to-SQL). Our previous benchmark research showed that by using a knowledge graph, the acc…
▽ More
There is increasing evidence that question-answering (QA) systems with Large Language Models (LLMs), which employ a knowledge graph/semantic representation of an enterprise SQL database (i.e. Text-to-SPARQL), achieve higher accuracy compared to systems that answer questions directly on SQL databases (i.e. Text-to-SQL). Our previous benchmark research showed that by using a knowledge graph, the accuracy improved from 16% to 54%. The question remains: how can we further improve the accuracy and reduce the error rate? Building on the observations of our previous research where the inaccurate LLM-generated SPARQL queries followed incorrect paths, we present an approach that consists of 1) Ontology-based Query Check (OBQC): detects errors by leveraging the ontology of the knowledge graph to check if the LLM-generated SPARQL query matches the semantic of ontology and 2) LLM Repair: use the error explanations with an LLM to repair the SPARQL query. Using the chat with the data benchmark, our primary finding is that our approach increases the overall accuracy to 72% including an additional 8% of "I don't know" unknown results. Thus, the overall error rate is 20%. These results provide further evidence that investing knowledge graphs, namely the ontology, provides higher accuracy for LLM powered question answering systems.
△ Less
Submitted 19 May, 2024;
originally announced May 2024.
-
A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model's Accuracy for Question Answering on Enterprise SQL Databases
Authors:
Juan Sequeda,
Dean Allemang,
Bryon Jacob
Abstract:
Enterprise applications of Large Language Models (LLMs) hold promise for question answering on enterprise SQL databases. However, the extent to which LLMs can accurately respond to enterprise questions in such databases remains unclear, given the absence of suitable Text-to-SQL benchmarks tailored to enterprise settings. Additionally, the potential of Knowledge Graphs (KGs) to enhance LLM-based qu…
▽ More
Enterprise applications of Large Language Models (LLMs) hold promise for question answering on enterprise SQL databases. However, the extent to which LLMs can accurately respond to enterprise questions in such databases remains unclear, given the absence of suitable Text-to-SQL benchmarks tailored to enterprise settings. Additionally, the potential of Knowledge Graphs (KGs) to enhance LLM-based question answering by providing business context is not well understood. This study aims to evaluate the accuracy of LLM-powered question answering systems in the context of enterprise questions and SQL databases, while also exploring the role of knowledge graphs in improving accuracy. To achieve this, we introduce a benchmark comprising an enterprise SQL schema in the insurance domain, a range of enterprise queries encompassing reporting to metrics, and a contextual layer incorporating an ontology and map**s that define a knowledge graph. Our primary finding reveals that question answering using GPT-4, with zero-shot prompts directly on SQL databases, achieves an accuracy of 16%. Notably, this accuracy increases to 54% when questions are posed over a Knowledge Graph representation of the enterprise SQL database. Therefore, investing in Knowledge Graph provides higher accuracy for LLM powered question answering systems.
△ Less
Submitted 13 November, 2023;
originally announced November 2023.
-
PG-Schema: Schemas for Property Graphs
Authors:
Renzo Angles,
Angela Bonifati,
Stefania Dumbrava,
George Fletcher,
Alastair Green,
Jan Hidders,
Bei Li,
Leonid Libkin,
Victor Marsault,
Wim Martens,
Filip Murlak,
Stefan Plantikow,
Ognjen Savković,
Michael Schmidt,
Juan Sequeda,
Sławek Staworko,
Dominik Tomaszuk,
Hannes Voigt,
Domagoj Vrgoč,
Mingxi Wu,
Dušan Živković
Abstract:
Property graphs have reached a high level of maturity, witnessed by multiple robust graph database systems as well as the ongoing ISO standardization effort aiming at creating a new standard Graph Query Language (GQL). Yet, despite documented demand, schema support is limited both in existing systems and in the first version of the GQL Standard. It is anticipated that the second version of the GQL…
▽ More
Property graphs have reached a high level of maturity, witnessed by multiple robust graph database systems as well as the ongoing ISO standardization effort aiming at creating a new standard Graph Query Language (GQL). Yet, despite documented demand, schema support is limited both in existing systems and in the first version of the GQL Standard. It is anticipated that the second version of the GQL Standard will include a rich DDL. Aiming to inspire the development of GQL and enhance the capabilities of graph database systems, we propose PG-Schema, a simple yet powerful formalism for specifying property graph schemas. It features PG-Types with flexible type definitions supporting multi-inheritance, as well as expressive constraints based on the recently proposed PG-Keys formalism. We provide the formal syntax and semantics of PG-Schema, which meet principled design requirements grounded in contemporary property graph management scenarios, and offer a detailed comparison of its features with those of existing schema languages and graph database systems.
△ Less
Submitted 8 July, 2023; v1 submitted 20 November, 2022;
originally announced November 2022.
-
Querying in the Age of Graph Databases and Knowledge Graphs
Authors:
Marcelo Arenas,
Claudio Gutierrez,
Juan F. Sequeda
Abstract:
Graphs have become the best way we know of representing knowledge. The computing community has investigated and developed the support for managing graphs by means of digital technology. Graph databases and knowledge graphs surface as the most successful solutions to this program. The goal of this document is to provide a conceptual map of the data management tasks underlying these developments, pa…
▽ More
Graphs have become the best way we know of representing knowledge. The computing community has investigated and developed the support for managing graphs by means of digital technology. Graph databases and knowledge graphs surface as the most successful solutions to this program. The goal of this document is to provide a conceptual map of the data management tasks underlying these developments, paying particular attention to data models and query languages for graphs.
△ Less
Submitted 24 June, 2021; v1 submitted 21 June, 2021;
originally announced June 2021.
-
The Future is Big Graphs! A Community View on Graph Processing Systems
Authors:
Sherif Sakr,
Angela Bonifati,
Hannes Voigt,
Alexandru Iosup,
Khaled Ammar,
Renzo Angles,
Walid Aref,
Marcelo Arenas,
Maciej Besta,
Peter A. Boncz,
Khuzaima Daudjee,
Emanuele Della Valle,
Stefania Dumbrava,
Olaf Hartig,
Bernhard Haslhofer,
Tim Hegeman,
Jan Hidders,
Katja Hose,
Adriana Iamnitchi,
Vasiliki Kalavri,
Hugo Kapp,
Wim Martens,
M. Tamer Özsu,
Eric Peukert,
Stefan Plantikow
, et al. (16 additional authors not shown)
Abstract:
Graphs are by nature unifying abstractions that can leverage interconnectedness to represent, explore, predict, and explain real- and digital-world phenomena. Although real users and consumers of graph instances and graph workloads understand these abstractions, future problems will require new abstractions and systems. What needs to happen in the next decade for big graph processing to continue t…
▽ More
Graphs are by nature unifying abstractions that can leverage interconnectedness to represent, explore, predict, and explain real- and digital-world phenomena. Although real users and consumers of graph instances and graph workloads understand these abstractions, future problems will require new abstractions and systems. What needs to happen in the next decade for big graph processing to continue to succeed?
△ Less
Submitted 11 December, 2020;
originally announced December 2020.
-
Knowledge Scientists: Unlocking the data-driven organization
Authors:
George Fletcher,
Paul Groth,
Juan Sequeda
Abstract:
Organizations across all sectors are increasingly undergoing deep transformation and restructuring towards data-driven operations. The central role of data highlights the need for reliable and clean data. Unreliable, erroneous, and incomplete data lead to critical bottlenecks in processing pipelines and, ultimately, service failures, which are disastrous for the competitive performance of the orga…
▽ More
Organizations across all sectors are increasingly undergoing deep transformation and restructuring towards data-driven operations. The central role of data highlights the need for reliable and clean data. Unreliable, erroneous, and incomplete data lead to critical bottlenecks in processing pipelines and, ultimately, service failures, which are disastrous for the competitive performance of the organization. Given its central importance, those organizations which recognize and react to the need for reliable data will have the advantage in the coming decade. We argue that the technologies for reliable data are driven by distinct concerns and expertise which complement those of the data scientist and the data engineer. Those organizations which identify the central importance of meaningful, explainable, reproducible, and maintainable data will be at the forefront of the democratization of reliable data. We call the new role which must be developed to fill this critical need the Knowledge Scientist. The organizational structures, tools, methodologies and techniques to support and make possible the work of knowledge scientists are still in their infancy. As organizations not only use data but increasingly rely on data, it is time to empower the people who are central to this transformation.
△ Less
Submitted 16 April, 2020;
originally announced April 2020.
-
Knowledge Graphs
Authors:
Aidan Hogan,
Eva Blomqvist,
Michael Cochez,
Claudia d'Amato,
Gerard de Melo,
Claudio Gutierrez,
José Emilio Labra Gayo,
Sabrina Kirrane,
Sebastian Neumaier,
Axel Polleres,
Roberto Navigli,
Axel-Cyrille Ngonga Ngomo,
Sabbir M. Rashid,
Anisa Rula,
Lukas Schmelzeisen,
Juan Sequeda,
Steffen Staab,
Antoine Zimmermann
Abstract:
In this paper we provide a comprehensive introduction to knowledge graphs, which have recently garnered significant attention from both industry and academia in scenarios that require exploiting diverse, dynamic, large-scale collections of data. After some opening remarks, we motivate and contrast various graph-based data models and query languages that are used for knowledge graphs. We discuss th…
▽ More
In this paper we provide a comprehensive introduction to knowledge graphs, which have recently garnered significant attention from both industry and academia in scenarios that require exploiting diverse, dynamic, large-scale collections of data. After some opening remarks, we motivate and contrast various graph-based data models and query languages that are used for knowledge graphs. We discuss the roles of schema, identity, and context in knowledge graphs. We explain how knowledge can be represented and extracted using a combination of deductive and inductive techniques. We summarise methods for the creation, enrichment, quality assessment, refinement, and publication of knowledge graphs. We provide an overview of prominent open knowledge graphs and enterprise knowledge graphs, their applications, and how they use the aforementioned techniques. We conclude with high-level future research directions for knowledge graphs.
△ Less
Submitted 11 September, 2021; v1 submitted 4 March, 2020;
originally announced March 2020.
-
G-CORE: A Core for Future Graph Query Languages
Authors:
Renzo Angles,
Marcelo Arenas,
Pablo Barceló,
Peter Boncz,
George H. L. Fletcher,
Claudio Gutierrez,
Tobias Lindaaker,
Marcus Paradies,
Stefan Plantikow,
Juan Sequeda,
Oskar van Rest,
Hannes Voigt
Abstract:
We report on a community effort between industry and academia to shape the future of graph query languages. We argue that existing graph database management systems should consider supporting a query language with two key characteristics. First, it should be composable, meaning, that graphs are the input and the output of queries. Second, the graph query language should treat paths as first-class…
▽ More
We report on a community effort between industry and academia to shape the future of graph query languages. We argue that existing graph database management systems should consider supporting a query language with two key characteristics. First, it should be composable, meaning, that graphs are the input and the output of queries. Second, the graph query language should treat paths as first-class citizens. Our result is G-CORE, a powerful graph query language design that fulfills these goals, and strikes a careful balance between path query expressivity and evaluation complexity.
△ Less
Submitted 6 December, 2017; v1 submitted 5 December, 2017;
originally announced December 2017.
-
On Directly Map** Relational Databases to RDF and OWL (Extended Version)
Authors:
Juan F. Sequeda,
Marcelo Arenas,
Daniel P. Miranker
Abstract:
Map** relational databases to RDF is a fundamental problem for the development of the Semantic Web. We present a solution, inspired by draft methods defined by the W3C where relational databases are directly mapped to RDF and OWL. Given a relational database schema and its integrity constraints, this direct map** produces an OWL ontology, which, provides the basis for generating RDF instances.…
▽ More
Map** relational databases to RDF is a fundamental problem for the development of the Semantic Web. We present a solution, inspired by draft methods defined by the W3C where relational databases are directly mapped to RDF and OWL. Given a relational database schema and its integrity constraints, this direct map** produces an OWL ontology, which, provides the basis for generating RDF instances. The semantics of this map** is defined using Datalog. Two fundamental properties are information preservation and query preservation. We prove that our map** satisfies both conditions, even for relational databases that contain null values. We also consider two desirable properties: monotonicity and semantics preservation. We prove that our map** is monotone and also prove that no monotone map**, including ours, is semantic preserving. We realize that monotonicity is an obstacle for semantic preservation and thus present a non-monotone direct map** that is semantics preserving.
△ Less
Submitted 22 February, 2012; v1 submitted 16 February, 2012;
originally announced February 2012.
-
Towards a Query Language for the Web of Data (A Vision Paper)
Authors:
Juan Sequeda,
Olaf Hartig
Abstract:
Research on querying the Web of Data is still in its infancy. In this paper, we provide an initial set of general features that we envision should be considered in order to define a query language for the Web of Data. Furthermore, for each of these features, we pose questions that have not been addressed before in the context of querying the Web of Data. We believe that addressing these questions…
▽ More
Research on querying the Web of Data is still in its infancy. In this paper, we provide an initial set of general features that we envision should be considered in order to define a query language for the Web of Data. Furthermore, for each of these features, we pose questions that have not been addressed before in the context of querying the Web of Data. We believe that addressing these questions and studying these features may guide the next 10 years of research on the Web of Data.
△ Less
Submitted 13 October, 2011;
originally announced October 2011.
