-
Controlled Randomness Improves the Performance of Transformer Models
Authors:
Tobias Deußer,
Cong Zhao,
Wolfgang Krämer,
David Leonhard,
Christian Bauckhage,
Rafet Sifa
Abstract:
During the pre-training step of natural language models, the main objective is to learn a general representation of the pre-training dataset, usually requiring large amounts of textual data to capture the complexity and diversity of natural language. Contrasting this, in most cases, the size of the data available to solve the specific downstream task is often dwarfed by the aforementioned pre-trai…
▽ More
During the pre-training step of natural language models, the main objective is to learn a general representation of the pre-training dataset, usually requiring large amounts of textual data to capture the complexity and diversity of natural language. Contrasting this, in most cases, the size of the data available to solve the specific downstream task is often dwarfed by the aforementioned pre-training dataset, especially in domains where data is scarce. We introduce controlled randomness, i.e. noise, into the training process to improve fine-tuning language models and explore the performance of targeted noise in addition to the parameters of these models. We find that adding such noise can improve the performance in our two downstream tasks of joint named entity recognition and relation extraction and text summarization.
△ Less
Submitted 20 October, 2023;
originally announced October 2023.
-
Improving Zero-Shot Text Matching for Financial Auditing with Large Language Models
Authors:
Lars Hillebrand,
Armin Berger,
Tobias Deußer,
Tim Dilmaghani,
Mohamed Khaled,
Bernd Kliem,
Rüdiger Loitz,
Maren Pielka,
David Leonhard,
Christian Bauckhage,
Rafet Sifa
Abstract:
Auditing financial documents is a very tedious and time-consuming process. As of today, it can already be simplified by employing AI-based solutions to recommend relevant text passages from a report for each legal requirement of rigorous accounting standards. However, these methods need to be fine-tuned regularly, and they require abundant annotated data, which is often lacking in industrial envir…
▽ More
Auditing financial documents is a very tedious and time-consuming process. As of today, it can already be simplified by employing AI-based solutions to recommend relevant text passages from a report for each legal requirement of rigorous accounting standards. However, these methods need to be fine-tuned regularly, and they require abundant annotated data, which is often lacking in industrial environments. Hence, we present ZeroShotALI, a novel recommender system that leverages a state-of-the-art large language model (LLM) in conjunction with a domain-specifically optimized transformer-based text-matching solution. We find that a two-step approach of first retrieving a number of best matching document sections per legal requirement with a custom BERT-based model and second filtering these selections using an LLM yields significant performance improvements over existing approaches.
△ Less
Submitted 14 August, 2023; v1 submitted 11 August, 2023;
originally announced August 2023.
-
sustain.AI: a Recommender System to analyze Sustainability Reports
Authors:
Lars Hillebrand,
Maren Pielka,
David Leonhard,
Tobias Deußer,
Tim Dilmaghani,
Bernd Kliem,
Rüdiger Loitz,
Milad Morad,
Christian Temath,
Thiago Bell,
Robin Stenzel,
Rafet Sifa
Abstract:
We present sustainAI, an intelligent, context-aware recommender system that assists auditors and financial investors as well as the general public to efficiently analyze companies' sustainability reports. The tool leverages an end-to-end trainable architecture that couples a BERT-based encoding module with a multi-label classification head to match relevant text passages from sustainability report…
▽ More
We present sustainAI, an intelligent, context-aware recommender system that assists auditors and financial investors as well as the general public to efficiently analyze companies' sustainability reports. The tool leverages an end-to-end trainable architecture that couples a BERT-based encoding module with a multi-label classification head to match relevant text passages from sustainability reports to their respective law regulations from the Global Reporting Initiative (GRI) standards. We evaluate our model on two novel German sustainability reporting data sets and consistently achieve a significantly higher recommendation performance compared to multiple strong baselines. Furthermore, sustainAI is publicly available for everyone at https://sustain.ki.nrw/.
△ Less
Submitted 26 May, 2023; v1 submitted 15 May, 2023;
originally announced May 2023.
