-
DID:RING: Ring Signatures using Decentralised Identifiers For Privacy-Aware Identity
Authors:
Dimitrios Kasimatis,
Sam Grierson,
William J. Buchanan,
Chris Eckl,
Pavlos Papadopoulos,
Nikolaos Pitropakis,
Craig Thomson,
Baraq Ghaleb
Abstract:
Decentralised identifiers have become a standardised element of digital identity architecture, with supra-national organisations such as the European Union adopting them as a key component for a unified European digital identity ledger. This paper delves into enhancing security and privacy features within decentralised identifiers by integrating ring signatures as an alternative verification metho…
▽ More
Decentralised identifiers have become a standardised element of digital identity architecture, with supra-national organisations such as the European Union adopting them as a key component for a unified European digital identity ledger. This paper delves into enhancing security and privacy features within decentralised identifiers by integrating ring signatures as an alternative verification method. This allows users to identify themselves through digital signatures without revealing which public key they used. To this end, the study proposed a novel decentralised identity method showcased in a decentralised identifier-based architectural framework. Additionally, the investigation assesses the repercussions of employing this new method in the verification process, focusing specifically on privacy and security aspects. Although ring signatures are an established asset of cryptographic protocols, this paper seeks to leverage their capabilities in the evolving domain of digital identities.
△ Less
Submitted 11 March, 2024; v1 submitted 8 March, 2024;
originally announced March 2024.
-
AI in Energy Digital Twining: A Reinforcement Learning-based Adaptive Digital Twin Model for Green Cities
Authors:
Lal Verda Cakir,
Kubra Duran,
Craig Thomson,
Matthew Broadbent,
Berk Canberk
Abstract:
Digital Twins (DT) have become crucial to achieve sustainable and effective smart urban solutions. However, current DT modelling techniques cannot support the dynamicity of these smart city environments. This is caused by the lack of right-time data capturing in traditional approaches, resulting in inaccurate modelling and high resource and energy consumption challenges. To fill this gap, we explo…
▽ More
Digital Twins (DT) have become crucial to achieve sustainable and effective smart urban solutions. However, current DT modelling techniques cannot support the dynamicity of these smart city environments. This is caused by the lack of right-time data capturing in traditional approaches, resulting in inaccurate modelling and high resource and energy consumption challenges. To fill this gap, we explore spatiotemporal graphs and propose the Reinforcement Learning-based Adaptive Twining (RL-AT) mechanism with Deep Q Networks (DQN). By doing so, our study contributes to advancing Green Cities and showcases tangible benefits in accuracy, synchronisation, resource optimization, and energy efficiency. As a result, we note the spatiotemporal graphs are able to offer a consistent accuracy and 55% higher querying performance when implemented using graph databases. In addition, our model demonstrates right-time data capturing with 20% lower overhead and 25% lower energy consumption.
△ Less
Submitted 28 January, 2024;
originally announced January 2024.
-
Scalable Multi-domain Trust Infrastructures for Segmented Networks
Authors:
Sam Grierson,
William J Buchanan,
Craig Thomson,
Baraq Ghaleb,
Leandros Maglaras,
Chris Eckl
Abstract:
Within a trust infrastructure, a private key is often used to digitally sign a transaction, which can be verified with an associated public key. Using PKI (Public Key Infrastructure), a trusted entity can produce a digital signature, verifying the authenticity of the public key. However, what happens when external entities are not trusted to verify the public key or in cases where there is no Inte…
▽ More
Within a trust infrastructure, a private key is often used to digitally sign a transaction, which can be verified with an associated public key. Using PKI (Public Key Infrastructure), a trusted entity can produce a digital signature, verifying the authenticity of the public key. However, what happens when external entities are not trusted to verify the public key or in cases where there is no Internet connection within an isolated or autonomously acting collection of devices? For this, a trusted entity can be elected to generate a key pair and then split the private key amongst trusted devices. Each node can then sign part of the transaction using their split of the shared secret. The aggregated signature can then define agreement on a consensus within the infrastructure. Unfortunately, this process has two significant problems. The first is when no trusted node can act as a dealer of the shares. The second is the difficulty of scaling the digital signature scheme. This paper outlines a method of creating a leaderless approach to defining trust domains to overcome weaknesses in the scaling of the elliptic curve digital signature algorithm. Instead, it proposes the usage of the Edwards curve digital signature algorithm for the definition of multiple trust zones. The paper shows that the computational overhead of the distributed key generation phase increases with the number of nodes in the trust domain but that the distributed signing has a relatively constant computational overhead.
△ Less
Submitted 10 October, 2023; v1 submitted 7 October, 2023;
originally announced October 2023.
-
Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP
Authors:
Anya Belz,
Craig Thomson,
Ehud Reiter,
Gavin Abercrombie,
Jose M. Alonso-Moral,
Mohammad Arvan,
Anouck Braggaar,
Mark Cieliebak,
Elizabeth Clark,
Kees van Deemter,
Tanvi Dinkar,
Ondřej Dušek,
Steffen Eger,
Qixiang Fang,
Mingqi Gao,
Albert Gatt,
Dimitra Gkatzia,
Javier González-Corbelle,
Dirk Hovy,
Manuela Hürlimann,
Takumi Ito,
John D. Kelleher,
Filip Klubicka,
Emiel Krahmer,
Huiyuan Lai
, et al. (17 additional authors not shown)
Abstract:
We report our efforts in identifying a set of previous human evaluations in NLP that would be suitable for a coordinated study examining what makes human evaluations in NLP more/less reproducible. We present our results and findings, which include that just 13\% of papers had (i) sufficiently low barriers to reproduction, and (ii) enough obtainable information, to be considered for reproduction, a…
▽ More
We report our efforts in identifying a set of previous human evaluations in NLP that would be suitable for a coordinated study examining what makes human evaluations in NLP more/less reproducible. We present our results and findings, which include that just 13\% of papers had (i) sufficiently low barriers to reproduction, and (ii) enough obtainable information, to be considered for reproduction, and that all but one of the experiments we selected for reproduction was discovered to have flaws that made the meaningfulness of conducting a reproduction questionable. As a result, we had to change our coordinated study design from a reproduce approach to a standardise-then-reproduce-twice approach. Our overall (negative) finding that the great majority of human evaluations in NLP is not repeatable and/or not reproducible and/or too flawed to justify reproduction, paints a dire picture, but presents an opportunity for a rethink about how to design and report human evaluations in NLP.
△ Less
Submitted 7 August, 2023; v1 submitted 2 May, 2023;
originally announced May 2023.
-
GEMv2: Multilingual NLG Benchmarking in a Single Line of Code
Authors:
Sebastian Gehrmann,
Abhik Bhattacharjee,
Abinaya Mahendiran,
Alex Wang,
Alexandros Papangelis,
Aman Madaan,
Angelina McMillan-Major,
Anna Shvets,
Ashish Upadhyay,
Bingsheng Yao,
Bryan Wilie,
Chandra Bhagavatula,
Chaobin You,
Craig Thomson,
Cristina Garbacea,
Dakuo Wang,
Daniel Deutsch,
Deyi Xiong,
Di **,
Dimitra Gkatzia,
Dragomir Radev,
Elizabeth Clark,
Esin Durmus,
Faisal Ladhak,
Filip Ginter
, et al. (52 additional authors not shown)
Abstract:
Evaluation in machine learning is usually informed by past choices, for example which datasets or metrics to use. This standardization enables the comparison on equal footing using leaderboards, but the evaluation choices become sub-optimal as better alternatives arise. This problem is especially pertinent in natural language generation which requires ever-improving suites of datasets, metrics, an…
▽ More
Evaluation in machine learning is usually informed by past choices, for example which datasets or metrics to use. This standardization enables the comparison on equal footing using leaderboards, but the evaluation choices become sub-optimal as better alternatives arise. This problem is especially pertinent in natural language generation which requires ever-improving suites of datasets, metrics, and human evaluation to make definitive claims. To make following best model evaluation practices easier, we introduce GEMv2. The new version of the Generation, Evaluation, and Metrics Benchmark introduces a modular infrastructure for dataset, model, and metric developers to benefit from each others work. GEMv2 supports 40 documented datasets in 51 languages. Models for all datasets can be evaluated online and our interactive data card creation and rendering tools make it easier to add new datasets to the living benchmark.
△ Less
Submitted 24 June, 2022; v1 submitted 22 June, 2022;
originally announced June 2022.
-
Generation Challenges: Results of the Accuracy Evaluation Shared Task
Authors:
Craig Thomson,
Ehud Reiter
Abstract:
The Shared Task on Evaluating Accuracy focused on techniques (both manual and automatic) for evaluating the factual accuracy of texts produced by neural NLG systems, in a sports-reporting domain. Four teams submitted evaluation techniques for this task, using very different approaches and techniques. The best-performing submissions did encouragingly well at this difficult task. However, all automa…
▽ More
The Shared Task on Evaluating Accuracy focused on techniques (both manual and automatic) for evaluating the factual accuracy of texts produced by neural NLG systems, in a sports-reporting domain. Four teams submitted evaluation techniques for this task, using very different approaches and techniques. The best-performing submissions did encouragingly well at this difficult task. However, all automatic submissions struggled to detect factual errors which are semantically or pragmatically complex (for example, based on incorrect computation or inference).
△ Less
Submitted 15 August, 2021; v1 submitted 12 August, 2021;
originally announced August 2021.
-
Underreporting of errors in NLG output, and what to do about it
Authors:
Emiel van Miltenburg,
Miruna-Adriana Clinciu,
Ondřej Dušek,
Dimitra Gkatzia,
Stephanie Inglis,
Leo Leppänen,
Saad Mahamood,
Emma Manning,
Stephanie Schoch,
Craig Thomson,
Luou Wen
Abstract:
We observe a severe under-reporting of the different kinds of errors that Natural Language Generation systems make. This is a problem, because mistakes are an important indicator of where systems should still be improved. If authors only report overall performance metrics, the research community is left in the dark about the specific weaknesses that are exhibited by `state-of-the-art' research. Ne…
▽ More
We observe a severe under-reporting of the different kinds of errors that Natural Language Generation systems make. This is a problem, because mistakes are an important indicator of where systems should still be improved. If authors only report overall performance metrics, the research community is left in the dark about the specific weaknesses that are exhibited by `state-of-the-art' research. Next to quantifying the extent of error under-reporting, this position paper provides recommendations for error identification, analysis and reporting.
△ Less
Submitted 8 August, 2021; v1 submitted 2 August, 2021;
originally announced August 2021.
-
A Gold Standard Methodology for Evaluating Accuracy in Data-To-Text Systems
Authors:
Craig Thomson,
Ehud Reiter
Abstract:
Most Natural Language Generation systems need to produce accurate texts. We propose a methodology for high-quality human evaluation of the accuracy of generated texts, which is intended to serve as a gold-standard for accuracy evaluations of data-to-text systems. We use our methodology to evaluate the accuracy of computer generated basketball summaries. We then show how our gold standard evaluatio…
▽ More
Most Natural Language Generation systems need to produce accurate texts. We propose a methodology for high-quality human evaluation of the accuracy of generated texts, which is intended to serve as a gold-standard for accuracy evaluations of data-to-text systems. We use our methodology to evaluate the accuracy of computer generated basketball summaries. We then show how our gold standard evaluation can be used to validate automated metrics
△ Less
Submitted 8 November, 2020;
originally announced November 2020.
-
Shared Task on Evaluating Accuracy in Natural Language Generation
Authors:
Ehud Reiter,
Craig Thomson
Abstract:
We propose a shared task on methodologies and algorithms for evaluating the accuracy of generated texts. Participants will measure the accuracy of basketball game summaries produced by NLG systems from basketball box score data.
We propose a shared task on methodologies and algorithms for evaluating the accuracy of generated texts. Participants will measure the accuracy of basketball game summaries produced by NLG systems from basketball box score data.
△ Less
Submitted 6 November, 2020; v1 submitted 22 June, 2020;
originally announced June 2020.
