-
Elevating Academic Administration: A Comprehensive Faculty Dashboard for Tracking Student Evaluations and Research
Authors:
Musa Azeem,
Muhammad Tukhtasunov,
Savannah Noblitt,
Mitchel Jonker,
Kevin Protzman
Abstract:
The USC Faculty Dashboard is a web application designed to revolutionize how department heads, professors, and instructors monitor progress and make decisions, providing a centralized hub for efficient data storage and analysis. Currently, there's a gap in tools tailored for department heads to concisely manage the performance of their department, which our platform aims to fill. The USC Faculty D…
▽ More
The USC Faculty Dashboard is a web application designed to revolutionize how department heads, professors, and instructors monitor progress and make decisions, providing a centralized hub for efficient data storage and analysis. Currently, there's a gap in tools tailored for department heads to concisely manage the performance of their department, which our platform aims to fill. The USC Faculty Dashboard offers easy access to upload and view student evaluation and research information, empowering department heads to evaluate the performance of faculty members and seamlessly track their research grants, publications, and expenditures. Furthermore, professors and instructors gain personalized performance analysis tools, with full access to their own data as well as curated access to peer data to assess their relative performance. The source code as well as the link to the deployed application can be found at https://github.com/SCCapstone/K3MS.
△ Less
Submitted 9 July, 2024;
originally announced July 2024.
-
Two-Stage Testing in a high dimensional setting
Authors:
Marianne A Jonker,
Luc van Schijndel,
Eric Cator
Abstract:
In a high dimensional regression setting in which the number of variables ($p$) is much larger than the sample size ($n$), the number of possible two-way interactions between the variables is immense. If the number of variables is in the order of one million, which is usually the case in e.g., genetics, the number of two-way interactions is of the order one million squared. In the pursuit of detec…
▽ More
In a high dimensional regression setting in which the number of variables ($p$) is much larger than the sample size ($n$), the number of possible two-way interactions between the variables is immense. If the number of variables is in the order of one million, which is usually the case in e.g., genetics, the number of two-way interactions is of the order one million squared. In the pursuit of detecting two-way interactions, testing all pairs for interactions one-by-one is computational unfeasible and the multiple testing correction will be severe. In this paper we describe a two-stage testing procedure consisting of a screening and an evaluation stage. It is proven that, under some assumptions, the tests-statistics in the two stages are asymptotically independent. As a result, multiplicity correction in the second stage is only needed for the number of statistical tests that are actually performed in that stage. This increases the power of the testing procedure. Also, since the testing procedure in the first stage is computational simple, the computational burden is lowered. Simulations have been performed for multiple settings and regression models (generalized linear models and Cox PH model) to study the performance of the two-stage testing procedure. The results show type I error control and an increase in power compared to the procedure in which the pairs are tested one-by-one.
△ Less
Submitted 25 June, 2024;
originally announced June 2024.
-
Towards General Negotiation Strategies with End-to-End Reinforcement Learning
Authors:
Bram M. Renting,
Thomas M. Moerland,
Holger H. Hoos,
Catholijn M. Jonker
Abstract:
The research field of automated negotiation has a long history of designing agents that can negotiate with other agents. Such negotiation strategies are traditionally based on manual design and heuristics. More recently, reinforcement learning approaches have also been used to train agents to negotiate. However, negotiation problems are diverse, causing observation and action dimensions to change,…
▽ More
The research field of automated negotiation has a long history of designing agents that can negotiate with other agents. Such negotiation strategies are traditionally based on manual design and heuristics. More recently, reinforcement learning approaches have also been used to train agents to negotiate. However, negotiation problems are diverse, causing observation and action dimensions to change, which cannot be handled by default linear policy networks. Previous work on this topic has circumvented this issue either by fixing the negotiation problem, causing policies to be non-transferable between negotiation problems or by abstracting the observations and actions into fixed-size representations, causing loss of information and expressiveness due to feature design. We developed an end-to-end reinforcement learning method for diverse negotiation problems by representing observations and actions as a graph and applying graph neural networks in the policy. With empirical evaluations, we show that our method is effective and that we can learn to negotiate with other agents on never-before-seen negotiation problems. Our result opens up new opportunities for reinforcement learning in negotiation agents.
△ Less
Submitted 21 June, 2024;
originally announced June 2024.
-
DarkDNS: Revisiting the Value of Rapid Zone Update
Authors:
Raffaele Sommese,
Gautam Akiwate,
Antonia Affinito,
Moritz Muller,
Mattijs Jonker,
KC Claffy
Abstract:
Malicious actors exploit the DNS namespace to launch spam campaigns, phishing attacks, malware, and other harmful activities. Combating these threats requires visibility into domain existence, ownership and nameservice activity that the DNS protocol does not itself provide. To facilitate visibility and security-related study of the expanding gTLD namespace, ICANN introduced the Centralized Zone Da…
▽ More
Malicious actors exploit the DNS namespace to launch spam campaigns, phishing attacks, malware, and other harmful activities. Combating these threats requires visibility into domain existence, ownership and nameservice activity that the DNS protocol does not itself provide. To facilitate visibility and security-related study of the expanding gTLD namespace, ICANN introduced the Centralized Zone Data Service (CZDS) that shares daily zone file snapshots of new gTLD zones. However, a remarkably high concentration of malicious activity is associated with domains that do not live long enough make it into these daily snapshots. Using public and private sources of newly observed domains to identify this activity, we discover that even with the best available data there is a considerable visibility gap. We find that the daily snapshots miss at least 1% of newly registered and short-lived domains, which are almost always registered with malicious intent. In reducing this critical visibility gap using public sources of data, we demonstrate how more timely access to TLD zone changes can help better prevent abuse. We hope that this work sparks a discussion in the community on how to effectively and safely revive the concept of sharing Rapid Zone Updates for security research.
△ Less
Submitted 20 May, 2024;
originally announced May 2024.
-
Bayesian Federated Inference for Survival Models
Authors:
Hassan Pazira,
Emanuele Massa,
Jetty AM Weijers,
Anthony CC Coolen,
Marianne A Jonker
Abstract:
In cancer research, overall survival and progression free survival are often analyzed with the Cox model. To estimate accurately the parameters in the model, sufficient data and, more importantly, sufficient events need to be observed. In practice, this is often a problem. Merging data sets from different medical centers may help, but this is not always possible due to strict privacy legislation a…
▽ More
In cancer research, overall survival and progression free survival are often analyzed with the Cox model. To estimate accurately the parameters in the model, sufficient data and, more importantly, sufficient events need to be observed. In practice, this is often a problem. Merging data sets from different medical centers may help, but this is not always possible due to strict privacy legislation and logistic difficulties. Recently, the Bayesian Federated Inference (BFI) strategy for generalized linear models was proposed. With this strategy the statistical analyses are performed in the local centers where the data were collected (or stored) and only the inference results are combined to a single estimated model; merging data is not necessary. The BFI methodology aims to compute from the separate inference results in the local centers what would have been obtained if the analysis had been based on the merged data sets. In this paper we generalize the BFI methodology as initially developed for generalized linear models to survival models. Simulation studies and real data analyses show excellent performance; i.e., the results obtained with the BFI methodology are very similar to the results obtained by analyzing the merged data. An R package for doing the analyses is available.
△ Less
Submitted 26 April, 2024;
originally announced April 2024.
-
A Hybrid Intelligence Method for Argument Mining
Authors:
Michiel van der Meer,
Enrico Liscio,
Catholijn M. Jonker,
Aske Plaat,
Piek Vossen,
Pradeep K. Murukannaiah
Abstract:
Large-scale survey tools enable the collection of citizen feedback in opinion corpora. Extracting the key arguments from a large and noisy set of opinions helps in understanding the opinions quickly and accurately. Fully automated methods can extract arguments but (1) require large labeled datasets that induce large annotation costs and (2) work well for known viewpoints, but not for novel points…
▽ More
Large-scale survey tools enable the collection of citizen feedback in opinion corpora. Extracting the key arguments from a large and noisy set of opinions helps in understanding the opinions quickly and accurately. Fully automated methods can extract arguments but (1) require large labeled datasets that induce large annotation costs and (2) work well for known viewpoints, but not for novel points of view. We propose HyEnA, a hybrid (human + AI) method for extracting arguments from opinionated texts, combining the speed of automated processing with the understanding and reasoning capabilities of humans. We evaluate HyEnA on three citizen feedback corpora. We find that, on the one hand, HyEnA achieves higher coverage and precision than a state-of-the-art automated method when compared to a common set of diverse opinions, justifying the need for human insight. On the other hand, HyEnA requires less human effort and does not compromise quality compared to (fully manual) expert analysis, demonstrating the benefit of combining human and artificial intelligence.
△ Less
Submitted 11 March, 2024;
originally announced March 2024.
-
Value Preferences Estimation and Disambiguation in Hybrid Participatory Systems
Authors:
Enrico Liscio,
Luciano C. Siebert,
Catholijn M. Jonker,
Pradeep K. Murukannaiah
Abstract:
Understanding citizens' values in participatory systems is crucial for citizen-centric policy-making. We envision a hybrid participatory system where participants make choices and provide motivations for those choices, and AI agents estimate their value preferences by interacting with them. We focus on situations where a conflict is detected between participants' choices and motivations, and propo…
▽ More
Understanding citizens' values in participatory systems is crucial for citizen-centric policy-making. We envision a hybrid participatory system where participants make choices and provide motivations for those choices, and AI agents estimate their value preferences by interacting with them. We focus on situations where a conflict is detected between participants' choices and motivations, and propose methods for estimating value preferences while addressing detected inconsistencies by interacting with the participants. We operationalize the philosophical stance that "valuing is deliberatively consequential." That is, if a participant's choice is based on a deliberation of value preferences, the value preferences can be observed in the motivation the participant provides for the choice. Thus, we propose and compare value estimation methods that prioritize the values estimated from motivations over the values estimated from choices alone. Then, we introduce a disambiguation strategy that addresses the detected inconsistencies between choices and motivations by directly interacting with the participants. We evaluate the proposed methods on a dataset of a large-scale survey on energy transition. The results show that explicitly addressing inconsistencies between choices and motivations improves the estimation of an individual's value preferences. The disambiguation strategy does not show substantial improvements when compared to similar baselines--however, we discuss how the novelty of the approach can open new research avenues and propose improvements to address the current limitations.
△ Less
Submitted 26 February, 2024;
originally announced February 2024.
-
Bayesian Federated Inference for regression models with heterogeneous multi-center populations
Authors:
Marianne A Jonker,
Hassan Pazira,
Anthony CC Coolen
Abstract:
To estimate accurately the parameters of a regression model, the sample size must be large enough relative to the number of possible predictors for the model. In practice, sufficient data is often lacking, which can lead to overfitting of the model and, as a consequence, unreliable predictions of the outcome of new patients. Pooling data from different data sets collected in different (medical) ce…
▽ More
To estimate accurately the parameters of a regression model, the sample size must be large enough relative to the number of possible predictors for the model. In practice, sufficient data is often lacking, which can lead to overfitting of the model and, as a consequence, unreliable predictions of the outcome of new patients. Pooling data from different data sets collected in different (medical) centers would alleviate this problem, but is often not feasible due to privacy regulation or logistic problems. An alternative route would be to analyze the local data in the centers separately and combine the statistical inference results with the Bayesian Federated Inference (BFI) methodology. The aim of this approach is to compute from the inference results in separate centers what would have been found if the statistical analysis was performed on the combined data. We explain the methodology under homogeneity and heterogeneity across the populations in the separate centers, and give real life examples for better understanding. Excellent performance of the proposed methodology is shown. An R-package to do all the calculations has been developed and is illustrated in this paper. The mathematical details are given in the Appendix.
△ Less
Submitted 5 February, 2024;
originally announced February 2024.
-
An Empirical Analysis of Diversity in Argument Summarization
Authors:
Michiel van der Meer,
Piek Vossen,
Catholijn M. Jonker,
Pradeep K. Murukannaiah
Abstract:
Presenting high-level arguments is a crucial task for fostering participation in online societal discussions. Current argument summarization approaches miss an important facet of this task -- capturing diversity -- which is important for accommodating multiple perspectives. We introduce three aspects of diversity: those of opinions, annotators, and sources. We evaluate approaches to a popular argu…
▽ More
Presenting high-level arguments is a crucial task for fostering participation in online societal discussions. Current argument summarization approaches miss an important facet of this task -- capturing diversity -- which is important for accommodating multiple perspectives. We introduce three aspects of diversity: those of opinions, annotators, and sources. We evaluate approaches to a popular argument summarization task called Key Point Analysis, which shows how these approaches struggle to (1) represent arguments shared by few people, (2) deal with data from various sources, and (3) align with subjectivity in human-provided annotations. We find that both general-purpose LLMs and dedicated KPA models exhibit this behavior, but have complementary strengths. Further, we observe that diversification of training data may ameliorate generalization. Addressing diversity in argument summarization requires a mix of strategies to deal with subjectivity.
△ Less
Submitted 14 February, 2024; v1 submitted 2 February, 2024;
originally announced February 2024.
-
Enabling the Digital Democratic Revival: A Research Program for Digital Democracy
Authors:
Davide Grossi,
Ulrike Hahn,
Michael Mäs,
Andreas Nitsche,
Jan Behrens,
Niclas Boehmer,
Markus Brill,
Ulle Endriss,
Umberto Grandi,
Adrian Haret,
Jobst Heitzig,
Nicolien Janssens,
Catholijn M. Jonker,
Marijn A. Keijzer,
Axel Kistner,
Martin Lackner,
Alexandra Lieben,
Anna Mikhaylovskaya,
Pradeep K. Murukannaiah,
Carlo Proietti,
Manon Revel,
Élise Rouméas,
Ehud Shapiro,
Gogulapati Sreedurga,
Björn Swierczek
, et al. (4 additional authors not shown)
Abstract:
This white paper outlines a long-term scientific vision for the development of digital-democracy technology. We contend that if digital democracy is to meet the ambition of enabling a participatory renewal in our societies, then a comprehensive multi-methods research effort is required that could, over the years, support its development in a democratically principled, empirically and computational…
▽ More
This white paper outlines a long-term scientific vision for the development of digital-democracy technology. We contend that if digital democracy is to meet the ambition of enabling a participatory renewal in our societies, then a comprehensive multi-methods research effort is required that could, over the years, support its development in a democratically principled, empirically and computationally informed way. The paper is co-authored by an international and interdisciplinary team of researchers and arose from the Lorentz Center Workshop on ``Algorithmic Technology for Democracy'' (Leiden, October 2022).
△ Less
Submitted 30 January, 2024;
originally announced January 2024.
-
A Systematic Review on Fostering Appropriate Trust in Human-AI Interaction
Authors:
Siddharth Mehrotra,
Chadha Degachi,
Oleksandra Vereschak,
Catholijn M. Jonker,
Myrthe L. Tielman
Abstract:
Appropriate Trust in Artificial Intelligence (AI) systems has rapidly become an important area of focus for both researchers and practitioners. Various approaches have been used to achieve it, such as confidence scores, explanations, trustworthiness cues, or uncertainty communication. However, a comprehensive understanding of the field is lacking due to the diversity of perspectives arising from v…
▽ More
Appropriate Trust in Artificial Intelligence (AI) systems has rapidly become an important area of focus for both researchers and practitioners. Various approaches have been used to achieve it, such as confidence scores, explanations, trustworthiness cues, or uncertainty communication. However, a comprehensive understanding of the field is lacking due to the diversity of perspectives arising from various backgrounds that influence it and the lack of a single definition for appropriate trust. To investigate this topic, this paper presents a systematic review to identify current practices in building appropriate trust, different ways to measure it, types of tasks used, and potential challenges associated with it. We also propose a Belief, Intentions, and Actions (BIA) map** to study commonalities and differences in the concepts related to appropriate trust by (a) describing the existing disagreements on defining appropriate trust, and (b) providing an overview of the concepts and definitions related to appropriate trust in AI from the existing literature. Finally, the challenges identified in studying appropriate trust are discussed, and observations are summarized as current trends, potential gaps, and research opportunities for future work. Overall, the paper provides insights into the complex concept of appropriate trust in human-AI interaction and presents research opportunities to advance our understanding on this topic.
△ Less
Submitted 8 November, 2023;
originally announced November 2023.
-
Packed to the Brim: Investigating the Impact of Highly Responsive Prefixes on Internet-wide Measurement Campaigns
Authors:
Patrick Sattler,
Johannes Zirngibl,
Mattijs Jonker,
Oliver Gasser,
Georg Carle,
Ralph Holz
Abstract:
Internet-wide scans are an important tool to evaluate the deployment of services. To enable large-scale application layer scans, a fast, stateless port scan (e.g., using ZMap) is often performed ahead of time to collect responsive targets. It is a common expectation that port scans on the entire IPv4 address space provide a relatively unbiased view as they cover the complete address space. Previou…
▽ More
Internet-wide scans are an important tool to evaluate the deployment of services. To enable large-scale application layer scans, a fast, stateless port scan (e.g., using ZMap) is often performed ahead of time to collect responsive targets. It is a common expectation that port scans on the entire IPv4 address space provide a relatively unbiased view as they cover the complete address space. Previous work, however, has found prefixes where all addresses share particular properties. In IPv6, aliased prefixes and fully responsive prefixes, i.e., prefixes where all addresses are responsive, are a well-known phenomenon. However, there is no such in-depth analysis for prefixes with these responsiveness patterns in IPv4. This paper delves into the underlying factors of this phenomenon in the context of IPv4 and evaluates port scans on a total of 161 ports (142 TCP & 19 UDP ports) from three different vantage points. To account for packet loss and other scanning artifacts, we propose the notion of a new category of prefixes, which we call highly responsive prefixes (HRPs). Our findings show that the share of HRPs can make up 70 % of responsive addresses on selected ports. Regarding specific ports, we observe that CDNs contribute to the largest fraction of HRPs on TCP/80 and TCP/443, while TCP proxies emerge as the primary cause of HRPs on other ports. Our analysis also reveals that application layer handshakes to targets outside HRPs are, depending on the chosen service, up to three times more likely to be successful compared to handshakes with targets located in HRPs. To improve future scanning campaigns conducted by the research community, we make our study's data publicly available and provide a tool for detecting HRPs. Furthermore, we propose an approach for a more efficient, ethical, and sustainable application layer target selection.
△ Less
Submitted 25 October, 2023;
originally announced October 2023.
-
Do Differences in Values Influence Disagreements in Online Discussions?
Authors:
Michiel van der Meer,
Piek Vossen,
Catholijn M. Jonker,
Pradeep K. Murukannaiah
Abstract:
Disagreements are common in online discussions. Disagreement may foster collaboration and improve the quality of a discussion under some conditions. Although there exist methods for recognizing disagreement, a deeper understanding of factors that influence disagreement is lacking in the literature. We investigate a hypothesis that differences in personal values are indicative of disagreement in on…
▽ More
Disagreements are common in online discussions. Disagreement may foster collaboration and improve the quality of a discussion under some conditions. Although there exist methods for recognizing disagreement, a deeper understanding of factors that influence disagreement is lacking in the literature. We investigate a hypothesis that differences in personal values are indicative of disagreement in online discussions. We show how state-of-the-art models can be used for estimating values in online discussions and how the estimated values can be aggregated into value profiles. We evaluate the estimated value profiles based on human-annotated agreement labels. We find that the dissimilarity of value profiles correlates with disagreement in specific cases. We also find that including value information in agreement prediction improves performance.
△ Less
Submitted 24 October, 2023;
originally announced October 2023.
-
This Is a Local Domain: On Amassing Country-Code Top-Level Domains from Public Data
Authors:
Raffaele Sommese,
Roland van Rijswijk-Deij,
Mattijs Jonker
Abstract:
Domain lists are a key ingredient for representative censuses of the Web. Unfortunately, such censuses typically lack a view on domains under country-code top-level domains (ccTLDs). This introduces unwanted bias: many countries have a rich local Web that remains hidden if their ccTLDs are not considered. The reason ccTLDs are rarely considered is that gaining access -- if possible at all -- is of…
▽ More
Domain lists are a key ingredient for representative censuses of the Web. Unfortunately, such censuses typically lack a view on domains under country-code top-level domains (ccTLDs). This introduces unwanted bias: many countries have a rich local Web that remains hidden if their ccTLDs are not considered. The reason ccTLDs are rarely considered is that gaining access -- if possible at all -- is often laborious. To tackle this, we ask: what can we learn about ccTLDs from public sources? We extract domain names under ccTLDs from 6 years of public data from Certificate Transparency logs and Common Crawl. We compare this against ground truth for 19 ccTLDs for which we have the full DNS zone. We find that public data covers 43%-80% of these ccTLDs, and that coverage grows over time. By also comparing port scan data we then show that these public sources reveal a significant part of the Web presence under a ccTLD. We conclude that in the absence of full access to ccTLDs, domain names learned from public sources can be a good proxy when performing Web censuses.
△ Less
Submitted 4 September, 2023;
originally announced September 2023.
-
Reflective Hybrid Intelligence for Meaningful Human Control in Decision-Support Systems
Authors:
Catholijn M. Jonker,
Luciano Cavalcante Siebert,
Pradeep K. Murukannaiah
Abstract:
With the growing capabilities and pervasiveness of AI systems, societies must collectively choose between reduced human autonomy, endangered democracies and limited human rights, and AI that is aligned to human and social values, nurturing collaboration, resilience, knowledge and ethical behaviour. In this chapter, we introduce the notion of self-reflective AI systems for meaningful human control…
▽ More
With the growing capabilities and pervasiveness of AI systems, societies must collectively choose between reduced human autonomy, endangered democracies and limited human rights, and AI that is aligned to human and social values, nurturing collaboration, resilience, knowledge and ethical behaviour. In this chapter, we introduce the notion of self-reflective AI systems for meaningful human control over AI systems. Focusing on decision support systems, we propose a framework that integrates knowledge from psychology and philosophy with formal reasoning methods and machine learning approaches to create AI systems responsive to human values and social norms. We also propose a possible research approach to design and develop self-reflective capability in AI systems. Finally, we argue that self-reflective AI systems can lead to self-reflective hybrid systems (human + AI), thus increasing meaningful human control and empowering human moral reasoning by providing comprehensible information and insights on possible human moral blind spots.
△ Less
Submitted 12 July, 2023;
originally announced July 2023.
-
Assessing Network Operator Actions to Enhance Digital Sovereignty and Strengthen Network Resilience: A Longitudinal Analysis during the Russia-Ukraine Conflict
Authors:
Muhammad Yasir Muzayan Haq,
Abhishta Abhishta,
Raffaele Sommese,
Mattijs Jonker,
Lambert J. M. Nieuwenhuis
Abstract:
We conduct longitudinal and temporal analyses on active DNS measurement data to investigate how the Russia-Ukraine conflict impacted the network infrastructures supporting domain names under ICANN's CZDS new gTLDs. Our findings revealed changes in the physical locations of network infrastructures, utilization of managed DNS services, infrastructure redundancy, and distribution, which started right…
▽ More
We conduct longitudinal and temporal analyses on active DNS measurement data to investigate how the Russia-Ukraine conflict impacted the network infrastructures supporting domain names under ICANN's CZDS new gTLDs. Our findings revealed changes in the physical locations of network infrastructures, utilization of managed DNS services, infrastructure redundancy, and distribution, which started right after the first reported Russian military movements in February 2022. We also found that domains from different countries had varying location preferences when moving their hosting infrastructure. These observed changes suggest that network operators took proactive measures in anticipation of an armed conflict to promote resilience and protect the sovereignty of their networks in response to the conflict.
△ Less
Submitted 28 May, 2023;
originally announced May 2023.
-
How mature are survival data at the time of an interim analysis in a clinical trial with a survival outcome?
Authors:
Marianne A Jonker,
Steven Teerenstra
Abstract:
In a clinical trial with a survival outcome, an interim analysis is often performed to allow for early stop** for efficacy. If the interim analysis is early in the trial, one might conclude that a new treatment is more effective (compared to e.g.\ a placebo) and stop the trial, whereas the survival curves in the trial arms are not mature for the research question under investigation, for example…
▽ More
In a clinical trial with a survival outcome, an interim analysis is often performed to allow for early stop** for efficacy. If the interim analysis is early in the trial, one might conclude that a new treatment is more effective (compared to e.g.\ a placebo) and stop the trial, whereas the survival curves in the trial arms are not mature for the research question under investigation, for example because the curves are still close to 1 at that time. This means that the decision is based on a small percentage of the events in the long run only; possibly the events of the more frail patients in the trial who may not be representative for the whole group of patients. It may not be sensible to conclude effectiveness based on so little information. Criteria to determine the moment the interim analysis will be performed, should be chosen with care, and include the maturity of the data at the time of the interim analysis. Here, the expected survival rates at the interim analysis play a role. In this paper we will derive the asymptotic distribution of the Kaplan-Meier curves at the (random) moment the interim analysis will be performed for a one and two arm clinical trial. Based on this distribution, an interval in which the Kaplan Meier curves will fall into (with probability 95\%) is derived and could be used to plan the moment of the interim analysis in the design stage of the trial, so before the trial starts.
△ Less
Submitted 6 May, 2023;
originally announced May 2023.
-
Registered Report : Perception of Other's Musical Preferences Based on Their Personal Values
Authors:
Sandy Manolios,
Catholijn M. Jonker,
Cynthia C. S. Liem
Abstract:
The present work is part of a research line seeking to uncover the mysteries of what lies behind people's musical preferences in order to provide better music recommendations. More specifically, it takes the angle of personal values. Personal values are what we as people strive for, and are a popular tool in marketing research to understand customer preferences for certain types of product. Theref…
▽ More
The present work is part of a research line seeking to uncover the mysteries of what lies behind people's musical preferences in order to provide better music recommendations. More specifically, it takes the angle of personal values. Personal values are what we as people strive for, and are a popular tool in marketing research to understand customer preferences for certain types of product. Therefore, it makes sense to explore their usefulness in the music domain. Based on a previous qualitative work using the Means-End theory, we designed a survey in an attempt to more quantitatively approach the relationship between personal values and musical preferences. We support our approach with a simulation study as a tool to improve the experimental procedure and decisions.
△ Less
Submitted 20 February, 2023;
originally announced February 2023.
-
Bayesian Federated Inference for estimating Statistical Models based on Non-shared Multicenter Data sets
Authors:
Marianne A. Jonker,
Hassan Pazira,
Anthony CC Coolen
Abstract:
Identifying predictive factors for an outcome of interest via a multivariable analysis is often difficult when the data set is small. Combining data from different medical centers into a single (larger) database would alleviate this problem, but is in practice challenging due to regulatory and logistic problems. Federated Learning (FL) is a machine learning approach that aims to construct from loc…
▽ More
Identifying predictive factors for an outcome of interest via a multivariable analysis is often difficult when the data set is small. Combining data from different medical centers into a single (larger) database would alleviate this problem, but is in practice challenging due to regulatory and logistic problems. Federated Learning (FL) is a machine learning approach that aims to construct from local inferences in separate data centers what would have been inferred had the data sets been merged. It seeks to harvest the statistical power of larger data sets without actually creating them. The FL strategy is not always efficient and precise. Therefore, in this paper we refine and implement an alternative Bayesian Federated Inference (BFI) framework for multicenter data with the same aim as FL. The BFI framework is designed to cope with small data sets by inferring locally not only the optimal parameter values, but also additional features of the posterior parameter distribution, capturing information beyond what is used in FL. BFI has the additional benefit that a single inference cycle across the centers is sufficient, whereas FL needs multiple cycles. We quantify the performance of the proposed methodology on simulated and real life data.
△ Less
Submitted 9 March, 2024; v1 submitted 15 February, 2023;
originally announced February 2023.
-
Forward Pass: On the Security Implications of Email Forwarding Mechanism and Policy
Authors:
Enze Liu,
Gautam Akiwate,
Mattijs Jonker,
Ariana Mirian,
Grant Ho,
Geoffrey M. Voelker,
Stefan Savage
Abstract:
The critical role played by email has led to a range of extension protocols (e.g., SPF, DKIM, DMARC) designed to protect against the spoofing of email sender domains. These protocols are complex as is, but are further complicated by automated email forwarding -- used by individual users to manage multiple accounts and by mailing lists to redistribute messages. In this paper, we explore how such em…
▽ More
The critical role played by email has led to a range of extension protocols (e.g., SPF, DKIM, DMARC) designed to protect against the spoofing of email sender domains. These protocols are complex as is, but are further complicated by automated email forwarding -- used by individual users to manage multiple accounts and by mailing lists to redistribute messages. In this paper, we explore how such email forwarding and its implementations can break the implicit assumptions in widely deployed anti-spoofing protocols. Using large-scale empirical measurements of 20 email forwarding services (16 leading email providers and four popular mailing list services), we identify a range of security issues rooted in forwarding behavior and show how they can be combined to reliably evade existing anti-spoofing controls. We further show how these issues allow attackers to not only deliver spoofed email messages to prominent email providers (e.g., Gmail, Microsoft Outlook, and Zoho), but also reliably spoof email on behalf of tens of thousands of popular domains including sensitive domains used by organizations in government (e.g., state.gov), finance (e.g., transunion.com), law (e.g., perkinscoie.com) and news (e.g., washingtonpost.com) among others.
△ Less
Submitted 19 April, 2023; v1 submitted 14 February, 2023;
originally announced February 2023.
-
AI Alignment Dialogues: An Interactive Approach to AI Alignment in Support Agents
Authors:
Pei-Yu Chen,
Myrthe L. Tielman,
Dirk K. J. Heylen,
Catholijn M. Jonker,
M. Birna van Riemsdijk
Abstract:
AI alignment is about ensuring AI systems only pursue goals and activities that are beneficial to humans. Most of the current approach to AI alignment is to learn what humans value from their behavioural data. This paper proposes a different way of looking at the notion of alignment, namely by introducing AI Alignment Dialogues: dialogues with which users and agents try to achieve and maintain ali…
▽ More
AI alignment is about ensuring AI systems only pursue goals and activities that are beneficial to humans. Most of the current approach to AI alignment is to learn what humans value from their behavioural data. This paper proposes a different way of looking at the notion of alignment, namely by introducing AI Alignment Dialogues: dialogues with which users and agents try to achieve and maintain alignment via interaction. We argue that alignment dialogues have a number of advantages in comparison to data-driven approaches, especially for behaviour support agents, which aim to support users in achieving their desired future behaviours rather than their current behaviours. The advantages of alignment dialogues include allowing the users to directly convey higher-level concepts to the agent, and making the agent more transparent and trustworthy. In this paper we outline the concept and high-level structure of alignment dialogues. Moreover, we conducted a qualitative focus group user study from which we developed a model that describes how alignment dialogues affect users, and created design suggestions for AI alignment dialogues. Through this we establish foundations for AI alignment dialogues and shed light on what requires further development and research.
△ Less
Submitted 5 October, 2023; v1 submitted 16 January, 2023;
originally announced January 2023.
-
Automated Configuration and Usage of Strategy Portfolios for Bargaining
Authors:
Bram M. Renting,
Holger H. Hoos,
Catholijn M. Jonker
Abstract:
Bargaining can be used to resolve mixed-motive games in multi-agent systems. Although there is an abundance of negotiation strategies implemented in automated negotiating agents, most agents are based on single fixed strategies, while it is widely acknowledged that there is no single best-performing strategy for all negotiation settings.
In this paper, we focus on bargaining settings where oppon…
▽ More
Bargaining can be used to resolve mixed-motive games in multi-agent systems. Although there is an abundance of negotiation strategies implemented in automated negotiating agents, most agents are based on single fixed strategies, while it is widely acknowledged that there is no single best-performing strategy for all negotiation settings.
In this paper, we focus on bargaining settings where opponents are repeatedly encountered, but the bargaining problems change. We introduce a novel method that automatically creates and deploys a portfolio of complementary negotiation strategies using a training set and optimise pay-off in never-before-seen bargaining settings through per-setting strategy selection. Our method relies on the following contributions. We introduce a feature representation that captures characteristics for both the opponent and the bargaining problem. We model the behaviour of an opponent during a negotiation based on its actions, which is indicative of its negotiation strategy, in order to be more effective in future encounters.
Our combination of feature-based methods generalises to new negotiation settings, as in practice, over time, it selects effective counter strategies in future encounters. Our approach is tested in an ANAC-like tournament, and we show that we are capable of winning such a tournament with a 5.6% increase in pay-off compared to the runner-up agent.
△ Less
Submitted 20 December, 2022;
originally announced December 2022.
-
Exploring Effectiveness of Explanations for Appropriate Trust: Lessons from Cognitive Psychology
Authors:
Ruben S. Verhagen,
Siddharth Mehrotra,
Mark A. Neerincx,
Catholijn M. Jonker,
Myrthe L. Tielman
Abstract:
The rapid development of Artificial Intelligence (AI) requires developers and designers of AI systems to focus on the collaboration between humans and machines. AI explanations of system behavior and reasoning are vital for effective collaboration by fostering appropriate trust, ensuring understanding, and addressing issues of fairness and bias. However, various contextual and subjective factors c…
▽ More
The rapid development of Artificial Intelligence (AI) requires developers and designers of AI systems to focus on the collaboration between humans and machines. AI explanations of system behavior and reasoning are vital for effective collaboration by fostering appropriate trust, ensuring understanding, and addressing issues of fairness and bias. However, various contextual and subjective factors can influence an AI system explanation's effectiveness. This work draws inspiration from findings in cognitive psychology to understand how effective explanations can be designed. We identify four components to which explanation designers can pay special attention: perception, semantics, intent, and user & context. We illustrate the use of these four explanation components with an example of estimating food calories by combining text with visuals, probabilities with exemplars, and intent communication with both user and context in mind. We propose that the significant challenge for effective AI explanations is an additional step between explanation generation using algorithms not producing interpretable explanations and explanation communication. We believe this extra step will benefit from carefully considering the four explanation components outlined in our work, which can positively affect the explanation's effectiveness.
△ Less
Submitted 5 October, 2022;
originally announced October 2022.
-
Penalization-induced shrinking without rotation in high dimensional GLM regression: a cavity analysis
Authors:
Emanuele Massa,
Marianne Jonker,
Anthony Coolen
Abstract:
In high dimensional regression, where the number of covariates is of the order of the number of observations, ridge penalization is often used as a remedy against overfitting. Unfortunately, for correlated covariates such regularisation typically induces in generalized linear models not only shrinking of the estimated parameter vector, but also an unwanted \emph{rotation} relative to the true vect…
▽ More
In high dimensional regression, where the number of covariates is of the order of the number of observations, ridge penalization is often used as a remedy against overfitting. Unfortunately, for correlated covariates such regularisation typically induces in generalized linear models not only shrinking of the estimated parameter vector, but also an unwanted \emph{rotation} relative to the true vector. We show analytically how this problem can be removed by using a generalization of ridge penalization, and we analyse the asymptotic properties of the corresponding estimators in the high dimensional regime, using the cavity method. Our results also provide a quantitative rationale for tuning the parameter that controlling the amount of shrinking. We compare our theoretical predictions with simulated data and find excellent agreement.
△ Less
Submitted 9 September, 2022;
originally announced September 2022.
-
No Time for Downtime: Understanding Post-Attack Behaviors by Customers of Managed DNS Providers
Authors:
Muhammad Yasir Muzayan Haq,
Mattijs Jonker,
Roland van Rijswijk-Deij,
KC Claffy,
Lambert J. M. Nieuwenhuis,
Abhishta Abhishta
Abstract:
We leverage large-scale DNS measurement data on authoritative name servers to study the reactions of domain owners affected by the 2016 DDoS attack on Dyn. We use industry sources of information about domain names to study the influence of factors such as industry sector and website popularity on the willingness of domain managers to invest in high availability of online services. Specifically, we…
▽ More
We leverage large-scale DNS measurement data on authoritative name servers to study the reactions of domain owners affected by the 2016 DDoS attack on Dyn. We use industry sources of information about domain names to study the influence of factors such as industry sector and website popularity on the willingness of domain managers to invest in high availability of online services. Specifically, we correlate business characteristics of domain owners with their resilience strategies in the wake of DoS attacks affecting their domains. Our analysis revealed correlations between two properties of domains -- industry sector and popularity -- and post-attack strategies. Specifically, owners of more popular domains were more likely to re-act to increase the diversity of their authoritative DNS service for their domains. Similarly, domains in certain industry sectors were more likely to seek out such diversity in their DNS service. For example, domains categorized as General News were nearly 6 times more likely to re-act than domains categorized as Internet Services. Our results can inform managed DNS and other network service providers regarding the potential impact of downtime on their customer portfolio.
△ Less
Submitted 25 May, 2022;
originally announced May 2022.
-
MOPaC: The Multiple Offers Protocol for Multilateral Negotiations with Partial Consensus
Authors:
Pradeep K. Murukannaiah,
Catholijn M. Jonker
Abstract:
Existing protocols for multilateral negotiation require a full consensus among the negotiating parties. In contrast, we propose a protocol for multilateral negotiation that allows partial consensus, wherein only a subset of the negotiating parties can reach an agreement. We motivate problems that require such a protocol and describe the protocol formally.
Existing protocols for multilateral negotiation require a full consensus among the negotiating parties. In contrast, we propose a protocol for multilateral negotiation that allows partial consensus, wherein only a subset of the negotiating parties can reach an agreement. We motivate problems that require such a protocol and describe the protocol formally.
△ Less
Submitted 13 May, 2022;
originally announced May 2022.
-
Correction of overfitting bias in regression models
Authors:
Emanuele Massa,
Marianne Jonker,
Kit Roes,
Anthony Coolen
Abstract:
Regression analysis based on many covariates is becoming increasingly common. However, when the number of covariates $p$ is of the same order as the number of observations $n$, maximum likelihood regression becomes unreliable due to overfitting. This typically leads to systematic estimation biases and increased estimator variances. It is crucial for inference and prediction to quantify these effec…
▽ More
Regression analysis based on many covariates is becoming increasingly common. However, when the number of covariates $p$ is of the same order as the number of observations $n$, maximum likelihood regression becomes unreliable due to overfitting. This typically leads to systematic estimation biases and increased estimator variances. It is crucial for inference and prediction to quantify these effects correctly. Several methods have been proposed in literature to overcome overfitting bias or adjust estimates. The vast majority of these focus on the regression parameters. But failure to estimate correctly also the nuisance parameters may lead to significant errors in confidence statements and outcome prediction.
In this paper we present a jacknife method for deriving a compact set of non-linear equations which describe the statistical properties of the ML estimator in the regime where $p=O(n)$ and under the hypothesis of normally distributed covariates. These equations enable one to compute the overfitting bias of maximum likelihood (ML) estimators in parametric regression models as functions of $ζ= p/n$. We then use these equations to compute shrinkage factors in order to remove the overfitting bias of maximum likelihood (ML) estimators. This new derivation offers various benefits over the replica approach in terms of increased transparency and reduced assumptions. To illustrate the theory we performed simulation studies for multiple regression models. In all cases we find excellent agreement between theory and simulations.
△ Less
Submitted 4 September, 2023; v1 submitted 12 April, 2022;
originally announced April 2022.
-
Saving Brian's Privacy: the Perils of Privacy Exposure through Reverse DNS
Authors:
Olivier van der Toorn,
Raffaele Sommese,
Anna Sperotto,
Roland van Rijswijk-Deij,
Mattijs Jonker
Abstract:
Given the importance of privacy, many Internet protocols are nowadays designed with privacy in mind (e.g., using TLS for confidentiality). Foreseeing all privacy issues at the time of protocol design is, however, challenging and may become near impossible when interaction out of protocol bounds occurs. One demonstrably not well understood interaction occurs when DHCP exchanges are accompanied by a…
▽ More
Given the importance of privacy, many Internet protocols are nowadays designed with privacy in mind (e.g., using TLS for confidentiality). Foreseeing all privacy issues at the time of protocol design is, however, challenging and may become near impossible when interaction out of protocol bounds occurs. One demonstrably not well understood interaction occurs when DHCP exchanges are accompanied by automated changes to the global DNS (e.g., to dynamically add hostnames for allocated IP addresses). As we will substantiate, this is a privacy risk: one may be able to infer device presence and network dynamics from virtually anywhere on the Internet -- and even identify and track individuals -- even if other mechanisms to limit tracking by outsiders (e.g., blocking **s) are in place.
We present a first of its kind study into this risk. We identify networks that expose client identifiers in reverse DNS records and study the relation between the presence of clients and said records. Our results show a strong link: in 9 out of 10 cases, records linger for at most an hour, for a selection of academic, enterprise and ISP networks alike. We also demonstrate how client patterns and network dynamics can be learned, by tracking devices owned by persons named Brian over time, revealing shifts in work patterns caused by COVID-19 related work-from-home measures, and by determining a good time to stage a heist.
△ Less
Submitted 20 September, 2022; v1 submitted 2 February, 2022;
originally announced February 2022.
-
Towards a Real-time Measure of the Perception of Anthropomorphism in Human-robot Interaction
Authors:
Maria Tsfasman,
Avinash Saravanan,
Dekel Viner,
Daan Goslinga,
Sarah de Wolf,
Chirag Raman,
Catholijn M. Jonker,
Catharine Oertel
Abstract:
How human-like do conversational robots need to look to enable long-term human-robot conversation? One essential aspect of long-term interaction is a human's ability to adapt to the varying degrees of a conversational partner's engagement and emotions. Prosodically, this can be achieved through (dis)entrainment. While speech-synthesis has been a limiting factor for many years, restrictions in this…
▽ More
How human-like do conversational robots need to look to enable long-term human-robot conversation? One essential aspect of long-term interaction is a human's ability to adapt to the varying degrees of a conversational partner's engagement and emotions. Prosodically, this can be achieved through (dis)entrainment. While speech-synthesis has been a limiting factor for many years, restrictions in this regard are increasingly mitigated. These advancements now emphasise the importance of studying the effect of robot embodiment on human entrainment. In this study, we conducted a between-subjects online human-robot interaction experiment in an educational use-case scenario where a tutor was either embodied through a human or a robot face. 43 English-speaking participants took part in the study for whom we analysed the degree of acoustic-prosodic entrainment to the human or robot face, respectively. We found that the degree of subjective and objective perception of anthropomorphism positively correlates with acoustic-prosodic entrainment.
△ Less
Submitted 24 January, 2022;
originally announced January 2022.
-
Meaningful human control: actionable properties for AI system development
Authors:
Luciano Cavalcante Siebert,
Maria Luce Lupetti,
Evgeni Aizenberg,
Niek Beckers,
Arkady Zgonnikov,
Herman Veluwenkamp,
David Abbink,
Elisa Giaccardi,
Geert-Jan Houben,
Catholijn M. Jonker,
Jeroen van den Hoven,
Deborah Forster,
Reginald L. Lagendijk
Abstract:
How can humans remain in control of artificial intelligence (AI)-based systems designed to perform tasks autonomously? Such systems are increasingly ubiquitous, creating benefits - but also undesirable situations where moral responsibility for their actions cannot be properly attributed to any particular person or group. The concept of meaningful human control has been proposed to address responsi…
▽ More
How can humans remain in control of artificial intelligence (AI)-based systems designed to perform tasks autonomously? Such systems are increasingly ubiquitous, creating benefits - but also undesirable situations where moral responsibility for their actions cannot be properly attributed to any particular person or group. The concept of meaningful human control has been proposed to address responsibility gaps and mitigate them by establishing conditions that enable a proper attribution of responsibility for humans; however, clear requirements for researchers, designers, and engineers are yet inexistent, making the development of AI-based systems that remain under meaningful human control challenging. In this paper, we address the gap between philosophical theory and engineering practice by identifying, through an iterative process of abductive thinking, four actionable properties for AI-based systems under meaningful human control, which we discuss making use of two applications scenarios: automated vehicles and AI-based hiring. First, a system in which humans and AI algorithms interact should have an explicitly defined domain of morally loaded situations within which the system ought to operate. Second, humans and AI agents within the system should have appropriate and mutually compatible representations. Third, responsibility attributed to a human should be commensurate with that human's ability and authority to control the system. Fourth, there should be explicit links between the actions of the AI agents and actions of humans who are aware of their moral responsibility. We argue that these four properties will support practically-minded professionals to take concrete steps toward designing and engineering for AI systems that facilitate meaningful human control.
△ Less
Submitted 19 May, 2022; v1 submitted 25 November, 2021;
originally announced December 2021.
-
Towards Social Situation Awareness in Support Agents
Authors:
Ilir Kola,
Pradeep K. Murukannaiah,
Catholijn M. Jonker,
M. Birna van Riemsdijk
Abstract:
Artificial agents that support people in their daily activities (e.g., virtual coaches and personal assistants) are increasingly prevalent. Since many daily activities are social in nature, support agents should understand a user's social situation to offer comprehensive support. However, there are no systematic approaches for develo** support agents that are social situation aware. We identify…
▽ More
Artificial agents that support people in their daily activities (e.g., virtual coaches and personal assistants) are increasingly prevalent. Since many daily activities are social in nature, support agents should understand a user's social situation to offer comprehensive support. However, there are no systematic approaches for develo** support agents that are social situation aware. We identify key requirements for a support agent to be social situation aware and propose steps to realize those requirements. These steps are presented through a conceptual architecture centered on two key ideas: (1) conceptualizing social situation awareness as an instantiation of `general' situation awareness, and (2) using situation taxonomies for such instantiation. This enables support agents to represent a user's social situation, comprehend its meaning, and assess its impact on the user's behavior. We discuss empirical results supporting the effectiveness of the proposed approach and illustrate how the architecture can be used in support agents through two use cases.
△ Less
Submitted 4 April, 2022; v1 submitted 19 October, 2021;
originally announced October 2021.
-
Using Psychological Characteristics of Situations for Social Situation Comprehension in Support Agents
Authors:
Ilir Kola,
Catholijn M. Jonker,
M. Birna van Riemsdijk
Abstract:
Support agents that help users in their daily lives need to take into account not only the user's characteristics, but also the social situation of the user. Existing work on including social context uses some type of situation cue as an input to information processing techniques in order to assess the expected behavior of the user. However, research shows that it is important to also determine th…
▽ More
Support agents that help users in their daily lives need to take into account not only the user's characteristics, but also the social situation of the user. Existing work on including social context uses some type of situation cue as an input to information processing techniques in order to assess the expected behavior of the user. However, research shows that it is important to also determine the meaning of a situation, a step which we refer to as social situation comprehension. We propose using psychological characteristics of situations, which have been proposed in social science for ascribing meaning to situations, as the basis for social situation comprehension. Using data from user studies, we evaluate this proposal from two perspectives. First, from a technical perspective, we show that psychological characteristics of situations can be used as input to predict the priority of social situations, and that psychological characteristics of situations can be predicted from the features of a social situation. Second, we investigate the role of the comprehension step in human-machine meaning making. We show that psychological characteristics can be successfully used as a basis for explanations given to users about the decisions of an agenda management personal assistant agent.
△ Less
Submitted 13 July, 2022; v1 submitted 15 October, 2021;
originally announced October 2021.
-
From Organisational Structure to Organisational Behaviour Formalisation
Authors:
Catholijn M. Jonker,
Jan Treur
Abstract:
To understand how an organisational structure relates to organisational behaviour is an interesting fundamental challenge in the area of organisation modelling. Specifications of organisational structure usually have a diagrammatic form that abstracts from more detailed dynamics. Dynamic properties of agent systems, on the other hand, are often specified in the form of a set of logical formulae in…
▽ More
To understand how an organisational structure relates to organisational behaviour is an interesting fundamental challenge in the area of organisation modelling. Specifications of organisational structure usually have a diagrammatic form that abstracts from more detailed dynamics. Dynamic properties of agent systems, on the other hand, are often specified in the form of a set of logical formulae in some temporal language. This paper addresses the question how these two perspectives can be combined in one framework. It is shown how for different aggregation levels and other elements within an organisation structure, sets of dynamic properties can be specified. Organisational structure provides a structure of (interlevel) relationships between these multiple sets of dynamic properties. Thus organisational structure is reflected in the formalisation of the dynamics of organisational behaviour. To illustrate the effectiveness of the approach a formal foundation is presented for the integrated specification of both structure and behaviour of an AGR organisation model.
△ Less
Submitted 29 September, 2021;
originally announced September 2021.
-
Hosting Industry Centralization and Consolidation
Authors:
Luciano Zembruzki,
Raffaele Sommese,
Lisandro Zambenedetti Granville,
Arthur Selle Jacobs,
Mattijs Jonker,
Giovane C. M. Moura
Abstract:
There have been growing concerns about the concentration and centralization of Internet infrastructure. In this work, we scrutinize the hosting industry on the Internet by using active measurements, covering 19 Top-Level Domains (TLDs). We show how the market is heavily concentrated: 1/3 of the domains are hosted by only 5 hosting providers, all US-based companies. For the country-code TLDs (ccTLD…
▽ More
There have been growing concerns about the concentration and centralization of Internet infrastructure. In this work, we scrutinize the hosting industry on the Internet by using active measurements, covering 19 Top-Level Domains (TLDs). We show how the market is heavily concentrated: 1/3 of the domains are hosted by only 5 hosting providers, all US-based companies. For the country-code TLDs (ccTLDs), however, hosting is primarily done by local, national hosting providers and not by the large American cloud and content providers. We show how shared languages (and borders) shape the hosting market -- German hosting companies have a notable presence in Austrian and Swiss markets, given they all share German as official language. While hosting concentration has been relatively high and stable over the past four years, we see that American hosting companies have been continuously increasing their presence in the market related to high traffic, popular domains within ccTLDs -- except for Russia, notably.
△ Less
Submitted 25 January, 2022; v1 submitted 2 September, 2021;
originally announced September 2021.
-
The Far Side of DNS Amplification: Tracing the DDoS Attack Ecosystem from the Internet Core
Authors:
Marcin Nawrocki,
Mattijs Jonker,
Thomas C. Schmidt,
Matthias Wählisch
Abstract:
In this paper, we shed new light on the DNS amplification ecosystem, by studying complementary data sources, bolstered by orthogonal methodologies. First, we introduce a passive attack detection method for the Internet core, i.e., at Internet eXchange Points (IXPs). Surprisingly, IXPs and honeypots observe mostly disjoint sets of attacks: 96% of IXP-inferred attacks were invisible to a sizable hon…
▽ More
In this paper, we shed new light on the DNS amplification ecosystem, by studying complementary data sources, bolstered by orthogonal methodologies. First, we introduce a passive attack detection method for the Internet core, i.e., at Internet eXchange Points (IXPs). Surprisingly, IXPs and honeypots observe mostly disjoint sets of attacks: 96% of IXP-inferred attacks were invisible to a sizable honeypot platform. Second, we assess the effectiveness of observed DNS attacks by studying IXP traces jointly with diverse data from independent measurement infrastructures. We find that attackers efficiently detect new reflectors and purposefully rotate between them. At the same time, we reveal that attackers are a small step away from bringing about significantly higher amplification factors (14x). Third, we identify and fingerprint a major attack entity by studying patterns in attack traces. We show that this entity dominates the DNS amplification ecosystem by carrying out 59% of the attacks, and provide an in-depth analysis of its behavior over time. Finally, our results reveal that operators of various .gov names do not adhere to DNSSEC key rollover best practices, which exacerbates amplification potential. We can verifiably connect this operational behavior to misuses and attacker decision-making.
△ Less
Submitted 6 October, 2021; v1 submitted 2 September, 2021;
originally announced September 2021.
-
Reason Against the Machine: Future Directions for Mass Online Deliberation
Authors:
Ruth Shortall,
Anatol Itten,
Michiel van der Meer,
Pradeep K. Murukannaiah,
Catholijn M. Jonker
Abstract:
Designers of online deliberative platforms aim to counter the degrading quality of online debates. Support technologies such as machine learning and natural language processing open avenues for widening the circle of people involved in deliberation, moving from small groups to "crowd" scale. Numerous design features of large-scale online discussion systems allow larger numbers of people to discuss…
▽ More
Designers of online deliberative platforms aim to counter the degrading quality of online debates. Support technologies such as machine learning and natural language processing open avenues for widening the circle of people involved in deliberation, moving from small groups to "crowd" scale. Numerous design features of large-scale online discussion systems allow larger numbers of people to discuss shared problems, enhance critical thinking, and formulate solutions. We review the transdisciplinary literature on the design of digital mass deliberation platforms and examine the commonly featured design aspects (e.g., argumentation support, automated facilitation, and gamification) that attempt to facilitate scaling up. We find that the literature is largely focused on develo** technical fixes for scaling up deliberation, but may neglect the more nuanced requirements of high quality deliberation. Current design research is carried out with a small, atypical segment of the world's population, and much research is still needed on how to facilitate and accommodate different genders or cultures in deliberation, how to deal with the implications of pre-existing social inequalities, how to build motivation and self-efficacy in certain groups, and how to deal with differences in cognitive abilities and cultural or linguistic differences. Few studies bridge disciplines between deliberative theory, design and engineering. As a result, scaling up deliberation will likely advance in separate systemic siloes. We make design and process recommendations to correct this course and suggest avenues for future research
△ Less
Submitted 31 January, 2022; v1 submitted 27 July, 2021;
originally announced July 2021.
-
A Data-Driven Method for Recognizing Automated Negotiation Strategies
Authors:
Ming Li,
Pradeep K. Murukannaiah,
Catholijn M. Jonker
Abstract:
Understanding an opponent agent helps in negotiating with it. Existing works on understanding opponents focus on preference modeling (or estimating the opponent's utility function). An important but largely unexplored direction is recognizing an opponent's negotiation strategy, which captures the opponent's tactics, e.g., to be tough at the beginning but to concede toward the deadline. Recognizing…
▽ More
Understanding an opponent agent helps in negotiating with it. Existing works on understanding opponents focus on preference modeling (or estimating the opponent's utility function). An important but largely unexplored direction is recognizing an opponent's negotiation strategy, which captures the opponent's tactics, e.g., to be tough at the beginning but to concede toward the deadline. Recognizing complex, state-of-the-art, negotiation strategies is extremely challenging, and simple heuristics may not be adequate for this purpose. We propose a novel data-driven approach for recognizing an opponent's s negotiation strategy. Our approach includes a data generation method for an agent to generate domain-independent sequences by negotiating with a variety of opponents across domains, a feature engineering method for representing negotiation data as time series with time-step features and overall features, and a hybrid (recurrent neural network-based) deep learning method for recognizing an opponent's strategy from the time series of bids. We perform extensive experiments, spanning four problem scenarios, to demonstrate the effectiveness of our approach.
△ Less
Submitted 7 October, 2021; v1 submitted 3 July, 2021;
originally announced July 2021.
-
Synthesising Reinforcement Learning Policies through Set-Valued Inductive Rule Learning
Authors:
Youri Coppens,
Denis Steckelmacher,
Catholijn M. Jonker,
Ann Nowé
Abstract:
Today's advanced Reinforcement Learning algorithms produce black-box policies, that are often difficult to interpret and trust for a person. We introduce a policy distilling algorithm, building on the CN2 rule mining algorithm, that distills the policy into a rule-based decision system. At the core of our approach is the fact that an RL process does not just learn a policy, a map** from states t…
▽ More
Today's advanced Reinforcement Learning algorithms produce black-box policies, that are often difficult to interpret and trust for a person. We introduce a policy distilling algorithm, building on the CN2 rule mining algorithm, that distills the policy into a rule-based decision system. At the core of our approach is the fact that an RL process does not just learn a policy, a map** from states to actions, but also produces extra meta-information, such as action values indicating the quality of alternative actions. This meta-information can indicate whether more than one action is near-optimal for a certain state. We extend CN2 to make it able to leverage knowledge about equally-good actions to distill the policy into fewer rules, increasing its interpretability by a person. Then, to ensure that the rules explain a valid, non-degenerate policy, we introduce a refinement algorithm that fine-tunes the rules to obtain good performance when executed in the environment. We demonstrate the applicability of our algorithm on the Mario AI benchmark, a complex task that requires modern reinforcement learning algorithms including neural networks. The explanations we produce capture the learned policy in only a few rules, that allow a person to understand what the black-box agent learned. Source code: https://gitlab.ai.vub.ac.be/yocoppen/svcn2
△ Less
Submitted 10 June, 2021;
originally announced June 2021.
-
More Similar Values, More Trust? -- the Effect of Value Similarity on Trust in Human-Agent Interaction
Authors:
Siddharth Mehrotra,
Catholijn M. Jonker,
Myrthe L. Tielman
Abstract:
As AI systems are increasingly involved in decision making, it also becomes important that they elicit appropriate levels of trust from their users. To achieve this, it is first important to understand which factors influence trust in AI. We identify that a research gap exists regarding the role of personal values in trust in AI. Therefore, this paper studies how human and agent Value Similarity (…
▽ More
As AI systems are increasingly involved in decision making, it also becomes important that they elicit appropriate levels of trust from their users. To achieve this, it is first important to understand which factors influence trust in AI. We identify that a research gap exists regarding the role of personal values in trust in AI. Therefore, this paper studies how human and agent Value Similarity (VS) influences a human's trust in that agent. To explore this, 89 participants teamed up with five different agents, which were designed with varying levels of value similarity to that of the participants. In a within-subjects, scenario-based experiment, agents gave suggestions on what to do when entering the building to save a hostage. We analyzed the agent's scores on subjective value similarity, trust and qualitative data from open-ended questions. Our results show that agents rated as having more similar values also scored higher on trust, indicating a positive effect between the two. With this result, we add to the existing understanding of human-agent trust by providing insight into the role of value-similarity.
△ Less
Submitted 19 May, 2021;
originally announced May 2021.
-
Modelling Human Routines: Conceptualising Social Practice Theory for Agent-Based Simulation
Authors:
Rijk Mercuur,
Virginia Dignum,
Catholijn M. Jonker
Abstract:
Our routines play an important role in a wide range of social challenges such as climate change, disease outbreaks and coordinating staff and patients in a hospital. To use agent-based simulations (ABS) to understand the role of routines in social challenges we need an agent framework that integrates routines. This paper provides the domain-independent Social Practice Agent (SoPrA) framework that…
▽ More
Our routines play an important role in a wide range of social challenges such as climate change, disease outbreaks and coordinating staff and patients in a hospital. To use agent-based simulations (ABS) to understand the role of routines in social challenges we need an agent framework that integrates routines. This paper provides the domain-independent Social Practice Agent (SoPrA) framework that satisfies requirements from the literature to simulate our routines. By choosing the appropriate concepts from the literature on agent theory, social psychology and social practice theory we ensure SoPrA correctly depicts current evidence on routines. By creating a consistent, modular and parsimonious framework suitable for multiple domains we enhance the usability of SoPrA. SoPrA provides ABS researchers with a conceptual, formal and computational framework to simulate routines and gain new insights into social systems.
△ Less
Submitted 22 December, 2020;
originally announced December 2020.
-
Sensitivity of the SHiP experiment to dark photons decaying to a pair of charged particles
Authors:
SHiP Collaboration,
C. Ahdida,
A. Akmete,
R. Albanese,
A. Alexandrov,
A. Anokhina,
S. Aoki,
G. Arduini,
E. Atkin,
N. Azorskiy,
J. J. Back,
A. Bagulya,
F. Baaltasar Dos Santos,
A. Baranov,
F. Bardou,
G. J. Barker,
M. Battistin,
J. Bauche,
A. Bay,
V. Bayliss,
G. Bencivenni,
A. Y. Berdnikov,
Y. A. Berdnikov,
M. Bertani,
C. Betancourt
, et al. (309 additional authors not shown)
Abstract:
Dark photons are hypothetical massive vector particles that could mix with ordinary photons. The simplest theoretical model is fully characterised by only two parameters: the mass of the dark photon m$_{γ^{\mathrm{D}}}$ and its mixing parameter with the photon, $\varepsilon$. The sensitivity of the SHiP detector is reviewed for dark photons in the mass range between 0.002 and 10 GeV. Different pro…
▽ More
Dark photons are hypothetical massive vector particles that could mix with ordinary photons. The simplest theoretical model is fully characterised by only two parameters: the mass of the dark photon m$_{γ^{\mathrm{D}}}$ and its mixing parameter with the photon, $\varepsilon$. The sensitivity of the SHiP detector is reviewed for dark photons in the mass range between 0.002 and 10 GeV. Different production mechanisms are simulated, with the dark photons decaying to pairs of visible fermions, including both leptons and quarks. Exclusion contours are presented and compared with those of past experiments. The SHiP detector is expected to have a unique sensitivity for m$_{γ^{\mathrm{D}}}$ ranging between 0.8 and 3.3$^{+0.2}_{-0.5}$ GeV, and $\varepsilon^2$ ranging between $10^{-11}$ and $10^{-17}$.
△ Less
Submitted 1 March, 2021; v1 submitted 10 November, 2020;
originally announced November 2020.
-
Model-based Reinforcement Learning: A Survey
Authors:
Thomas M. Moerland,
Joost Broekens,
Aske Plaat,
Catholijn M. Jonker
Abstract:
Sequential decision making, commonly formalized as Markov Decision Process (MDP) optimization, is a important challenge in artificial intelligence. Two key approaches to this problem are reinforcement learning (RL) and planning. This paper presents a survey of the integration of both fields, better known as model-based reinforcement learning. Model-based RL has two main steps. First, we systematic…
▽ More
Sequential decision making, commonly formalized as Markov Decision Process (MDP) optimization, is a important challenge in artificial intelligence. Two key approaches to this problem are reinforcement learning (RL) and planning. This paper presents a survey of the integration of both fields, better known as model-based reinforcement learning. Model-based RL has two main steps. First, we systematically cover approaches to dynamics model learning, including challenges like dealing with stochasticity, uncertainty, partial observability, and temporal abstraction. Second, we present a systematic categorization of planning-learning integration, including aspects like: where to start planning, what budgets to allocate to planning and real data collection, how to plan, and how to integrate planning in the learning and acting loop. After these two sections, we also discuss implicit model-based RL as an end-to-end alternative for model learning and planning, and we cover the potential benefits of model-based RL. Along the way, the survey also draws connections to several related RL fields, like hierarchical RL and transfer learning. Altogether, the survey presents a broad conceptual overview of the combination of planning and learning for MDP optimization.
△ Less
Submitted 31 March, 2022; v1 submitted 30 June, 2020;
originally announced June 2020.
-
A Unifying Framework for Reinforcement Learning and Planning
Authors:
Thomas M. Moerland,
Joost Broekens,
Aske Plaat,
Catholijn M. Jonker
Abstract:
Sequential decision making, commonly formalized as optimization of a Markov Decision Process, is a key challenge in artificial intelligence. Two successful approaches to MDP optimization are reinforcement learning and planning, which both largely have their own research communities. However, if both research fields solve the same problem, then we might be able to disentangle the common factors in…
▽ More
Sequential decision making, commonly formalized as optimization of a Markov Decision Process, is a key challenge in artificial intelligence. Two successful approaches to MDP optimization are reinforcement learning and planning, which both largely have their own research communities. However, if both research fields solve the same problem, then we might be able to disentangle the common factors in their solution approaches. Therefore, this paper presents a unifying algorithmic framework for reinforcement learning and planning (FRAP), which identifies underlying dimensions on which MDP planning and learning algorithms have to decide. At the end of the paper, we compare a variety of well-known planning, model-free and model-based RL algorithms along these dimensions. Altogether, the framework may help provide deeper insight in the algorithmic design space of planning and reinforcement learning.
△ Less
Submitted 31 March, 2022; v1 submitted 26 June, 2020;
originally announced June 2020.
-
The Second Type of Uncertainty in Monte Carlo Tree Search
Authors:
Thomas M Moerland,
Joost Broekens,
Aske Plaat,
Catholijn M Jonker
Abstract:
Monte Carlo Tree Search (MCTS) efficiently balances exploration and exploitation in tree search based on count-derived uncertainty. However, these local visit counts ignore a second type of uncertainty induced by the size of the subtree below an action. We first show how, due to the lack of this second uncertainty type, MCTS may completely fail in well-known sparse exploration problems, known from…
▽ More
Monte Carlo Tree Search (MCTS) efficiently balances exploration and exploitation in tree search based on count-derived uncertainty. However, these local visit counts ignore a second type of uncertainty induced by the size of the subtree below an action. We first show how, due to the lack of this second uncertainty type, MCTS may completely fail in well-known sparse exploration problems, known from the reinforcement learning community. We then introduce a new algorithm, which estimates the size of the subtree below an action, and leverages this information in the UCB formula to better direct exploration. Subsequently, we generalize these ideas by showing that loops, i.e., the repeated occurrence of (approximately) the same state in the same trace, are actually a special case of subtree depth variation. Testing on a variety of tasks shows that our algorithms increase sample efficiency, especially when the planning budget per timestep is small.
△ Less
Submitted 19 May, 2020;
originally announced May 2020.
-
Think Too Fast Nor Too Slow: The Computational Trade-off Between Planning And Reinforcement Learning
Authors:
Thomas M. Moerland,
Anna Deichler,
Simone Baldi,
Joost Broekens,
Catholijn M. Jonker
Abstract:
Planning and reinforcement learning are two key approaches to sequential decision making. Multi-step approximate real-time dynamic programming, a recently successful algorithm class of which AlphaZero [Silver et al., 2018] is an example, combines both by nesting planning within a learning loop. However, the combination of planning and learning introduces a new question: how should we balance time…
▽ More
Planning and reinforcement learning are two key approaches to sequential decision making. Multi-step approximate real-time dynamic programming, a recently successful algorithm class of which AlphaZero [Silver et al., 2018] is an example, combines both by nesting planning within a learning loop. However, the combination of planning and learning introduces a new question: how should we balance time spend on planning, learning and acting? The importance of this trade-off has not been explicitly studied before. We show that it is actually of key importance, with computational results indicating that we should neither plan too long nor too short. Conceptually, we identify a new spectrum of planning-learning algorithms which ranges from exhaustive search (long planning) to model-free RL (no planning), with optimal performance achieved midway.
△ Less
Submitted 15 May, 2020;
originally announced May 2020.
-
Automated Configuration of Negotiation Strategies
Authors:
Bram M. Renting,
Holger H. Hoos,
Catholijn M. Jonker
Abstract:
Bidding and acceptance strategies have a substantial impact on the outcome of negotiations in scenarios with linear additive and nonlinear utility functions. Over the years, it has become clear that there is no single best strategy for all negotiation settings, yet many fixed strategies are still being developed. We envision a shift in the strategy design question from: What is a good strategy?, t…
▽ More
Bidding and acceptance strategies have a substantial impact on the outcome of negotiations in scenarios with linear additive and nonlinear utility functions. Over the years, it has become clear that there is no single best strategy for all negotiation settings, yet many fixed strategies are still being developed. We envision a shift in the strategy design question from: What is a good strategy?, towards: What could be a good strategy? For this purpose, we developed a method leveraging automated algorithm configuration to find the best strategies for a specific set of negotiation settings. By empowering automated negotiating agents using automated algorithm configuration, we obtain a flexible negotiation agent that can be configured automatically for a rich space of opponents and negotiation scenarios.
To critically assess our approach, the agent was tested in an ANAC-like bilateral automated negotiation tournament setting against past competitors. We show that our automatically configured agent outperforms all other agents, with a 5.1% increase in negotiation payoff compared to the next-best agent. We note that without our agent in the tournament, the top-ranked agent wins by a margin of only 0.01%.
△ Less
Submitted 31 March, 2020;
originally announced April 2020.
-
SND@LHC
Authors:
SHiP Collaboration,
C. Ahdida,
A. Akmete,
R. Albanese,
A. Alexandrov,
M. Andreini,
A. Anokhina,
S. Aoki,
G. Arduini,
E. Atkin,
N. Azorskiy,
J. J. Back,
A. Bagulya,
F. Baaltasar Dos Santos,
A. Baranov,
F. Bardou,
G. J. Barker,
M. Battistin,
J. Bauche,
A. Bay,
V. Bayliss,
G. Bencivenni,
A. Y. Berdnikov,
Y. A. Berdnikov,
M. Bertani
, et al. (319 additional authors not shown)
Abstract:
We propose to build and operate a detector that, for the first time, will measure the process $pp\toνX$ at the LHC and search for feebly interacting particles (FIPs) in an unexplored domain. The TI18 tunnel has been identified as a suitable site to perform these measurements due to very low machine-induced background. The detector will be off-axis with respect to the ATLAS interaction point (IP1)…
▽ More
We propose to build and operate a detector that, for the first time, will measure the process $pp\toνX$ at the LHC and search for feebly interacting particles (FIPs) in an unexplored domain. The TI18 tunnel has been identified as a suitable site to perform these measurements due to very low machine-induced background. The detector will be off-axis with respect to the ATLAS interaction point (IP1) and, given the pseudo-rapidity range accessible, the corresponding neutrinos will mostly come from charm decays: the proposed experiment will thus make the first test of the heavy flavour production in a pseudo-rapidity range that is not accessible by the current LHC detectors. In order to efficiently reconstruct neutrino interactions and identify their flavour, the detector will combine in the target region nuclear emulsion technology with scintillating fibre tracking layers and it will adopt a muon identification system based on scintillating bars that will also play the role of a hadronic calorimeter. The time of flight measurement will be achieved thanks to a dedicated timing detector. The detector will be a small-scale prototype of the scattering and neutrino detector (SND) of the SHiP experiment: the operation of this detector will provide an important test of the neutrino reconstruction in a high occupancy environment.
△ Less
Submitted 20 February, 2020;
originally announced February 2020.
-
A powerful MAF-neutral allele-based test for case-control association studies
Authors:
Marianne A. Jonker,
Jakub Pecanka
Abstract:
In a case-control study aimed at locating autosomal disease variants for a disease of interest, association between markers and the disease status is often tested by comparing the marker minor allele frequencies (MAFs) between cases and controls. For most common allele-based tests the statistical power is highly dependent on the actual values of these MAFs, where associated markers with low MAFs h…
▽ More
In a case-control study aimed at locating autosomal disease variants for a disease of interest, association between markers and the disease status is often tested by comparing the marker minor allele frequencies (MAFs) between cases and controls. For most common allele-based tests the statistical power is highly dependent on the actual values of these MAFs, where associated markers with low MAFs have less power to be detected compared to associated markers with high MAFs. Therefore, the popular strategy of selecting markers for follow-up studies based primarily on their p-values is likely to preferentially select markers with high MAFs. We propose a new test which does not favor markers with high MAFs and improves the power for markers with low to moderate MAFs without sacrificing performance for markers with high MAFs and is therefore superior to most existing tests in this regard. An explicit formula for the asymptotic power function of the proposed test is derived theoretically, which allows for fast and easy computation of the corresponding p-values. The performance of the proposed test is compared with several existing tests both in the asymptotic and the finite sample size settings.
△ Less
Submitted 12 February, 2020;
originally announced February 2020.
-
The Compact Linear Collider (CLIC) - 2018 Summary Report
Authors:
The CLIC,
CLICdp collaborations,
:,
T. K. Charles,
P. J. Giansiracusa,
T. G. Lucas,
R. P. Rassool,
M. Volpi,
C. Balazs,
K. Afanaciev,
V. Makarenko,
A. Patapenka,
I. Zhuk,
C. Collette,
M. J. Boland,
A. C. Abusleme Hoffman,
M. A. Diaz,
F. Garay,
Y. Chi,
X. He,
G. Pei,
S. Pei,
G. Shu,
X. Wang,
J. Zhang
, et al. (671 additional authors not shown)
Abstract:
The Compact Linear Collider (CLIC) is a TeV-scale high-luminosity linear $e^+e^-$ collider under development at CERN. Following the CLIC conceptual design published in 2012, this report provides an overview of the CLIC project, its current status, and future developments. It presents the CLIC physics potential and reports on design, technology, and implementation aspects of the accelerator and the…
▽ More
The Compact Linear Collider (CLIC) is a TeV-scale high-luminosity linear $e^+e^-$ collider under development at CERN. Following the CLIC conceptual design published in 2012, this report provides an overview of the CLIC project, its current status, and future developments. It presents the CLIC physics potential and reports on design, technology, and implementation aspects of the accelerator and the detector. CLIC is foreseen to be built and operated in stages, at centre-of-mass energies of 380 GeV, 1.5 TeV and 3 TeV, respectively. CLIC uses a two-beam acceleration scheme, in which 12 GHz accelerating structures are powered via a high-current drive beam. For the first stage, an alternative with X-band klystron powering is also considered. CLIC accelerator optimisation, technical developments and system tests have resulted in an increased energy efficiency (power around 170 MW) for the 380 GeV stage, together with a reduced cost estimate at the level of 6 billion CHF. The detector concept has been refined using improved software tools. Significant progress has been made on detector technology developments for the tracking and calorimetry systems. A wide range of CLIC physics studies has been conducted, both through full detector simulations and parametric studies, together providing a broad overview of the CLIC physics potential. Each of the three energy stages adds cornerstones of the full CLIC physics programme, such as Higgs width and couplings, top-quark properties, Higgs self-coupling, direct searches, and many precision electroweak measurements. The interpretation of the combined results gives crucial and accurate insight into new physics, largely complementary to LHC and HL-LHC. The construction of the first CLIC energy stage could start by 2026. First beams would be available by 2035, marking the beginning of a broad CLIC physics programme spanning 25-30 years.
△ Less
Submitted 6 May, 2019; v1 submitted 14 December, 2018;
originally announced December 2018.
-
Towards Agent-based Models of Rumours in Organizations: A Social Practice Theory Approach
Authors:
Amir Ebrahimi Fard,
Rijk Mercuur,
Virginia Dignum,
Catholijn M. Jonker,
Bartel van de Walle
Abstract:
Rumour is a collective emergent phenomenon with a potential for provoking a crisis. Modelling approaches have been deployed since five decades ago; however, the focus was mostly on epidemic behaviour of the rumours which does not take into account the differences of the agents. We use social practice theory to model agent decision making in organizational rumourmongering. Such an approach provides…
▽ More
Rumour is a collective emergent phenomenon with a potential for provoking a crisis. Modelling approaches have been deployed since five decades ago; however, the focus was mostly on epidemic behaviour of the rumours which does not take into account the differences of the agents. We use social practice theory to model agent decision making in organizational rumourmongering. Such an approach provides us with an opportunity to model rumourmongering agents with a layer of cognitive realism and study the impacts of various intervention strategies for prevention and control of rumours in organizations.
△ Less
Submitted 10 April, 2019; v1 submitted 3 December, 2018;
originally announced December 2018.