-
Using AI Uncertainty Quantification to Improve Human Decision-Making
Authors:
Laura R. Marusich,
Jonathan Z. Bakdash,
Yan Zhou,
Murat Kantarcioglu
Abstract:
AI Uncertainty Quantification (UQ) has the potential to improve human decision-making beyond AI predictions alone by providing additional probabilistic information to users. The majority of past research on AI and human decision-making has concentrated on model explainability and interpretability, with little focus on understanding the potential impact of UQ on human decision-making. We evaluated…
▽ More
AI Uncertainty Quantification (UQ) has the potential to improve human decision-making beyond AI predictions alone by providing additional probabilistic information to users. The majority of past research on AI and human decision-making has concentrated on model explainability and interpretability, with little focus on understanding the potential impact of UQ on human decision-making. We evaluated the impact on human decision-making for instance-level UQ, calibrated using a strict scoring rule, in two online behavioral experiments. In the first experiment, our results showed that UQ was beneficial for decision-making performance compared to only AI predictions. In the second experiment, we found UQ had generalizable benefits for decision-making across a variety of representations for probabilistic information. These results indicate that implementing high quality, instance-level UQ for AI may improve decision-making with real systems compared to AI predictions alone.
△ Less
Submitted 6 February, 2024; v1 submitted 19 September, 2023;
originally announced September 2023.
-
Augmented Reality and Mixed Reality Measurement Under Different Environments: A Survey on Head-Mounted Devices
Authors:
Hung-Jui Guo,
Jonathan Z. Bakdash,
Laura R. Marusich,
Balakrishnan Prabhakaran
Abstract:
Augmented Reality (AR) and Mixed Reality (MR) have been two of the most explosive research topics in the last few years. Head-Mounted Devices (HMDs) are essential intermediums for using AR and MR technology, playing an important role in the research progress in these two areas. Behavioral research with users is one way of evaluating the technical progress and effectiveness of HMDs. In addition, AR…
▽ More
Augmented Reality (AR) and Mixed Reality (MR) have been two of the most explosive research topics in the last few years. Head-Mounted Devices (HMDs) are essential intermediums for using AR and MR technology, playing an important role in the research progress in these two areas. Behavioral research with users is one way of evaluating the technical progress and effectiveness of HMDs. In addition, AR and MR technology is dependent upon virtual interactions with the real environment. Thus, conditions in real environments can be a significant factor for AR and MR measurements with users. In this paper, we survey 87 environmental-related HMD papers with measurements from users, spanning over 32 years. We provide a thorough review of AR- and MR-related user experiments with HMDs under different environmental factors. Then, we summarize trends in this literature over time using a new classification method with four environmental factors, the presence or absence of user feedback in behavioral experiments, and ten main categories to subdivide these papers (e.g., domain and method of user assessment). We also categorize characteristics of the behavioral experiments, showing similarities and differences among papers.
△ Less
Submitted 28 October, 2022;
originally announced October 2022.
-
Dynamic X-Ray Vision in Mixed Reality
Authors:
Hung-Jui Guo,
Jonathan Z. Bakdash,
Laura R. Marusich,
Balakrishnan Prabhakaran
Abstract:
X-ray vision, a technique that allows users to see through walls and other obstacles, is a popular technique for Augmented Reality (AR) and Mixed Reality (MR). In this paper, we demonstrate a dynamic X-ray vision window that is rendered in real-time based on the user's current position and changes with movement in the physical environment. Moreover, the location and transparency of the window are…
▽ More
X-ray vision, a technique that allows users to see through walls and other obstacles, is a popular technique for Augmented Reality (AR) and Mixed Reality (MR). In this paper, we demonstrate a dynamic X-ray vision window that is rendered in real-time based on the user's current position and changes with movement in the physical environment. Moreover, the location and transparency of the window are also dynamically rendered based on the user's eye gaze. We build this X-ray vision window for a current state-of-the-art MR Head-Mounted Device (HMD) -- HoloLens 2 by integrating several different features: scene understanding, eye tracking, and clip** primitive.
△ Less
Submitted 14 September, 2022;
originally announced September 2022.
-
Does Explainable Artificial Intelligence Improve Human Decision-Making?
Authors:
Yasmeen Alufaisan,
Laura R. Marusich,
Jonathan Z. Bakdash,
Yan Zhou,
Murat Kantarcioglu
Abstract:
Explainable AI provides insight into the "why" for model predictions, offering potential for users to better understand and trust a model, and to recognize and correct AI predictions that are incorrect. Prior research on human and explainable AI interactions has focused on measures such as interpretability, trust, and usability of the explanation. Whether explainable AI can improve actual human de…
▽ More
Explainable AI provides insight into the "why" for model predictions, offering potential for users to better understand and trust a model, and to recognize and correct AI predictions that are incorrect. Prior research on human and explainable AI interactions has focused on measures such as interpretability, trust, and usability of the explanation. Whether explainable AI can improve actual human decision-making and the ability to identify the problems with the underlying model are open questions. Using real datasets, we compare and evaluate objective human decision accuracy without AI (control), with an AI prediction (no explanation), and AI prediction with explanation. We find providing any kind of AI prediction tends to improve user decision accuracy, but no conclusive evidence that explainable AI has a meaningful impact. Moreover, we observed the strongest predictor for human decision accuracy was AI accuracy and that users were somewhat able to detect when the AI was correct versus incorrect, but this was not significantly affected by including an explanation. Our results indicate that, at least in some situations, the "why" information provided in explainable AI may not enhance user decision-making, and further research may be needed to understand how to integrate explainable AI into real systems.
△ Less
Submitted 19 June, 2020;
originally announced June 2020.
-
Malware in the Future? Forecasting of Analyst Detection of Cyber Events
Authors:
Jonathan Z. Bakdash,
Steve Hutchinson,
Erin G. Zaroukian,
Laura R. Marusich,
Saravanan Thirumuruganathan,
Charmaine Sample,
Blaine Hoffman,
Gautam Das
Abstract:
There have been extensive efforts in government, academia, and industry to anticipate, forecast, and mitigate cyber attacks. A common approach is time-series forecasting of cyber attacks based on data from network telescopes, honeypots, and automated intrusion detection/prevention systems. This research has uncovered key insights such as systematicity in cyber attacks. Here, we propose an alternat…
▽ More
There have been extensive efforts in government, academia, and industry to anticipate, forecast, and mitigate cyber attacks. A common approach is time-series forecasting of cyber attacks based on data from network telescopes, honeypots, and automated intrusion detection/prevention systems. This research has uncovered key insights such as systematicity in cyber attacks. Here, we propose an alternate perspective of this problem by performing forecasting of attacks that are analyst-detected and -verified occurrences of malware. We call these instances of malware cyber event data. Specifically, our dataset was analyst-detected incidents from a large operational Computer Security Service Provider (CSSP) for the U.S. Department of Defense, which rarely relies only on automated systems. Our data set consists of weekly counts of cyber events over approximately seven years. Since all cyber events were validated by analysts, our dataset is unlikely to have false positives which are often endemic in other sources of data. Further, the higher-quality data could be used for a number for resource allocation, estimation of security resources, and the development of effective risk-management strategies. We used a Bayesian State Space Model for forecasting and found that events one week ahead could be predicted. To quantify bursts, we used a Markov model. Our findings of systematicity in analyst-detected cyber attacks are consistent with previous work using other sources. The advanced information provided by a forecast may help with threat awareness by providing a probable value and range for future cyber events one week ahead. Other potential applications for cyber event forecasting include proactive allocation of resources and capabilities for cyber defense (e.g., analyst staffing and sensor configuration) in CSSPs. Enhanced threat awareness may improve cybersecurity.
△ Less
Submitted 8 June, 2018; v1 submitted 11 July, 2017;
originally announced July 2017.
