-
GPT-4's assessment of its performance in a USMLE-based case study
Authors:
Uttam Dhakal,
Aniket Kumar Singh,
Suman Devkota,
Yogesh Sapkota,
Bishal Lamichhane,
Suprinsa Paudyal,
Chandra Dhakal
Abstract:
This study investigates GPT-4's assessment of its performance in healthcare applications. A simple prompting technique was used to prompt the LLM with questions taken from the United States Medical Licensing Examination (USMLE) questionnaire and it was tasked to evaluate its confidence score before posing the question and after asking the question. The questionnaire was categorized into two groups…
▽ More
This study investigates GPT-4's assessment of its performance in healthcare applications. A simple prompting technique was used to prompt the LLM with questions taken from the United States Medical Licensing Examination (USMLE) questionnaire and it was tasked to evaluate its confidence score before posing the question and after asking the question. The questionnaire was categorized into two groups-questions with feedback (WF) and questions with no feedback(NF) post-question. The model was asked to provide absolute and relative confidence scores before and after each question. The experimental findings were analyzed using statistical tools to study the variability of confidence in WF and NF groups. Additionally, a sequential analysis was conducted to observe the performance variation for the WF and NF groups. Results indicate that feedback influences relative confidence but doesn't consistently increase or decrease it. Understanding the performance of LLM is paramount in exploring its utility in sensitive areas like healthcare. This study contributes to the ongoing discourse on the reliability of AI, particularly of LLMs like GPT-4, within healthcare, offering insights into how feedback mechanisms might be optimized to enhance AI-assisted medical education and decision support.
△ Less
Submitted 26 March, 2024; v1 submitted 14 February, 2024;
originally announced February 2024.
-
Efficient Neural Representation of Volumetric Data using Coordinate-Based Networks
Authors:
Sudarshan Devkota,
Sumanta Pattanaik
Abstract:
In this paper, we propose an efficient approach for the compression and representation of volumetric data utilizing coordinate-based networks and multi-resolution hash encoding. Efficient compression of volumetric data is crucial for various applications, such as medical imaging and scientific simulations. Our approach enables effective compression by learning a map** between spatial coordinates…
▽ More
In this paper, we propose an efficient approach for the compression and representation of volumetric data utilizing coordinate-based networks and multi-resolution hash encoding. Efficient compression of volumetric data is crucial for various applications, such as medical imaging and scientific simulations. Our approach enables effective compression by learning a map** between spatial coordinates and intensity values. We compare different encoding schemes and demonstrate the superiority of multi-resolution hash encoding in terms of compression quality and training efficiency. Furthermore, we leverage optimization-based meta-learning, specifically using the Reptile algorithm, to learn weight initialization for neural representations tailored to volumetric data, enabling faster convergence during optimization. Additionally, we compare our approach with state-of-the-art methods to showcase improved image quality and compression ratios. These findings highlight the potential of coordinate-based networks and multi-resolution hash encoding for an efficient and accurate representation of volumetric data, paving the way for advancements in large-scale data visualization and other applications.
△ Less
Submitted 16 January, 2024;
originally announced January 2024.
-
The Confidence-Competence Gap in Large Language Models: A Cognitive Study
Authors:
Aniket Kumar Singh,
Suman Devkota,
Bishal Lamichhane,
Uttam Dhakal,
Chandra Dhakal
Abstract:
Large Language Models (LLMs) have acquired ubiquitous attention for their performances across diverse domains. Our study here searches through LLMs' cognitive abilities and confidence dynamics. We dive deep into understanding the alignment between their self-assessed confidence and actual performance. We exploit these models with diverse sets of questionnaires and real-world scenarios and extract…
▽ More
Large Language Models (LLMs) have acquired ubiquitous attention for their performances across diverse domains. Our study here searches through LLMs' cognitive abilities and confidence dynamics. We dive deep into understanding the alignment between their self-assessed confidence and actual performance. We exploit these models with diverse sets of questionnaires and real-world scenarios and extract how LLMs exhibit confidence in their responses. Our findings reveal intriguing instances where models demonstrate high confidence even when they answer incorrectly. This is reminiscent of the Dunning-Kruger effect observed in human psychology. In contrast, there are cases where models exhibit low confidence with correct answers revealing potential underestimation biases. Our results underscore the need for a deeper understanding of their cognitive processes. By examining the nuances of LLMs' self-assessment mechanism, this investigation provides noteworthy revelations that serve to advance the functionalities and broaden the potential applications of these formidable language models.
△ Less
Submitted 27 September, 2023;
originally announced September 2023.
-
Reference Based Color Transfer for Medical Volume Rendering
Authors:
Sudarshan Devkota,
Summanta Pattanaik
Abstract:
The benefits of medical imaging are enormous. Medical images provide considerable amounts of anatomical information and this facilitates medical practitioners in performing effective disease diagnosis and deciding upon the best course of medical treatment. A transition from traditional monochromatic medical images like CT scans, X-Rays or MRI images to a colored 3D representation of the anatomical…
▽ More
The benefits of medical imaging are enormous. Medical images provide considerable amounts of anatomical information and this facilitates medical practitioners in performing effective disease diagnosis and deciding upon the best course of medical treatment. A transition from traditional monochromatic medical images like CT scans, X-Rays or MRI images to a colored 3D representation of the anatomical structure further enhances the capabilities of medical professionals in extracting valuable medical information. The proposed framework in our research starts with performing color transfer by finding deep semantic correspondence between two medical images: a colored reference image, and a monochromatic CT scan or an MRI image. We extend this idea of reference-based colorization technique to perform colored volume rendering from a stack of grayscale medical images. Furthermore, we also propose to use an effective reference image recommendation system to aid in the selection of good reference images. With our approach, we successfully perform colored medical volume visualization and essentially eliminate the painstaking process of user interaction with a transfer function to obtain color and opacity parameters for volume rendering.
△ Less
Submitted 14 October, 2022;
originally announced October 2022.
-
Deep Learning based Super-Resolution for Medical Volume Visualization with Direct Volume Rendering
Authors:
Sudarshan Devkota,
Sumanta Pattanaik
Abstract:
Modern-day display systems demand high-quality rendering. However, rendering at higher resolution requires a large number of data samples and is computationally expensive. Recent advances in deep learning-based image and video super-resolution techniques motivate us to investigate such networks for high-fidelity upscaling of frames rendered at a lower resolution to a higher resolution. While our w…
▽ More
Modern-day display systems demand high-quality rendering. However, rendering at higher resolution requires a large number of data samples and is computationally expensive. Recent advances in deep learning-based image and video super-resolution techniques motivate us to investigate such networks for high-fidelity upscaling of frames rendered at a lower resolution to a higher resolution. While our work focuses on super-resolution of medical volume visualization performed with direct volume rendering, it is also applicable for volume visualization with other rendering techniques. We propose a learning-based technique where our proposed system uses color information along with other supplementary features gathered from our volume renderer to learn efficient upscaling of a low-resolution rendering to a higher-resolution space. Furthermore, to improve temporal stability, we also implement the temporal reprojection technique for accumulating history samples in volumetric rendering.
△ Less
Submitted 14 October, 2022;
originally announced October 2022.
-
Multicriteria Scalable Graph Drawing via Stochastic Gradient Descent, $(SGD)^2$
Authors:
Reyan Ahmed,
Felice De Luca,
Sabin Devkota,
Stephen Kobourov,
Mingwei Li
Abstract:
Readability criteria, such as distance or neighborhood preservation, are often used to optimize node-link representations of graphs to enable the comprehension of the underlying data. With few exceptions, graph drawing algorithms typically optimize one such criterion, usually at the expense of others. We propose a layout approach, Multicriteria Scalable Graph Drawing via Stochastic Gradient Descen…
▽ More
Readability criteria, such as distance or neighborhood preservation, are often used to optimize node-link representations of graphs to enable the comprehension of the underlying data. With few exceptions, graph drawing algorithms typically optimize one such criterion, usually at the expense of others. We propose a layout approach, Multicriteria Scalable Graph Drawing via Stochastic Gradient Descent, $(SGD)^2$, that can handle multiple readability criteria. $(SGD)^2$ can optimize any criterion that can be described by a differentiable function. Our approach is flexible and can be used to optimize several criteria that have already been considered earlier (e.g., obtaining ideal edge lengths, stress, neighborhood preservation) as well as other criteria which have not yet been explicitly optimized in such fashion (e.g., node resolution, angular resolution, aspect ratio). The approach is scalable and can handle large graphs. A variation of the underlying approach can also be used to optimize many desirable properties in planar graphs, while maintaining planarity. Finally, we provide quantitative and qualitative evidence of the effectiveness of $(SGD)^2$: we analyze the interactions between criteria, measure the quality of layouts generated from $(SGD)^2$ as well as the runtime behavior, and analyze the impact of sample sizes. The source code is available on github and we also provide an interactive demo for small graphs.
△ Less
Submitted 2 December, 2021;
originally announced December 2021.
-
Domain-Centered Support for Layout, Tasks, and Specification for Control Flow Graph Visualization
Authors:
Sabin Devkota,
Matthew P. LeGendre,
Adam Kunen,
Pascal Aschwanden,
Katherine E. Isaacs
Abstract:
Computing professionals in areas like compilers, performance analysis, and security often analyze and manipulate control flow graphs (CFGs) in their work. CFGs are directed networks that describe possible orderings of instructions in the execution of a program. Visualizing a CFG is a common activity in develo** or debugging computational approaches that use them. However, general graph drawing l…
▽ More
Computing professionals in areas like compilers, performance analysis, and security often analyze and manipulate control flow graphs (CFGs) in their work. CFGs are directed networks that describe possible orderings of instructions in the execution of a program. Visualizing a CFG is a common activity in develo** or debugging computational approaches that use them. However, general graph drawing layouts, including the hierarchical ones frequently applied to CFGs, do not capture CFG-specific structures or tasks and thus the resulting drawing may not match the needs of their audience, especially for more complicated programs. While several algorithms offer flexibility in specifying the layout, they often require expertise with graph drawing layouts and primitives that these potential users do not have. To bring domain-specific CFG drawing to this audience, we develop CFGConf, a library designed to match the abstraction level of CFG experts. CFGConf provides a JSON interface that produces drawings that can stand-alone or be integrated into multi-view visualization systems. We developed CFGConf through an interactive design process with experts while incorporating lessons learned from previous CFG visualization systems, a survey of CFG drawing conventions in computing systems conferences, and existing design principles for notations. We evaluate CFGConf in terms of expressiveness, usability, and notational efficiency through a user study and illustrative examples. CFG experts were able to use the library to produce the domain-aware layouts and appreciated the task-aware nature of the specification.
△ Less
Submitted 25 September, 2022; v1 submitted 6 August, 2021;
originally announced August 2021.
-
CcNav: Understanding Compiler Optimizations in Binary Code
Authors:
Sabin Devkota,
Pascal Aschwanden,
Adam Kunen,
Matthew Legendre,
Katherine E. Isaacs
Abstract:
Program developers spend significant time on optimizing and tuning programs. During this iterative process, they apply optimizations, analyze the resulting code, and modify the compilation until they are satisfied. Understanding what the compiler did with the code is crucial to this process but is very time-consuming and labor-intensive. Users need to navigate through thousands of lines of binary…
▽ More
Program developers spend significant time on optimizing and tuning programs. During this iterative process, they apply optimizations, analyze the resulting code, and modify the compilation until they are satisfied. Understanding what the compiler did with the code is crucial to this process but is very time-consuming and labor-intensive. Users need to navigate through thousands of lines of binary code and correlate it to source code concepts to understand the results of the compilation and to identify optimizations. We present a design study in collaboration with program developers and performance analysts. Our collaborators work with various artifacts related to the program such as binary code, source code, control flow graphs, and call graphs. Through interviews, feedback, and pair-analytics sessions, we analyzed their tasks and workflow. Based on this task analysis and through a human-centric design process, we designed a visual analytics system Compilation Navigator (CcNav) to aid exploration of the effects of compiler optimizations on the program. CcNav provides a streamlined workflow and a unified context that integrates disparate artifacts. CcNav supports consistent interactions across all the artifacts making it easy to correlate binary code with source code concepts. CcNav enables users to navigate and filter large binary code to identify and summarize optimizations such as inlining, vectorization, loop unrolling, and code hoisting. We evaluate CcNav through guided sessions and semi-structured interviews. We reflect on our design process, particularly the immersive elements, and on the transferability of design studies through our experience with a previous design study on program analysis.
△ Less
Submitted 7 November, 2020; v1 submitted 2 September, 2020;
originally announced September 2020.
-
Graph Drawing via Gradient Descent, $(GD)^2$
Authors:
Reyan Ahmed,
Felice De Luca,
Sabin Devkota,
Stephen Kobourov,
Mingwei Li
Abstract:
Readability criteria, such as distance or neighborhood preservation, are often used to optimize node-link representations of graphs to enable the comprehension of the underlying data. With few exceptions, graph drawing algorithms typically optimize one such criterion, usually at the expense of others. We propose a layout approach, Graph Drawing via Gradient Descent, $(GD)^2$, that can handle multi…
▽ More
Readability criteria, such as distance or neighborhood preservation, are often used to optimize node-link representations of graphs to enable the comprehension of the underlying data. With few exceptions, graph drawing algorithms typically optimize one such criterion, usually at the expense of others. We propose a layout approach, Graph Drawing via Gradient Descent, $(GD)^2$, that can handle multiple readability criteria. $(GD)^2$ can optimize any criterion that can be described by a smooth function. If the criterion cannot be captured by a smooth function, a non-smooth function for the criterion is combined with another smooth function, or auto-differentiation tools are used for the optimization. Our approach is flexible and can be used to optimize several criteria that have already been considered earlier (e.g., obtaining ideal edge lengths, stress, neighborhood preservation) as well as other criteria which have not yet been explicitly optimized in such fashion (e.g., vertex resolution, angular resolution, aspect ratio). We provide quantitative and qualitative evidence of the effectiveness of $(GD)^2$ with experimental data and a functional prototype: \url{http://hdc.cs.arizona.edu/~mwli/graph-drawing/}.
△ Less
Submitted 12 August, 2020;
originally announced August 2020.
-
Stress-Plus-X (SPX) Graph Layout
Authors:
Sabin Devkota,
Reyan Ahmed,
Felice De Luca,
Katherine E. Isaacs,
Stephen Kobourov
Abstract:
Stress, edge crossings, and crossing angles play an important role in the quality and readability of graph drawings. Most standard graph drawing algorithms optimize one of these criteria which may lead to layouts that are deficient in other criteria. We introduce an optimization framework, Stress-Plus-X (SPX), that simultaneously optimizes stress together with several other criteria: edge crossing…
▽ More
Stress, edge crossings, and crossing angles play an important role in the quality and readability of graph drawings. Most standard graph drawing algorithms optimize one of these criteria which may lead to layouts that are deficient in other criteria. We introduce an optimization framework, Stress-Plus-X (SPX), that simultaneously optimizes stress together with several other criteria: edge crossings, minimum crossing angle, and upwardness (for directed acyclic graphs). SPX achieves results that are close to the state-of-the-art algorithms that optimize these metrics individually. SPX is flexible and extensible and can optimize a subset or all of these criteria simultaneously. Our experimental analysis shows that our joint optimization approach is successful in drawing graphs with good performance across readability criteria.
△ Less
Submitted 23 August, 2019; v1 submitted 4 August, 2019;
originally announced August 2019.
-
L-Graphs and Monotone L-Graphs
Authors:
Abu Reyan Ahmed,
Felice De Luca,
Sabin Devkota,
Alon Efrat,
Md Iqbal Hossain,
Stephen Kobourov,
Jixian Li,
Sammi Abida Salma,
Eric Welch
Abstract:
In an $\mathsf{L}$-embedding of a graph, each vertex is represented by an $\mathsf{L}$-segment, and two segments intersect each other if and only if the corresponding vertices are adjacent in the graph. If the corner of each $\mathsf{L}$-segment in an $\mathsf{L}$-embedding lies on a straight line, we call it a monotone $\mathsf{L}$-embedding. In this paper we give a full characterization of monot…
▽ More
In an $\mathsf{L}$-embedding of a graph, each vertex is represented by an $\mathsf{L}$-segment, and two segments intersect each other if and only if the corresponding vertices are adjacent in the graph. If the corner of each $\mathsf{L}$-segment in an $\mathsf{L}$-embedding lies on a straight line, we call it a monotone $\mathsf{L}$-embedding. In this paper we give a full characterization of monotone $\mathsf{L}$-embeddings by introducing a new class of graphs which we call "non-jum**" graphs. We show that a graph admits a monotone $\mathsf{L}$-embedding if and only if the graph is a non-jum** graph. Further, we show that outerplanar graphs, convex bipartite graphs, interval graphs, 3-leaf power graphs, and complete graphs are subclasses of non-jum** graphs. Finally, we show that distance-hereditary graphs and $k$-leaf power graphs ($k\le 4$) admit $\mathsf{L}$-embeddings.
△ Less
Submitted 4 March, 2017;
originally announced March 2017.
