-
Evaluating Large Language Models for Public Health Classification and Extraction Tasks
Authors:
Joshua Harris,
Timothy Laurence,
Leo Loman,
Fan Grayson,
Toby Nonnenmacher,
Harry Long,
Loes WalsGriffith,
Amy Douglas,
Holly Fountain,
Stelios Georgiou,
Jo Hardstaff,
Kathryn Hopkins,
Y-Ling Chi,
Galena Kuyumdzhieva,
Lesley Larkin,
Samuel Collins,
Hamish Mohammed,
Thomas Finnie,
Luke Hounsome,
Steven Riley
Abstract:
Advances in Large Language Models (LLMs) have led to significant interest in their potential to support human experts across a range of domains, including public health. In this work we present automated evaluations of LLMs for public health tasks involving the classification and extraction of free text. We combine six externally annotated datasets with seven new internally annotated datasets to e…
▽ More
Advances in Large Language Models (LLMs) have led to significant interest in their potential to support human experts across a range of domains, including public health. In this work we present automated evaluations of LLMs for public health tasks involving the classification and extraction of free text. We combine six externally annotated datasets with seven new internally annotated datasets to evaluate LLMs for processing text related to: health burden, epidemiological risk factors, and public health interventions. We initially evaluate five open-weight LLMs (7-70 billion parameters) across all tasks using zero-shot in-context learning. We find that Llama-3-70B-Instruct is the highest performing model, achieving the best results on 15/17 tasks (using micro-F1 scores). We see significant variation across tasks with all open-weight LLMs scoring below 60% micro-F1 on some challenging tasks, such as Contact Classification, while all LLMs achieve greater than 80% micro-F1 on others, such as GI Illness Classification. For a subset of 12 tasks, we also evaluate GPT-4 and find comparable results to Llama-3-70B-Instruct, which scores equally or outperforms GPT-4 on 6 of the 12 tasks. Overall, based on these initial results we find promising signs that LLMs may be useful tools for public health experts to extract information from a wide variety of free text sources, and support public health surveillance, research, and interventions.
△ Less
Submitted 23 May, 2024;
originally announced May 2024.
-
Multimodal deep learning for map** forest dominant height by fusing GEDI with earth observation data
Authors:
Man Chen,
Wenquan Dong,
Hao Yu,
Iain Woodhouse,
Casey M. Ryan,
Haoyu Liu,
Selena Georgiou,
Edward T. A. Mitchard
Abstract:
The integration of multisource remote sensing data and deep learning models offers new possibilities for accurately map** high spatial resolution forest height. We found that GEDI relative heights (RH) metrics exhibited strong correlation with the mean of the top 10 highest trees (dominant height) measured in situ at the corresponding footprint locations. Consequently, we proposed a novel deep l…
▽ More
The integration of multisource remote sensing data and deep learning models offers new possibilities for accurately map** high spatial resolution forest height. We found that GEDI relative heights (RH) metrics exhibited strong correlation with the mean of the top 10 highest trees (dominant height) measured in situ at the corresponding footprint locations. Consequently, we proposed a novel deep learning framework termed the multi-modal attention remote sensing network (MARSNet) to estimate forest dominant height by extrapolating dominant height derived from GEDI, using Setinel-1 data, ALOS-2 PALSAR-2 data, Sentinel-2 optical data and ancillary data. MARSNet comprises separate encoders for each remote sensing data modality to extract multi-scale features, and a shared decoder to fuse the features and estimate height. Using individual encoders for each remote sensing imagery avoids interference across modalities and extracts distinct representations. To focus on the efficacious information from each dataset, we reduced the prevalent spatial and band redundancies in each remote sensing data by incorporating the extended spatial and band reconstruction convolution modules in the encoders. MARSNet achieved commendable performance in estimating dominant height, with an R2 of 0.62 and RMSE of 2.82 m, outperforming the widely used random forest approach which attained an R2 of 0.55 and RMSE of 3.05 m. Finally, we applied the trained MARSNet model to generate wall-to-wall maps at 10 m resolution for Jilin, China. Through independent validation using field measurements, MARSNet demonstrated an R2 of 0.58 and RMSE of 3.76 m, compared to 0.41 and 4.37 m for the random forest baseline. Our research demonstrates the effectiveness of a multimodal deep learning approach fusing GEDI with SAR and passive optical imagery for enhancing the accuracy of high resolution dominant height estimation.
△ Less
Submitted 20 November, 2023;
originally announced November 2023.
-
A Survey on Machine Learning Techniques for Source Code Analysis
Authors:
Tushar Sharma,
Maria Kechagia,
Stefanos Georgiou,
Rohit Tiwari,
Indira Vats,
Hadi Moazen,
Federica Sarro
Abstract:
The advancements in machine learning techniques have encouraged researchers to apply these techniques to a myriad of software engineering tasks that use source code analysis, such as testing and vulnerability detection. Such a large number of studies hinders the community from understanding the current research landscape. This paper aims to summarize the current knowledge in applied machine learni…
▽ More
The advancements in machine learning techniques have encouraged researchers to apply these techniques to a myriad of software engineering tasks that use source code analysis, such as testing and vulnerability detection. Such a large number of studies hinders the community from understanding the current research landscape. This paper aims to summarize the current knowledge in applied machine learning for source code analysis. We review studies belonging to twelve categories of software engineering tasks and corresponding machine learning techniques, tools, and datasets that have been applied to solve them. To do so, we conducted an extensive literature search and identified 479 primary studies published between 2011 and 2021. We summarize our observations and findings with the help of the identified studies. Our findings suggest that the use of machine learning techniques for source code analysis tasks is consistently increasing. We synthesize commonly used steps and the overall workflow for each task and summarize machine learning techniques employed. We identify a comprehensive list of available datasets and tools useable in this context. Finally, the paper discusses perceived challenges in this area, including the availability of standard datasets, reproducibility and replicability, and hardware resources.
△ Less
Submitted 13 September, 2022; v1 submitted 18 October, 2021;
originally announced October 2021.
-
The significance of user-defined identifiers in Java source code authorship identification
Authors:
Georgia Frantzeskou,
Stephen G. MacDonell,
Efstathios Stamatatos,
Stelios Georgiou,
Stefanos Gritzalis
Abstract:
When writing source code, programmers have varying levels of freedom when it comes to the creation and use of identifiers. Do they habitually use the same identifiers, names that are different to those used by others? Is it then possible to tell who the author of a piece of code is by examining these identifiers? If so, can we use the presence or absence of identifiers to assist in correctly class…
▽ More
When writing source code, programmers have varying levels of freedom when it comes to the creation and use of identifiers. Do they habitually use the same identifiers, names that are different to those used by others? Is it then possible to tell who the author of a piece of code is by examining these identifiers? If so, can we use the presence or absence of identifiers to assist in correctly classifying programs to authors? Is it possible to hide the provenance of programs by identifier renaming? In this study, we assess the importance of three types of identifiers in source code author classification for two different Java program data sets. We do this through a sequence of experiments in which we disguise one type of identifier at a time. These experiments are performed using as a tool the Source Code Author Profiles (SCAP) method. The results show that, although identifiers when examined as a whole do not seem to reflect program authorship for these data sets, when examined separately there is evidence that class names do signal the author of the program. In contrast, simple variables and method names used in Java programs do not appear to reflect program authorship. On the contrary, our analysis suggests that such identifiers are so common as to mask authorship. We believe that these results have applicability in relation to the robustness of code plagiarism analysis and that the underlying methods could be valuable in cases of litigation arising from disputes over program authorship.
△ Less
Submitted 28 January, 2021;
originally announced January 2021.
-
A True AR Authoring Tool for Interactive Virtual Museums
Authors:
Efstratios Geronikolakis,
Paul Zikas,
Steve Kateros,
Nick Lydatakis,
Stelios Georgiou,
Mike Kentros,
George Papagiannakis
Abstract:
In this work, a new and innovative way of spatial computing that appeared recently in the bibliography called True Augmented Reality (AR), is employed in cultural heritage preservation. This innovation could be adapted by the Virtual Museums of the future to enhance the quality of experience. It emphasises, the fact that a visitor will not be able to tell, at a first glance, if the artefact that h…
▽ More
In this work, a new and innovative way of spatial computing that appeared recently in the bibliography called True Augmented Reality (AR), is employed in cultural heritage preservation. This innovation could be adapted by the Virtual Museums of the future to enhance the quality of experience. It emphasises, the fact that a visitor will not be able to tell, at a first glance, if the artefact that he/she is looking at is real or not and it is expected to draw the visitors' interest. True AR is not limited to artefacts but extends even to buildings or life-sized character simulations of statues. It provides the best visual quality possible so that the users will not be able to tell the real objects from the augmented ones. Such applications can be beneficial for future museums, as with True AR, 3D models of various exhibits, monuments, statues, characters and buildings can be reconstructed and presented to the visitors in a realistic and innovative way. We also propose our Virtual Reality Sample application, a True AR playground featuring basic components and tools for generating interactive Virtual Museum applications, alongside a 3D reconstructed character (the priest of Asinou church) facilitating the storyteller of the augmented experience.
△ Less
Submitted 21 October, 2019; v1 submitted 20 September, 2019;
originally announced September 2019.
-
iMapD: intrinsic Map Dynamics exploration for uncharted effective free energy landscapes
Authors:
Eliodoro Chiavazzo,
Ronald R. Coifman,
Roberto Covino,
C. William Gear,
Anastasia S. Georgiou,
Gerhard Hummer,
Ioannis G. Kevrekidis
Abstract:
We describe and implement iMapD, a computer-assisted approach for accelerating the exploration of uncharted effective Free Energy Surfaces (FES), and more generally for the extraction of coarse-grained, macroscopic information from atomistic or stochastic (here Molecular Dynamics, MD) simulations. The approach functionally links the MD simulator with nonlinear manifold learning techniques. The add…
▽ More
We describe and implement iMapD, a computer-assisted approach for accelerating the exploration of uncharted effective Free Energy Surfaces (FES), and more generally for the extraction of coarse-grained, macroscopic information from atomistic or stochastic (here Molecular Dynamics, MD) simulations. The approach functionally links the MD simulator with nonlinear manifold learning techniques. The added value comes from biasing the simulator towards new, unexplored phase space regions by exploiting the smoothness of the (gradually, as the exploration progresses) revealed intrinsic low-dimensional geometry of the FES.
△ Less
Submitted 31 December, 2016;
originally announced January 2017.
-
Window functions and sigmoidal behaviour of memristive systems
Authors:
Panayiotis S. Georgiou,
Sophia N. Yaliraki,
Emmanuel M. Drakakis,
Mauricio Barahona
Abstract:
A common approach to model memristive systems is to include empirical window functions to describe edge effects and non-linearities in the change of the memristance. We demonstrate that under quite general conditions, each window function can be associated with a sigmoidal curve relating the normalised time-dependent memristance to the time integral of the input. Conversely, this explicit relation…
▽ More
A common approach to model memristive systems is to include empirical window functions to describe edge effects and non-linearities in the change of the memristance. We demonstrate that under quite general conditions, each window function can be associated with a sigmoidal curve relating the normalised time-dependent memristance to the time integral of the input. Conversely, this explicit relation allows us to derive window functions suitable for the mesoscopic modelling of memristive systems from a variety of well-known sigmoidals. Such sigmoidal curves are defined in terms of measured variables and can thus be extracted from input and output signals of a device and then transformed to its corresponding window. We also introduce a new generalised window function that allows the flexible modelling of asymmetric edge effects in a simple manner.
△ Less
Submitted 14 January, 2016;
originally announced May 2016.
-
Quantitative Measure of Hysteresis for Memristors Through Explicit Dynamics
Authors:
Panayiotis S. Georgiou,
Sophia N. Yaliraki,
Emmanuel M. Drakakis,
Mauricio Barahona
Abstract:
We introduce a mathematical framework for the analysis of the input-output dynamics of externally driven memristors. We show that, under general assumptions, their dynamics comply with a Bernoulli differential equation and hence can be nonlinearly transformed into a formally solvable linear equation. The Bernoulli formalism, which applies to both charge- and flux-controlled memristors when either…
▽ More
We introduce a mathematical framework for the analysis of the input-output dynamics of externally driven memristors. We show that, under general assumptions, their dynamics comply with a Bernoulli differential equation and hence can be nonlinearly transformed into a formally solvable linear equation. The Bernoulli formalism, which applies to both charge- and flux-controlled memristors when either current- or voltage-driven, can, in some cases, lead to expressions of the output of the device as an explicit function of the input. We apply our framework to obtain analytical solutions of the i-v characteristics of the recently proposed model of the Hewlett-Packard memristor under three different drives without the need for numerical simulations. Our explicit solutions allow us to identify a dimensionless lumped parameter that combines device-specific parameters with properties of the input drive. This parameter governs the memristive behavior of the device and, consequently, the amount of hysteresis in the i-v. We proceed further by defining formally a quantitative measure for the hysteresis of the device for which we obtain explicit formulas in terms of the aforementioned parameter and we discuss the applicability of the analysis for the design and analysis of memristor devices.
△ Less
Submitted 17 July, 2011; v1 submitted 30 October, 2010;
originally announced November 2010.
