-
Leveraging Large Language Models for Actionable Course Evaluation Student Feedback to Lecturers
Authors:
Mike Zhang,
Euan D Lindsay,
Frederik Bode Thorbensen,
Danny Bøgsted Poulsen,
Johannes Bjerva
Abstract:
End of semester student evaluations of teaching are the dominant mechanism for providing feedback to academics on their teaching practice. For large classes, however, the volume of feedback makes these tools impractical for this purpose. This paper explores the use of open-source generative AI to synthesise factual, actionable and appropriate summaries of student feedback from these survey respons…
▽ More
End of semester student evaluations of teaching are the dominant mechanism for providing feedback to academics on their teaching practice. For large classes, however, the volume of feedback makes these tools impractical for this purpose. This paper explores the use of open-source generative AI to synthesise factual, actionable and appropriate summaries of student feedback from these survey responses. In our setup, we have 742 student responses ranging over 75 courses in a Computer Science department. For each course, we synthesise a summary of the course evaluations and actionable items for the instructor. Our results reveal a promising avenue for enhancing teaching practices in the classroom setting. Our contribution lies in demonstrating the feasibility of using generative AI to produce insightful feedback for teachers, thus providing a cost-effective means to support educators' development. Overall, our work highlights the possibility of using generative AI to produce factual, actionable, and appropriate feedback for teachers in the classroom setting.
△ Less
Submitted 2 July, 2024; v1 submitted 1 July, 2024;
originally announced July 2024.
-
Temporal distribution of clusters of investors and their application in prediction with expert advice
Authors:
Wojciech Wisniewski,
Yuri Kalnishkan,
David Lindsay,
Siân Lindsay
Abstract:
Financial organisations such as brokers face a significant challenge in servicing the investment needs of thousands of their traders worldwide. This task is further compounded since individual traders will have their own risk appetite and investment goals. Traders may look to capture short-term trends in the market which last only seconds to minutes, or they may have longer-term views which last s…
▽ More
Financial organisations such as brokers face a significant challenge in servicing the investment needs of thousands of their traders worldwide. This task is further compounded since individual traders will have their own risk appetite and investment goals. Traders may look to capture short-term trends in the market which last only seconds to minutes, or they may have longer-term views which last several days to months. To reduce the complexity of this task, client trades can be clustered. By examining such clusters, we would likely observe many traders following common patterns of investment, but how do these patterns vary through time? Knowledge regarding the temporal distributions of such clusters may help financial institutions manage the overall portfolio of risk that accumulates from underlying trader positions. This study contributes to the field by demonstrating that the distribution of clusters derived from the real-world trades of 20k Foreign Exchange (FX) traders (from 2015 to 2017) is described in accordance with Ewens' Sampling Distribution. Further, we show that the Aggregating Algorithm (AA), an on-line prediction with expert advice algorithm, can be applied to the aforementioned real-world data in order to improve the returns of portfolios of trader risk. However we found that the AA 'struggles' when presented with too many trader ``experts'', especially when there are many trades with similar overall patterns. To help overcome this challenge, we have applied and compared the use of Statistically Validated Networks (SVN) with a hierarchical clustering approach on a subset of the data, demonstrating that both approaches can be used to significantly improve results of the AA in terms of profitability and smoothness of returns.
△ Less
Submitted 4 June, 2024;
originally announced June 2024.
-
Transductive Confidence Machine and its application to Medical Data Sets
Authors:
David Lindsay
Abstract:
The Transductive Confidence Machine Nearest Neighbours (TCMNN) algorithm and a supporting, simple user interface was developed. Different settings of the TCMNN algorithms' parameters were tested on medical data sets, in addition to the use of different Minkowski metrics and polynomial kernels. The effect of increasing the number of nearest neighbours and marking results with significance was also…
▽ More
The Transductive Confidence Machine Nearest Neighbours (TCMNN) algorithm and a supporting, simple user interface was developed. Different settings of the TCMNN algorithms' parameters were tested on medical data sets, in addition to the use of different Minkowski metrics and polynomial kernels. The effect of increasing the number of nearest neighbours and marking results with significance was also investigated. SVM implementation of the Transductive Confidence Machine was compared with Nearest Neighbours implementation. The application of neural networks was investigated as a useful comparison to the transductive algorithms.
△ Less
Submitted 24 May, 2024;
originally announced May 2024.
-
Effective Confidence Region Prediction Using Probability Forecasters
Authors:
David Lindsay,
Sian Lindsay
Abstract:
Confidence region prediction is a practically useful extension to the commonly studied pattern recognition problem. Instead of predicting a single label, the constraint is relaxed to allow prediction of a subset of labels given a desired confidence level 1-delta. Ideally, effective region predictions should be (1) well calibrated - predictive regions at confidence level 1-delta should err with rel…
▽ More
Confidence region prediction is a practically useful extension to the commonly studied pattern recognition problem. Instead of predicting a single label, the constraint is relaxed to allow prediction of a subset of labels given a desired confidence level 1-delta. Ideally, effective region predictions should be (1) well calibrated - predictive regions at confidence level 1-delta should err with relative frequency at most delta and (2) be as narrow (or certain) as possible. We present a simple technique to generate confidence region predictions from conditional probability estimates (probability forecasts). We use this 'conversion' technique to generate confidence region predictions from probability forecasts output by standard machine learning algorithms when tested on 15 multi-class datasets. Our results show that approximately 44% of experiments demonstrate well-calibrated confidence region predictions, with the K-Nearest Neighbour algorithm tending to perform consistently well across all data. Our results illustrate the practical benefits of effective confidence region prediction with respect to medical diagnostics, where guarantees of capturing the true disease label can be given.
△ Less
Submitted 24 May, 2024;
originally announced May 2024.
-
Learning from String Sequences
Authors:
David Lindsay,
Sian Lindsay
Abstract:
The Universal Similarity Metric (USM) has been demonstrated to give practically useful measures of "similarity" between sequence data. Here we have used the USM as an alternative distance metric in a K-Nearest Neighbours (K-NN) learner to allow effective pattern recognition of variable length sequence data. We compare this USM approach with the commonly used string-to-word vector approach. Our exp…
▽ More
The Universal Similarity Metric (USM) has been demonstrated to give practically useful measures of "similarity" between sequence data. Here we have used the USM as an alternative distance metric in a K-Nearest Neighbours (K-NN) learner to allow effective pattern recognition of variable length sequence data. We compare this USM approach with the commonly used string-to-word vector approach. Our experiments have used two data sets of divergent domains: (1) spam email filtering and (2) protein subcellular localization. Our results with this data reveal that the USM-based K-NN learner (1) gives predictions with higher classification accuracy than those output by techniques that use the string-to-word vector approach, and (2) can be used to generate reliable probability forecasts.
△ Less
Submitted 10 May, 2024;
originally announced May 2024.
-
A Framework for Responsible Development of Automated Student Feedback with Generative AI
Authors:
Euan D Lindsay,
Aditya Johri,
Johannes Bjerva
Abstract:
Providing rich feedback to students is essential for supporting student learning. Recent advances in generative AI, particularly within large language modelling (LLM), provide the opportunity to deliver repeatable, scalable and instant automatically generated feedback to students, making abundant a previously scarce and expensive learning resource. Such an approach is feasible from a technical per…
▽ More
Providing rich feedback to students is essential for supporting student learning. Recent advances in generative AI, particularly within large language modelling (LLM), provide the opportunity to deliver repeatable, scalable and instant automatically generated feedback to students, making abundant a previously scarce and expensive learning resource. Such an approach is feasible from a technical perspective due to these recent advances in Artificial Intelligence (AI) and Natural Language Processing (NLP); while the potential upside is a strong motivator, doing so introduces a range of potential ethical issues that must be considered as we apply these technologies. The attractiveness of AI systems is that they can effectively automate the most mundane tasks; but this risks introducing a "tyranny of the majority", where the needs of minorities in the long tail are overlooked because they are difficult to automate.
Develo** machine learning models that can generate valuable and authentic feedback requires the input of human domain experts. The choices we make in capturing this expertise -- whose, which, when, and how -- will have significant consequences for the nature of the resulting feedback. How we maintain our models will affect how that feedback remains relevant given temporal changes in context, theory, and prior learning profiles of student cohorts. These questions are important from an ethical perspective; but they are also important from an operational perspective. Unless they can be answered, our AI generated systems will lack the trust necessary for them to be useful features in the contemporary learning environment.
This article will outline the frontiers of automated feedback, identify the ethical issues involved in the provision of automated feedback and present a framework to assist academics to develop such systems responsibly.
△ Less
Submitted 29 August, 2023;
originally announced August 2023.
-
MOOSE: Enabling Massively Parallel Multiphysics Simulation
Authors:
Cody J. Permann,
Derek R. Gaston,
David Andrs,
Robert W. Carlsen,
Fande Kong,
Alexander D. Lindsay,
Jason M. Miller,
John W. Peterson,
Andrew E. Slaughter,
Roy H. Stogner,
Richard C. Martineau
Abstract:
Harnessing modern parallel computing resources to achieve complex multi-physics simulations is a daunting task. The Multiphysics Object Oriented Simulation Environment (MOOSE) aims to enable such development by providing simplified interfaces for specification of partial differential equations, boundary conditions, material properties, and all aspects of a simulation without the need to consider t…
▽ More
Harnessing modern parallel computing resources to achieve complex multi-physics simulations is a daunting task. The Multiphysics Object Oriented Simulation Environment (MOOSE) aims to enable such development by providing simplified interfaces for specification of partial differential equations, boundary conditions, material properties, and all aspects of a simulation without the need to consider the parallel, adaptive, nonlinear, finite-element solve that is handled internally. Through the use of interfaces and inheritance, each portion of a simulation becomes reusable and composable in a manner that allows disparate research groups to share code and create an ecosystem of growing capability that lowers the barrier for the creation of multiphysics simulation codes. Included within the framework is a unique capability for building multiscale, multiphysics simulations through simultaneous execution of multiple sub-applications with data transfers between the scales. Other capabilities include automatic differentiation, scaling to a large number of processors, hybrid parallelism, and mesh adaptivity. To date, MOOSE-based applications have been created in areas of science and engineering such as nuclear physics, geothermal science, magneto-hydrodynamics, seismic events, compressible and incompressible fluid flow, microstructure evolution, and advanced manufacturing processes.
△ Less
Submitted 11 November, 2019;
originally announced November 2019.
-
Transformative effects of IoT, Blockchain and Artificial Intelligence on cloud computing: Evolution, vision, trends and open challenges
Authors:
Sukhpal Singh Gill,
Shreshth Tuli,
Minxian Xu,
Inderpreet Singh,
Karan Vijay Singh,
Dominic Lindsay,
Shikhar Tuli,
Daria Smirnova,
Manmeet Singh,
Udit Jain,
Haris Pervaiz,
Bhanu Sehgal,
Sukhwinder Singh Kaila,
Sanjay Misra,
Mohammad Sadegh Aslanpour,
Harshit Mehta,
Vlado Stankovski,
Peter Garraghan
Abstract:
Cloud computing plays a critical role in modern society and enables a range of applications from infrastructure to social media. Such system must cope with varying load and evolving usage reflecting societies interaction and dependency on automated computing systems whilst satisfying Quality of Service (QoS) guarantees. Enabling these systems are a cohort of conceptual technologies, synthesized to…
▽ More
Cloud computing plays a critical role in modern society and enables a range of applications from infrastructure to social media. Such system must cope with varying load and evolving usage reflecting societies interaction and dependency on automated computing systems whilst satisfying Quality of Service (QoS) guarantees. Enabling these systems are a cohort of conceptual technologies, synthesized to meet demand of evolving computing applications. In order to understand current and future challenges of such system, there is a need to identify key technologies enabling future applications. In this study, we aim to explore how three emerging paradigms (Blockchain, IoT and Artificial Intelligence) will influence future cloud computing systems. Further, we identify several technologies driving these paradigms and invite international experts to discuss the current status and future directions of cloud computing. Finally, we proposed a conceptual model for cloud futurology to explore the influence of emerging paradigms and technologies on evolution of cloud computing.
△ Less
Submitted 21 October, 2019;
originally announced November 2019.
-
A scalable multilevel domain decomposition preconditioner with a subspace-based coarsening algorithm for the neutron transport calculations
Authors:
Fande Kong,
Yaqi Wang,
Derek R. Gaston,
Alexander D. Lindsay,
Cody J. Permann,
Richard C. Martineau
Abstract:
The multigroup neutron transport equations has been widely used to study the interactions of neutrons with their background materials in nuclear reactors. High-resolution simulations of the multigroup neutron transport equations using modern supercomputers require the development of scalable parallel solving techniques. In this paper, we study a scalable transport method for solving the algebraic…
▽ More
The multigroup neutron transport equations has been widely used to study the interactions of neutrons with their background materials in nuclear reactors. High-resolution simulations of the multigroup neutron transport equations using modern supercomputers require the development of scalable parallel solving techniques. In this paper, we study a scalable transport method for solving the algebraic system arising from the discretization of the multigroup neutron transport equations. The proposed transport method consists of a fully coupled Newton solver for the generalized eigenvalue problems and GMRES together with a novel multilevel domain decomposition preconditioner for the Jacobian system. The multilevel preconditioner has been successfully used for many problems, but the construction of coarse spaces for certain problems, especially for unstructured mesh problems, is expensive and often unscalable. We introduce a new subspace-based coarsening algorithm to address this issue by exploring the structure of the matrix in the discretized version of the neutron transport problems. We numerically demonstrate that the proposed transport method is highly scalable with more than 10,000 processor cores for the 3D C5G7 benchmark problem on unstructured meshes with billions of unknowns. Compared with the traditional multilevel domain decomposition method, the new approach equipped with the subspace-based coarsening algorithm is much faster on the construction of coarse spaces.
△ Less
Submitted 18 June, 2019;
originally announced June 2019.
-
A highly parallel multilevel Newton-Krylov-Schwarz method with subspace-based coarsening and partition-based balancing for the multigroup neutron transport equations on 3D unstructured meshes
Authors:
Fande Kong,
Yaqi Wang,
Derek R. Gaston,
Cody J. Permann,
Andrew E. Slaughter,
Alexander D. Lindsay,
Richard C. Martineau
Abstract:
The multigroup neutron transport equations have been widely used to study the motion of neutrons and their interactions with the background materials. Numerical simulation of the multigroup neutron transport equations is computationally challenging because the equations is defined on a high dimensional phase space (1D in energy, 2D in angle, and 3D in spatial space), and furthermore, for realistic…
▽ More
The multigroup neutron transport equations have been widely used to study the motion of neutrons and their interactions with the background materials. Numerical simulation of the multigroup neutron transport equations is computationally challenging because the equations is defined on a high dimensional phase space (1D in energy, 2D in angle, and 3D in spatial space), and furthermore, for realistic applications, the computational spatial domain is complex and the materials are heterogeneous. The multilevel domain decomposition methods is one of the most popular algorithms for solving the multigroup neutron transport equations, but the construction of coarse spaces is expensive and often not strongly scalable when the number of processor cores is large. In this paper, we study a highly parallel multilevel Newton-Krylov-Schwarz method equipped with several novel components, such as subspace-based coarsening, partition-based balancing and hierarchical mesh partitioning, that enable the overall simulation strongly scalable in terms of the compute time. Compared with the traditional coarsening method, the subspace-based coarsening algorithm significantly reduces the cost of the preconditioner setup that is often unscalable. In addition, the partition-based balancing strategy enhances the parallel efficiency of the overall solver by assigning a nearly-equal amount of work to each processor core. The hierarchical mesh partitioning is able to generate a large number of subdomains and meanwhile minimizes the off-node communication. We numerically show that the proposed algorithm is scalable with more than 10,000 processor cores for a realistic application with a few billions unknowns on 3D unstructured meshes.
△ Less
Submitted 8 March, 2019;
originally announced March 2019.
-
A New Framework for Network Disruption
Authors:
Susan E. Martonosi,
Doug Altner,
Michael Ernst,
Elizabeth Ferme,
Kira Langsjoen,
Danika Lindsay,
Sean Plott,
Andrew S. Ronan
Abstract:
Traditional network disruption approaches focus on disconnecting or lengthening paths in the network. We present a new framework for network disruption that attempts to reroute flow through critical vertices via vertex deletion, under the assumption that this will render those vertices vulnerable to future attacks. We define the load on a critical vertex to be the number of paths in the network th…
▽ More
Traditional network disruption approaches focus on disconnecting or lengthening paths in the network. We present a new framework for network disruption that attempts to reroute flow through critical vertices via vertex deletion, under the assumption that this will render those vertices vulnerable to future attacks. We define the load on a critical vertex to be the number of paths in the network that must flow through the vertex. We present graph-theoretic and computational techniques to maximize this load, firstly by removing either a single vertex from the network, secondly by removing a subset of vertices.
△ Less
Submitted 13 September, 2011;
originally announced September 2011.
