-
WisPerMed at "Discharge Me!": Advancing Text Generation in Healthcare with Large Language Models, Dynamic Expert Selection, and Priming Techniques on MIMIC-IV
Authors:
Hendrik Damm,
Tabea M. G. Pakull,
Bahadır Eryılmaz,
Helmut Becker,
Ahmad Idrissi-Yaghir,
Henning Schäfer,
Sergej Schultenkämper,
Christoph M. Friedrich
Abstract:
This study aims to leverage state of the art language models to automate generating the "Brief Hospital Course" and "Discharge Instructions" sections of Discharge Summaries from the MIMIC-IV dataset, reducing clinicians' administrative workload. We investigate how automation can improve documentation accuracy, alleviate clinician burnout, and enhance operational efficacy in healthcare facilities.…
▽ More
This study aims to leverage state of the art language models to automate generating the "Brief Hospital Course" and "Discharge Instructions" sections of Discharge Summaries from the MIMIC-IV dataset, reducing clinicians' administrative workload. We investigate how automation can improve documentation accuracy, alleviate clinician burnout, and enhance operational efficacy in healthcare facilities. This research was conducted within our participation in the Shared Task Discharge Me! at BioNLP @ ACL 2024. Various strategies were employed, including few-shot learning, instruction tuning, and Dynamic Expert Selection (DES), to develop models capable of generating the required text sections. Notably, utilizing an additional clinical domain-specific dataset demonstrated substantial potential to enhance clinical language processing. The DES method, which optimizes the selection of text outputs from multiple predictions, proved to be especially effective. It achieved the highest overall score of 0.332 in the competition, surpassing single-model outputs. This finding suggests that advanced deep learning methods in combination with DES can effectively automate parts of electronic health record documentation. These advancements could enhance patient care by freeing clinician time for patient interactions. The integration of text selection strategies represents a promising avenue for further research.
△ Less
Submitted 18 May, 2024;
originally announced May 2024.
-
ScaleFold: Reducing AlphaFold Initial Training Time to 10 Hours
Authors:
Feiwen Zhu,
Arkadiusz Nowaczynski,
Rundong Li,
Jie Xin,
Yifei Song,
Michal Marcinkiewicz,
Sukru Burc Eryilmaz,
Jun Yang,
Michael Andersch
Abstract:
AlphaFold2 has been hailed as a breakthrough in protein folding. It can rapidly predict protein structures with lab-grade accuracy. However, its implementation does not include the necessary training code. OpenFold is the first trainable public reimplementation of AlphaFold. AlphaFold training procedure is prohibitively time-consuming, and gets diminishing benefits from scaling to more compute res…
▽ More
AlphaFold2 has been hailed as a breakthrough in protein folding. It can rapidly predict protein structures with lab-grade accuracy. However, its implementation does not include the necessary training code. OpenFold is the first trainable public reimplementation of AlphaFold. AlphaFold training procedure is prohibitively time-consuming, and gets diminishing benefits from scaling to more compute resources. In this work, we conducted a comprehensive analysis on the AlphaFold training procedure based on Openfold, identified that inefficient communications and overhead-dominated computations were the key factors that prevented the AlphaFold training from effective scaling. We introduced ScaleFold, a systematic training method that incorporated optimizations specifically for these factors. ScaleFold successfully scaled the AlphaFold training to 2080 NVIDIA H100 GPUs with high resource utilization. In the MLPerf HPC v3.0 benchmark, ScaleFold finished the OpenFold benchmark in 7.51 minutes, shown over $6\times$ speedup than the baseline. For training the AlphaFold model from scratch, ScaleFold completed the pretraining in 10 hours, a significant improvement over the seven days required by the original AlphaFold pretraining baseline.
△ Less
Submitted 17 April, 2024;
originally announced April 2024.
-
Edge AI without Compromise: Efficient, Versatile and Accurate Neurocomputing in Resistive Random-Access Memory
Authors:
Weier Wan,
Rajkumar Kubendran,
Clemens Schaefer,
S. Burc Eryilmaz,
Wenqiang Zhang,
Dabin Wu,
Stephen Deiss,
Priyanka Raina,
He Qian,
Bin Gao,
Siddharth Joshi,
Huaqiang Wu,
H. -S. Philip Wong,
Gert Cauwenberghs
Abstract:
Realizing today's cloud-level artificial intelligence functionalities directly on devices distributed at the edge of the internet calls for edge hardware capable of processing multiple modalities of sensory data (e.g. video, audio) at unprecedented energy-efficiency. AI hardware architectures today cannot meet the demand due to a fundamental "memory wall": data movement between separate compute an…
▽ More
Realizing today's cloud-level artificial intelligence functionalities directly on devices distributed at the edge of the internet calls for edge hardware capable of processing multiple modalities of sensory data (e.g. video, audio) at unprecedented energy-efficiency. AI hardware architectures today cannot meet the demand due to a fundamental "memory wall": data movement between separate compute and memory units consumes large energy and incurs long latency. Resistive random-access memory (RRAM) based compute-in-memory (CIM) architectures promise to bring orders of magnitude energy-efficiency improvement by performing computation directly within memory. However, conventional approaches to CIM hardware design limit its functional flexibility necessary for processing diverse AI workloads, and must overcome hardware imperfections that degrade inference accuracy. Such trade-offs between efficiency, versatility and accuracy cannot be addressed by isolated improvements on any single level of the design. By co-optimizing across all hierarchies of the design from algorithms and architecture to circuits and devices, we present NeuRRAM - the first multimodal edge AI chip using RRAM CIM to simultaneously deliver a high degree of versatility for diverse model architectures, record energy-efficiency $5\times$ - $8\times$ better than prior art across various computational bit-precisions, and inference accuracy comparable to software models with 4-bit weights on all measured standard AI benchmarks including accuracy of 99.0% on MNIST and 85.7% on CIFAR-10 image classification, 84.7% accuracy on Google speech command recognition, and a 70% reduction in image reconstruction error on a Bayesian image recovery task. This work paves a way towards building highly efficient and reconfigurable edge AI hardware platforms for the more demanding and heterogeneous AI applications of the future.
△ Less
Submitted 17 August, 2021;
originally announced August 2021.
-
Opportunities for Analog Coding in Emerging Memory Systems
Authors:
Jesse H. Engel,
S. Burc Eryilmaz,
SangBum Kim,
Matthew BrightSky,
Chung Lam,
Hsiang-Lan Lung,
Bruno A. Olshausen,
H. -S. Philip Wong
Abstract:
The exponential growth in data generation and large-scale data analysis creates an unprecedented need for inexpensive, low-latency, and high-density information storage. This need has motivated significant research into multi-level memory systems that can store multiple bits of information per device. Although both the memory state of these devices and much of the data they store are intrinsically…
▽ More
The exponential growth in data generation and large-scale data analysis creates an unprecedented need for inexpensive, low-latency, and high-density information storage. This need has motivated significant research into multi-level memory systems that can store multiple bits of information per device. Although both the memory state of these devices and much of the data they store are intrinsically analog-valued, both are quantized for use with digital systems and discrete error correcting codes. Using phase change memory as a prototypical multi-level storage technology, we herein demonstrate that analog-valued devices can achieve higher capacities when paired with analog codes. Further, we find that storing analog signals directly through joint-coding can achieve low distortion with reduced coding complexity. By jointly optimizing for signal statistics, device statistics, and a distortion metric, finite-length analog encodings can perform comparable to digital systems with asymptotically infinite large encodings. These results show that end-to-end analog memory systems have not only the potential to reach higher storage capacities than discrete systems, but also to significantly lower coding complexity, leading to faster and more energy efficient storage.
△ Less
Submitted 21 January, 2017;
originally announced January 2017.
-
Training a Probabilistic Graphical Model with Resistive Switching Electronic Synapses
Authors:
S. Burc Eryilmaz,
Emre Neftci,
Siddharth Joshi,
SangBum Kim,
Matthew BrightSky,
Hsiang-Lan Lung,
Chung Lam,
Gert Cauwenberghs,
H. -S. Philip Wong
Abstract:
Current large scale implementations of deep learning and data mining require thousands of processors, massive amounts of off-chip memory, and consume gigajoules of energy. Emerging memory technologies such as nanoscale two-terminal resistive switching memory devices offer a compact, scalable and low power alternative that permits on-chip co-located processing and memory in fine-grain distributed p…
▽ More
Current large scale implementations of deep learning and data mining require thousands of processors, massive amounts of off-chip memory, and consume gigajoules of energy. Emerging memory technologies such as nanoscale two-terminal resistive switching memory devices offer a compact, scalable and low power alternative that permits on-chip co-located processing and memory in fine-grain distributed parallel architecture. Here we report first use of resistive switching memory devices for implementing and training a Restricted Boltzmann Machine (RBM), a generative probabilistic graphical model as a key component for unsupervised learning in deep networks. We experimentally demonstrate a 45-synapse RBM realized with 90 resistive switching phase change memory (PCM) elements trained with a bio-inspired variant of the Contrastive Divergence (CD) algorithm, implementing Hebbian and anti-Hebbian weight updates. The resistive PCM devices show a two-fold to ten-fold reduction in error rate in a missing pixel pattern completion task trained over 30 epochs, compared to untrained case. Measured programming energy consumption is 6.1 nJ per epoch with the resistive switching PCM devices, a factor of ~150 times lower than conventional processor-memory systems. We analyze and discuss the dependence of learning performance on cycle-to-cycle variations as well as number of gradual levels in the PCM analog memory devices.
△ Less
Submitted 9 October, 2016; v1 submitted 27 September, 2016;
originally announced September 2016.
-
Device and System Level Design Considerations for Analog-Non-Volatile-Memory Based Neuromorphic Architectures
Authors:
Sukru Burc Eryilmaz,
Duygu Kuzum,
Shimeng Yu,
H. -S. Philip Wong
Abstract:
This paper gives an overview of recent progress in the brain inspired computing field with a focus on implementation using emerging memories as electronic synapses. Design considerations and challenges such as requirements and design targets on multilevel states, device variability, programming energy, array-level connectivity, fan-in/fanout, wire energy, and IR drop are presented. Wires are incre…
▽ More
This paper gives an overview of recent progress in the brain inspired computing field with a focus on implementation using emerging memories as electronic synapses. Design considerations and challenges such as requirements and design targets on multilevel states, device variability, programming energy, array-level connectivity, fan-in/fanout, wire energy, and IR drop are presented. Wires are increasingly important in design decisions, especially for large systems, and cycle-to-cycle variations have large impact on learning performance.
△ Less
Submitted 6 May, 2016; v1 submitted 25 December, 2015;
originally announced December 2015.
-
Brain-like associative learning using a nanoscale non-volatile phase change synaptic device array
Authors:
Sukru Burc Eryilmaz,
Duygu Kuzum,
Rakesh Jeyasingh,
SangBum Kim,
Matthew BrightSky,
Chung Lam,
H. -S. Philip Wong
Abstract:
Recent advances in neuroscience together with nanoscale electronic device technology have resulted in huge interests in realizing brain-like computing hardwares using emerging nanoscale memory devices as synaptic elements. Although there has been experimental work that demonstrated the operation of nanoscale synaptic element at the single device level, network level studies have been limited to si…
▽ More
Recent advances in neuroscience together with nanoscale electronic device technology have resulted in huge interests in realizing brain-like computing hardwares using emerging nanoscale memory devices as synaptic elements. Although there has been experimental work that demonstrated the operation of nanoscale synaptic element at the single device level, network level studies have been limited to simulations. In this work, we demonstrate, using experiments, array level associative learning using phase change synaptic devices connected in a grid like configuration similar to the organization of the biological brain. Implementing Hebbian learning with phase change memory cells, the synaptic grid was able to store presented patterns and recall missing patterns in an associative brain-like fashion. We found that the system is robust to device variations, and large variations in cell resistance states can be accommodated by increasing the number of training epochs. We illustrated the tradeoff between variation tolerance of the network and the overall energy consumption, and found that energy consumption is decreased significantly for lower variation tolerance.
△ Less
Submitted 13 July, 2014; v1 submitted 19 June, 2014;
originally announced June 2014.
-
Supervised classification-based stock prediction and portfolio optimization
Authors:
Sercan Arik,
Sukru Burc Eryilmaz,
Adam Goldberg
Abstract:
As the number of publicly traded companies as well as the amount of their financial data grows rapidly, it is highly desired to have tracking, analysis, and eventually stock selections automated. There have been few works focusing on estimating the stock prices of individual companies. However, many of those have worked with very small number of financial parameters. In this work, we apply machine…
▽ More
As the number of publicly traded companies as well as the amount of their financial data grows rapidly, it is highly desired to have tracking, analysis, and eventually stock selections automated. There have been few works focusing on estimating the stock prices of individual companies. However, many of those have worked with very small number of financial parameters. In this work, we apply machine learning techniques to address automated stock picking, while using a larger number of financial parameters for individual companies than the previous studies. Our approaches are based on the supervision of prediction parameters using company fundamentals, time-series properties, and correlation information between different stocks. We examine a variety of supervised learning techniques and found that using stock fundamentals is a useful approach for the classification problem, when combined with the high dimensional data handling capabilities of support vector machine. The portfolio our system suggests by predicting the behavior of stocks results in a 3% larger growth on average than the overall market within a 3-month time period, as the out-of-sample test suggests.
△ Less
Submitted 3 June, 2014;
originally announced June 2014.
-
Plasmonic Nanoslit Array Enhanced Metal-Semiconductor-Metal Optical Detectors
Authors:
Sukru Burc Eryilmaz,
Onur Tidin,
Ali K. Okyay
Abstract:
Metallic nanoslit arrays integrated on germanium metal-semiconductor-metal photodetectors show many folds of absorption enhancement for transverse-magnetic polarization in the telecommunication C-band. Such high enhancement is attributed to resonant interference of surface plasmon modes at the metal-semiconductor interface. Horizontal surface plasmon modes were reported earlier to inhibit photodet…
▽ More
Metallic nanoslit arrays integrated on germanium metal-semiconductor-metal photodetectors show many folds of absorption enhancement for transverse-magnetic polarization in the telecommunication C-band. Such high enhancement is attributed to resonant interference of surface plasmon modes at the metal-semiconductor interface. Horizontal surface plasmon modes were reported earlier to inhibit photodetector performance. We computationally show, however, that horizontal modes enhance the efficiency of surface devices despite reducing transmitted light in the far field.
△ Less
Submitted 13 June, 2014; v1 submitted 29 May, 2014;
originally announced May 2014.
-
Experimental Demonstration of Array-level Learning with Phase Change Synaptic Devices
Authors:
S. Burc Eryilmaz,
Duygu Kuzum,
Rakesh G. D. Jeyasingh,
SangBum Kim,
Matthew BrightSky,
Chung Lam,
H. -S. Philip Wong
Abstract:
The computational performance of the biological brain has long attracted significant interest and has led to inspirations in operating principles, algorithms, and architectures for computing and signal processing. In this work, we focus on hardware implementation of brain-like learning in a brain-inspired architecture. We demonstrate, in hardware, that 2-D crossbar arrays of phase change synaptic…
▽ More
The computational performance of the biological brain has long attracted significant interest and has led to inspirations in operating principles, algorithms, and architectures for computing and signal processing. In this work, we focus on hardware implementation of brain-like learning in a brain-inspired architecture. We demonstrate, in hardware, that 2-D crossbar arrays of phase change synaptic devices can achieve associative learning and perform pattern recognition. Device and array-level studies using an experimental 10x10 array of phase change synaptic devices have shown that pattern recognition is robust against synaptic resistance variations and large variations can be tolerated by increasing the number of training iterations. Our measurements show that increase in initial variation from 9 % to 60 % causes required training iterations to increase from 1 to 11.
△ Less
Submitted 3 June, 2014; v1 submitted 29 May, 2014;
originally announced May 2014.
