Showing 1–14 of 14 results for author: Herzog, R

Inpainting Pathology in Lumbar Spine MRI with Latent Diffusion

Authors: Colin Hansen, Simas Glinskis, Ashwin Raju, Micha Kornreich, **Hyeong Park, Jayashri Pawar, Richard Herzog, Li Zhang, Benjamin Odry

Abstract: Data driven models for automated diagnosis in radiology suffer from insufficient and imbalanced datasets due to low representation of pathology in a population and the cost of expert annotations. Datasets can be bolstered through data augmentation. However, even when utilizing a full suite of transformations during model training, typical data augmentations do not address variations in human anato… ▽ More Data driven models for automated diagnosis in radiology suffer from insufficient and imbalanced datasets due to low representation of pathology in a population and the cost of expert annotations. Datasets can be bolstered through data augmentation. However, even when utilizing a full suite of transformations during model training, typical data augmentations do not address variations in human anatomy. An alternative direction is to synthesize data using generative models, which can potentially craft datasets with specific attributes. While this holds promise, commonly used generative models such as Generative Adversarial Networks may inadvertently produce anatomically inaccurate features. On the other hand, diffusion models, which offer greater stability, tend to memorize training data, raising concerns about privacy and generative diversity. Alternatively, inpainting has the potential to augment data through directly inserting pathology in medical images. However, this approach introduces a new challenge: accurately merging the generated pathological features with the surrounding anatomical context. While inpainting is a well established method for addressing simple lesions, its application to pathologies that involve complex structural changes remains relatively unexplored. We propose an efficient method for inpainting pathological features onto healthy anatomy in MRI through voxelwise noise scheduling in a latent diffusion model. We evaluate the method's ability to insert disc herniation and central canal stenosis in lumbar spine sagittal T2 MRI, and it achieves superior Frechet Inception Distance compared to state-of-the-art methods. △ Less

Submitted 4 June, 2024; originally announced June 2024.

Shape Constraints in Symbolic Regression using Penalized Least Squares

Authors: Viktor Martinek, Julia Reuter, Ophelia Frotscher, Sanaz Mostaghim, Markus Richter, Roland Herzog

Abstract: We study the addition of shape constraints and their consideration during the parameter estimation step of symbolic regression (SR). Shape constraints serve as a means to introduce prior knowledge about the shape of the otherwise unknown model function into SR. Unlike previous works that have explored shape constraints in SR, we propose minimizing shape constraint violations during parameter estim… ▽ More We study the addition of shape constraints and their consideration during the parameter estimation step of symbolic regression (SR). Shape constraints serve as a means to introduce prior knowledge about the shape of the otherwise unknown model function into SR. Unlike previous works that have explored shape constraints in SR, we propose minimizing shape constraint violations during parameter estimation using gradient-based numerical optimization. We test three algorithm variants to evaluate their performance in identifying three symbolic expressions from a synthetically generated data set. This paper examines two benchmark scenarios: one with varying noise levels and another with reduced amounts of training data. The results indicate that incorporating shape constraints into the expression search is particularly beneficial when data is scarce. Compared to using shape constraints only in the selection process, our approach of minimizing violations during parameter estimation shows a statistically significant benefit in some of our test cases, without being significantly worse in any instance. △ Less

Submitted 31 May, 2024; originally announced May 2024.

Unit-Aware Genetic Programming for the Development of Empirical Equations

Authors: Julia Reuter, Viktor Martinek, Roland Herzog, Sanaz Mostaghim

Abstract: When develo** empirical equations, domain experts require these to be accurate and adhere to physical laws. Often, constants with unknown units need to be discovered alongside the equations. Traditional unit-aware genetic programming (GP) approaches cannot be used when unknown constants with undetermined units are included. This paper presents a method for dimensional analysis that propagates un… ▽ More When develo** empirical equations, domain experts require these to be accurate and adhere to physical laws. Often, constants with unknown units need to be discovered alongside the equations. Traditional unit-aware genetic programming (GP) approaches cannot be used when unknown constants with undetermined units are included. This paper presents a method for dimensional analysis that propagates unknown units as ''jokers'' and returns the magnitude of unit violations. We propose three methods, namely evolutive culling, a repair mechanism, and a multi-objective approach, to integrate the dimensional analysis in the GP algorithm. Experiments on datasets with ground truth demonstrate comparable performance of evolutive culling and the multi-objective approach to a baseline without dimensional analysis. Extensive analysis of the results on datasets without ground truth reveals that the unit-aware algorithms make only low sacrifices in accuracy, while producing unit-adherent solutions. Overall, we presented a promising novel approach for develo** unit-adherent empirical equations. △ Less

Submitted 29 May, 2024; originally announced May 2024.

Comments: Submitted to Conference Proceedings of PPSN2024

Adaptive Step Sizes for Preconditioned Stochastic Gradient Descent

Authors: Frederik Köhne, Leonie Kreis, Anton Schiela, Roland Herzog

Abstract: This paper proposes a novel approach to adaptive step sizes in stochastic gradient descent (SGD) by utilizing quantities that we have identified as numerically traceable -- the Lipschitz constant for gradients and a concept of the local variance in search directions. Our findings yield a nearly hyperparameter-free algorithm for stochastic optimization, which has provable convergence properties whe… ▽ More This paper proposes a novel approach to adaptive step sizes in stochastic gradient descent (SGD) by utilizing quantities that we have identified as numerically traceable -- the Lipschitz constant for gradients and a concept of the local variance in search directions. Our findings yield a nearly hyperparameter-free algorithm for stochastic optimization, which has provable convergence properties when applied to quadratic problems and exhibits truly problem adaptive behavior on classical image classification tasks. Our framework enables the potential inclusion of a preconditioner, thereby enabling the implementation of adaptive step sizes for stochastic second-order optimization methods. △ Less

Submitted 28 November, 2023; originally announced November 2023.

Sensitivity-Based Layer Insertion for Residual and Feedforward Neural Networks

Authors: Evelyn Herberg, Roland Herzog, Frederik Köhne, Leonie Kreis, Anton Schiela

Abstract: The training of neural networks requires tedious and often manual tuning of the network architecture. We propose a systematic method to insert new layers during the training process, which eliminates the need to choose a fixed network size before training. Our technique borrows techniques from constrained optimization and is based on first-order sensitivity information of the objective with respec… ▽ More The training of neural networks requires tedious and often manual tuning of the network architecture. We propose a systematic method to insert new layers during the training process, which eliminates the need to choose a fixed network size before training. Our technique borrows techniques from constrained optimization and is based on first-order sensitivity information of the objective with respect to the virtual parameters that additional layers, if inserted, would offer. We consider fully connected feedforward networks with selected activation functions as well as residual neural networks. In numerical experiments, the proposed sensitivity-based layer insertion technique exhibits improved training decay, compared to not inserting the layer. Furthermore, the computational effort is reduced in comparison to inserting the layer from the beginning. The code is available at \url{https://github.com/LeonieKreis/layer_insertion_sensitivity_based}. △ Less

Submitted 27 November, 2023; originally announced November 2023.

Frobenius-Type Norms and Inner Products of Matrices and Linear Maps with Applications to Neural Network Training

Authors: Roland Herzog, Frederik Köhne, Leonie Kreis, Anton Schiela

Abstract: The Frobenius norm is a frequent choice of norm for matrices. In particular, the underlying Frobenius inner product is typically used to evaluate the gradient of an objective with respect to matrix variable, such as those occuring in the training of neural networks. We provide a broader view on the Frobenius norm and inner product for linear maps or matrices, and establish their dependence on inne… ▽ More The Frobenius norm is a frequent choice of norm for matrices. In particular, the underlying Frobenius inner product is typically used to evaluate the gradient of an objective with respect to matrix variable, such as those occuring in the training of neural networks. We provide a broader view on the Frobenius norm and inner product for linear maps or matrices, and establish their dependence on inner products in the domain and co-domain spaces. This shows that the classical Frobenius norm is merely one special element of a family of more general Frobenius-type norms. The significant extra freedom furnished by this realization can be used, among other things, to precondition neural network training. △ Less

Submitted 26 November, 2023; originally announced November 2023.

Introducing Thermodynamics-Informed Symbolic Regression -- A Tool for Thermodynamic Equations of State Development

Authors: Viktor Martinek, Ophelia Frotscher, Markus Richter, Roland Herzog

Abstract: Thermodynamic equations of state (EOS) are essential for many industries as well as in academia. Even leaving aside the expensive and extensive measurement campaigns required for the data acquisition, the development of EOS is an intensely time-consuming process, which does often still heavily rely on expert knowledge and iterative fine-tuning. To improve upon and accelerate the EOS development pr… ▽ More Thermodynamic equations of state (EOS) are essential for many industries as well as in academia. Even leaving aside the expensive and extensive measurement campaigns required for the data acquisition, the development of EOS is an intensely time-consuming process, which does often still heavily rely on expert knowledge and iterative fine-tuning. To improve upon and accelerate the EOS development process, we introduce thermodynamics-informed symbolic regression (TiSR), a symbolic regression (SR) tool aimed at thermodynamic EOS modeling. TiSR is already a capable SR tool, which was used in the research of https://doi.org/10.1007/s10765-023-03197-z. It aims to combine an SR base with the extensions required to work with often strongly scattered experimental data, different residual pre- and post-processing options, and additional features required to consider thermodynamic EOS development. Although TiSR is not ready for end users yet, this paper is intended to report on its current state, showcase the progress, and discuss (distant and not so distant) future directions. TiSR is available at https://github.com/scoop-group/TiSR and can be cited as https://doi.org/10.5281/zenodo.8317547. △ Less

Submitted 6 September, 2023; originally announced September 2023.

Time Regularization in Optimal Time Variable Learning

Authors: Evelyn Herberg, Roland Herzog, Frederik Köhne

Abstract: Recently, optimal time variable learning in deep neural networks (DNNs) was introduced in arXiv:2204.08528. In this manuscript we extend the concept by introducing a regularization term that directly relates to the time horizon in discrete dynamical systems. Furthermore, we propose an adaptive pruning approach for Residual Neural Networks (ResNets), which reduces network complexity without comprom… ▽ More Recently, optimal time variable learning in deep neural networks (DNNs) was introduced in arXiv:2204.08528. In this manuscript we extend the concept by introducing a regularization term that directly relates to the time horizon in discrete dynamical systems. Furthermore, we propose an adaptive pruning approach for Residual Neural Networks (ResNets), which reduces network complexity without compromising expressiveness, while simultaneously decreasing training time. The results are illustrated by applying the proposed concepts to classification tasks on the well known MNIST and Fashion MNIST data sets. Our PyTorch code is available on https://github.com/frederikkoehne/time_variable_learning. △ Less

Submitted 6 December, 2023; v1 submitted 28 June, 2023; originally announced June 2023.

Identifying public values and spatial conflicts in urban planning

Authors: Rico H. Herzog, Juliana E. Gonçalves, Geertje Slingerland, Reinout Kleinhans, Holger Prang, Frances Brazier, Trivik Verma

Abstract: Identifying the diverse and often competing values of citizens, and resolving the consequent public value conflicts, are of significant importance for inclusive and integrated urban development. Scholars have highlighted that relational, value-laden urban space gives rise to many diverse conflicts that vary both spatially and temporally. Although notions of public value conflicts have been conceiv… ▽ More Identifying the diverse and often competing values of citizens, and resolving the consequent public value conflicts, are of significant importance for inclusive and integrated urban development. Scholars have highlighted that relational, value-laden urban space gives rise to many diverse conflicts that vary both spatially and temporally. Although notions of public value conflicts have been conceived in theory, there are very few empirical studies that identify such values and their conflicts in urban space. Building on public value theory and using a case-study mixed-methods approach, this paper proposes a new approach to empirically investigate public value conflicts in urban space. Using unstructured participatory data of 4,528 citizen contributions from a Public Participation Geographic Information Systems in Hamburg, Germany, natural language processing and spatial clustering techniques are used to identify areas of potential value conflicts. Four expert workshops assess and interpret these quantitative findings. Integrating both quantitative and qualitative results, 19 general public values and a total of 9 archetypical conflicts are identified. On the basis of these results, this paper proposes a new conceptual tool of Public Value Spheres that extends the theoretical notion of public-value conflicts and helps to further account for the value-laden nature of urban space. △ Less

Submitted 18 July, 2022; v1 submitted 11 July, 2022; originally announced July 2022.

Explaining the Effectiveness of Multi-Task Learning for Efficient Knowledge Extraction from Spine MRI Reports

Authors: Arijit Sehanobish, McCullen Sandora, Nabila Abraham, Jayashri Pawar, Danielle Torres, Anasuya Das, Murray Becker, Richard Herzog, Benjamin Odry, Ron Vianu

Abstract: Pretrained Transformer based models finetuned on domain specific corpora have changed the landscape of NLP. However, training or fine-tuning these models for individual tasks can be time consuming and resource intensive. Thus, a lot of current research is focused on using transformers for multi-task learning (Raffel et al.,2020) and how to group the tasks to help a multi-task model to learn effect… ▽ More Pretrained Transformer based models finetuned on domain specific corpora have changed the landscape of NLP. However, training or fine-tuning these models for individual tasks can be time consuming and resource intensive. Thus, a lot of current research is focused on using transformers for multi-task learning (Raffel et al.,2020) and how to group the tasks to help a multi-task model to learn effective representations that can be shared across tasks (Standley et al., 2020; Fifty et al., 2021). In this work, we show that a single multi-tasking model can match the performance of task specific models when the task specific models show similar representations across all of their hidden layers and their gradients are aligned, i.e. their gradients follow the same direction. We hypothesize that the above observations explain the effectiveness of multi-task learning. We validate our observations on our internal radiologist-annotated datasets on the cervical and lumbar spine. Our method is simple and intuitive, and can be used in a wide range of NLP problems. △ Less

Submitted 5 May, 2022; originally announced May 2022.

Comments: To appear at NAACL-2022, Industry Track. Follow-up of previous work: arXiv:2204.04544

Efficient Extraction of Pathologies from C-Spine Radiology Reports using Multi-Task Learning

Authors: Arijit Sehanobish, Nathaniel Brown, Ishita Daga, Jayashri Pawar, Danielle Torres, Anasuya Das, Murray Becker, Richard Herzog, Benjamin Odry, Ron Vianu

Abstract: Pretrained Transformer based models finetuned on domain specific corpora have changed the landscape of NLP. Generally, if one has multiple tasks on a given dataset, one may finetune different models or use task specific adapters. In this work, we show that a multi-task model can beat or achieve the performance of multiple BERT-based models finetuned on various tasks and various task specific adapt… ▽ More Pretrained Transformer based models finetuned on domain specific corpora have changed the landscape of NLP. Generally, if one has multiple tasks on a given dataset, one may finetune different models or use task specific adapters. In this work, we show that a multi-task model can beat or achieve the performance of multiple BERT-based models finetuned on various tasks and various task specific adapter augmented BERT-based models. We validate our method on our internal radiologist's report dataset on cervical spine. We hypothesize that the tasks are semantically close and related and thus multitask learners are powerful classifiers. Our work opens the scope of using our method to radiologist's reports on various body parts. △ Less

Submitted 9 April, 2022; originally announced April 2022.

Comments: Accepted at 6th International Workshop on Health Intelligence, AAAI-2022. To appear in as a book chapter published by Springer in Studies in Computational Intelligence

Spatio-Temporal Outdoor Lighting Aggregation on Image Sequences using Transformer Networks

Authors: Haebom Lee, Christian Homeyer, Robert Herzog, Jan Rexilius, Carsten Rother

Abstract: In this work, we focus on outdoor lighting estimation by aggregating individual noisy estimates from images, exploiting the rich image information from wide-angle cameras and/or temporal image sequences. Photographs inherently encode information about the scene's lighting in the form of shading and shadows. Recovering the lighting is an inverse rendering problem and as that ill-posed. Recent work… ▽ More In this work, we focus on outdoor lighting estimation by aggregating individual noisy estimates from images, exploiting the rich image information from wide-angle cameras and/or temporal image sequences. Photographs inherently encode information about the scene's lighting in the form of shading and shadows. Recovering the lighting is an inverse rendering problem and as that ill-posed. Recent work based on deep neural networks has shown promising results for single image lighting estimation, but suffers from robustness. We tackle this problem by combining lighting estimates from several image views sampled in the angular and temporal domain of an image sequence. For this task, we introduce a transformer architecture that is trained in an end-2-end fashion without any statistical post-processing as required by previous work. Thereby, we propose a positional encoding that takes into account the camera calibration and ego-motion estimation to globally register the individual estimates when computing attention between visual words. We show that our method leads to improved lighting estimation while requiring less hyper-parameters compared to the state-of-the-art. △ Less

Submitted 18 February, 2022; originally announced February 2022.

Comments: 11 pages, 7 figures, 1 table, currently under a review process

Using Probabilistic Movement Primitives in Analyzing Human Motion Difference under Transcranial Current Stimulation

Authors: Honghu Xue, Rebecca Herzog, Till M Berger, Tobias Bäumer, Anne Weissbach, Elmar Rueckert

Abstract: In medical tasks such as human motion analysis, computer-aided auxiliary systems have become preferred choice for human experts for its high efficiency. However, conventional approaches are typically based on user-defined features such as movement onset times, peak velocities, motion vectors or frequency domain analyses. Such approaches entail careful data post-processing or specific domain knowle… ▽ More In medical tasks such as human motion analysis, computer-aided auxiliary systems have become preferred choice for human experts for its high efficiency. However, conventional approaches are typically based on user-defined features such as movement onset times, peak velocities, motion vectors or frequency domain analyses. Such approaches entail careful data post-processing or specific domain knowledge to achieve a meaningful feature extraction. Besides, they are prone to noise and the manual-defined features could hardly be re-used for other analyses. In this paper, we proposed probabilistic movement primitives (ProMPs), a widely-used approach in robot skill learning, to model human motions. The benefit of ProMPs is that the features are directly learned from the data and ProMPs can capture important features describing the trajectory shape, which can easily be extended to other tasks. Distinct from previous research, where classification tasks are mostly investigated, we applied ProMPs together with a variant of Kullback-Leibler (KL) divergence to quantify the effect of different transcranial current stimulation methods on human motions. We presented an initial result with 10 participants. The results validate ProMPs as a robust and effective feature extractor for human motions. △ Less

Submitted 5 July, 2021; originally announced July 2021.

Journal ref: https://www.frontiersin.org/articles/10.3389/frobt.2021.721890/full

Mesh Denoising and Inpainting using the Total Variation of the Normal and a Shape Newton Approach

Authors: Lukas Baumgärtner, Ronny Bergmann, Roland Herzog, Stephan Schmidt, José Vidal-Núñez, Manuel Weiß

Abstract: We present a novel approach to denoising and inpainting problems for surface meshes. The purpose of these problems is to remove noise or fill in missing parts while preserving important features such as sharp edges. A discrete variant of the total variation of the unit normal vector field serves as a regularizing functional to achieve these goals. In order to solve the resulting problem, we use a… ▽ More We present a novel approach to denoising and inpainting problems for surface meshes. The purpose of these problems is to remove noise or fill in missing parts while preserving important features such as sharp edges. A discrete variant of the total variation of the unit normal vector field serves as a regularizing functional to achieve these goals. In order to solve the resulting problem, we use a version of the split Bregman (ADMM) iteration adapted to the problem. A new formulation of the total variation regularizer, as well as the use of an inexact Newton method for the shape optimization step, bring significant speed-up compared to earlier methods. Numerical examples are included, demonstrating the performance of our algorithm with some complex 3D geometries. △ Less

Submitted 12 March, 2024; v1 submitted 21 December, 2020; originally announced December 2020.