-
OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training
Authors:
Sami Jaghouar,
Jack Min Ong,
Johannes Hagemann
Abstract:
OpenDiLoCo is an open-source implementation and replication of the Distributed Low-Communication (DiLoCo) training method for large language models. We provide a reproducible implementation of the DiLoCo experiments, offering it within a scalable, decentralized training framework using the Hivemind library. We demonstrate its effectiveness by training a model across two continents and three countr…
▽ More
OpenDiLoCo is an open-source implementation and replication of the Distributed Low-Communication (DiLoCo) training method for large language models. We provide a reproducible implementation of the DiLoCo experiments, offering it within a scalable, decentralized training framework using the Hivemind library. We demonstrate its effectiveness by training a model across two continents and three countries, while maintaining 90-95% compute utilization. Additionally, we conduct ablations studies focusing on the algorithm's compute efficiency, scalability in the number of workers and show that its gradients can be all-reduced using FP16 without any performance degradation. Furthermore, we scale OpenDiLoCo to 3x the size of the original work, demonstrating its effectiveness for billion parameter models.
△ Less
Submitted 10 July, 2024;
originally announced July 2024.
-
Cylindrical compression of thin wires by irradiation with a Joule-class short pulse laser
Authors:
Alejandro Laso Garcia,
Long Yang,
Victorien Bouffetier,
Karen Apple,
Carsten Baehtz,
Johannes Hagemann,
Hauke Höppner,
Oliver Humphries,
Mikhail Mishchenko,
Motoaki Nakatsutsumi,
Alexander Pelka,
Thomas R. Preston,
Lisa Randolph,
Ulf Zastrau,
Thomas E. Cowan,
Lingen Huang,
Toma Toncian
Abstract:
Equation of state measurements at Jovian or stellar conditions are currently conducted by dynamic shock compression driven by multi-kilojoule multi-beam nanosecond-duration lasers. These experiments require precise design of the target and specific tailoring of the spatial and temporal laser profiles to reach the highest pressures. At the same time, the studies are limited by the low repetition ra…
▽ More
Equation of state measurements at Jovian or stellar conditions are currently conducted by dynamic shock compression driven by multi-kilojoule multi-beam nanosecond-duration lasers. These experiments require precise design of the target and specific tailoring of the spatial and temporal laser profiles to reach the highest pressures. At the same time, the studies are limited by the low repetition rate of the lasers. Here, we show that by the irradiation of a thin wire with single beam Joule-class short-pulse laser, a converging cylindrical shock is generated compressing the wire material to conditions relevant for the above applications. The shockwave was observed using Phase Contrast Imaging employing a hard X-ray Free Electron Laser with unprecedented temporal and spatial sensitivity. The data collected for Cu wires is in agreement with hydrodynamic simulations of an ablative shock launched by a highly-impulsive and transient resistive heating of the wire surface. The subsequent cylindrical shockwave travels towards the wire axis and is predicted to reach a compression factor of 9 and pressures above 800 Mbar. Simulations for astrophysical relevant materials underline the potential of this compression technique as a new tool for high energy density studies at high repetition rates.
△ Less
Submitted 10 February, 2024;
originally announced February 2024.
-
In situ Imaging of Precipitate Formation in Additively Manufactured Al-Alloys by Scanning X-ray Fluorescence
Authors:
Isac Lazar,
Bharat Mehta,
Vendulka Bertschová,
Sri Bala Aditya Malladi,
Zhe Ren,
Srashtasrita Das,
Johannes Hagemann,
Gerald Falkenberg,
Karin Frisk,
Anders Mikkelsen,
Lars Nyborg
Abstract:
A new family of high-strength Al-alloys has recently been developed, tailored for the powder bed fusion-laser beam process. In these alloys, Mn, Cr and Zr are incorporated in solid solution at amounts up to three times that of equilibrium in the as-printed state. Mn and Cr-enriched precipitates that form during printing and heat treatment influence the material's mechanical properties. In this stu…
▽ More
A new family of high-strength Al-alloys has recently been developed, tailored for the powder bed fusion-laser beam process. In these alloys, Mn, Cr and Zr are incorporated in solid solution at amounts up to three times that of equilibrium in the as-printed state. Mn and Cr-enriched precipitates that form during printing and heat treatment influence the material's mechanical properties. In this study, direct imaging of these precipitates was accomplished through the utilisation of in situ synchrotron-based scanning X-ray fluorescence. During heat treatment, a selective accumulation of Cr and Mn in two distinct types of precipitates at grain boundaries was observed. Additionally, the microstructure at the melt-pool boundary, containing precipitates found in the as-printed state, remains thermally stable during the heat treatment. The study demonstrates the significant value of employing high-sensitivity in-situ X-ray fluorescence microscopy in exploring the kinetics of sub-micrometre scale precipitation.
△ Less
Submitted 24 November, 2023;
originally announced November 2023.
-
Efficient Parallelization Layouts for Large-Scale Distributed Model Training
Authors:
Johannes Hagemann,
Samuel Weinbach,
Konstantin Dobler,
Maximilian Schall,
Gerard de Melo
Abstract:
Efficiently training large language models requires parallelizing across hundreds of hardware accelerators and invoking various compute and memory optimizations. When combined, many of these strategies have complex interactions regarding the final training efficiency. Prior work tackling this problem did not have access to the latest set of optimizations, such as FlashAttention or sequence paralle…
▽ More
Efficiently training large language models requires parallelizing across hundreds of hardware accelerators and invoking various compute and memory optimizations. When combined, many of these strategies have complex interactions regarding the final training efficiency. Prior work tackling this problem did not have access to the latest set of optimizations, such as FlashAttention or sequence parallelism. In this work, we conduct a comprehensive ablation study of possible training configurations for large language models. We distill this large study into several key recommendations for the most efficient training. For instance, we find that using a micro-batch size of 1 usually enables the most efficient training layouts. Larger micro-batch sizes necessitate activation checkpointing or higher degrees of model parallelism and also lead to larger pipeline bubbles. Our most efficient configurations enable us to achieve state-of-the-art training efficiency results over a range of model sizes, most notably a Model FLOPs utilization of 70.5% when training a Llama 13B model.
△ Less
Submitted 10 December, 2023; v1 submitted 9 November, 2023;
originally announced November 2023.
-
Less Is More: A Comparison of Active Learning Strategies for 3D Medical Image Segmentation
Authors:
Josafat-Mattias Burmeister,
Marcel Fernandez Rosas,
Johannes Hagemann,
Jonas Kordt,
Jasper Blum,
Simon Shabo,
Benjamin Bergner,
Christoph Lippert
Abstract:
Since labeling medical image data is a costly and labor-intensive process, active learning has gained much popularity in the medical image segmentation domain in recent years. A variety of active learning strategies have been proposed in the literature, but their effectiveness is highly dependent on the dataset and training scenario. To facilitate the comparison of existing strategies and provide…
▽ More
Since labeling medical image data is a costly and labor-intensive process, active learning has gained much popularity in the medical image segmentation domain in recent years. A variety of active learning strategies have been proposed in the literature, but their effectiveness is highly dependent on the dataset and training scenario. To facilitate the comparison of existing strategies and provide a baseline for evaluating novel strategies, we evaluate the performance of several well-known active learning strategies on three datasets from the Medical Segmentation Decathlon. Additionally, we consider a strided sampling strategy specifically tailored to 3D image data. We demonstrate that both random and strided sampling act as strong baselines and discuss the advantages and disadvantages of the studied methods. To allow other researchers to compare their work to our results, we provide an open-source framework for benchmarking active learning strategies on a variety of medical segmentation datasets.
△ Less
Submitted 2 July, 2022;
originally announced July 2022.
-
Parallel Statistical Multi-resolution Estimation
Authors:
Jan Lebert,
Lutz Künneke,
Johannes Hagemann,
Stephan C. Kramer
Abstract:
We discuss several strategies to implement Dykstra's projection algorithm on NVIDIA's compute unified device architecture (CUDA). Dykstra's algorithm is the central step in and the computationally most expensive part of statistical multi-resolution methods. It projects a given vector onto the intersection of convex sets. Compared with a CPU implementation our CUDA implementation is one order of ma…
▽ More
We discuss several strategies to implement Dykstra's projection algorithm on NVIDIA's compute unified device architecture (CUDA). Dykstra's algorithm is the central step in and the computationally most expensive part of statistical multi-resolution methods. It projects a given vector onto the intersection of convex sets. Compared with a CPU implementation our CUDA implementation is one order of magnitude faster. For a further speed up and to reduce memory consumption we have developed a new variant, which we call incomplete Dykstra's algorithm. Implemented in CUDA it is one order of magnitude faster than the CUDA implementation of the standard Dykstra algorithm. As sample application we discuss using the incomplete Dykstra's algorithm as preprocessor for the recently developed super-resolution optical fluctuation imaging (SOFI) method (Dertinger et al. 2009). We show that statistical multi-resolution estimation can enhance the resolution improvement of the plain SOFI algorithm just as the Fourier-reweighting of SOFI. The results are compared in terms of their power spectrum and their Fourier ring correlation (Saxton and Baumeister 1982). The Fourier ring correlation indicates that the resolution for typical second order SOFI images can be improved by about 30 per cent. Our results show that a careful parallelization of Dykstra's algorithm enables its use in large-scale statistical multi-resolution analyses.
△ Less
Submitted 10 March, 2015;
originally announced March 2015.
-
Real-Time Phase Masks for Interactive Stimulation of Optogenetic Neurons
Authors:
Stephan C. Kramer,
Johannes Hagemann,
D. Russell Luke
Abstract:
Experiments with networks of optogenetically altered neurons require stimulation with high spatio-temporal selectivity. Computer-assisted holography is an energy-efficient method for robust and reliable addressing of single neurons on the millisecond-timescale inherent to biologial information processing. We show that real-time control of neurons can be achieved by a CUDA-based hologram computatio…
▽ More
Experiments with networks of optogenetically altered neurons require stimulation with high spatio-temporal selectivity. Computer-assisted holography is an energy-efficient method for robust and reliable addressing of single neurons on the millisecond-timescale inherent to biologial information processing. We show that real-time control of neurons can be achieved by a CUDA-based hologram computation.
△ Less
Submitted 1 February, 2013;
originally announced February 2013.
