Autoregressive Image Diffusion: Generation of Image Sequence and Application in MRI
Abstract
Magnetic resonance imaging (MRI) is a widely used non-invasive imaging modality. However, a persistent challenge lies in balancing image quality with imaging speed. This trade-off is primarily constrained by k-space measurements, which traverse specific trajectories in the spatial Fourier domain (k-space). These measurements are often undersampled to shorten acquisition times, resulting in image artifacts and compromised quality. Generative models learn image distributions and can be used to reconstruct high-quality images from undersampled k-space data. In this work, we present the autoregressive image diffusion (AID) model for image sequences and use it to sample the posterior for accelerated MRI reconstruction. The algorithm incorporates both undersampled k-space and pre-existing information. Models trained with fastMRI dataset are evaluated comprehensively. The results show that the AID model can robustly generate sequentially coherent image sequences. In 3D and dynamic MRI, the AID can outperform the standard diffusion model and reduce hallucinations, due to the learned inter-image dependencies.
1 Introduction
Magnetic resonance imaging (MRI) is a non-invasive imaging modality widely used in clinical practice to visualize soft tissue. Despite its utility, a persistent challenge in MRI is the trade-off between image quality and imaging speed. The trade-off is influenced by the k-space (spatial Fourier domain) measurements, which traverse spatial frequency data points along given sampling trajectories. To reduce acquisition time, the k-space measurements are often undersampled, resulting in image artifacts and reduced image quality.
In recent years, deep learning-based methods have emerged to improve image reconstruction in MRI. These methods are formulated as an inverse problem building upon compressed sensing techniques [1, 2] and benefit from the learned prior information instead of hand-crafted priors [3, 4, 5]. Another successful approach involves learning an image prior parameterized by a generative neural network [6, 7], which is used as regularization on the image. Generative priors offer flexibility in handling changes in the forward model and perform well in reconstructing high-quality images from undersampled data.
Diffusion models [8, 9, 10], a class of generative models, have gained attention in recent years and are making an impact in many fields, including MRI reconstruction [11, 12]. These models learn to reverse a diffusion process that transforms random noise into structured images, producing high-quality, detailed images. Various approaches, including denoising diffusion probabilistic models (DDPMs) [10], denoising score matching [9], and continuous formulations based on stochastic differential equations (SDEs) [13], have been proposed for deriving diffusion models.
Recent studies demonstrate the effectiveness of diffusion models in accelerated MRI and their flexibility in handling various sampling patterns [11, 14, 15, 16, 17]. For example, training score-based generative models using Langevin dynamics yields competitive reconstruction results for both in-distribution and out-of-distribution data [11]. Additionally, score-based diffusion models trained solely on magnitude images can reconstruct complex-valued data [15]. Comprehensive approaches using data-driven Markov chains facilitate efficient MRI reconstruction across variable sampling schemes and enable the generation of uncertainty maps [16].
Autoregressive models are statistical models that predict the current value of a variable based on its past values, capturing temporal dependencies and patterns within the data. They are widely used in various fields such as time series analysis, signal processing, and sequence modeling. In natural language processing, autoregressive models like generative pre-trained transform (GPT) [18, 19] predict each token in a sequence based on previously generated tokens, enabling the generation of coherent and contextually relevant text. Similarly, in image modeling, autoregressive models like PixelCNN [20] and ImageGPT [21] generate images by predicting each pixel value based on previously generated pixel values, often in a left-to-right, top-to-bottom order. Instead of directly modeling pixels, which can be computationally expensive for high-resolution images, the study [22] proposes to first compress the image into a smaller representation using vector quantized variational autoencoder (VQVAE). This VQVAE learns a codebook of visually meaningful image components. Then, a transformer is applied to model the autoregressive relationship between these components, effectively capturing the global structure of the image. By predicting each image component based on previous ones, the model generates high-resolution images in a sequential manner, maintaining consistency and coherence across the entire image.
As in the clinical practice of MRI, we often involves acquiring volumetric image sequences to monitor disease progression and treatment response. Modeling these image sequences and generating realistic sequences is a challenging problem. Autoregressive models can be employed to model the joint distribution of image sequences and extract the dependencies between images. The diffusion process is effective in modelling images by treating each image independently. Therefore, we aim to combine these two models and propose autoregressive image diffusion (AID) model to generate sequences of images.
The contributions of this work are the following aspects. We present how to derive the autoregressive image diffusion training loss starting from a common diffusion loss and how to optimize loss in parallel for efficient training. We present the algorithm to sample the posterior for accelerated MRI reconstruction when using AID to facilitate the incorporation of pre-existing information. We performed experiments to evaluate its ability in generating images when different the amount of initial information is given and to validate its effectiveness in MRI reconstruction. The results show that the AID model can stably generate highly coherent image sequences even without any pre-existing information. When used as a generative prior in MRI reconstruction, the AID outperforms the standard diffusion model and reduces the hallucinations in the reconstructed images, benefiting from the learned prior knowledge about the relationship between images and pre-existing information.
2 Methods
2.1 Autoregressive image diffusion
Given a dataset consisting of multiple sequences of images, each sequence represented as , our goal is to model the joint distribution of these images. This joint distribution is autoregressively factorized into the product of conditional probabilities:
(1) |
where and the image is known. The model parameterized by is trained by minimizing the negative log-likelihood of the data:
(2) |
Sohl-Dickstein et al. [8] and Ho et al. [10] introduced the denoising diffusion probabilistic model (DDPM). This model gradually introduces fixed Gaussian noise to an observed data point using known scales , generating a series of progressively noisier values . The final noisy output follows a Gaussian distribution with zero and identity covariance matrix , containing no information about the original data point. The series of positive noise scales must be increasing, ensuring that the first noisy output closely resembles the original data , while the final value represents pure noise. We apply this process to the conditional probability in Equation 2 by adding the noise to the image independent of the position in the sequence, i.e., and are conditionally independent given . Then the transition from to is defined as:
(3) |
Here, represents the image at time , is the image at the previous time step, and denotes all images preceding at the initial time step. The parameter controls the drift and diffusion of this process. The objective is to learn to reverse this process. The reverse process is defined as:
(4) |
where and are parameterized by a neural network , taking , , and as inputs. Using the variational lower bound, the reverse process can be learned by minimizing the negative log-likelihood of the data:
(5) |
Given the initial image and that and are conditionally independent given , at an arbitrary time step is sampled from a Gaussian distribution:
(6) |
using and . The posterior distribution given and is then calculated as:
(7) |
where and .
The training objective Equation 5 is further written as minimizing the Kullback-Leibler (KL) divergence between the forward and reverse processes in Equation 4 and Equation 7, as proposed by Sohl-Dickstein et al. [8]. (See Appendix A for details.)
In practice, the approach proposed by Ho et al. [10] involves reparameterizing and predicting the noise for . The expression for is given by , with fixed. We realized this with a neural network shown in Figure 1, which predicts the noise for at each time step given . In the end, the objective function in Equation 2 for training autoregressive image diffusion is written as
(8) |
where the expectation is taken over the noise and the time step . To generate an image sequence, we begin with the noise and update it iteratively using Equation 4 with the given , following the sequence . This process yields a clean sample . Subsequently, we can sample in the same manner using the generated images , and continue this process iteratively to generate the entire sequence of images.
2.2 Architecture
To optimize the objective function in Equation 8 efficiently, ordered images are loaded as sequences of a certain length during the training phase. We take the first images as the conditioning sequence and the last images as the target sequence, as shown in Figure 1. We adopt an architecture built on an Unet [23] with capabilities of temporal-spatial conditioning (TSC), designed to process the conditioning sequence and predict the noise for the target sequence. The term "temporal" refers to conditioning in previous frames along the dimensions, while the "spatial" refers to the conditioning in the previous frame among the dimensions. Additionally, the TSC block is conditioned on the time steps of the diffusion process.
The only interaction between images in the conditioning sequence occurs during the attention operation. To maintain proper conditioning with autoregressive property, we implemented a standard upper triangular mask on the matrix of attention logits. This causal attention module is used in DiTBlock [18, 24]. The modified DiTBlock is followed by a ResNet block [25], which is a standard building block in the Unet architecture. The features output by the TSC block are then passed to the corresponding encoder block in the Unet, which process the target sequence. The change in tensor dimensions inside TSC Block is handled by the einops library111https://github.com/arogozhnikov/einops and illustrated in Figure 1.
During training, the net predicts the noise in parallel for each noisy image that is sampled from the target sequence, given the conditioning sequence. During generation of sequence, the net iteratively refines the noisy input to produce a clean image, which is then appended to the conditioning sequence.
2.3 Application in MRI inverse problem
Image reconstruction is formulated as a Bayesian problem where the posterior of image is expressed as
(9) |
Here, represents the measured k-space data, denotes the image, and is a generative prior. The minimum mean square error (MMSE) estimator for the posterior minimizes the mean square error, given by:
(10) |
2.4 Likelihood function for k-space
The image is represented as a complex matrix , where is the image size, and is a vector of complex-valued k-space samples from receive coils. Assuming circularly-symmetric normal noise with zero mean and covariance matrix , the likelihood of observing given is formulated as a complex normal distribution:
(11) |
where is the identity matrix, is the standard deviation of the noise, represents the mean, and is the length of the k-space data vector. The operator maps the image to k-space and is composed of the coil sensitivity maps , the two-dimensional Fourier transform , and the k-space sampling mask , defined as . For more details and visual understanding on the forward operator, please refer to Appendix C.
2.5 Sampling the posterior
Given a sequence of k-space , each posterior in is expressed as
(12) |
when the acquisition of is independent of the image , and are conditionally independent given . Following the Reference [8], we have
(13) |
The details for Equation 13 is in Appendix B. To sample the above posterior, the learned reverse process in Equation 4 is used, and the algorithm is constructed with two gradient updates for the log of the prior and k-space likelihood: the DDIM (Denoising Diffusion Implicit Model) reverse step proposed by Song et al. [26], and a data fidelity step derived from the likelihood function Equation 11, which are described as follows:
(14) | ||||
(15) |
where is the step size, and is the gradient of the log-likelihood of Equation 11. Then, the reconstruction of a sequence images from the undersampled k-space data is achieved by sequentially sampling the posterior in using autoregressive diffusion model. The algorithm is summarized in Algorithm 1.
3 Experiments and Results
3.1 Model training
Two autoregressive diffusion models were trained on separate datasets: one in image space and the other in latent space. The image space model was trained on brain images that are from the fastMRI training dataset, which includes T1-weighted (some with post-contrast), T2-weighted, and FLAIR images [27]. These complex images were reconstructed from fully sampled multi-channel k-space volumes, with coil sensitivity maps computed using the BART toolbox [28]. The images were then normalized to a maximum magnitude of 1, and the real and imaginary parts were treated as separate channels when input into the neural network. The number of images in each volume ranged from 12 to 16. Images were loaded without reordering and resized to 320320 pixels if they were not already that size.
The latent space model is trained with the cardiac dataset that contains cine images reconstructed by the SSA-FARY method [29]. Firstly, a VQVAE was trained on the cine images that were preprocessed similarly to images in fastMRI dataset. The cine images have a size of 256256 pixels. Then, it generates latent space for the training AID. (See the details for configuration of VQVAE in Appendix F). All the training was performed on 4 NVIDIA A100 GPUs with 80GB memory. The models were trained using the Adam optimizer with a learning rate of and a batch size of 1 for image space model and 4 for latent space model. Two models were trained for 440,000 iterations. It took around 2 hours to train brain model for 10k steps and 1.2 hours for cardiac model. The length of conditioning sequence for brain and cardiac models are 10 and 42. The network as illustrated in Figure 1 was implemented based on OpenAI’s guide diffusion codebase222https://github.com/openai/guided-diffusion. Our code will be released upon publication. We also trained a standard diffusion model, Guide, on the brain dataset for comparison. The Guide model was trained using the same hyperparameters as the AID model, except the batch size is 10. The Guide model uses the same Unet blocks as AID.
3.2 Generating sequence of images
To test different aspects of the autoregressive diffusion models, we generate the sequence of images using the following two approaches.
Retrospective sampling: This method generates a new sequence of images based on the given sequence . is sampled from Equation 4 given .
(a) |
|
|
---|---|---|
(b) |
|
Prospective sampling: A fixed-length sliding window is initialized with the given sequence . is generated from Equation 4 with the current window as conditioning. Subsequently, the window is updated by adding the newly generated and removing the earliest image . This autoregressive sampling process is repeated until the stop condition is met. We refer to this process as a warm start. In a cold start, the window is initialized with zeros, and each element in it is updated with newly generated images from the beginning to the end, after which the generation is warmed up.
In the retrospective sampling, the model generates a sequence of images that are sequentially coherent and visually similar to the conditioning sequence, as shown in Figure 2 (a). The prospective sampling generates a sequence of images that extends the initial images in the sliding window and constitutes multiple volumes, as shown in Figure 2 (b). As for a cold start, Figure 3 demonstrates the model’s ability to generate a sequence of images using black background as initial status. This shows the model’s generative capabilities from a minimal initial condition, thereby proving its robustness and flexibility. Due to the limit of space, the samples with similar quality from the model trained on the cardiac dataset are shown in Appendix D.
3.3 MRI reconstruction
The MMSE estimator in Equation 10 cannot be computed in a closed form, and numerical approximations are typically required. Once the samples from the posterior is obtained with Algorithm 1, a consistent estimate of can be computed by averaging those samples, i.e. the empirical mean of samples converges in probability to due to weak law of large numbers. The variance of those samples provides a solution to the error assessment in the reconstruction assuming the trained model is trusted. To highlight the regions with large uncertainty, we compute the pseudo-confidence intervals based on the assumption that each pixel’s intensity is normally distributed. This involves determining the standard error from the variance, then multiplying it by the t-score corresponding to a 95% confidence level.
Unfolding of aliased single-coil image: To investigate how the trained model, AID, reduces the folding artifacts in the reconstruction, we designed the single coil unfolding experiment. The single-channel k-space is simulated out of multichannel k-space data. The odd lines in k-space are retained, . Ten samples were drawn from the posterior using Algorithm 1 with parameters: . The experiment was repeated using a standard diffusion model, Guide. The results are shown in Figure 4. The AID model significantly reduces the errors in the region of folding artifacts compared to the Guide model. The mean over samples, , is highlighted with a confidence interval computed from the variance of samples. The highlighted mean image shows the reconstruction by AID is more trustworthy in the folding region. In general, the highlighted region lies in the folding region, where large errors remains, as we expected.
(a) | (b) |
---|---|
Reconstruction from undersampled data: To further investigate the model’s performance in reconstruction, we conducted experiments on 20 volumes from the fastMRI validation dataset where k-space data was retrospectively undersampled using various sampling masks. We created four types of sampling masks: random with autocalibration signal (ACS), random without ACS, equispaced with ACS, and equispaced without ACS. The undersampling factor is 12. Setting parameters: for Algorithm 1, the images were reconstructed from the undersampled k-space data using the AID and Guide as prior, respectively. We used peak-signal-noise-ratio (PSNR in dB) and normalized root-mean-square error (NRMSE) to evaluate the reconstruction quality against the reference image that is reconstructed from full k-space. The comparison of metrics across experimental conditions is illustrated in Figure 5. The proposed AID model outperforms the Guide model in terms of PSNR and NRMSE, demonstrating its superior performance in image reconstruction from undersampled k-space data. The results are consistent across different undersampling factors and sampling masks, indicating the model’s robustness and flexibility in handling various types of undersampled k-space data.
(a) | (b) |
---|---|
For the visual impression of the improvement by the AID model in reconstruction, we show the reconstructed images in Figure 6 and more of them in Appendix E. The images reconstructed using AID are more visually similar to the reference images than using Guide, even which also provides aliased-free images. Furthermore, it is worth noting that more visually notable hallucinations were introduced by the Guide model than the AID model, which means AID is more trustworthy.
4 Discussion
In this work, we propose an autoregressive image diffusion model for generating image sequences, with specific applications to accelerated MRI reconstruction. We conducted comprehensive evaluation of its performance as an image prior in reconstruction algorithms, comparing it to a standard diffusion model. Due to the learned prior information on inter-image dependencies, the proposed model outperforms the standard diffusion model across various scenarios. Our model is particularly well-suited for medical applications where image sequences are often acquired (e.g., in volumetric format) from patients in clinical practice. For instance, when different contrast images are acquired during an examination session [levac2023conditional], our model is designed to capture the relationships between these images. This enables more accurate and coherent reconstructions from undersampled k-space data using the proposed Algorithm 1. Additionally, other medical imaging tasks like dynamic MRI, multi-contrast, super-resolution, and denoising could benefit from our model’s ability by leveraging inter-image dependencies [li2024rethinking]. Furthermore, the proposed algorithm holds great promise for facilitating the incorporation pre-existing information from other imaging modalities into MRI image reconstruction. This opens up a wide range of potential medical applications, with the potential to improve patient care and reduce healthcare costs by enabling faster and more accurate image acquisition and diagnosis.
Limitation and future work: We did not evaluate the model on a common image dataset such as ImageNet or Cifar-10, nor did we compute metrics such as FID and Inception Score, which could be a limitation of our work. We plan to address these limitations in future work by running the model on a large dataset and comparing it with other state-of-the-art models. Additionally, given the model’s suitability for modeling image sequences, it is worth exploring its potential for optimizing MRI k-space acquisition strategies, as the acquisition process constitutes a sequence of operations.
5 Conclusion
The proposed autoregressive image diffusion model offers an approach to generating image sequences, with significant potential as a trustworthy prior in accelerated MRI reconstruction. In various experiments, it outperforms the standard diffusion model in terms of both image quality and robustness by taking the advantage of the prior information on inter-image dependencies.
References
- Lustig et al. [2007] M. Lustig, D. Donoho, and J. M. Pauly. Sparse MRI: The application of compressed sensing for rapid MR imaging. Magn. Reson. Med., 58(6):1182–1195, 2007.
- Block et al. [2007] K. T. Block, M. Uecker, and J. Frahm. Undersampled radial MRI with multiple coils. Iterative image reconstruction using a total variation constraint. Magn. Reson. Med., 57(6):1086–1098, 2007. ISSN 1522-2594. doi: 10.1002/mrm.21236.
- Yang et al. [2016] Yan Yang, Jian Sun, Huibin Li, and Zongben Xu. Deep ADMM-Net for Compressive Sensing MRI. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
- Hammernik et al. [2017] Kerstin Hammernik, Teresa Klatzer, Erich Kobler, Michael P. Recht, Daniel K. Sodickson, Thomas Pock, and Florian Knoll. Learning a variational network for reconstruction of accelerated MRI data. Magn. Reson. Med., 79(6):3055–3071, 2017. ISSN 1522-2594. doi: 10.1002/mrm.26977.
- Mardani et al. [2018] Morteza Mardani, Enhao Gong, Joseph Y Cheng, Shreyas S Vasanawala, Greg Zaharchuk, Lei Xing, and John M Pauly. Deep generative adversarial neural networks for compressive sensing mRI. IEEE transactions on medical imaging, 38(1):167–179, 2018.
- Tezcan et al. [2019] Kerem C Tezcan, Christian F Baumgartner, Roger Luechinger, Klaas P Pruessmann, and Ender Konukoglu. MR image reconstruction using deep density priors. IEEE transactions on medical imaging, 38(7):1633–1642, 2019. doi: 10.1109/TMI.2018.2887072.
- Luo et al. [2020] Guanxiong Luo, Na Zhao, Wenhao Jiang, Edward S. Hui, and Peng Cao. MRI reconstruction using deep bayesian estimation. Magn. Reson. Med., 84(4):2246–2261, apr 2020. doi: 10.1002/mrm.28274.
- Sohl-Dickstein et al. [2015] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256–2265. PMLR, 2015.
- Song and Ermon [2019] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 11895–11907, 2019.
- Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.
- Jalal et al. [2021] Ajil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G Dimakis, and Jon Tamir. Robust compressed sensing mri with deep generative priors. Advances in Neural Information Processing Systems, 34:14938–14954, 2021.
- Yang et al. [2023] Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 56(4):1–39, 2023.
- Song et al. [2021] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
- Güngör et al. [2023] Alper Güngör, Salman UH Dar, Şaban Öztürk, Yilmaz Korkmaz, Hasan A Bedel, Gokberk Elmas, Muzaffer Ozbey, and Tolga Çukur. Adaptive diffusion priors for accelerated mri reconstruction. Medical Image Analysis, 88:102872, 2023.
- Chung and Ye [2022] Hyung** Chung and Jong Chul Ye. Score-based diffusion models for accelerated mri. Medical image analysis, 80:102479, 2022.
- Luo et al. [2023] Guanxiong Luo, Moritz Blumenthal, Martin Heide, and Martin Uecker. Bayesian mri reconstruction with joint uncertainty estimation using diffusion models. Magnetic Resonance in Medicine, 90(1):295–311, 2023.
- Zach et al. [2023] Martin Zach, Florian Knoll, and Thomas Pock. Stable deep mri reconstruction using generative priors. IEEE Transactions on Medical Imaging, 2023.
- Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
- Radford et al. [2018] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
- Van den Oord et al. [2016] Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelcnn decoders. Advances in neural information processing systems, 29, 2016.
- Chen et al. [2020] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference on machine learning, pages 1691–1703. PMLR, 2020.
- Esser et al. [2021] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021.
- Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234–241. Springer, 2015.
- Peebles and Xie [2023] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195–4205, 2023.
- He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
- Song et al. [2020] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
- Knoll et al. [2020] Florian Knoll, Jure Zbontar, Anuroop Sriram, Matthew J Muckley, Mary Bruno, Aaron Defazio, Marc Parente, Krzysztof J Geras, Joe Katsnelson, Hersh Chandarana, et al. fastmri: A publicly available raw k-space and dicom dataset of knee images for accelerated mr image reconstruction using machine learning. Radiology: Artificial Intelligence, 2(1):e190007, 2020.
- Blumenthal et al. [2023] Moritz Blumenthal, Martin Heide, Christian Holme, Martin Juschitz, Bernhard Rapp, Philip Schaten, Nick Scholand, Jon Tamir, Christian Tönnes, and Martin Uecker. mrirecon/bart: version 0.9.00, December 2023. URL https://doi.org/10.5281/zenodo.10277939.
- Rosenzweig et al. [2020] Sebastian Rosenzweig, Nick Scholand, H Christian M Holme, and Martin Uecker. Cardiac and respiratory self-gating in radial mri using an adapted singular spectrum analysis (ssa-fary). IEEE transactions on medical imaging, 39(10):3029–3041, 2020.
Appendix A Loss function derivation
Below is a derivation of Equation 5, the reduced variance variational bound for diffusion models. This adapted from Sohl-Dickstein et al. [8] and Ho et al. [10]. We include it here only for completeness. In the forward process, and are conditionally independent given .
(16) | ||||
(17) | ||||
(18) | ||||
(19) | ||||
(20) |
Appendix B Posterior derivation
When samples drawn from the posterior started from the standard Gaussian noise, with Equation 12 we have
(21) |
for all the reverse time steps. Because
(22) |
and
(23) | ||||
(24) |
then we have
(25) |
Therefore, we have
(26) |
is a constant for evidence. Then with gradient based method, the posterior is sampled from the likelihood and the reverse process ,
Appendix C Likelihood function for k-space
The autocalibration signal (ACS) region are lines through the center of k-space, however, are fully sampled. The sensitivity of a coil is a spatial profile that describes the receiving field that induces signals in the coil. The simultaneous data acquisition, with each coil’s sensitivity corresponding to a different subregion, leads to a complete image without aliasing artifacts shown in Figure 4.
Appendix D Cardiac samples
Appendix E Reconstruction from undersampled data
Appendix F VQVAE configuration
The VQVAE is trained on the cardiac dataset to generate the latent space for the training of the autoregressive diffusion model, using the official implementation333https://github.com/CompVis/taming-transformers.git. The VQVAE is trained with the following configuration:
base_learning_rate: 4.5e-06 params: embed_dim: 3 n_embed: 8192 ddconfig: double_z: false z_channels: 3 resolution: 256 in_channels: 3 out_ch: 3 ch: 128 ch_mult: [1, 2, 4] num_res_blocks: 2 attn_resolutions: [] dropout: 0.0 lossconfig: target: losses.vqperceptual.VQLPIPSWithDiscriminator params: disc_conditional: false disc_in_channels: 3 disc_start: 30001 disc_weight: 0.8 codebook_weight: 1.0