Semantic-guided Adversarial Diffusion Model for Self-supervised Shadow Removal

Abstract

Existing unsupervised methods have addressed the challenges of inconsistent paired data and tedious acquisition of ground-truth labels in shadow removal tasks. However, GAN-based training often faces issues such as mode collapse and unstable optimization. Furthermore, due to the complex map** between shadow and shadow-free domains, merely relying on adversarial learning is not enough to capture the underlying relationship between two domains, resulting in low quality of the generated images. To address these problems, we propose a semantic-guided adversarial diffusion framework for self-supervised shadow removal, which consists of two stages. At first stage a semantic-guided generative adversarial network (SG-GAN) is proposed to carry out a coarse result and construct paired synthetic data through a cycle-consistent structure. Then the coarse result is refined with a diffusion-based restoration module (DBRM) to enhance the texture details and edge artifact at second stage. Meanwhile, we propose a multi-modal semantic prompter (MSP) that aids in extracting accurate semantic information from real images and text, guiding the shadow removal network to restore images better in SG-GAN. We conduct experiments on multiple public datasets, and the experimental results demonstrate the effectiveness of our method. <ccs2012> <concept> <concept_id>10010147.10010371.10010382.10010383</concept_id> <concept_desc>Computing methodologies Image processing</concept_desc> <concept_significance>500</concept_significance> </concept> <concept> <concept_id>10010147.10010178.10010224.10010240.10010241</concept_id> <concept_desc>Computing methodologies Image representations</concept_desc> <concept_significance>500</concept_significance> </concept> </ccs2012>

[500]Computing methodologies Image processing [500]Computing methodologies Image representations

1 Introduction

In the realm of human perception, visual, auditory, and tactile senses collectively serve as conduits through which individuals interact with the world. Among these senses, vision stands out as the most intuitive and information-rich mode of perception. Visual perception occurs when light interacts with the retina, giving rise to visual experiences. Shadows, a natural consequence of obstructed light sources, play a crucial role in sha** the visual landscape. While shadows convey valuable cues regarding object shapes and light direction, their presence often complicates the semantic understanding of images in computer vision tasks such as image segmentation[LS18] and object detection[FY19]. The shadow regions may be misclassified as objects or part of object thereof may significantly impair the accuracy and performance of these tasks. Consequently, shadow detection and removal are essential for enhancing the efficacy of computer-based visual tasks.

Figure 1: The shadow removal results of our method and other two GAN-based methods: G2R-ShadowNet[LYW^∗21] and Mask-ShadowGAN[HJFH19]. Gan-based methods have obvious shadow boundaries and artifacts.

Early shadow removal methods[AHO11] [FHLD06] [GDH13] [XSXM13]focus on exploring shadow removal in images according to different physical properties of shadows. Due to the insufficient accuracy and limitations of the underlying physical model, traditional physical model-based shadow removal algorithms are unable to effectively address shadows in complex real-world scenes.

Learning-based methods[ZLY^∗20] [CPS20][ZXF^∗22] typically train networks using paired shadow images and corresponding shadow-free images in fully supervised manner. However, inconsistencies exist in large-scale paired shadow removal datasets due to uncontrollable nature of outdoor illumination. Some methods have been proposed, which do not require paired data for training and generate supervisory signals through shadow generation using a cycle-consistent architecture. Nevertheless, the gap between synthetic and real images limits the full application of these methods in real-world shadow removal task. Although existing unsupervised methods have achieved notable results in shadow removal, GAN-based methods are susceptible to unstable optimization and mode collapse during training [XKV21] [ZCHY24]. Additionally, adversarial training alone is insufficient to fully learn the complex map** between shadow and shadow-free domains. As shown in Figure 1, the resulting shadow-free images often exhibit visible boundary artifacts and lack detailed texture restoration, leaving room for improvement in visual quality.

Figure 2: Visualization of different domains and domain transitions. The red arrow represents the previous adversarial generation process, and the green arrow represents the diffusion generation process.

In recent years, diffusion models, e.g.the Denoising Diffusion Probabilistic Model (DDPM) [HJA20], have achieved significant breakthroughs in visual generative tasks as a new branch of generative models [ZYY23] [LRJ^∗23] [ZLM^∗24] [LLK23] [LWL^∗24] [RBL^∗22] [SCC^∗22]. Owing to their superior generative capabilities, many studies have explored the application of diffusion models in image restoration tasks to enhance texture recovery. ShadowDiffusion [GWY^∗23a] proposes a diffusion sampling strategy to explicitly integrate the shadow degradation prior into the inherent iterative process of the dynamic mask sensing diffusion model. Liu [LKX^∗24] reconstructs the local illumination of the shadow region by diffusion model. Although diffusion models provide a more stable training process and are more effective in capturing the pixel distribution of images in contrast to GANs. These diffusion-based methods still require paired data for training and are encumbered by the limitations inherent to the supervised paradigm. Consequently, we aim to explore self-supervised paradigm based on diffusion models.

As shown in Figure 2, we consider first using the GAN-based method to carry out coarse shadow removal, and in this process construct paired synthetic data through a cycle-consistent structure. Then we use diffusion model to process the paired data and refine the coarse results to make them closer to our target. Through this process, the diffusion model can be used in self-supervised shadow removal. In this paper, we propose a novel semantic-guided adversarial diffusion framework , which provides a self-supervised solution to shadow removal with unpaired data. Specifically, our framework comprises two major stages: coarse processing stage and refined restoration stage, which correspond to the semantic-guided generative adversarial network (SG-GAN) and the diffusion-based restoration module (DBRM) respectively. In the first stage, SG-GAN, consists of two sub-branches. We train mask-guided generators to synthesize shadow images, assisting the shadow removal network in adversarial training. Then, we propose a multi-modal semantic prompter (MSP) that uses pre-trained visual-language models to extract features and semantic information from real images and text, enhancing shadow removal performance in both branches. In the second stage, we use paired data as the input which is constructed by the cycle-consistent structure in the branch of SG-GAN. We exploit DBRM to further refine the coarse results obtained in SG-GAN, which might contain edge artifacts and texture blurring, making them closer to the target images. This process overcomes the obstacle that the diffusion model relies on paired data for training, which otherwise makes its use in unsupervised methods challenging.

In summary, our main contributions of this work are as follows:

• We propose a semantic-guided adversarial diffusion model for self-supervised shadow removal to solve the difficulty of diffusion model processing unpaired data by constructing paired images with a cycle-consistent structure. Our methods can learn to remove shadow from unpaired data and solve the problem of obvious shadow boundary and texture detail missing in results.

• A general-purpose multi-modal semantic prompter is introduced to bridge the inherent gap between real-world shadow images and synthetic shadow images. Meanwhile, the effectiveness of this module is validated in other methods.

• We conduct extensive experiments on two public datasets. The experimental results show that our proposed method achieves competitive performance and is superior to previous unsupervised shadow removal methods.

2 Related work

2.1 Shadow removal

Traditional shadow removal methods rely on image gradients[FHLD06], illumination information[XSXM13], and image intensity regions[GDH13] to remove shadows. These early shadow removal methods often model the image without shadows or transfer color and texture features from non-shadow regions to shadow regions to achieve shadow removal. However, due to the lack of accuracy in underlying physical models, these methods usually cannot handle shadows in complex real-world scenes. With the emergence of deep learning methods, deep learning-based approaches[YYB^∗21][YZG^∗22][YWZ^∗22][FSZY24] have shown greater advantages in dealing with more complex and varied scenes.

Qu[QTH^∗17a] proposed an end-to-end deep neural network DeshadowNet for shadow removal, which predicts output shadows from three different directional information using multiple contextual architectures. Wang[WLY18] designed ST-CGAN to detect and remove shadows and created the first large-scale shadow benchmark dataset ISTD with three pairs of 1870 paired images. Due to the difficulty and inconsistency in obtaining paired shadow images in practice, to eliminate the reliance on paired data, Hu[HJFH19] proposed the Mask-ShadowGAN method based on the idea of CycleGAN, treating the shadow removal problem as an image-to-image style transfer problem. LG-ShadowNet[LYM^∗21] proposed a shadow image enhancement method based on a simple physical lighting model and image decomposition formula for shadow and pseudo-shadow removal. Liu[LYW^∗21] introduced G2R-ShadowNet for shadow removal using a training dataset constructed from a set of shadow images and their corresponding shadow masks. Most of these methods adopt adversarial learning, which results in obvious shadow boundaries and lack of texture details.

Figure 3: Overall pipeline of our method. At the coarse processing stage, the SG-GAN, which consists of S2F, F2F, and MSP, predicts the coarse shadow removal results ${\widetilde{R}}^{2}_{s}$. At the refined restoration stage, DBRM takes paired data $R_{n}$ and ${\widetilde{R}}^{2}_{s}$ from the previous stage as input, where the coarse result ${\widetilde{R}}^{2}_{s}$ is refined.

2.2 Diffusion model

Diffusion Models[SDWMG15] [HJA20] are generative models which learn distributions of real images by the Gaussian noise blurring process and the reverse denoising process. They have been successfully applied to various computer vision tasks, e.g., image super-resolution [SHC^∗22], inpainting [LDR^∗22], color harmonization [XHL23] and image restoration [GWY^∗23b] [ZCDH24].

In recent years, with the proposed diffusion model, diffusion models have also used in shadow removal tasks. For example, [MFL^∗24] enhanced the diffusion process by conditioning on a learned latent feature space from shadow-free images, while integrating noise features to avoid local optima during training. Liu[LKX^∗24] used the diffusion model to reconstruct the local illumination of the shadow region under the condition of the global illumination of the shadow image. However, these methods using the diffusion model inevitably need to use paired data to provide supervisory information for network training.

Additionally, methods combining adversarial learning with diffusion models have emerged. [KOY22] employed DDPM with adversarial learning for unsupervised vessel segmentation and has achieved promising results. However, as far as we know, no study have yet combined these approaches in the domain of shadow removal.

3 Preliminary

In this section, we briefly review the key concepts underlying SDE-based diffusion models and outline the process of generating samples with reverse-time SDEs. Let $p_{0}$ denote the initial data distribution, and $t\in[0,T]$ denote the continuous time variable. We consider a diffusion process ${x(t)}_{t=0}^{T}$ defined by an SDE of the form:

$$dx=f(x,t)dt+g(t)dw,\quad x(0)\sim p_{0}(x),$$ (1)

where $f$ and $g$ are the drift and dispersion functions, respectively, $w$ is a standard Wiener process, and $x(0)\in\mathbb{R}^{d}$ is the initial condition. Typically, the terminal state $x(T)$ follows a Gaussian distribution with fixed mean and variance. The idea is to design such an SDE that gradually transforms the data distribution into Gaussian noise[DBMH^∗22, LZB^∗22].

We can reverse the process to sample data from noise by simulating the SDE backward in time[SSDK^∗20]. [And82] shows that a reverse-time representation of the SDE (1) is given by:

$$dx=\left[f(x,t)-g(t)^{2}\nabla_{x}\log p_{t}(x)\right]dt+g(t)d\hat{w},$$ (2)

where $x(T)\sim p_{T}(x)$. Here, $\hat{w}$ is a reverse-time Wiener process, and $p_{t}(x)$ is the marginal probability density function of $x(t)$ at time $t$. Since the score function $\nabla_{x}\log p_{t}(x)$ is generally intractable, SDE-based diffusion models approximate it by training a time-dependent neural network $s_{\theta}(x,t)$ using a score matching objective[SSDK^∗20].

4 Proposed method

Figure 4: Real shadow images from ISTD and corresponding synthetic shadow images obtained from our generator $G_{s}$, the histograms show the inconsistencies in the intensity distribution between them.

4.1 Overall framework

To enhance the effectiveness of shadow removal networks and address edge artifacts and blur texture in unsupervised methods, we propose a semantic-guided adversarial diffusion model for self-supervised shadow removal. Figure 3 illustrates the overall network architecture of our method, which consists of two stages: coarse processing stage and refined restoration stage.These two stages are composed of the semantic-guided generative adversarial network (SG-GAN) and the diffusion-based restoration module (DBRM) respectively. In SG-GAN, we use a general-purpose multi-modal semantic prompter (MSP) to extract semantic information by the pre-trained clip model helps the network in better restoration. In terms of the structure of SG-GAN, it can be divided into two branches which utilizes a set of unpaired shadow images, shadow-free images, and shadow masks as inputs for training. It includes shadow generation and removal generators $G_{s}$ and $G_{r}$, as well as shadow and shadow-free image discriminators $D_{s}$ and $D_{r}$. However, SG-GAN relies on discriminators and consistency constraints, which are insufficient to achieve optimal results. The outcomes after shadow removal still contain undesirable noise.Therefore, in refined restoration stage, DBRM uses the coarse shadow removal results obtained from SG-GAN to construct paired data with the clean inputs for network training. By the powerful generative capabilities of the diffusion model, DBRM further refines the coarse results to remove artifacts and enhance texture details.

4.2 Multi-modal semantic prompter

Existing unsupervised methods [LYW^∗21] [HJFH19] primarily rely on generated shadow images (synthetic data) to train shadow removal networks without the guidance from real prior information. However, as shown in Figure 4, even though the real shadow image and the synthesized shadow image look very similar, data distribution difference still exists in. The shadow removal network trained with synthetic images might reduce its effectiveness on real-world shadow images.

Useful prompts can help to correct task networks toward better performance. To minimize the impact of this gap, we designed a multi-modal semantic prompter (MSP). As shown in Figure 3, MSP extracts features using the image encoder $C_{image}$ and text encoder $C_{text}$ from a pre-trained CLIP model [RKH^∗21]. The prior information extracted by the image encoder is fused with the features extracted by $G_{r}$ in residual blocks through semantic fusion blocks (SFB), while the text features extracted by the text encoder are used to define a clip loss (introduced in Sec.4.3) to further constrain the image recovered by $G_{r}$.

SFB aims to better perceive prior information, using this more reliably perceived content to assist $G_{r}$ in restoration. It also controls the propagation of perceived prior information, enabling it to adaptively learn more useful features in $G_{r}$ for better restoration. Given a recovery tensor $X_{k-1}$ from the $(k-1)_{th}$ residual block in $G_{r}$ and a semantic tensor $Y$ extracted by $C_{image}$, they are first fused through a cross attention mechanism to obtain $Z_{k-1}$:

$$Z_{k-1}=A_{c-attn}(X_{k-1},Y)$$ (3)

where $A_{c-attn}$ refers to the cross attention. Then, using a $1\times 1$ convolution and a gated control function sigmoid, the input for the next residual block and can be obtained:

$$\mathcal{S}(X_{k-1},Y)=\sigma(W_{p}(Z_{k-1}))\odot X_{k-1}+X_{k-1}$$ (4)

where $\sigma(\cdot)$ represents the sigmoid function and $W_{p}(\cdot)$ means the 1 × 1 point-wise convolution.

4.3 Semantic-guided generative adversarial network

SG-GAN consists of two branches: shadow to shadow-free (S2F) and shadow-free to shadow-free (F2F), of which the input is a real-world image and a shadow mask. The input shadow mask is a binary map where 0 represents non-shadow (black) regions and 1 represents shadow (white) regions.

S2F and F2F sub-branches In S2F, a shadow mask image $M^{1}$ consisting entirely of zeros and a real shadow image $R_{s}$ are used as inputs to the generator $G_{s}$ to produce an image ${\widetilde{R}}^{1}_{s}$ without additional shadows:

$${\widetilde{R}}^{1}_{s}=G_{s}(R_{s},M^{1})$$ (5)

The generated shadow image ${\widetilde{R}}^{1}_{s}$ is then transformed into a shadow-free image ${\widetilde{R}}^{1}_{n}$ using the generator $G_{r}$. Subsequently, the discriminator $D_{r}$ is employed to distinguish whether ${\widetilde{R}}_{n}$ is a genuine shadow-free image:

$${\widetilde{R}}^{1}_{n}=G_{r}({\widetilde{R}}^{1}_{s}),D_{r}({\widetilde{R}}^{1}_{n})=real/fake?$$ (6)

In F2F, a real shadow-free image $R_{n}$ and a shadow mask $M^{2}$ which indicates the shadow region are required as input. We use the method described in [LYW^∗21] to generate the shadow mask $M^{2}$. Then, with above inputs, generator $G_{s}$ generates a shadow image ${\widetilde{R}}^{2}_{s}$ to deceive the discriminator $D_{s}$, making it difficult for $D_{s}$ to discern whether it is a genuine shadow image:

$${\widetilde{R}}^{2}_{s}=G_{s}(Rn,M^{2}),D_{s}({\widetilde{R}}^{2}_{s})=real/fake?$$ (7)

Then the synthesized ${\widetilde{R}}^{2}_{s}$ as input to generator $G_{r}$ for shadow removal to obtain a coarse shadow-free image ${\widetilde{R}}^{2}_{n}$.

In the above process, we find that the input to generator $G_{r}$ is always the synthesized shadow image generated by $G_{s}$. In order to improve the shadow removal performance of $G_{r}$, we add MSP in $G_{r}$ to reduce the impact of synthesized data in both S2F and F2F.

The architectures of $G_{r}$ and $G_{r}$ are same, which mainly follow the generator proposed by Hu [HJFH19]. They consist of three convolution layers with a stride of 2, followed by nine residual blocks for feature extraction, and finally three deconvolution layers to upsample feature map. Instance normalization is applied after each convolution operation. For the discriminators $D_{r}$ and $D_{s}$ we adopt the architecture proposed in PatchGAN [IZZE17].

Loss function In S2F, we use identity loss to make the generated shadow image ${\widetilde{R}}^{1}_{s}$ close to the input shadow image $R_{s}$ :

$$\mathcal{L}_{\textit{identity}}(G_{s})=\mathbb{E}_{R_{s}\sim p(R_{s})}\left[\|G_{s}(R_{s},M^{1})-R_{s}\|_{1}\right]$$ (8)

For generator $G_{r}$ and its discriminator$D_{r}$, the objective function is optimized as follows:

$$\mathcal{L}_{\textit{GAN}_{r}} =\mathbb{E}_{R_{n}\sim p(R_{n})}\left[\log(D_{r}(R_{n}))\right]$$ (9)

$$\quad+\mathbb{E}_{R_{s}\sim p(R_{s})}\left[\log(1-D_{r}(G_{r}(\widetilde{R}^{1}_{s})))\right]$$

We incorporated the MSP into $G_{r}$. This process introduces a clip loss to help $G_{r}$ better remove shadows. As shown in Figure 3, the clip loss constraint between the output $\mathcal{S}(X,C_{image}(R_{s})$ of the last SFB and the semantic features extracted from input text $T$ by clip text encoder $C_{text}$ is defined as follows:

$$\mathcal{L}_{\textit{clip}_{S2F}}=\frac{e^{cos\left(\mathcal{S}(X,C_{image}(R_{s}),C_{text}(T))\right)/\tau}}{e^{cos\left(\mathcal{S}(X,C_{image}(R_{s}),C_{text}(T)\right)/\tau}+e^{\frac{1}{\tau}}}$$ (10)

where $X$ is the output of the penultimate residual block in $G_{r}$, $\tau$ denotes the temperature parameter which we set to 0.5 in experiment. In F2F, the adversarial loss for generator $G_{s}$ and discriminator $D_{s}$ is formulated as:

$$\mathcal{L}_{\text{GAN}_{s}} =\mathbb{E}_{R_{s}\sim p(R_{s})}\left[\log(D_{s}(R_{s}))\right]$$ (11)

$$+\mathbb{E}_{R_{n}\sim p(R_{n})}\left[\log(1-D_{s}(G_{s}(R_{n},M^{2})))\right]$$

MSP is also used in F2F, so we define a loss similair to $\mathcal{L}_{\textit{clip}_{S2F}}$ that makes the shadow removal result close to the input text $T$:

$$\mathcal{L}_{\textit{clip}_{F2F}}=\frac{e^{cos\left(\mathcal{S}(X,C_{image}(R_{s}),C_{text}(T)\right)/t}}{e^{cos\left(\mathcal{S}(X,C_{image}(R_{s}),C_{text}(T))\right)/t}+e^{\frac{1}{t}}}$$ (12)

To ensure that ${\widetilde{R}^{2}}_{n}$ is similar to the original input real shadow-free image $R_{n}$, we use cycle consistency loss to optimize the map** functions in $G_{s}$ and $G_{r}$:

$$\mathcal{L}_{\textit{cycle}}(G_{s},G_{r})=\mathbb{E}_{R_{n}\sim p(R_{n})}\left[\|G_{r}(G_{s}(R_{n},M^{2}))-R_{n}\|_{1}\right]$$ (13)

To further emphasize that the shadow region in ${\widetilde{R}}^{2}_{n}$ guided by the shadow mask matches the content in the input image $R_{n}$, we apply a shadow loss as follow:

$$\mathcal{L}_{\textit{shadow}}(G_{s},G_{r})=\mathbb{E}_{R_{n}\sim p(R_{n})}\left[\|M\odot(G_{r}(G_{s}(R_{n},M^{2}))-R_{n})\|_{1}\right]$$ (14)

In summary, the total loss in SG-GAN is defined as:

$$\mathcal{L}= \ \mathrm{\lambda}_{1}\mathcal{L}_{\text{identity}}+\mathrm{\lambda}_{2}(\mathcal{L}_{\text{GAN}_{s}}+\mathcal{L}_{\text{GAN}_{r}})$$ (15)

$$\ +\mathrm{\lambda}_{3}(\mathcal{L}_{\textit{clip}_{S2F}}+\mathcal{L}_{\textit{clip}_{F2F}})+\mathrm{\lambda}_{4}(\mathcal{L}_{\text{cycle}}+\mathcal{L}_{\text{shadow}})$$

where $\lambda_{1}$, $\lambda_{2}$, $\lambda_{3}$, and $\lambda_{4}$ are weights balancing different loss terms. In our experiment, we set $\lambda_{1}$, $\lambda_{2}$, $\lambda_{3}$ , and $\lambda_{4}$ to 5, 1, 0.2, and 10, respectively.

4.4 Diffusion-based restoration module

Figure 5: Visualisation comparisons results on five real-world challenging samples from the ISTD dataset.

At this stage, we employ IR-SDE [QTH^∗17b] as the diffusion framework for diffusion model. It allows us to better understand and control the image generation process by simulating the image degradation process. The key idea of our diffusion model is to combine a mean-reverting SDE with a maximum likelihood objective for neural network training, which naturally transforms high-quality images into degraded low-quality images regardless of the complexity of the degradation.

Diffusion framework According to [QTH^∗17b], the forward process of a mean-reverting SDE is defined as:

$$dx_{t}=\theta_{t}(\mu-x_{t})dt+\sigma_{t}dw,$$ (16)

where $\theta_{t}$ and $\sigma_{t}$ are time-varying positive parameters representing the mean regression rate and stochastic volatility, respectively. Here, $\mu$ is the state mean, and $w$ denotes Brownian motion.

In our model, a coarse shadow removal result ${\widetilde{R}}^{2}_{n}$ is obtained in the second branch of SG-GAN. We use ${\widetilde{R}}^{2}_{n}$ and the corresponding input clean image $R_{n}$ to construct paired data (LQ and GT) for our DBRM. Given paired images, we set $R_{n}$ as the initial state $x_{0}$ and ${\widetilde{R}}^{2}_{n}$ as $\mu$, with a fixed noise level $\lambda$.

By ensuring $\sigma_{t}^{2}/\theta_{t}=2\lambda^{2}$ for all $t$, the forward process solution is:

$$x(t)={\widetilde{R}}^{2}_{n}+(x(s)-{\widetilde{R}}^{2}_{n})e^{-\bar{\theta}{s:t}}+\int{s}^{t}\sigma_{z}e^{-\bar{\theta}{z:t}}dw(z),$$ (17)

where $\bar{\theta}{s:t}=\int_{s}^{t}\theta_{z},dz$. The transition kernel is:

$$p(x(t)\mid x(s))=\mathcal{N}(x(t)\mid m_{s:t}(x(s)),v_{s:t}),$$ (18)

where $m_{s:t}$ is the mean and $v_{s:t}$ is the variance. The forward SDE iteratively transforms the GT image $R_{n}$ into the LQ image $\widetilde{R}^{2}_{n}$ with added noise:

$$p(x_{t}\mid x_{t-1})=\mathcal{N}\left(x_{t}\mid m_{t-1}(x_{t-1}),v_{t-1}\right).$$ (19)

A notable property of this process is that noisy data $x_{t}$ can be sampled from $x_{0}$ in closed form:

$$p_{t}(x)=\mathcal{N}\left(x(t)\mid m_{t}(x),v_{t}\right)$$ (20)

with $m_{t}=\widetilde{R}^{2}_{n}+(x(0)-\widetilde{R}^{2}_{n})e^{-\bar{\theta}t}$, $v_{t}=\lambda^{2}\left(1-e^{-2\bar{\theta}t}\right)$ and $\bar{\theta}_{t}$ referring to $m_{0:t}$, $v_{0:t}$ and $\bar{\theta}_{0:t}$, respectively.

The reverse-time representation from [And82] is:

$$dx=[\theta_{t}({\widetilde{R}}^{2}_{n}-x)-\sigma_{t}^{2}\nabla_{x}\log p_{t}(x)]dt+\sigma_{t}dw,$$ (21)

where $\nabla_{x}\log p_{t}(x)$ is the score of the marginal distribution at time $t$. Given the GT image $R_{n}$, we compute the score function as:

$$\nabla_{x}\log p_{t}(x)=-\frac{x(t)-m_{t}}{v_{t}}.$$ (22)

We then train a conditional time-dependent neural network $\tilde{\epsilon}_{\phi}(x_{t},\widetilde{R}^{2}_{n},t)$ to estimate noise. Sampling $x_{t}$ according to $x_{t}=m_{t}(x)+\sqrt{v_{t}}\ \epsilon_{t}$, where $\epsilon_{t}\sim\mathcal{N}(0,I)$. The score can then be directly computed from the noise:

$$\nabla_{x}\log p_{t}(x)=-\frac{\epsilon_{t}}{\sqrt{v_{t}}}.$$ (23)

Figure 6: Visualisation comparisons results on five real-world challenging samples from the AISTD dataset.

Network architecture As shown the architecture of DBRM in Figure 3, our noise prediction network is based on nonlinear activated free block (NAF block) [CCZS22]. NAF block replaces nonlinear activations (such as ReLU and GELU) with a SimpleGate unit. Given an input, SimpleGate splits it into two features along the channel dimension and then uses a linear gate to compute the output. The SimpleGate unit is added after the depth-wise convolution and between the two fully connected layers. Meanwhile, we add an additional multi-layer perceptual processing time embedding for each NAF block.

Loss function An alternative maximum likelihood objective aims to find the optimal trajectory $x_{1:T}$ given the high-quality image $x_{0}$, stabilizing training and recovering more accurate images. Following IR-SDE [QTH^∗17b], we train our prediction network with a maximum likelihood loss which specifies the optimal reverse path $x^{*}_{t-1}$ for all times:

$$x^{*}_{t-1}= \frac{1-e^{-2\bar{\theta}_{t-1}}}{1-e^{-2\bar{\theta}_{t}}}e^{-\theta^{\prime}_{t}}(x_{t}-\widetilde{R}^{2}_{n})$$ (24)

$$+\frac{1-e^{-2\theta^{\prime}_{t}}}{1-e^{-2\bar{\theta}_{t}}}e^{-\bar{\theta}_{t-1}}(R_{n}-\widetilde{R}^{2}_{n})+\widetilde{R}^{2}_{n}$$

where $\theta^{\prime}_{i}=\int_{i-1}^{i}\theta_{t}\,dt$. Then, we define the maximum likelihood loss :

$$\mathcal{L}_{\textit{diff}}=\sum_{t=1}^{T}\gamma_{t}\mathbb{E}\left[\left\|x_{t}-(dx_{t})\tilde{\epsilon}_{\phi}-{x^{*}_{t-1}}\right\|\right]$$ (25)

where $\gamma_{1},...,\gamma_{T}$ are positive weights and ${\{x_{t}\}}^{T}_{t=0}$ denotes the discretization of the diffusion process. $(dx)\tilde{\epsilon}_{\phi}$ denotes the reverse-time SDE in Eq.17 and its score is predicted by the noise network $\tilde{\epsilon}_{\phi}$. $x_{t}-(dx_{t})\tilde{\epsilon}_{\phi}$ is the reverse $x_{t-1}$. Once trained, we can use the network $\tilde{\epsilon}_{\phi}$ to generate high-quality images by sampling a noisy state $x_{T}$ and iteratively solving the Eq.17 with a numerical scheme.

5 Experiment

Scheme Methods Shadow Region Non-shadow Region All Image RMSE$\downarrow$ PSNR$\uparrow$ SSIM$\uparrow$ RMSE$\downarrow$ PSNR$\uparrow$ SSIM$\uparrow$ RMSE$\downarrow$ PSNR$\uparrow$ SSIM$\uparrow$ LPIPS$\downarrow$ Supervised DHAN 7.53 35.82 0.989 5.33 30.95 0.971 5.68 29.09 0.953 0.027 Sp+M-Net 7.13 35.08 0.984 3.16 36.38 0.979 3.92 31.89 0.953 0.079 AEFNet 7.91 34.71 0.975 5.51 28.61 0.880 5.88 27.19 0.945 0.045 ShadowDiffusion 4.10 40.06 0.996 4.18 33.00 0.973 4.16 32.20 0.967 - Unsupervised Mask-ShadowGAN 13.00 30.53 0.977 6.07 28.86 0.960 6.96 25.72 0.925 0.064 DC-ShadowNet 11.89 31.27 0.969 7.84 27.20 0.910 7.03 25.51 0.865 0.104 LG-ShadowNet 10.72 31.53 0.979 6.30 27.67 0.967 6.08 26.62 0.936 0.056 S3R-Net 12.16 - - 6.38 - - 7.12 - - - Ours 10.53 32.68 0.966 5.54 30.96 0.970 6.29 27.94 0.930 0.038 Shadow image 32.10 22.40 0.936 7.09 27.32 0.976 10.89 20.56 0.893 0.116

Table 1: Quantitative comparison results of our methods with the state-of-the-art methods on ISTD dataset. The best and second performances for supervised learning and unsupervised learning methods are highlighted in Bold and underlined, respectively. ‘-’ denotes the results are not publicly available.

Scheme	Methods	Shadow Region			Non-shadow Region			All
		RMSE$\downarrow$	PSNR$\uparrow$	SSIM$\uparrow$	RMSE$\downarrow$	PSNR$\uparrow$	SSIM$\uparrow$	RMSE$\downarrow$	PSNR$\uparrow$	SSIM$\uparrow$	LPIPS$\downarrow$
Supervised	DHAN	9.57	32.92	0.987	7.41	27.15	0.972	7.77	25.66	0.954	0.026
	Sp+M-Net	5.91	37.60	0.99 0	2.99	36.02	0.976	3.46	32.94	0.962	0.085
	AEFNet	6.55	36.04	0.978	3.77	31.16	0.892	4.22	29.45	0.861	0.046
	ShadowDiffusion	4.60	40.13	0.997	2.74	36.48	0.979	2.91	35.66	0.974	-
Unsupervised	Mask-ShadowGAN	11.28	31.50	0.981	3.90	32.63	0.967	4.97	28.11	0.936	0.063
	DC-ShadowNet	10.81	32.15	0.978	3.46	35.50	0.974	4.61	29.09	0.940	0.051
	LG-ShadowNet	9.90	32.42	0.982	3.18	34.01	0.976	4.25	29.31	0.947	0.049
	S3R-Net	12.86	-	-	4.43	-	-	5.71	-	-	-
	Ours	9.48	33.63	0.968	3.09	36.01	0.978	4.06	31.09	0.940	0.034
	Shadow image	36.95	20.83	0.927	2.42	37.46	0.985	8.40	20.46	0.894	0.117

Table 2: Quantitative comparison results of our methods with the state-of-the-art methods on AISTD dataset.

5.1 Datasets and evaluation metrics

Dataset We utilize two state-of-the-art shadow removal datasets: ISTD [WLY18] and AISTD [LS19]. ISTD comprising 1870 triplets of shadow, shadow-free images and shadow masks, with 1330 triplets for training and 540 triplets for testing. AISTD is the adjust dataset which further corrects the color inconsistency problem of images from the ISTD.

Evaluation metrics In our experiments, we follow [LS21] calculate root mean square error (RMSE) in the LAB color space , and employ the structure similarity (SSIM), peak signal-to-noise ratio (PSNR), and learned perceptual image patch similarity (LPIPS) as evaluation metrics for comparisons. Generally, higher PSNR and SSIM values are preferred, while lower RMSE and LPIPS values indicate better performance. We provide the metrics measured on the shadow region, non-shadow region and whole image for reference.

5.2 Experimental settings

We implement our methods using PyTorch [PGM^∗19] and a single NVIDIA GeForce GTX 3090 GPU. At first stage, we initialise our SG-GAN using a Gaussian distribution with a mean of 0 and a standard deviation of 0.02. We employ the Adam optimiser to train our network with the first and the second momentum setting to 0.5 and 0.999, respectively. We train the whole model for 200 epochs and the base learning rate is set to $2\times 10^{-4}$ for the first 100 epochs and then we apply a linear decay strategy to decrease it to 0 for the rest epochs. Additionally, horizontal flip** and random crop** strategy purposed in [LYW^∗21] is applied to the training data for data augmentation. The network training involves both two branch, and they impact each other. At second stage, for training the diffusion model, we fix the noise level at 50 and set the number of diffusion denoising steps to 100. The batch sizes are set to 8 and the training patches are $256\times 256$ pixels. We use the Lion optimizer [CLH^∗24] with $\beta_{1}$ = 0.9 and $\beta_{2}$ = 0.99. The initial learning rate is set to $3\times 10^{-5}$ and decayed to 1e-7 by the Cosine scheduler. The noise level is fixed to 50 and the number of diffusion denoising steps is set to 100. We train DBRM for 400 000 iterations, which takes for about 4 days on the GPU.

5.3 Comparison with the state-of-the-arts

In this subsection, we compare our full model on the ISTD and AISTD datasets with several state-of-the-art methods, including supervised methods which are trained with paried shadow and shadow-free images: DHAN [CPS20], SP+M-Net [LS19], AEFNet [FZG^∗21] and ShadowDiffusion [GWY^∗23a]; unsupervised methods training without paired shadow and shadow-free images: G2R-ShadowNet [LYW^∗21], Mask-ShadowGAN [HJFH19], DC-ShadowNet [JST21], LG-ShadowNet [LYM^∗21] and S3R-Net [KMP^∗24]. All of the shadow removal results by the competing methods are quoted from the original papers or reproduced using their official implementations.

Figure 7: Visual comparisons result for ablation study on the use of each loss terms in SG-GAN.

Table 1 shows the quantitative results on the ISTD dataset. The supervised methods share the same type of training data, including shadow and shadow-free image pairs. They learn the map** from shadow images to shadow-free images based on training pairs. However, their performance might be largely degraded when extended to some unseen scenes. Our method achieves results in shadow-free region and all image that are comparable to other deep neural networks trained on paired images, and in some metrics, it even surpasses some supervised methods. For instance, LPIPS metric of our results for the all image is better than those of Sp+M-net [LS19] and AEFNet [FZG^∗21]. Moreover, several unsupervised methods, such as Mask-ShadowGAN [HJFH19], LG-ShadowNet [LYM^∗21], and DC-ShadowNet[JST21], we can see that our method significantly outperforms these three methods. Our method outperforms LG-ShadowNet [LYM^∗21] which is suboptimal on most metrics, especially the results for non-shadow region and the all image improve by about 2.1dB and 2.2dB in PSNR, respectively, except a SSIM and RMSE value is slightly lower. Additionally, our LPIPS and PSNR are clearly better than all the unsupervised methods in the table. When compared to the state-of-the-art unsupervised method S3R [KMP^∗24], our method performs better on RMSE. Note that the code of S3R [KMP^∗24] is not publicly available, so we could only report their results on the dataset used in their published paper. Qualitative results on four challenging samples drawn from the testing set of ISTD are shown in Figure 5. Compared with other methods, our method can produce more realistic results with less artifacts and better preserve the texture details occluded by shadows. Moreover, the colour in the shadow region is more consistent with the surrounding area using our method.

We also retrain and test our model on the AISTD dataset, with quantitative results presented in Table 2. The comparative results are shown in Table 2 and Figure 6. Among the unsupervised methods, just like before our method performs the best on AISTD, followed by LG-ShadowNet [LYM^∗21]. In Figure 6, the outputs generated by our competitors exhibit sharp shadow edges, whereas our results have significantly smoother shadow boundaries. Additionally, our samples show a noticeably sharper appearance in shadow region. We believe we have demonstrated that our approach can produce the most visually pleasing results to the human eye.

Methods	RMSE$\downarrow$	PSNR$\uparrow$	SSIM$\uparrow$	LPIPS$\downarrow$
w/o DBRM	7.27	25.44	0.935	0.054
w/o S2F	10.36	21.45	0.892	0.122
w/o MSP	7.99	24.89	0.902	0.065
Ours	6.29	27.94	0.930	0.038

Table 3: Ablation study on the choices of different component for our method on ISTD testing set.

Figure 8: Visual comparisons result of ablation study on the use of each component for our method.

5.4 Ablation study

To validate the efficacy of each pivotal component within our proposed method, we trained and evaluated several model variants on the ISTD dataset. First, We propose to utilize the DBRM to suppress the artifacts of shadow removal results. So we validate DBRM by removing it from our complete model, retaining only SG-GAN. Additionally, we train SG-GAN without the S2F and MSP components to evaluate their individual contributions. The quantitative results are reported in Table 3. Subsequently, we conduct ablation studies on loss functions we proposed. These studies involved training SG-GAN without specific loss terms to demonstrate the effectiveness of each loss function. The quantitative results are reported in Table 4.

Methods	RMSE$\downarrow$	PSNR$\uparrow$	SSIM$\uparrow$	LPIPS$\downarrow$
w/o $\mathcal{L}_{GAN_{s}}$	8.75	24.34	0.838	0.119
w/o $\mathcal{L}_{GAN_{r}}$	10.05	21.64	0.900	0.115
w/o $\mathcal{L}_{\textit{cycle}}$	8.40	24.58	0.825	0.126
w/o $\mathcal{L}_{\textit{shadow}}$	8.03	24.79	0.877	0.111
w/o $\mathcal{L}_{\textit{identity}}$	13.54	19.39	0.845	0.182
w/o $\mathcal{L}_{\textit{clip}}$	7.41	25.07	0.927	0.068
SG-GAN	7.27	25.44	0.935	0.054

Table 4: Ablation study on the choices of the loss functions for our SA-GAN on ISTD testing set.

Methods	RMSE$\downarrow$	PSNR$\uparrow$	SSIM$\uparrow$	LPIPS$\downarrow$
Ours w/o MSP	6.40	27.16	0.917	0.046
Ours	6.29	27.94	0.930	0.038
Msak-ShadowGAN	6.96	25.72	0.925	0.064
Mask-ShadowGAN w/ MSP	6.73	26.19	0.933	0.060
G2R-ShadowNet	7.33	25.72	0.941	0.047
G2R-ShadowNet w/ MSP	6.83	25.87	0.952	0.053

Table 5: Ablation studies on the effectiveness of the MSP on the ISTD testing set.

As show in Table 3, we observe that performances of our method reduces across all metrics except SSIM when DBRM is omitted. Comparing row 2 to row 4, we find that the S2F branch is crucial, providing substantial performance gains in terms of all the metrics. Then, when MSP is excluded, there is also a certain drop in performance. In Table 4, we find that $\mathcal{L}_{GAN_{s}}$ is important and brings performance improvement. Rows 2 and 5 show a significant decline in SG-GAN’s performance without $\mathcal{L}_{GAN_{r}}$ and $\mathcal{L}_{identity}$, especially for the $\mathcal{L}_{identity}$. Then, when $\mathcal{L}_{\textit{clip}}$, $\mathcal{L}_{cycle}$, and $\mathcal{L}_{shadow}$ is respectively removed from the total loss, the performance of SG-GAN shows a slight decrease. As shown in Figure 8 and Figure 7, the qualitative results are generally consistent with the aforementioned quantitative results in demonstrating the effectiveness of each component. Compared to the model trained with all components, other variants trained with subsets of these components may exhibit noticeable artifacts in the results. For instance, the result of SG-GAN w/o $\mathcal{L}_{\textit{shadow}}$ in Figure 7 and the result of w/o S2F in Figure 8, artifact in the shadow region is very prominent.

5.5 Effectiveness of general-purpose MSP

We propose a general MSP to mitigate the impact of synthetic images on the performance of the shadow removal network. We experimentally validate the effectiveness of MSP on the ISTD dataset. We conduct experiments not only on our own model but also by integrating our proposed MSP into two unsupervised methods, G2R-ShadowNet [LYW^∗21] and Mask-ShadowGAN[HJFH19], which require synthetic shadows for shadow removal network training.

Quantitative results are shown in Table 5. The results clearly demonstrate that the performance of our method and the other two methods has shown varying degrees of improvement after integrating MSP, particularly in the case of G2R-ShadowNet [LYW^∗21]. It is noteworthy that both our method and Mask-ShadowGAN [HJFH19] are trained in RGB space, whereas G2R-ShadowNet [LYW^∗21] is trained in LAB space. This outcome indicates the universal applicability of our MSP across different color spaces.

6 Conclusion

In summary, we propose a novel adversarial diffusion framework for self-supervised shadow removal. Our method consists of two stages: a coarse processing stage and a refined restoration stage, with the main networks being SG-GAN and DBRM, respectively. In SG-GAN, shadows are first generated and then removed, constructing paired training sets used for refinement in DBRM. Additionally, we design a general-purpose MSP module to mitigate the impact of synthetic data on network performance. The coarse results are refined by the powerful generative capabilities of DBRM, restoring texture details and edge artifacts in shadow regions. Extensive experiments demonstrate the effectiveness of our proposed method, showing competitive performance on the ISTD and AISTD datasets compared to other state-of-the-art techniques.