IDA-UIE: An Iterative Framework for Deep Network-based Degradation Aware Underwater Image Enhancement

The model is based on the principles of linear superposition and the water medium modeling defined in the Jaffe–McGlamery model [2]. The irradiance entering the camera is a linear combination of three distinct components: the direct component ($E_{d}$), the forward-scattered component ($E_{f}$), and the backscatter component ($E_{b}$). The total irradiance ($E_{T}$) can be expressed as follows:

$$E_{T}=E_{d}+E_{f}+E_{b}$$

(1)

The direct component, denoted as $E_{d}$, refers to the light that is reflected by an object and reaches the camera without undergoing any scattering. Forward scatter, represented by $E_{f}$, occurs when the light reflected from an object scatters in its direction before reaching the camera. In contrast, backscatter happens when the light scatters directly towards the camera after reflecting off particles in the water. These models are often used for image restoration but require high-speed computations and longer execution times.

$E_{d}$ signifies the light that is directly reflected by the object without any scattering in the water. This component is particularly beneficial for underwater imaging and can be expressed as:

$$E_{d}(x,y)=E(x,y)e^{-cd(x,y)}$$

(2)

The expression $E(x,y)$ represents the irradiance at position $(x,y)$. The total attenuation coefficient (c) of the medium quantifies the combined effects of scattering and absorption on light loss within the medium. The variable $d(x,y)$ denotes the distance between the object and the camera. Furthermore, $E_{f}$ refers to the forward scatter component, which is light reflected by an object and scattered at a small angle before reaching the camera:

$$E_{f}(x,y)=E_{d}(x,y)*g(x,y)$$

(3)

To denote the convolution operator, the symbol $\ast$ is used, and $g(x,y)$ represents the point spread function (PSF). To avoid the mathematically complex issue of solving the deconvolution through PSF estimation, researchers typically assume that the underwater scene is close to the camera, thereby neglecting the impact of forward scattering.

$E_{b}$ represents the backscattered light reflected by particles in the water. This component does not include light from the object itself, as it is primarily caused by the scattering of floating particles. $B_{\infty}$ denotes the underwater background light.

$$E_{b}(x,y)=B_{\infty}(\lambda)(1-e^{-cd(x,y)})$$

(4)

The presence of suspended particles in water can be mathematically modeled using the radiative transfer equation, which describes the interaction of light with matter. For underwater images, this equation can model the propagation of light through the water column, including the effects of scattering and absorption by suspended particles [46].

A common approach to enhancing underwater images involves using a dehazing algorithm that estimates the transmission map of the image. This map represents the fraction of light that has successfully transmitted through the water column. The transmission map can be estimated using the following equation:

$$t(x)=e^{(-\beta d(x))}$$

(5)

where $t(x)$ is the transmission at pixel $x$, $d(x)$ is the distance between the camera and pixel $x$, and $\beta$ is the scattering coefficient of the water. The scattering coefficient depends on the concentration and size distribution of suspended particles in the water and can be estimated using empirical or theoretical models.

Once the transmission map is estimated, it can be used to remove the effects of haze and recover the original colors and contrast of the image using the following equation:

$$I(x)=\frac{(I(x)-A)}{t(x)}+A$$

(6)

where $I(x)$ is the intensity of the image at pixel $x$, $A$ represents the atmospheric light (the color of the light in the absence of scattering), and $t(x)$ is the estimated transmission at pixel $x$.

Color correction algorithms can also be employed to compensate for the color distortion caused by suspended particles. A common approach is to estimate the color of the ambient light in the underwater environment using a white-balancing algorithm and then adjust the color balance of the image accordingly.

In summary, the key to mathematically enhancing underwater images lies in modeling the effects of suspended particles on light transmission using the radiative transfer equation and applying appropriate image enhancement techniques to mitigate these effects.

Absorption and scattering of light in water can lead to blurriness, reduced contrast, and an overall decline in image quality. These effects are further exacerbated in high-turbidity underwater conditions or when powerful artificial light sources are used [51]. Such light sources can cause non-uniform lighting in fluorescence, resulting in reflections that obscure image details and create bright spots as shown in Fig 5.

A common method to model non-uniform illumination in underwater environments is by using the Beer-Lambert law. This law describes how light intensity attenuates as it travels through a medium, stating that the intensity of light decreases exponentially with distance:

$$I=I_{0}*e^{(-k*d)}$$

(7)

where $I$ is the intensity of the light after passing through the medium, $I_{0}$ is the initial intensity of the light, $k$ is the extinction coefficient of the medium (a measure of how much the medium absorbs or scatters light), and $d$ is the distance the light has traveled through the medium.

In underwater environments, the extinction coefficient can vary depending on factors such as water depth, water clarity, and the presence of suspended particles or plankton. Thus, the Beer-Lambert law can be used to model the non-uniformity of underwater illumination [36].

Other factors contributing to non-uniform illumination in underwater environments include the angle of incidence of the light, the direction and intensity of light fluorescence, and the presence of shadows and reflections. Modeling these factors may require more complex mathematical formulas, such as ray tracing or radiative transfer models.

To address these challenges, various techniques and algorithms have been developed for underwater image enhancement. These include classical methods such as histogram equalization and Retinex, as well as deep learning-based approaches utilizing CNNs, GANs, and U-Net [47]. These techniques aim to improve contrast, sharpness, and color balance in images while minimizing the effects of scattering, absorption, and other factors. However, there is still significant work required to further enhance the quality and clarity of underwater images, particularly under challenging conditions [29].

Image denoising is a technique used to reduce or remove noise from digital images. It aims to improve the visual quality of an image by suppressing unwanted noise while preserving important details and structures [27].

Image quality is often evaluated based on the level of contrast in the image. Contrast refers to the difference in luminance reflected from two adjacent planes and is a key factor in making objects distinguishable from the background. Vision is more sensitive to contrast than absolute luminance, which allows us to perceive the world despite variations in illumination conditions. If an image has highly concentrated contrast in a particular range, such as being very dark, critical information may be lost in those areas. Therefore, optimizing the contrast is necessary to represent all the details in the input image. To address issues related to contrast in underwater image processing, numerous algorithms for achieving contrast enhancement have been developed [2].

The colors present in underwater images are mainly blue and green due to their shorter wavelengths. The histogram distribution of these images indicates that the green channel’s mean is more significant than that of the red channel, and the RGB channels’ distribution range does not cover the full range of [0, 255]. To correct the issue of color cast, color correction techniques are used to improve the visual information content of underwater images. A manual correction approach is found to be better than automatic enhancement techniques in terms of the significance level. An enhancement method that uses fuzzy logic and bacterial foraging optimization is proposed to remove the color cast, which gives better results than existing algorithms. Additionally, a method for non-uniform illumination correction is proposed, which uses maximum-likelihood estimation to map the image to Rayleigh distribution. An adaptive linear stretch method that adjusts regions with low light distributions with a threshold depending on the histogram is also proposed.

Underwater images often require image enhancement for improved quality. As such, there are several methods available in the literature to address this issue. In this study, a new underwater image enhancement method is proposed. This method employs the HSV, V transform algorithm, and histogram equalization techniques. Initially, the RGB image is separated into its R, G, and B components, and then converted into the HSV color space. The V element is then extended within a specified interval before converting the image back to the RGB color space. Histogram equalization is then applied to each of the R, G, and B components, and the components are combined to form a color image. Finally, a Gaussian low-pass filter is applied to the image. The performance of the proposed method is compared to that of other studies using mean value and entropy metric, which demonstrate that the proposed method significantly improves underwater image quality [31].

Ordinary AHE tends to over-amplify the contrast in near-constant regions of the image. It is originally developed for the enhancement of low-contrast images [34]. CLAHE is a variant of adaptive histogram equalization in which contrast amplification is limited to reduce this problem of noise amplification. In order to limit noise amplification, CLAHE is used [44]. In CLAHE, the contrast-limited procedure is applied to each neighborhood from which the transformation function is derived. Rather than taking the whole image, CLAHE prevents over-amplification by dividing the image into small data regions called tiles and performing contrast enhancement [4]. These tiles are then rejoined to get an overall enhanced image. It is applied to both grayscale and colored images [30] [4].

Underwater images often suffer from low contrast and color distortion due to the variable attenuation of light and non-uniform absorption of red, green, and blue components. To address these issues, a Retinex-based approach for underwater image enhancement has been proposed. The approach involves using contrast-limited adaptive histogram equalization (CLAHE) to enhance the contrast of the darker components of the underwater image while limiting noise, which may blur visual information. Next, a Retinex-based enhancement is performed on the CLAHE-processed image to restore distorted colors [30] [4]. To restore distorted edges and achieve smoothing of the blurred parts of the image, bilateral filtering is performed on the Retinex-processed image. To optimize the individual strengths of CLAHE, Retinex, and bilateral filtering algorithms within a single framework, suitable parameter values are determined. Comparing the performance of the proposed approach with existing methods, both qualitatively and quantitatively, indicates that it results in better enhancement of underwater images [29].

Haze arises from particles suspended in bodies of water such as sand, minerals, and plankton. This phenomenon disrupts the clarity of underwater images by reducing contrast, causing poor visibility, absorbing natural light, and limiting color variation. Enhancing the quality and visibility of underwater images requires the dehazing process [22]. This research introduces the Dark Channel Prior (DCP) algorithm, which capitalizes on the observation that most local patches in haze-free outdoor images contain pixels with very low intensity in at least one color channel. By utilizing DCP, underwater images exhibit significantly improved visibility and superior color accuracy. Moreover, this approach reduces computational complexity and enhances dehazing efficiency. Underwater images experience distortions primarily due to light dispersion and color effects. The dispersion of light and its scattering in water reduces the visibility and contrast of captured images. Additionally, color changes caused by the presence of particles such as sand, minerals, and plankton in the water, along with the absorption and scattering of natural light, further impact underwater images. When light reflects from objects in the water, it encounters suspended particles, leading to light absorption and scattering [22]. To address these issues, the Dark Channel Prior (DCP) method is applied. This method estimates the atmospheric light and utilizes a mathematical function to handle both sky and non-sky regions. It identifies affected patches in the images, estimates the scene depth, and removes the haze to enhance the clarity of the image. To improve the accuracy of the depth map generated by the block-based dark channel prior, image matting is employed. This combination of techniques enhances accuracy and enables more precise identification of object contours [22]. The application of image matting to the underwater depth map, derived through the general dark-channel methodology, represents a novel approach. Subsequently, the following section presents a list of existing works in this field.

Underwater images often suffer from low contrast and poor visibility, making it crucial to enhance them before further processing. Image enhancement techniques aim to improve the quality and contrast of degraded underwater photos and videos. Standard cameras used for capturing underwater scenes face challenges such as limited available light, low resolution, and blurriness, necessitating the improvement of the initial images or videos obtained from image processing equipment. Researchers have proposed various solutions to address these challenges.

One commonly used approach for enhancing underwater images is the dark channel prior (DCP), which aims to improve the Peak Signal to Noise Ratio (PSNR). However, DCP has significant drawbacks, including the tendency to darken images, reduce contrast, and introduce halo effects. To overcome these limitations, the suggested technique incorporates contrast-limited adaptive histogram equalization (CLAHE) and the Adaptive Color Correction technique.

To evaluate the proposed approach, experiments were conducted using photographs obtained from the Japan Agency for Marine-Earth Science and Technology (JAMSTEC) as well as from the internet. Performance measures such as entropy (MOE), enhancement (EME), mean square error (MSE), and PSNR were used during the evaluation. The results demonstrate that the proposed framework outperforms other methods in terms of MSE and PSNR, achieving values of 0.26 and 32, respectively.

Mean Filter The mean filter is a method employed to decrease image noise. It involves performing a local averaging operation, making it one of the most basic linear filters. In this technique, each pixel’s value is substituted with the average value of all the pixels in its surrounding neighborhood. If we denote a noisy image as $f(i,j)$, the resulting smoothed image can be obtained as $g(x,y)$ by following this process.

$$g(x,y)=\frac{1}{n}_{i,j\in S}\sum f(i,j)$$

(8)

Bilateral Filter A bilateral filter is a non-linear image-smoothing filter that preserves edges while reducing noise. It operates by replacing the intensity of each pixel with a weighted average of the intensities of nearby pixels. The weights are determined using a Gaussian distribution.

$$BF[I]_{p}=\sum_{q\in S}G_{\sigma_{s}}(||p-q||)G_{\sigma_{r}}(|I_{p}-I_{q}|)I_{q}$$

(9)

Gaussian Filter A Gaussian Filter serves as a low-pass filter employed to diminish noise (high-frequency components) and blur specific areas within an image. This filter is constructed as an Odd-sized Symmetric Kernel (a Matrix in Digital Image Processing terms), which is applied to each pixel in the Region of Interest to achieve the intended outcome. The kernel is designed to be gentle regarding significant colour changes (edges), as the pixels near the centre of the kernel hold more significance in determining the final value compared to those at the edges.

$$G(x,y)=\frac{1}{2\pi\sigma^{2}}e^{-\frac{x^{2}+y^{2}}{2\sigma^{2}}}$$

(10)

Median Filter The median filter, frequently employed in digital filtering, is a non-linear technique aimed at eliminating noise from images or signals. It serves as a common pre-processing step to enhance subsequent processing outcomes, such as image edge detection.

The following models benefit from the well-known encoder–decoder architecture to advance underwater image enhancement research. P2P Network Recently, [21] proposed an approach to improve the quality of underwater images using pixel-to-pixel (P2P) networks. Their model, resembling REDNet [20], adopts a symmetric architecture consisting of an encoder and a decoder. The encoder is constructed with three convolutional layers, while the decoder is formed by three deconvolutional layers. ReLU activation is applied to each network element except for the last one as shown in Fig 9.

To achieve a data-driven image enhancement model, the super-parameters of our network play a crucial role. The convolutional part retains the first three layers while discarding the fully connected layers. The reason behind this decision is that the full connection layers are designed for feature map** from two dimensions to one, primarily for input to classifiers. However, the objective is to create a pixel-to-pixel network for image enhancement, which differs from classification tasks. Utilizing full connection layers would result in the loss of important two-dimensional information, making it unsuitable for underwater image enhancement[21]. Additionally, chose to abandon the pooling layers. Although pooling and unpooling layers can enhance object recognition and semantic segmentation by sharpening object edges, they are unnecessary and detrimental to image enhancement and denoising tasks. This is primarily because pooling layers lead to denser feature graphs during the multi-to-one map** operation, causing the loss of spatial information within a receptive field. Furthermore, the corresponding unpooling layers introduce considerable noise information. During the unpooling map**, only one value originates from the original feature map, while the remaining values are artificially generated (typically filled with zeros) [21].

A combination of multi-scale structure similarity and L1 is used for the loss function. To calculate SSIM, the appropriate selection of the size of the Gaussian kernel to compute the image mean value and variance is particularly crucial. If the selection is small, the local structure of the image cannot be ll-maintained by the calculated SSIM loss, and artifacts will appear. If the selection is large, the noise will be generated by the network at the edge of the image.

Underwater images are crucial for obtaining and interpreting information about the underwater environment. The reliability of underwater intelligent systems depends on high-quality underwater images. Unfortunately, these images often suffer from low contrast, color casts, blurring, low light, and uneven illumination, which severely limit their usefulness. To address this issue, numerous methods have been proposed, including those that utilize deep learning technologies. Hover, the performance of these methods is often unsatisfactory due to a lack of sufficient training data and effective network structures [26].

To tackle these challenges, this report proposes a conditional generative adversarial network (cGAN) for enhancing underwater images. The proposed approach uses a multi-scale generator to produce clear underwater images and a dual discriminator to capture local and global semantic information, ensuring that the generated results are both realistic and natural. Experimental results, obtained from both real-world and synthetic underwater images, show that the proposed method outperforms existing state-of-the-art underwater image enhancement methods [26].

Multi-Scale Generator. cGAN’s multi-scale generator comprises three main components: a multi-scale feature extraction unit, a feature refinement unit, and a residual map estimation unit. The multi-scale feature extraction unit is constructed using three sets of multi-scale convolutions with different kernel sizes (7x7, 5x5, and 3x3), each set consisting of five convolutional layers with increasing filter numbers ranging from 16 to 256 [26]. A non-linear activation ReLU follows each convolutional layer. The multi-scale feature extractor aims to obtain statistical information from inputs on various scales by acquiring different receptive fields. The multi-scale features are then down-sampled by half of their original size, concatenated, and fed to the feature refinement unit to capture global features and reduce computational costs. The refined features are processed through successive convolutional layers before being down-sampled and fed to three successive convolutional layers, each with 64 filters, and then up-sampled to their original size. Finally, a residual map is estimated by a convolutional layer without non-linear activity, which is used to achieve the final enhanced result via element-wise addition. Zero padding is applied to each convolutional layer to maintain input and output sizes. With the exception of the multi-scale feature extractor’s convolutional layers, all convolutional layers have 3x3 kernel sizes. Unlike the common encoder-decoder and cGAN network structures, the generator includes a multi-scale feature extraction unit designed to enhance network capability and adapt to varying underwater sources. Additionally, the generator has a shallow and lightweight structure and does not use skip connections.

Dual Discriminator. The dual discriminator comprises two sub-discriminators with identical network structures but different weights. Additionally, the inputs to these sub-discriminators have different sizes - one is the original size, while the other is half the original size. The dual discriminator aims to guide the generator in producing realistic images at both the global semantic and local detail levels. This design is necessary because the existing discriminator cannot effectively guide the generator to create realistic details. By providing multi-resolution inputs to different discriminators, the visual quality of the results can be improved. Specifically, the sub-discriminator contains eight convolutional layers with an increasing number of 3x3 filters, increasing from 64 to 512 by a factor of 2. Stridden convolutions are used to reduce the image resolutions, and the 512 feature maps are fed to two fully connected layers to predict the probability of the inputs being real or fake. Unlike the multi-scale generator, the first convolution in the sub-discriminator is followed by Leaky ReLU non-linear activation, while the other convolutions are followed by batch normalization and Leaky ReLU. The last fully connected layer uses the Sigmoid non-linear activation to predict the probability, which is commonly used in image classification tasks. These two sub-discriminators are employed to guide the multiscale generator [26].

A variation of the standard GAN network structure is the cycle-consistent adversarial network (CycleGAN), which uses two mirror-symmetric GAN generators and two matching discriminators arranged in a ring network. The CycleGAN framework involves training two GAN networks, denoted as G and F, along with two discriminators, $D_{x}$ and $D_{y}$. The generators G and F are utilized to discover the map** relationships between the X and Y domains and the Y and X domains, respectively. The necessary conditions for the input picture and the produced image to correlate are F(G(x)) $\approx$ x and G(F(y)) $\approx$ y. To achieve cyclic consistency, Cyc1e GAN is suggested as the cyclic consistency loss function. This network structure overcomes the challenge faced by GANs, which require paired data for training, and performs all with underwater photos that do not have paired data [33]. The CycleGAN is a GAN designed for unpaired image-to-image translation, where the task is to translate images from a Source domain X to a target domain Y. It consists of two GANs, one for translating from domain X to Y and one from Y to X [32]. The two discriminators represent the functions:

$$D_{A}:X\to R;D_{B}:Y\to R$$

(11)

the two generators represent the function:

$$G_{A}:X\to Y;G_{B}:Y\to X$$

(12)

Generator A generator’s goal is to alter the input picture and produce it as the output. A neural network structure is made up of three parts an encoder, a transformer, and a decoder. While increasing the number of channels, the encoder reduces the size of the input pictures. It is made up of 3 layers of blocks, similar to the Discriminator, with a 2D Convolution Layer, an Instance Normalization Layer, and a Leaky ReLU Layer in each block. The first layer block just adds 64 input channels; it has no effect on the image’s size. However, each of the next 2 layers of blocks reduces the input size by 50% while increasing the number of channels. The transformer then receives the altered input [45] [47]. The transformer maintains the input’s size while adding the needed characteristics [43]. It has six ResNet blocks, also known as residual netblocks. Each ResNet has two layers blocks: the first layer block has a Leaky ReLU Layer, an Instance Normalization Layer, and a 2D Convolution Layer (with stride=1). A 2D Convolution Layer (with stride=1) and an Instance Normalization Layers are both included in the second layer block. The decoder receives the modified input after that [41]. To create the final output image, the decoder shrinks the input to its original size and collapses all channels into RGB. Two Transpose Convolution Layers are stacked together to accomplish the enlargement operation. A transpose convolution layer might be thought of as a simple combination of a 2D Up-sampling layer and a 2D convolution layer with stride=1. In general, it will reduce the number of channels while increasing the size of the input. The output layer will eventually receive the 256x256 pixel data with 64 channels generated by the two transpose convolution layers, collapse the channels into RGB, and output it as the final output image [38] [39].

Deep learning techniques have shown promise in enhancing underwater images, but there are gaps in the research that need to be addressed. One challenge is the high number of parameters involved, leading to overfitting and reduced generalization ability. There is a need for research to develop efficient deep-learning models with fewer parameters that still achieve good performance. Another gap is the interpretability of deep learning models, which are often considered "black-box" models, making it difficult to understand how they make decisions. There is a need for develo** methods to interpret these models to identify strengths and weaknesses and improve performance. Overall, the research gap in deep learning for underwater image enhancement is in develo** efficient and interpretable models with good performance.

Several methods have been proposed to tackle challenges in Underwater Image Enhancement (UIE). Challenges like light attenuation and scattering often result in color casts and diminished visibility [15]. One particularly noteworthy approach introduced a novel quality assessment method centered around colorfulness, contrast, and visibility metrics, providing an effective means to evaluate UIE outcomes [15]. However, the diverse underwater landscapes pose a challenge to existing color constancy methods. To address this, an adaptive UIE technique leveraging hue channel statistics and deep learning networks trained on authentic datasets with ground truth annotations was developed in [16].

Texture and color enhancement are pivotal for effective underwater image enhancement, and the Texture-Aware and Color-Consistent Network (TACC-Net) has emerged as a standout performer in this regard. By decoupling features to enhance texture and ensure color consistency, TACC-Net has significantly improved visual quality [17]. Meanwhile, issues such as light absorption and turbulence continue to impair image quality in underwater target imaging, affecting clarity and resolution. To address these challenges, a study has proposed a block mixed filter denoising technique and underscored the importance of objective quality evaluation for image enhancement methods [18].

This paper [23] introduces a novel strategy to enhance underwater videos and images using fusion principles. The unique aspect of this strategy is that it derives both the inputs and the weight measures solely from the degraded version of the image, without the need for specialized hardware or prior knowledge of underwater conditions or scene structure.

The approach involves the derivation of two inputs from the original underwater image or frame. The first input is a color-corrected version that addresses the color distortion commonly caused by underwater environments. The second input is a contrast-enhanced version, which aims to improve the visibility of details often lost in the hazy underwater images. These inputs help to mitigate the color and contrast issues inherent in underwater imaging [23].

Additionally, four weight maps are defined to increase the visibility of distant objects, which are usually degraded due to medium scattering and absorption in underwater environments. These weight maps help in selectively emphasizing important features in the image, enhancing the overall clarity.

The fusion framework integrates these inputs and weight maps to produce an enhanced image. This approach ensures that the finest details and edges in the image are significantly improved. The enhanced images and videos are characterized by a reduced noise level, as effective edge-preserving noise reduction strategies are applied to minimize noise while retaining important details. Dark areas in the image are better exposed, making hidden details more visible. The overall contrast of the image is enhanced, making it more visually appealing and informative.

For videos, the framework also ensures temporal coherence between adjacent frames. This means that the enhancement process maintains consistency across frames, preventing flickering or abrupt changes that can distract viewers.

The utility of this enhancement technique is demonstrated across several challenging applications, showing its versatility and effectiveness in various underwater imaging scenarios.

Autonomous underwater vehicles (AUVs) rely on a variety of sensors, including acoustic, inertial, and visual sensors, for intelligent decision-making. Among these, vision is particularly attractive due to its non-intrusive, passive nature and high information content, especially at shallower depths. However, several factors adversely affect the quality of visual data obtained underwater. Light refraction and absorption, suspended particles in the water, and color distortion all contribute to producing noisy and distorted images. Consequently, AUVs that depend on visual sensing face significant challenges and often exhibit poor performance on vision-driven tasks [50].

This paper [50] proposes a method to enhance the quality of visual underwater scenes using Generative Adversarial Networks (GANs). The goal is to improve the visual input for vision-driven behaviors further down the autonomy pipeline of AUVs. GANs are well-suited for this task because of their ability to generate high-quality images that closely resemble real-world scenes, making them ideal for underwater image restoration [50].

The key challenges in underwater visual data include light refraction and absorption, suspended particles, and color distortion. Underwater environments significantly alter light paths, causing refraction and absorption that lead to reduced clarity and visibility. Particles in the water scatter light, creating a hazy appearance and further degrading image quality. The underwater medium absorbs different wavelengths of light at different rates, causing color distortions that affect the accuracy of visual data [50].

The proposed method leverages the power of GANs to address these challenges. GANs consist of two neural networks: a generator and a discriminator. The generator creates enhanced images from the degraded input, while the discriminator evaluates the authenticity of the generated images, driving the generator to produce increasingly realistic enhancements. This adversarial process results in images that are not only visually appealing but also more useful for subsequent vision-driven tasks.

To train the GANs effectively, a dataset specifically tailored for underwater image restoration is required. Recently proposed methods allow for the generation of such datasets by simulating various underwater conditions and degradations. This synthetic dataset includes images with different types of distortions commonly found in underwater environments, providing a comprehensive training set for the GANs.

For visually-guided underwater robots, improving the quality of visual data can lead to increased safety and reliability. Enhanced visual perception enables AUVs to perform better in tasks such as navigation, object detection, and diver tracking. The proposed GAN-based approach not only generates visually appealing images but also enhances the accuracy of vision-driven algorithms.

The effectiveness of the proposed method is demonstrated through both quantitative and qualitative evaluations. Enhanced images show significant improvements in clarity, color accuracy, and detail preservation compared to the original degraded images. Additionally, these improvements translate to increased accuracy for a diver tracking algorithm, showcasing the practical benefits of the enhanced visual data.

In this work, [13] a conditional generative adversarial network-based model is presented for real-time underwater image enhancement. The model’s adversarial training is supervised by an objective function that evaluates perceptual image quality based on global content, color, local texture, and style information. A large-scale dataset, EUVP, is introduced, consisting of paired and unpaired collections of underwater images of varying quality, captured using seven different cameras under various visibility conditions during oceanic explorations and human-robot collaborative experiments.

Several qualitative and quantitative evaluations were performed, demonstrating that the proposed model effectively learns to enhance underwater image quality from both paired and unpaired training datasets. The enhanced images improve the performance of standard models for underwater object detection, human pose estimation, and saliency prediction. These results validate the suitability of the proposed model for real-time preprocessing in the autonomy pipeline of visually-guided underwater robots[13].

In this paper [12], the simultaneous enhancement and super-resolution (SESR) problem for underwater robot vision is introduced and tackled, providing an efficient solution for near real-time applications. The proposed solution, Deep SESR, is a generative model based on a residual-in-residual network that learns to restore perceptual image qualities at 2x, 3x, or 4x higher spatial resolution. The model is trained using a multi-modal objective function that addresses chrominance-specific underwater color degradation, lack of image sharpness, and loss in high-level feature representation. Additionally, the model is supervised to learn salient foreground regions in the image, which guides it to enhance global contrast.

An end-to-end training pipeline is designed to jointly learn saliency prediction and SESR on a shared hierarchical feature space for fast inference. This approach ensures that the model can process images quickly, making it suitable for near real-time applications [12].

The paper [12] also introduces UFO-120, the first dataset designed to facilitate large-scale SESR learning, containing over 1500 training samples and a benchmark test set of 120 samples. Experimental evaluations on UFO-120 and other standard datasets demonstrate that Deep SESR outperforms existing solutions for underwater image enhancement and super-resolution. The model’s generalization performance is validated on several test cases, including underwater images with diverse spectral and spatial degradation levels and terrestrial images with unseen natural objects.

Furthermore, the computational feasibility of Deep SESR for single-board deployments is analyzed, demonstrating its operational benefits for visually-guided underwater robots. The model’s ability to enhance and super-resolve images in near real-time provides significant advantages for underwater robotics, enabling more accurate and reliable visual perception in challenging underwater environments.

Underwater image enhancement is vital for marine engineering and aquatic robotics, but existing algorithms are mainly tested on synthetic datasets or limited real-world images. To evaluate these algorithms’ real-world performance, a comprehensive perceptual study using large-scale real-world images is conducted. This study introduces the Underwater Image Enhancement Benchmark (UIEB), containing 950 real-world underwater images, with 890 having corresponding reference images and 60 considered challenging due to the lack of satisfactory references [14].

The study also proposes Water-Net, an underwater image enhancement network trained on the UIEB [24]. The benchmark evaluations and Water-Net demonstrate the strengths and limitations of current algorithms, providing insights for future research. This work advances the assessment and benchmarking of underwater image enhancement algorithms, contributing to the field’s progress [14].

Enhancing underwater images is crucial for applications such as underwater exploration. Traditional methods often rely on paired underwater and reference images for training, which are challenging to acquire. These methods frequently suffer from information loss, resulting in blurred details and limited applicability across diverse underwater conditions [49].

This paper [49] introduces a novel approach using the Multi-Scale Structural and Color Enhanced Generative Adversarial Network (MSSCE-GAN) for unpaired underwater image enhancement. The method includes modules for detail feature recovery and attention enhancement, addressing various distortions prevalent in underwater imagery.

Key to this approach is its ability to generate superior enhanced images without requiring paired training data. Experimental evaluations demonstrate significant improvements over existing techniques in terms of effectiveness and generalizability across multiple underwater image datasets.

Underwater images typically suffer from low contrast and significant color distortions due to varying light attenuation as it travels through water. This phenomenon affects different colors asymmetrically, complicating image restoration tasks. Despite numerous attempts using deep learning for underwater image restoration (UIR), existing methods often overlook this asymmetry in network design [11].

This article introduces two novel contributions to address these challenges in UIR. Firstly, it proposes adapting receptive field sizes based on the wavelength-dependent attenuation of color channels, aiming for improved performance. Secondly, it incorporates an attentive skip mechanism to refine multi-contextual features effectively, enhancing model representational power while suppressing irrelevant features.

The proposed framework, Deep WaveNet, is optimized using pixel-wise and feature-based cost functions. Extensive experiments demonstrate its superiority over state-of-the-art methods on benchmark datasets. Furthermore, the study validates the enhanced images through various high-level vision tasks, such as underwater image semantic segmentation and diver’s 2D pose estimation [11].

Underwater Dark:

This dataset comprises 5550 pairs of images for training, each with a size of $256\times 256$ pixels. Each pair consists of two images, one contains poor-quality or gray images, and the other contains enhanced or colored images. The filenames for each pair are identical. Additionally, 570 images are set aside for validation. In total, the dataset contains 11,670 images as shown in Table 2 .

Underwater ImageNet:

The Underwater ImageNet dataset consists of 3700 pairs of images for training, each with a size of $256\times 256$ pixels. Similar to the Underwater Dark dataset, one contains poor-quality images and the other contains enhanced or better-quality images. The filenames for corresponding pairs match. The dataset also includes 1270 images for validation, resulting in a total of 8670 images.

Underwater Scenes:

This dataset comprises 2185 pairs of images, each with a size of $320\times 240$ pixels for training, with each pair containing a poor-quality image and a corresponding enhanced or better-quality image. The filenames for corresponding pairs are consistent. Additionally, 130 images are allocated for validation purposes. In total, the dataset encompasses 4500 images.

In the dataset for unpaired training, there are 3195 images representing poor quality images, while the set comprises 3140 images of enhanced or better quality. These images come in sizes of $960\times 540$, $640\times 480$, and $320\times 240$ pixels. Additionally, there are 330 images allocated for validation purposes. These images are not paired, meaning that there is no one-to-one correspondence between the poor-quality and enhanced-quality images. This dataset arrangement allows for the training of models aimed at enhancing image quality without relying on direct paired examples as shown in Table 3.

Low illumination in images can result from various factors such as poor lighting conditions. Simulating low illumination is crucial for testing the robustness of image processing algorithms in real-world scenarios. The degradation is achieved by reducing the overall brightness of the image, mimicking the effect of dim lighting conditions. This reduction in brightness can lead to loss of details and visibility of objects in the image and variation of illumination shown in Fig 22.

where $I_{ID}$ is the modified image after applying varying illumination. This factor determines the extent of brightness reduction. A lower value results in a darker image.

High contrast simulates images with intense differences between light and dark areas. This effect is achieved by adjusting the pixel values to increase the contrast.The equation for increasing contrast is:

$$I_{CD}(x,y)=\alpha\times I(x,y)+\beta,\quad\forall\alpha\sim U(a,b),\quad\beta=m$$

(41)

The hazy effect simulates the presence of haze or fog in the image. It is achieved by adding a semi-transparent haze layer over the original image. The mathematical expression for applying the haze effect to the image is:

$$I_{DH}(x,y)=(1-\gamma)\times I(x,y)+\gamma\times\gamma_{c}(x,y),\quad\forall\gamma\sim U(a,b),\quad\forall\gamma_{c}\sim U(l,m)$$

(42)

Here, $\gamma_{L}(x,y)$ creates a haze layer with the same dimensions as the degraded image, where each pixel is set to the randomly generated haze color.

Blend the original image and the haze layer using the formula above that generate $I_{DH}$ . The original image is multiplied by $(1-\gamma)$ to reduce its intensity, and the haze layer is multiplied by $\gamma$ to control the strength of the haze effect as shown in Fig 24. The resulting image represents the original scene with the added haze effect. This process mimics the visual appearance of images captured in hazy conditions, where distant objects appear less distinct due to scattering of light by haze particles in the atmosphere.

The blurry effect is simulated using a Gaussian blur filter applied to the entire image. The equation for applying Gaussian blur to an image is as follows:

$$G(x,y)=\frac{1}{2\pi\sigma^{2}}\exp\left(-\frac{x^{2}+y^{2}}{2\sigma^{2}}\right)$$

(43)

where $(x,y)$ are the coordinates in the kernel, and $\sigma$ is the standard deviation of the Gaussian distribution. The Gaussian kernel $G(x,y)$ is typically normalized so that the sum of all elements equals 1. The convolution operation between the input image $I$ and the Gaussian kernel $G$ is denoted by $I*G$. It’s defined as:

$$(I*G)(x,y)=\sum_{i=-k}^{k}\sum_{j=-l}^{l}I(i,j)\cdot G(x-i,y-j)$$

(44)

where $(x,y)$ are the coordinates of the output pixel, $(i,j)$ are the coordinates of the input pixel, and $I(i,j)$ is the intensity of the input pixel. The convolution operation involves sliding the Gaussian kernel over each pixel of the input image and computing a weighted sum of pixel intensities in the neighborhood defined by the kernel as shown in Fig 25. $I_{DB}(x,y)$ is blurred image and defined as:

$$I_{DB}(x,y)=(I*G)(x,y)$$

(45)

The GaussianBlur function convolves the image with a $G(x,y)$ to compute the blurred result. The standard deviation of the $G(x,y)$ is implicitly determined by the kernel size. Overall, the Gaussian blur operation smooths out the sharp transitions between pixel values in the input image, resulting in a blurred version of the original image. The degree of blurring is controlled by the standard deviation $\sigma$ of the Gaussian kernel, with larger values of $\sigma$ resulting in more significant blurring.