-
A provable control of sensitivity of neural networks through a direct parameterization of the overall bi-Lipschitzness
Authors:
Yuri Kinoshita,
Taro Toyoizumi
Abstract:
While neural networks can enjoy an outstanding flexibility and exhibit unprecedented performance, the mechanism behind their behavior is still not well-understood. To tackle this fundamental challenge, researchers have tried to restrict and manipulate some of their properties in order to gain new insights and better control on them. Especially, throughout the past few years, the concept of \emph{b…
▽ More
While neural networks can enjoy an outstanding flexibility and exhibit unprecedented performance, the mechanism behind their behavior is still not well-understood. To tackle this fundamental challenge, researchers have tried to restrict and manipulate some of their properties in order to gain new insights and better control on them. Especially, throughout the past few years, the concept of \emph{bi-Lipschitzness} has been proved as a beneficial inductive bias in many areas. However, due to its complexity, the design and control of bi-Lipschitz architectures are falling behind, and a model that is precisely designed for bi-Lipschitzness realizing a direct and simple control of the constants along with solid theoretical analysis is lacking. In this work, we investigate and propose a novel framework for bi-Lipschitzness that can achieve such a clear and tight control based on convex neural networks and the Legendre-Fenchel duality. Its desirable properties are illustrated with concrete experiments. We also apply this framework to uncertainty estimation and monotone problem settings to illustrate its broad range of applications.
△ Less
Submitted 15 April, 2024;
originally announced April 2024.
-
Power Series Composition in Near-Linear Time
Authors:
Yasunori Kinoshita,
Baitian Li
Abstract:
We present an algebraic algorithm that computes the composition of two power series in $\mathop{\tilde{\mathrm O}}(n)$ time complexity. The previous best algorithms are $\mathop{\mathrm O}(n^{1+o(1)})$ by Kedlaya and Umans (FOCS 2008) and an $\mathop{\mathrm O}(n^{1.43})$ algebraic algorithm by Neiger, Salvy, Schost and Villard (JACM 2023).
Our algorithm builds upon the recent Graeffe iteration…
▽ More
We present an algebraic algorithm that computes the composition of two power series in $\mathop{\tilde{\mathrm O}}(n)$ time complexity. The previous best algorithms are $\mathop{\mathrm O}(n^{1+o(1)})$ by Kedlaya and Umans (FOCS 2008) and an $\mathop{\mathrm O}(n^{1.43})$ algebraic algorithm by Neiger, Salvy, Schost and Villard (JACM 2023).
Our algorithm builds upon the recent Graeffe iteration approach to manipulate rational power series introduced by Bostan and Mori (SOSA 2021).
△ Less
Submitted 7 April, 2024;
originally announced April 2024.
-
Controlling Posterior Collapse by an Inverse Lipschitz Constraint on the Decoder Network
Authors:
Yuri Kinoshita,
Kenta Oono,
Kenji Fukumizu,
Yuichi Yoshida,
Shin-ichi Maeda
Abstract:
Variational autoencoders (VAEs) are one of the deep generative models that have experienced enormous success over the past decades. However, in practice, they suffer from a problem called posterior collapse, which occurs when the encoder coincides, or collapses, with the prior taking no information from the latent structure of the input data into consideration. In this work, we introduce an invers…
▽ More
Variational autoencoders (VAEs) are one of the deep generative models that have experienced enormous success over the past decades. However, in practice, they suffer from a problem called posterior collapse, which occurs when the encoder coincides, or collapses, with the prior taking no information from the latent structure of the input data into consideration. In this work, we introduce an inverse Lipschitz neural network into the decoder and, based on this architecture, provide a new method that can control in a simple and clear manner the degree of posterior collapse for a wide range of VAE models equipped with a concrete theoretical guarantee. We also illustrate the effectiveness of our method through several numerical experiments.
△ Less
Submitted 2 February, 2024; v1 submitted 25 April, 2023;
originally announced April 2023.
-
Template matching with white balance adjustment under multiple illuminants
Authors:
Teruaki Akazawa,
Yuma Kinoshita,
Hitoshi Kiya
Abstract:
In this paper, we propose a novel template matching method with a white balancing adjustment, called N-white balancing, which was proposed for multi-illuminant scenes. To reduce the influence of lighting effects, N-white balancing is applied to images for multi-illumination color constancy, and then a template matching method is carried out by using adjusted images. In experiments, the effectivene…
▽ More
In this paper, we propose a novel template matching method with a white balancing adjustment, called N-white balancing, which was proposed for multi-illuminant scenes. To reduce the influence of lighting effects, N-white balancing is applied to images for multi-illumination color constancy, and then a template matching method is carried out by using adjusted images. In experiments, the effectiveness of the proposed method is demonstrated to be effective in object detection tasks under various illumination conditions.
△ Less
Submitted 3 August, 2022;
originally announced August 2022.
-
Image and Model Transformation with Secret Key for Vision Transformer
Authors:
Hitoshi Kiya,
Ryota Iijima,
MaungMaung Aprilpyone,
Yuma Kinoshita
Abstract:
In this paper, we propose a combined use of transformed images and vision transformer (ViT) models transformed with a secret key. We show for the first time that models trained with plain images can be directly transformed to models trained with encrypted images on the basis of the ViT architecture, and the performance of the transformed models is the same as models trained with plain images when…
▽ More
In this paper, we propose a combined use of transformed images and vision transformer (ViT) models transformed with a secret key. We show for the first time that models trained with plain images can be directly transformed to models trained with encrypted images on the basis of the ViT architecture, and the performance of the transformed models is the same as models trained with plain images when using test images encrypted with the key. In addition, the proposed scheme does not require any specially prepared data for training models or network modification, so it also allows us to easily update the secret key. In an experiment, the effectiveness of the proposed scheme is evaluated in terms of performance degradation and model protection performance in an image classification task on the CIFAR-10 dataset.
△ Less
Submitted 20 July, 2022; v1 submitted 12 July, 2022;
originally announced July 2022.
-
Privacy-Preserving Image Classification Using Vision Transformer
Authors:
Zheng Qi,
AprilPyone MaungMaung,
Yuma Kinoshita,
Hitoshi Kiya
Abstract:
In this paper, we propose a privacy-preserving image classification method that is based on the combined use of encrypted images and the vision transformer (ViT). The proposed method allows us not only to apply images without visual information to ViT models for both training and testing but to also maintain a high classification accuracy. ViT utilizes patch embedding and position embedding for im…
▽ More
In this paper, we propose a privacy-preserving image classification method that is based on the combined use of encrypted images and the vision transformer (ViT). The proposed method allows us not only to apply images without visual information to ViT models for both training and testing but to also maintain a high classification accuracy. ViT utilizes patch embedding and position embedding for image patches, so this architecture is shown to reduce the influence of block-wise image transformation. In an experiment, the proposed method for privacy-preserving image classification is demonstrated to outperform state-of-the-art methods in terms of classification accuracy and robustness against various attacks.
△ Less
Submitted 24 May, 2022;
originally announced May 2022.
-
Improved Convergence Rate of Stochastic Gradient Langevin Dynamics with Variance Reduction and its Application to Optimization
Authors:
Yuri Kinoshita,
Taiji Suzuki
Abstract:
The stochastic gradient Langevin Dynamics is one of the most fundamental algorithms to solve sampling problems and non-convex optimization appearing in several machine learning applications. Especially, its variance reduced versions have nowadays gained particular attention. In this paper, we study two variants of this kind, namely, the Stochastic Variance Reduced Gradient Langevin Dynamics and th…
▽ More
The stochastic gradient Langevin Dynamics is one of the most fundamental algorithms to solve sampling problems and non-convex optimization appearing in several machine learning applications. Especially, its variance reduced versions have nowadays gained particular attention. In this paper, we study two variants of this kind, namely, the Stochastic Variance Reduced Gradient Langevin Dynamics and the Stochastic Recursive Gradient Langevin Dynamics. We prove their convergence to the objective distribution in terms of KL-divergence under the sole assumptions of smoothness and Log-Sobolev inequality which are weaker conditions than those used in prior works for these algorithms. With the batch size and the inner loop length set to $\sqrt{n}$, the gradient complexity to achieve an $ε$-precision is $\tilde{O}((n+dn^{1/2}ε^{-1})γ^2 L^2α^{-2})$, which is an improvement from any previous analyses. We also show some essential applications of our result to non-convex optimization.
△ Less
Submitted 18 November, 2022; v1 submitted 30 March, 2022;
originally announced March 2022.
-
An Overview of Compressible and Learnable Image Transformation with Secret Key and Its Applications
Authors:
Hitoshi Kiya,
AprilPyone MaungMaung,
Yuma Kinoshita,
Shoko Imaizumi,
Sayaka Shiota
Abstract:
This article presents an overview of image transformation with a secret key and its applications. Image transformation with a secret key enables us not only to protect visual information on plain images but also to embed unique features controlled with a key into images. In addition, numerous encryption methods can generate encrypted images that are compressible and learnable for machine learning.…
▽ More
This article presents an overview of image transformation with a secret key and its applications. Image transformation with a secret key enables us not only to protect visual information on plain images but also to embed unique features controlled with a key into images. In addition, numerous encryption methods can generate encrypted images that are compressible and learnable for machine learning. Various applications of such transformation have been developed by using these properties. In this paper, we focus on a class of image transformation referred to as learnable image encryption, which is applicable to privacy-preserving machine learning and adversarially robust defense. Detailed descriptions of both transformation algorithms and performances are provided. Moreover, we discuss robustness against various attacks.
△ Less
Submitted 15 April, 2022; v1 submitted 26 January, 2022;
originally announced January 2022.
-
Self-Supervised Intrinsic Image Decomposition Network Considering Reflectance Consistency
Authors:
Yuma Kinoshita,
Hitoshi Kiya
Abstract:
We propose a novel intrinsic image decomposition network considering reflectance consistency. Intrinsic image decomposition aims to decompose an image into illumination-invariant and illumination-variant components, referred to as ``reflectance'' and ``shading,'' respectively. Although there are three consistencies that the reflectance and shading should satisfy, most conventional work does not su…
▽ More
We propose a novel intrinsic image decomposition network considering reflectance consistency. Intrinsic image decomposition aims to decompose an image into illumination-invariant and illumination-variant components, referred to as ``reflectance'' and ``shading,'' respectively. Although there are three consistencies that the reflectance and shading should satisfy, most conventional work does not sufficiently account for consistency with respect to reflectance, owing to the use of a white-illuminant decomposition model and the lack of training images capturing the same objects under various illumination-brightness and -color conditions. For this reason, the three consistencies are considered in the proposed network by using a color-illuminant model and training the network with losses calculated from images taken under various illumination conditions. In addition, the proposed network can be trained in a self-supervised manner because various illumination conditions can easily be simulated. Experimental results show that our network can decompose images into reflectance and shading components.
△ Less
Submitted 5 November, 2021;
originally announced November 2021.
-
Spatially varying white balancing for mixed and non-uniform illuminants
Authors:
Teruaki Akazawa,
Yuma Kinoshita,
Hitoshi Kiya
Abstract:
In this paper, we propose a novel white balance adjustment, called "spatially varying white balancing," for single, mixed, and non-uniform illuminants. By using n diagonal matrices along with a weight, the proposed method can reduce lighting effects on all spatially varying colors in an image under such illumination conditions. In contrast, conventional white balance adjustments do not consider th…
▽ More
In this paper, we propose a novel white balance adjustment, called "spatially varying white balancing," for single, mixed, and non-uniform illuminants. By using n diagonal matrices along with a weight, the proposed method can reduce lighting effects on all spatially varying colors in an image under such illumination conditions. In contrast, conventional white balance adjustments do not consider the correcting of all colors except under a single illuminant. Also, multi-color balance adjustments can map multiple colors into corresponding ground truth colors, although they may cause the rank deficiency problem to occur as a non-diagonal matrix is used, unlike white balancing. In an experiment, the effectiveness of the proposed method is shown under mixed and non-uniform illuminants, compared with conventional white and multi-color balancing. Moreover, under a single illuminant, the proposed method has almost the same performance as the conventional white balancing.
△ Less
Submitted 3 September, 2021;
originally announced September 2021.
-
Separated-Spectral-Distribution Estimation Based on Bayesian Inference with Single RGB Camera
Authors:
Yuma Kinoshita,
Hitoshi Kiya
Abstract:
In this paper, we propose a novel method for separately estimating spectral distributions from images captured by a typical RGB camera. The proposed method allows us to separately estimate a spectral distribution of illumination, reflectance, or camera sensitivity, while recent hyperspectral cameras are limited to capturing a joint spectral distribution from a scene. In addition, the use of Bayesi…
▽ More
In this paper, we propose a novel method for separately estimating spectral distributions from images captured by a typical RGB camera. The proposed method allows us to separately estimate a spectral distribution of illumination, reflectance, or camera sensitivity, while recent hyperspectral cameras are limited to capturing a joint spectral distribution from a scene. In addition, the use of Bayesian inference makes it possible to take into account prior information of both spectral distributions and image noise as probability distributions. As a result, the proposed method can estimate spectral distributions in a unified way, and it can enhance the robustness of the estimation against noise, which conventional spectral-distribution estimation methods cannot. The use of Bayesian inference also enables us to obtain the confidence of estimation results. In an experiment, the proposed method is shown not only to outperform conventional estimation methods in terms of RMSE but also to be robust against noise.
△ Less
Submitted 1 June, 2021;
originally announced June 2021.
-
Multi-color balance for color constancy
Authors:
Teruaki Akazawa,
Yuma Kinoshita,
Hitoshi Kiya
Abstract:
In this paper, we propose a novel multi-color balance adjustment for color constancy. The proposed method, called "n-color balancing," allows us not only to perfectly correct n target colors on the basis of corresponding ground truth colors but also to correct colors other than the n colors. In contrast, although white-balancing can perfectly adjust white, colors other than white are not considere…
▽ More
In this paper, we propose a novel multi-color balance adjustment for color constancy. The proposed method, called "n-color balancing," allows us not only to perfectly correct n target colors on the basis of corresponding ground truth colors but also to correct colors other than the n colors. In contrast, although white-balancing can perfectly adjust white, colors other than white are not considered in the framework of white-balancing in general. In an experiment, the proposed multi-color balancing is demonstrated to outperform both conventional white and multi-color balance adjustments including Bradford's model.
△ Less
Submitted 21 May, 2021;
originally announced May 2021.
-
Multi-color balancing for correctly adjusting the intensity of target colors
Authors:
Teruaki Akazawa,
Yuma Kinoshita,
Hitoshi Kiya
Abstract:
In this paper, we propose a novel multi-color balance method for reducing color distortions caused by lighting effects. The proposed method allows us to adjust three target-colors chosen by a user in an input image so that each target color is the same as the corresponding destination (benchmark) one. In contrast, white balancing is a typical technique for reducing the color distortions, however,…
▽ More
In this paper, we propose a novel multi-color balance method for reducing color distortions caused by lighting effects. The proposed method allows us to adjust three target-colors chosen by a user in an input image so that each target color is the same as the corresponding destination (benchmark) one. In contrast, white balancing is a typical technique for reducing the color distortions, however, they cannot remove lighting effects on colors other than white. In an experiment, the proposed method is demonstrated to be able to remove lighting effects on selected three colors, and is compared with existing white balance adjustments.
△ Less
Submitted 21 May, 2021; v1 submitted 3 February, 2021;
originally announced February 2021.
-
CycleGAN without checkerboard artifacts for counter-forensics of fake-image detection
Authors:
Takayuki Osakabe,
Miki Tanaka,
Yuma Kinoshita,
Hitoshi Kiya
Abstract:
In this paper, we propose a novel CycleGAN without checkerboard artifacts for counter-forensics of fake-image detection. Recent rapid advances in image manipulation tools and deep image synthesis techniques, such as Generative Adversarial Networks (GANs) have easily generated fake images, so detecting manipulated images has become an urgent issue. Most state-of-the-art forgery detection methods as…
▽ More
In this paper, we propose a novel CycleGAN without checkerboard artifacts for counter-forensics of fake-image detection. Recent rapid advances in image manipulation tools and deep image synthesis techniques, such as Generative Adversarial Networks (GANs) have easily generated fake images, so detecting manipulated images has become an urgent issue. Most state-of-the-art forgery detection methods assume that images include checkerboard artifacts which are generated by using DNNs. Accordingly, we propose a novel CycleGAN without any checkerboard artifacts for counter-forensics of fake-mage detection methods for the first time, as an example of GANs without checkerboard artifacts.
△ Less
Submitted 1 December, 2020;
originally announced December 2020.
-
Checkerboard-Artifact-Free Image-Enhancement Network Considering Local and Global Features
Authors:
Yuma Kinoshita,
Hitoshi Kiya
Abstract:
In this paper, we propose a novel convolutional neural network (CNN) that never causes checkerboard artifacts, for image enhancement. In research fields of image-to-image translation problems, it is well-known that images generated by usual CNNs are distorted by checkerboard artifacts which mainly caused in forward-propagation of upsampling layers. However, checkerboard artifacts in image enhancem…
▽ More
In this paper, we propose a novel convolutional neural network (CNN) that never causes checkerboard artifacts, for image enhancement. In research fields of image-to-image translation problems, it is well-known that images generated by usual CNNs are distorted by checkerboard artifacts which mainly caused in forward-propagation of upsampling layers. However, checkerboard artifacts in image enhancement have never been discussed. In this paper, we point out that applying U-Net based CNNs to image enhancement causes checkerboard artifacts. In contrast, the proposed network that contains fixed convolutional layers can perfectly prevent the artifacts. In addition, the proposed network architecture, which can handle both local and global features, enables us to improve the performance of image enhancement. Experimental results show that the use of fixed convolutional layers can prevent checkerboard artifacts and the proposed network outperforms state-of-the-art CNN-based image-enhancement methods in terms of various objective quality metrics: PSNR, SSIM, and NIQE.
△ Less
Submitted 12 October, 2020;
originally announced October 2020.
-
Image Transformation Network for Privacy-Preserving Deep Neural Networks and Its Security Evaluation
Authors:
Hiroki Ito,
Yuma Kinoshita,
Hitoshi Kiya
Abstract:
We propose a transformation network for generating visually-protected images for privacy-preserving DNNs. The proposed transformation network is trained by using a plain image dataset so that plain images are transformed into visually protected ones. Conventional perceptual encryption methods have a weak visual-protection performance and some accuracy degradation in image classification. In contra…
▽ More
We propose a transformation network for generating visually-protected images for privacy-preserving DNNs. The proposed transformation network is trained by using a plain image dataset so that plain images are transformed into visually protected ones. Conventional perceptual encryption methods have a weak visual-protection performance and some accuracy degradation in image classification. In contrast, the proposed network enables us not only to strongly protect visual information but also to maintain the image classification accuracy that using plain images achieves. In an image classification experiment, the proposed network is demonstrated to strongly protect visual information on plain images without any performance degradation under the use of CIFAR datasets. In addition, it is shown that the visually protected images are robust against a DNN-based attack, called inverse transformation network attack (ITN-Attack) in an experiment.
△ Less
Submitted 7 August, 2020;
originally announced August 2020.
-
A Privacy-Preserving Machine Learning Scheme Using EtC Images
Authors:
Ayana Kawamura,
Yuma Kinoshita,
Takayuki Nakachi,
Sayaka Shiota,
Hitoshi Kiya
Abstract:
We propose a privacy-preserving machine learning scheme with encryption-then-compression (EtC) images, where EtC images are images encrypted by using a block-based encryption method proposed for EtC systems with JPEG compression. In this paper, a novel property of EtC images is first discussed, although EtC ones was already shown to be compressible as a property. The novel property allows us to di…
▽ More
We propose a privacy-preserving machine learning scheme with encryption-then-compression (EtC) images, where EtC images are images encrypted by using a block-based encryption method proposed for EtC systems with JPEG compression. In this paper, a novel property of EtC images is first discussed, although EtC ones was already shown to be compressible as a property. The novel property allows us to directly apply EtC images to machine learning algorithms non-specialized for computing encrypted data. In addition, the proposed scheme is demonstrated to provide no degradation in the performance of some typical machine learning algorithms including the support vector machine algorithm with kernel trick and random forests under the use of z-score normalization. A number of facial recognition experiments with are carried out to confirm the effectiveness of the proposed scheme.
△ Less
Submitted 17 July, 2020;
originally announced July 2020.
-
Fixed smooth convolutional layer for avoiding checkerboard artifacts in CNNs
Authors:
Yuma Kinoshita,
Hitoshi Kiya
Abstract:
In this paper, we propose a fixed convolutional layer with an order of smoothness not only for avoiding checkerboard artifacts in convolutional neural networks (CNNs) but also for enhancing the performance of CNNs, where the smoothness of its filter kernel can be controlled by a parameter. It is well-known that a number of CNNs generate checkerboard artifacts in both of two process: forward-propag…
▽ More
In this paper, we propose a fixed convolutional layer with an order of smoothness not only for avoiding checkerboard artifacts in convolutional neural networks (CNNs) but also for enhancing the performance of CNNs, where the smoothness of its filter kernel can be controlled by a parameter. It is well-known that a number of CNNs generate checkerboard artifacts in both of two process: forward-propagation of upsampling layers and backward-propagation of strided convolutional layers. The proposed layer can perfectly prevent checkerboard artifacts caused by strided convolutional layers or upsampling layers including transposed convolutional layers. In an image-classification experiment with four CNNs: a simple CNN, VGG8, ResNet-18, and ResNet-101, applying the fixed layers to these CNNs is shown to improve the classification performance of all CNNs. In addition, the fixed layer are applied to generative adversarial networks (GANs), for the first time. From image-generation results, a smoother fixed convolutional layer is demonstrated to enable us to improve the quality of images generated with GANs.
△ Less
Submitted 6 February, 2020;
originally announced February 2020.
-
On the Security of Pixel-Based Image Encryption for Privacy-Preserving Deep Neural Networks
Authors:
Warit Sirichotedumrong,
Yuma Kinoshita,
Hitoshi Kiya
Abstract:
This paper aims to evaluate the safety of a pixel-based image encryption method, which has been proposed to apply images with no visual information to deep neural networks (DNN), in terms of robustness against ciphertext-only attacks (COA). In addition, we propose a novel DNN-based COA that aims to reconstruct the visual information of encrypted images. The effectiveness of the proposed attack is…
▽ More
This paper aims to evaluate the safety of a pixel-based image encryption method, which has been proposed to apply images with no visual information to deep neural networks (DNN), in terms of robustness against ciphertext-only attacks (COA). In addition, we propose a novel DNN-based COA that aims to reconstruct the visual information of encrypted images. The effectiveness of the proposed attack is evaluated under two encryption key conditions: same encryption key, and different encryption keys. The results show that the proposed attack can recover the visual information of the encrypted images if images are encrypted under same encryption key. Otherwise, the pixel-based image encryption method has robustness against COA.
△ Less
Submitted 9 December, 2019;
originally announced December 2019.
-
Privacy-Preserving Machine Learning Using EtC Images
Authors:
Ayana Kawamura,
Yuma Kinoshita,
Hitoshi Kiya
Abstract:
In this paper, we propose a novel privacy-preserving machine learning scheme with encrypted images, called EtC (Encryption-then-Compression) images. Using machine learning algorithms in cloud environments has been spreading in many fields. However, there are serious issues with it for end users, due to semi-trusted cloud providers. Accordingly, we propose using EtC images, which have been proposed…
▽ More
In this paper, we propose a novel privacy-preserving machine learning scheme with encrypted images, called EtC (Encryption-then-Compression) images. Using machine learning algorithms in cloud environments has been spreading in many fields. However, there are serious issues with it for end users, due to semi-trusted cloud providers. Accordingly, we propose using EtC images, which have been proposed for EtC systems with JPEG compression. In this paper, a novel property of EtC images is considered under the use of z-score normalization. It is demonstrated that the use of EtC images allows us not only to protect visual information of images, but also to preserve both the Euclidean distance and the inner product between vectors. In addition, dimensionality reduction is shown to can be applied to EtC images for fast and accurate matching. In an experiment, the proposed scheme is applied to a facial recognition algorithm with classifiers for confirming the effectiveness of the scheme under the use of support vector machine (SVM) with the kernel trick.
△ Less
Submitted 1 November, 2019;
originally announced November 2019.
-
An Image Fusion Scheme for Single-Shot High Dynamic Range Imaging with Spatially Varying Exposures
Authors:
Chihiro Go,
Yuma Kinoshita,
Sayaka Shiota,
Hitoshi Kiya
Abstract:
This paper proposes a novel multi-exposure image fusion (MEF) scheme for single-shot high dynamic range imaging with spatially varying exposures (SVE). Single-shot imaging with SVE enables us not only to produce images without color saturation regions from a single-shot image, but also to avoid ghost artifacts in the producing ones. However, the number of exposures is generally limited to two, and…
▽ More
This paper proposes a novel multi-exposure image fusion (MEF) scheme for single-shot high dynamic range imaging with spatially varying exposures (SVE). Single-shot imaging with SVE enables us not only to produce images without color saturation regions from a single-shot image, but also to avoid ghost artifacts in the producing ones. However, the number of exposures is generally limited to two, and moreover it is difficult to decide the optimum exposure values before the photographing. In the proposed scheme, a scene segmentation method is applied to input multi-exposure images, and then the luminance of the input images is adjusted according to both of the number of scenes and the relationship between exposure values and pixel values. The proposed method with the luminance adjustment allows us to improve the above two issues. In this paper, we focus on dual-ISO imaging as one of single-shot imaging. In an experiment, the proposed scheme is demonstrated to be effective for single-shot high dynamic range imaging with SVE, compared with conventional MEF schemes with exposure compensation.
△ Less
Submitted 22 August, 2019;
originally announced August 2019.
-
Single-Shot High Dynamic Range Imaging with Spatially Varying Exposures Considering Hue Distortion
Authors:
Chihiro Go,
Yuma Kinoshita,
Sayaka Shiota,
Hitoshi Kiya
Abstract:
We proposes a novel single-shot high dynamic range imaging scheme with spatially varying exposures (SVE) considering hue distortion. Single-shot imaging with SVE enables us to capture multi-exposure images from a single-shot image, so high dynamic range images can be produced without ghost artifacts. However, SVE images have some pixels at which a range supported by camera sensors is exceeded. The…
▽ More
We proposes a novel single-shot high dynamic range imaging scheme with spatially varying exposures (SVE) considering hue distortion. Single-shot imaging with SVE enables us to capture multi-exposure images from a single-shot image, so high dynamic range images can be produced without ghost artifacts. However, SVE images have some pixels at which a range supported by camera sensors is exceeded. Therefore, generated images have some color distortion, so that conventional imaging with SVE has never considered the influence of this range limitation. To overcome this issue, we consider estimating the correct hue of a scene from raw images, and propose a method with the estimated hue information for correcting the hue of SVE images on the constant hue plain in the RGB color space.
△ Less
Submitted 31 July, 2019;
originally announced August 2019.
-
A Color Compensation Method Using Inverse Camera Response Function for Multi-exposure Image Fusion
Authors:
Artit Visavakitcharoen,
Yuma Kinoshita,
Hitoshi Kiya
Abstract:
Multi-exposure image fusion is a method for producing an image with a wide dynamic range by fusing multiple images taken under various exposure values. In this paper, we discuss color distortion included in fused images, and propose a novel color compensation method for multi-exposure image fusion. In the proposed method, an inverse camera response function (CRF) is estimated by using multi-exposu…
▽ More
Multi-exposure image fusion is a method for producing an image with a wide dynamic range by fusing multiple images taken under various exposure values. In this paper, we discuss color distortion included in fused images, and propose a novel color compensation method for multi-exposure image fusion. In the proposed method, an inverse camera response function (CRF) is estimated by using multi-exposure images, and then a high dynamic range (HDR) radiance map is recovered. The color information of the radiance map is applied to images fused by conventional multi-exposure imaging to correct the color distortion. The proposed method can be applied to any existing fusion approaches for improving the quality of the fused images.
△ Less
Submitted 26 July, 2019;
originally announced July 2019.
-
Convolutional Neural Networks Considering Local and Global features for Image Enhancement
Authors:
Yuma Kinoshita,
Hitoshi Kiya
Abstract:
In this paper, we propose a novel convolutional neural network (CNN) architecture considering both local and global features for image enhancement. Most conventional image enhancement methods, including Retinex-based methods, cannot restore lost pixel values caused by clip** and quantizing. CNN-based methods have recently been proposed to solve the problem, but they still have a limited performa…
▽ More
In this paper, we propose a novel convolutional neural network (CNN) architecture considering both local and global features for image enhancement. Most conventional image enhancement methods, including Retinex-based methods, cannot restore lost pixel values caused by clip** and quantizing. CNN-based methods have recently been proposed to solve the problem, but they still have a limited performance due to network architectures not handling global features. To handle both local and global features, the proposed architecture consists of three networks: a local encoder, a global encoder, and a decoder. In addition, high dynamic range (HDR) images are used for generating training data for our networks. The use of HDR images makes it possible to train CNNs with better-quality images than images directly captured with cameras. Experimental results show that the proposed method can produce higher-quality images than conventional image enhancement methods including CNN-based methods, in terms of various objective quality metrics: TMQI, entropy, NIQE, and BRISQUE.
△ Less
Submitted 7 May, 2019;
originally announced May 2019.
-
Privacy-Preserving Deep Neural Networks with Pixel-based Image Encryption Considering Data Augmentation in the Encrypted Domain
Authors:
Warit Sirichotedumrong,
Takahiro Maekawa,
Yuma Kinoshita,
Hitoshi Kiya
Abstract:
We present a novel privacy-preserving scheme for deep neural networks (DNNs) that enables us not to only apply images without visual information to DNNs for both training and testing but to also consider data augmentation in the encrypted domain for the first time. In this paper, a novel pixel-based image encryption method is first proposed for privacy-preserving DNNs. In addition, a novel adaptat…
▽ More
We present a novel privacy-preserving scheme for deep neural networks (DNNs) that enables us not to only apply images without visual information to DNNs for both training and testing but to also consider data augmentation in the encrypted domain for the first time. In this paper, a novel pixel-based image encryption method is first proposed for privacy-preserving DNNs. In addition, a novel adaptation network is considered that reduces the influence of image encryption. In an experiment, the proposed method is applied to a well-known network, ResNet-18, for image classification. The experimental results demonstrate that conventional privacy-preserving machine learning methods including the state-of-the-arts cannot be applied to data augmentation in the encrypted domain and that the proposed method outperforms them in terms of classification accuracy.
△ Less
Submitted 6 May, 2019;
originally announced May 2019.
-
A Noise-aware Enhancement Method for Underexposed Images
Authors:
Chien-Cheng Chien,
Yuma Kinoshita,
Hitoshi Kiya
Abstract:
A novel method of contrast enhancement is proposed for underexposed images, in which heavy noise is hidden. Under low light conditions, images taken by digital cameras have low contrast in dark or bright regions. This is due to a limited dynamic range that imaging sensors have. For these reasons, various contrast enhancement methods have been proposed so far. These methods, however, have two probl…
▽ More
A novel method of contrast enhancement is proposed for underexposed images, in which heavy noise is hidden. Under low light conditions, images taken by digital cameras have low contrast in dark or bright regions. This is due to a limited dynamic range that imaging sensors have. For these reasons, various contrast enhancement methods have been proposed so far. These methods, however, have two problems: (1) The loss of details in bright regions due to over-enhancement of contrast. (2) The noise is amplified in dark regions because conventional enhancement methods do not consider noise included in images. The proposed method aims to overcome these problems. In the proposed method, a shadow-up function is applied to adaptive gamma correction with weighting distribution, and a denoising filter is also used to avoid noise being amplified in dark regions. As a result, the proposed method allows us not only to enhance contrast of dark regions, but also to avoid amplifying noise, even under strong noise environments.
△ Less
Submitted 24 April, 2019;
originally announced April 2019.
-
Scene Segmentation-Based Luminance Adjustment for Multi-Exposure Image Fusion
Authors:
Yuma Kinoshita,
Hitoshi Kiya
Abstract:
We propose a novel method for adjusting luminance for multi-exposure image fusion. For the adjustment, two novel scene segmentation approaches based on luminance distribution are also proposed. Multi-exposure image fusion is a method for producing images that are expected to be more informative and perceptually appealing than any of the input ones, by directly fusing photos taken with different ex…
▽ More
We propose a novel method for adjusting luminance for multi-exposure image fusion. For the adjustment, two novel scene segmentation approaches based on luminance distribution are also proposed. Multi-exposure image fusion is a method for producing images that are expected to be more informative and perceptually appealing than any of the input ones, by directly fusing photos taken with different exposures. However, existing fusion methods often produce unclear fused images when input images do not have a sufficient number of different exposure levels. In this paper, we point out that adjusting the luminance of input images makes it possible to improve the quality of the final fused images. This insight is the basis of the proposed method. The proposed method enables us to produce high-quality images, even when undesirable inputs are given. Visual comparison results show that the proposed method can produce images that clearly represent a whole scene. In addition, multi-exposure image fusion with the proposed method outperforms state-of-the-art fusion methods in terms of MEF-SSIM, discrete entropy, tone mapped image quality index, and statistical naturalness.
△ Less
Submitted 18 March, 2019;
originally announced March 2019.
-
Deep Inverse Tone Map** Using LDR Based Learning for Estimating HDR Images with Absolute Luminance
Authors:
Yuma Kinoshita,
Hitoshi Kiya
Abstract:
In this paper, a novel inverse tone map** method using a convolutional neural network (CNN) with LDR based learning is proposed. In conventional inverse tone map** with CNNs, generated HDR images cannot have absolute luminance, although relative luminance can. Moreover, loss functions suitable for learning HDR images are problematic, so it is difficult to train CNNs by directly using HDR image…
▽ More
In this paper, a novel inverse tone map** method using a convolutional neural network (CNN) with LDR based learning is proposed. In conventional inverse tone map** with CNNs, generated HDR images cannot have absolute luminance, although relative luminance can. Moreover, loss functions suitable for learning HDR images are problematic, so it is difficult to train CNNs by directly using HDR images. In contrast, the proposed method enables us not only to estimate absolute luminance, but also to train a CNN by using LDR images. The CNN used in the proposed method learns a transformation from various input LDR images to LDR images mapped by Reinhard's global operator. Experimental results show that HDR images generated by the proposed method have higher-quality than HDR ones generated by conventional inverse tone map** methods,in terms of HDR-VDP-2.2 and PU encoding + MS-SSIM.
△ Less
Submitted 28 February, 2019;
originally announced March 2019.
-
Analysis of Quantum Multi-Prover Zero-Knowledge Systems: Elimination of the Honest Condition and Computational Zero-Knowledge Systems for QMIP*
Authors:
Yusuke Kinoshita
Abstract:
Zero-knowledge and multi-prover systems are both central notions in classical and quantum complexity theory. There is, however, little research in quantum multi-prover zero-knowledge systems. This paper studies complexity-theoretical aspects of the quantum multi-prover zero-knowledge systems. This paper has two results:
1.QMIP* systems with honest zero-knowledge can be converted into general zer…
▽ More
Zero-knowledge and multi-prover systems are both central notions in classical and quantum complexity theory. There is, however, little research in quantum multi-prover zero-knowledge systems. This paper studies complexity-theoretical aspects of the quantum multi-prover zero-knowledge systems. This paper has two results:
1.QMIP* systems with honest zero-knowledge can be converted into general zero-knowledge systems without any assumptions.
2.QMIP* has computational quantum zero-knowledge systems if a natural computational conjecture holds.
One of the main tools is a test (called the GHZ test) that uses GHZ states shared by the provers, which prevents the verifier's attack in the above two results. Another main tool is what we call the Local Hamiltonian based Interactive protocol (LHI protocol). The LHI protocol makes previous research for Local Hamiltonians applicable to check the history state of interactive proofs, and we then apply Broadbent et al.'s zero-knowledge protocol for QMA \cite{BJSW} to quantum multi-prover systems in order to obtain the second result.
△ Less
Submitted 27 February, 2019;
originally announced February 2019.
-
Image Enhancement Network Trained by Using HDR images
Authors:
Yuma Kinoshita,
Hitoshi Kiya
Abstract:
In this paper, a novel image enhancement network is proposed, where HDR images are used for generating training data for our network. Most of conventional image enhancement methods, including Retinex based methods, do not take into account restoring lost pixel values caused by clip** and quantizing. In addition, recently proposed CNN based methods still have a limited scope of application or a l…
▽ More
In this paper, a novel image enhancement network is proposed, where HDR images are used for generating training data for our network. Most of conventional image enhancement methods, including Retinex based methods, do not take into account restoring lost pixel values caused by clip** and quantizing. In addition, recently proposed CNN based methods still have a limited scope of application or a limited performance, due to network architectures. In contrast, the proposed method have a higher performance and a simpler network architecture than existing CNN based methods. Moreover, the proposed method enables us to restore lost pixel values. Experimental results show that the proposed method can provides higher-quality images than conventional image enhancement methods including a CNN based method, in terms of TMQI and NIQE.
△ Less
Submitted 24 January, 2019; v1 submitted 17 January, 2019;
originally announced January 2019.
-
A Retinex-based Image Enhancement Scheme with Noise Aware Shadow-up Function
Authors:
Chien Cheng Chien,
Yuma Kinoshita,
Sayaka Shiota,
Hitoshi Kiya
Abstract:
This paper proposes a novel image contrast enhancement method based on both a noise aware shadow-up function and Retinex (retina and cortex) decomposition. Under low light conditions, images taken by digital cameras have low contrast in dark or bright regions. This is due to a limited dynamic range that imaging sensors have. For this reason, various contrast enhancement methods have been proposed.…
▽ More
This paper proposes a novel image contrast enhancement method based on both a noise aware shadow-up function and Retinex (retina and cortex) decomposition. Under low light conditions, images taken by digital cameras have low contrast in dark or bright regions. This is due to a limited dynamic range that imaging sensors have. For this reason, various contrast enhancement methods have been proposed. Our proposed method can enhance the contrast of images without not only over-enhancement but also noise amplification. In the proposed method, an image is decomposed into illumination layer and reflectance layer based on the retinex theory, and lightness information of the illumination layer is adjusted. A shadow-up function is used for preventing over-enhancement. The proposed map** function, designed by using a noise aware histogram, allows not only to enhance contrast of dark region, but also to avoid amplifying noise, even under strong noise environments.
△ Less
Submitted 8 November, 2018;
originally announced November 2018.
-
A Pseudo Multi-Exposure Fusion Method Using Single Image
Authors:
Yuma Kinoshita,
Sayaka Shiota,
Hitoshi Kiya
Abstract:
This paper proposes a novel pseudo multi-exposure image fusion method based on a single image. Multi-exposure image fusion is used to produce images without saturation regions, by using photos with different exposures. However, it is difficult to take photos suited for the multi-exposure image fusion when we take a photo of dynamic scenes or record a video. In addition, the multi-exposure image fu…
▽ More
This paper proposes a novel pseudo multi-exposure image fusion method based on a single image. Multi-exposure image fusion is used to produce images without saturation regions, by using photos with different exposures. However, it is difficult to take photos suited for the multi-exposure image fusion when we take a photo of dynamic scenes or record a video. In addition, the multi-exposure image fusion cannot be applied to existing images with a single exposure or videos. The proposed method enables us to produce pseudo multi-exposure images from a single image. To produce multi-exposure images, the proposed method utilizes the relationship between the exposure values and pixel values, which is obtained by assuming that a digital camera has a linear response function. Moreover, it is shown that the use of a local contrast enhancement method allows us to produce pseudo multi-exposure images with higher quality. Most of conventional multi-exposure image fusion methods are also applicable to the proposed multi-exposure images. Experimental results show the effectiveness of the proposed method by comparing the proposed one with conventional ones.
△ Less
Submitted 1 August, 2018;
originally announced August 2018.
-
Multi-Exposure Image Fusion Based on Exposure Compensation
Authors:
Yuma Kinoshita,
Taichi Yoshida,
Sayaka Shiota,
Hitoshi Kiya
Abstract:
This paper proposes a novel multi-exposure image fusion method based on exposure compensation. Multi-exposure image fusion is a method to produce images without color saturation regions, by using photos with different exposures. However, in conventional works, it is unclear how to determine appropriate exposure values, and moreover, it is difficult to set appropriate exposure values at the time of…
▽ More
This paper proposes a novel multi-exposure image fusion method based on exposure compensation. Multi-exposure image fusion is a method to produce images without color saturation regions, by using photos with different exposures. However, in conventional works, it is unclear how to determine appropriate exposure values, and moreover, it is difficult to set appropriate exposure values at the time of photographing due to time constraints. In the proposed method, the luminance of the input multi-exposure images is adjusted on the basis of the relationship between exposure values and pixel values, where the relationship is obtained by assuming that a digital camera has a linear response function. The use of a local contrast enhancement method is also considered to improve input multi-exposure images. The compensated images are finally combined by one of existing multi-exposure image fusion methods. In some experiments, the effectiveness of the proposed method are evaluated in terms of the tone mapped image quality index, statistical naturalness, and discrete entropy, by comparing the proposed one with conventional ones.
△ Less
Submitted 23 June, 2018;
originally announced June 2018.
-
Automatic Exposure Compensation for Multi-Exposure Image Fusion
Authors:
Yuma Kinoshita,
Sayaka Shiota,
Hitoshi Kiya
Abstract:
This paper proposes a novel luminance adjustment method based on automatic exposure compensation for multi-exposure image fusion. Multi-exposure image fusion is a method to produce images without saturation regions, by using photos with different exposures. In conventional works, it has been pointed out that the quality of those multi-exposure images can be improved by adjusting the luminance of t…
▽ More
This paper proposes a novel luminance adjustment method based on automatic exposure compensation for multi-exposure image fusion. Multi-exposure image fusion is a method to produce images without saturation regions, by using photos with different exposures. In conventional works, it has been pointed out that the quality of those multi-exposure images can be improved by adjusting the luminance of them. However, how to determine the degree of adjustment has never been discussed. This paper therefore proposes a way to automatically determines the degree on the basis of the luminance distribution of input multi-exposure images. Moreover, new weights, called "simple weights", for image fusion are also considered for the proposed luminance adjustment method. Experimental results show that the multi-exposure images adjusted by the proposed method have better quality than the input multi-exposure ones in terms of the well-exposedness. It is also confirmed that the proposed simple weights provide the highest score of statistical naturalness and discrete entropy in all fusion methods.
△ Less
Submitted 28 May, 2018;
originally announced May 2018.
