Showing 1–18 of 18 results for author: Davy, A

Fast, nonlocal and neural: a lightweight high quality solution to image denoising

Authors: Yu Guo, Axel Davy, Gabriele Facciolo, Jean-Michel Morel, Qiyu **

Abstract: With the widespread application of convolutional neural networks (CNNs), the traditional model based denoising algorithms are now outperformed. However, CNNs face two problems. First, they are computationally demanding, which makes their deployment especially difficult for mobile terminals. Second, experimental evidence shows that CNNs often over-smooth regular textures present in images, in contr… ▽ More With the widespread application of convolutional neural networks (CNNs), the traditional model based denoising algorithms are now outperformed. However, CNNs face two problems. First, they are computationally demanding, which makes their deployment especially difficult for mobile terminals. Second, experimental evidence shows that CNNs often over-smooth regular textures present in images, in contrast to traditional non-local models. In this letter, we propose a solution to both issues by combining a nonlocal algorithm with a lightweight residual CNN. This solution gives full latitude to the advantages of both models. We apply this framework to two GPU implementations of classic nonlocal algorithms (NLM and BM3D) and observe a substantial gain in both cases, performing better than the state-of-the-art with low computational requirements. Our solution is between 10 and 20 times faster than CNNs with equivalent performance and attains higher PSNR. In addition the final method shows a notable gain on images containing complex textures like the ones of the MIT Moire dataset. △ Less

Submitted 6 March, 2024; originally announced March 2024.

Comments: 5 pages. This paper was accepted by IEEE Signal Processing Letters on July 1, 2021

Journal ref: IEEE Signal Processing Letters, 2021, 28:1515-1519

On the Importance of Large Objects in CNN Based Object Detection Algorithms

Authors: Ahmed Ben Saad, Gabriele Facciolo, Axel Davy

Abstract: Object detection models, a prominent class of machine learning algorithms, aim to identify and precisely locate objects in images or videos. However, this task might yield uneven performances sometimes caused by the objects sizes and the quality of the images and labels used for training. In this paper, we highlight the importance of large objects in learning features that are critical for all siz… ▽ More Object detection models, a prominent class of machine learning algorithms, aim to identify and precisely locate objects in images or videos. However, this task might yield uneven performances sometimes caused by the objects sizes and the quality of the images and labels used for training. In this paper, we highlight the importance of large objects in learning features that are critical for all sizes. Given these findings, we propose to introduce a weighting term into the training loss. This term is a function of the object area size. We show that giving more weight to large objects leads to improved detection scores across all object sizes and so an overall improvement in Object Detectors performances (+2 p.p. of mAP on small objects, +2 p.p. on medium and +4 p.p. on large on COCO val 2017 with InternImage-T). Additional experiments and ablation studies with different models and on a different dataset further confirm the robustness of our findings. △ Less

Submitted 20 November, 2023; originally announced November 2023.

Journal ref: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Jan 2024, WAIKOLOA, HAWAII, United States

L1BSR: Exploiting Detector Overlap for Self-Supervised Single-Image Super-Resolution of Sentinel-2 L1B Imagery

Authors: Ngoc Long Nguyen, Jérémy Anger, Axel Davy, Pablo Arias, Gabriele Facciolo

Abstract: High-resolution satellite imagery is a key element for many Earth monitoring applications. Satellites such as Sentinel-2 feature characteristics that are favorable for super-resolution algorithms such as aliasing and band-misalignment. Unfortunately the lack of reliable high-resolution (HR) ground truth limits the application of deep learning methods to this task. In this work we propose L1BSR, a… ▽ More High-resolution satellite imagery is a key element for many Earth monitoring applications. Satellites such as Sentinel-2 feature characteristics that are favorable for super-resolution algorithms such as aliasing and band-misalignment. Unfortunately the lack of reliable high-resolution (HR) ground truth limits the application of deep learning methods to this task. In this work we propose L1BSR, a deep learning-based method for single-image super-resolution and band alignment of Sentinel-2 L1B 10m bands. The method is trained with self-supervision directly on real L1B data by leveraging overlap** areas in L1B images produced by adjacent CMOS detectors, thus not requiring HR ground truth. Our self-supervised loss is designed to enforce the super-resolved output image to have all the bands correctly aligned. This is achieved via a novel cross-spectral registration network (CSR) which computes an optical flow between images of different spectral bands. The CSR network is also trained with self-supervision using an Anchor-Consistency loss, which we also introduce in this work. We demonstrate the performance of the proposed approach on synthetic and real L1B data, where we show that it obtains comparable results to supervised methods. △ Less

Submitted 17 April, 2023; v1 submitted 13 April, 2023; originally announced April 2023.

Comments: EarthVision 2023

A Graph Neural Networks based Framework for Topology-Aware Proactive SLA Management in a Latency Critical NFV Application Use-case

Authors: Nikita Jalodia, Mohit Taneja, Alan Davy

Abstract: Recent advancements in the rollout of 5G and 6G have led to the emergence of a new range of latency-critical applications delivered via a Network Function Virtualization (NFV) enabled paradigm of flexible and softwarized communication networks. Evolving verticals like telecommunications, smart grid, virtual reality (VR), industry 4.0, automated vehicles, etc. are driven by the vision of low latenc… ▽ More Recent advancements in the rollout of 5G and 6G have led to the emergence of a new range of latency-critical applications delivered via a Network Function Virtualization (NFV) enabled paradigm of flexible and softwarized communication networks. Evolving verticals like telecommunications, smart grid, virtual reality (VR), industry 4.0, automated vehicles, etc. are driven by the vision of low latency and high reliability, and there is a wide gap to efficiently bridge the Quality of Service (QoS) constraints for both the service providers and the end-user. In this work, we look to tackle the over-provisioning of latency-critical services by proposing a proactive SLA management framework leveraging Graph Neural Networks (GNN) and Deep Reinforcement Learning (DRL) to balance the trade-off between efficiency and reliability. To summarize our key contributions: 1) we compose a graph-based spatio-temporal multivariate time-series forecasting model with multiple time-step predictions in a multi-output scenario, delivering 74.62% improved performance over the established baseline state-of-art model on the use-case; and 2) we leverage realistic SLA definitions for the use-case to achieve a dynamic SLA-aware oversight for scaling policy management with DRL. △ Less

Submitted 10 November, 2022; originally announced December 2022.

Improving Pixel-Level Contrastive Learning by Leveraging Exogenous Depth Information

Authors: Ahmed Ben Saad, Kristina Prokopetc, Josselin Kherroubi, Axel Davy, Adrien Courtois, Gabriele Facciolo

Abstract: Self-supervised representation learning based on Contrastive Learning (CL) has been the subject of much attention in recent years. This is due to the excellent results obtained on a variety of subsequent tasks (in particular classification), without requiring a large amount of labeled samples. However, most reference CL algorithms (such as SimCLR and MoCo, but also BYOL and Barlow Twins) are not a… ▽ More Self-supervised representation learning based on Contrastive Learning (CL) has been the subject of much attention in recent years. This is due to the excellent results obtained on a variety of subsequent tasks (in particular classification), without requiring a large amount of labeled samples. However, most reference CL algorithms (such as SimCLR and MoCo, but also BYOL and Barlow Twins) are not adapted to pixel-level downstream tasks. One existing solution known as PixPro proposes a pixel-level approach that is based on filtering of pairs of positive/negative image crops of the same image using the distance between the crops in the whole image. We argue that this idea can be further enhanced by incorporating semantic information provided by exogenous data as an additional selection filter, which can be used (at training time) to improve the selection of the pixel-level positive/negative samples. In this paper we will focus on the depth information, which can be obtained by using a depth estimation network or measured from available data (stereovision, parallax motion, LiDAR, etc.). Scene depth can provide meaningful cues to distinguish pixels belonging to different objects based on their depth. We show that using this exogenous information in the contrastive loss leads to improved results and that the learned representations better follow the shapes of objects. In addition, we introduce a multi-scale loss that alleviates the issue of finding the training parameters adapted to different object sizes. We demonstrate the effectiveness of our ideas on the Breakout Segmentation on Borehole Images where we achieve an improvement of 1.9\% over PixPro and nearly 5\% over the supervised baseline. We further validate our technique on the indoor scene segmentation tasks with ScanNet and outdoor scenes with CityScapes ( 1.6\% and 1.1\% improvement over PixPro respectively). △ Less

Submitted 18 November, 2022; originally announced November 2022.

Comments: Accepted for WACV 2023

Self-Supervised Super-Resolution for Multi-Exposure Push-Frame Satellites

Authors: Ngoc Long Nguyen, Jérémy Anger, Axel Davy, Pablo Arias, Gabriele Facciolo

Abstract: Modern Earth observation satellites capture multi-exposure bursts of push-frame images that can be super-resolved via computational means. In this work, we propose a super-resolution method for such multi-exposure sequences, a problem that has received very little attention in the literature. The proposed method can handle the signal-dependent noise in the inputs, process sequences of any length,… ▽ More Modern Earth observation satellites capture multi-exposure bursts of push-frame images that can be super-resolved via computational means. In this work, we propose a super-resolution method for such multi-exposure sequences, a problem that has received very little attention in the literature. The proposed method can handle the signal-dependent noise in the inputs, process sequences of any length, and be robust to inaccuracies in the exposure times. Furthermore, it can be trained end-to-end with self-supervision, without requiring ground truth high resolution frames, which makes it especially suited to handle real data. Central to our method are three key contributions: i) a base-detail decomposition for handling errors in the exposure times, ii) a noise-level-aware feature encoding for improved fusion of frames with varying signal-to-noise ratio and iii) a permutation invariant fusion strategy by temporal pooling operators. We evaluate the proposed method on synthetic and real data and show that it outperforms by a significant margin existing single-exposure approaches that we adapted to the multi-exposure case. △ Less

Submitted 4 May, 2022; originally announced May 2022.

Comments: CVPR 2022

Proba-V-ref: Repurposing the Proba-V challenge for reference-aware super resolution

Authors: Ngoc Long Nguyen, Jérémy Anger, Axel Davy, Pablo Arias, Gabriele Facciolo

Abstract: The PROBA-V Super-Resolution challenge distributes real low-resolution image series and corresponding high-resolution targets to advance research on Multi-Image Super Resolution (MISR) for satellite images. However, in the PROBA-V dataset the low-resolution image corresponding to the high-resolution target is not identified. We argue that in doing so, the challenge ranks the proposed methods not o… ▽ More The PROBA-V Super-Resolution challenge distributes real low-resolution image series and corresponding high-resolution targets to advance research on Multi-Image Super Resolution (MISR) for satellite images. However, in the PROBA-V dataset the low-resolution image corresponding to the high-resolution target is not identified. We argue that in doing so, the challenge ranks the proposed methods not only by their MISR performance, but mainly by the heuristics used to guess which image in the series is the most similar to the high-resolution target. We demonstrate this by improving the performance obtained by the two winners of the challenge only by using a different reference image, which we compute following a simple heuristic. Based on this, we propose PROBA-V-REF a variant of the PROBA-V dataset, in which the reference image in the low-resolution series is provided, and show that the ranking between the methods changes in this setting. This is relevant to many practical use cases of MISR where the goal is to super-resolve a specific image of the series, i.e. the reference is known. The proposed PROBA-V-REF should better reflect the performance of the different methods for this reference-aware MISR problem. △ Less

Submitted 26 January, 2021; v1 submitted 25 January, 2021; originally announced January 2021.

Comments: 5 pages

Synchrotron Microtomography and Neutron Radiography Characterization of the Microstruture and Water Absorption of Concrete from Pompeii

Authors: Ke Xu, Anton S. Tremsin, Jiaqi Li, Daniela M. Ushizima, Catherine A. Davy, Amine Bouterf, Ying Tsun Su, Milena Marroccoli, Anna Maria Mauro, Massimo Osanna, Antonio Telesca, Paulo J. M. Monteiro

Abstract: There is renewed interest in using advanced techniques to characterize ancient Roman concrete. In the present work, samples were drilled from the "Hospitium" in Pompeii and were analyzed by synchrotron microtomography (uCT) and neutron radiography to study how the microstructure, including the presence of induced cracks, affects their water adsorption. The water distribution and absorptivity were… ▽ More There is renewed interest in using advanced techniques to characterize ancient Roman concrete. In the present work, samples were drilled from the "Hospitium" in Pompeii and were analyzed by synchrotron microtomography (uCT) and neutron radiography to study how the microstructure, including the presence of induced cracks, affects their water adsorption. The water distribution and absorptivity were quantified by neutron radiography. The 3D crack propagation, pore size distribution and orientation, tortuosity, and connectivity were analyzed from uCT results using advanced imaging methods. The concrete characterization also included classical methods (e.g., differential thermal-thermogravimetric, X-ray diffractometry, and scanning electron microscopy). Ductile fracture patterns were observed once cracks were introduced. When compared to Portland cement mortar/concrete, Pompeii samples had relatively high porosity, low connectivity, and similar coefficient of capillary penetration. In addition, the permeability was predicted from models based on percolation theory and the pore structure data to evaluate the fluid transport properties. △ Less

Submitted 2 December, 2020; v1 submitted 26 May, 2020; originally announced May 2020.

Journal ref: Cement and Concrete Research 139 (2021) 106282

Self-Supervised training for blind multi-frame video denoising

Authors: Valéry Dewil, Jérémy Anger, Axel Davy, Thibaud Ehret, Pablo Arias, Gabriele Facciolo

Abstract: We propose a self-supervised approach for training multi-frame video denoising networks. These networks predict frame t from a window of frames around t. Our self-supervised approach benefits from the video temporal consistency by penalizing a loss between the predicted frame t and a neighboring target frame, which are aligned using an optical flow. We use the proposed strategy for online internal… ▽ More We propose a self-supervised approach for training multi-frame video denoising networks. These networks predict frame t from a window of frames around t. Our self-supervised approach benefits from the video temporal consistency by penalizing a loss between the predicted frame t and a neighboring target frame, which are aligned using an optical flow. We use the proposed strategy for online internal learning, where a pre-trained network is fine-tuned to denoise a new unknown noise type from a single video. After a few frames, the proposed fine-tuning reaches and sometimes surpasses the performance of a state-of-the-art network trained with supervision. In addition, for a wide range of noise types, it can be applied blindly without knowing the noise distribution. We demonstrate this by showing results on blind denoising of different synthetic and realistic noises. △ Less

Submitted 20 April, 2021; v1 submitted 15 April, 2020; originally announced April 2020.

Comments: 14 pages

Joint Demosaicking and Denoising by Fine-Tuning of Bursts of Raw Images

Authors: Thibaud Ehret, Axel Davy, Pablo Arias, Gabriele Facciolo

Abstract: Demosaicking and denoising are the first steps of any camera image processing pipeline and are key for obtaining high quality RGB images. A promising current research trend aims at solving these two problems jointly using convolutional neural networks. Due to the unavailability of ground truth data these networks cannot be currently trained using real RAW images. Instead, they resort to simulated… ▽ More Demosaicking and denoising are the first steps of any camera image processing pipeline and are key for obtaining high quality RGB images. A promising current research trend aims at solving these two problems jointly using convolutional neural networks. Due to the unavailability of ground truth data these networks cannot be currently trained using real RAW images. Instead, they resort to simulated data. In this paper we present a method to learn demosaicking directly from mosaicked images, without requiring ground truth RGB data. We apply this to learn joint demosaicking and denoising only from RAW images, thus enabling the use of real data. In addition we show that for this application fine-tuning a network to a specific burst improves the quality of restoration for both demosaicking and denoising. △ Less

Submitted 10 September, 2019; v1 submitted 13 May, 2019; originally announced May 2019.

Comments: ICCV 2019

MAC Protocols for Terahertz Communication: A Comprehensive Survey

Authors: Saim Ghafoor, Noureddine Boujnah, Mubashir Husain Rehmani, Alan Davy

Abstract: Terahertz communication is emerging as a future technology to support Terabits per second link with highlighting features as high throughput and negligible latency. However, the unique features of the Terahertz band such as high path loss, scattering and reflection pose new challenges and results in short communication distance. The antenna directionality, in turn, is required to enhance the commu… ▽ More Terahertz communication is emerging as a future technology to support Terabits per second link with highlighting features as high throughput and negligible latency. However, the unique features of the Terahertz band such as high path loss, scattering and reflection pose new challenges and results in short communication distance. The antenna directionality, in turn, is required to enhance the communication distance and to overcome the high path loss. However, these features in combine negate the use of traditional Medium access protocols. Therefore novel MAC protocol designs are required to fully exploit their potential benefits including efficient channel access, control message exchange, link establishment, mobility management, and line-of-sight blockage mitigation. An in-depth survey of Terahertz MAC protocols is presented in this paper. The paper highlights the key features of the Terahertz band which should be considered while designing an efficient Terahertz MAC protocol, and the decisions which if taken at Terahertz MAC layer can enhance the network performance. Different Terahertz applications at macro and nano scales are highlighted with design requirements for their MAC protocols. The MAC protocol design issues and considerations are highlighted. Further, the existing MAC protocols are also classified based on network topology, channel access mechanisms, and link establishment strategies as Transmitter and Receiver initiated communication. The open challenges and future research directions on Terahertz MAC protocols are also highlighted. △ Less

Submitted 16 July, 2020; v1 submitted 25 April, 2019; originally announced April 2019.

Comments: Submitted to IEEE Communication Surveys and Tutorials Journal

Reducing Anomaly Detection in Images to Detection in Noise

Authors: Axel Davy, Thibaud Ehret, Jean-Michel Morel, Mauricio Delbracio

Abstract: Anomaly detectors address the difficult problem of detecting automatically exceptions in an arbitrary background image. Detection methods have been proposed by the thousands because each problem requires a different background model. By analyzing the existing approaches, we show that the problem can be reduced to detecting anomalies in residual images (extracted from the target image) in which noi… ▽ More Anomaly detectors address the difficult problem of detecting automatically exceptions in an arbitrary background image. Detection methods have been proposed by the thousands because each problem requires a different background model. By analyzing the existing approaches, we show that the problem can be reduced to detecting anomalies in residual images (extracted from the target image) in which noise and anomalies prevail. Hence, the general and impossible background modeling problem is replaced by simpler noise modeling, and allows the calculation of rigorous thresholds based on the a contrario detection theory. Our approach is therefore unsupervised and works on arbitrary images. △ Less

Submitted 25 April, 2019; originally announced April 2019.

Comments: ICIP 2018

Journal ref: 25th IEEE International Conference on Image Processing (2018) 1058-1062

Model-blind Video Denoising Via Frame-to-frame Training

Authors: Thibaud Ehret, Axel Davy, Jean-Michel Morel, Gabriele Facciolo, Pablo Arias

Abstract: Modeling the processing chain that has produced a video is a difficult reverse engineering task, even when the camera is available. This makes model based video processing a still more complex task. In this paper we propose a fully blind video denoising method, with two versions off-line and on-line. This is achieved by fine-tuning a pre-trained AWGN denoising network to the video with a novel fra… ▽ More Modeling the processing chain that has produced a video is a difficult reverse engineering task, even when the camera is available. This makes model based video processing a still more complex task. In this paper we propose a fully blind video denoising method, with two versions off-line and on-line. This is achieved by fine-tuning a pre-trained AWGN denoising network to the video with a novel frame-to-frame training strategy. Our denoiser can be used without knowledge of the origin of the video or burst and the post processing steps applied from the camera sensor. The on-line process only requires a couple of frames before achieving visually-pleasing results for a wide range of perturbations. It nonetheless reaches state of the art performance for standard Gaussian noise, and can be used off-line with still better performance. △ Less

Submitted 25 February, 2020; v1 submitted 30 November, 2018; originally announced November 2018.

Comments: CVPR 2019

Non-Local Video Denoising by CNN

Authors: Axel Davy, Thibaud Ehret, Jean-Michel Morel, Pablo Arias, Gabriele Facciolo

Abstract: Non-local patch based methods were until recently state-of-the-art for image denoising but are now outperformed by CNNs. Yet they are still the state-of-the-art for video denoising, as video redundancy is a key factor to attain high denoising performance. The problem is that CNN architectures are hardly compatible with the search for self-similarities. In this work we propose a new and efficient w… ▽ More Non-local patch based methods were until recently state-of-the-art for image denoising but are now outperformed by CNNs. Yet they are still the state-of-the-art for video denoising, as video redundancy is a key factor to attain high denoising performance. The problem is that CNN architectures are hardly compatible with the search for self-similarities. In this work we propose a new and efficient way to feed video self-similarities to a CNN. The non-locality is incorporated into the network via a first non-trainable layer which finds for each patch in the input image its most similar patches in a search region. The central values of these patches are then gathered in a feature vector which is assigned to each image pixel. This information is presented to a CNN which is trained to predict the clean image. We apply the proposed architecture to image and video denoising. For the latter patches are searched for in a 3D spatio-temporal volume. The proposed architecture achieves state-of-the-art results. To the best of our knowledge, this is the first successful application of a CNN to video denoising. △ Less

Submitted 2 July, 2019; v1 submitted 30 November, 2018; originally announced November 2018.

Comments: A shorter version of this work has been accepted at ICIP 2019 (A NON-LOCAL CNN FOR VIDEO DENOISING). The results of v2 were improved compared to v1 and the code was updated accordingly. Code is available at: https://github.com/axeldavy/vnlnet

Image Anomalies: a Review and Synthesis of Detection Methods

Authors: Thibaud Ehret, Axel Davy, Jean-Michel Morel, Mauricio Delbracio

Abstract: We review the broad variety of methods that have been proposed for anomaly detection in images. Most methods found in the literature have in mind a particular application. Yet we show that the methods can be classified mainly by the structural assumption they make on the "normal" image. Five different structural assumptions emerge. Our analysis leads us to reformulate the best representative algor… ▽ More We review the broad variety of methods that have been proposed for anomaly detection in images. Most methods found in the literature have in mind a particular application. Yet we show that the methods can be classified mainly by the structural assumption they make on the "normal" image. Five different structural assumptions emerge. Our analysis leads us to reformulate the best representative algorithms by attaching to them an a contrario detection that controls the number of false positives and thus derive universal detection thresholds. By combining the most general structural assumptions expressing the background's normality with the best proposed statistical detection tools, we end up proposing generic algorithms that seem to generalize or reconcile most methods. We compare the six best representatives of our proposed classes of algorithms on anomalous images taken from classic papers on the subject, and on a synthetic database. Our conclusion is that it is possible to perform automatic anomaly detection on a single image. △ Less

Submitted 3 June, 2019; v1 submitted 7 August, 2018; originally announced August 2018.

Comments: Thibaud Ehret and Axel Davy contributed equally to this work

Software Defined Networks based Smart Grid Communication: A Comprehensive Survey

Authors: Mubashir Husain Rehmani, Alan Davy, Brendan Jennings, Chadi Assi

Abstract: The current power grid is no longer a feasible solution due to ever-increasing user demand of electricity, old infrastructure, and reliability issues and thus require transformation to a better grid a.k.a., smart grid (SG). The key features that distinguish SG from the conventional electrical power grid are its capability to perform two-way communication, demand side management, and real time pric… ▽ More The current power grid is no longer a feasible solution due to ever-increasing user demand of electricity, old infrastructure, and reliability issues and thus require transformation to a better grid a.k.a., smart grid (SG). The key features that distinguish SG from the conventional electrical power grid are its capability to perform two-way communication, demand side management, and real time pricing. Despite all these advantages that SG will bring, there are certain issues which are specific to SG communication system. For instance, network management of current SG systems is complex, time consuming, and done manually. Moreover, SG communication (SGC) system is built on different vendor specific devices and protocols. Therefore, the current SG systems are not protocol independent, thus leading to interoperability issue. Software defined network (SDN) has been proposed to monitor and manage the communication networks globally. This article serves as a comprehensive survey on SDN-based SGC. In this article, we first discuss taxonomy of advantages of SDNbased SGC.We then discuss SDN-based SGC architectures, along with case studies. Our article provides an in-depth discussion on routing schemes for SDN-based SGC. We also provide detailed survey of security and privacy schemes applied to SDN-based SGC. We furthermore present challenges, open issues, and future research directions related to SDN-based SGC. △ Less

Submitted 27 March, 2019; v1 submitted 14 January, 2018; originally announced January 2018.

Comments: Accepted

Journal ref: IEEE Communications Surveys and Tutorials, 2019

A survey of exemplar-based texture synthesis

Authors: Lara Raad, Axel Davy, Agnès Desolneux, Jean-Michel Morel

Abstract: Exemplar-based texture synthesis is the process of generating, from an input sample, new texture images of arbitrary size and which are perceptually equivalent to the sample. The two main approaches are statistics-based methods and patch re-arrangement methods. In the first class, a texture is characterized by a statistical signature; then, a random sampling conditioned to this signature produces… ▽ More Exemplar-based texture synthesis is the process of generating, from an input sample, new texture images of arbitrary size and which are perceptually equivalent to the sample. The two main approaches are statistics-based methods and patch re-arrangement methods. In the first class, a texture is characterized by a statistical signature; then, a random sampling conditioned to this signature produces genuinely different texture images. The second class boils down to a clever "copy-paste" procedure, which stitches together large regions of the sample. Hybrid methods try to combine ideas from both approaches to avoid their hurdles. The recent approaches using convolutional neural networks fit to this classification, some being statistical and others performing patch re-arrangement in the feature space. They produce impressive synthesis on various kinds of textures. Nevertheless, we found that most real textures are organized at multiple scales, with global structures revealed at coarse scales and highly varying details at finer ones. Thus, when confronted with large natural images of textures the results of state-of-the-art methods degrade rapidly, and the problem of modeling them remains wide open. △ Less

Submitted 24 November, 2017; v1 submitted 22 July, 2017; originally announced July 2017.

Comments: v2: Added comments and typos fixes. New section added to describe FRAME. New method presented: CNNMRF

Brain Tumor Segmentation with Deep Neural Networks

Authors: Mohammad Havaei, Axel Davy, David Warde-Farley, Antoine Biard, Aaron Courville, Yoshua Bengio, Chris Pal, Pierre-Marc Jodoin, Hugo Larochelle

Abstract: In this paper, we present a fully automatic brain tumor segmentation method based on Deep Neural Networks (DNNs). The proposed networks are tailored to glioblastomas (both low and high grade) pictured in MR images. By their very nature, these tumors can appear anywhere in the brain and have almost any kind of shape, size, and contrast. These reasons motivate our exploration of a machine learning s… ▽ More In this paper, we present a fully automatic brain tumor segmentation method based on Deep Neural Networks (DNNs). The proposed networks are tailored to glioblastomas (both low and high grade) pictured in MR images. By their very nature, these tumors can appear anywhere in the brain and have almost any kind of shape, size, and contrast. These reasons motivate our exploration of a machine learning solution that exploits a flexible, high capacity DNN while being extremely efficient. Here, we give a description of different model choices that we've found to be necessary for obtaining competitive performance. We explore in particular different architectures based on Convolutional Neural Networks (CNN), i.e. DNNs specifically adapted to image data. We present a novel CNN architecture which differs from those traditionally used in computer vision. Our CNN exploits both local features as well as more global contextual features simultaneously. Also, different from most traditional uses of CNNs, our networks use a final layer that is a convolutional implementation of a fully connected layer which allows a 40 fold speed up. We also describe a 2-phase training procedure that allows us to tackle difficulties related to the imbalance of tumor labels. Finally, we explore a cascade architecture in which the output of a basic CNN is treated as an additional source of information for a subsequent CNN. Results reported on the 2013 BRATS test dataset reveal that our architecture improves over the currently published state-of-the-art while being over 30 times faster. △ Less

Submitted 20 May, 2016; v1 submitted 13 May, 2015; originally announced May 2015.