-
Robust and Explainable Fine-Grained Visual Classification with Transfer Learning: A Dual-Carriageway Framework
Authors:
Zheming Zuo,
Joseph Smith,
Jonathan Stonehouse,
Boguslaw Obara
Abstract:
In the realm of practical fine-grained visual classification applications rooted in deep learning, a common scenario involves training a model using a pre-existing dataset. Subsequently, a new dataset becomes available, prompting the desire to make a pivotal decision for achieving enhanced and leveraged inference performance on both sides: Should one opt to train datasets from scratch or fine-tune…
▽ More
In the realm of practical fine-grained visual classification applications rooted in deep learning, a common scenario involves training a model using a pre-existing dataset. Subsequently, a new dataset becomes available, prompting the desire to make a pivotal decision for achieving enhanced and leveraged inference performance on both sides: Should one opt to train datasets from scratch or fine-tune the model trained on the initial dataset using the newly released dataset? The existing literature reveals a lack of methods to systematically determine the optimal training strategy, necessitating explainability. To this end, we present an automatic best-suit training solution searching framework, the Dual-Carriageway Framework (DCF), to fill this gap. DCF benefits from the design of a dual-direction search (starting from the pre-existing or the newly released dataset) where five different training settings are enforced. In addition, DCF is not only capable of figuring out the optimal training strategy with the capability of avoiding overfitting but also yields built-in quantitative and visual explanations derived from the actual input and weights of the trained model. We validated DCF's effectiveness through experiments with three convolutional neural networks (ResNet18, ResNet34 and Inception-v3) on two temporally continued commercial product datasets. Results showed fine-tuning pathways outperformed training-from-scratch ones by up to 2.13% and 1.23% on the pre-existing and new datasets, respectively, in terms of mean accuracy. Furthermore, DCF identified reflection padding as the superior padding method, enhancing testing accuracy by 3.72% on average. This framework stands out for its potential to guide the development of robust and explainable AI solutions in fine-grained visual classification tasks.
△ Less
Submitted 9 May, 2024;
originally announced May 2024.
-
How Quality Affects Deep Neural Networks in Fine-Grained Image Classification
Authors:
Joseph Smith,
Zheming Zuo,
Jonathan Stonehouse,
Boguslaw Obara
Abstract:
In this paper, we propose a No-Reference Image Quality Assessment (NRIQA) guided cut-off point selection (CPS) strategy to enhance the performance of a fine-grained classification system. Scores given by existing NRIQA methods on the same image may vary and not be as independent of natural image augmentations as expected, which weakens their connection and explainability to fine-grained image clas…
▽ More
In this paper, we propose a No-Reference Image Quality Assessment (NRIQA) guided cut-off point selection (CPS) strategy to enhance the performance of a fine-grained classification system. Scores given by existing NRIQA methods on the same image may vary and not be as independent of natural image augmentations as expected, which weakens their connection and explainability to fine-grained image classification. Taking the three most commonly adopted image augmentation configurations -- crop**, rotating, and blurring -- as the entry point, we formulate a two-step mechanism for selecting the most discriminative subset from a given image dataset by considering both the confidence of model predictions and the density distribution of image qualities over several NRIQA methods. Concretely, the cut-off points yielded by those methods are aggregated via majority voting to inform the process of image subset selection. The efficacy and efficiency of such a mechanism have been confirmed by comparing the models being trained on high-quality images against a combination of high- and low-quality ones, with a range of 0.7% to 4.2% improvement on a commercial product dataset in terms of mean accuracy through four deep neural classifiers. The robustness of the mechanism has been proven by the observations that all the selected high-quality images can work jointly with 70% low-quality images with 1.3% of classification precision sacrificed when using ResNet34 in an ablation study.
△ Less
Submitted 9 May, 2024;
originally announced May 2024.
-
Robust 3D U-Net Segmentation of Macular Holes
Authors:
Jonathan Frawley,
Chris G. Willcocks,
Maged Habib,
Caspar Geenen,
David H. Steel,
Boguslaw Obara
Abstract:
Macular holes are a common eye condition which result in visual impairment. We look at the application of deep convolutional neural networks to the problem of macular hole segmentation. We use the 3D U-Net architecture as a basis and experiment with a number of design variants. Manually annotating and measuring macular holes is time consuming and error prone. Previous automated approaches to macul…
▽ More
Macular holes are a common eye condition which result in visual impairment. We look at the application of deep convolutional neural networks to the problem of macular hole segmentation. We use the 3D U-Net architecture as a basis and experiment with a number of design variants. Manually annotating and measuring macular holes is time consuming and error prone. Previous automated approaches to macular hole segmentation take minutes to segment a single 3D scan. Our proposed model generates significantly more accurate segmentations in less than a second. We found that an approach of architectural simplification, by greatly simplifying the network capacity and depth, exceeds both expert performance and state-of-the-art models such as residual 3D U-Nets.
△ Less
Submitted 7 April, 2021; v1 submitted 1 March, 2021;
originally announced March 2021.
-
Not Half Bad: Exploring Half-Precision in Graph Convolutional Neural Networks
Authors:
John Brennan,
Stephen Bonner,
Amir Atapour-Abarghouei,
Philip T Jackson,
Boguslaw Obara,
Andrew Stephen McGough
Abstract:
With the growing significance of graphs as an effective representation of data in numerous applications, efficient graph analysis using modern machine learning is receiving a growing level of attention. Deep learning approaches often operate over the entire adjacency matrix -- as the input and intermediate network layers are all designed in proportion to the size of the adjacency matrix -- leading…
▽ More
With the growing significance of graphs as an effective representation of data in numerous applications, efficient graph analysis using modern machine learning is receiving a growing level of attention. Deep learning approaches often operate over the entire adjacency matrix -- as the input and intermediate network layers are all designed in proportion to the size of the adjacency matrix -- leading to intensive computation and large memory requirements as the graph size increases. It is therefore desirable to identify efficient measures to reduce both run-time and memory requirements allowing for the analysis of the largest graphs possible. The use of reduced precision operations within the forward and backward passes of a deep neural network along with novel specialised hardware in modern GPUs can offer promising avenues towards efficiency. In this paper, we provide an in-depth exploration of the use of reduced-precision operations, easily integrable into the highly popular PyTorch framework, and an analysis of the effects of Tensor Cores on graph convolutional neural networks. We perform an extensive experimental evaluation of three GPU architectures and two widely-used graph analysis tasks (vertex classification and link prediction) using well-known benchmark and synthetically generated datasets. Thus allowing us to make important observations on the effects of reduced-precision operations and Tensor Cores on computational and memory usage of graph convolutional neural networks -- often neglected in the literature.
△ Less
Submitted 23 October, 2020;
originally announced October 2020.
-
Camera Bias in a Fine Grained Classification Task
Authors:
Philip T. Jackson,
Stephen Bonner,
Ning Jia,
Christopher Holder,
Jon Stonehouse,
Boguslaw Obara
Abstract:
We show that correlations between the camera used to acquire an image and the class label of that image can be exploited by convolutional neural networks (CNN), resulting in a model that "cheats" at an image classification task by recognizing which camera took the image and inferring the class label from the camera. We show that models trained on a dataset with camera / label correlations do not g…
▽ More
We show that correlations between the camera used to acquire an image and the class label of that image can be exploited by convolutional neural networks (CNN), resulting in a model that "cheats" at an image classification task by recognizing which camera took the image and inferring the class label from the camera. We show that models trained on a dataset with camera / label correlations do not generalize well to images in which those correlations are absent, nor to images from unencountered cameras. Furthermore, we investigate which visual features they are exploiting for camera recognition. Our experiments present evidence against the importance of global color statistics, lens deformation and chromatic aberration, and in favor of high frequency features, which may be introduced by image processing algorithms built into the cameras.
△ Less
Submitted 16 July, 2020;
originally announced July 2020.
-
Segmentation of Macular Edema Datasets with Small Residual 3D U-Net Architectures
Authors:
Jonathan Frawley,
Chris G. Willcocks,
Maged Habib,
Caspar Geenen,
David H. Steel,
Boguslaw Obara
Abstract:
This paper investigates the application of deep convolutional neural networks with prohibitively small datasets to the problem of macular edema segmentation. In particular, we investigate several different heavily regularized architectures. We find that, contrary to popular belief, neural architectures within this application setting are able to achieve close to human-level performance on unseen t…
▽ More
This paper investigates the application of deep convolutional neural networks with prohibitively small datasets to the problem of macular edema segmentation. In particular, we investigate several different heavily regularized architectures. We find that, contrary to popular belief, neural architectures within this application setting are able to achieve close to human-level performance on unseen test images without requiring large numbers of training examples. Annotating these 3D datasets is difficult, with multiple criteria required. It takes an experienced clinician two days to annotate a single 3D image, whereas our trained model achieves similar performance in less than a second. We found that an approach which uses targeted dataset augmentation, alongside architectural simplification with an emphasis on residual design, has acceptable generalization performance - despite relying on fewer than 15 training examples.
△ Less
Submitted 10 May, 2020;
originally announced May 2020.
-
Temporal Neighbourhood Aggregation: Predicting Future Links in Temporal Graphs via Recurrent Variational Graph Convolutions
Authors:
Stephen Bonner,
Amir Atapour-Abarghouei,
Philip T Jackson,
John Brennan,
Ibad Kureshi,
Georgios Theodoropoulos,
Andrew Stephen McGough,
Boguslaw Obara
Abstract:
Graphs have become a crucial way to represent large, complex and often temporal datasets across a wide range of scientific disciplines. However, when graphs are used as input to machine learning models, this rich temporal information is frequently disregarded during the learning process, resulting in suboptimal performance on certain temporal infernce tasks. To combat this, we introduce Temporal N…
▽ More
Graphs have become a crucial way to represent large, complex and often temporal datasets across a wide range of scientific disciplines. However, when graphs are used as input to machine learning models, this rich temporal information is frequently disregarded during the learning process, resulting in suboptimal performance on certain temporal infernce tasks. To combat this, we introduce Temporal Neighbourhood Aggregation (TNA), a novel vertex representation model architecture designed to capture both topological and temporal information to directly predict future graph states. Our model exploits hierarchical recurrence at different depths within the graph to enable exploration of changes in temporal neighbourhoods, whilst requiring no additional features or labels to be present. The final vertex representations are created using variational sampling and are optimised to directly predict the next graph in the sequence. Our claims are reinforced by extensive experimental evaluation on both real and synthetic benchmark datasets, where our approach demonstrates superior performance compared to competing methods, out-performing them at predicting new temporal edges by as much as 23% on real-world datasets, whilst also requiring fewer overall model parameters.
△ Less
Submitted 21 November, 2019; v1 submitted 21 August, 2019;
originally announced August 2019.
-
TMIXT: A process flow for Transcribing MIXed handwritten and machine-printed Text
Authors:
Fady Medhat,
Mahnaz Mohammadi,
Sardar Jaf,
Chris G. Willcocks,
Toby P. Breckon,
Peter Matthews,
Andrew Stephen McGough,
Georgios Theodoropoulos,
Boguslaw Obara
Abstract:
Handling large corpuses of documents is of significant importance in many fields, no more so than in the areas of crime investigation and defence, where an organisation may be presented with a large volume of scanned documents which need to be processed in a finite time. However, this problem is exacerbated both by the volume, in terms of scanned documents and the complexity of the pages, which ne…
▽ More
Handling large corpuses of documents is of significant importance in many fields, no more so than in the areas of crime investigation and defence, where an organisation may be presented with a large volume of scanned documents which need to be processed in a finite time. However, this problem is exacerbated both by the volume, in terms of scanned documents and the complexity of the pages, which need to be processed. Often containing many different elements, which each need to be processed and understood. Text recognition, which is a primary task of this process, is usually dependent upon the type of text, being either handwritten or machine-printed. Accordingly, the recognition involves prior classification of the text category, before deciding on the recognition method to be applied. This poses a more challenging task if a document contains both handwritten and machine-printed text. In this work, we present a generic process flow for text recognition in scanned documents containing mixed handwritten and machine-printed text without the need to classify text in advance. We realize the proposed process flow using several open-source image processing and text recognition packages1. The evaluation is performed using a specially developed variant, presented in this work, of the IAM handwriting database, where we achieve an average transcription accuracy of nearly 80% for pages containing both printed and handwritten text.
△ Less
Submitted 28 April, 2019;
originally announced April 2019.
-
Phenotypic Profiling of High Throughput Imaging Screens with Generic Deep Convolutional Features
Authors:
Philip T. Jackson,
Yinhai Wang,
Sinead Knight,
Hongming Chen,
Thierry Dorval,
Martin Brown,
Claus Bendtsen,
Boguslaw Obara
Abstract:
While deep learning has seen many recent applications to drug discovery, most have focused on predicting activity or toxicity directly from chemical structure. Phenotypic changes exhibited in cellular images are also indications of the mechanism of action (MoA) of chemical compounds. In this paper, we show how pre-trained convolutional image features can be used to assist scientists in discovering…
▽ More
While deep learning has seen many recent applications to drug discovery, most have focused on predicting activity or toxicity directly from chemical structure. Phenotypic changes exhibited in cellular images are also indications of the mechanism of action (MoA) of chemical compounds. In this paper, we show how pre-trained convolutional image features can be used to assist scientists in discovering interesting chemical clusters for further investigation. Our method reduces the dimensionality of raw fluorescent stained images from a high throughput imaging (HTI) screen, producing an embedding space that groups together images with similar cellular phenotypes. Running standard unsupervised clustering on this embedding space yields a set of distinct phenotypic clusters. This allows scientists to further select and focus on interesting clusters for downstream analyses. We validate the consistency of our embedding space qualitatively with t-sne visualizations, and quantitatively by measuring embedding variance among images that are known to be similar. Results suggested the usefulness of our proposed workflow using deep learning and clustering and it can lead to robust HTI screening and compound triage.
△ Less
Submitted 15 March, 2019;
originally announced March 2019.
-
2D and 3D Vascular Structures Enhancement via Multiscale Fractional Anisotropy Tensor
Authors:
Haifa F. Alhasson,
Shuaa S. Alharbi,
Boguslaw Obara
Abstract:
The detection of vascular structures from noisy images is a fundamental process for extracting meaningful information in many applications. Most well-known vascular enhancing techniques often rely on Hessian-based filters. This paper investigates the feasibility and deficiencies of detecting curve-like structures using a Hessian matrix. The main contribution is a novel enhancement function, which…
▽ More
The detection of vascular structures from noisy images is a fundamental process for extracting meaningful information in many applications. Most well-known vascular enhancing techniques often rely on Hessian-based filters. This paper investigates the feasibility and deficiencies of detecting curve-like structures using a Hessian matrix. The main contribution is a novel enhancement function, which overcomes the deficiencies of established methods. Our approach has been evaluated quantitatively and qualitatively using synthetic examples and a wide range of real 2D and 3D biomedical images. Compared with other existing approaches, the experimental results prove that our proposed approach achieves high-quality curvilinear structure enhancement.
△ Less
Submitted 1 February, 2019;
originally announced February 2019.
-
Temporal Graph Offset Reconstruction: Towards Temporally Robust Graph Representation Learning
Authors:
Stephen Bonner,
John Brennan,
Ibad Kureshi,
Georgios Theodoropoulos,
Andrew Stephen McGough,
Boguslaw Obara
Abstract:
Graphs are a commonly used construct for representing relationships between elements in complex high dimensional datasets. Many real-world phenomenon are dynamic in nature, meaning that any graph used to represent them is inherently temporal. However, many of the machine learning models designed to capture knowledge about the structure of these graphs ignore this rich temporal information when cre…
▽ More
Graphs are a commonly used construct for representing relationships between elements in complex high dimensional datasets. Many real-world phenomenon are dynamic in nature, meaning that any graph used to represent them is inherently temporal. However, many of the machine learning models designed to capture knowledge about the structure of these graphs ignore this rich temporal information when creating representations of the graph. This results in models which do not perform well when used to make predictions about the future state of the graph -- especially when the delta between time stamps is not small. In this work, we explore a novel training procedure and an associated unsupervised model which creates graph representations optimised to predict the future state of the graph. We make use of graph convolutional neural networks to encode the graph into a latent representation, which we then use to train our temporal offset reconstruction method, inspired by auto-encoders, to predict a later time point -- multiple time steps into the future. Using our method, we demonstrate superior performance for the task of future link prediction compared with none-temporal state-of-the-art baselines. We show our approach to be capable of outperforming non-temporal baselines by 38% on a real world dataset.
△ Less
Submitted 20 November, 2018;
originally announced November 2018.
-
Curvilinear Structure Enhancement by Multiscale Top-Hat Tensor in 2D/3D Images
Authors:
Shuaa S. Alharbi,
Cigdem Sazak,
Carl J. Nelson,
Boguslaw Obara
Abstract:
A wide range of biomedical applications requires enhancement, detection, quantification and modelling of curvilinear structures in 2D and 3D images. Curvilinear structure enhancement is a crucial step for further analysis, but many of the enhancement approaches still suffer from contrast variations and noise. This can be addressed using a multiscale approach that produces a better quality enhancem…
▽ More
A wide range of biomedical applications requires enhancement, detection, quantification and modelling of curvilinear structures in 2D and 3D images. Curvilinear structure enhancement is a crucial step for further analysis, but many of the enhancement approaches still suffer from contrast variations and noise. This can be addressed using a multiscale approach that produces a better quality enhancement for low contrast and noisy images compared with a single-scale approach in a wide range of biomedical images. Here, we propose the Multiscale Top-Hat Tensor (MTHT) approach, which combines multiscale morphological filtering with a local tensor representation of curvilinear structures in 2D and 3D images. The proposed approach is validated on synthetic and real data and is also compared to the state-of-the-art approaches. Our results show that the proposed approach achieves high-quality curvilinear structure enhancement in synthetic examples and in a wide range of 2D and 3D images.
△ Less
Submitted 4 January, 2019; v1 submitted 23 September, 2018;
originally announced September 2018.
-
Style Augmentation: Data Augmentation via Style Randomization
Authors:
Philip T. Jackson,
Amir Atapour-Abarghouei,
Stephen Bonner,
Toby Breckon,
Boguslaw Obara
Abstract:
We introduce style augmentation, a new form of data augmentation based on random style transfer, for improving the robustness of convolutional neural networks (CNN) over both classification and regression based tasks. During training, our style augmentation randomizes texture, contrast and color, while preserving shape and semantic content. This is accomplished by adapting an arbitrary style trans…
▽ More
We introduce style augmentation, a new form of data augmentation based on random style transfer, for improving the robustness of convolutional neural networks (CNN) over both classification and regression based tasks. During training, our style augmentation randomizes texture, contrast and color, while preserving shape and semantic content. This is accomplished by adapting an arbitrary style transfer network to perform style randomization, by sampling input style embeddings from a multivariate normal distribution instead of inferring them from a style image. In addition to standard classification experiments, we investigate the effect of style augmentation (and data augmentation generally) on domain transfer tasks. We find that data augmentation significantly improves robustness to domain shift, and can be used as a simple, domain agnostic alternative to domain adaptation. Comparing style augmentation against a mix of seven traditional augmentation techniques, we find that it can be readily combined with them to improve network performance. We validate the efficacy of our technique with domain transfer experiments in classification and monocular depth estimation, illustrating consistent improvements in generalization.
△ Less
Submitted 12 April, 2019; v1 submitted 14 September, 2018;
originally announced September 2018.
-
Exploring the Semantic Content of Unsupervised Graph Embeddings: An Empirical Study
Authors:
Stephen Bonner,
Ibad Kureshi,
John Brennan,
Georgios Theodoropoulos,
Andrew Stephen McGough,
Boguslaw Obara
Abstract:
Graph embeddings have become a key and widely used technique within the field of graph mining, proving to be successful across a broad range of domains including social, citation, transportation and biological. Graph embedding techniques aim to automatically create a low-dimensional representation of a given graph, which captures key structural elements in the resulting embedding space. However, t…
▽ More
Graph embeddings have become a key and widely used technique within the field of graph mining, proving to be successful across a broad range of domains including social, citation, transportation and biological. Graph embedding techniques aim to automatically create a low-dimensional representation of a given graph, which captures key structural elements in the resulting embedding space. However, to date, there has been little work exploring exactly which topological structures are being learned in the embeddings process. In this paper, we investigate if graph embeddings are approximating something analogous with traditional vertex level graph features. If such a relationship can be found, it could be used to provide a theoretical insight into how graph embedding approaches function. We perform this investigation by predicting known topological features, using supervised and unsupervised methods, directly from the embedding space. If a map** between the embeddings and topological features can be found, then we argue that the structural information encapsulated by the features is represented in the embedding space. To explore this, we present extensive experimental evaluation from five state-of-the-art unsupervised graph embedding techniques, across a range of empirical graph datasets, measuring a selection of topological features. We demonstrate that several topological features are indeed being approximated by the embedding space, allowing key insight into how graph embeddings create good representations.
△ Less
Submitted 19 June, 2018;
originally announced June 2018.
-
The Multiscale Bowler-Hat Transform for Vessel Enhancement in 3D Biomedical Images
Authors:
Cigdem Sazak,
Carl J. Nelson,
Boguslaw Obara
Abstract:
Enhancement and detection of 3D vessel-like structures has long been an open problem as most existing image processing methods fail in many aspects, including a lack of uniform enhancement between vessels of different radii and a lack of enhancement at the junctions.
Here, we propose a method based on mathematical morphology to enhance 3D vessel-like structures in biomedical images. The proposed…
▽ More
Enhancement and detection of 3D vessel-like structures has long been an open problem as most existing image processing methods fail in many aspects, including a lack of uniform enhancement between vessels of different radii and a lack of enhancement at the junctions.
Here, we propose a method based on mathematical morphology to enhance 3D vessel-like structures in biomedical images. The proposed method, 3D bowler-hat transform, combines sphere and line structuring elements to enhance vessel-like structures. The proposed method is validated on synthetic and real data and compared with state-of-the-art methods.
Our results show that the proposed method achieves a high-quality vessel-like structures enhancement in both synthetic and real biomedical images, and is able to cope with variations in vessels thickness throughout vascular networks while remaining robust at junctions.
△ Less
Submitted 14 February, 2018;
originally announced February 2018.
-
The Multiscale Bowler-Hat Transform for Blood Vessel Enhancement in Retinal Images
Authors:
Çiğdem Sazak,
Carl J. Nelson,
Boguslaw Obara
Abstract:
Enhancement, followed by segmentation, quantification and modelling, of blood vessels in retinal images plays an essential role in computer-aid retinopathy diagnosis. In this paper, we introduce a new vessel enhancement method which is the bowler-hat transform based on mathematical morphology. The proposed method combines different structuring elements to detect innate features of vessel-like stru…
▽ More
Enhancement, followed by segmentation, quantification and modelling, of blood vessels in retinal images plays an essential role in computer-aid retinopathy diagnosis. In this paper, we introduce a new vessel enhancement method which is the bowler-hat transform based on mathematical morphology. The proposed method combines different structuring elements to detect innate features of vessel-like structures. We evaluate the proposed method qualitatively and quantitatively, and compare it with the existing, state-of-the-art methods using both synthetic and real datasets. Our results show that the proposed method achieves high-quality vessel-like structure enhancement in both synthetic examples and in clinically relevant retinal images, and is shown to be able to detect fine vessels while remaining robust at junctions.
△ Less
Submitted 28 March, 2018; v1 submitted 16 September, 2017;
originally announced September 2017.
