-
Neural Sculpting: Uncovering hierarchically modular task structure in neural networks through pruning and network analysis
Authors:
Shreyas Malakarjun Patil,
Loizos Michael,
Constantine Dovrolis
Abstract:
Natural target functions and tasks typically exhibit hierarchical modularity -- they can be broken down into simpler sub-functions that are organized in a hierarchy. Such sub-functions have two important features: they have a distinct set of inputs (input-separability) and they are reused as inputs higher in the hierarchy (reusability). Previous studies have established that hierarchically modular…
▽ More
Natural target functions and tasks typically exhibit hierarchical modularity -- they can be broken down into simpler sub-functions that are organized in a hierarchy. Such sub-functions have two important features: they have a distinct set of inputs (input-separability) and they are reused as inputs higher in the hierarchy (reusability). Previous studies have established that hierarchically modular neural networks, which are inherently sparse, offer benefits such as learning efficiency, generalization, multi-task learning, and transfer. However, identifying the underlying sub-functions and their hierarchical structure for a given task can be challenging. The high-level question in this work is: if we learn a task using a sufficiently deep neural network, how can we uncover the underlying hierarchy of sub-functions in that task? As a starting point, we examine the domain of Boolean functions, where it is easier to determine whether a task is hierarchically modular. We propose an approach based on iterative unit and edge pruning (during training), combined with network analysis for module detection and hierarchy inference. Finally, we demonstrate that this method can uncover the hierarchical modularity of a wide range of Boolean functions and two vision tasks based on the MNIST digits dataset.
△ Less
Submitted 27 October, 2023; v1 submitted 28 May, 2023;
originally announced May 2023.
-
PHEW: Constructing Sparse Networks that Learn Fast and Generalize Well without Training Data
Authors:
Shreyas Malakarjun Patil,
Constantine Dovrolis
Abstract:
Methods that sparsify a network at initialization are important in practice because they greatly improve the efficiency of both learning and inference. Our work is based on a recently proposed decomposition of the Neural Tangent Kernel (NTK) that has decoupled the dynamics of the training process into a data-dependent component and an architecture-dependent kernel - the latter referred to as Path…
▽ More
Methods that sparsify a network at initialization are important in practice because they greatly improve the efficiency of both learning and inference. Our work is based on a recently proposed decomposition of the Neural Tangent Kernel (NTK) that has decoupled the dynamics of the training process into a data-dependent component and an architecture-dependent kernel - the latter referred to as Path Kernel. That work has shown how to design sparse neural networks for faster convergence, without any training data, using the Synflow-L2 algorithm. We first show that even though Synflow-L2 is optimal in terms of convergence, for a given network density, it results in sub-networks with "bottleneck" (narrow) layers - leading to poor performance as compared to other data-agnostic methods that use the same number of parameters. Then we propose a new method to construct sparse networks, without any training data, referred to as Paths with Higher-Edge Weights (PHEW). PHEW is a probabilistic network formation method based on biased random walks that only depends on the initial weights. It has similar path kernel properties as Synflow-L2 but it generates much wider layers, resulting in better generalization and performance. PHEW achieves significant improvements over the data-independent SynFlow and SynFlow-L2 methods at a wide range of network densities.
△ Less
Submitted 23 June, 2021; v1 submitted 21 October, 2020;
originally announced October 2020.
-
A Multi-task Framework for Skin Lesion Detection and Segmentation
Authors:
Sulaiman Vesal,
Shreyas Malakarjun Patil,
Nishant Ravikumar,
Andreas Maier
Abstract:
Early detection and segmentation of skin lesions is crucial for timely diagnosis and treatment, necessary to improve the survival rate of patients. However, manual delineation is time consuming and subject to intra- and inter-observer variations among dermatologists. This underlines the need for an accurate and automatic approach to skin lesion segmentation. To tackle this issue, we propose a mult…
▽ More
Early detection and segmentation of skin lesions is crucial for timely diagnosis and treatment, necessary to improve the survival rate of patients. However, manual delineation is time consuming and subject to intra- and inter-observer variations among dermatologists. This underlines the need for an accurate and automatic approach to skin lesion segmentation. To tackle this issue, we propose a multi-task convolutional neural network (CNN) based, joint detection and segmentation framework, designed to initially localize the lesion and subsequently, segment it. A `Faster region-based convolutional neural network' (Faster-RCNN) which comprises a region proposal network (RPN), is used to generate bounding boxes/region proposals, for lesion localization in each image. The proposed regions are subsequently refined using a softmax classifier and a bounding-box regressor. The refined bounding boxes are finally cropped and segmented using `SkinNet', a modified version of U-Net. We trained and evaluated the performance of our network, using the ISBI 2017 challenge and the PH2 datasets, and compared it with the state-of-the-art, using the official test data released as part of the challenge for the former. Our approach outperformed others in terms of Dice coefficients ($>0.93$), Jaccard index ($>0.88$), accuracy ($>0.96$) and sensitivity ($>0.95$), across five-fold cross validation experiments.
△ Less
Submitted 5 August, 2018;
originally announced August 2018.
-
Siamese LSTM based Fiber Structural Similarity Network (FS2Net) for Rotation Invariant Brain Tractography Segmentation
Authors:
Shreyas Malakarjun Patil,
Aditya Nigam,
Arnav Bhavsar,
Chiranjoy Chattopadhyay
Abstract:
In this paper, we propose a novel deep learning architecture combining stacked Bi-directional LSTM and LSTMs with the Siamese network architecture for segmentation of brain fibers, obtained from tractography data, into anatomically meaningful clusters. The proposed network learns the structural difference between fibers of different classes, which enables it to classify fibers with high accuracy.…
▽ More
In this paper, we propose a novel deep learning architecture combining stacked Bi-directional LSTM and LSTMs with the Siamese network architecture for segmentation of brain fibers, obtained from tractography data, into anatomically meaningful clusters. The proposed network learns the structural difference between fibers of different classes, which enables it to classify fibers with high accuracy. Importantly, capturing such deep inter and intra class structural relationship also ensures that the segmentation is robust to relative rotation among test and training data, hence can be used with unregistered data. Our extensive experimentation over order of hundred-thousands of fibers show that the proposed model achieves state-of-the-art results, even in cases of large relative rotations between test and training data.
△ Less
Submitted 28 December, 2017;
originally announced December 2017.
-
BrainSegNet : A Segmentation Network for Human Brain Fiber Tractography Data into Anatomically Meaningful Clusters
Authors:
Tushar Gupta,
Shreyas Malakarjun Patil,
Mukkaram Tailor,
Daksh Thapar,
Aditya Nigam
Abstract:
The segregation of brain fiber tractography data into distinct and anatomically meaningful clusters can help to comprehend the complex brain structure and early investigation and management of various neural disorders. We propose a novel stacked bidirectional long short-term memory(LSTM) based segmentation network, (BrainSegNet) for human brain fiber tractography data classification. We perform a…
▽ More
The segregation of brain fiber tractography data into distinct and anatomically meaningful clusters can help to comprehend the complex brain structure and early investigation and management of various neural disorders. We propose a novel stacked bidirectional long short-term memory(LSTM) based segmentation network, (BrainSegNet) for human brain fiber tractography data classification. We perform a two-level hierarchical classification a) White vs Grey matter (Macro) and b) White matter clusters (Micro). BrainSegNet is trained over three brain tractography data having over 250,000 fibers each. Our experimental evaluation shows that our model achieves state-of-the-art results. We have performed inter as well as intra class testing over three patient's brain tractography data and achieved a high classification accuracy for both macro and micro levels both under intra as well as inter brain testing scenario.
△ Less
Submitted 14 October, 2017;
originally announced October 2017.
-
UBSegNet: Unified Biometric Region of Interest Segmentation Network
Authors:
Ranjeet Ranjan Jha,
Daksh Thapar,
Shreyas Malakarjun Patil,
Aditya Nigam
Abstract:
Digital human identity management, can now be seen as a social necessity, as it is essentially required in almost every public sector such as, financial inclusions, security, banking, social networking e.t.c. Hence, in today's rampantly emerging world with so many adversarial entities, relying on a single biometric trait is being too optimistic. In this paper, we have proposed a novel end-to-end,…
▽ More
Digital human identity management, can now be seen as a social necessity, as it is essentially required in almost every public sector such as, financial inclusions, security, banking, social networking e.t.c. Hence, in today's rampantly emerging world with so many adversarial entities, relying on a single biometric trait is being too optimistic. In this paper, we have proposed a novel end-to-end, Unified Biometric ROI Segmentation Network (UBSegNet), for extracting region of interest from five different biometric traits viz. face, iris, palm, knuckle and 4-slap fingerprint. The architecture of the proposed UBSegNet consists of two stages: (i) Trait classification and (ii) Trait localization. For these stages, we have used a state of the art region based convolutional neural network (RCNN), comprising of three major parts namely convolutional layers, region proposal network (RPN) along with classification and regression heads. The model has been evaluated over various huge publicly available biometric databases. To the best of our knowledge this is the first unified architecture proposed, segmenting multiple biometric traits. It has been tested over around 5000 * 5 = 25,000 images (5000 images per trait) and produces very good results. Our work on unified biometric segmentation, opens up the vast opportunities in the field of multiple biometric traits based authentication systems.
△ Less
Submitted 26 September, 2017;
originally announced September 2017.
