Showing 1–28 of 28 results for author: Noy, A

Correlating fluorescence microscopy, optical and magnetic tweezers to study single chiral biopolymers such as DNA

Authors: Jack W Shepherd, Sebastien Guilbaud, Zhaokun Zhou, Jamieson Howard, Matthew Burman, Charley Schaefer, Adam Kerrigan, Clare Steele-King, Agnes Noy, Mark C Leake

Abstract: Biopolymer topology is critical for determining interactions inside cell environments, exemplified by DNA where its response to mechanical perturbation is as important as biochemical properties to its cellular roles. The dynamic structures of chiral biopolymers exhibit complex dependence with extension and torsion, however the physical mechanisms underpinning the emergence of structural motifs upo… ▽ More Biopolymer topology is critical for determining interactions inside cell environments, exemplified by DNA where its response to mechanical perturbation is as important as biochemical properties to its cellular roles. The dynamic structures of chiral biopolymers exhibit complex dependence with extension and torsion, however the physical mechanisms underpinning the emergence of structural motifs upon physiological twisting and stretching are poorly understood due to technological limitations in correlating force, torque and spatial localization information. We present COMBI-Tweez (Combined Optical and Magnetic BIomolecule TWEEZers), a transformative tool that overcomes these challenges by integrating optical trap**, time-resolved electromagnetic tweezers, and fluorescence microscopy, demonstrated on single DNA molecules, that can controllably form and visualise higher order structural motifs including plectonemes. This technology combined with cutting-edge MD simulations provides quantitative insight into complex dynamic structures relevant to DNA cellular processes and can be adapted to study a range of filamentous biopolymers. △ Less

Submitted 5 March, 2024; v1 submitted 18 January, 2024; originally announced January 2024.

Fluidification of entanglements by a DNA bending protein

Authors: Yair A. G. Fosado, Jamieson Howard, Simon Weir, Agnes Noy, Mark C Leake, Davide Michieletto

Abstract: In spite of the nanoscale and single-molecule insights into how nucleoid associated proteins (NAPs) interact with DNA, their role in modulating the mesoscale viscoelasticity of the entangled genome in vivo has been overlooked so far. By combining microrheology and molecular dynamics simulation we find that the important NAP called Integration Host Factor (IHF) lowers the viscosity of entangled… ▽ More In spite of the nanoscale and single-molecule insights into how nucleoid associated proteins (NAPs) interact with DNA, their role in modulating the mesoscale viscoelasticity of the entangled genome in vivo has been overlooked so far. By combining microrheology and molecular dynamics simulation we find that the important NAP called Integration Host Factor (IHF) lowers the viscosity of entangled $λ$DNA 20-fold at physiological concentrations and stoichiometries. We argue that IHF may act as a "genomic fluidiser", reducing the effective viscosity of the nucleoid $\sim$200-fold. Our results suggest a previously unappreciated key role of IHF in regulating DNA dynamics and re-organisation in vivo △ Less

Submitted 7 February, 2023; v1 submitted 6 May, 2022; originally announced May 2022.

End-to-End Audio Strikes Back: Boosting Augmentations Towards An Efficient Audio Classification Network

Authors: Avi Gazneli, Gadi Zimerman, Tal Ridnik, Gilad Sharir, Asaf Noy

Abstract: While efficient architectures and a plethora of augmentations for end-to-end image classification tasks have been suggested and heavily investigated, state-of-the-art techniques for audio classifications still rely on numerous representations of the audio signal together with large architectures, fine-tuned from large datasets. By utilizing the inherited lightweight nature of audio and novel audio… ▽ More While efficient architectures and a plethora of augmentations for end-to-end image classification tasks have been suggested and heavily investigated, state-of-the-art techniques for audio classifications still rely on numerous representations of the audio signal together with large architectures, fine-tuned from large datasets. By utilizing the inherited lightweight nature of audio and novel audio augmentations, we were able to present an efficient end-to-end network with strong generalization ability. Experiments on a variety of sound classification sets demonstrate the effectiveness and robustness of our approach, by achieving state-of-the-art results in various settings. Public code is available at: \href{https://github.com/Alibaba-MIIL/AudioClassfication}{this http url} △ Less

Submitted 5 July, 2022; v1 submitted 25 April, 2022; originally announced April 2022.

Diverse Imagenet Models Transfer Better

Authors: Niv Nayman, Avram Golbert, Asaf Noy, Tan **, Lihi Zelnik-Manor

Abstract: A commonly accepted hypothesis is that models with higher accuracy on Imagenet perform better on other downstream tasks, leading to much research dedicated to optimizing Imagenet accuracy. Recently this hypothesis has been challenged by evidence showing that self-supervised models transfer better than their supervised counterparts, despite their inferior Imagenet accuracy. This calls for identifyi… ▽ More A commonly accepted hypothesis is that models with higher accuracy on Imagenet perform better on other downstream tasks, leading to much research dedicated to optimizing Imagenet accuracy. Recently this hypothesis has been challenged by evidence showing that self-supervised models transfer better than their supervised counterparts, despite their inferior Imagenet accuracy. This calls for identifying the additional factors, on top of Imagenet accuracy, that make models transferable. In this work we show that high diversity of the features learnt by the model promotes transferability jointly with Imagenet accuracy. Encouraged by the recent transferability results of self-supervised models, we propose a method that combines self-supervised and supervised pretraining to generate models with both high diversity and high accuracy, and as a result high transferability. We demonstrate our results on several architectures and multiple downstream tasks, including both single-label and multi-label classification. △ Less

Submitted 19 April, 2022; originally announced April 2022.

MSC Class: 68T07; 68T10; 68T45 ACM Class: I.2.10; I.2.6; I.4.10

Solving ImageNet: a Unified Scheme for Training any Backbone to Top Results

Authors: Tal Ridnik, Hussam Lawen, Emanuel Ben-Baruch, Asaf Noy

Abstract: ImageNet serves as the primary dataset for evaluating the quality of computer-vision models. The common practice today is training each architecture with a tailor-made scheme, designed and tuned by an expert. In this paper, we present a unified scheme for training any backbone on ImageNet. The scheme, named USI (Unified Scheme for ImageNet), is based on knowledge distillation and modern tricks. It… ▽ More ImageNet serves as the primary dataset for evaluating the quality of computer-vision models. The common practice today is training each architecture with a tailor-made scheme, designed and tuned by an expert. In this paper, we present a unified scheme for training any backbone on ImageNet. The scheme, named USI (Unified Scheme for ImageNet), is based on knowledge distillation and modern tricks. It requires no adjustments or hyper-parameters tuning between different models, and is efficient in terms of training times. We test USI on a wide variety of architectures, including CNNs, Transformers, Mobile-oriented and MLP-only. On all models tested, USI outperforms previous state-of-the-art results. Hence, we are able to transform training on ImageNet from an expert-oriented task to an automatic seamless routine. Since USI accepts any backbone and trains it to top results, it also enables to perform methodical comparisons, and identify the most efficient backbones along the speed-accuracy Pareto curve. Implementation is available at:https://github.com/Alibaba-MIIL/Solving_ImageNet △ Less

Submitted 12 May, 2022; v1 submitted 7 April, 2022; originally announced April 2022.

ML-Decoder: Scalable and Versatile Classification Head

Authors: Tal Ridnik, Gilad Sharir, Avi Ben-Cohen, Emanuel Ben-Baruch, Asaf Noy

Abstract: In this paper, we introduce ML-Decoder, a new attention-based classification head. ML-Decoder predicts the existence of class labels via queries, and enables better utilization of spatial data compared to global average pooling. By redesigning the decoder architecture, and using a novel group-decoding scheme, ML-Decoder is highly efficient, and can scale well to thousands of classes. Compared to u… ▽ More In this paper, we introduce ML-Decoder, a new attention-based classification head. ML-Decoder predicts the existence of class labels via queries, and enables better utilization of spatial data compared to global average pooling. By redesigning the decoder architecture, and using a novel group-decoding scheme, ML-Decoder is highly efficient, and can scale well to thousands of classes. Compared to using a larger backbone, ML-Decoder consistently provides a better speed-accuracy trade-off. ML-Decoder is also versatile - it can be used as a drop-in replacement for various classification heads, and generalize to unseen classes when operated with word queries. Novel query augmentations further improve its generalization ability. Using ML-Decoder, we achieve state-of-the-art results on several classification tasks: on MS-COCO multi-label, we reach 91.4% mAP; on NUS-WIDE zero-shot, we reach 31.1% ZSL mAP; and on ImageNet single-label, we reach with vanilla ResNet50 backbone a new top score of 80.7%, without extra data or distillation. Public code is available at: https://github.com/Alibaba-MIIL/ML_Decoder △ Less

Submitted 31 December, 2021; v1 submitted 25 November, 2021; originally announced November 2021.

BINAS: Bilinear Interpretable Neural Architecture Search

Authors: Niv Nayman, Yonathan Aflalo, Asaf Noy, Rong **, Lihi Zelnik-Manor

Abstract: Practical use of neural networks often involves requirements on latency, energy and memory among others. A popular approach to find networks under such requirements is through constrained Neural Architecture Search (NAS). However, previous methods use complicated predictors for the accuracy of the network. Those predictors are hard to interpret and sensitive to many hyperparameters to be tuned, he… ▽ More Practical use of neural networks often involves requirements on latency, energy and memory among others. A popular approach to find networks under such requirements is through constrained Neural Architecture Search (NAS). However, previous methods use complicated predictors for the accuracy of the network. Those predictors are hard to interpret and sensitive to many hyperparameters to be tuned, hence, the resulting accuracy of the generated models is often harmed. In this work we resolve this by introducing Bilinear Interpretable Neural Architecture Search (BINAS), that is based on an accurate and simple bilinear formulation of both an accuracy estimator and the expected resource requirement, together with a scalable search method with theoretical guarantees. The simplicity of our proposed estimator together with the intuitive way it is constructed bring interpretability through many insights about the contribution of different design choices. For example, we find that in the examined search space, adding depth and width is more effective at deeper stages of the network and at the beginning of each resolution stage. Our experiments show that BINAS generates comparable to or better architectures than other state-of-the-art NAS methods within a reduced marginal search cost, while strictly satisfying the resource constraints. △ Less

Submitted 27 April, 2022; v1 submitted 24 October, 2021; originally announced October 2021.

Comments: The full code is released at https://github.com/Alibaba-MIIL/BINAS

MSC Class: 68T09; 68T45 ACM Class: G.1.6; G.3; I.2.8; I.2.10; I.5.1

Multi-label Classification with Partial Annotations using Class-aware Selective Loss

Authors: Emanuel Ben-Baruch, Tal Ridnik, Itamar Friedman, Avi Ben-Cohen, Nadav Zamir, Asaf Noy, Lihi Zelnik-Manor

Abstract: Large-scale multi-label classification datasets are commonly, and perhaps inevitably, partially annotated. That is, only a small subset of labels are annotated per sample. Different methods for handling the missing labels induce different properties on the model and impact its accuracy. In this work, we analyze the partial labeling problem, then propose a solution based on two key ideas. First, un… ▽ More Large-scale multi-label classification datasets are commonly, and perhaps inevitably, partially annotated. That is, only a small subset of labels are annotated per sample. Different methods for handling the missing labels induce different properties on the model and impact its accuracy. In this work, we analyze the partial labeling problem, then propose a solution based on two key ideas. First, un-annotated labels should be treated selectively according to two probability quantities: the class distribution in the overall dataset and the specific label likelihood for a given data sample. We propose to estimate the class distribution using a dedicated temporary model, and we show its improved efficiency over a naive estimation computed using the dataset's partial annotations. Second, during the training of the target model, we emphasize the contribution of annotated labels over originally un-annotated labels by using a dedicated asymmetric loss. With our novel approach, we achieve state-of-the-art results on OpenImages dataset (e.g. reaching 87.3 mAP on V6). In addition, experiments conducted on LVIS and simulated-COCO demonstrate the effectiveness of our approach. Code is available at https://github.com/Alibaba-MIIL/PartialLabelingCSL. △ Less

Submitted 21 October, 2021; originally announced October 2021.

PETA: Photo Albums Event Recognition using Transformers Attention

Authors: Tamar Glaser, Emanuel Ben-Baruch, Gilad Sharir, Nadav Zamir, Asaf Noy, Lihi Zelnik-Manor

Abstract: In recent years the amounts of personal photos captured increased significantly, giving rise to new challenges in multi-image understanding and high-level image understanding. Event recognition in personal photo albums presents one challenging scenario where life events are recognized from a disordered collection of images, including both relevant and irrelevant images. Event recognition in images… ▽ More In recent years the amounts of personal photos captured increased significantly, giving rise to new challenges in multi-image understanding and high-level image understanding. Event recognition in personal photo albums presents one challenging scenario where life events are recognized from a disordered collection of images, including both relevant and irrelevant images. Event recognition in images also presents the challenge of high-level image understanding, as opposed to low-level image object classification. In absence of methods to analyze multiple inputs, previous methods adopted temporal mechanisms, including various forms of recurrent neural networks. However, their effective temporal window is local. In addition, they are not a natural choice given the disordered characteristic of photo albums. We address this gap with a tailor-made solution, combining the power of CNNs for image representation and transformers for album representation to perform global reasoning on image collection, offering a practical and efficient solution for photo albums event recognition. Our solution reaches state-of-the-art results on 3 prominent benchmarks, achieving above 90\% mAP on all datasets. We further explore the related image-importance task in event recognition, demonstrating how the learned attentions correlate with the human-annotated importance for this subjective task, thus opening the door for new applications. △ Less

Submitted 26 September, 2021; originally announced September 2021.

Comments: 8 pages, 10 including references, 3 figures, was submitted to WACV 2022

ImageNet-21K Pretraining for the Masses

Authors: Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, Lihi Zelnik-Manor

Abstract: ImageNet-1K serves as the primary dataset for pretraining deep learning models for computer vision tasks. ImageNet-21K dataset, which is bigger and more diverse, is used less frequently for pretraining, mainly due to its complexity, low accessibility, and underestimation of its added value. This paper aims to close this gap, and make high-quality efficient pretraining on ImageNet-21K available for… ▽ More ImageNet-1K serves as the primary dataset for pretraining deep learning models for computer vision tasks. ImageNet-21K dataset, which is bigger and more diverse, is used less frequently for pretraining, mainly due to its complexity, low accessibility, and underestimation of its added value. This paper aims to close this gap, and make high-quality efficient pretraining on ImageNet-21K available for everyone. Via a dedicated preprocessing stage, utilization of WordNet hierarchical structure, and a novel training scheme called semantic softmax, we show that various models significantly benefit from ImageNet-21K pretraining on numerous datasets and tasks, including small mobile-oriented models. We also show that we outperform previous ImageNet-21K pretraining schemes for prominent new models like ViT and Mixer. Our proposed pretraining pipeline is efficient, accessible, and leads to SoTA reproducible results, from a publicly available dataset. The training code and pretrained models are available at: https://github.com/Alibaba-MIIL/ImageNet21K △ Less

Submitted 5 August, 2021; v1 submitted 22 April, 2021; originally announced April 2021.

Comments: Accepted to NeurIPS 2021 (Datasets and Benchmarks)

An Image is Worth 16x16 Words, What is a Video Worth?

Authors: Gilad Sharir, Asaf Noy, Lihi Zelnik-Manor

Abstract: Leading methods in the domain of action recognition try to distill information from both the spatial and temporal dimensions of an input video. Methods that reach State of the Art (SotA) accuracy, usually make use of 3D convolution layers as a way to abstract the temporal information from video frames. The use of such convolutions requires sampling short clips from the input video, where each clip… ▽ More Leading methods in the domain of action recognition try to distill information from both the spatial and temporal dimensions of an input video. Methods that reach State of the Art (SotA) accuracy, usually make use of 3D convolution layers as a way to abstract the temporal information from video frames. The use of such convolutions requires sampling short clips from the input video, where each clip is a collection of closely sampled frames. Since each short clip covers a small fraction of an input video, multiple clips are sampled at inference in order to cover the whole temporal length of the video. This leads to increased computational load and is impractical for real-world applications. We address the computational bottleneck by significantly reducing the number of frames required for inference. Our approach relies on a temporal transformer that applies global attention over video frames, and thus better exploits the salient information in each frame. Therefore our approach is very input efficient, and can achieve SotA results (on Kinetics dataset) with a fraction of the data (frames per video), computation and latency. Specifically on Kinetics-400, we reach $80.5$ top-1 accuracy with $\times 30$ less frames per video, and $\times 40$ faster inference than the current leading method. Code is available at: https://github.com/Alibaba-MIIL/STAM △ Less

Submitted 27 May, 2021; v1 submitted 25 March, 2021; originally announced March 2021.

HardCoRe-NAS: Hard Constrained diffeRentiable Neural Architecture Search

Authors: Niv Nayman, Yonathan Aflalo, Asaf Noy, Lihi Zelnik-Manor

Abstract: Realistic use of neural networks often requires adhering to multiple constraints on latency, energy and memory among others. A popular approach to find fitting networks is through constrained Neural Architecture Search (NAS), however, previous methods enforce the constraint only softly. Therefore, the resulting networks do not exactly adhere to the resource constraint and their accuracy is harmed.… ▽ More Realistic use of neural networks often requires adhering to multiple constraints on latency, energy and memory among others. A popular approach to find fitting networks is through constrained Neural Architecture Search (NAS), however, previous methods enforce the constraint only softly. Therefore, the resulting networks do not exactly adhere to the resource constraint and their accuracy is harmed. In this work we resolve this by introducing Hard Constrained diffeRentiable NAS (HardCoRe-NAS), that is based on an accurate formulation of the expected resource requirement and a scalable search method that satisfies the hard constraint throughout the search. Our experiments show that HardCoRe-NAS generates state-of-the-art architectures, surpassing other NAS methods, while strictly satisfying the hard resource constraints without any tuning required. △ Less

Submitted 23 February, 2021; originally announced February 2021.

Comments: Niv Nayman and Yonathan Aflalo contributed equally. An implementation of HardCoRe-NAS is available at: https://github.com/Alibaba-MIIL/HardCoReNAS

MSC Class: 68T09; 68T45 ACM Class: G.1.6; G.3; I.2.8; I.2.10; I.5.1

A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks

Authors: Asaf Noy, Yi Xu, Yonathan Aflalo, Lihi Zelnik-Manor, Rong **

Abstract: Deep neural networks' remarkable ability to correctly fit training data when optimized by gradient-based algorithms is yet to be fully understood. Recent theoretical results explain the convergence for ReLU networks that are wider than those used in practice by orders of magnitude. In this work, we take a step towards closing the gap between theory and practice by significantly improving the known… ▽ More Deep neural networks' remarkable ability to correctly fit training data when optimized by gradient-based algorithms is yet to be fully understood. Recent theoretical results explain the convergence for ReLU networks that are wider than those used in practice by orders of magnitude. In this work, we take a step towards closing the gap between theory and practice by significantly improving the known theoretical bounds on both the network width and the convergence time. We show that convergence to a global minimum is guaranteed for networks with widths quadratic in the sample size and linear in their depth at a time logarithmic in both. Our analysis and convergence bounds are derived via the construction of a surrogate network with fixed activation patterns that can be transformed at any time to an equivalent ReLU network of a reasonable size. This construction can be viewed as a novel technique to accelerate training, while its tight finite-width equivalence to Neural Tangent Kernel (NTK) suggests it can be utilized to study generalization as well. △ Less

Submitted 8 February, 2021; v1 submitted 11 January, 2021; originally announced January 2021.

WeMix: How to Better Utilize Data Augmentation

Authors: Yi Xu, Asaf Noy, Ming Lin, Qi Qian, Hao Li, Rong **

Abstract: Data augmentation is a widely used training trick in deep learning to improve the network generalization ability. Despite many encouraging results, several recent studies did point out limitations of the conventional data augmentation scheme in certain scenarios, calling for a better theoretical understanding of data augmentation. In this work, we develop a comprehensive analysis that reveals pros… ▽ More Data augmentation is a widely used training trick in deep learning to improve the network generalization ability. Despite many encouraging results, several recent studies did point out limitations of the conventional data augmentation scheme in certain scenarios, calling for a better theoretical understanding of data augmentation. In this work, we develop a comprehensive analysis that reveals pros and cons of data augmentation. The main limitation of data augmentation arises from the data bias, i.e. the augmented data distribution can be quite different from the original one. This data bias leads to a suboptimal performance of existing data augmentation methods. To this end, we develop two novel algorithms, termed "AugDrop" and "MixLoss", to correct the data bias in the data augmentation. Our theoretical analysis shows that both algorithms are guaranteed to improve the effect of data augmentation through the bias correction, which is further validated by our empirical studies. Finally, we propose a generic algorithm "WeMix" by combining AugDrop and MixLoss, whose effectiveness is observed from extensive empirical evaluations. △ Less

Submitted 2 October, 2020; originally announced October 2020.

Asymmetric Loss For Multi-Label Classification

Authors: Emanuel Ben-Baruch, Tal Ridnik, Nadav Zamir, Asaf Noy, Itamar Friedman, Matan Protter, Lihi Zelnik-Manor

Abstract: In a typical multi-label setting, a picture contains on average few positive labels, and many negative ones. This positive-negative imbalance dominates the optimization process, and can lead to under-emphasizing gradients from positive labels during training, resulting in poor accuracy. In this paper, we introduce a novel asymmetric loss ("ASL"), which operates differently on positive and negative… ▽ More In a typical multi-label setting, a picture contains on average few positive labels, and many negative ones. This positive-negative imbalance dominates the optimization process, and can lead to under-emphasizing gradients from positive labels during training, resulting in poor accuracy. In this paper, we introduce a novel asymmetric loss ("ASL"), which operates differently on positive and negative samples. The loss enables to dynamically down-weights and hard-thresholds easy negative samples, while also discarding possibly mislabeled samples. We demonstrate how ASL can balance the probabilities of different samples, and how this balancing is translated to better mAP scores. With ASL, we reach state-of-the-art results on multiple popular multi-label datasets: MS-COCO, Pascal-VOC, NUS-WIDE and Open Images. We also demonstrate ASL applicability for other tasks, such as single-label classification and object detection. ASL is effective, easy to implement, and does not increase the training time or complexity. Implementation is available at: https://github.com/Alibaba-MIIL/ASL. △ Less

Submitted 29 July, 2021; v1 submitted 29 September, 2020; originally announced September 2020.

Comments: Accepted to ICCV 2021

ACM Class: I.2.6; I.2.10; I.0; I.4.0

TResNet: High Performance GPU-Dedicated Architecture

Authors: Tal Ridnik, Hussam Lawen, Asaf Noy, Emanuel Ben Baruch, Gilad Sharir, Itamar Friedman

Abstract: Many deep learning models, developed in recent years, reach higher ImageNet accuracy than ResNet50, with fewer or comparable FLOPS count. While FLOPs are often seen as a proxy for network efficiency, when measuring actual GPU training and inference throughput, vanilla ResNet50 is usually significantly faster than its recent competitors, offering better throughput-accuracy trade-off. In this work… ▽ More Many deep learning models, developed in recent years, reach higher ImageNet accuracy than ResNet50, with fewer or comparable FLOPS count. While FLOPs are often seen as a proxy for network efficiency, when measuring actual GPU training and inference throughput, vanilla ResNet50 is usually significantly faster than its recent competitors, offering better throughput-accuracy trade-off. In this work, we introduce a series of architecture modifications that aim to boost neural networks' accuracy, while retaining their GPU training and inference efficiency. We first demonstrate and discuss the bottlenecks induced by FLOPs-optimizations. We then suggest alternative designs that better utilize GPU structure and assets. Finally, we introduce a new family of GPU-dedicated models, called TResNet, which achieve better accuracy and efficiency than previous ConvNets. Using a TResNet model, with similar GPU throughput to ResNet50, we reach 80.8 top-1 accuracy on ImageNet. Our TResNet models also transfer well and achieve state-of-the-art accuracy on competitive single-label classification datasets such as Stanford cars (96.0%), CIFAR-10 (99.0%), CIFAR-100 (91.5%) and Oxford-Flowers (99.1%). They also perform well on multi-label classification and object detection tasks. Implementation is available at: https://github.com/mrT23/TResNet. △ Less

Submitted 27 August, 2020; v1 submitted 30 March, 2020; originally announced March 2020.

Comments: 11 pages, 5 figures

Knapsack Pruning with Inner Distillation

Authors: Yonathan Aflalo, Asaf Noy, Ming Lin, Itamar Friedman, Lihi Zelnik

Abstract: Neural network pruning reduces the computational cost of an over-parameterized network to improve its efficiency. Popular methods vary from $\ell_1$-norm sparsification to Neural Architecture Search (NAS). In this work, we propose a novel pruning method that optimizes the final accuracy of the pruned network and distills knowledge from the over-parameterized parent network's inner layers. To enabl… ▽ More Neural network pruning reduces the computational cost of an over-parameterized network to improve its efficiency. Popular methods vary from $\ell_1$-norm sparsification to Neural Architecture Search (NAS). In this work, we propose a novel pruning method that optimizes the final accuracy of the pruned network and distills knowledge from the over-parameterized parent network's inner layers. To enable this approach, we formulate the network pruning as a Knapsack Problem which optimizes the trade-off between the importance of neurons and their associated computational cost. Then we prune the network channels while maintaining the high-level structure of the network. The pruned network is fine-tuned under the supervision of the parent network using its inner network knowledge, a technique we refer to as the Inner Knowledge Distillation. Our method leads to state-of-the-art pruning results on ImageNet, CIFAR-10 and CIFAR-100 using ResNet backbones. To prune complex network structures such as convolutions with skip-links and depth-wise convolutions, we propose a block grou** approach to cope with these structures. Through this we produce compact architectures with the same FLOPs as EfficientNet-B0 and MobileNetV3 but with higher accuracy, by $1\%$ and $0.3\%$ respectively on ImageNet, and faster runtime on GPU. △ Less

Submitted 3 June, 2020; v1 submitted 19 February, 2020; originally announced February 2020.

Eradicating Attacks on the Internal Network with Internal Network Policy

Authors: Yehuda Afek, Anat Bremler-Barr, Alon Noy

Abstract: In this paper we present three attacks on private internal networks behind a NAT and a corresponding new protection mechanism, Internal Network Policy, to mitigate a wide range of attacks that penetrate internal networks behind a NAT. In the attack scenario, a victim is tricked to visit the attacker's website, which contains a malicious script that lets the attacker access the victim's internal ne… ▽ More In this paper we present three attacks on private internal networks behind a NAT and a corresponding new protection mechanism, Internal Network Policy, to mitigate a wide range of attacks that penetrate internal networks behind a NAT. In the attack scenario, a victim is tricked to visit the attacker's website, which contains a malicious script that lets the attacker access the victim's internal network in different ways, including opening a port in the NAT or sending a sophisticated request to local devices. The first attack utilizes DNS Rebinding in a particular way, while the other two demonstrate different methods of attacking the network, based on application security vulnerabilities. Following the attacks, we provide a new browser security policy, Internal Network Policy (INP), which protects against these types of vulnerabilities and attacks. This policy is implemented in the browser just like Same Origin Policy (SOP) and prevents malicious access to internal resources by external entities. △ Less

Submitted 3 October, 2019; v1 submitted 2 October, 2019; originally announced October 2019.

XNAS: Neural Architecture Search with Expert Advice

Authors: Niv Nayman, Asaf Noy, Tal Ridnik, Itamar Friedman, Rong **, Lihi Zelnik-Manor

Abstract: This paper introduces a novel optimization method for differential neural architecture search, based on the theory of prediction with expert advice. Its optimization criterion is well fitted for an architecture-selection, i.e., it minimizes the regret incurred by a sub-optimal selection of operations. Unlike previous search relaxations, that require hard pruning of architectures, our method is des… ▽ More This paper introduces a novel optimization method for differential neural architecture search, based on the theory of prediction with expert advice. Its optimization criterion is well fitted for an architecture-selection, i.e., it minimizes the regret incurred by a sub-optimal selection of operations. Unlike previous search relaxations, that require hard pruning of architectures, our method is designed to dynamically wipe out inferior architectures and enhance superior ones. It achieves an optimal worst-case regret bound and suggests the use of multiple learning-rates, based on the amount of information carried by the backward gradients. Experiments show that our algorithm achieves a strong performance over several image classification datasets. Specifically, it obtains an error rate of 1.6% for CIFAR-10, 24% for ImageNet under mobile settings, and achieves state-of-the-art results on three additional datasets. △ Less

Submitted 19 June, 2019; originally announced June 2019.

ASAP: Architecture Search, Anneal and Prune

Authors: Asaf Noy, Niv Nayman, Tal Ridnik, Nadav Zamir, Sivan Doveh, Itamar Friedman, Raja Giryes, Lihi Zelnik-Manor

Abstract: Automatic methods for Neural Architecture Search (NAS) have been shown to produce state-of-the-art network models. Yet, their main drawback is the computational complexity of the search process. As some primal methods optimized over a discrete search space, thousands of days of GPU were required for convergence. A recent approach is based on constructing a differentiable search space that enables… ▽ More Automatic methods for Neural Architecture Search (NAS) have been shown to produce state-of-the-art network models. Yet, their main drawback is the computational complexity of the search process. As some primal methods optimized over a discrete search space, thousands of days of GPU were required for convergence. A recent approach is based on constructing a differentiable search space that enables gradient-based optimization, which reduces the search time to a few days. While successful, it still includes some noncontinuous steps, e.g., the pruning of many weak connections at once. In this paper, we propose a differentiable search space that allows the annealing of architecture weights, while gradually pruning inferior operations. In this way, the search converges to a single output network in a continuous manner. Experiments on several vision datasets demonstrate the effectiveness of our method with respect to the search cost and accuracy of the achieved model. Specifically, with $0.2$ GPU search days we achieve an error rate of $1.68\%$ on CIFAR-10. △ Less

Submitted 10 October, 2019; v1 submitted 8 April, 2019; originally announced April 2019.

The emergence of sequence-dependent structural motifs in stretched, torsionally constrained DNA

Authors: Jack W Shepherd, Robert J Greenall, Matt I J Probert, Agnes Noy, Mark C Leake

Abstract: The double-helical structure of DNA results from canonical base pairing and stacking interactions. However, variations from steady-state conformations result from mechanical perturbations in cells. These different topologies have physiological relevance but their dependence on sequence remains unclear. Here, we use molecular dynamics simulations to show that sequence differences result in markedly… ▽ More The double-helical structure of DNA results from canonical base pairing and stacking interactions. However, variations from steady-state conformations result from mechanical perturbations in cells. These different topologies have physiological relevance but their dependence on sequence remains unclear. Here, we use molecular dynamics simulations to show that sequence differences result in markedly different structural motifs upon physiological twisting and stretching. We simulated overextension on four different sequences of DNA ((AA)12, (AT)12, (GG)12 and (GC)12) with supercoiling densities within the physiological range. We found that DNA denatures in the majority of stretching simulations, surprisingly including those with overtwisted DNA. GC-rich sequences were observed to be more stable than AT-rich, with the specific response dependent on base pair ordering. Furthermore, we found that (AT)12 forms stable periodic structures with non-canonical hydrogen bonds in some regions and non-canonical stacking in others, whereas (GC)12 forms a stacking motif of four base pairs independent of supercoiling density. Our results demonstrate that 20-30% DNA extension is sufficient for breaking B-DNA around and significantly above cellular supercoiling, and that the DNA sequence is crucial for understanding structural changes under mechanical stress. Our findings have important implications for the activities of protein machinery interacting with DNA in all cells. △ Less

Submitted 4 April, 2019; originally announced April 2019.

Reply to Comment on 'Length Scale Dependence of DNA Mechanical Properties'

Authors: Agnes Noy, Ramin Golestanian

Abstract: Reply to Comment on 'Length Scale Dependence of DNA Mechanical Properties' Reply to Comment on 'Length Scale Dependence of DNA Mechanical Properties' △ Less

Submitted 14 October, 2013; v1 submitted 8 January, 2013; originally announced January 2013.

Comments: Revised version to appear in Phys. Rev. Lett

Length scale dependence of DNA mechanical properties

Authors: Agnes Noy, Ramin Golestanian

Abstract: Although mechanical properties of DNA are well characterized at the kilo base-pair range, a number of recent experiments have suggested that DNA is more flexible at shorter length scales, which correspond to the regime that is crucial for cellular processes such as DNA packaging and gene regulation. Here, we perform a systematic study of the effective elastic properties of DNA at different length… ▽ More Although mechanical properties of DNA are well characterized at the kilo base-pair range, a number of recent experiments have suggested that DNA is more flexible at shorter length scales, which correspond to the regime that is crucial for cellular processes such as DNA packaging and gene regulation. Here, we perform a systematic study of the effective elastic properties of DNA at different length scales by probing the conformation and fluctuations of DNA from single base-pair level up to four helical turns, using trajectories from atomistic simulation. We find evidence that supports cooperative softening of the stretch modulus and identify the essential modes that give rise to this effect. The bend correlation exhibits modulations that reflect the helical periodicity, while it yields a reasonable value for the effective persistence length, and the twist modulus undergoes a smooth crossover---from a relatively smaller value at the single base-pair level to the bulk value---over half a DNA-turn. △ Less

Submitted 26 October, 2012; originally announced October 2012.

Comments: 5 pages, 4 figures, accepted for publication in Phys. Rev. Lett

Three-dimensional coherent X-ray diffraction imaging of a ceramic nanofoam: determination of structural deformation mechanisms

Authors: A. Barty, S. Marchesini, H. N. Chapman, C. Cui, M. R. Howells, D. A. Shapiro, A. M. Minor, J. C. H. Spence, U. Weierstall, J. Ilavsky, A. Noy, S. P. Hau-Riege, A. B. Artyukhin, T. Baumann, T. Willey, J. Stolken, T. van Buuren, J. H. Kinney

Abstract: Ultra-low density polymers, metals, and ceramic nanofoams are valued for their high strength-to-weight ratio, high surface area and insulating properties ascribed to their structural geometry. We obtain the labrynthine internal structure of a tantalum oxide nanofoam by X-ray diffractive imaging. Finite element analysis from the structure reveals mechanical properties consistent with bulk samples… ▽ More Ultra-low density polymers, metals, and ceramic nanofoams are valued for their high strength-to-weight ratio, high surface area and insulating properties ascribed to their structural geometry. We obtain the labrynthine internal structure of a tantalum oxide nanofoam by X-ray diffractive imaging. Finite element analysis from the structure reveals mechanical properties consistent with bulk samples and with a diffusion limited cluster aggregation model, while excess mass on the nodes discounts the dangling fragments hypothesis of percolation theory. △ Less

Submitted 25 June, 2008; v1 submitted 30 August, 2007; originally announced August 2007.

Comments: 8 pages, 5 figures, 30 references

Report number: UCRL-JRNL-231416

Journal ref: Phys. Rev. Lett. 101, 055501 (2008).

Progress in Three-Dimensional Coherent X-Ray Diffraction Imaging

Authors: S. Marchesini, H. N. Chapman, A. Barty, A. Noy, S. P. Hau-Riege, J. M. Kinney, C. Cui, M. R. Howells, R. Rosen, J. C. H. Spence, U. Weierstall, D. Shapiro, T. Beetz, C. Jacobsen, E. Lima, A. M. Minor, H. He

Abstract: The Fourier inversion of phased coherent diffraction patterns offers images without the resolution and depth-of-focus limitations of lens-based tomographic systems. We report on our recent experimental images inverted using recent developments in phase retrieval algorithms, and summarize efforts that led to these accomplishments. These include ab-initio reconstruction of a two-dimensional test p… ▽ More The Fourier inversion of phased coherent diffraction patterns offers images without the resolution and depth-of-focus limitations of lens-based tomographic systems. We report on our recent experimental images inverted using recent developments in phase retrieval algorithms, and summarize efforts that led to these accomplishments. These include ab-initio reconstruction of a two-dimensional test pattern, infinite depth of focus image of a thick object, and its high-resolution (~10 nm resolution) three-dimensional image. Developments on the structural imaging of low density aerogel samples are discussed. △ Less

Submitted 6 October, 2005; v1 submitted 4 October, 2005; originally announced October 2005.

Comments: 5 pages, X-Ray Microscopy 2005, Himeji, Japan

Report number: UCRL-PROC-215874

Journal ref: IPAP Conf. Series 7 pp.353-356, 2006

High-resolution ab initio three-dimensional X-ray diffraction microscopy

Authors: H. N. Chapman, A. Barty, S. Marchesini, A. Noy, C. Cui, M. R. Howells, R. Rosen, H. He, J. C. H. Spence, U. Weierstall, T. Beetz, C. Jacobsen, D. Shapiro

Abstract: Coherent X-ray diffraction microscopy is a method of imaging non-periodic isolated objects at resolutions only limited, in principle, by the largest scattering angles recorded. We demonstrate X-ray diffraction imaging with high resolution in all three dimensions, as determined by a quantitative analysis of the reconstructed volume images. These images are retrieved from the 3D diffraction data u… ▽ More Coherent X-ray diffraction microscopy is a method of imaging non-periodic isolated objects at resolutions only limited, in principle, by the largest scattering angles recorded. We demonstrate X-ray diffraction imaging with high resolution in all three dimensions, as determined by a quantitative analysis of the reconstructed volume images. These images are retrieved from the 3D diffraction data using no a priori knowledge about the shape or composition of the object, which has never before been demonstrated on a non-periodic object. We also construct 2D images of thick objects with infinite depth of focus (without loss of transverse spatial resolution). These methods can be used to image biological and materials science samples at high resolution using X-ray undulator radiation, and establishes the techniques to be used in atomic-resolution ultrafast imaging at X-ray free-electron laser sources. △ Less

Submitted 8 September, 2005; originally announced September 2005.

Comments: 22 pages, 11 figures, submitted

Report number: UCRL-JRNL-214796

Journal ref: J. Opt. Soc. Am. A 23, 1179-1200 (2006)

SPEDEN: Reconstructing single particles from their diffraction patterns

Authors: S. P. Hau-Riege, H. Szoke, H. N. Chapman, A. Szoke, S. Marchesini, A. Noy, H. He, M. R. Howells, U. Weierstall, J. C. H. Spence

Abstract: Speden is a computer program that reconstructs the electron density of single particles from their x-ray diffraction patterns, using a single-particle adaptation of the Holographic Method in crystallography. (Szoke, A., Szoke, H., and Somoza, J.R., 1997. Acta Cryst. A53, 291-313.) The method, like its parent, is unique that it does not rely on ``back'' transformation from the diffraction pattern… ▽ More Speden is a computer program that reconstructs the electron density of single particles from their x-ray diffraction patterns, using a single-particle adaptation of the Holographic Method in crystallography. (Szoke, A., Szoke, H., and Somoza, J.R., 1997. Acta Cryst. A53, 291-313.) The method, like its parent, is unique that it does not rely on ``back'' transformation from the diffraction pattern into real space and on interpolation within measured data. It is designed to deal successfully with sparse, irregular, incomplete and noisy data. It is also designed to use prior information for ensuring sensible results and for reliable convergence. This article describes the theoretical basis for the reconstruction algorithm, its implementation and quantitative results of tests on synthetic and experimentally obtained data. The program could be used for determining the structure of radiation tolerant samples and, eventually, of large biological molecular structures without the need for crystallization. △ Less

Submitted 18 March, 2004; originally announced March 2004.

Comments: 12 pages, 10 figures

Report number: UCRL-ID-154574

Journal ref: Acta Cryst. A60, 294-305 (2004).

X-ray image reconstruction from a diffraction pattern alone

Authors: S. Marchesini, H. He, H. N. Chapman, S. P. Hau-Riege, A. Noy, M. R. Howells, U. Weierstall, J. C. H. Spence

Abstract: A solution to the inversion problem of scattering would offer aberration-free diffraction-limited 3D images without the resolution and depth-of-field limitations of lens-based tomographic systems. Powerful algorithms are increasingly being used to act as lenses to form such images. Current image reconstruction methods, however, require the knowledge of the shape of the object and the low spatial… ▽ More A solution to the inversion problem of scattering would offer aberration-free diffraction-limited 3D images without the resolution and depth-of-field limitations of lens-based tomographic systems. Powerful algorithms are increasingly being used to act as lenses to form such images. Current image reconstruction methods, however, require the knowledge of the shape of the object and the low spatial frequencies unavoidably lost in experiments. Diffractive imaging has thus previously been used to increase the resolution of images obtained by other means. We demonstrate experimentally here a new inversion method, which reconstructs the image of the object without the need for any such prior knowledge. △ Less

Submitted 12 September, 2003; v1 submitted 24 June, 2003; originally announced June 2003.

Comments: 5 pages, 3 figures, improved figures and captions, changed title

Report number: UCRL-JC-153571

Journal ref: Phys. Rev. B 68, 140101(R) (2003)