-
AUTOKD: Automatic Knowledge Distillation Into A Student Architecture Family
Authors:
Roy Henha Eyono,
Fabio Maria Carlucci,
Pedro M Esperança,
Binxin Ru,
Phillip Torr
Abstract:
State-of-the-art results in deep learning have been improving steadily, in good part due to the use of larger models. However, widespread use is constrained by device hardware limitations, resulting in a substantial performance gap between state-of-the-art models and those that can be effectively deployed on small devices. While Knowledge Distillation (KD) theoretically enables small student model…
▽ More
State-of-the-art results in deep learning have been improving steadily, in good part due to the use of larger models. However, widespread use is constrained by device hardware limitations, resulting in a substantial performance gap between state-of-the-art models and those that can be effectively deployed on small devices. While Knowledge Distillation (KD) theoretically enables small student models to emulate larger teacher models, in practice selecting a good student architecture requires considerable human expertise. Neural Architecture Search (NAS) appears as a natural solution to this problem but most approaches can be inefficient, as most of the computation is spent comparing architectures sampled from the same distribution, with negligible differences in performance. In this paper, we propose to instead search for a family of student architectures sharing the property of being good at learning from a given teacher. Our approach AutoKD, powered by Bayesian Optimization, explores a flexible graph-based search space, enabling us to automatically learn the optimal student architecture distribution and KD parameters, while being 20x more sample efficient compared to existing state-of-the-art. We evaluate our method on 3 datasets; on large images specifically, we reach the teacher performance while using 3x less memory and 10x less parameters. Finally, while AutoKD uses the traditional KD loss, it outperforms more advanced KD variants using hand-designed students.
△ Less
Submitted 5 November, 2021;
originally announced November 2021.
-
Current State and Future Directions for Learning in Biological Recurrent Neural Networks: A Perspective Piece
Authors:
Luke Y. Prince,
Roy Henha Eyono,
Ellen Boven,
Arna Ghosh,
Joe Pemberton,
Franz Scherr,
Claudia Clopath,
Rui Ponte Costa,
Wolfgang Maass,
Blake A. Richards,
Cristina Savin,
Katharina Anna Wilmes
Abstract:
We provide a brief review of the common assumptions about biological learning with findings from experimental neuroscience and contrast them with the efficiency of gradient-based learning in recurrent neural networks. The key issues discussed in this review include: synaptic plasticity, neural circuits, theory-experiment divide, and objective functions. We conclude with recommendations for both th…
▽ More
We provide a brief review of the common assumptions about biological learning with findings from experimental neuroscience and contrast them with the efficiency of gradient-based learning in recurrent neural networks. The key issues discussed in this review include: synaptic plasticity, neural circuits, theory-experiment divide, and objective functions. We conclude with recommendations for both theoretical and experimental neuroscientists when designing new studies that could help bring clarity to these issues.
△ Less
Submitted 5 January, 2022; v1 submitted 11 May, 2021;
originally announced May 2021.
-
Behavioural Analytics: Beyond Risk-based MFA
Authors:
Roy Henha Eyono
Abstract:
This paper investigates how to effectively stop an attacker from using compromised user credentials to gain authorized entry to systems that they are otherwise not authorised to access. The proposed solution extends previous work to move beyond a risk-based multi-factor authentication system. It adds a behavioural analytics component that uses keystroke dynamics to grant or deny users access. Give…
▽ More
This paper investigates how to effectively stop an attacker from using compromised user credentials to gain authorized entry to systems that they are otherwise not authorised to access. The proposed solution extends previous work to move beyond a risk-based multi-factor authentication system. It adds a behavioural analytics component that uses keystroke dynamics to grant or deny users access. Given the increasing number of compromised user credential stores, we make the assumption that criminals already know the user credentials. Hence, to test our solution, users were given authentic user credentials and asked to login to our proof-of-concept. Despite the fact that all illegitimate users in our test cases were given the correct user credentials for legitimate users, none of these were granted access by the system. This demonstrates zero- tolerance to false positives. The results demonstrate the uniqueness of keystroke dynamics and its use to prevent users with stolen credentials from accessing systems they are not authorized to access.
△ Less
Submitted 8 January, 2018;
originally announced January 2018.
