-
Assessing the Portability of Parameter Matrices Trained by Parameter-Efficient Finetuning Methods
Authors:
Mohammed Sabry,
Anya Belz
Abstract:
As the cost of training ever larger language models has grown, so has the interest in reusing previously learnt knowledge. Transfer learning methods have shown how reusing non-task-specific knowledge can help in subsequent task-specific learning. In this paper, we investigate the inverse: porting whole functional modules that encode task-specific knowledge from one model to another. We designed a…
▽ More
As the cost of training ever larger language models has grown, so has the interest in reusing previously learnt knowledge. Transfer learning methods have shown how reusing non-task-specific knowledge can help in subsequent task-specific learning. In this paper, we investigate the inverse: porting whole functional modules that encode task-specific knowledge from one model to another. We designed a study comprising 1,440 training/testing runs to test the portability of modules trained by parameter-efficient finetuning (PEFT) techniques, using sentiment analysis as an example task. We test portability in a wide range of scenarios, involving different PEFT techniques and different pretrained host models, among other dimensions. We compare the performance of ported modules with that of equivalent modules trained (i) from scratch, and (ii) from parameters sampled from the same distribution as the ported module. We find that the ported modules far outperform the two alternatives tested, but that there are interesting performance differences between the four PEFT techniques. We conclude that task-specific knowledge in the form of structurally modular sets of parameters as produced by PEFT techniques is highly portable, but that degree of success depends on type of PEFT and on differences between originating and receiving pretrained models.
△ Less
Submitted 25 January, 2024;
originally announced January 2024.
-
PEFT-Ref: A Modular Reference Architecture and Typology for Parameter-Efficient Finetuning Techniques
Authors:
Mohammed Sabry,
Anya Belz
Abstract:
Recent parameter-efficient finetuning (PEFT) techniques aim to improve over the considerable cost of fully finetuning large pretrained language models (PLM). As different PEFT techniques proliferate, it is becoming difficult to compare them, in particular in terms of (i) the structure and functionality they add to the PLM, (ii) the different types and degrees of efficiency improvements achieved, (…
▽ More
Recent parameter-efficient finetuning (PEFT) techniques aim to improve over the considerable cost of fully finetuning large pretrained language models (PLM). As different PEFT techniques proliferate, it is becoming difficult to compare them, in particular in terms of (i) the structure and functionality they add to the PLM, (ii) the different types and degrees of efficiency improvements achieved, (iii) performance at different downstream tasks, and (iv) how differences in structure and functionality relate to efficiency and task performance. To facilitate such comparisons, this paper presents a reference architecture which standardises aspects shared by different PEFT techniques, while isolating differences to specific locations and interactions with the standard components. Through this process of standardising and isolating differences, a modular view of PEFT techniques emerges, supporting not only direct comparison of different techniques and their efficiency and task performance, but also systematic exploration of reusability and composability of the different types of finetuned modules. We demonstrate how the reference architecture can be applied to understand properties and relative advantages of PEFT techniques, hence to inform selection of techniques for specific tasks, and design choices for new PEFT techniques.
△ Less
Submitted 19 October, 2023; v1 submitted 24 April, 2023;
originally announced April 2023.
-
A Novel Power-optimized CMOS sEMG Device with Ultra Low-noise integrated with ConvNet (VGG16) for Biomedical Applications
Authors:
Ahmed Ayman - Mohamed Sabry
Abstract:
The needle bio-potential sensors for measuring muscle and brain activity need invasive surgical targeted muscle reinnervation (TMR) and a demanding process to maintain, but surface bio-potential sensors lack clear bio-signal reading (Signal-Interference). In this research, a novel power-optimized complementary metal-oxide-semiconductor (CMOS) Surface Electromyography (sEMG) is developed to improve…
▽ More
The needle bio-potential sensors for measuring muscle and brain activity need invasive surgical targeted muscle reinnervation (TMR) and a demanding process to maintain, but surface bio-potential sensors lack clear bio-signal reading (Signal-Interference). In this research, a novel power-optimized complementary metal-oxide-semiconductor (CMOS) Surface Electromyography (sEMG) is developed to improve the efficiency and quality of captured bio-signal for biomedical application: The early diagnosis of neurological disorders (Dystonia) and a novel compatible mind-controlled prosthetic leg with human daily activities. A novel sEMG composed of CMOS Op-Amp based PIC16F877A 8-bit CMOS Flash-based Microcontroller is utilized to minimize power consumption and data processing time. sEMG Circuit is implemented with developed analog filter along with infinite impulse response (IIR) digital filter via Fast Fourier Transform (FFT), Z-transform, and difference equations. The analysis shows a significant improvement of 169.2% noise-reduction in recorded EMG signal using developed digital filter compared to analog one according to numerical root mean square error (RMSE). Moreover, digital IIR was tested in two stages: algorithmic and real-world. As a result, IIR's algorithmic (MATLAB) and real-world RMSEs were 0.03616 and 0.05224, respectively. A notable advancement of 20.8% in data processing duration in EMG signal analysis. Optimizing VGG, AlexNet, and ResNet ConvNet as trained and tested on 15 public EEG (62-electrode) and 18 subjects' observed EMG data. The results indicate that VGG16-1D is 98.43% higher. During real testing, the accuracy was 95.8 +/- 4.6% for 16 subjects (6 Amputees-10 Dystonia). This study demonstrates the potential for sEMG, paving the way for biomedical applications.
△ Less
Submitted 10 May, 2023; v1 submitted 3 January, 2023;
originally announced January 2023.
-
AfriVEC: Word Embedding Models for African Languages. Case Study of Fon and Nobiin
Authors:
Bonaventure F. P. Dossou,
Mohammed Sabry
Abstract:
From Word2Vec to GloVe, word embedding models have played key roles in the current state-of-the-art results achieved in Natural Language Processing. Designed to give significant and unique vectorized representations of words and entities, those models have proven to efficiently extract similarities and establish relationships reflecting semantic and contextual meaning among words and entities. Afr…
▽ More
From Word2Vec to GloVe, word embedding models have played key roles in the current state-of-the-art results achieved in Natural Language Processing. Designed to give significant and unique vectorized representations of words and entities, those models have proven to efficiently extract similarities and establish relationships reflecting semantic and contextual meaning among words and entities. African Languages, representing more than 31% of the worldwide spoken languages, have recently been subject to lots of research. However, to the best of our knowledge, there are currently very few to none word embedding models for those languages words and entities, and none for the languages under study in this paper. After describing Glove, Word2Vec, and Poincaré embeddings functionalities, we build Word2Vec and Poincaré word embedding models for Fon and Nobiin, which show promising results. We test the applicability of transfer learning between these models as a landmark for African Languages to jointly involve in mitigating the scarcity of their resources, and attempt to provide linguistic and social interpretations of our results. Our main contribution is to arouse more interest in creating word embedding models proper to African Languages, ready for use, and that can significantly improve the performances of Natural Language Processing downstream tasks on them. The official repository and implementation is at https://github.com/bonaventuredossou/afrivec
△ Less
Submitted 18 March, 2021; v1 submitted 8 March, 2021;
originally announced March 2021.
-
ArchiveSafe: Mass-Leakage-Resistant Storage from Proof-of-Work
Authors:
Moe Sabry,
Reza Samavi,
Douglas Stebila
Abstract:
Data breaches-mass leakage of stored information-are a major security concern. Encryption can provide confidentiality, but encryption depends on a key which, if compromised, allows the attacker to decrypt everything, effectively instantly. Security of encrypted data thus becomes a question of protecting the encryption keys. In this paper, we propose using keyless encryption to construct a mass lea…
▽ More
Data breaches-mass leakage of stored information-are a major security concern. Encryption can provide confidentiality, but encryption depends on a key which, if compromised, allows the attacker to decrypt everything, effectively instantly. Security of encrypted data thus becomes a question of protecting the encryption keys. In this paper, we propose using keyless encryption to construct a mass leakage resistant archiving system, where decryption of a file is only possible after the requester, whether an authorized user or an adversary, completes a proof of work in the form of solving a cryptographic puzzle. This proposal is geared towards protection of infrequently-accessed archival data, where any one file may not require too much work to decrypt, decryption of a large number of files-mass leakage-becomes increasingly expensive for an attacker. We present a prototype implementation realized as a user-space file system driver for Linux. We report experimental results of system behaviour under different file sizes and puzzle difficulty levels. Our keyless encryption technique can be added as a layer on top of traditional encryption: together they provide strong security against adversaries without the key and resistance against mass decryption by an attacker.
△ Less
Submitted 14 October, 2020; v1 submitted 31 August, 2020;
originally announced September 2020.
-
On the Reduction of Variance and Overestimation of Deep Q-Learning
Authors:
Mohammed Sabry,
Amr M. A. Khalifa
Abstract:
The breakthrough of deep Q-Learning on different types of environments revolutionized the algorithmic design of Reinforcement Learning to introduce more stable and robust algorithms, to that end many extensions to deep Q-Learning algorithm have been proposed to reduce the variance of the target values and the overestimation phenomena. In this paper, we examine new methodology to solve these issues…
▽ More
The breakthrough of deep Q-Learning on different types of environments revolutionized the algorithmic design of Reinforcement Learning to introduce more stable and robust algorithms, to that end many extensions to deep Q-Learning algorithm have been proposed to reduce the variance of the target values and the overestimation phenomena. In this paper, we examine new methodology to solve these issues, we propose using Dropout techniques on deep Q-Learning algorithm as a way to reduce variance and overestimation. We also present experiments conducted on benchmark environments, demonstrating the effectiveness of our methodology in enhancing stability and reducing both variance and overestimation in model performance.
△ Less
Submitted 14 April, 2024; v1 submitted 14 October, 2019;
originally announced October 2019.
-
TEA-DNN: the Quest for Time-Energy-Accuracy Co-optimized Deep Neural Networks
Authors:
Lile Cai,
Anne-Maelle Barneche,
Arthur Herbout,
Chuan Sheng Foo,
Jie Lin,
Vijay Ramaseshan Chandrasekhar,
Mohamed M. Sabry
Abstract:
Embedded deep learning platforms have witnessed two simultaneous improvements. First, the accuracy of convolutional neural networks (CNNs) has been significantly improved through the use of automated neural-architecture search (NAS) algorithms to determine CNN structure. Second, there has been increasing interest in develo** hardware accelerators for CNNs that provide improved inference performa…
▽ More
Embedded deep learning platforms have witnessed two simultaneous improvements. First, the accuracy of convolutional neural networks (CNNs) has been significantly improved through the use of automated neural-architecture search (NAS) algorithms to determine CNN structure. Second, there has been increasing interest in develo** hardware accelerators for CNNs that provide improved inference performance and energy consumption compared to GPUs. Such embedded deep learning platforms differ in the amount of compute resources and memory-access bandwidth, which would affect performance and energy consumption of CNNs. It is therefore critical to consider the available hardware resources in the network architecture search. To this end, we introduce TEA-DNN, a NAS algorithm targeting multi-objective optimization of execution time, energy consumption, and classification accuracy of CNN workloads on embedded architectures. TEA-DNN leverages energy and execution time measurements on embedded hardware when exploring the Pareto-optimal curves across accuracy, execution time, and energy consumption and does not require additional effort to model the underlying hardware. We apply TEA-DNN for image classification on actual embedded platforms (NVIDIA Jetson TX2 and Intel Movidius Neural Compute Stick). We highlight the Pareto-optimal operating points that emphasize the necessity to explicitly consider hardware characteristics in the search process. To the best of our knowledge, this is the most comprehensive study of Pareto-optimal models across a range of hardware platforms using actual measurements on hardware to obtain objective values.
△ Less
Submitted 21 October, 2019; v1 submitted 29 November, 2018;
originally announced November 2018.
-
Towards On-Chip MEMS-Based Optical Autocorrelator
Authors:
Ahmed M. Othman,
Hussein E. Kotb,
Yasser M. Sabry,
Osama Terra,
Diaa A. Khalil
Abstract:
We propose a compact MEMS-based optical autocorrelator based on a micromachined Michelson interferometer in silicon and the two-photon absorption non-linearity in a photodetector. The miniaturized autocorrelator has a scanning range of 1.2 ps and operates in the wavelength range of 1100-2000 nm. The device measures the interferometric autocorrelation due to its collinear nature, from which the int…
▽ More
We propose a compact MEMS-based optical autocorrelator based on a micromachined Michelson interferometer in silicon and the two-photon absorption non-linearity in a photodetector. The miniaturized autocorrelator has a scanning range of 1.2 ps and operates in the wavelength range of 1100-2000 nm. The device measures the interferometric autocorrelation due to its collinear nature, from which the intensity autocorrelation can be calculated. The field autocorrelation can also be measured, from which the optical pulse spectrum can be calculated. A theoretical model based on Gaussian beam propagation is developed to study the effect of optical beam divergence, pulse dispersion, tilt angle between the interferometer mirrors, and amplitude mismatch between the interfering pulses. This model explains many of the effects observed in experimental measurements due to the use of a MEMS interferometer. The experimental results of autocorrelation signals for several pulses in the order of 100 fs are compared to a commercial autocorrelator and a good match is found.
△ Less
Submitted 12 September, 2018;
originally announced September 2018.
-
Flexible profile approach to the conjugate heat transfer problem
Authors:
M. -N. Sabry
Abstract:
The flexible profile approach proposed earlier to create CTM (compact or reduced order thermal models) is extended to cover the area of conjugate heat transfer. The flexible profile approach is a methodology that allows building a highly boundary conditions independent CTM, with any desired degree of accuracy, that may adequately replace detailed 3D models for the whole spectrum of applications…
▽ More
The flexible profile approach proposed earlier to create CTM (compact or reduced order thermal models) is extended to cover the area of conjugate heat transfer. The flexible profile approach is a methodology that allows building a highly boundary conditions independent CTM, with any desired degree of accuracy, that may adequately replace detailed 3D models for the whole spectrum of applications in which the modeled object may be used. The extension to conjugate problems radically solves the problem of interfacing two different domains. Each domain, fluid or solid, can be "compacted" independently creating two CTM that can be joined together to produce reliable results for any arbitrary set of external boundary conditions.
△ Less
Submitted 7 January, 2008;
originally announced January 2008.
