-
MU-Bench: A Multitask Multimodal Benchmark for Machine Unlearning
Authors:
Jiali Cheng,
Hadi Amiri
Abstract:
Recent advancements in Machine Unlearning (MU) have introduced solutions to selectively remove certain training samples, such as those with outdated or sensitive information, from trained models. Despite these advancements, evaluation of MU methods have been inconsistent, employing different trained models and architectures, and sample removal strategies, which hampers accurate comparison. In addi…
▽ More
Recent advancements in Machine Unlearning (MU) have introduced solutions to selectively remove certain training samples, such as those with outdated or sensitive information, from trained models. Despite these advancements, evaluation of MU methods have been inconsistent, employing different trained models and architectures, and sample removal strategies, which hampers accurate comparison. In addition, prior MU approaches have mainly focused on singular tasks or modalities, which is not comprehensive. To address these limitations, we develop MU-Bench, the first comprehensive benchmark for MU that (i) unifies the sets of deleted samples and trained models, and (ii) provides broad coverage of tasks and data modalities, including previously unexplored domains such as speech and video classification. Our evaluation show that RandLabel and SalUn are the most effective general MU approaches on MU-Bench, and BadT and SCRUB are capable of achieving random performance on the deletion set. We analyze several under-investigated aspects of unlearning, including scalability, the impacts of parameter-efficient fine-tuning and curriculum learning, and susceptibility to dataset biases. MU-Bench provides an easy-to-use package that includes dataset splits, models, and implementations, together with a leader board to enable unified and scalable MU research.
△ Less
Submitted 20 June, 2024;
originally announced June 2024.
-
Complexity-Guided Curriculum Learning for Text Graphs
Authors:
Nidhi Vakil,
Hadi Amiri
Abstract:
Curriculum learning provides a systematic approach to training. It refines training progressively, tailors training to task requirements, and improves generalization through exposure to diverse examples. We present a curriculum learning approach that builds on existing knowledge about text and graph complexity formalisms for training with text graph data. The core part of our approach is a novel d…
▽ More
Curriculum learning provides a systematic approach to training. It refines training progressively, tailors training to task requirements, and improves generalization through exposure to diverse examples. We present a curriculum learning approach that builds on existing knowledge about text and graph complexity formalisms for training with text graph data. The core part of our approach is a novel data scheduler, which employs "spaced repetition" and complexity formalisms to guide the training process. We demonstrate the effectiveness of the proposed approach on several text graph tasks and graph neural network architectures. The proposed model gains more and uses less data; consistently prefers text over graph complexity indices throughout training, while the best curricula derived from text and graph complexity indices are equally effective; and it learns transferable curricula across GNN models and datasets. In addition, we find that both node-level (local) and graph-level (global) graph complexity indices, as well as shallow and traditional text complexity indices play a crucial role in effective curriculum learning.
△ Less
Submitted 22 November, 2023;
originally announced November 2023.
-
Multimodal Machine Unlearning
Authors:
Jiali Cheng,
Hadi Amiri
Abstract:
Machine Unlearning is the process of removing specific training data samples and their corresponding effects from an already trained model. It has significant practical benefits, such as purging private, inaccurate, or outdated information from trained models without the need for complete re-training. Unlearning within a multimodal setting presents unique challenges due to the intrinsic dependenci…
▽ More
Machine Unlearning is the process of removing specific training data samples and their corresponding effects from an already trained model. It has significant practical benefits, such as purging private, inaccurate, or outdated information from trained models without the need for complete re-training. Unlearning within a multimodal setting presents unique challenges due to the intrinsic dependencies between different data modalities and the expensive cost of training on large multimodal datasets and architectures. Current approaches to machine unlearning have not fully addressed these challenges. To bridge this gap, we introduce MMUL, a machine unlearning approach specifically designed for multimodal data and models. MMUL formulates the multimodal unlearning task by focusing on three key properties: (a): modality decoupling, which effectively decouples the association between individual unimodal data points within multimodal inputs marked for deletion, rendering them as unrelated data points within the model's context, (b): unimodal knowledge retention, which retains the unimodal representation capability of the model post-unlearning, and (c): multimodal knowledge retention, which retains the multimodal representation capability of the model post-unlearning. MMUL is efficient to train and is not constrained by the requirement of using a strongly convex loss. Experiments on two multimodal models and four multimodal benchmark datasets, including vision-language and graph-language datasets, show that MMUL outperforms existing baselines, gaining an average improvement of +17.6 points against the best-performing unimodal baseline in distinguishing between deleted and remaining data. In addition, MMUL can largely maintain pre-existing knowledge of the original model post unlearning, with a performance gap of only 0.3 points compared to retraining a new model from scratch.
△ Less
Submitted 18 November, 2023;
originally announced November 2023.
-
Ling-CL: Understanding NLP Models through Linguistic Curricula
Authors:
Mohamed Elgaar,
Hadi Amiri
Abstract:
We employ a characterization of linguistic complexity from psycholinguistic and language acquisition research to develop data-driven curricula to understand the underlying linguistic knowledge that models learn to address NLP tasks. The novelty of our approach is in the development of linguistic curricula derived from data, existing knowledge about linguistic complexity, and model behavior during…
▽ More
We employ a characterization of linguistic complexity from psycholinguistic and language acquisition research to develop data-driven curricula to understand the underlying linguistic knowledge that models learn to address NLP tasks. The novelty of our approach is in the development of linguistic curricula derived from data, existing knowledge about linguistic complexity, and model behavior during training. By analyzing several benchmark NLP datasets, our curriculum learning approaches identify sets of linguistic metrics (indices) that inform the challenges and reasoning required to address each task. Our work will inform future research in all NLP areas, allowing linguistic complexity to be considered early in the research and development process. In addition, our work prompts an examination of gold standards and fair evaluation in NLP.
△ Less
Submitted 30 October, 2023;
originally announced October 2023.
-
Large Language Models for Spatial Trajectory Patterns Mining
Authors:
Zheng Zhang,
Hossein Amiri,
Zhenke Liu,
Andreas Züfle,
Liang Zhao
Abstract:
Identifying anomalous human spatial trajectory patterns can indicate dynamic changes in mobility behavior with applications in domains like infectious disease monitoring and elderly care. Recent advancements in large language models (LLMs) have demonstrated their ability to reason in a manner akin to humans. This presents significant potential for analyzing temporal patterns in human mobility. In…
▽ More
Identifying anomalous human spatial trajectory patterns can indicate dynamic changes in mobility behavior with applications in domains like infectious disease monitoring and elderly care. Recent advancements in large language models (LLMs) have demonstrated their ability to reason in a manner akin to humans. This presents significant potential for analyzing temporal patterns in human mobility. In this paper, we conduct empirical studies to assess the capabilities of leading LLMs like GPT-4 and Claude-2 in detecting anomalous behaviors from mobility data, by comparing to specialized methods. Our key findings demonstrate that LLMs can attain reasonable anomaly detection performance even without any specific cues. In addition, providing contextual clues about potential irregularities could further enhances their prediction efficacy. Moreover, LLMs can provide reasonable explanations for their judgments, thereby improving transparency. Our work provides insights on the strengths and limitations of LLMs for human spatial trajectory analysis.
△ Less
Submitted 7 October, 2023;
originally announced October 2023.
-
Linking Symptom Inventories using Semantic Textual Similarity
Authors:
Eamonn Kennedy,
Shashank Vadlamani,
Hannah M Lindsey,
Kelly S Peterson,
Kristen Dams OConnor,
Kenton Murray,
Ronak Agarwal,
Houshang H Amiri,
Raeda K Andersen,
Talin Babikian,
David A Baron,
Erin D Bigler,
Karen Caeyenberghs,
Lisa Delano-Wood,
Seth G Disner,
Ekaterina Dobryakova,
Blessen C Eapen,
Rachel M Edelstein,
Carrie Esopenko,
Helen M Genova,
Elbert Geuze,
Naomi J Goodrich-Hunsaker,
Jordan Grafman,
Asta K Haberg,
Cooper B Hodges
, et al. (57 additional authors not shown)
Abstract:
An extensive library of symptom inventories has been developed over time to measure clinical symptoms, but this variety has led to several long standing issues. Most notably, results drawn from different settings and studies are not comparable, which limits reproducibility. Here, we present an artificial intelligence (AI) approach using semantic textual similarity (STS) to link symptoms and scores…
▽ More
An extensive library of symptom inventories has been developed over time to measure clinical symptoms, but this variety has led to several long standing issues. Most notably, results drawn from different settings and studies are not comparable, which limits reproducibility. Here, we present an artificial intelligence (AI) approach using semantic textual similarity (STS) to link symptoms and scores across previously incongruous symptom inventories. We tested the ability of four pre-trained STS models to screen thousands of symptom description pairs for related content - a challenging task typically requiring expert panels. Models were tasked to predict symptom severity across four different inventories for 6,607 participants drawn from 16 international data sources. The STS approach achieved 74.8% accuracy across five tasks, outperforming other models tested. This work suggests that incorporating contextual, semantic information can assist expert decision-making processes, yielding gains for both general and disease-specific clinical assessment.
△ Less
Submitted 8 September, 2023;
originally announced September 2023.
-
Curriculum Learning for Graph Neural Networks: A Multiview Competence-based Approach
Authors:
Nidhi Vakil,
Hadi Amiri
Abstract:
A curriculum is a planned sequence of learning materials and an effective one can make learning efficient and effective for both humans and machines. Recent studies developed effective data-driven curriculum learning approaches for training graph neural networks in language applications. However, existing curriculum learning approaches often employ a single criterion of difficulty in their trainin…
▽ More
A curriculum is a planned sequence of learning materials and an effective one can make learning efficient and effective for both humans and machines. Recent studies developed effective data-driven curriculum learning approaches for training graph neural networks in language applications. However, existing curriculum learning approaches often employ a single criterion of difficulty in their training paradigms. In this paper, we propose a new perspective on curriculum learning by introducing a novel approach that builds on graph complexity formalisms (as difficulty criteria) and model competence during training. The model consists of a scheduling scheme which derives effective curricula by accounting for different views of sample difficulty and model competence during training. The proposed solution advances existing research in curriculum learning for graph neural networks with the ability to incorporate a fine-grained spectrum of graph difficulty criteria in their training paradigms. Experimental results on real-world link prediction and node classification tasks illustrate the effectiveness of the proposed approach.
△ Less
Submitted 17 July, 2023;
originally announced July 2023.
-
HuCurl: Human-induced Curriculum Discovery
Authors:
Mohamed Elgaar,
Hadi Amiri
Abstract:
We introduce the problem of curriculum discovery and describe a curriculum learning framework capable of discovering effective curricula in a curriculum space based on prior knowledge about sample difficulty. Using annotation entropy and loss as measures of difficulty, we show that (i): the top-performing discovered curricula for a given model and dataset are often non-monotonic as opposed to mono…
▽ More
We introduce the problem of curriculum discovery and describe a curriculum learning framework capable of discovering effective curricula in a curriculum space based on prior knowledge about sample difficulty. Using annotation entropy and loss as measures of difficulty, we show that (i): the top-performing discovered curricula for a given model and dataset are often non-monotonic as opposed to monotonic curricula in existing literature, (ii): the prevailing easy-to-hard or hard-to-easy transition curricula are often at the risk of underperforming, and (iii): the curricula discovered for smaller datasets and models perform well on larger datasets and models respectively. The proposed framework encompasses some of the existing curriculum learning approaches and can discover curricula that outperform them across several NLP tasks.
△ Less
Submitted 14 July, 2023;
originally announced July 2023.
-
Toward Real-Time Image Annotation Using Marginalized Coupled Dictionary Learning
Authors:
Seyed Mahdi Roostaiyan,
Mohammad Mehdi Hosseini,
Mahya Mohammadi Kashani,
S. Hamid Amiri
Abstract:
In most image retrieval systems, images include various high-level semantics, called tags or annotations. Virtually all the state-of-the-art image annotation methods that handle imbalanced labeling are search-based techniques which are time-consuming. In this paper, a novel coupled dictionary learning approach is proposed to learn a limited number of visual prototypes and their corresponding seman…
▽ More
In most image retrieval systems, images include various high-level semantics, called tags or annotations. Virtually all the state-of-the-art image annotation methods that handle imbalanced labeling are search-based techniques which are time-consuming. In this paper, a novel coupled dictionary learning approach is proposed to learn a limited number of visual prototypes and their corresponding semantics simultaneously. This approach leads to a real-time image annotation procedure. Another contribution of this paper is that utilizes a marginalized loss function instead of the squared loss function that is inappropriate for image annotation with imbalanced labels. We have employed a marginalized loss function in our method to leverage a simple and effective method of prototype updating. Meanwhile, we have introduced ${\ell}_1$ regularization on semantic prototypes to preserve the sparse and imbalanced nature of labels in learned semantic prototypes. Finally, comprehensive experimental results on various datasets demonstrate the efficiency of the proposed method for image annotation tasks in terms of accuracy and time. The reference implementation is publicly available on https://github.com/hamid-amiri/MCDL-Image-Annotation.
△ Less
Submitted 17 April, 2023; v1 submitted 13 April, 2023;
originally announced April 2023.
-
Design interpretable experience of dynamical feed forward machine learning model for forecasting NASDAQ
Authors:
Pouriya Khalilian,
Sara Azizi,
Mohammad Hossein Amiri,
Javad T. Firouzjaee
Abstract:
National Association of Securities Dealers Automated Quotations(NASDAQ) is an American stock exchange based. It is one of the most valuable stock economic indices in the world and is located in New York City \cite{pagano2008quality}. The volatility of the stock market and the influence of economic indicators such as crude oil, gold, and the dollar in the stock market, and NASDAQ shares are also af…
▽ More
National Association of Securities Dealers Automated Quotations(NASDAQ) is an American stock exchange based. It is one of the most valuable stock economic indices in the world and is located in New York City \cite{pagano2008quality}. The volatility of the stock market and the influence of economic indicators such as crude oil, gold, and the dollar in the stock market, and NASDAQ shares are also affected and have a volatile and chaotic nature \cite{firouzjaee2022lstm}.In this article, we have examined the effect of oil, dollar, gold, and the volatility of the stock market in the economic market, and then we have also examined the effect of these indicators on NASDAQ stocks. Then we started to analyze the impact of the feedback on the past prices of NASDAQ stocks and its impact on the current price. Using PCA and Linear Regression algorithm, we have designed an optimal dynamic learning experience for modeling these stocks. The results obtained from the quantitative analysis are consistent with the results of the qualitative analysis of economic studies, and the modeling done with the optimal dynamic experience of machine learning justifies the current price of NASDAQ shares.
△ Less
Submitted 22 December, 2022;
originally announced December 2022.
-
DLDNN: Deterministic Lateral Displacement Design Automation by Neural Networks
Authors:
Farzad Vatandoust,
Hoseyn A. Amiri,
Sima Mas-hafi
Abstract:
Size-based separation of bioparticles/cells is crucial to a variety of biomedical processing steps for applications such as exosomes and DNA isolation. Design and improvement of such microfluidic devices is a challenge to best answer the demand for producing homogeneous end-result for study and use. Deterministic lateral displacement (DLD) exploits a similar principle that has drawn extensive atte…
▽ More
Size-based separation of bioparticles/cells is crucial to a variety of biomedical processing steps for applications such as exosomes and DNA isolation. Design and improvement of such microfluidic devices is a challenge to best answer the demand for producing homogeneous end-result for study and use. Deterministic lateral displacement (DLD) exploits a similar principle that has drawn extensive attention over years. However, the lack of predictive understanding of the particle trajectory and its induced mode makes designing a DLD device an iterative procedure. Therefore, this paper investigates a fast versatile design automation platform to address this issue. To do so, convolutional and artificial neural networks were employed to learn velocity fields and critical diameters of a wide range of DLD configurations. Later, these networks were combined with a multi-objective evolutionary algorithm to construct the automation tool. After ensuring the accuracy of the neural networks, the developed tool was tested for 12 critical conditions. Reaching the imposed conditions, the automation components performed reliably with errors of less than 4%. Moreover, this tool is generalizable to other field-based problems and since the neural network is an integral part of this method, it enables transfer learning for similar physics. All the codes generated and used in this study alongside the pre-trained neural network models are available on https://github.com/HoseynAAmiri/DLDNN.
△ Less
Submitted 30 August, 2022;
originally announced August 2022.
-
Generic and Trend-aware Curriculum Learning for Relation Extraction in Graph Neural Networks
Authors:
Nidhi Vakil,
Hadi Amiri
Abstract:
We present a generic and trend-aware curriculum learning approach for graph neural networks. It extends existing approaches by incorporating sample-level loss trends to better discriminate easier from harder samples and schedule them for training. The model effectively integrates textual and structural information for relation extraction in text graphs. Experimental results show that the model pro…
▽ More
We present a generic and trend-aware curriculum learning approach for graph neural networks. It extends existing approaches by incorporating sample-level loss trends to better discriminate easier from harder samples and schedule them for training. The model effectively integrates textual and structural information for relation extraction in text graphs. Experimental results show that the model provides robust estimations of sample difficulty and shows sizable improvement over the state-of-the-art approaches across several datasets.
△ Less
Submitted 8 June, 2022; v1 submitted 17 May, 2022;
originally announced May 2022.
-
A Robust and Efficient Method for Improving Accuracy of License Plate Characters Recognition
Authors:
Reza Azad,
Hamid Reza Shayegh,
Hamed Amiri
Abstract:
License Plate Recognition (LPR) plays an important role on the traffic monitoring and parking management. A robust and efficient method for enhancing accuracy of license plate characters recognition based on K Nearest Neighbours (K-NN) classifier is presented in this paper. The system first prepares a contour form of the extracted character, then the angle and distance feature information about th…
▽ More
License Plate Recognition (LPR) plays an important role on the traffic monitoring and parking management. A robust and efficient method for enhancing accuracy of license plate characters recognition based on K Nearest Neighbours (K-NN) classifier is presented in this paper. The system first prepares a contour form of the extracted character, then the angle and distance feature information about the character is extracted and finally K-NN classifier is used to character recognition. Angle and distance features of a character have been computed based on distribution of points on the bitmap image of character. In K-NN method, the Euclidean distance between testing point and reference points is calculated in order to find the k-nearest neighbours. We evaluated our method on the available dataset that contain 1200 sample. Using 70% samples for training, we tested our method on whole samples and obtained 99% correct recognition rate.Further, we achieved average 99.41% accuracy using three/strategy validation technique on 1200 dataset.
△ Less
Submitted 16 September, 2014; v1 submitted 24 July, 2014;
originally announced July 2014.
-
Repairing and Inpainting Damaged Images using Diffusion Tensor
Authors:
Faouzi Benzarti,
Hamid Amiri
Abstract:
Removing or repairing the imperfections of a digital images or videos is a very active and attractive field of research belonging to the image inpainting technique. This later has a wide range of applications, such as removing scratches in old photographic image, removing text and logos or creating cartoon and artistic effects. In this paper, we propose an efficient method to repair a damaged imag…
▽ More
Removing or repairing the imperfections of a digital images or videos is a very active and attractive field of research belonging to the image inpainting technique. This later has a wide range of applications, such as removing scratches in old photographic image, removing text and logos or creating cartoon and artistic effects. In this paper, we propose an efficient method to repair a damaged image based on a non linear diffusion tensor. The idea is to track perfectly the local geometry of the damaged image and allowing diffusion only in the isophotes curves direction. To illustrate the effective performance of our method, we present some experimental results on test and real photographic color images
△ Less
Submitted 9 May, 2013;
originally announced May 2013.
-
Speckle Noise Reduction in Medical Ultrasound Images
Authors:
Faouzi Benzarti,
Hamid Amiri
Abstract:
Ultrasound imaging is an incontestable vital tool for diagnosis, it provides in non-invasive manner the internal structure of the body to detect eventually diseases or abnormalities tissues. Unfortunately, the presence of speckle noise in these images affects edges and fine details which limit the contrast resolution and make diagnostic more difficult. In this paper, we propose a denoising approac…
▽ More
Ultrasound imaging is an incontestable vital tool for diagnosis, it provides in non-invasive manner the internal structure of the body to detect eventually diseases or abnormalities tissues. Unfortunately, the presence of speckle noise in these images affects edges and fine details which limit the contrast resolution and make diagnostic more difficult. In this paper, we propose a denoising approach which combines logarithmic transformation and a non linear diffusion tensor. Since speckle noise is multiplicative and nonwhite process, the logarithmic transformation is a reasonable choice to convert signaldependent or pure multiplicative noise to an additive one. The key idea from using diffusion tensor is to adapt the flow diffusion towards the local orientation by applying anisotropic diffusion along the coherent structure direction of interesting features in the image. To illustrate the effective performance of our algorithm, we present some experimental results on synthetically and real echographic images.
△ Less
Submitted 6 May, 2013;
originally announced May 2013.
-
Robust Noise Filtering in Image Sequences
Authors:
Soumaya Hichri,
Faouzi Benzarti,
Hamid Amiri
Abstract:
Image sequences filtering have recently become a very important technical problem especially with the advent of new technology in multimedia and video systems applications. Often image sequences are corrupted by some amount of noise introduced by the image sensor and therefore inherently present in the imaging process. The main problem in the image sequences is how to deal with spatio-temporal and…
▽ More
Image sequences filtering have recently become a very important technical problem especially with the advent of new technology in multimedia and video systems applications. Often image sequences are corrupted by some amount of noise introduced by the image sensor and therefore inherently present in the imaging process. The main problem in the image sequences is how to deal with spatio-temporal and non stationary signals. In this paper, we propose a robust method for noise removal of image sequence based on coupled spatial and temporal anisotropic diffusion. The idea is to achieve an adaptive smoothing in both spatial and temporal directions, by solving a nonlinear diffusion equation. This allows removing noise while preserving all spatial and temporal discontinuities
△ Less
Submitted 17 April, 2013;
originally announced April 2013.
-
About Multichannel Speech Signal Extraction and Separation Techniques
Authors:
Adel Hidri,
Souad Meddeb,
Hamid Amiri
Abstract:
The extraction of a desired speech signal from a noisy environment has become a challenging issue. In the recent years, the scientific community has particularly focused on multichannel techniques which are dealt with in this review. In fact, this study tries to classify these multichannel techniques into three main ones: Beamforming, Independent Com-ponent Analysis (ICA) and Time Frequency (T-F)…
▽ More
The extraction of a desired speech signal from a noisy environment has become a challenging issue. In the recent years, the scientific community has particularly focused on multichannel techniques which are dealt with in this review. In fact, this study tries to classify these multichannel techniques into three main ones: Beamforming, Independent Com-ponent Analysis (ICA) and Time Frequency (T-F) masking. This paper also highlights their advantages and drawbacks. However these previously mentioned techniques could not afford satisfactory results. This fact leads to the idea that a combination of those techniques, which is depicted along this study, may probably provide more efficient results. In-deed, giving the fact that those approaches are still be considered as being not totally efficient, has led us to review these mentioned above in the hope that further researches will provide this domain with suitable innovations.
△ Less
Submitted 31 December, 2012;
originally announced December 2012.
-
Discrimination between Arabic and Latin from bilingual documents
Authors:
Sofiene Haboubi,
Samia Maddouri,
Hamid Amiri
Abstract:
2011 International Conference on Communications, Computing and Control Applications (CCCA)
2011 International Conference on Communications, Computing and Control Applications (CCCA)
△ Less
Submitted 7 April, 2012;
originally announced April 2012.
-
Identification of arabic word from bilingual text using character features
Authors:
Sofiene Haboubi,
Samia Maddouri,
Hamid Amiri
Abstract:
The identification of the language of the script is an important stage in the process of recognition of the writing. There are several works in this research area, which treat various languages. Most of the used methods are global or statistical. In this present paper, we study the possibility of using the features of scripts to identify the language. The identification of the language of the scri…
▽ More
The identification of the language of the script is an important stage in the process of recognition of the writing. There are several works in this research area, which treat various languages. Most of the used methods are global or statistical. In this present paper, we study the possibility of using the features of scripts to identify the language. The identification of the language of the script by characteristics returns the identification in the case of multilingual documents less difficult. We present by this work, a study on the possibility of using the structural features to identify the Arabic language from an Arabic / Latin text.
△ Less
Submitted 17 March, 2011;
originally announced March 2011.
