Showing 1–4 of 4 results for author: Lê, N C

Self-Organization Towards $1/f$ Noise in Deep Neural Networks

Authors: Nicholas Chong Jia Le, Ling Feng

Abstract: The presence of $1/f$ noise, also known as pink noise, is a well-established phenomenon in biological neural networks, and is thought to play an important role in information processing in the brain. In this study, we find that such $1/f$ noise is also found in deep neural networks trained on natural language, resembling that of their biological counterparts. Specifically, we trained Long Short-Te… ▽ More The presence of $1/f$ noise, also known as pink noise, is a well-established phenomenon in biological neural networks, and is thought to play an important role in information processing in the brain. In this study, we find that such $1/f$ noise is also found in deep neural networks trained on natural language, resembling that of their biological counterparts. Specifically, we trained Long Short-Term Memory (LSTM) networks on the `IMDb' AI benchmark dataset, then measured the neuron activations. The detrended fluctuation analysis (DFA) on the time series of the different neurons demonstrate clear $1/f$ patterns, which is absent in the time series of the inputs to the LSTM. Interestingly, when the neural network is at overcapacity, having more than enough neurons to achieve the learning task, the activation patterns deviate from $1/f$ noise and shifts towards white noise. This is because many of the neurons are not effectively used, showing little fluctuations when fed with input data. We further examine the exponent values in the $1/f$ noise in ``internal" and ``external" activations in the LSTM cell, finding some resemblance in the variations of the exponents in fMRI signals of the human brain. Our findings further supports the hypothesis that $1/f$ noise is a signature of optimal learning. With deep learning models approaching or surpassing humans in certain tasks, and being more ``experimentable'' than their biological counterparts, our study suggests that they are good candidates to understand the fundamental origins of $1/f$ noise. △ Less

Submitted 1 April, 2024; v1 submitted 20 January, 2023; originally announced January 2023.

MAGNeto: An Efficient Deep Learning Method for the Extractive Tags Summarization Problem

Authors: Hieu Trong Phung, Anh Tuan Vu, Tung Dinh Nguyen, Lam Thanh Do, Giang Nam Ngo, Trung Thanh Tran, Ngoc C. Lê

Abstract: In this work, we study a new image annotation task named Extractive Tags Summarization (ETS). The goal is to extract important tags from the context lying in an image and its corresponding tags. We adjust some state-of-the-art deep learning models to utilize both visual and textual information. Our proposed solution consists of different widely used blocks like convolutional and self-attention lay… ▽ More In this work, we study a new image annotation task named Extractive Tags Summarization (ETS). The goal is to extract important tags from the context lying in an image and its corresponding tags. We adjust some state-of-the-art deep learning models to utilize both visual and textual information. Our proposed solution consists of different widely used blocks like convolutional and self-attention layers, together with a novel idea of combining auxiliary loss functions and the gating mechanism to glue and elevate these fundamental components and form a unified architecture. Besides, we introduce a loss function that aims to reduce the imbalance of the training data and a simple but effective data augmentation technique dedicated to alleviates the effect of outliers on the final results. Last but not least, we explore an unsupervised pre-training strategy to further boost the performance of the model by making use of the abundant amount of available unlabeled data. Our model shows the good results as 90% $F_\text{1}$ score on the public NUS-WIDE benchmark, and 50% $F_\text{1}$ score on a noisy large-scale real-world private dataset. Source code for reproducing the experiments is publicly available at: https://github.com/pixta-dev/labteam △ Less

Submitted 9 November, 2020; originally announced November 2020.

On the Vietnamese Name Entity Recognition: A Deep Learning Method Approach

Authors: Ngoc C. Lê, Ngoc-Yen Nguyen, Anh-Duong Trinh

Abstract: Named entity recognition (NER) plays an important role in text-based information retrieval. In this paper, we combine Bidirectional Long Short-Term Memory (Bi-LSTM) \cite{hochreiter1997,schuster1997} with Conditional Random Field (CRF) \cite{lafferty2001} to create a novel deep learning model for the NER problem. Each word as input of the deep learning model is represented by a Word2vec-trained ve… ▽ More Named entity recognition (NER) plays an important role in text-based information retrieval. In this paper, we combine Bidirectional Long Short-Term Memory (Bi-LSTM) \cite{hochreiter1997,schuster1997} with Conditional Random Field (CRF) \cite{lafferty2001} to create a novel deep learning model for the NER problem. Each word as input of the deep learning model is represented by a Word2vec-trained vector. A word embedding set trained from about one million articles in 2018 collected through a Vietnamese news portal (baomoi.com). In addition, we concatenate a Word2Vec\cite{mikolov2013}-trained vector with semantic feature vector (Part-Of-Speech (POS) tagging, chunk-tag) and hidden syntactic feature vector (extracted by Bi-LSTM nerwork) to achieve the (so far best) result in Vietnamese NER system. The result was conducted on the data set VLSP2016 (Vietnamese Language and Speech Processing 2016 \cite{vlsp2016}) competition. △ Less

Submitted 18 November, 2019; originally announced December 2019.

An Application of Random Walk on Fake Account Detection Problem: A Hybrid Approach

Authors: Ngoc C. Lê, Manh-Tuan Dao, Hoang-Linh Nguyen, Tuyet-Nhi Nguyen, Hue Vu

Abstract: Social networks play a significant role in today's world. The importance of social networks, for example Facebook or Twitter, are undeniable. However, they also have many issues. One of which is the need for a defense mechanism against fake accounts. It is obviously not a trivial task to separate fake accounts from authentic ones. In this paper, we propose a ranking scheme, comprising of both grap… ▽ More Social networks play a significant role in today's world. The importance of social networks, for example Facebook or Twitter, are undeniable. However, they also have many issues. One of which is the need for a defense mechanism against fake accounts. It is obviously not a trivial task to separate fake accounts from authentic ones. In this paper, we propose a ranking scheme, comprising of both graph based and feature based approaches to aid the detection of fake Facebook profiles. Utilizing Support Vector Machine (SVM) \cite{cortes1995} and SybilWalk \cite{JWZ17}, the model achieved high accuracy over the set of ten thousands Vietnamese Facebook accounts. △ Less

Submitted 18 November, 2019; originally announced November 2019.