-
The Accuracy of Domain Specific and Descriptive Analysis Generated by Large Language Models
Authors:
Denish Omondi Otieno,
Faranak Abri,
Sima Siami-Namini,
Akbar Siami Namin
Abstract:
Large language models (LLMs) have attracted considerable attention as they are capable of showcasing impressive capabilities generating comparable high-quality responses to human inputs. LLMs, can not only compose textual scripts such as emails and essays but also executable programming code. Contrary, the automated reasoning capability of these LLMs in performing statistically-driven descriptive…
▽ More
Large language models (LLMs) have attracted considerable attention as they are capable of showcasing impressive capabilities generating comparable high-quality responses to human inputs. LLMs, can not only compose textual scripts such as emails and essays but also executable programming code. Contrary, the automated reasoning capability of these LLMs in performing statistically-driven descriptive analysis, particularly on user-specific data and as personal assistants to users with limited background knowledge in an application domain who would like to carry out basic, as well as advanced statistical and domain-specific analysis is not yet fully explored. More importantly, the performance of these LLMs has not been compared and discussed in detail when domain-specific data analysis tasks are needed. This study, consequently, explores whether LLMs can be used as generative AI-based personal assistants to users with minimal background knowledge in an application domain infer key data insights. To demonstrate the performance of the LLMs, the study reports a case study through which descriptive statistical analysis, as well as Natural Language Processing (NLP) based investigations, are performed on a number of phishing emails with the objective of comparing the accuracy of the results generated by LLMs to the ones produced by analysts. The experimental results show that LangChain and the Generative Pre-trained Transformer (GPT-4) excel in numerical reasoning tasks i.e., temporal statistical analysis, achieve competitive correlation with human judgments on feature engineering tasks while struggle to some extent on domain specific knowledge reasoning, where domain-specific knowledge is required.
△ Less
Submitted 29 May, 2024;
originally announced May 2024.
-
The Performance of Sequential Deep Learning Models in Detecting Phishing Websites Using Contextual Features of URLs
Authors:
Saroj Gopali,
Akbar S. Namin,
Faranak Abri,
Keith S. Jones
Abstract:
Cyber attacks continue to pose significant threats to individuals and organizations, stealing sensitive data such as personally identifiable information, financial information, and login credentials. Hence, detecting malicious websites before they cause any harm is critical to preventing fraud and monetary loss. To address the increasing number of phishing attacks, protective mechanisms must be hi…
▽ More
Cyber attacks continue to pose significant threats to individuals and organizations, stealing sensitive data such as personally identifiable information, financial information, and login credentials. Hence, detecting malicious websites before they cause any harm is critical to preventing fraud and monetary loss. To address the increasing number of phishing attacks, protective mechanisms must be highly responsive, adaptive, and scalable. Fortunately, advances in the field of machine learning, coupled with access to vast amounts of data, have led to the adoption of various deep learning models for timely detection of these cyber crimes. This study focuses on the detection of phishing websites using deep learning models such as Multi-Head Attention, Temporal Convolutional Network (TCN), BI-LSTM, and LSTM where URLs of the phishing websites are treated as a sequence. The results demonstrate that Multi-Head Attention and BI-LSTM model outperform some other deep learning-based algorithms such as TCN and LSTM in producing better precision, recall, and F1-scores.
△ Less
Submitted 15 April, 2024;
originally announced April 2024.
-
Exploiting Large Language Models (LLMs) through Deception Techniques and Persuasion Principles
Authors:
Sonali Singh,
Faranak Abri,
Akbar Siami Namin
Abstract:
With the recent advent of Large Language Models (LLMs), such as ChatGPT from OpenAI, BARD from Google, Llama2 from Meta, and Claude from Anthropic AI, gain widespread use, ensuring their security and robustness is critical. The widespread use of these language models heavily relies on their reliability and proper usage of this fascinating technology. It is crucial to thoroughly test these models t…
▽ More
With the recent advent of Large Language Models (LLMs), such as ChatGPT from OpenAI, BARD from Google, Llama2 from Meta, and Claude from Anthropic AI, gain widespread use, ensuring their security and robustness is critical. The widespread use of these language models heavily relies on their reliability and proper usage of this fascinating technology. It is crucial to thoroughly test these models to not only ensure its quality but also possible misuses of such models by potential adversaries for illegal activities such as hacking. This paper presents a novel study focusing on exploitation of such large language models against deceptive interactions. More specifically, the paper leverages widespread and borrows well-known techniques in deception theory to investigate whether these models are susceptible to deceitful interactions.
This research aims not only to highlight these risks but also to pave the way for robust countermeasures that enhance the security and integrity of language models in the face of sophisticated social engineering tactics. Through systematic experiments and analysis, we assess their performance in these critical security domains. Our results demonstrate a significant finding in that these large language models are susceptible to deception and social engineering attacks.
△ Less
Submitted 24 November, 2023;
originally announced November 2023.
-
A Comparative Study of Detecting Anomalies in Time Series Data Using LSTM and TCN Models
Authors:
Saroj Gopali,
Faranak Abri,
Sima Siami-Namini,
Akbar Siami Namin
Abstract:
There exist several data-driven approaches that enable us model time series data including traditional regression-based modeling approaches (i.e., ARIMA). Recently, deep learning techniques have been introduced and explored in the context of time series analysis and prediction. A major research question to ask is the performance of these many variations of deep learning techniques in predicting ti…
▽ More
There exist several data-driven approaches that enable us model time series data including traditional regression-based modeling approaches (i.e., ARIMA). Recently, deep learning techniques have been introduced and explored in the context of time series analysis and prediction. A major research question to ask is the performance of these many variations of deep learning techniques in predicting time series data. This paper compares two prominent deep learning modeling techniques. The Recurrent Neural Network (RNN)-based Long Short-Term Memory (LSTM) and the convolutional Neural Network (CNN)-based Temporal Convolutional Networks (TCN) are compared and their performance and training time are reported. According to our experimental results, both modeling techniques perform comparably having TCN-based models outperform LSTM slightly. Moreover, the CNN-based TCN model builds a stable model faster than the RNN-based LSTM models.
△ Less
Submitted 16 December, 2021;
originally announced December 2021.
-
Toward Explainable Users: Using NLP to Enable AI to Understand Users' Perceptions of Cyber Attacks
Authors:
Faranak Abri,
Luis Felipe Gutierrez,
Chaitra T. Kulkarni,
Akbar Siami Namin,
Keith S. Jones
Abstract:
To understand how end-users conceptualize consequences of cyber security attacks, we performed a card sorting study, a well-known technique in Cognitive Sciences, where participants were free to group the given consequences of chosen cyber attacks into as many categories as they wished using rationales they see fit. The results of the open card sorting study showed a large amount of inter-particip…
▽ More
To understand how end-users conceptualize consequences of cyber security attacks, we performed a card sorting study, a well-known technique in Cognitive Sciences, where participants were free to group the given consequences of chosen cyber attacks into as many categories as they wished using rationales they see fit. The results of the open card sorting study showed a large amount of inter-participant variation making the research team wonder how the consequences of security attacks were comprehended by the participants. As an exploration of whether it is possible to explain user's mental model and behavior through Artificial Intelligence (AI) techniques, the research team compared the card sorting data with the outputs of a number of Natural Language Processing (NLP) techniques with the goal of understanding how participants perceived and interpreted the consequences of cyber attacks written in natural languages. The results of the NLP-based exploration methods revealed an interesting observation implying that participants had mostly employed checking individual keywords in each sentence to group cyber attack consequences together and less considered the semantics behind the description of consequences of cyber attacks. The results reported in this paper are seemingly useful and important for cyber attacks comprehension from user's perspectives. To the best of our knowledge, this paper is the first introducing the use of AI techniques in explaining and modeling users' behavior and their perceptions about a context. The novel idea introduced here is about explaining users using AI.
△ Less
Submitted 3 June, 2021;
originally announced June 2021.
-
Phishing Detection through Email Embeddings
Authors:
Luis Felipe Gutiérrez,
Faranak Abri,
Miriam Armstrong,
Akbar Siami Namin,
Keith S. Jones
Abstract:
The problem of detecting phishing emails through machine learning techniques has been discussed extensively in the literature. Conventional and state-of-the-art machine learning algorithms have demonstrated the possibility of building classifiers with high accuracy. The existing research studies treat phishing and genuine emails through general indicators and thus it is not exactly clear what phis…
▽ More
The problem of detecting phishing emails through machine learning techniques has been discussed extensively in the literature. Conventional and state-of-the-art machine learning algorithms have demonstrated the possibility of building classifiers with high accuracy. The existing research studies treat phishing and genuine emails through general indicators and thus it is not exactly clear what phishing features are contributing to variations of the classifiers. In this paper, we crafted a set of phishing and legitimate emails with similar indicators in order to investigate whether these cues are captured or disregarded by email embeddings, i.e., vectorizations. We then fed machine learning classifiers with the carefully crafted emails to find out about the performance of email embeddings developed. Our results show that using these indicators, email embeddings techniques is effective for classifying emails as phishing or legitimate.
△ Less
Submitted 28 December, 2020;
originally announced December 2020.
-
Predicting Emotions Perceived from Sounds
Authors:
Faranak Abri,
Luis Felipe Gutiérrez,
Akbar Siami Namin,
David R. W. Sears,
Keith S. Jones
Abstract:
Sonification is the science of communication of data and events to users through sounds. Auditory icons, earcons, and speech are the common auditory display schemes utilized in sonification, or more specifically in the use of audio to convey information. Once the captured data are perceived, their meanings, and more importantly, intentions can be interpreted more easily and thus can be employed as…
▽ More
Sonification is the science of communication of data and events to users through sounds. Auditory icons, earcons, and speech are the common auditory display schemes utilized in sonification, or more specifically in the use of audio to convey information. Once the captured data are perceived, their meanings, and more importantly, intentions can be interpreted more easily and thus can be employed as a complement to visualization techniques. Through auditory perception it is possible to convey information related to temporal, spatial, or some other context-oriented information. An important research question is whether the emotions perceived from these auditory icons or earcons are predictable in order to build an automated sonification platform. This paper conducts an experiment through which several mainstream and conventional machine learning algorithms are developed to study the prediction of emotions perceived from sounds. To do so, the key features of sounds are captured and then are modeled using machine learning algorithms using feature reduction techniques. We observe that it is possible to predict perceived emotions with high accuracy. In particular, the regression based on Random Forest demonstrated its superiority compared to other machine learning algorithms.
△ Less
Submitted 4 December, 2020;
originally announced December 2020.
-
Fake Reviews Detection through Analysis of Linguistic Features
Authors:
Faranak Abri,
Luis Felipe Gutierrez,
Akbar Siami Namin,
Keith S. Jones,
David R. W. Sears
Abstract:
Online reviews play an integral part for success or failure of businesses. Prior to purchasing services or goods, customers first review the online comments submitted by previous customers. However, it is possible to superficially boost or hinder some businesses through posting counterfeit and fake reviews. This paper explores a natural language processing approach to identify fake reviews. We pre…
▽ More
Online reviews play an integral part for success or failure of businesses. Prior to purchasing services or goods, customers first review the online comments submitted by previous customers. However, it is possible to superficially boost or hinder some businesses through posting counterfeit and fake reviews. This paper explores a natural language processing approach to identify fake reviews. We present a detailed analysis of linguistic features for distinguishing fake and trustworthy online reviews. We study 15 linguistic features and measure their significance and importance towards the classification schemes employed in this study. Our results indicate that fake reviews tend to include more redundant terms and pauses, and generally contain longer sentences. The application of several machine learning classification algorithms revealed that we were able to discriminate fake from real reviews with high accuracy using these linguistic features.
△ Less
Submitted 8 October, 2020;
originally announced October 2020.
-
Cloud as an Attack Platform
Authors:
Moitrayee Chatterjee,
Prerit Datta,
Faranak Abri,
Akbar Siami Namin,
Keith S. Jones
Abstract:
We present an exploratory study of responses from $75$ security professionals and ethical hackers in order to understand how they abuse cloud platforms for attack purposes. The participants were recruited at the Black Hat and DEF CON conferences. We presented the participants' with various attack scenarios and asked them to explain the steps they would have carried out for launching the attack in…
▽ More
We present an exploratory study of responses from $75$ security professionals and ethical hackers in order to understand how they abuse cloud platforms for attack purposes. The participants were recruited at the Black Hat and DEF CON conferences. We presented the participants' with various attack scenarios and asked them to explain the steps they would have carried out for launching the attack in each scenario. Participants' responses were studied to understand attackers' mental models, which would improve our understanding of necessary security controls and recommendations regarding precautionary actions to circumvent the exploitation of clouds for malicious activities. We observed that in 93.78% of the responses, participants are abusing cloud services to establish their attack environment and launch attacks.
△ Less
Submitted 14 June, 2020;
originally announced June 2020.
-
Fake Reviews Detection through Ensemble Learning
Authors:
Luis Gutierrez-Espinoza,
Faranak Abri,
Akbar Siami Namin,
Keith S. Jones,
David R. W. Sears
Abstract:
Customers represent their satisfactions of consuming products by sharing their experiences through the utilization of online reviews. Several machine learning-based approaches can automatically detect deceptive and fake reviews. Recently, there have been studies reporting the performance of ensemble learning-based approaches in comparison to conventional machine learning techniques. Motivated by t…
▽ More
Customers represent their satisfactions of consuming products by sharing their experiences through the utilization of online reviews. Several machine learning-based approaches can automatically detect deceptive and fake reviews. Recently, there have been studies reporting the performance of ensemble learning-based approaches in comparison to conventional machine learning techniques. Motivated by the recent trends in ensemble learning, this paper evaluates the performance of ensemble learning-based approaches to identify bogus online information. The application of a number of ensemble learning-based approaches to a collection of fake restaurant reviews that we developed show that these ensemble learning-based approaches detect deceptive information better than conventional machine learning algorithms.
△ Less
Submitted 14 June, 2020;
originally announced June 2020.
-
Launching Stealth Attacks using Cloud
Authors:
Moitrayee Chatterjee,
Prerit Datta,
Faranak Abri,
Akbar Siami Namin,
Keith S. Jones
Abstract:
Cloud computing offers users scalable platforms and low resource cost. At the same time, the off-site location of the resources of this service model makes it more vulnerable to certain types of adversarial actions. Cloud computing has not only gained major user base, but also, it has the features that attackers can leverage to remain anonymous and stealth. With convenient access to data and techn…
▽ More
Cloud computing offers users scalable platforms and low resource cost. At the same time, the off-site location of the resources of this service model makes it more vulnerable to certain types of adversarial actions. Cloud computing has not only gained major user base, but also, it has the features that attackers can leverage to remain anonymous and stealth. With convenient access to data and technology, cloud has turned into an attack platform among other utilization. This paper reports our study to show that cyber attackers heavily abuse the public cloud platforms to setup their attack environments and launch stealth attacks. The paper first reviews types of attacks launched through cloud environment. It then reports case studies through which the processes of launching cyber attacks using clouds are demonstrated.
△ Less
Submitted 14 June, 2020;
originally announced June 2020.
-
The Performance of Machine and Deep Learning Classifiers in Detecting Zero-Day Vulnerabilities
Authors:
Faranak Abri,
Sima Siami-Namini,
Mahdi Adl Khanghah,
Fahimeh Mirza Soltani,
Akbar Siami Namin
Abstract:
The detection of zero-day attacks and vulnerabilities is a challenging problem. It is of utmost importance for network administrators to identify them with high accuracy. The higher the accuracy is, the more robust the defense mechanism will be. In an ideal scenario (i.e., 100% accuracy) the system can detect zero-day malware without being concerned about mistakenly tagging benign files as malware…
▽ More
The detection of zero-day attacks and vulnerabilities is a challenging problem. It is of utmost importance for network administrators to identify them with high accuracy. The higher the accuracy is, the more robust the defense mechanism will be. In an ideal scenario (i.e., 100% accuracy) the system can detect zero-day malware without being concerned about mistakenly tagging benign files as malware or enabling disruptive malicious code running as none-malicious ones. This paper investigates different machine learning algorithms to find out how well they can detect zero-day malware. Through the examination of 34 machine/deep learning classifiers, we found that the random forest classifier offered the best accuracy. The paper poses several research questions regarding the performance of machine and deep learning algorithms when detecting zero-day malware with zero rates for false positive and false negative.
△ Less
Submitted 21 November, 2019;
originally announced November 2019.
