-
Unimodal Intermediate Training for Multimodal Meme Sentiment Classification
Authors:
Muzhaffar Hazman,
Susan McKeever,
Josephine Griffith
Abstract:
Internet Memes remain a challenging form of user-generated content for automated sentiment classification. The availability of labelled memes is a barrier to develo** sentiment classifiers of multimodal memes. To address the shortage of labelled memes, we propose to supplement the training of a multimodal meme classifier with unimodal (image-only and text-only) data. In this work, we present a n…
▽ More
Internet Memes remain a challenging form of user-generated content for automated sentiment classification. The availability of labelled memes is a barrier to develo** sentiment classifiers of multimodal memes. To address the shortage of labelled memes, we propose to supplement the training of a multimodal meme classifier with unimodal (image-only and text-only) data. In this work, we present a novel variant of supervised intermediate training that uses relatively abundant sentiment-labelled unimodal data. Our results show a statistically significant performance improvement from the incorporation of unimodal text data. Furthermore, we show that the training set of labelled memes can be reduced by 40% without reducing the performance of the downstream model.
△ Less
Submitted 1 August, 2023;
originally announced August 2023.
-
Meme Sentiment Analysis Enhanced with Multimodal Spatial Encoding and Facial Embedding
Authors:
Muzhaffar Hazman,
Susan McKeever,
Josephine Griffith
Abstract:
Internet memes are characterised by the interspersing of text amongst visual elements. State-of-the-art multimodal meme classifiers do not account for the relative positions of these elements across the two modalities, despite the latent meaning associated with where text and visual elements are placed. Against two meme sentiment classification datasets, we systematically show performance gains fr…
▽ More
Internet memes are characterised by the interspersing of text amongst visual elements. State-of-the-art multimodal meme classifiers do not account for the relative positions of these elements across the two modalities, despite the latent meaning associated with where text and visual elements are placed. Against two meme sentiment classification datasets, we systematically show performance gains from incorporating the spatial position of visual objects, faces, and text clusters extracted from memes. In addition, we also present facial embedding as an impactful enhancement to image representation in a multimodal meme classifier. Finally, we show that incorporating this spatial information allows our fully automated approaches to outperform their corresponding baselines that rely on additional human validation of OCR-extracted text.
△ Less
Submitted 3 March, 2023;
originally announced March 2023.
-
Exploring the Impact of Noise and Degradations on Heart Sound Classification Models
Authors:
Davoud Shariat Panah,
Andrew Hines,
Susan McKeever
Abstract:
The development of data-driven heart sound classification models has been an active area of research in recent years. To develop such data-driven models in the first place, heart sound signals need to be captured using a signal acquisition device. However, it is almost impossible to capture noise-free heart sound signals due to the presence of internal and external noises in most situations. Such…
▽ More
The development of data-driven heart sound classification models has been an active area of research in recent years. To develop such data-driven models in the first place, heart sound signals need to be captured using a signal acquisition device. However, it is almost impossible to capture noise-free heart sound signals due to the presence of internal and external noises in most situations. Such noises and degradations in heart sound signals can potentially reduce the accuracy of data-driven classification models. Although different techniques have been proposed in the literature to address the noise issue, how and to what extent different noise and degradations in heart sound signals impact the accuracy of data-driven classification models remains unexplored. To answer this question, we produced a synthetic heart sound dataset including normal and abnormal heart sounds contaminated with a large variety of noise and degradations. We used this dataset to investigate the impact of noise and degradation in heart sound recordings on the performance of different classification models. The results show different noises and degradations affect the performance of heart sound classification models to a different extent; some are more problematic for classification models, and others are less destructive. Comparing the findings of this study with the results of a survey we previously carried out with a group of clinicians shows noise and degradations that are more detrimental to classification models are also more disruptive to accurate auscultation. The findings of this study can be leveraged to develop targeted heart sound quality enhancement approaches - which adapt the type and aggressiveness of quality enhancement based on the characteristics of noise and degradation in heart sound signals.
△ Less
Submitted 14 November, 2022;
originally announced November 2022.
-
A Discipline of Programming with Quantities
Authors:
Steve McKeever
Abstract:
In scientific and engineering applications, physical quantities embodied as units of measurement (UoM) are frequently used. The loss of the Mars climate orbiter, attributed to a confusion between the metric and imperial unit systems, popularised the disastrous consequences of incorrectly handling measurement values. Dimensional analysis can be used to ensure expressions containing annotated values…
▽ More
In scientific and engineering applications, physical quantities embodied as units of measurement (UoM) are frequently used. The loss of the Mars climate orbiter, attributed to a confusion between the metric and imperial unit systems, popularised the disastrous consequences of incorrectly handling measurement values. Dimensional analysis can be used to ensure expressions containing annotated values are evaluated correctly. This has led to the development of a large number of libraries, languages and validators to ensure developers can specify and verify UoM information in their designs and codes. Many tools can also automatically convert values between commensurable UoM, such as yards and metres. However these systems do not differentiate between quantities and dimensions. For instance torque and work, which share the same UoM, can not be interchanged because they do not represent the same entity. We present a named quantity layer that complements dimensional analysis by ensuring that values of different quantities are safely managed. Our technique is a mixture of analysis and discipline, where expressions involving multiplications are relegated to functions, in order to ensure that named quantities are handled soundly.
△ Less
Submitted 22 October, 2022;
originally announced October 2022.
-
Addressing the "Leaky Pipeline": A Review and Categorisation of Actions to Recruit and Retain Women in Computing Education
Authors:
Alina Berry,
Susan McKeever,
Brenda Murphy,
Sarah Jane Delany
Abstract:
Gender imbalance in computing education is a well-known issue around the world. The term "leaky pipeline" is often used to describe the lack of retention of women before they progress to senior roles. Numerous initiatives have targeted the problem of the leaky pipeline in recent decades. This paper provides a comprehensive review of initiatives related to techniques used to boost recruitment and r…
▽ More
Gender imbalance in computing education is a well-known issue around the world. The term "leaky pipeline" is often used to describe the lack of retention of women before they progress to senior roles. Numerous initiatives have targeted the problem of the leaky pipeline in recent decades. This paper provides a comprehensive review of initiatives related to techniques used to boost recruitment and retention of women in undergraduate computing and related courses in higher education. The primary aim was to identify interventions or initiatives (which we called "actions") that have shown some effectiveness. A secondary objective was to structure our findings as a categorisation, in order to enable future action discussion, comparison and planning. A particular challenge faced in a significant portion of the work was the lack of evaluation: i.e. the assessment of the direct relationship between the initiatives and the outcomes on retention or recruitment. The actions were categorised into four groups: Policy, Pedagogy, Influence and Support and Promotion and Engagement. Policy actions need support and potentially structural change at institution level. Pedagogy actions are initiatives related to the teaching of computing courses. The Influence and Support category includes actions associated with ways to influence women to choose computing and once enrolled to support and encourage them to stay. Finally, Promotion and Engagement actions are initiatives to promote computing based courses and involve engagement and outreach activities. We present our categorisation, identifying the literature related to actions under each category and subcategory. We discuss the challenges with evaluating the direct impact of actions and outline how this work leads towards the next phase of our work - a toolkit of actions to promote retention and recruitment of women in computing undergraduate courses.
△ Less
Submitted 13 June, 2022;
originally announced June 2022.
-
Increasing Gender Balance Across Academic Staffing in Computer Science -- case study
Authors:
Susan Mckeever,
Deirdre Lillis
Abstract:
As at 2019, Technological University Dublin* Computer Science is the top university in Ireland in terms of gender balance of female academic staff in computer science schools. In an academic team of approximately 55 full-time equivalents, 36% of our academic staff are female, 50% of our senior academic leadership team (2 of 4) are female and 75% of our School Executive are female (3 of 4), includi…
▽ More
As at 2019, Technological University Dublin* Computer Science is the top university in Ireland in terms of gender balance of female academic staff in computer science schools. In an academic team of approximately 55 full-time equivalents, 36% of our academic staff are female, 50% of our senior academic leadership team (2 of 4) are female and 75% of our School Executive are female (3 of 4), including a female Head of School. This is as a result of our seven year SUCCESS programme which had a four strand approach: Source, Career, Environment and Support. The Source strand explicitly encouraged females to apply for each recruitment drive; Career focused on female career and skills development initiatives; Environment created a female-friendly culture and reputation, both within the School, across our organisation and across the third level sector in Ireland and Support addressed practical supports for the specific difficulties experienced by female staff. As a result we have had 0% turnover in female staff in the past five years (in contrast to 10% male staff turnover). We will continue to work across these four strands to preserve our pipeline of female staff and ensure their success over the coming years in an academic and ICT sector that remains challenging for females.
△ Less
Submitted 12 October, 2021;
originally announced October 2021.
-
Addressing the Recruitment and Retention of Female Students in Computer Science at Third Level
Authors:
Susan McKeever,
Deirdre Lillis
Abstract:
In the School of Computing at the Dublin Institute of Technology (DIT), Ireland, we undertook our Computer Science for All (CS4All) initiative, a five year strategy to implement structural reforms at Faculty level, to address recruitment and retention issues of female undergraduate computer science (CS) students. Since 2012, under CS4All we implemented a variety of reforms to improve student reten…
▽ More
In the School of Computing at the Dublin Institute of Technology (DIT), Ireland, we undertook our Computer Science for All (CS4All) initiative, a five year strategy to implement structural reforms at Faculty level, to address recruitment and retention issues of female undergraduate computer science (CS) students. Since 2012, under CS4All we implemented a variety of reforms to improve student retention, set up a new CS program to attract more female students, and delivered changes to promote a sense of community amongst our female students. We have made significant improvements. For example, we have achieved a dramatic improvement in retention rising from 45% to 89% in first year progression rates. Our new hybrid CS International program has more than double the percentage of females first year enrolments in comparison to our other undergraduate programs. As at 2018, we continue to roll out the remaining parts of CS4All within our School.
△ Less
Submitted 12 October, 2021;
originally announced October 2021.
-
Wider Vision: Enriching Convolutional Neural Networks via Alignment to External Knowledge Bases
Authors:
Xuehao Liu,
Sarah Jane Delany,
Susan McKeever
Abstract:
Deep learning models suffer from opaqueness. For Convolutional Neural Networks (CNNs), current research strategies for explaining models focus on the target classes within the associated training dataset. As a result, the understanding of hidden feature map activations is limited by the discriminative knowledge gleaned during training. The aim of our work is to explain and expand CNNs models via t…
▽ More
Deep learning models suffer from opaqueness. For Convolutional Neural Networks (CNNs), current research strategies for explaining models focus on the target classes within the associated training dataset. As a result, the understanding of hidden feature map activations is limited by the discriminative knowledge gleaned during training. The aim of our work is to explain and expand CNNs models via the mirroring or alignment of CNN to an external knowledge base. This will allow us to give a semantic context or label for each visual feature. We can match CNN feature activations to nodes in our external knowledge base. This supports knowledge-based interpretation of the features associated with model decisions. To demonstrate our approach, we build two separate graphs. We use an entity alignment method to align the feature nodes in a CNN with the nodes in a ConceptNet based knowledge graph. We then measure the proximity of CNN graph nodes to semantically meaningful knowledge base nodes. Our results show that in the aligned embedding space, nodes from the knowledge graph are close to the CNN feature nodes that have similar meanings, indicating that nodes from an external knowledge base can act as explanatory semantic references for features in the model. We analyse a variety of graph building methods in order to improve the results from our embedding space. We further demonstrate that by using hierarchical relationships from our external knowledge base, we can locate new unseen classes outside the CNN training set in our embeddings space, based on visual feature activations. This suggests that we can adapt our approach to identify unseen classes based on CNN feature activations. Our demonstrated approach of aligning a CNN with an external knowledge base paves the way to reason about and beyond the trained model, with future adaptations to explainable models and zero-shot learning.
△ Less
Submitted 22 February, 2021;
originally announced February 2021.
-
Feature Fusion of Raman Chemical Imaging and Digital Histopathology using Machine Learning for Prostate Cancer Detection
Authors:
Trevor Doherty,
Susan McKeever,
Nebras Al-Attar,
Tiarnan Murphy,
Claudia Aura,
Arman Rahman,
Amanda O'Neill,
Stephen P Finn,
Elaine Kay,
William M. Gallagher,
R. William G. Watson,
Aoife Gowen,
Patrick Jackman
Abstract:
The diagnosis of prostate cancer is challenging due to the heterogeneity of its presentations, leading to the over diagnosis and treatment of non-clinically important disease. Accurate diagnosis can directly benefit a patient's quality of life and prognosis. Towards addressing this issue, we present a learning model for the automatic identification of prostate cancer. While many prostate cancer st…
▽ More
The diagnosis of prostate cancer is challenging due to the heterogeneity of its presentations, leading to the over diagnosis and treatment of non-clinically important disease. Accurate diagnosis can directly benefit a patient's quality of life and prognosis. Towards addressing this issue, we present a learning model for the automatic identification of prostate cancer. While many prostate cancer studies have adopted Raman spectroscopy approaches, none have utilised the combination of Raman Chemical Imaging (RCI) and other imaging modalities. This study uses multimodal images formed from stained Digital Histopathology (DP) and unstained RCI. The approach was developed and tested on a set of 178 clinical samples from 32 patients, containing a range of non-cancerous, Gleason grade 3 (G3) and grade 4 (G4) tissue microarray samples. For each histological sample, there is a pathologist labelled DP - RCI image pair. The hypothesis tested was whether multimodal image models can outperform single modality baseline models in terms of diagnostic accuracy. Binary non-cancer/cancer models and the more challenging G3/G4 differentiation were investigated. Regarding G3/G4 classification, the multimodal approach achieved a sensitivity of 73.8% and specificity of 88.1% while the baseline DP model showed a sensitivity and specificity of 54.1% and 84.7% respectively. The multimodal approach demonstrated a statistically significant 12.7% AUC advantage over the baseline with a value of 85.8% compared to 73.1%, also outperforming models based solely on RCI and median Raman spectra. Feature fusion of DP and RCI does not improve the more trivial task of tumour identification but does deliver an observed advantage in G3/G4 discrimination. Building on these promising findings, future work could include the acquisition of larger datasets for enhanced model generalization.
△ Less
Submitted 18 January, 2021;
originally announced January 2021.
-
Explaining Deep Learning Models for Structured Data using Layer-Wise Relevance Propagation
Authors:
hsan Ullah,
Andre Rios,
Vaibhav Gala,
Susan Mckeever
Abstract:
Trust and credibility in machine learning models is bolstered by the ability of a model to explain itsdecisions. While explainability of deep learning models is a well-known challenge, a further chal-lenge is clarity of the explanation itself, which must be interpreted by downstream users. Layer-wiseRelevance Propagation (LRP), an established explainability technique developed for deep models inco…
▽ More
Trust and credibility in machine learning models is bolstered by the ability of a model to explain itsdecisions. While explainability of deep learning models is a well-known challenge, a further chal-lenge is clarity of the explanation itself, which must be interpreted by downstream users. Layer-wiseRelevance Propagation (LRP), an established explainability technique developed for deep models incomputer vision, provides intuitive human-readable heat maps of input images. We present the novelapplication of LRP for the first time with structured datasets using a deep neural network (1D-CNN),for Credit Card Fraud detection and Telecom Customer Churn prediction datasets. We show how LRPis more effective than traditional explainability concepts of Local Interpretable Model-agnostic Ex-planations (LIME) and Shapley Additive Explanations (SHAP) for explainability. This effectivenessis both local to a sample level and holistic over the whole testing set. We also discuss the significantcomputational time advantage of LRP (1-2s) over LIME (22s) and SHAP (108s), and thus its poten-tial for real time application scenarios. In addition, our validation of LRP has highlighted features forenhancing model performance, thus opening up a new area of research of using XAI as an approachfor feature subset selection
△ Less
Submitted 26 November, 2020;
originally announced November 2020.
-
"Why did you do that?": Explaining black box models with Inductive Synthesis
Authors:
Görkem Paçacı,
David Johnson,
Steve McKeever,
Andreas Hamfelt
Abstract:
By their nature, the composition of black box models is opaque. This makes the ability to generate explanations for the response to stimuli challenging. The importance of explaining black box models has become increasingly important given the prevalence of AI and ML systems and the need to build legal and regulatory frameworks around them. Such explanations can also increase trust in these uncerta…
▽ More
By their nature, the composition of black box models is opaque. This makes the ability to generate explanations for the response to stimuli challenging. The importance of explaining black box models has become increasingly important given the prevalence of AI and ML systems and the need to build legal and regulatory frameworks around them. Such explanations can also increase trust in these uncertain systems. In our paper we present RICE, a method for generating explanations of the behaviour of black box models by (1) probing a model to extract model output examples using sensitivity analysis; (2) applying CNPInduce, a method for inductive logic program synthesis, to generate logic programs based on critical input-output pairs; and (3) interpreting the target program as a human-readable explanation. We demonstrate the application of our method by generating explanations of an artificial neural network trained to follow simple traffic rules in a hypothetical self-driving car simulation. We conclude with a discussion on the scalability and usability of our approach and its potential applications to explanation-critical scenarios.
△ Less
Submitted 17 April, 2019;
originally announced April 2019.
