-
The Penalized Inverse Probability Measure for Conformal Classification
Authors:
Paul Melki,
Lionel Bombrun,
Boubacar Diallo,
Jérôme Dias,
Jean-Pierre da Costa
Abstract:
The deployment of safe and trustworthy machine learning systems, and particularly complex black box neural networks, in real-world applications requires reliable and certified guarantees on their performance. The conformal prediction framework offers such formal guarantees by transforming any point into a set predictor with valid, finite-set, guarantees on the coverage of the true at a chosen leve…
▽ More
The deployment of safe and trustworthy machine learning systems, and particularly complex black box neural networks, in real-world applications requires reliable and certified guarantees on their performance. The conformal prediction framework offers such formal guarantees by transforming any point into a set predictor with valid, finite-set, guarantees on the coverage of the true at a chosen level of confidence. Central to this methodology is the notion of the nonconformity score function that assigns to each example a measure of ''strangeness'' in comparison with the previously seen observations. While the coverage guarantees are maintained regardless of the nonconformity measure, the point predictor and the dataset, previous research has shown that the performance of a conformal model, as measured by its efficiency (the average size of the predicted sets) and its informativeness (the proportion of prediction sets that are singletons), is influenced by the choice of the nonconformity score function. The current work introduces the Penalized Inverse Probability (PIP) nonconformity score, and its regularized version RePIP, that allow the joint optimization of both efficiency and informativeness. Through toy examples and empirical results on the task of crop and weed image classification in agricultural robotics, the current work shows how PIP-based conformal classifiers exhibit precisely the desired behavior in comparison with other nonconformity measures and strike a good balance between informativeness and efficiency.
△ Less
Submitted 13 June, 2024;
originally announced June 2024.
-
Group-Conditional Conformal Prediction via Quantile Regression Calibration for Crop and Weed Classification
Authors:
Paul Melki,
Lionel Bombrun,
Boubacar Diallo,
Jérôme Dias,
Jean-Pierre da Costa
Abstract:
As deep learning predictive models become an integral part of a large spectrum of precision agricultural systems, a barrier to the adoption of such automated solutions is the lack of user trust in these highly complex, opaque and uncertain models. Indeed, deep neural networks are not equipped with any explicit guarantees that can be used to certify the system's performance, especially in highly va…
▽ More
As deep learning predictive models become an integral part of a large spectrum of precision agricultural systems, a barrier to the adoption of such automated solutions is the lack of user trust in these highly complex, opaque and uncertain models. Indeed, deep neural networks are not equipped with any explicit guarantees that can be used to certify the system's performance, especially in highly varying uncontrolled environments such as the ones typically faced in computer vision for agriculture.Fortunately, certain methods developed in other communities can prove to be important for agricultural applications. This article presents the conformal prediction framework that provides valid statistical guarantees on the predictive performance of any black box prediction machine, with almost no assumptions, applied to the problem of deep visual classification of weeds and crops in real-world conditions. The framework is exposed with a focus on its practical aspects and special attention accorded to the Adaptive Prediction Sets (APS) approach that delivers marginal guarantees on the model's coverage. Marginal results are then shown to be insufficient to guarantee performance on all groups of individuals in the population as characterized by their environmental and pedo-climatic auxiliary data gathered during image acquisition.To tackle this shortcoming, group-conditional conformal approaches are presented: the ''classical'' method that consists of iteratively applying the APS procedure on all groups, and a proposed elegant reformulation and implementation of the procedure using quantile regression on group membership indicators. Empirical results showing the validity of the proposed approach are presented and compared to the marginal APS then discussed.
△ Less
Submitted 29 August, 2023;
originally announced August 2023.
-
Metadata Interpretation Driven Development
Authors:
Júlio G. S. F. da Costa,
Reinaldo A. Petta,
Samuel Xavier-de-Souza
Abstract:
Despite decades of engineering and scientific research efforts, separation of concerns in software development remains not fully achieved. The challenge has been to avoid the crosscutting of concerns phenomenon, which has no apparent complete solution. In this paper, we show that business-domain coding plays an even larger role in this challenge. We then introduce a new approach called \emph{Metad…
▽ More
Despite decades of engineering and scientific research efforts, separation of concerns in software development remains not fully achieved. The challenge has been to avoid the crosscutting of concerns phenomenon, which has no apparent complete solution. In this paper, we show that business-domain coding plays an even larger role in this challenge. We then introduce a new approach called \emph{Metadata Interpretation Driven Development} (MIDD), which suggests a way to enhance the current way of realizing separation of concerns by eliminating the need to code functional concerns. We propose to code non-functional concerns as metadata interpreters. This interpretation occurs at run-time and is possible because it assumes the existence of such metadata in artefacts created in previous stages of the process, such as the modelling phase. We show how this can increase the (re)use of the constructs. Furthermore, we show that a single interpreter, due to its semantic disconnection from the domain, can simultaneously serve different business domains with no concerns regarding the need to rewrite or refactor code. Although high-reuse software construction is considered a relatively mature field, changes in the software services scenario demand constant evolution of the actual solutions. The emergence of new software architectures, such as serverless computing, reinforces the need to rethink software construction. This approach is presented as a response to this need.
△ Less
Submitted 8 October, 2021; v1 submitted 2 May, 2021;
originally announced May 2021.
-
Learning low-frequency temporal patterns for quantitative trading
Authors:
Joel da Costa,
Tim Gebbie
Abstract:
We consider the viability of a modularised mechanistic online machine learning framework to learn signals in low-frequency financial time series data. The framework is proved on daily sampled closing time-series data from JSE equity markets. The input patterns are vectors of pre-processed sequences of daily, weekly and monthly or quarterly sampled feature changes. The data processing is split into…
▽ More
We consider the viability of a modularised mechanistic online machine learning framework to learn signals in low-frequency financial time series data. The framework is proved on daily sampled closing time-series data from JSE equity markets. The input patterns are vectors of pre-processed sequences of daily, weekly and monthly or quarterly sampled feature changes. The data processing is split into a batch processed step where features are learnt using a stacked autoencoder via unsupervised learning, and then both batch and online supervised learning are carried out using these learnt features, with the output being a point prediction of measured time-series feature fluctuations. Weight initializations are implemented with restricted Boltzmann machine pre-training, and variance based initializations. Historical simulations are then run using an online feedforward neural network initialised with the weights from the batch training and validation step. The validity of results are considered under a rigorous assessment of backtest overfitting using both combinatorially symmetrical cross validation and probabilistic and deflated Sharpe ratios. Results are used to develop a view on the phenomenology of financial markets and the value of complex historical data-analysis for trading under the unstable adaptive dynamics that characterise financial markets.
△ Less
Submitted 12 August, 2020;
originally announced August 2020.
-
Análise Estática de Código-Fonte
Authors:
Joenio Marques da Costa
Abstract:
This article presents a theoretical summary of the source code static analysis, its definition, uses and applications, how static analysis is performed, their intermediate representation formats, models and most common analysis techniques, ends up presenting a set of free and freely available downloadable static analysis tools, academic software tools developed by scientists during their research…
▽ More
This article presents a theoretical summary of the source code static analysis, its definition, uses and applications, how static analysis is performed, their intermediate representation formats, models and most common analysis techniques, ends up presenting a set of free and freely available downloadable static analysis tools, academic software tools developed by scientists during their research work (The paper is written in Brazillian Portuguese).
△ Less
Submitted 29 June, 2019;
originally announced July 2019.
