Fast kernel methods for Data Quality Monitoring as a goodness-of-fit test
Authors:
Gaia Grosso,
Nicolò Lai,
Marco Letizia,
Jacopo Pazzini,
Marco Rando,
Lorenzo Rosasco,
Andrea Wulzer,
Marco Zanetti
Abstract:
We here propose a machine learning approach for monitoring particle detectors in real-time. The goal is to assess the compatibility of incoming experimental data with a reference dataset, characterising the data behaviour under normal circumstances, via a likelihood-ratio hypothesis test. The model is based on a modern implementation of kernel methods, nonparametric algorithms that can learn any c…
▽ More
We here propose a machine learning approach for monitoring particle detectors in real-time. The goal is to assess the compatibility of incoming experimental data with a reference dataset, characterising the data behaviour under normal circumstances, via a likelihood-ratio hypothesis test. The model is based on a modern implementation of kernel methods, nonparametric algorithms that can learn any continuous function given enough data. The resulting approach is efficient and agnostic to the type of anomaly that may be present in the data. Our study demonstrates the effectiveness of this strategy on multivariate data from drift tube chamber muon detectors.
△ Less
Submitted 9 March, 2023;
originally announced March 2023.
Using Big Data Technologies for HEP Analysis
Authors:
Matteo Cremonesi,
Claudio Bellini,
Bianny Bian,
Luca Canali,
Vasileios Dimakopoulos,
Peter Elmer,
Ian Fisk,
Maria Girone,
Oliver Gutsche,
Siew-Yan Hoh,
Bo Jayatilaka,
Viktor Khristenko,
Andrea Luiselli,
Andrew Melo,
Evangelos Evangelos,
Dominick Olivito,
Jacopo Pazzini,
Jim Pivarski,
Alexey Svyatkovskiy,
Marco Zanetti
Abstract:
The HEP community is approaching an era were the excellent performances of the particle accelerators in delivering collision at high rate will force the experiments to record a large amount of information. The growing size of the datasets could potentially become a limiting factor in the capability to produce scientific results timely and efficiently. Recently, new technologies and new approaches…
▽ More
The HEP community is approaching an era were the excellent performances of the particle accelerators in delivering collision at high rate will force the experiments to record a large amount of information. The growing size of the datasets could potentially become a limiting factor in the capability to produce scientific results timely and efficiently. Recently, new technologies and new approaches have been developed in industry to answer to the necessity to retrieve information as quickly as possible to analyze PB and EB datasets. Providing the scientists with these modern computing tools will lead to rethinking the principles of data analysis in HEP, making the overall scientific process faster and smoother.
In this paper, we are presenting the latest developments and the most recent results on the usage of Apache Spark for HEP analysis. The study aims at evaluating the efficiency of the application of the new tools both quantitatively, by measuring the performances, and qualitatively, focusing on the user experience. The first goal is achieved by develo** a data reduction facility: working together with CERN Openlab and Intel, CMS replicates a real physics search using Spark-based technologies, with the ambition of reducing 1 PB of public data in 5 hours, collected by the CMS experiment, to 1 TB of data in a format suitable for physics analysis.
The second goal is achieved by implementing multiple physics use-cases in Apache Spark using as input preprocessed datasets derived from official CMS data and simulation. By performing different end-analyses up to the publication plots on different hardware, feasibility, usability and portability are compared to the ones of a traditional ROOT-based workflow.
△ Less
Submitted 21 January, 2019;
originally announced January 2019.
