-
SemEval-2024 Task 2: Safe Biomedical Natural Language Inference for Clinical Trials
Authors:
Mael Jullien,
Marco Valentino,
André Freitas
Abstract:
Large Language Models (LLMs) are at the forefront of NLP achievements but fall short in dealing with shortcut learning, factual inconsistency, and vulnerability to adversarial inputs.These shortcomings are especially critical in medical contexts, where they can misrepresent actual model capabilities. Addressing this, we present SemEval-2024 Task 2: Safe Biomedical Natural Language Inference for Cl…
▽ More
Large Language Models (LLMs) are at the forefront of NLP achievements but fall short in dealing with shortcut learning, factual inconsistency, and vulnerability to adversarial inputs.These shortcomings are especially critical in medical contexts, where they can misrepresent actual model capabilities. Addressing this, we present SemEval-2024 Task 2: Safe Biomedical Natural Language Inference for ClinicalTrials. Our contributions include the refined NLI4CT-P dataset (i.e., Natural Language Inference for Clinical Trials - Perturbed), designed to challenge LLMs with interventional and causal reasoning tasks, along with a comprehensive evaluation of methods and results for participant submissions. A total of 106 participants registered for the task contributing to over 1200 individual submissions and 25 system overview papers. This initiative aims to advance the robustness and applicability of NLI models in healthcare, ensuring safer and more dependable AI assistance in clinical decision-making. We anticipate that the dataset, models, and outcomes of this task can support future research in the field of biomedical NLI. The dataset, competition leaderboard, and website are publicly available.
△ Less
Submitted 7 April, 2024;
originally announced April 2024.
-
NLI4CT: Multi-Evidence Natural Language Inference for Clinical Trial Reports
Authors:
Maël Jullien,
Marco Valentino,
Hannah Frost,
Paul O'Regan,
Donal Landers,
André Freitas
Abstract:
How can we interpret and retrieve medical evidence to support clinical decisions? Clinical trial reports (CTR) amassed over the years contain indispensable information for the development of personalized medicine. However, it is practically infeasible to manually inspect over 400,000+ clinical trial reports in order to find the best evidence for experimental treatments. Natural Language Inference…
▽ More
How can we interpret and retrieve medical evidence to support clinical decisions? Clinical trial reports (CTR) amassed over the years contain indispensable information for the development of personalized medicine. However, it is practically infeasible to manually inspect over 400,000+ clinical trial reports in order to find the best evidence for experimental treatments. Natural Language Inference (NLI) offers a potential solution to this problem, by allowing the scalable computation of textual entailment. However, existing NLI models perform poorly on biomedical corpora, and previously published datasets fail to capture the full complexity of inference over CTRs. In this work, we present a novel resource to advance research on NLI for reasoning on CTRs. The resource includes two main tasks. Firstly, to determine the inference relation between a natural language statement, and a CTR. Secondly, to retrieve supporting facts to justify the predicted relation. We provide NLI4CT, a corpus of 2400 statements and CTRs, annotated for these tasks. Baselines on this corpus expose the limitations of existing NLI models, with 6 state-of-the-art NLI models achieving a maximum F1 score of 0.627. To the best of our knowledge, we are the first to design a task that covers the interpretation of full CTRs. To encourage further work on this challenging dataset, we make the corpus, competition leaderboard, website and code to replicate the baseline experiments available at: https://github.com/ai-systems/nli4ct
△ Less
Submitted 28 October, 2023; v1 submitted 5 May, 2023;
originally announced May 2023.
-
SemEval-2023 Task 7: Multi-Evidence Natural Language Inference for Clinical Trial Data
Authors:
Maël Jullien,
Marco Valentino,
Hannah Frost,
Paul O'Regan,
Donal Landers,
André Freitas
Abstract:
This paper describes the results of SemEval 2023 task 7 -- Multi-Evidence Natural Language Inference for Clinical Trial Data (NLI4CT) -- consisting of 2 tasks, a Natural Language Inference (NLI) task, and an evidence selection task on clinical trial data. The proposed challenges require multi-hop biomedical and numerical reasoning, which are of significant importance to the development of systems…
▽ More
This paper describes the results of SemEval 2023 task 7 -- Multi-Evidence Natural Language Inference for Clinical Trial Data (NLI4CT) -- consisting of 2 tasks, a Natural Language Inference (NLI) task, and an evidence selection task on clinical trial data. The proposed challenges require multi-hop biomedical and numerical reasoning, which are of significant importance to the development of systems capable of large-scale interpretation and retrieval of medical evidence, to provide personalized evidence-based care.
Task 1, the entailment task, received 643 submissions from 40 participants, and Task 2, the evidence selection task, received 364 submissions from 23 participants. The tasks are challenging, with the majority of submitted systems failing to significantly outperform the majority class baseline on the entailment task, and we observe significantly better performance on the evidence selection task than on the entailment task. Increasing the number of model parameters leads to a direct increase in performance, far more significant than the effect of biomedical pre-training. Future works could explore the limitations of large models for generalization and numerical inference, and investigate methods to augment clinical datasets to allow for more rigorous testing and to facilitate fine-tuning.
We envisage that the dataset, models, and results of this task will be useful to the biomedical NLI and evidence retrieval communities. The dataset, competition leaderboard, and website are publicly available.
△ Less
Submitted 11 May, 2023; v1 submitted 4 May, 2023;
originally announced May 2023.
-
Do Transformers Encode a Foundational Ontology? Probing Abstract Classes in Natural Language
Authors:
Mael Jullien,
Marco Valentino,
Andre Freitas
Abstract:
With the methodological support of probing (or diagnostic classification), recent studies have demonstrated that Transformers encode syntactic and semantic information to some extent. Following this line of research, this paper aims at taking semantic probing to an abstraction extreme with the goal of answering the following research question: can contemporary Transformer-based models reflect an u…
▽ More
With the methodological support of probing (or diagnostic classification), recent studies have demonstrated that Transformers encode syntactic and semantic information to some extent. Following this line of research, this paper aims at taking semantic probing to an abstraction extreme with the goal of answering the following research question: can contemporary Transformer-based models reflect an underlying Foundational Ontology? To this end, we present a systematic Foundational Ontology (FO) probing methodology to investigate whether Transformers-based models encode abstract semantic information. Following different pre-training and fine-tuning regimes, we present an extensive evaluation of a diverse set of large-scale language models over three distinct and complementary FO tagging experiments. Specifically, we present and discuss the following conclusions: (1) The probing results indicate that Transformer-based models incidentally encode information related to Foundational Ontologies during the pre-training pro-cess; (2) Robust FO taggers (accuracy of 90 percent)can be efficiently built leveraging on this knowledge.
△ Less
Submitted 25 January, 2022;
originally announced January 2022.
-
Spanwise dispersion optimizes the efficiency of dense microfluidic trap arrays
Authors:
Nicolas Ruyssen,
Gabriel Fina,
Rachele Allena,
Marie-Caroline Jullien,
Jacques Fattaccioli
Abstract:
Microfluidic Trap Arrays (MTAs) have proved efficient tools for several applications requiring working at the single cell level like cancer understanding and treatment or immune synapse research. Unfortunately, it generally appears that many traps stay empty, even after a long time of injection which can drastically reduce the number of samples available for post-treatment. It has been shown that…
▽ More
Microfluidic Trap Arrays (MTAs) have proved efficient tools for several applications requiring working at the single cell level like cancer understanding and treatment or immune synapse research. Unfortunately, it generally appears that many traps stay empty, even after a long time of injection which can drastically reduce the number of samples available for post-treatment. It has been shown that these unfilled traps were due to the symmetrical nature of the flow around the traps, with a break in symmetry improving capture efficiency. In this work, we use a numerical approach to show that it is possible to generate optimal geometries that significantly improve capture efficiency. This efficiency is associated with an increase in the lateral dispersion of the objects; we show that adding disorder to the layout of the traps is the most optimal solution and may stay very efficient independently of the trap array size. These numerical results are corroborated by experiments, validating our approach.
△ Less
Submitted 8 January, 2024; v1 submitted 14 October, 2021;
originally announced October 2021.
-
Enhancing the capture efficiency and homogeneity of single-layer flow-through trap** microfluidic devices using oblique hydrodynamic streams
Authors:
Olivier Mesdjian,
Nicolas Ruyssen,
Marie-Caroline Jullien,
Rachele Allena,
Jacques Fattaccioli
Abstract:
With the aim to parallelize and monitor biological or biochemical phenomena, trap** and immobilization of objects such as particles, droplets or cells in microfluidic devices has been an intense area of research and engineering so far. Either being passive or active, these microfluidic devices are usually composed of arrays of elementary traps with various levels of sophistication. For a given a…
▽ More
With the aim to parallelize and monitor biological or biochemical phenomena, trap** and immobilization of objects such as particles, droplets or cells in microfluidic devices has been an intense area of research and engineering so far. Either being passive or active, these microfluidic devices are usually composed of arrays of elementary traps with various levels of sophistication. For a given array, it is important to have an efficient and fast immobilization of the highest number of objects, while optimizing the spatial homogeneity of the trap** over the whole chip. For passive devices, this has been achieved with two-layers structures, making the fabrication process more complex. In this work, we designed small microfluidic traps by single-layer direct laser writing into a photoresist, and we show that even in this simplest case, the orientation of the main flow of particles with respect to the traps have a drastic effect on the trap** efficiency and homogeneity. To better understand this phenomenon, we have considered two different flow geometries: parallel and oblique with respect to the traps array, and compared quantitatively the immobilization of particles with various sizes and densities. Using image analysis, we show that diagonal flows gives a spatial distribution of the trap loading that is more homogeneous over the whole chip as compared to the straight ones, and by performing FEM and trap** simulation, we propose a qualitative explanation of this phenomenon.
△ Less
Submitted 16 September, 2021; v1 submitted 28 June, 2021;
originally announced June 2021.
-
Wettability patterning in microfluidic devices using thermally-enhanced hydrophobic recovery of PDMS
Authors:
Marc Pascual,
Margaux Kerdraon,
Quentin Rezard,
Marie-Caroline Jullien,
Lorène Champougny
Abstract:
Spatial control of wettability is key to many applications of microfluidic devices, ranging from double emulsion generation to localized cell adhesion. A number of techniques, often based on masking, have been developed to produce spatially-resolved wettability patterns at the surface of poly(dimethylsiloxane) (PDMS) elastomers. A major impediment they face is the natural hydrophobic recovery of P…
▽ More
Spatial control of wettability is key to many applications of microfluidic devices, ranging from double emulsion generation to localized cell adhesion. A number of techniques, often based on masking, have been developed to produce spatially-resolved wettability patterns at the surface of poly(dimethylsiloxane) (PDMS) elastomers. A major impediment they face is the natural hydrophobic recovery of PDMS: hydrophilized PDMS surfaces tend to return to hydrophobicity with time, mainly because of diffusion of low molecular weight silicone species to the surface. Instead of trying to avoid this phenomenon, we propose in this work to take advantage of hydrophobic recovery to modulate spatially the surface wettability of PDMS. Because temperature speeds up the rate of hydrophobic recovery, we show that space-resolved hydrophobic patterns can be produced by locally heating a plasma-hydrophilized PDMS surface with microresistors. Importantly, local wettability is quantified in microchannels using a fluorescent probe. This "thermo-patterning" technique provides a simple route to in situ wettability patterning in closed PDMS chips, without requiring further surface chemistry.
△ Less
Submitted 29 October, 2019;
originally announced October 2019.
-
Ultrasound transmission through monodisperse 2D microfoams
Authors:
Lorène Champougny,
Juliette Pierre,
Antoine Devulder,
Valentin Leroy,
Marie-Caroline Jullien
Abstract:
While the acoustic properties of solid foams have been abundantly characterized, sound propagation in liquid foams remains poorly understood. Recent studies have investigated the transmission of ultrasound through three-dimensional polydisperse liquid foams (Pierre et al., 2013, 2014, 2017). However, further progress requires to characterize the acoustic response of better controlled foam structur…
▽ More
While the acoustic properties of solid foams have been abundantly characterized, sound propagation in liquid foams remains poorly understood. Recent studies have investigated the transmission of ultrasound through three-dimensional polydisperse liquid foams (Pierre et al., 2013, 2014, 2017). However, further progress requires to characterize the acoustic response of better controlled foam structures. In this work, we study experimentally the transmission of ultrasounds through a single layer of monodisperse bubbles generated by microfluidics techniques. In such a material, we show that the sound velocity is only sensitive to the gas phase. Nevertheless, the structure of the liquid network has to be taken into account through a transfer parameter analogous to the one in a layer of porous material. Finally, we observe that the attenuation cannot be explained by thermal dissipation alone, but is compatible with viscous dissipation in the gas pores of the monolayer.
△ Less
Submitted 18 January, 2019;
originally announced January 2019.
-
Global sensitivity analysis for models with spatially dependent outputs
Authors:
Amandine Marrel,
Bertrand Iooss,
Michel Jullien,
Beatrice Laurent,
Elena Volkova
Abstract:
The global sensitivity analysis of a complex numerical model often calls for the estimation of variance-based importance measures, named Sobol' indices. Metamodel-based techniques have been developed in order to replace the cpu time-expensive computer code with an inexpensive mathematical function, which predicts the computer code output. The common metamodel-based sensitivity analysis methods are…
▽ More
The global sensitivity analysis of a complex numerical model often calls for the estimation of variance-based importance measures, named Sobol' indices. Metamodel-based techniques have been developed in order to replace the cpu time-expensive computer code with an inexpensive mathematical function, which predicts the computer code output. The common metamodel-based sensitivity analysis methods are well-suited for computer codes with scalar outputs. However, in the environmental domain, as in many areas of application, the numerical model outputs are often spatial maps, which may also vary with time. In this paper, we introduce an innovative method to obtain a spatial map of Sobol' indices with a minimal number of numerical model computations. It is based upon the functional decomposition of the spatial output onto a wavelet basis and the metamodeling of the wavelet coefficients by the Gaussian process. An analytical example is presented to clarify the various steps of our methodology. This technique is then applied to a real hydrogeological case: for each model input variable, a spatial map of Sobol' indices is thus obtained.
△ Less
Submitted 23 September, 2010; v1 submitted 6 November, 2009;
originally announced November 2009.
-
An Active Chaotic Micromixer Integrating Thermal Actuation Associating PDMS and Silicon Microtechnology
Authors:
O. Français,
M. -C. Jullien,
L. Rousseau,
P. Poulichet,
S. Desportes,
A. Chouai,
J. -P. Lefevre,
J. Delaire
Abstract:
Due to scaling laws, in microfluidic, flows are laminar. Consequently, mixing between two liquids is mainly obtained by natural diffusion which may take a long time or equivalently requires centimetre length channels. To reduce time and length for mixing, it is possible to generate chaotic-like flows either by modifying the channel geometry or by creating an external perturbation of the flow. In…
▽ More
Due to scaling laws, in microfluidic, flows are laminar. Consequently, mixing between two liquids is mainly obtained by natural diffusion which may take a long time or equivalently requires centimetre length channels. To reduce time and length for mixing, it is possible to generate chaotic-like flows either by modifying the channel geometry or by creating an external perturbation of the flow. In this paper, an active micromixer is presented consisting on thermal actuation with heating resistors. In order to disturb the liquid flow, an oscillating transverse flow is generated by heating the liquid. Depending on the value of boiling point, either bubble expansion or volumetric dilation controlled the transverse flow amplitude. A chaotic like mixing is then induced under particular conditions depending on volume expansion, liquid velocity, frequency of actuation... This solution presents the advantage to achieve mixing in a very short time (1s) and along a short channel distance (channel width). It can also be integrated in a more complex device due to actuator integration with microfluidics.
△ Less
Submitted 21 November, 2007;
originally announced November 2007.
-
Vorticity statistics in the two-dimensional enstrophy cascade
Authors:
J Paret,
M. C. Jullien,
P Tabeling
Abstract:
We report the first extensive experimental observation of the two-dimensional enstrophy cascade, along with the determination of the high order vorticity statistics. The energy spectra we obtain are remarkably close to the Kraichnan Batchelor expectation. The distributions of the vorticity increments, in the inertial range, deviate only little from gaussianity and the corresponding structure fun…
▽ More
We report the first extensive experimental observation of the two-dimensional enstrophy cascade, along with the determination of the high order vorticity statistics. The energy spectra we obtain are remarkably close to the Kraichnan Batchelor expectation. The distributions of the vorticity increments, in the inertial range, deviate only little from gaussianity and the corresponding structure functions exponents are indistinguishable from zero. It is thus shown that there is no sizeable small scale intermittency in the enstrophy cascade, in agreement with recent theoretical analyses.
△ Less
Submitted 21 April, 1999;
originally announced April 1999.
