-
LLaRA: Supercharging Robot Learning Data for Vision-Language Policy
Authors:
Xiang Li,
Cristina Mata,
Jongwoo Park,
Kumara Kahatapitiya,
Yoo Sung Jang,
**ghuan Shang,
Kanchana Ranasinghe,
Ryan Burgert,
Mu Cai,
Yong Jae Lee,
Michael S. Ryoo
Abstract:
Large Language Models (LLMs) equipped with extensive world knowledge and strong reasoning skills can tackle diverse tasks across domains, often by posing them as conversation-style instruction-response pairs. In this paper, we propose LLaRA: Large Language and Robotics Assistant, a framework which formulates robot action policy as conversations, and provides improved responses when trained with au…
▽ More
Large Language Models (LLMs) equipped with extensive world knowledge and strong reasoning skills can tackle diverse tasks across domains, often by posing them as conversation-style instruction-response pairs. In this paper, we propose LLaRA: Large Language and Robotics Assistant, a framework which formulates robot action policy as conversations, and provides improved responses when trained with auxiliary data that complements policy learning. LLMs with visual inputs, i.e., Vision Language Models (VLMs), have the capacity to process state information as visual-textual prompts and generate optimal policy decisions in text. To train such action policy VLMs, we first introduce an automated pipeline to generate diverse high-quality robotics instruction data from existing behavior cloning data. A VLM finetuned with the resulting collection of datasets based on a conversation-style formulation tailored for robotics tasks, can generate meaningful robot action policy decisions. Our experiments across multiple simulated and real-world environments demonstrate the state-of-the-art performance of the proposed LLaRA framework. The code, datasets, and pretrained models are available at https://github.com/LostXine/LLaRA.
△ Less
Submitted 28 June, 2024;
originally announced June 2024.
-
FAU-Net: An Attention U-Net Extension with Feature Pyramid Attention for Prostate Cancer Segmentation
Authors:
Pablo Cesar Quihui-Rubio,
Daniel Flores-Araiza,
Miguel Gonzalez-Mendoza,
Christian Mata,
Gilberto Ochoa-Ruiz
Abstract:
This contribution presents a deep learning method for the segmentation of prostate zones in MRI images based on U-Net using additive and feature pyramid attention modules, which can improve the workflow of prostate cancer detection and diagnosis. The proposed model is compared to seven different U-Net-based architectures. The automatic segmentation performance of each model of the central zone (CZ…
▽ More
This contribution presents a deep learning method for the segmentation of prostate zones in MRI images based on U-Net using additive and feature pyramid attention modules, which can improve the workflow of prostate cancer detection and diagnosis. The proposed model is compared to seven different U-Net-based architectures. The automatic segmentation performance of each model of the central zone (CZ), peripheral zone (PZ), transition zone (TZ) and Tumor were evaluated using Dice Score (DSC), and the Intersection over Union (IoU) metrics. The proposed alternative achieved a mean DSC of 84.15% and IoU of 76.9% in the test set, outperforming most of the studied models in this work except from R2U-Net and attention R2U-Net architectures.
△ Less
Submitted 3 September, 2023;
originally announced September 2023.
-
Assessing the performance of deep learning-based models for prostate cancer segmentation using uncertainty scores
Authors:
Pablo Cesar Quihui-Rubio,
Daniel Flores-Araiza,
Gilberto Ochoa-Ruiz,
Miguel Gonzalez-Mendoza,
Christian Mata
Abstract:
This study focuses on comparing deep learning methods for the segmentation and quantification of uncertainty in prostate segmentation from MRI images. The aim is to improve the workflow of prostate cancer detection and diagnosis. Seven different U-Net-based architectures, augmented with Monte-Carlo dropout, are evaluated for automatic segmentation of the central zone, peripheral zone, transition z…
▽ More
This study focuses on comparing deep learning methods for the segmentation and quantification of uncertainty in prostate segmentation from MRI images. The aim is to improve the workflow of prostate cancer detection and diagnosis. Seven different U-Net-based architectures, augmented with Monte-Carlo dropout, are evaluated for automatic segmentation of the central zone, peripheral zone, transition zone, and tumor, with uncertainty estimation. The top-performing model in this study is the Attention R2U-Net, achieving a mean Intersection over Union (IoU) of 76.3% and Dice Similarity Coefficient (DSC) of 85% for segmenting all zones. Additionally, Attention R2U-Net exhibits the lowest uncertainty values, particularly in the boundaries of the transition zone and tumor, when compared to the other models.
△ Less
Submitted 8 August, 2023;
originally announced August 2023.
-
Comparison of automatic prostate zones segmentation models in MRI images using U-net-like architectures
Authors:
Pablo Cesar Quihui-Rubio,
Gilberto Ochoa-Ruiz,
Miguel Gonzalez-Mendoza,
Gerardo Rodriguez-Hernandez,
Christian Mata
Abstract:
Prostate cancer is the second-most frequently diagnosed cancer and the sixth leading cause of cancer death in males worldwide. The main problem that specialists face during the diagnosis of prostate cancer is the localization of Regions of Interest (ROI) containing a tumor tissue. Currently, the segmentation of this ROI in most cases is carried out manually by expert doctors, but the procedure is…
▽ More
Prostate cancer is the second-most frequently diagnosed cancer and the sixth leading cause of cancer death in males worldwide. The main problem that specialists face during the diagnosis of prostate cancer is the localization of Regions of Interest (ROI) containing a tumor tissue. Currently, the segmentation of this ROI in most cases is carried out manually by expert doctors, but the procedure is plagued with low detection rates (of about 27-44%) or overdiagnosis in some patients. Therefore, several research works have tackled the challenge of automatically segmenting and extracting features of the ROI from magnetic resonance images, as this process can greatly facilitate many diagnostic and therapeutic applications. However, the lack of clear prostate boundaries, the heterogeneity inherent to the prostate tissue, and the variety of prostate shapes makes this process very difficult to automate.In this work, six deep learning models were trained and analyzed with a dataset of MRI images obtained from the Centre Hospitalaire de Dijon and Universitat Politecnica de Catalunya. We carried out a comparison of multiple deep learning models (i.e. U-Net, Attention U-Net, Dense-UNet, Attention Dense-UNet, R2U-Net, and Attention R2U-Net) using categorical cross-entropy loss function. The analysis was performed using three metrics commonly used for image segmentation: Dice score, Jaccard index, and mean squared error. The model that give us the best result segmenting all the zones was R2U-Net, which achieved 0.869, 0.782, and 0.00013 for Dice, Jaccard and mean squared error, respectively.
△ Less
Submitted 19 July, 2022;
originally announced July 2022.
-
Impact of loss function in Deep Learning methods for accurate retinal vessel segmentation
Authors:
Daniela Herrera,
Gilberto Ochoa-Ruiz,
Miguel Gonzalez-Mendoza,
Christian Mata
Abstract:
The retinal vessel network studied through fundus images contributes to the diagnosis of multiple diseases not only found in the eye. The segmentation of this system may help the specialized task of analyzing these images by assisting in the quantification of morphological characteristics. Due to its relevance, several Deep Learning-based architectures have been tested for tackling this problem au…
▽ More
The retinal vessel network studied through fundus images contributes to the diagnosis of multiple diseases not only found in the eye. The segmentation of this system may help the specialized task of analyzing these images by assisting in the quantification of morphological characteristics. Due to its relevance, several Deep Learning-based architectures have been tested for tackling this problem automatically. However, the impact of loss function selection on the segmentation of the intricate retinal blood vessel system hasn't been systematically evaluated. In this work, we present the comparison of the loss functions Binary Cross Entropy, Dice, Tversky, and Combo loss using the deep learning architectures (i.e. U-Net, Attention U-Net, and Nested UNet) with the DRIVE dataset. Their performance is assessed using four metrics: the AUC, the mean squared error, the dice score, and the Hausdorff distance. The models were trained with the same number of parameters and epochs. Using dice score and AUC, the best combination was SA-UNet with Combo loss, which had an average of 0.9442 and 0.809 respectively. The best average of Hausdorff distance and mean square error were obtained using the Nested U-Net with the Dice loss function, which had an average of 6.32 and 0.0241 respectively. The results showed that there is a significant difference in the selection of loss function
△ Less
Submitted 1 June, 2022;
originally announced June 2022.
-
StandardSim: A Synthetic Dataset For Retail Environments
Authors:
Cristina Mata,
Nick Locascio,
Mohammed Azeem Sheikh,
Kenny Kihara,
Dan Fischetti
Abstract:
Autonomous checkout systems rely on visual and sensory inputs to carry out fine-grained scene understanding in retail environments. Retail environments present unique challenges compared to typical indoor scenes owing to the vast number of densely packed, unique yet similar objects. The problem becomes even more difficult when only RGB input is available, especially for data-hungry tasks such as i…
▽ More
Autonomous checkout systems rely on visual and sensory inputs to carry out fine-grained scene understanding in retail environments. Retail environments present unique challenges compared to typical indoor scenes owing to the vast number of densely packed, unique yet similar objects. The problem becomes even more difficult when only RGB input is available, especially for data-hungry tasks such as instance segmentation. To address the lack of datasets for retail, we present StandardSim, a large-scale photorealistic synthetic dataset featuring annotations for semantic segmentation, instance segmentation, depth estimation, and object detection. Our dataset provides multiple views per scene, enabling multi-view representation learning. Further, we introduce a novel task central to autonomous checkout called change detection, requiring pixel-level classification of takes, puts and shifts in objects over time. We benchmark widely-used models for segmentation and depth estimation on our dataset, show that our test set constitutes a difficult benchmark compared to current smaller-scale datasets and that our training set provides models with crucial information for autonomous checkout tasks.
△ Less
Submitted 4 February, 2022;
originally announced February 2022.
-
Experimental Large-Scale Jet Flames' Geometrical Features Extraction for Risk Management Using Infrared Images and Deep Learning Segmentation Methods
Authors:
Carmina Pérez-Guerrero,
Adriana Palacios,
Gilberto Ochoa-Ruiz,
Christian Mata,
Joaquim Casal,
Miguel Gonzalez-Mendoza,
Luis Eduardo Falcón-Morales
Abstract:
Jet fires are relatively small and have the least severe effects among the diverse fire accidents that can occur in industrial plants; however, they are usually involved in a process known as the domino effect, that leads to more severe events, such as explosions or the initiation of another fire, making the analysis of such fires an important part of risk analysis. This research work explores the…
▽ More
Jet fires are relatively small and have the least severe effects among the diverse fire accidents that can occur in industrial plants; however, they are usually involved in a process known as the domino effect, that leads to more severe events, such as explosions or the initiation of another fire, making the analysis of such fires an important part of risk analysis. This research work explores the application of deep learning models in an alternative approach that uses the semantic segmentation of jet fires flames to extract main geometrical attributes, relevant for fire risk assessments. A comparison is made between traditional image processing methods and some state-of-the-art deep learning models. It is found that the best approach is a deep learning architecture known as UNet, along with its two improvements, Attention UNet and UNet++. The models are then used to segment a group of vertical jet flames of varying pipe outlet diameters to extract their main geometrical characteristics. Attention UNet obtained the best general performance in the approximation of both height and area of the flames, while also showing a statistically significant difference between it and UNet++. UNet obtained the best overall performance for the approximation of the lift-off distances; however, there is not enough data to prove a statistically significant difference between Attention UNet and UNet++. The only instance where UNet++ outperformed the other models, was while obtaining the lift-off distances of the jet flames with 0.01275 m pipe outlet diameter. In general, the explored models show good agreement between the experimental and predicted values for relatively large turbulent propane jet flames, released in sonic and subsonic regimes; thus, making these radiation zones segmentation models, a suitable approach for different jet flame risk management scenarios.
△ Less
Submitted 19 January, 2022;
originally announced January 2022.
-
Comparing Machine Learning based Segmentation Models on Jet Fire Radiation Zones
Authors:
Carmina Pérez-Guerrero,
Adriana Palacios,
Gilberto Ochoa-Ruiz,
Christian Mata,
Miguel Gonzalez-Mendoza,
Luis Eduardo Falcón-Morales
Abstract:
Risk assessment is relevant in any workplace, however there is a degree of unpredictability when dealing with flammable or hazardous materials so that detection of fire accidents by itself may not be enough. An example of this is the im**ement of jet fires, where the heat fluxes of the flame could reach nearby equipment and dramatically increase the probability of a domino effect with catastroph…
▽ More
Risk assessment is relevant in any workplace, however there is a degree of unpredictability when dealing with flammable or hazardous materials so that detection of fire accidents by itself may not be enough. An example of this is the im**ement of jet fires, where the heat fluxes of the flame could reach nearby equipment and dramatically increase the probability of a domino effect with catastrophic results. Because of this, the characterization of such fire accidents is important from a risk management point of view. One such characterization would be the segmentation of different radiation zones within the flame, so this paper presents an exploratory research regarding several traditional computer vision and Deep Learning segmentation approaches to solve this specific problem. A data set of propane jet fires is used to train and evaluate the different approaches and given the difference in the distribution of the zones and background of the images, different loss functions, that seek to alleviate data imbalance, are also explored. Additionally, different metrics are correlated to a manual ranking performed by experts to make an evaluation that closely resembles the expert's criteria. The Hausdorff Distance and Adjusted Random Index were the metrics with the highest correlation and the best results were obtained from the UNet architecture with a Weighted Cross-Entropy Loss. These results can be used in future research to extract more geometric information from the segmentation masks or could even be implemented on other types of fire accidents.
△ Less
Submitted 1 November, 2021; v1 submitted 7 July, 2021;
originally announced July 2021.
-
Complex Relations in a Deep Structured Prediction Model for Fine Image Segmentation
Authors:
Cristina Mata,
Guy Ben-Yosef,
Boris Katz
Abstract:
Many deep learning architectures for semantic segmentation involve a Fully Convolutional Neural Network (FCN) followed by a Conditional Random Field (CRF) to carry out inference over an image. These models typically involve unary potentials based on local appearance features computed by FCNs, and binary potentials based on the displacement between pixels. We show that while current methods succeed…
▽ More
Many deep learning architectures for semantic segmentation involve a Fully Convolutional Neural Network (FCN) followed by a Conditional Random Field (CRF) to carry out inference over an image. These models typically involve unary potentials based on local appearance features computed by FCNs, and binary potentials based on the displacement between pixels. We show that while current methods succeed in segmenting whole objects, they perform poorly in situations involving a large number of object parts. We therefore suggest incorporating into the inference algorithm additional higher-order potentials inspired by the way humans identify and localize parts. We incorporate two relations that were shown to be useful to human object identification - containment and attachment - into the energy term of the CRF and evaluate their performance on the Pascal VOC Parts dataset. Our experimental results show that the segmentation of fine parts is positively affected by the addition of these two relations, and that the segmentation of fine parts can be further influenced by complex structural features.
△ Less
Submitted 23 May, 2018;
originally announced May 2018.
-
Semi-automated labelling of medical images: benefits of a collaborative work in the evaluation of prostate cancer in MRI
Authors:
Christian Mata,
Alain Lalande,
Paul Walker,
Arnau Oliver,
Joan Martí
Abstract:
Purpose: The goal of this study is to show the advantage of a collaborative work in the annotation and evaluation of prostate cancer tissues from T2-weighted MRI compared to the commonly used double blind evaluation.
Methods: The variability of medical findings focused on the prostate gland (central gland, peripheral and tumoural zones) by two independent experts was firstly evaluated, and secon…
▽ More
Purpose: The goal of this study is to show the advantage of a collaborative work in the annotation and evaluation of prostate cancer tissues from T2-weighted MRI compared to the commonly used double blind evaluation.
Methods: The variability of medical findings focused on the prostate gland (central gland, peripheral and tumoural zones) by two independent experts was firstly evaluated, and secondly compared with a consensus of these two experts. Using a prostate MRI database, experts drew regions of interest (ROIs) corresponding to healthy prostate (peripheral and central zones) and cancer using a semi-automated tool. One of the experts then drew the ROI with knowledge of the other expert's ROI.
Results: The surface area of each ROI as the Hausdorff distance and the Dice coefficient for each contour were evaluated between the different experiments, taking the drawing of the second expert as the reference. The results showed that the significant differences between the two experts became non-significant with a collaborative work.
Conclusions: This study shows that collaborative work with a dedicated tool allows a better consensus between expertise than using a double blind evaluation. Although we show this for prostate cancer evaluation in T2-weighted MRI, the results of this research can be extrapolated to other diseases and kind of medical images.
△ Less
Submitted 29 August, 2017;
originally announced August 2017.
-
An oversampling technique for the multiscale finite volume method to simulate electromagnetic responses in the frequency domain
Authors:
Luz Angelica Caudillo Mata,
Eldad Haber,
Christoph Schwarzbach
Abstract:
In order to reduce the computational cost of the simulation of electromagnetic responses in geophysical settings that involve highly heterogeneous media, we develop a multiscale finite volume method with oversampling for the quasi-static Maxwell's equations in the frequency domain. We assume a coarse mesh nested within a fine mesh that accurately discretizes the problem. For each coarse cell, we i…
▽ More
In order to reduce the computational cost of the simulation of electromagnetic responses in geophysical settings that involve highly heterogeneous media, we develop a multiscale finite volume method with oversampling for the quasi-static Maxwell's equations in the frequency domain. We assume a coarse mesh nested within a fine mesh that accurately discretizes the problem. For each coarse cell, we independently solve a local version of the original Maxwell's system subject to linear boundary conditions on an extended domain, which includes the coarse cell and a neighborhood of fine cells around it. The local Maxwell's system is solved using the fine mesh contained in the extended domain and the mimetic finite volume method. Next, these local solutions (basis functions) together with a weak-continuity condition are used to construct a coarse-mesh version of the global problem. The basis functions can be used to obtain the fine-mesh details from the solution of the coarse-mesh problem. Our approach leads to a significant reduction in the size of the final system of equations and the computational time, while accurately approximating the behavior of the fine-mesh solutions. We demonstrate the performance of our method using a synthetic 3D example of a mineral deposit.
△ Less
Submitted 6 October, 2016;
originally announced October 2016.
