-
Open Vocabulary Semantic Scene Sketch Understanding
Authors:
Ahmed Bourouis,
Judith Ellen Fan,
Yulia Gryaditskaya
Abstract:
We study the underexplored but fundamental vision problem of machine understanding of abstract freehand scene sketches. We introduce a sketch encoder that results in semantically-aware feature space, which we evaluate by testing its performance on a semantic sketch segmentation task. To train our model we rely only on the availability of bitmap sketches with their brief captions and do not require…
▽ More
We study the underexplored but fundamental vision problem of machine understanding of abstract freehand scene sketches. We introduce a sketch encoder that results in semantically-aware feature space, which we evaluate by testing its performance on a semantic sketch segmentation task. To train our model we rely only on the availability of bitmap sketches with their brief captions and do not require any pixel-level annotations. To obtain generalization to a large set of sketches and categories, we build on a vision transformer encoder pretrained with the CLIP model. We freeze the text encoder and perform visual-prompt tuning of the visual encoder branch while introducing a set of critical modifications. Firstly, we augment the classical key-query (k-q) self-attention blocks with value-value (v-v) self-attention blocks. Central to our model is a two-level hierarchical network design that enables efficient semantic disentanglement: The first level ensures holistic scene sketch encoding, and the second level focuses on individual categories. We, then, in the second level of the hierarchy, introduce a cross-attention between textual and visual branches. Our method outperforms zero-shot CLIP pixel accuracy of segmentation results by 37 points, reaching an accuracy of $85.5\%$ on the FS-COCO sketch dataset. Finally, we conduct a user study that allows us to identify further improvements needed over our method to reconcile machine and human understanding of scene sketches.
△ Less
Submitted 30 March, 2024; v1 submitted 18 December, 2023;
originally announced December 2023.
-
A New Architecture of a Ubiquitous Health Monitoring System: A Prototype Of Cloud Mobile Health Monitoring System
Authors:
Abderrahim Bourouis,
Mohamed Feham,
Abdelhamid Bouchachia
Abstract:
Wireless Body Area Sensor Networks (WBASN) is an emerging technology which uses wireless sensors to implement real-time wearable health monitoring of patients to enhance independent living. In this paper we propose a prototype of cloud mobile health monitoring system. The system uses WBASN and Smartphone application that uses cloud computing, location data and a neural network to determine the sta…
▽ More
Wireless Body Area Sensor Networks (WBASN) is an emerging technology which uses wireless sensors to implement real-time wearable health monitoring of patients to enhance independent living. In this paper we propose a prototype of cloud mobile health monitoring system. The system uses WBASN and Smartphone application that uses cloud computing, location data and a neural network to determine the state of patients.
△ Less
Submitted 31 May, 2012;
originally announced May 2012.
-
Ubiquitous Mobile Health Monitoring System for Elderly (UMHMSE)
Authors:
Abderrahim Bourouis,
Mohamed Feham,
Abdelhamid Bouchachia
Abstract:
Recent research in ubiquitous computing uses technologies of Body Area Networks (BANs) to monitor the person's kinematics and physiological parameters. In this paper we propose a real time mobile health system for monitoring elderly patients from indoor or outdoor environments. The system uses a bio- signal sensor worn by the patient and a Smartphone as a central node. The sensor data is collected…
▽ More
Recent research in ubiquitous computing uses technologies of Body Area Networks (BANs) to monitor the person's kinematics and physiological parameters. In this paper we propose a real time mobile health system for monitoring elderly patients from indoor or outdoor environments. The system uses a bio- signal sensor worn by the patient and a Smartphone as a central node. The sensor data is collected and transmitted to the intelligent server through GPRS/UMTS to be analyzed. The prototype (UMHMSE) monitors the elderly mobility, location and vital signs such as Sp02 and Heart Rate. Remote users (family and medical personnel) might have a real time access to the collected information through a web application.
△ Less
Submitted 19 July, 2011;
originally announced July 2011.
