UG4 group project

Title: Developing a Multimodal AI system for Non-contact vital sign monitoring

The demand for seamless, continuous health and wellness tracking is rapidly growing across sectors such as fitness, occupational wellbeing, and smart home environments. However, current solutions rely heavily on wearable devices, which can be intrusive, uncomfortable, or easily forgotten. This project aims to address this by developing a robust, completely non-contact monitoring system. By fusing data from standard optical cameras (RGB) and millimetre-wave (mmWave) radar, we can accurately extract physiological vital signs—such as cardiac and respiratory waveforms—from subjects without requiring any physical sensors on the body.

Project Overview In this project, the group will be responsible for building a prototype of a multimodal sensing system based on RBG and mmWave sensing. Moving beyond basic off-the-shelf algorithms, you will tackle the core challenge of non-contact monitoring: extracting microscopic physiological signals (such as the subtle skin-colour variations of a heartbeat or the sub-millimetre chest movements of breathing) in the presence of real-world noise, varying lighting, and natural human motion.

Key Objectives & Work Packages

1. Sensor Integration & Synchronisation: Set up a synchronised data-capture pipeline interfacing an RGB camera and a mmWave radar module.

2. Signal Processing & Computer Vision: Extract remote photoplethysmography (rPPG) signals from RGB video feeds [1].

   • Process complex radar phase-data to isolate cardiac and respiratory micro-vibrations [2].

3. Cross-Modal Data Fusion: Develop a multimodal AI algorithm to combine data from both sensors. The goal is to create a system that dynamically relies on the most accurate sensor at any given moment (e.g., relying heavily on radar if room lighting drops or the subject turns away from the camera).

4. Data Collection & Validation: Conduct a controlled data collection study on a cohort of healthy subjects. You will evaluate your prototype’s accuracy against ground-truth commercial wearables (e.g., chest strap heart monitors or pulse oximeters).

Skills Required / Developed This project is highly technical and requires a strong, ambitious team. You will utilize and develop skills in:

• Python Programming

• Computer Vision (OpenCV)

• Digital Signal Processing (DSP)

• Machine Learning / AI

• Sensor Hardware Interfacing

Impact You will be building the core architecture for next-generation smart environments. This technology has the potential to revolutionise how we interact with our living and working spaces, providing invisible, frictionless insights into human health, fitness, and well-being. Excellent work may lead to co-authorship on publications.

[1]: https://github.com/ubicomplab/rPPG-Toolbox

[2]: vital sign monitoring in dynamic environment via mmWave radar and camera fusion, 2024