Title: A foundation model for joint segmentation, detection and recognition of biomedical objects across nine modalities

Abstract: Biomedical image analysis is fundamental for biomedical discovery. Holistic image analysis comprises interdependent subtasks such as segmentation, detection and recognition, which are tackled separately by traditional approaches. Here, we propose BiomedParse, a biomedical foundation model that can jointly conduct segmentation, detection and recognition across nine imaging modalities. This joint learning improves the accuracy for individual tasks and enables new applications such as segmenting all relevant objects in an image through a textual description. To train BiomedParse, we created a large dataset comprising over 6 million triples of image, segmentation mask and textual description by leveraging natural language labels or descriptions accompanying existing datasets. We showed that BiomedParse outperformed existing methods on image segmentation across nine imaging modalities, with larger improvement on objects with irregular shapes. We further showed that BiomedParse can simultaneously segment and label all objects in an image. In summary, BiomedParse is an all-in-one tool for biomedical image analysis on all major image modalities, paving the path for efficient and accurate image-based biomedical discovery.

Bio: Theodore Zhao is a Senior Applied Scientist at Microsoft Health and Life Sciences Research, working on multimodal biomedical imaging models as well as biomedical natural language processing. Theodore earned his PhD in Applied Mathematics degree from University of Washington, where his research applied machine learning, stochastic modeling and optimization to applications in finance and healthcare. His research interests focus on machine learning, self-supervised learning, multimodal models, and mathematical modeling.

Paper Link: A foundation model for joint segmentation, detection and recognition of biomedical objects across nine modalities | Nature MethodsLink opens in a new windowLink opens in a new window

How to attend: Either turn up to the event on the day, or if you want to attend online then please contact Adam Shephard (adam.shephard@warwick.ac.uk) for more details.