Title: A vision–language foundation model for precision oncology

Abstract: Clinical decision-making is driven by multimodal data, including clinical notes and pathological characteristics. However, the scarcity of well-annotated multimodal datasets in clinical settings has hindered the development of useful models. We developed the Multimodal transformer with Unified maSKed modeling (MUSK), a vision–language foundation model designed to leverage large-scale, unlabelled, unpaired image and text data. MUSK was pretrained on 50 million pathology images from 11,577 patients and one billion pathology-related text tokens using unified masked modelling. It was further pretrained on one million pathology image–text pairs to efficiently align the vision and language features. With minimal or no further training, MUSK was tested in a wide range of applications and demonstrated superior performance across 23 patch-level and slide-level benchmarks, including image-to-text and text-to-image retrieval, visual question answering, image classification and molecular biomarker prediction. Furthermore, MUSK showed strong performance in outcome prediction, including melanoma relapse prediction, pan-cancer prognosis prediction and immunotherapy response prediction in lung and gastro-oesophageal cancers.

Bio: Jinxi Xiang is a multidisciplinary researcher specializing in signal processing and machine learning for healthcare applications. He integrated machine learning with medical imaging during his doctoral study at Tsinghua University (09/2016-06/2021). At Tencent AI Lab (07/2021-01/2024), he developed AI tools for clinical pathology using the techniques of image/video coding and multimodal learning. Since January 2024, he has been a postdoctoral researcher at Stanford University, focusing on computational pathology for cancer diagnosis and personalized treatment.

Paper Link: A vision–language foundation model for precision oncology | NatureLink opens in a new window

How to attend: Either turn up to the event on the day, or if you want to attend online then please contact Adam Shephard (adam.shephard@warwick.ac.uk) for more details.