Hoifung Poon’s Post

Excited to attend #CVPR 2024 tomorrow (Summit 324 14:20-15:00) and talk about the research frontier in multimodal #GenerativeAI for precision health: https://lnkd.in/gheD5Svn. The confluence of digital transformation and GenAI revolution opens unprecedented opportunities for optimizing patient care and accelerating biomed discovery, but challenges abound in the forward path. Generic frontier models such as GPT-4 are amazingly proficient in understanding biomedical text (e.g., MedPrompt https://lnkd.in/gmgWPiS3), but they exhibit major competency gaps in other modalities such as medical imaging and multi-omics. In this talk, I'll present some learning in bridging such competency gaps: BiomedCLIP: For any modality, its study typically involves natural languages, which means that modality-text pairs are often abundantly available. We explore using publicly available data for vision-language pretraining (15 million PMC image-caption pairs): https://lnkd.in/ghF55fyU. Excited to see many subsequent amazing works, e.g., Twitter (PLIP, James Zou), Youtube (QUILT, Linda Shapiro), PMC/textbook (CONCH, Faisal Mahmood). LLaVA-Med: Standard contrastive learning treats all modalities equally. We instead explore using text as the interlingua modality and focus on learning an adapter for "translating" into the text semantic space, which is very data efficient. To train a multimodal GenAI copilot, we leverage GPT4 to synthesize instruction-following data from available image-text pairs. LLaVA-Med also features a modular design with late-fusion, with plug-and-play modality-specific encoders/decoders, thus can easily scale to general use cases (e.g., combining X-ray, CT, MRI, digital pathology, multi-omics). As a PoC, LLaVA-Med 1.0 was trained using just a tiny fraction of BiomedCLIP data: https://lnkd.in/deNH4XkR. We have since substantially improved the model and just release LLaVA-Med 1.5: https://lnkd.in/gndEMbC9. We are also exploring specialization such as LLaVA-Rad https://lnkd.in/g7cCZqse. Excited to see many subsequent amazing works, e.g., MAIRA (Javier Alvarez Valle), PRISM (SIQI LIU, Kristen Severson), PathChat (Faisal Mahmood). Multimodal generation: The LLaVA-Med framework can incorporate modality-specific decoders for generating multimodal output, e.g., BiomedJourney https://lnkd.in/gFnvqFTw, BiomedParse https://lnkd.in/gzFPe5aH (Mu Wei will present in the same workshop 10:40-11:20). Whole-slide modeling: Digital pathology poses unique computational challenges due to its enormous size. We propose the first whole-slide pathology foundation model, GigaPath, in our #Nature paper https://lnkd.in/gHjmT7We. So much more remains to be done!

I think we may be approaching a tipping point in AI for medical imaging. Incredible progress at the @CVPR workshop on foundation models for medical imaging. Our teams were pleased to share work @MSFTResearch on #Biomeparse and #GigaPath to standing-room only audience.

It was my great pleasure to participate in the AIPM (https://lnkd.in/ehxWt4ih) panel on “Generative AI in Drug Development” live event at the Interdisciplinary Science and Engineering Complex at Northeastern University last week where I had the opportunity to share more about Pfizer's efforts to leverage innovations in human-led AI/ML to enhance R&D productivity. I joined a dynamic panel with FDA’s Robert Ball, Microsoft’s Hoifung Poon, Johns Hopkins School of Public Health's Brian Caffo and Google Venture’s Anthony Philippakis, with expert moderation by David Sontag  of MIT &  Layer Health where we discussed applications of Large Language Models for FDA’s Sentinel program, foundation models for pathology related multi-modal data analysis, automating regulatory corpus with #GenerativeAI for pharmaceutical product submissions, where tried-and-true classical AI methods may be better suited, and using generative AI to advance precision medicine in Cardiology. Thank you to the AIPM organizing committee and my fellow panelists for the valuable discussion. Photo Credit: Demissie Alemayehu