SIQI LIU

Conditional Computation in LLMs (MoEs): Understanding the Concept of MoEs The Mixture of Experts (MoE) works on the idea of conditional computation, allowing the model to only run through some parts (an Expert) for a given input compared to a dense model where all parameters are used. It consists of a gating mechanism that selects the experts for the given input, enabling the model to scale without increasing the required computation. Popular LLM architectures like Mixtral and Grok use the same concept. What is MoEs? The Mixture of Experts (MoE) makes a simple modification to the usual architecture by replacing all or some of the Feed Forward layers with an MoE layer. The MoE layer consists of: 1. Router: A parameter gating mechanism that selects the experts used to process the given input by producing a probability distribution over experts. 2. Experts: Standalone NN modules with independent sets of parameters trained jointly with routers. It can use more complex architecture to create hierarchical MoE modules by implementing each expert as another MoE. - Implementation in Switch Transformers : Training was found to be 7x faster than training the base T5, keeping computation and training time constant. - Gshard and MeshTransformer : These frameworks propose efficient methods to train the MoE architectures using the parallelisation of models across GPUs/TPUs. Challenges with MoEs and Solutions 1. Inconsistent Batch Size for Experts: Can be overcome with the use of "soft" constraints in training loss. 2. Load Balancing Across Experts: The issue of repeatedly using the same few experts while training can be overcome by including better balancing mechanisms. 3. Model Instability Issues: Use of router-z Loss as an auxiliary loss to encourage smaller values within the router mechanism. 4. Model Overfitting: Requires different hyperparameter settings for effective fine-tuning. It was found that models benefit from high learning rates and smaller batch sizes. Recommended Reading/References 1. Hugging Face Blog on MoE - https://lnkd.in/g36gQJTx 2. Cameron Wolfe's Substack on Conditional Computation - https://lnkd.in/gtpjMHnJ

60

1 Comment

Kolmogorov–Arnold Networks seem like a paradigm shift in ML architectures. Curious to see how they evolve. "While MLPs have fixed activation functions on nodes ("neurons"), KANs have learnable activation functions on edges ("weights"). KANs have no linear weights at all -- every weight parameter is replaced by a univariate function parametrized as a spline. " https://lnkd.in/gdPk_qCQ

29

I recently read “Towards Monosemanticity: Decomposing Language Models with Dictionary Learning” by Bricken et al 2023. Published at transformer-circuits.pub In this paper the authors discuss the use of autoencoders to extract interpretable features from a one-layer transformer model. The approach aims to improve the understanding and transparency of artificial neural networks. To do this they use the concept of dictionary learning which is a method used to decompose the activations of the transformer model into a set of basis features that are sparse and interpretable. This involves training sparse autoencoders to identify a dictionary of monosemantic features (or features with a single meaning). These monosemantic features in single units (artificial neurons) reminds me of the “grandmother neuron” theory in neuroscience which proposes that specific neurons respond to specific highly complex stimuli. However, in real brains, it is widely believed that complex representations are encoded by the collective activity of groups of neurons (and not monosemantic). But going back to the paper, I really like the visualization tools and illustration to make such complex networks more transparent (as opposed to black boxes). Paper: https://lnkd.in/gR87pJcs

3

This paper investigates how shortcut learning in medical AI models—where models rely on spurious correlations rather than task-relevant features—leads to overestimation of their generalizability to new healthcare settings. 1️⃣ Healthcare AI models often learn from hidden biases in large, passively collected datasets, resulting in up to 20% overestimation of their performance. 2️⃣ Shortcut learning happens when AI models use spurious data correlations instead of meaningful task-related features, affecting generalizability. 3️⃣ The authors propose a method to estimate external accuracy by measuring and calibrating for data acquisition biases-induced shortcut learning, improving accuracy estimates to within 4%. 4️⃣ Experiments across 13 datasets and various medical data types (X-rays, CTs, ECGs, clinical notes, and lung sounds) demonstrate the prevalence and impact of shortcut learning. 5️⃣ The study emphasizes the need for better validation methods to ensure AI models are truly generalizable and reliable across different healthcare environments. ✍🏻 Cathy Ong Ly, Balagopal Unnikrishnan, Tony Tadic, Tirth Patel, Joe Duhamel, Sonja Kandel, Yasbanoo Moayedi, Michael Brudno, Andrew Hope, Dr. Heather Ross, Chris McIntosh. Shortcut learning in medical AI hinders generalization: method for estimating AI model generalization without external data. npj Digit. Med. 7, 124 (2024). DOI: 10.1038/s41746-024-01118-4

167

10 Comments

👉🏼 GPT-4 Turbo with Vision fails to outperform text-only GPT-4 Turbo in the Japan Diagnostic Radiology Board Examination 🤓 Yuichiro Hirano 👇🏻 https://lnkd.in/eguBsnAm 🔍 Focus on data insights: - GPT-4TV correctly answered 45% of questions - GPT-4 T correctly answered 41% of questions - No significant performance difference between the two models (P = 0.44) 💡 Main outcomes and implications: - No significant enhancement in accuracy observed with GPT-4TV using image input - GPT-4TV responses received lower legitimacy scores compared to GPT-4 T responses 📚 Field significance: - The study highlights the importance of evaluating the performance of multimodal language models in specific contexts like diagnostic radiology examinations 🗄️: #datainsights #radiology #GPT-4Turbo #performanceevaluation

1

This weekend I read “Zero-shot Tokenizer Transfer” by Minixhofer et al, 2024. In this paper the authors tackle the limitation of the dependence of language models (LM) to their tokenizer. In their approach, the authors propose to “mitigate” this limitation by changing the original tokenizer to an arbitrary one such that the performance of the LM does not degrade. To do this, the authors train a network that takes the tokenizer as input and predicts the embeddings for a base LM. Once trained, the authors claim that the network generalizes to new tokenizers for different LLMs. To me this paper is interesting because studies the implications of how the units of the vocabulary at hand (tokens) have to the language model. Being able to map across representations seems to be an important computing operation in artificial and biological networks. Paper: https://lnkd.in/gU-6Ja7h

5

🎉 Exciting News! 🎉 I am thrilled to share that our research paper titled "Segmentation of Liver Tumors by Monai and PyTorch in CT Images with Deep Learning Techniques" has been published! In this work, my co-author Jing Zhang and I delve into the cutting-edge application of deep learning techniques for medical imaging. We leveraged the powerful MONAI framework and PyTorch to enhance the segmentation of liver tumors in CT scans, aiming to improve accuracy and efficiency in medical diagnostics. 🔬 Key Highlights: Utilized state-of-the-art deep learning models to address the challenges of liver tumor segmentation. Integrated advanced image processing techniques to refine tumor boundaries and enhance model performance. Demonstrated the potential of MONAI and PyTorch in developing robust and scalable medical imaging solutions. 📘 Why It Matters: Liver cancer is a critical health issue, and early, precise detection is crucial for effective treatment. Our work contributes to the field by providing a more reliable method for identifying liver tumors, which could potentially lead to better patient outcomes. I am immensely proud of this accomplishment and grateful to all those who supported and collaborated on this journey. Special thanks to my co-author Jing Zhang for the invaluable partnership. 📖 You can read the full paper here https://lnkd.in/emaB73AM. Thank you to everyone who has been a part of this journey. Let’s continue pushing the boundaries of what’s possible in medical imaging and AI! #DeepLearning #MedicalImaging #AI #Healthcare #Research #CTScans #PyTorch #MONAI #LiverCancer

2

This is a wonderful paper. It shows how clever design (with simplicity and elegance) in architecture can improve the performance of transformers in arithmetic, rather than by simply throwing in more data and compute. https://lnkd.in/gZNnifc3

14

Last weekend I read the paper "Med42 - Evaluating Fine-Tuning Strategies for Medical LLMs: Full-Parameter vs. Parameter-Efficient Approaches" by Christophe et al., 2024. In this paper the authors compare two approaches to train Llama 2 models (7B, 70B) for medical tasks. The training approaches are called full-training and fine-tuning. They found that the full training approach (all parameters) performs better on the selected medical evaluation tasks than the selective fine-tunning (LoRa) approach (under certain conditions). The authors make their trained model (Med42) available in HF. To me this paper is important because they dive in the details and specific decisions to evaluate both training approaches in a specific (and relevant to me) domain. I am looking forward to apply a similar training and evaluation for LLMs in dentistry (any pointer to training datasets is highly appreciated). Paper: https://lnkd.in/gQBSZfmt Enjoy!

10

Always though medical sciences would be the perfect fit for ML and AI. Why? Because doctors are too expensive, completly overwhelmed, and stuck in the past with obsolete education. Also they can be too old to “see” properly, and even to connect the dots with family , environment, work related and so on. AI learns from billions of records, while human doctors can see less than 10000 cases in their lifetime.

3

1 Comment

It was a great serendipity to meet Eivind Antonsen Segtnan and to support Segtnan.ai effort to practically implement their recent research in hospitals to make disease testing up to 80% more efficient. Btw, their paper just got published in Nature Scientific Reports: https://lnkd.in/dgR56Jj2. Congrats! TL;DR This solution has the potential to save millions when testing for diseases such as Covid-19, HIV, and chlamydia, by grouping and testing samples from multiple individuals together. Segtnan.ai's mobile app integrates AI to tailor risk assessments and employ data-driven methods to optimize group size. Their groundbreaking solution can significantly increase testing capacity and save both time and money in the healthcare system.

11

2 Comments

Love this paper. There are countless low-hanging fruits in deploying LLMs in important fields, including healthcare. I particularly liked this use case. Social Determinants of Health (SDoH) have been close to my heart since my work with RWJF (with Oktawia Wojcik) and Sutter Health (with Sylvia Sudat) to extract them from EHRs many years ago. This is a clear-cut instance where AI is not only effective but better than humans in annotating and surfacing important context for the healthcare provider. #AI #healthcare https://lnkd.in/eGMkjGFQ

22

2 Comments