Multimodal Medical LLM (In progress)

Medical MLLM Agent (Project in Progress), this service is a collaboration with Guangzhou LBP Medicine Science & Technology Co., Ltd.(688393.SH). Based on the MLLM Agent, it aims to complete multimodal retrieval and generation of a case library. It can search patient case records, diagnostic images, and other medical records, and provide MLLM consultation, offering reliable diagnostic assistance to medical personnel and patients. We are still conducting this research.

Project Collaborator: Jiale Gu:jialegu@hkust-gz.edu.cn, Feiyu Huang:feiyuhuang@hkust-gz.edu.cn, Ri Su,Yonghao Si, Zhao Chen

I am responsible for the following tasks

• Fine tuning of large language models (Unfortunately, it was overtaken by newer LLM models ).

• Corpus data collection (>1T token)

• Dialogue Intent Recognition: Based on the content of the user’s message and historical information, identify the required routing service. Response Time: 20~100ms

Some Case

• Image-Text dialogue

• case library retrieval