Projects

PUMC-Colposcopy report interpretation assistant

This project, in collaboration with Peking Union Medical College (北京协和医学院), aims to develop an intelligent medical system focused on interpreting colposcopy reports, independently completed by me. The primary goal of the project is to help patients better understand their examination reports and provide diagnostic support to doctors, thus reducing doctor-patient pressure and enhancing clinical efficiency. The system’s technical architecture and features are composed of the following key modules:

• Large Language Model and Multimodal Processing: Facilitates seamless integration of text and image data, enhancing information extraction and interpretation across various data types.
• RAG (Retrieval-Augmented Generation): Optimizes retrieval speed and accuracy from the knowledge base, ensuring the reliability and authority of the information provided.
• CoT (Chain of Thought) Reasoning: Adopts a step-by-step reasoning approach to generate interpretations and recommendations, simulating doctors' thought processes for advanced intelligent diagnostic support.

Semi-automatic intelligent labeling

This is the first open-source project applying the SAM model to real-time annotation of whole slide images (WSI) in the medical field which I developed. This project utilizes models from the SAM family (e.g., SAM, SAM2, etc.) to assist medical personnel in annotating WSI case slides, significantly improving the speed and efficiency of medical researchers' annotation work. It provides accurate and reliable data for model training.

• The first open-source project applying the SAM model to real-time annotation WSI
• Surpport SAM family
• significantly improving the speed and efficiency of annotation work

Medical MLLM Agent

Medical MLLM Agent (Project in Progress), this service is a collaboration with Guangzhou LBP Medicine Science & Technology Co., Ltd.(688393.SH). Based on the MLLM Agent, it aims to complete multimodal retrieval and generation of LBP's case library. It can search patient case records, diagnostic images, and other medical records, and provide MLLM consultation, offering reliable diagnostic assistance to medical personnel and patients. We are still conducting this research.

Medical Data Manager

This is the medical data management platform I developed, supporting storage and management operations for medical data. The specific features are as follows:

• Data upload, download, and modification management.
• Whole slide image（WSI） and regular image preview.
• 🌟 Initiate data-driven deep learning tasks.

MDI Data Evaluation

This project aims to briefly evaluate the annotation results of doctors on a cell electron microscope dataset using various methods, including feature information calculation, training, and data screening, to improve the quality and accuracy of the dataset annotations.