PUMC-Colposcopy report interpretation assistant

Introduction

This project, in collaboration with Peking Union Medical College (北京协和医学院), aims to develop an intelligent medical system focused on interpreting colposcopy reports, independently completed by me. The primary goal of the project is to help patients better understand their examination reports and provide diagnostic support to doctors, thus reducing doctor-patient pressure and enhancing clinical efficiency. The system’s technical architecture and features are composed of the following key modules.

if you are interested in this project or want to try it out, please contact us by email📮: czhangcn@connect.ust.hk .

Features

1. LLM & Multimodal Recognitions

Users can upload their colposcopy reports (pdf, txt, images) through mobile or desktop platforms. The system employs a multimodal large language model (MLLM) for text extraction and image information processing. Utilizing Prompt Engineering techniques, the system translates complex medical terminology into clear, accessible language, helping patients to understand their health status more easily, and alleviating the pressure on doctors during consultations.

2. Colposcopy Medical Knowledge Base with RAG

A colposcopy knowledge base, based on data from Peking Union Medical College, has been established. Through Retrieval-Augmented Generation (RAG), the system precisely retrieves authoritative medical knowledge from knowledge graphs and vector databases, providing a scientific basis for diagnosis.

3. CoT (Chain of Thought) Reasoning

The system incorporates Chain of Thought (CoT) reasoning methods aligned with doctors’ diagnostic logic, providing a complete workflow from report interpretation to diagnostic recommendations. This closed-loop process for information parsing and decision support enables the system to generate clinically compliant explanations and aid doctors in their diagnostic decisions.

Evaluation

Here is a simple evaluation comparing with gpt-4o which shows that our interpretation works better. On the left are the gpt-4o results, and on the right are our results.Each criteria is scored on a scale of 0-10, and the scoring criteria are:

• Comprehensiveness: How much detail does the answer provide to cover all aspects and details of the question?
• Diversity: How diverse and rich are the answers in providing different perspectives and insights on issues?
• Empowerment: To what extent do the answers help the reader understand and make informed judgments about the topic?
• Directness: How does the answer address the problem concretely and clearly?

Conclusion

This system not only improves patient comprehension of report content but also significantly alleviates the workload on doctors, providing a more efficient and intelligent solution for colposcopy examination and subsequent clinical care.