Cohort: 1 

Location: Hangzhou, China 

Application Title: Multimodal AI platform for clinical decision support and medical consultation services 

Team Members: Zhang Junjie (Executive Sponsor), Liu Junwei, Zhu Dongwei, Feng Lei, Wei Peng, Wang Jian

The challenge 

Healthcare delivery is often fragmented. Patients interact with multiple providers across different stages of care, while clinicians rely on separate systems for records, diagnostics and follow-up. As a result, information is spread across formats such as clinical notes, imaging results and sensor data, making it difficult to form a complete picture of a patient’s condition. 

Many “smart health” systems attempt to address this by adding AI tools to existing workflows. These tools may assist with a single task, such as triage or image analysis, but they rarely connect across the full patient journey. Clinicians must reconcile outputs manually, and patients experience care as a series of disconnected interactions. Privacy and compliance requirements further limit the ability to integrate data across institutions. 

This fragmentation reduces efficiency and makes it difficult to scale AI support beyond isolated use cases. 

The solution 

A unified platform spanning the patient journey 

Ant Group developed a multimodal AI platform designed to support clinical decision-making and patient consultation across different stages of care. 

At its core is a medical-grade large language model capable of processing text, medical images, audio and video. This allows the system to reason across symptoms, laboratory results and imaging data within a single framework. A privacy-preserving data layer encrypts and harmonises clinical information from hospitals, primary care clinics and consumer health devices, enabling secure use in regulated environments. 

The platform assists both clinicians and patients. Before a visit, an AI assistant helps patients record symptoms and schedule appointments. During the visit, it provides convenient medical treatment for users throughout the entire process, and offers customized intelligent triage services to enhance the user's healthcare experience. Following diagnosis and treatment, the system also delivers functions such as follow-up care, medication guidance, and rehabilitation planning services, thereby establishing a continuous feedback loop rather than a one-time interaction.

How it works 

Edge–cloud coordination with medical-grade governance 

The platform uses a hybrid edge-to-cloud architecture. Real-time interactions, such as voice-based consultation or digital-human interfaces, run at the edge to minimise latency. Model training and updates occur in the cloud, allowing improvements to be deployed quickly across institutions. 

The system is built in layers. General foundation models provide base capabilities. A domain-specific layer is trained on professional medical datasets and clinical literature. A top-tier medical model is then refined through reinforcement learning and cross-modal alignment to support clinical reasoning. 

Governance is handled through a multi-layer security framework that includes standardised evaluation processes and ethical oversight. This design supports use in sensitive contexts while maintaining transparency and compliance. 

How it differs from conventional approaches 

Moving beyond isolated clinical tools 

Traditional health AI systems are typically deployed as standalone applications. Each tool addresses a narrow task and relies on siloed data. Integration is left to hospitals, increasing operational burden. 

Ant Group’s approach provides a shared service layer that supports multiple clinical functions within a single system. Rather than bolting AI onto existing tools, the platform connects data, reasoning and interaction across the care pathway. This reduces manual reconciliation and supports more consistent use across departments. 

The hybrid architecture also allows on-premise deployment where required, addressing compliance constraints that limit adoption of cloud-only solutions. 

Impact 

• Auxiliary diagnostic accuracy above 90% across supported use cases 
• 80% reduction in medical literature search time for clinicians 
• Serving 160 million users, with nearly 1 million doctors across more than 5,000 institutions 

Why it matters 

Supporting scalable and trusted clinical AI 

As healthcare systems seek to improve access and efficiency, AI tools need to integrate into care delivery rather than operate at its edges. Ant Group’s platform shows how multimodal models, combined with privacy-first infrastructure, can support clinical reasoning across the patient journey. 

By reducing fragmentation and supporting continuous interaction between patients and clinicians, the approach offers a pathway toward more scalable and consistent AI-supported care.