Project Background

Across many financial systems, consumer complaints represent one of the most critical but under-leveraged sources of supervisory intelligence. Central banks receive thousands of complaints from the public each year. However, these are often submitted in unstructured formats: handwritten letters, scanned images, PDFs, and forms in multiple languages. Manually reviewing, classifying, and acting upon such data is time-consuming and prone to inconsistencies.

To overcome this challenge, a central bank in the Middle East decided to explore the use of AI technologies to automate the extraction and structuring of complaint data. Through the Public-Private Secondments for SupTech Innovation initiative, the World Economic Forum and the University of Cambridge SupTech Lab brought in a dedicated seconded team from Abhi, a fintech infrastructure company based in Pakistan. The team included experts in OCR, natural language processing, and large language models, who collaborated directly with the central bank's supervision and IT departments.

Aligned with the Bank's broader digital transformation strategy, the project’s objective was to co-design a proof of concept capable of:

• Extracting text from a variety of unstructured document formats.
• Analysing the extracted text to identify key complaint themes and metadata.
• Converting free-text complaints into structured data that could be analyzed or integrated into existing systems.
• Achieving high accuracy across multilingual inputs (Arabic and English).
• Ensuring that the solution could be deployed on-premise, with no reliance on cloud infrastructure.

Solution Overview

The proof of concept developed by Abhi was a modular, AI-driven pipeline capable of ingesting and analyzing consumer complaints in multiple formats and languages. The solution design combined several technologies into a single architecture:

Optical Character Recognition (OCR)

• The solution design leveraged a custom-trained OCR engine (Surya OCR) that performed well on scanned Arabic text, including noisy or handwritten documents, achieving 91.6% character-level accuracy for low-resolution and poorly scanned documents.
• Pre-processing steps such as skew correction, noise filtering, and contrast and resolution enhancement were applied to optimize OCR outputs and improve reliability, resulting in a 13% improvement in OCR accuracy.
• The system could handle various layouts, such as bank forms, handwritten notes, and multi-column PDFs.

Multilingual Text Extraction and Preprocessing

• Once text was extracted, it was normalized, language-tagged (Arabic or English), and segmented into tokens for further processing.
• Auto-orientation and layout parsing steps ensured clean, language-aware inputs to downstream NLP modules.
• The architecture supported a dual-language NLP pipeline to ensure semantic consistency across Arabic and English complaints.

Keyword and Entity Recognition

• A combination of rule-based matching and trained Named Entity Recognition (NER) models, such as Qwen-72B, was used to identify entities such as:
  • Financial product types (loans, cards, savings)
  • Complaint themes (fraud, delay, miscommunication)
  • Institution names or locations
• A taxonomy of keywords was developed by the central bank and used to standardize output fields.

Large Language Model (LLM) Integration

• To enhance prediction accuracy and interpretability, Abhi integrated a Retrieval-Augmented Generation (RAG) mechanism and Labeling for Retrieval Augmentation (LfRA) into the system.
• The system retrieved historical resolution examples using ChromaDB and learned from human-corrected outputs to refine future predictions.
• This LLM module was fine-tuned on complaint-style text and used to:
  • Disambiguate noisy OCR output
  • Identify implied intent and complaint tone
  • Improve field tagging through context-aware reasoning

Output Structuring

• All extracted and interpreted data was transformed into a structured JSON format that mapped each complaint to fields such as:
  • Date, product type, nature of complaint, involved institution, suggested next step (optional).
• The system maintained traceability and auditability by linking each output field back to its source text span.

Technology Stack

• OCR Engines
  • Surya OCR: Main engine optimized for handwritten Arabic text.
  • Tesseract: Backup engine for typed Arabic text.            
• Named Entity Recognition (NER)
  • Qwen 2.5 - 72B: Accessed via OpenRouter for contextual entity recognition and improved NLP accuracy. Extracts names, dates, complaint types, and other key fields.
• Sentiment Analysis
  • Rule-enhanced model: Flags complaints with strong negative sentiment or urgency indicators.
• Backend Framework
  • Django: Handles complaint-processing logic and exposes REST APIs for integration.

Evaluation and Accuracy

The solution design was tested on a diverse set of real complaint documents provided by the central bank, including samples categorized as A (complex), B (moderate), and C (simple). Abhi conducted side-by-side accuracy benchmarking with and without the LLM enhancement.

The team also conducted manual validation and native Arabic experts alongside central bank staff, who verified that extracted data aligned with the actual complaint contents.

Open-Source Delivery and On-Premise Readiness

A key achievement of the project was that all outputs were delivered as open-source, removing concerns about vendor lock-in. This included the full OCR and NLP pipeline, keyword taxonomies, model weights (where licensing allowed), and user documentation.

Recognizing the requirements for data privacy and infrastructure control, Abhi also provided:

• A detailed on-premise deployment plan, including system architecture diagrams and VM requirements.
• Containerized services (via Docker) to ease local deployment.
• Instructions for integrating with the Bank's existing IT stack and secure environments.
• Recommendations for future updates and retraining based on evolving complaint patterns.

This ensured that the solution could be managed entirely within the Bank's infrastructure — aligning with their compliance and data protection policies.

Impact and Lessons Learned

This project represents a strong example of how targeted technical secondments can generate solution designs that address concrete supervisory challenges. Key impacts include:

• Operational Efficiency: Complaints that once required manual reading and tagging can now be structured with high accuracy in seconds.
• Capacity Building: The internal supervision and IT teams were directly involved throughout, building familiarity with modern AI tooling.
• Technology Transfer: All code, models, and documentation were transferred in open format, enabling future upgrades.
• Replicability: The modular design means that similar authorities can adopt the approach in other jurisdictions.

Next Steps: Toward Resolution Intelligence

Following the proof of concept’s insight, the central bank is now exploring operational deployment and extending the system to link structured complaints to predefined resolution actions, taking into account urgency, content, and regulatory significance. This could enable faster triage, consistent response strategies, and improved consumer outcomes.

Abhi proposed integrating lightweight AI agents trained on tagged resolution pathways, the system could dynamically recommend actions based on complaint attributes. These agents could operate on top of the structured JSON output already produced, making the extension both cost-effective and immediately impactful.

This roadmap would allow the Bank to move from reactive complaint handling to intelligent supervisory resolution, aligning with its broader digital transformation goals.

About the Project Partners

Abhi Private Limited

Abhi is a leading fintech infrastructure company, headquartered in Pakistan, specializing in real-time payments and embedded finance. The seconded team included:

• Ejaz Anwer – Technical Lead & Deployment Architect
• Ahmad Khalid Butt – Solution Designer, NLP & Machine Learning Engineer
• Abdul Bari – Arabic Language Expert

Public-Private Secondments for SupTech Innovation

Co-led by the Cambridge SupTech Lab and the World Economic Forum, this initiative connects technical experts from the private sector with financial authorities through short-term secondments. The goal is to accelerate innovation in supervision and regulation by solving practical challenges with cutting-edge technology.