Development of a modular optical character recognition and large language model application

Liew, Zi Yue (2026) Development of a modular optical character recognition and large language model application. Final Year Project, UTAR.

PDF Download (5Mb)

Abstract

Optical Character Recognition (OCR) and Large Language Models (LLMs) can be combined to improve document digitization, especially when traditional OCR struggles with complex layouts, multilingual content, and limited contextual understanding. This project addresses these limitations by developing a modular OCR-LLM application designed for English, Malay, and Chinese document processing. The methodology involves a flexible pipeline that evaluates five free and open-source OCR engines (Tesseract, EasyOCR, PaddleOCR, DocTR, and DeepSeek-OCR) and two locally deployable 8-billion-parameter LLMs (DeepSeek-R1 and Llama 3.1), which are then deployed via a Streamlit web interface. Performance evaluation on single-column and unstructured layouts demonstrated that DeepSeek-OCR 2 achieved the highest robustness, maintaining 90.17% accuracy on simple layouts and 83.78% on complex structures. For contextual refinement, DeepSeek-R1 outperformed Llama 3.1 with a 94.77% BERT-based Semantic Similarity Score (BERTScore) and 81.38% Bilingual Evaluation Understudy with Representations from Transformers (BLEURT) score. While post-processing by the Artificial Intelligence (AI) agent improved the average F1-score to 91.34%, it introduced a trade-off by increasing the Character Error Rate (CER) and Word Error Rate (WER) due to semantic reformatting. Overall, this project demonstrates that combining free, open-source, and locally deployable OCR engines with LLMs in a modular architecture can produce a flexible, and user-friendly document processing system without Application Programming Interfaces (APIs) charges. Moving forward, future work should focus on GPU-based optimization, improved support for handwritten and highly degraded documents, stronger multilingual handling, and further enhancement of agentic AI capabilities for more functions. Keywords: Optical Character Recognition, Large Language Model, DeepSeek-OCR 2, Multilingual Document Processing, AI Agent, Document Digitization, Streamlit Web Application Subject Area: TA168 Systems engineering

Actions (login required)