Self-hosted multi-agent RAG system for contextual document processing

Eng, Zi Jun (2025) Self-hosted multi-agent RAG system for contextual document processing. Final Year Project, UTAR.

PDF Download (4Mb)

Abstract

The increasing use of Artificial Intelligence (AI) in document processing faces persistent challenges such as hallucination, privacy risks, and limited adaptability. This study presents a self-hosted multi-agent Retrieval-Augmented Generation (RAG) system designed to address these limitations by enhancing accuracy and preserving data privacy through a fully local and modular architecture. Built using Marker, Ollama, LangGraph, and Weaviate, the system enables flexible deployment and coordination between agents. Evaluation using the SQuAD dataset measured retrieval and generation performance through metrics such as Recall@3, Mean Reciprocal Rank (MRR), Context Recall, Faithfulness, and Answer Correctness. Two evaluation methods were employed: a calculation-based approach on 100 samples for quantitative assessment, and an LLM-as-Judge approach using GPT-4o on 20 samples for qualitative, human-like evaluation. Results show strong retrieval performance with a Recall@3 of 90%, MRR of 75%, and Context Recall of 100%, demonstrating accurate and consistent grounding. The generation results indicate improved faithfulness and contextual relevance, though challenges remain in scalability and factual precision. Overall, the findings show that the proposed multi-agent RAG system effectively mitigates hallucination and privacy concerns while maintaining adaptability, making it a promising approach for secure and accurate AI-driven document processing. Keywords: Artificial Intelligence (AI), Retrieval-Augmented Generation (RAG), Large Language Models (LLMs), Self-Hosted AI Subject Area: Q300-390 Cybernetics

Actions (login required)