Jarvis: development of a local AI-powered personal assistant for seamless integration within the Apple ecosystem

Low, Jia Hao (2026) Jarvis: development of a local AI-powered personal assistant for seamless integration within the Apple ecosystem. Final Year Project, UTAR.

PDF Download (2671Kb)

Abstract

This project introduces JARVIS, a powerful, fully localized, and privacy-focused AI personal assistant which built on Apple ecosystem (iOS, macOS, and iPadOS). While modern LLMs possess a high level of intelligence, they are fundamentally limited by cloud reliance, posing significant risks to user privacy and reliability in offline environments. This research addresses these limitations by developing a unified multimodal agentic pipeline utilizing the MLX framework to deploy a 4-bit quantized Qwen3-4B model entirely on-device. The core innovation of JARVIS lies in its Reason + Action (ReAct) agentic workflow, which transitions the system from a passive chatbot to an active autonomous agent. By implementing a zero-shot tool-calling orchestration layer, the LLM can dynamically reason through user queries to execute HomeKit commands for local hardware control and perform anonymized internet searches via the Brave Answer API. To ensure privacy, the agent employs a "minimal intent" query refiner to mitigate data leakage during external API calls. The multimodal interface is powered by WhisperKit for real-time, streaming STT across 99 languages, while the TTS engine utilizes Apple’s Natural Language framework to perform per-sentence language identification, ensuring seamless synchronization with neural voice models. Through deep optimization for Apple Silicon’s Unified Memory Architecture, the system achieves an industry-leading throughput, peaking at 13 TPS with an average of 11 TPS, ensuring real-time responsiveness that exceeds human speech rates. Experimental evaluations confirm that JARVIS maintains stability over multi-turn conversation using KV-cache management, successfully bridging the gap between high-level reasoning and local hardware integration. This project also serves as a proof-of-concept for "Sovereign AI", by demonstrating that complex, multimodal, and tool-augmented intelligence can be executed locally without compromising performance or user data sovereignty.

Actions (login required)