arXiv cs.AI· Zahraa Selim, Menna Hamed, Wesam Ahmed, Sohyla Said, Sara Basheer, Rami Zewail·· 1 小时前AI 评分34
面向消费级 GPU 的边缘端多模态 RAG：硬件感知量化与架构优化研究
Edge-Native Multimodal RAG: Hardware-Aware Quantization and Architectural Optimization for Consumer GPUs
AI 导读
一项 arXiv 论文提出面向消费级 Turing 架构 GPU（如 NVIDIA Tesla T4）的多模态 RAG 工程方案。研究对比 GPTQ 与 NF4 量化：GPTQ 吞吐量更高（51.6 tok/s）但在 2048+ token 时上下文不稳定，NF4 保持长上下文稳健（81 ms/tok）；结合解耦的 Florence-2 视觉集成仅增加 430 MB VRAM。
正文 · 原文
Abstract:Deployment of retrieval-augmented generation (RAG) for Edge-AI is constrained by memory and bandwidth limitations. A standard 7B parameter model consumes 14 GB Video Random Access Memory (VRAM) in half-precision (FP16), precluding co-location of vector indices and Key-Value (KV) caches. This paper presents an engineering integration and empirical evaluation of a multimodal RAG framework optimized for consumer-grade Turing hardware (e.g., NVIDIA Tesla T4). Our system-level analysis identifies a critical trade-off: while generative pre-trained transformer quantization (GPTQ) offers superior throughput (51.6 tok/s), it exhibits significant context instability at 2048+ tokens on Turing architectures. Conversely, NormalFloat 4-bit (NF4) maintains robust long-context performance with viable latency (81 ms/tok). By coupling NF4 with a decoupled Florence-2 vision integration, we achieve visual reasoning with only 430 MB VRAM overhead. Furthermore, empirical observations of prompt-induced search space pruning show that structured prompting reduces latency by 47% without altering computational intensity (1.88 GFLOPs/token). The integrated architecture maintains an 8.37 GB footprint, establishing a viable RAG pipeline for 12-16 GB GPUs. Finally, structured few-shot prompting effectively compensates for quantization-induced reasoning loss, restoring mathematical accuracy from 35.0% to 92.0% across 200 trials.
| Comments: | Accepted to the 14th International Japan-Africa Conference on Electronics, Communications and Computations (JAC-ECC 2026).6 pages, 3 figures, 6 tables |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.14924 [cs.AI] |
| (or arXiv:2611.14924v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.14924 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Rami Zewail Dr. [view email]
[v1]
Wed, 11 Nov 2026 02:26:07 UTC (88 KB)
来源：arXiv cs.AI · arxiv.org