arXiv cs.AI· Lubnaa Abdur Rahman, Ioannis Papathanail, Maria Panagiotou, Nefeli Dimitropoulou, Margret Har{\dh}ard\'ottir, Elisabeth J. den Brok, Bastiaan E. de Galan, Stavroula Georgia Mougiakakou·· 4 小时前AI 评分37
NutriVisRAG：用多模态范例为视觉语言模型的营养估算提供视觉依据
NutriVisRAG: Multimodal Exemplar Grounding for Nutrient Estimation with Vision-Language Models
AI 导读
NutriVisRAG 是一个检索增强生成框架，通过视觉编码器检索带营养标注的相似餐食图像，作为多模态 in-context 范例供 VLM 在推理时估算营养，无需任务特定微调。
正文 · 原文
Abstract:Diet is a major determinant of health, yet accurate dietary assessment from free-living meal images remains challenging. Existing retrieval-based Vision-Language Model (VLM) approaches to nutrient estimation typically rely on textual food representations, while, to our knowledge, direct grounding on visually similar annotated meals has not been investigated. We present NutriVisRAG, a retrieval-augmented generation framework that uses a vision encoder to retrieve similar meals with known nutritional annotations and provide them as multimodal in-context exemplars for VLM nutrient estimation at inference time, without task-specific fine-tuning. We evaluate NutriVisRAG across four datasets, four VLMs, three retrieval encoders, and two prompting strategies, including free-living meals from adults with insulin-treated diabetes, where carbohydrate estimates are compared against two dietitians' annotations. Across all 96 configurations (4 datasets x 4 VLMs x 3 retrieval encoders x 2 prompting strategies), NutriVisRAG reduced the primary estimation error in 56, with improvements more frequent under whole-meal (35/48) than per-item prompting (21/48). Across the evaluated configurations, the lowest NutriVisRAG error was below the best Vanilla error in 13 of 16 VLM--dataset pairs. For GPT-5.4-mini, under whole-meal prompting, the best-performing retrieval encoder for each dataset reduced the primary error on all four datasets relative to Vanilla, with reductions of 1.7--35.7%. Overall, our results show that grounding in multimodal exemplars comprising images and respective annotations can improve nutrient estimation at inference time without task-specific fine-tuning, although the benefits remain configuration-dependent.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.08054 [cs.CV] |
| (or arXiv:2611.08054v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.08054 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Lubnaa Abdur Rahman [view email]
[v1]
Thu, 5 Nov 2026 14:10:35 UTC (2,421 KB)
来源：arXiv cs.AI · arxiv.org