跳到正文
arXiv cs.AI· Faizan Faisal, Umair Yousaf, Muhammad Hassnain, Zeerak Babar·· 4 小时前AI 评分31

LEGAL-UQA:面向乌尔都语使用者的宪法问答英乌双语平行基准

LEGAL-UQA: Testing Constitutional Question-Answering Systems for Urdu Speakers with a Parallel English-Urdu Benchmark

AI 导读

arXiv 论文提出 LEGAL-UQA,一个包含 619 组问答对的英语-乌尔都语平行基准,题目关联巴基斯坦宪法条文,每组含双语问题、参考答案和原文段落。评测将答题与段落检索分开进行,发现模型排名取决于答案相似度的度量方式,简单词匹配检索效果优于所测嵌入方法;自动裁判 JEV 与人工评分的一致率低于给所有答案满分的基础线。审计还发现部分参考答案遗漏原文支持的信息。

正文 · 原文

View PDF HTML (experimental)

Abstract:Legal information assistants could help Urdu speakers understand their constitutional rights, but building reliable systems requires test data and evaluation methods suited to the language. We present LEGAL-UQA, a parallel English-Urdu benchmark of 619 question-answer pairs linked to passages from Pakistan's Constitution. Each example includes the question, reference answer, and source passage in both languages. We evaluate two components separately: answering Urdu questions when the source passage is supplied, and retrieving the passage from which each question was generated. We also compare JEV, a decision model used as an automated judge, with human grades. The findings show that model rankings depend on how answer similarity is measured, while a simple word-matching method retrieves source passages more effectively than the embedding methods tested. At the evaluated cutoff, JEV agrees with human grades less often than a baseline that gives every answer full credit. An audit also identifies cases where generated references omit information supported by the source. Together, these findings highlight the need to test retrieval and answering separately, include simple retrieval baselines, and validate automated judges and reference answers with human review when engineering legal QA software for Urdu speakers.
Comments: 8 pages
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
MSC classes: 68T50
Cite as: arXiv:2410.13013 [cs.CL]
  (or arXiv:2410.13013v2 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2410.13013

arXiv-issued DOI via DataCite

Submission history

From: Faizan Faisal [view email]
[v1] Wed, 16 Oct 2024 20:14:45 UTC (40 KB)
[v2] Fri, 13 Nov 2026 00:14:19 UTC (70 KB)

来源:arXiv cs.AI · arxiv.org