Abstract:This paper presents a principled and scalable framework for systematically generating complex Question Answering (QA) data. In the core of this framework is a graphlet-anchored generation process, where small subgraphs from a Knowledge Graph (KG) are used in a structured prompt to control the complexity and ensure the factual grounding of questions generated by Large Language Models. The first instantiation of this framework is BioGraphletQA, a new biomedical KGQA dataset of 119,856 QA pairs. Each entry is grounded in a graphlet of up to five nodes from the OREGANO KG, with most of the pairs being enriched with relevant document snippets from PubMed. We start by demonstrating the framework's value and the dataset's quality through evaluation by a domain expert on 106 QA pairs, confirming the high scientific validity and complexity of the generated data. Secondly, we establish its practical utility by showing that augmenting downstream benchmarks with our data improves accuracy on PubMedQA from 49.2% to 68.5% in a low-resource setting, and on MedQA from a 41.4% baseline to 44.8% in a full-resource setting. Our framework provides a robust and generalizable solution for creating critical resources to advance complex QA tasks, including MCQA and KGQA. All resources supporting this work, including the dataset (this https URL) and framework code (this https URL), are publicly available to facilitate use, reproducibility and extension.
| Comments: | 15 pages, 7 figures, conference (ECIR) |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2604.26048 [cs.CL] |
| (or arXiv:2604.26048v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2604.26048 arXiv-issued DOI via DataCite (pending registration) |
|
| Journal reference: | Advances in Information Retrieval: 48th European Conference on Information Retrieval, ECIR 2026, Delft, The Netherlands, March 29 - April 2, 2026, Proceedings, Part IV |
| Related DOI: | https://doi.org/10.1007/978-3-032-21321-1_62
DOI(s) linking to related resources |
Submission history
From: Richard Jonker [view email]
[v1]
Tue, 28 Apr 2026 18:33:21 UTC (869 KB)