Skip to main navigation Skip to search Skip to main content

Comparative Evaluation of RAG and GraphRAG for Open-Ended Question Answering

Research output: Contribution to conference › Paper › peer-review

19 Downloads (Pure)

Abstract

Retrieval-Augmented Generation (RAG) has become the basis in modern question answering (QA) systems, combining Large Language Models (LLMs) with external document retrieval. However, traditional RAG architectures often struggle with retrieving semantically structured or context-rich content, which can impact accuracy and relevance. This paper presents a comparative evaluation of a standard RAG model and a GraphRAG model. The GraphRAG approach combines graph-based document representation within the retrieval pipeline to assess their efficiency on a structured question answering task. This research leverages two modern Ollama-hosted models, Phi-4 and LLaMA 3.2 3B, as the base language models for both retrieval pipelines. Using a custom dataset derived from German coalition policy documents, this research evaluates performance through both lexical and semantic metrics. The results demonstrate that GraphRAG consistently outperforms traditional RAG in semantic alignment and contextual accuracy, particularly when paired with Phi-4. These findings aim to contribute to the growing body of work on hybrid retrieval strategies and support the case for graph-enhanced architectures in long-form QA systems, which are central to advancing structured and knowledge-aware retrieval methods in complex information domains.
Original languageEnglish
Pages73-80
Number of pages8
Publication statusPublished online - 19 Apr 2026
EventThe Eighteenth International Conference on Advanced Cognitive Technologies and Applications - Portugal, Lisbon, Portugal
Duration: 19 Apr 2026 → 23 Apr 2026
Conference number: 18
https://www.iaria.org/conferences2026/COGNITIVE26.html

Conference

ConferenceThe Eighteenth International Conference on Advanced Cognitive Technologies and Applications
Abbreviated titleCOGNITIVE 2026
Country/TerritoryPortugal
CityLisbon
Period19/04/26 → 23/04/26
Internet address

Funding

This research is funded by UKRI (UK Research and Innovation) under the Hartree National Centre for Digital Innovation Programme (Hartree Centre Northern Ireland Hub )

Funders
UKRI/ STFC Hartree Centre

    Keywords

    • Evaluation
    • GraphRAG
    • Generative AI
    • Large language models (LLMs)

    Fingerprint

    Dive into the research topics of 'Comparative Evaluation of RAG and GraphRAG for Open-Ended Question Answering'. Together they form a unique fingerprint.

    Cite this