Development and Evaluation of a University Chatbot Using Deep Learning: A RAG-Based Approach

Kabir Olawore; Michael McTear; Y Bi

doi:10.1007/978-3-031-88045-2_7

Development and Evaluation of a University Chatbot Using Deep Learning: A RAG-Based Approach

Kabir Olawore
, Michael McTear
, Y Bi

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

In university systems, traditional methods of information retrieval of-
ten prove inefficient, leading to frustration among students and staff. This paper
presents the development and evaluation of a university-specific chatbot that em-
ploys the Retrieval-Augmented Generation (RAG) approach to improve the ac-
curacy and relevance of its responses. Unlike conventional chatbots that depend
on intent classification and pre-designed system responses and conversation
flows, the proposed chatbot integrates Large Language Models (LLMs) with lo-
cal university data, enhancing its ability to handle complex queries with context-
aware responses and dynamically generated conversation flows. The system ar-
chitecture includes components such as LangChain for orchestration, a vector
store for embedding external knowledge, and a user interface developed using
Streamlit. Evaluation results demonstrate that the RAG-based chatbot substan-
tially outperforms traditional LLMs, including GPT-3.5, GPT-4 mini, and GPT-
4, in terms of answer accuracy and reliability. In this paper we also reflect on the
lessons learned during the chatbot’s development and deployment in a real-world
university setting

Original language	English
Title of host publication	Chatbots and Human-Centered AI
Subtitle of host publication	Conference Proceedings
Pages	96-111
Number of pages	16
ISBN (Electronic)	978-3-031-88045-2
DOIs	https://doi.org/10.1007/978-3-031-88045-2_7
Publication status	Published (in print/issue) - 4 Apr 2024
Event	CONVERSATIONS 2024 - Thessaloniki , Greece Duration: 4 Dec 2024 → 5 Dec 2024

Publication series

Name	Lecture Notes in Computer Science
Publisher	Springer
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	CONVERSATIONS 2024
Abbreviated title	CONVERSATIONS 2024
Country/Territory	Greece
Period	4/12/24 → 5/12/24

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 4 Quality Education

Keywords

Chatbot
large language models
retrieval-augmented generation

Access to Document

10.1007/978-3-031-88045-2_7

Embargoed Document

Sedimentology - 2025 - Garner - Sedimentology and geomorphology of tufa barrages on rock coast shore platforms
Author Accepted Manuscript, 18.1 MB
Embargo ends: 4/04/26

Cite this

@inproceedings{23b66e9c4624444ea3b299c4509becf4,

title = "Development and Evaluation of a University Chatbot Using Deep Learning: A RAG-Based Approach",

abstract = "In university systems, traditional methods of information retrieval of- ten prove inefficient, leading to frustration among students and staff. This paper presents the development and evaluation of a university-specific chatbot that em- ploys the Retrieval-Augmented Generation (RAG) approach to improve the ac- curacy and relevance of its responses. Unlike conventional chatbots that depend on intent classification and pre-designed system responses and conversation flows, the proposed chatbot integrates Large Language Models (LLMs) with lo- cal university data, enhancing its ability to handle complex queries with context- aware responses and dynamically generated conversation flows. The system ar- chitecture includes components such as LangChain for orchestration, a vector store for embedding external knowledge, and a user interface developed using Streamlit. Evaluation results demonstrate that the RAG-based chatbot substan- tially outperforms traditional LLMs, including GPT-3.5, GPT-4 mini, and GPT- 4, in terms of answer accuracy and reliability. In this paper we also reflect on the lessons learned during the chatbot{\textquoteright}s development and deployment in a real-world university setting",

keywords = "Chatbot, large language models, retrieval-augmented generation",

author = "Kabir Olawore and Michael McTear and Y Bi",

year = "2024",

month = apr,

day = "4",

doi = "10.1007/978-3-031-88045-2\_7",

language = "English",

isbn = "978-3-031-88044-5",

series = "Lecture Notes in Computer Science",

publisher = "Springer",

pages = "96--111",

booktitle = "Chatbots and Human-Centered AI",

note = "CONVERSATIONS 2024, CONVERSATIONS 2024 ; Conference date: 04-12-2024 Through 05-12-2024",

}

TY - GEN

T1 - Development and Evaluation of a University Chatbot Using Deep Learning: A RAG-Based Approach

AU - Olawore, Kabir

AU - McTear, Michael

AU - Bi, Y

PY - 2024/4/4

Y1 - 2024/4/4

N2 - In university systems, traditional methods of information retrieval of- ten prove inefficient, leading to frustration among students and staff. This paper presents the development and evaluation of a university-specific chatbot that em- ploys the Retrieval-Augmented Generation (RAG) approach to improve the ac- curacy and relevance of its responses. Unlike conventional chatbots that depend on intent classification and pre-designed system responses and conversation flows, the proposed chatbot integrates Large Language Models (LLMs) with lo- cal university data, enhancing its ability to handle complex queries with context- aware responses and dynamically generated conversation flows. The system ar- chitecture includes components such as LangChain for orchestration, a vector store for embedding external knowledge, and a user interface developed using Streamlit. Evaluation results demonstrate that the RAG-based chatbot substan- tially outperforms traditional LLMs, including GPT-3.5, GPT-4 mini, and GPT- 4, in terms of answer accuracy and reliability. In this paper we also reflect on the lessons learned during the chatbot’s development and deployment in a real-world university setting

AB - In university systems, traditional methods of information retrieval of- ten prove inefficient, leading to frustration among students and staff. This paper presents the development and evaluation of a university-specific chatbot that em- ploys the Retrieval-Augmented Generation (RAG) approach to improve the ac- curacy and relevance of its responses. Unlike conventional chatbots that depend on intent classification and pre-designed system responses and conversation flows, the proposed chatbot integrates Large Language Models (LLMs) with lo- cal university data, enhancing its ability to handle complex queries with context- aware responses and dynamically generated conversation flows. The system ar- chitecture includes components such as LangChain for orchestration, a vector store for embedding external knowledge, and a user interface developed using Streamlit. Evaluation results demonstrate that the RAG-based chatbot substan- tially outperforms traditional LLMs, including GPT-3.5, GPT-4 mini, and GPT- 4, in terms of answer accuracy and reliability. In this paper we also reflect on the lessons learned during the chatbot’s development and deployment in a real-world university setting

KW - Chatbot

KW - large language models

KW - retrieval-augmented generation

U2 - 10.1007/978-3-031-88045-2_7

DO - 10.1007/978-3-031-88045-2_7

M3 - Conference contribution

SN - 978-3-031-88044-5

T3 - Lecture Notes in Computer Science

SP - 96

EP - 111

BT - Chatbots and Human-Centered AI

T2 - CONVERSATIONS 2024

Y2 - 4 December 2024 through 5 December 2024

ER -

Development and Evaluation of a University Chatbot Using Deep Learning: A RAG-Based Approach

Abstract

Publication series

Conference

UN SDGs

Keywords

Access to Document

Embargoed Document

Fingerprint

Cite this