Development and Evaluation of a University Chatbot using Deep L:earning: A RAG-Based Approach

In university systems, traditional methods of information retrieval are often found to be inefficient, leading to frustration among students and staff. This paper presents the development and evaluation of a university-specific chatbot that employs the Retrieval-Augmented Generation (RAG) approach to improve the accuracy and relevance of its responses. Unlike conventional chatbots that depend on intent classification and pre-designed system responses and conversa-tion flows, the proposed chatbot integrates Large Language Models (LLMs) with local university data, enhancing its ability to handle complex queries with con-text-aware responses and dynamically generated conversation flows. The system architecture includes components such as LangChain for orchestration, a vector store for embedding external knowledge, and a user interface developed using Streamlit. Evaluation results demonstrate that the RAG-based chatbot substan-tially outperforms traditional LLMs, including GPT-3.5, GPT-4 mini, and GPT-4, in terms of answer accuracy and reliability. In this paper we also reflect on the lessons learned during the chatbot’s development and deployment in a real-world university setting.