Exploring the Use of the Retrieval-Augmented Generation for First-Level Software Support: a comparison between PaLM 2 and Llama 2-7B Chat models

With the advanced Large Language Model and modern Natural Language Processing, different industries are looking at how to use it better to automate their process. However, the cost and complexity of fine-tuning a Large Language Model might concern small to medium organizations.

Mourão, Edgard A. Jr., 2023

Art der Arbeit Master Thesis
Auftraggebende
Betreuende Dozierende Martin, Andreas, Pande, Charuta
Keywords LLM, FLM, RAG, Embedding, Retriever, Generative Model, Chatbot, Software first-level support
Views: 5
Recent research has shown that language model size is not directly linked to its performance.
This study aims first to analyze the efficiency of different sizes of Language Models combined with the Retrieval-Augmented Generation (RAG) technique in providing software first-level support from AMOS, an aircraft maintenance software from Swiss-AviationSoftware (Swiss-AS) and second to compare the effectiveness of two RAG chatbots: one using cloud-hosted Llama 2-7B Chat model and other using PaLM 2 text-bison model sourced through API as the generative models. An interview with three specialists was conducted to evaluate the effectiveness of both chatbots. Analysis of the responses demonstrates PaLM 2 text-bison as the generative model provides a straightforward answer, while Llama 2-7B Chat provides verbose and less focused answers.
The result indicates that the RAG technique efficiently delivers software first-level support. However, while effective, Llama 2-7B Chat does not emerge as the optimal generative model, falling short in answer accuracy when compared to the PaLM 2 text-bison model.
Studiengang: Business Information Systems (Master)
Vertraulichkeit: öffentlich
Art der Arbeit
Master Thesis
Autorinnen und Autoren
Mourão, Edgard A. Jr.
Betreuende Dozierende
Martin, Andreas, Pande, Charuta
Publikationsjahr
2023
Sprache der Arbeit
Englisch
Vertraulichkeit
öffentlich
Studiengang
Business Information Systems (Master)
Standort Studiengang
Olten
Keywords
LLM, FLM, RAG, Embedding, Retriever, Generative Model, Chatbot, Software first-level support