Recent research has shown that language model size is not directly linked to its performance.

This study aims first to analyze the efficiency of different sizes of Language Models combined with the Retrieval-Augmented Generation (RAG) technique in providing software first-level support from AMOS, an aircraft maintenance software from Swiss-AviationSoftware (Swiss-AS) and second to compare the effectiveness of two RAG chatbots: one using cloud-hosted Llama 2-7B Chat model and other using PaLM 2 text-bison model sourced through API as the generative models. An interview with three specialists was conducted to evaluate the effectiveness of both chatbots. Analysis of the responses demonstrates PaLM 2 text-bison as the generative model provides a straightforward answer, while Llama 2-7B Chat provides verbose and less focused answers.

The result indicates that the RAG technique efficiently delivers software first-level support. However, while effective, Llama 2-7B Chat does not emerge as the optimal generative model, falling short in answer accuracy when compared to the PaLM 2 text-bison model.