Exploring the Capabilities and Limitations of ChatGPT for Java Code Generation

This master's thesis investigates the use of generative artificial intelligence (AI), focussing on ChatGPT (GPT-3.5), to explore its potential to increase productivity in software development through the generation of Java code. As a relatively novel technology, ChatGPT, released in November 2022, represents a disruptive example of generative AI capable of generating diverse content, including Java code.

Schädler, Luca, 2023

Art der Arbeit Master Thesis
Auftraggebende
Betreuende Dozierende Scherb, Christopher
Keywords AI, generative AI, OpenAI, ChatGPT, Java, Code quality, Code security
Views: 13 - Downloads: 2
The study begins with a contextualisation of generative AI in the current landscape, acknowledging its position in the Gartner Hype Cycle for AI and recognising the associated risks in the "Trough of Disillusionment" phase. The research problem focuses on the critical evaluation of code quality and security, particularly for Java code generation, a relatively unexplored area in the literature. The thesis is that while ChatGPT can contribute to developer productivity by generating Java code, the inherent code quality and security challenges may not meet corporate standards. The main research question investigates the weaknesses of the Java code generated by ChatGPT in terms of code quality and security.
The study follows a pragmatic research philosophy that utilises an abductive approach and a mixed methods strategy. The design science research methodology is applied in five phases: problem awareness, suggestion, development, evaluation and conclusion. The artefact developed in this process is a list of identified quality and security issues in the Java code generated by ChatGPT. The analysis is based on experiments with two cases, "Fractionator" and "PlayerCardApp", which reveal notable code validity and maintenance issues. The code security analysis identifies weaknesses in the robustness of the application, leading to application crashes.
In conclusion, the thesis shows that ChatGPT has several weaknesses in code generation that affect quality and security metrics. The provided artefact, a list of identified issues, serves as a tool for companies and software developers to evaluate the suitability of ChatGPT-generated code for company standards. The study contributes to closing the gap in the literature on the quality and security of Java programming languages. The discussion section reflects on the challenges encountered during the work and confirms the initial assumptions about code problems. Ultimately, the results suggest that ChatGPT can support a software engineer. However, its role in code generation may not lead to a significant productivity gain but rather to a redistribution of effort and time in software development.
Studiengang: Business Information Systems (Master)
Vertraulichkeit: öffentlich
Art der Arbeit
Master Thesis
Autorinnen und Autoren
Schädler, Luca
Betreuende Dozierende
Scherb, Christopher
Publikationsjahr
2023
Sprache der Arbeit
Englisch
Vertraulichkeit
öffentlich
Studiengang
Business Information Systems (Master)
Standort Studiengang
Olten
Keywords
AI, generative AI, OpenAI, ChatGPT, Java, Code quality, Code security