Evaluating the Performance of a Large Language Model (LLM) as a “Process Management Expert”

A Case Study of ChatGPT

Jäger, Florian, 2024

Art der Arbeit Master Thesis
Auftraggebende
Betreuende Dozierende Telesko, Rainer
Keywords
Views: 4 - Downloads: 0
This master thesis investigates the potential of ChatGPT-4 to perform tasks typically managed by human process management experts. The primary objective is to benchmark ChatGPT's effectiveness, exploring its strengths and limitations while evaluating its capacity to complement human expertise in business process management (BPM).
The research adopts a mixed-methods approach, combining quantitative accuracy measures with qualitative data gathered from candidate responses. Controlled experiments were conducted where ChatGPT and human experts tackled identical BPM tasks derived from real-world scenarios.
The findings reveal that ChatGPT responds rapidly and qualitatively high to well-defined, non-complex prompts. It matches or exceeds junior experts' performance in knowledge-based routine tasks but struggles with complex tasks requiring deep contextual understanding and creativity, like BPMN diagrams. ChatGPT potentially significantly enhances efficiency by automating routine tasks, allowing human experts to focus on strategic aspects benefiting from human intuition. However, its limitations in handling complex BPM tasks and optimising processes highlight the necessity of continuous human oversight. Effective interaction with ChatGPT requires iterative prompts to refine responses, ensuring task execution with minimal follow-up. Integrating ChatGPT into BPM can improve operational efficiency and innovation, provided its limitations are managed appropriately. The thesis concludes that ChatGPT-4 demonstrates significant potential as a complementary tool in BPM, capable of handling selected tasks with sufficient quality.
Studiengang: Business Information Systems (Master)
Vertraulichkeit: öffentlich
Art der Arbeit
Master Thesis
Autorinnen und Autoren
Jäger, Florian
Betreuende Dozierende
Telesko, Rainer
Publikationsjahr
2024
Sprache der Arbeit
Englisch
Vertraulichkeit
öffentlich
Studiengang
Business Information Systems (Master)
Standort Studiengang
Olten