Evaluating the Performance of a Large Language Model (LLM) as a “Process Management Expert”
A Case Study of ChatGPT
Jäger, Florian, 2024
Type of Thesis Master Thesis
Client
Supervisor Telesko, Rainer
Views: 10 - Downloads: 4
This master thesis investigates the potential of ChatGPT-4 to perform tasks typically managed by human process management experts. The primary objective is to benchmark ChatGPT's effectiveness, exploring its strengths and limitations while
evaluating its capacity to complement human expertise in business process management (BPM).
The research adopts a mixed-methods approach, combining quantitative accuracy measures with qualitative data gathered from candidate responses. Controlled experiments were conducted where ChatGPT and human experts tackled identical BPM
tasks derived from real-world scenarios.
The findings reveal that ChatGPT responds rapidly and qualitatively high to well-defined, non-complex prompts. It matches or exceeds junior experts' performance in knowledge-based routine tasks but struggles with complex tasks requiring deep contextual understanding and creativity, like BPMN diagrams. ChatGPT potentially significantly enhances efficiency by automating routine tasks, allowing human experts to focus on strategic aspects benefiting from human intuition. However, its limitations in handling complex BPM tasks and optimising processes highlight the necessity of continuous human oversight. Effective interaction with ChatGPT requires iterative prompts to refine responses, ensuring task execution with minimal follow-up. Integrating ChatGPT into BPM can improve operational efficiency and innovation, provided its limitations are managed appropriately. The thesis concludes that ChatGPT-4 demonstrates significant potential as a complementary tool in BPM, capable of handling selected tasks with sufficient quality.
Studyprogram: Business Information Systems (Master)
Keywords
Confidentiality: öffentlich