Evaluating the Performance of a Large Language Model (LLM) as a “Process Management Expert”

A Case Study of ChatGPT

Jäger, Florian, 2024

Type of Thesis Master Thesis
Client
Supervisor Telesko, Rainer
Views: 10 - Downloads: 4
This master thesis investigates the potential of ChatGPT-4 to perform tasks typically managed by human process management experts. The primary objective is to benchmark ChatGPT's effectiveness, exploring its strengths and limitations while evaluating its capacity to complement human expertise in business process management (BPM).
The research adopts a mixed-methods approach, combining quantitative accuracy measures with qualitative data gathered from candidate responses. Controlled experiments were conducted where ChatGPT and human experts tackled identical BPM tasks derived from real-world scenarios.
The findings reveal that ChatGPT responds rapidly and qualitatively high to well-defined, non-complex prompts. It matches or exceeds junior experts' performance in knowledge-based routine tasks but struggles with complex tasks requiring deep contextual understanding and creativity, like BPMN diagrams. ChatGPT potentially significantly enhances efficiency by automating routine tasks, allowing human experts to focus on strategic aspects benefiting from human intuition. However, its limitations in handling complex BPM tasks and optimising processes highlight the necessity of continuous human oversight. Effective interaction with ChatGPT requires iterative prompts to refine responses, ensuring task execution with minimal follow-up. Integrating ChatGPT into BPM can improve operational efficiency and innovation, provided its limitations are managed appropriately. The thesis concludes that ChatGPT-4 demonstrates significant potential as a complementary tool in BPM, capable of handling selected tasks with sufficient quality.
Studyprogram: Business Information Systems (Master)
Keywords
Confidentiality: öffentlich
Type of Thesis
Master Thesis
Authors
Jäger, Florian
Supervisor
Telesko, Rainer
Publication Year
2024
Thesis Language
English
Confidentiality
Public
Studyprogram
Business Information Systems (Master)
Location
Olten