Multimodal AI Architecture for Operational Intelligence in IoT and Robotic Systems

The increasing availability of sensor data in industrial and robotic systems creates opportunities for decision support in maintenance, anomaly detection, and operational optimization. However, many existing approaches rely on isolated models and provide limited contextual reasoning and explainability.

Buga, Kyrylo, 2026

Art der Arbeit Master Thesis
Auftraggebende
Betreuende Dozierende Hanne, Thomas, Re, Barbara
Views: 1 - Downloads: 0
This thesis investigates the design of a multimodal AI architecture that integrates sensor data streams, operational events, and technical documentation using embedding-based retrieval (vector databases) and an LLM-based agent. The agent follows a structured workflow that retrieves relevant evidence and executes a small set of constrained analysis tools (e.g., querying sensor/event data, summarizing episodes, and retrieving SOP excerpts) to generate traceable outputs. The goal is to evaluate an operational assistant that correlates sensor streams (e.g., MQTT/ROS2 data), process events, and domain knowledge (SOPs, maintenance logs) to support incident understanding and diagnosis.
The framework is realised as a working prototype and evaluated on public benchmarks and a simulated MQTT deployment. A pre-trained time-series foundation model (MOMENT-1-large) reaches a Recall@5 of 0.955 on the C-MAPSS retrieval benchmark (edging a hand-crafted statistical baseline and separating the operationally hardest degrading regime by 9.2 points), yet a per-channel Z-score remains the stronger detector of stealthy control-system attacks on the HAI dataset (AUROC 0.881 versus 0.600 for a foundation-model novelty score); the architecture accordingly uses the embedding for similarity-driven retrieval and rule-based statistics for detection. End-to-end, the constrained tool registry and citation policy gate produce grounded, citation-backed answers: on a systematic thirteen-query assessment spanning eight operator-query categories the agent reaches a correctness of 0.85 and a grounding rate of 0.89 with zero hallucinations (both hallucination-bait prompts are correctly declined against the telemetry), the residual failures concentrating on underspecified queries that draw a conservative refusal instead of a clarifying question.
The results contribute a reproducible blueprint for AI-assisted operational intelligence in industrial and robotic environments.
Studiengang: Business Information Systems (Master)
Keywords
Vertraulichkeit: öffentlich
Art der Arbeit
Master Thesis
Autorinnen und Autoren
Buga, Kyrylo
Betreuende Dozierende
Hanne, Thomas, Re, Barbara
Publikationsjahr
2026
Sprache der Arbeit
Englisch
Vertraulichkeit
öffentlich
Studiengang
Business Information Systems (Master)
Standort Studiengang
Olten