A standardized, elaborated and well-argued performance metric for benchmarking anytime algorithms on noisy optimization problems is missing in the current benchmarking frameworks. This Thesis aims at developing the very as an artefact by following the Design Science Research methodology.

Existing noise-handling practices and benchmarking guidelines are reviewed. Benchmarking experiments including a test suite containing 13 artificial test functions are conducted with two state-of-the-art algorithms to assess the performance metric.

Based on the generated data and existing literature it is concluded that combining the loss value and the runtime as performance metric maximizes reliability, expressiveness and information content considering the relevance.