The goal was to create a crawler prototype which visits websites based on search terms and extracts information from web pages. By applying Natural Language Processing (NLP) and sentiment analysis, the prototype will provide help in decision-making.

The team used an agile approach to create a web app consisting of a frontend and a backend which carries out web-searching and data evaluation. The prototype uses NLP techniques and sentiment analysis using RapidMiner and MeaningCloud.

Scouti allows the user to search for keywords (e.g., antivirus) and select the industry of interest (e.g., healthcare). Scouti interface is simple and effective. After sending the query and receiving the links from the Google Search API, Scouti processes the data and the user is given an overview of suitable software packages that were found on the web. The technical documentation describing the prototype discusses further potential applications. The prototype shows the viability of NLP applications in search automation. As one of the customer interests was the feasibility of NLP in the evaluation of new technologies, we show that this solution is viable and scalable and can be adapted to other topics.