Data collection in the context of market monitoring can either be done manually or automatically. The method of manual data collection is still widespread among retail companies. In the following, we will first discuss some downsides of manual data collection compared to automated data collection.

Manual market monitoring

In the manual collection of market information from online stores, marketplaces or price comparison portals, an employee calls up the relevant website, searches for the relevant information, copies it into further systems (e.g. Excel spreadsheet or database) and, in the final step, creates the desired analyses. This outlined path is associated with a number of tasks compared to an automated survey.

High time investment

From search to analysis - To collect relevant data, an employee must call up the corresponding website and navigate his way - often through several category levels - to the desired product. Once the employee has found the product, he must highlight the relevant information and copy it. This process requires a significant amount of time for each product, regardless of whether the data is taken from the view or page source text. Different font types and sizes cause additional effort for data preparation. The same applies to the necessary linking (= matching) of the different assortments. Even partially automated solutions such as S-references in Excel are of limited help in this regard. The last step, generating relevant analyses from the collected raw data, is again time-consuming.

Small sample size

Lack of data leads to low quality - As a direct consequence of the high time required, only a very limited sample size can be collected and analyzed as part of manual monitoring. Often, only specific focus articles are compared, leaving large assortment areas completely without a comparison. This, in turn, makes it difficult to clearly decipher the pricing strategy and product range policy of the competition.

Lack of topicality

As a general rule, manual competition monitoring is only one of several topics with which the employee in question is concerned. As a result, the employee does not get around to carrying out the competition monitoring with the necessary regularity and quality. As a result, the timeliness of the collected sample often leaves much to be desired. This in turn means a blurred view of the competition, in which, for example, old prices are taken into account or any changes in the product range are not recognized at all.

High error ratio

Damage due to repetition and monotony - when performing a market analysis manually, a number of repetitive and monotonous steps have to be taken, which increase the risk of errors. At the same time, the impact of errors, even if small, is business-critical in market monitoring. A wrong product or a wrongly quoted price can subsequently lead to wrong conclusions and decisions.

Manual effort usually means higher costs than automated execution. This applies to a considerable extent to competitive intelligence. Automated monitoring using web scrapers, implemented review mechanisms and dedicated software can collect, review, prepare, analyze and visualize many times more data in the same period of time. The use of an automated solution can thus significantly reduce the costs incurred. This is especially true in a long-term view.

Employee motivation

The pursuit of variety and meaningfulness - Many people are more motivated when they can perform varied and meaningful tasks. The regular manual gathering and processing of market information is characterized by a high level of monotony. Accordingly, the motivation to perform such a task in the long term is usually low. By using a software solution for automated collection and processing, employees have significantly more time to devote to analyzing and interpreting the collected data. This often leads to higher work motivation and valuable results that provide the company with greater benefits.

Automated market monitoring

Automation is the key to streamlining your manual processes. Automated collection for market information is done with the help of web scrapers. These are computer programs that automatically perform repetitive tasks. In concrete terms, this means in the case of competitive analysis that a web scraper calls up the relevant website and collects the desired information there at defined time intervals. Since online stores, market places, and price portals are dynamic entities, web crawlers must be constantly adapted and maintained. This should be ensured by the respective service provider as part of a joint project.