Data mining: analysis methods for big data

Data plays a fundamental role in the e-commerce sector. In order to optimise sales processes, many online stores work hard to diligently collect data. With the help of analysis tools, they compile numbers and values on customer behavior, products, and shopping cart information. But a vast collection of data alone doesn’t offer any added value to an online business. Those looking to optimise sales methods and increase profit need to be able to evaluate information in an expedient manner. This is where an analytical approach, called data mining, comes into play.

Data mining offers the possibility to optimise e-commerce on a scientific basis. Here, large data sets that are accrued build the basis for explanations and prognoses. Statistically prepared and neatly visualised, these methods allow operators of online stores to identify important factors required for a successful online business. To this end, data mining is used in order to:

• To divide markets into segments
• Analyse shopping cart data
• Create consumer profiles
• Set up prognoses on contract periods
• Analyse demand
• Identify errors in the purchasing process

In order to be able to extract relevant business information from large data sets, many methods have been established that are based on identifying important relationships, patterns, and trends. These methods can also be used for statistical processes.

• Outlier detection: extreme values that stand out from the rest of data are known as outliers. In data mining, outlier detection is used in order to identify atypical data sets. In practice, these data mining methods can for instance reveal credit card fraud by exposing suspicious transactions.

• Cluster analysis: clusters refer to a group of objects that, in one way or another, are similar to one another. The goal of this analysis is to segment unstructured data. To this end, algorithms are used to search for similarities in the structures of large data sets, in order to identify new clusters. In contrast to the classification process (see below), cluster analyses aim to detect new possibilities for creating groups. In cases where a data set cannot be allocated to any clusters, this can be interpreted as an outlier. A classic application for cluster analyses involves identifying user groups.

• Classification: while cluster analyses primarily aim to identify new groups, classification involves making use of predefined classes. Allocating these occurs with the help of matching characteristics from the data set. A decision tree presents a common method for automatically classifying data. For each node, a property of the object is called up. The presence of this property determines the choice of the following node. For e-commerce related purposes, this process can be used in order to divide customers into different segments.

• Association analysis: an association analysis aims to identify relationships within data sets that can be formulated as inference rules. When it comes to e-commerce, these data mining methods can be used in order to identify the correlation of individual products within shopping carts along the pattern of ‘if product A is bought, then product B will also be bought’.

• Regression analysis: regression analyses help create models that explain dependent variables through various independent variables. In practice, this means that the prognosis for a product’s sales performance can be created by correlating the product price and the average customer income level in a regression model. 

In data mining, statistical procedures are employed that make it possible to carry out a fundamentally objective analysis of available data sets. The rather subjective nature of selecting an analysis method (as well as the various algorithms and parameters) with the intention of pursuing specific goals can, however, lead to falsified results. Such effects can be evaded by outsourcing data mining processes to external service providers.

One of the most important factors when it comes to the quality of the data gained through data mining is the quality of the data basis. Representative results can generally only be gained from representative data. For this reason, data mining normally requires data sets to be processed in advance; this allows missing values and biases to be ironed out.

Finally, it’s important to note that data mining only offers results in the form of patterns and cross-connections. Answers can only first be obtained when the analysis results are interpreted with regards to previous questions and goals.