HAN
10-ch03-083-124-9780123814791
2011/6/1
3:16
Page 86
#4
86
Chapter 3 Data Preprocessing
given concept may have different names in different databases, causing inconsistencies
and redundancies. For example, the attribute for customer identification may be referred
to as customer id in one data store and cust id in another. Naming inconsistencies may
also occur for attribute values. For example, the same first name could be registered as
“Bill” in one database, “William” in another, and “B.” in a third. Furthermore, you sus-
pect that some attributes may be inferred from others (e.g., annual revenue). Having
a large amount of redundant data may slow down or confuse the knowledge discov-
ery process. Clearly, in addition to data cleaning, steps must be taken to help avoid
redundancies during data integration. Typically, data cleaning and data integration are
performed as a preprocessing step when preparing data for a data warehouse. Addi-
tional data cleaning can be performed to detect and remove redundancies that may have
resulted from data integration.
“Hmmm,” you wonder, as you consider your data even further.
“The data set I have
selected for analysis is HUGE, which is sure to slow down the mining process. Is there a
way I can reduce the size of my data set without jeopardizing the data mining results?”
Data reduction obtains a reduced representation of the data set that is much smaller in
volume, yet produces the same (or almost the same) analytical results. Data reduction
strategies include dimensionality reduction and numerosity reduction.
In dimensionality reduction, data encoding schemes are applied so as to obtain a
reduced or “compressed” representation of the original data. Examples include data
compression techniques (e.g., wavelet transforms and principal components analysis),
attribute subset selection (e.g., removing irrelevant attributes), and
attribute construction
(e.g., where a small set of more useful attributes is derived from the original set).
In numerosity reduction, the data are replaced by alternative, smaller representa-
tions using parametric models (e.g., regression or log-linear models) or nonparametric
models (e.g., histograms, clusters, sampling, or data aggregation). Data reduction is the
topic of Section 3.4.
Getting back to your data, you have decided, say, that you would like to use a distance-
based mining algorithm for your analysis, such as neural networks, nearest-neighbor
classifiers, or clustering.
1
Such methods provide better results if the data to be ana-
lyzed have been
normalized, that is, scaled to a smaller range such as [0.0, 1.0]. Your
customer data, for example, contain the attributes age and annual salary. The annual
salary attribute usually takes much larger values than
age. Therefore, if the attributes
are left unnormalized, the distance measurements taken on annual salary will generally
outweigh distance measurements taken on age. Discretization and concept hierarchy gen-
eration can also be useful, where raw data values for attributes are replaced by ranges or
higher conceptual levels. For example, raw values for age may be replaced by higher-level
concepts, such as youth, adult, or senior.
Discretization and concept hierarchy generation are powerful tools for data min-
ing in that they allow data mining at multiple abstraction levels. Normalization, data
1
Neural networks and nearest-neighbor classifiers are described in Chapter 9, and clustering is discussed
in Chapters 10 and 11.
HAN
10-ch03-083-124-9780123814791
2011/6/1
3:16
Page 87
#5
3.2 Data Preprocessing: An Overview
87
discretization, and concept hierarchy generation are forms of data transformation.
You soon realize such data transformation operations are additional data preprocessing
procedures that would contribute toward the success of the mining process. Data
integration and data discretization are discussed in Sections 3.5.
Figure 3.1 summarizes the data preprocessing steps described here. Note that the pre-
vious categorization is not mutually exclusive. For example, the removal of redundant
data may be seen as a form of data cleaning, as well as data reduction.
In summary, real-world data tend to be dirty, incomplete, and inconsistent. Data pre-
processing techniques can improve data quality, thereby helping to improve the accuracy
and efficiency of the subsequent mining process. Data preprocessing is an important step
in the knowledge discovery process, because quality decisions must be based on qual-
ity data. Detecting data anomalies, rectifying them early, and reducing the data to be
analyzed can lead to huge payoffs for decision making.
Data cleaning
Data integration
Data reduction
Attributes
Attributes
A1
A2
A3
...
A126
T1
T2
T3
T4
...
T2000
Transactions
Transactions
T1
T4
...
T1456
A1
A3
...
A115
Data transformation
Ϫ2, 32, 100, 59, 48
Ϫ0.02, 0.32, 1.00, 0.59, 0.48
Figure 3.1
Forms of data preprocessing.