Chapter 15ASSESSMENT OF DATA MODELS.pptVIP

Chapter 15ASSESSMENT OF DATA MODELS Outline Introduction Models, Their Assessment and Selection The Bias-Variance Dilemma Simple Split and Cross-Validation Bootstrap Occam’s Razor Heuristic Minimum Description Length Principle Akaike’s Information Criterion and Bayesian Information Criterion Sensitivity, Specificity and ROC Analyses Interestingness Criteria Introduction Assessment of a data model is done by a data miner before selecting and presenting the model to the user. The user then makes a final decision whether to accept the model in its current form, and use it for some purpose, or ask to generate a new one (a frequent outcome) . The user’s expectation of a DM project is to find some new information/knowledge, hidden in the data, so that it can be used for some advantage. Introduction Problem: A data miner assesses quality of the generated model often by using the same data that were used to generate the model itself (albeit divided into training and testing data sets), while the owner/user depends not only on data and DM results but also on his deep (expert) domain knowledge. In spite of the KDP requirement that the data miner learns about the domain and the data as much as possible, his or her knowledge obviously constitutes only a subset of the knowledge of the experts (data owners). Introduction In practice: A data miner generates several models of the data and must decide which one is “best” in terms of how well it explains the data and/or its predictive power, before presenting it to the data owner. We will discuss the methods for selecting “best” models. When a data miner selects “best” model, such an undertaking is called model selection. The data owner utilizes his domain knowledge for model assessment. Data miners attempt to do the same by coming up with artificial measures, called interestingness. We focus mainly on the heuristic, data-reuse (data re-sampling), and analytic methods for model selection and validation. I