近红外分析的变量选择—梁逸曾.pptVIP

梁逸曾中南大学化学化工学院 College of Chemistry and Chemical Engineering, Central South University, China The original NIR spectra of corn moisture High-Dimensional Problems: p N 在分析之前，粗看这些不同样本的光谱就明显可知，它们不但相似度很高，而且每个样本所对应的变量数，即光谱的波长数目（或波数点数）都大大超过样本数，即m（变量数）n（样本数）。对于这些特点，我们可以简单地得出两个结论：1）光谱相似度很高，即这些光谱的共线性很严重，其化学秩很难准确确定(导致很难使用最小二乘回归)；2）模型变量数大大超过样本数，其过拟合风险很高。 共线性问题： 一个近红外或拉曼光谱大都有上千个波长或波数点，此数目一般都比样本数要多，致使矩阵（XtX）严重亏秩，很难使用常规最小二乘来求解。 回归模型的过拟合 这样就出现了PCR和PLS，解决校正矩阵（XtX）严重亏秩的问题；采用cross-validation解决过拟合问题； 从数学的角度来看？ 1. PLS解决了所有的问题吗？ 2. 共线性问题？ 3. 特征变量（潜变量）解决了变量选择的问题吗？ 4. High-Dimensional Problems: p N 5. ∑Cpi=2p-1；NP难题，不容易 6. 常规变量选择方法基本无效！ Explanation from Statistics 从化学的角度来看： Overfitting happened from PLS-LDA Recently in the 11th Scandinavian Symposium on Chemometrics (SSC11), Professor Bro performed a live simulation experiment and got very counter-intuitive result. He first randomly produced a data matrix of size 100×200. Then 50 samples are also randomly chosen and assigned the class label ‘0’, while the remaining 50 samples are assigned the class label ‘1’. Finally, partial least squares linear discriminant analysis (PLSLDA) [2-4] was employed to classify the two classes of samples. Overfitting happened from PLS-LDA Surprisingly, for each replicate simulation, the two classes of samples are classified very well. The undoubted fact that randomly produced binary classification data with no between-class difference can be classified very well greatly aroused the interests of the SSC11 participants. What’s the philosophy of the classification? Is it a pitfall or is there anything we didn’t pay much attention to [5, 6]? Is there anything connecting with the overfitting, since, in his example, the number of variables (200) is much larger than that of samples (100). Overfitting happened from PLS-LDA First let us see an interesting comparison. The organisation committee of the symposium Chimiometrie 2004 held in Paris propos