大数据问题中的技术策略.pptVIP

大数据管理与数据质量

-美国金融业中的对策汪时奇（博士）处理速度容量限制数据质量

Overview数据=Data=信息(并非数字集合)数据科学(约)=信息科学为何研究大数据?因为相关产品(如硬盘,memory,CPU等)价格指数下降因为信息爆炸因为大数据导致许多新问题大数据研究是多学科的综合(IT,DM,BI,BA,…)实业界对大数据问题的对策(见下文)

1.数据库策略1.1Database(DB)performance1.2DBspace

1.1DBperformanceAuditing–2tables:asmallactiveahugepassivePartitionIndex(good/bad;Cluster;Global/Local)Locktype(whenapplyrowlock)Transaction:1-phaseor2-phaseNormalizationInternaloptimization

(e.g.ExecutionPlan=hintinOracle)Constraints(e.g.Check)usagetoreplacetriggerTricks(e.g.Datefunction;Searchsmalltablefirst;…)

1.2DBspaceSpacearrangementforevendistribution

(e.g.1hugetableusesafewdatafiles)CleaningprocedurewithdefragmentPartitiondesignwithcleaningplan

2.Applications(软件)

(Javaexample)Usingadvancedlanguage(e.g.JavaorC#)2.1Memory（内存）2.2Disk/networkspace2.3Performance2.4Maintainability

2.1MemoryMinimizebigobjectscreationandcoexistenceGC(GarbageCollection)ornullbigobjectsonceoutofscopeChooseappropriateGCtypegc()TrytosplitonebigobjecttosmallobjectsUsemutableclassforfrequentlychangedbigobjects(e.g.StringBuilder,insteadofString)

2.2Disk/networkspaceSmartcleanandarchiveprocesses

e.g.archivezippedoldornotusedfilestolowspeednetworkspaceanddeleteveryoldfilesfromthatspaceSmartloggingsettingse.g.log4jsizerollinge.g.AvoidduplicatedortriviallogginginfoMonitorforspaces

2.3PerformanceAvoidredundanttreatment(inbigloops)

MaximizereuseMulti-threadingDBaccessingLogging--avoidslowoptions(e.g.line#)

2.4MaintainabilitySOAprinciplesLosecoupling,reusability,granularity,modularity,composability,componentization,interoperability,…JEEpatterns(DAO,DTO,BizDelegation,…)Designpatterns(23)andMVCCreationStructureBehavior(e.g.Visitor)OOPprinciplesAbstraction,encapsulation,polymorphism,…Open/Close

3.数据质量控制3.1Business3.2ProcessA.FailoverDR(DisasterRecovery)B