the babar database challenges, trends and projections.pptVIP

On the Verge of One Petabyte – the Story Behind the BaBar Database System Jacek Becla Stanford Linear Accelerator Center For the BaBar Computing Group Outline Talk will cover Our experience with running large scale DB system Achievements, issues New development and what drives it Main focus on period since last CHEP Providing Persistency for BaBar Growing complexity and demands Changing requirements Hitting unforeseen limits in many places Non-trivial maintenance Most problems are persistent-technology independent System becoming more and more distributed Very lively environment Production not as stable as one would imagine Some Numbers 750+ TB of data 0.5+ million DB files Several billion events 60+ million collections 1000+ simultaneous analysis jobs accessing DB common Data Availability is Essential PromptCalibration Rapid feedback, keeping up with detector Event Reco (ER) Data available for analysis within a week Reprocessing All data reprocessed before conferences Analysis Outages 4% driven mostly by power outages, hardware failures What Changed Since Sep01/last CHEP? Event Reconstruction 4 output physics streams ? 20 20 output streams ? 5 + 115 pointer collections Rolling calibrations now separated Runs now processed in parallel Raw and rec not persisted anymore Planning to run skim production separately What Changed Since Sep01/last CHEP? Simulation Production 1.5 ? 3 MC events per real event ~8 ? ~24 production sites Analysis Bridge federations now fully functional Significant system growth 29 data servers, 34 lock/journal servers 66TB disk space, 101 slave federations Some Challenges Setting up ER/REP in Padova All Linux based Recovery from Linux-client crashes leaving connections on server side Three data corruptions Understood and fixed – race condition: file descriptors closed/reopened incorrectly Never understood, went away after power outage (Dec02) Not sure who is at fault: Objectivity? Linux kernel? Problems with B-Tree index updates in Temporal Da