SHARKFEST 08 Foothill College March 31 - April 2, 2008.ppt

SHARKFEST 08 | Foothill College | March 31 - April 2, 2008 Protocol Analysis in a Complex Enterprise April 2nd, 2008 Hansang Bae Senior VP | Citigroup SHARKFEST 08 Foothill College March 31 - April 2, 2008 SHARKFEST 08 | Foothill College | March 31 - April 2, 2008 Challenges: As it turns out, size does matter! Citi’s branch network spans 5,000+ locations in the US Citi’s network infrastructure includes 30,000+ devices 300,000 users located in over 100 countries. Compliance/Security Quagmire It’s for your own protection, or so I’m told! Doing a full packet capture is difficult Wireshark is the only approved protocol analyzer at Citi. It dislodged past market leaders. Challenges (con’t): Capturing and Analyzing: Two pieces to the same puzzle Enormous amounts PCAP data are involved. In most cases, header analysis is adequate. Wireshark/WinPCAP is not well suited for this much volume Citi uses a commercial product for packet capturing. Working with the vendor, it took over three years of development before it was deemed “Citi-ready” Example One: Path MTU Infrastructure size makes it interesting. Very difficult problem without a proper protocol analyzer Example One: (Con’t) In depth understanding of routers and protocols were required. Usenet to the rescue! ICMP and IP.ADDR filters were key! So which side am I on in the “religious debate” about whether ICMP messages should be included in the “ip.addr” display filter? ..\..\..\Traces\Consumer\CBNA\ICMPRateLimit.pcap In retrospect, it was an easy problem to solve. Yet the sheer size made it difficult to spot. Example Two: Clock Drift MarketData driven business complains of extreme delays from UK to US. At first glance, application logs seem to confirm delays in the 200+ms delays. RTT is 70ms. Because it’s easy, let’s blame the firewall and the network! SLA tracking and further investigation of routers/switches gets us nowhere with problem resolution. Our analysis shows that something i