To Err Is Human….pptVIP

To Err Is Human… Chuck Thompson Manager, CS Technology Services Group Outline General Mistakes Environment Matters Real World Problems And Their Causes Detecting Security Incidents And Problems Final Thoughts General Mistakes Typos Cut and Paste Line Wraps (or lack of) Wrong Window Right Action, Wrong Location Incorrect Permissions Incomplete Changes Inadequate Preparation Environment Matters Epoch Redisplay Port to Lucid Emacs Software Distribution X11 Build Circa 1991 TeX/LaTeX PATH and TEXINPUTS Binary Path Differences Between OSes Crontab vs. Login Shell Environment It is very difficult to properly recreate production environments in test environments. Real World Problems And Their Causes All situations and incidents to be described really happened They all happened in or to this department Names may or may not be changed to protect the guilty Networking Related Problems UIUCnet 12 hour outage on August 27 Filtering of dcs-server1 on September 7 ServerIron MAC address corruption Router ACLs bug in early 2004 Switch uplink module problems in 2002 Cut-and-Paste ACL update failure DHCP server not running… due to a missing semicolon OS Bug / Patching Related Problems Primary CS server crashing due to DHCP-tickled bug Y2K induced problems (3 bad Solaris patches) BIND security patch (or lack of) from Sun Broken binaries after patching, before reboot DCL elock servers patching incident Services Startup Problems missing init scripts / links / chkconfig settings httpd.pid file not removed before reboot SSL-capable httpd startup with password protected SSL certificates /etc/init.d/sendmail vs. /etc/init.d/sendmail-dcs Disk Space Related Problems FA05 class websites server /var filled up in less than a week: 3GB of debugging info corrupted alias distribution problems Bluestem login failures Runaway programs doing excessive logging favorite of CS grad students ? /home/student out-of-space in April 2005 Mail Related Problems @ email handler queue backup in March 2005