Bad Data Lurking in Plain Text推荐.pdfVIP

CHAPTER 4 Bad Data Lurking in Plain Text Josh Levy, PhD This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface. —Doug McIlroy Bad data is often delivered with a warning or an apology such as, “This dump is a real mess, but maybe you’ll find something there.” Some bad data comes with a more vacuous label: “This is plain text, tab-delimited. It won’t give you any trouble.” In this article, I’ll present data problems I’ve encountered while performing seemingly simple analysis of data stored in plain text files and the strategies I’ve used to get past the problems and back to work. The problems I’ll discuss are: 1. Unknown character encoding 2. Misrepresented character encoding 3. Application-specific characters leaking into plain text I’ll use snippets of Python code to illustrate these problems and their solutions. My demo programs will run against a stock install of Python 2.7.2 without any additional re‐ quirements. There are, however, many excellent Open Source libraries for text process‐ ing in Python. Towards the end of the article, I’ll survey a few of my favorites. I’ll conclude with a set of exercises that the reader can perform on publicly available data. 53 Which Plain Text Encoding? McIlroy’s advice