A “Big Data” Study of Microstructural Volatility in Futures Markets.pdfVIP

6 A “Big Data” Study of Microstructural Volatility in Futures Markets Kesheng Wu, E. Wes Bethel, Ming Gu, David Leinweber, Oliver Rübel Lawrence Berkeley National Laboratory Factors such as electronic exchanges, decimalisation of stock prices and automated order slicing created an explosion in the amount of financial data in the first decade of the 21st century, and the number of trades per day has been increasing dramatically. A large portion of the trades happen near opening or closing of the trading day, which creates very high rates of trading activities in short bursts. This high data rate and even higher burst rate make it difficult to understand the market. Many researchers have argued that a better understanding of high-frequency trading, and better regulations, might have prevented events such as the US flash crash of May 6, 2010 (Easley et al 2011b; Menkveld and Yueshen 2013). However, academic researchers and government regulators typically lack the computing resources and the software tools to work with large vol- umes of data from high-frequency markets. We believe that the exist- ing investments in high-performance computing resources for sci- ence could effectively analyse financial market data. In this chapter, we use the concrete task of computing a leading indicator of market volatility to demonstrate that a modest machine could analyse over 100 GB of futures contracts quickly. Scientific research activities such as the Large Hadron Collider1 and climate modelling2 produce and analyse petabytes (1015 bytes) of data. The data rates from such activities are in fact multiple orders of magnitude higher than those from financial markets. In most cases, scientists are conducting their analysis tasks not