Entropy of Search Logs How hard is search With搜索日志的熵如何搜索与.pptVIP

Cross Validations Weekends are harder to predict Weekdays to predict weekdays weekends to predict weekdays Day time to predict day time nights to predict day time * * Partition by Day-Week (CV) Test data: weekdays and weekends Weekdays are easier to be predicted by history of weekdays Weekends are more difficult to predict Training data: weekdays and weekends * * * * * * * * There are two very different answers from the computational perspective: * * * * * * * * * * * * * * Entropy of Search Logs- How Big is the Web?- How Hard is Search? - With Personalization? With Backoff? Qiaozhu Mei?, Kenneth Church? ? University of Illinois at Urbana-Champaign ? Microsoft Research * * How Big is the Web? 5B? 20B? More? Less? What if a small cache of millions of pages Could capture much of the value of billions? Could a Big bet on a cluster in the clouds Turn into a big liability? Examples of Big Bets Computer Centers Clusters Capital (Hardware) Expense (Power) Dev (Mapreduce, GFS, Big Table, etc.) Sales Marketing Production Distribution Small * Millions (Not Billions) * Population Bound With all the talk about the Long Tail You’d think that the Web was astronomical Carl Sagan: Billions and Billions… Lower Distribution $$ ? Sell Less of More But there are limits to this process NetFlix: 55k movies (not even millions) Amazon: 8M products Vanity Searches: Infinite??? Personal Home Pages Phone Book Population Business Home Pages Yellow Pages Population Millions, not Billions (until market saturates) * It Will Take Decades to Reach Population Bound Most people (and products) don’t have a web page (yet) Currently, I can find famous people (and academics) but not my neighbors There aren’t that many famous people (and academics)… Millions, not billions (for the foreseeable future) * Equilibrium: Supply = Demand If there is a page on the web, And no one sees it, Did it make a sound? How big is the web? Should we count “silent” pages That don’t make a sound? How