Beautiful Data

I’ve just finished reading Beautiful Data (published by O’Reilly in 2009), a collection of essays edited by Toby Segaran and Jeff Hammerbacher. The 20 essays from 39 contributors address a diverse array of topics relating to data and how it’s collected, analysed and interpreted.

Since this is a collection of essays, the writing style and level of technical detail varies considerably between chapters. To be honest, I didn’t find every chapter absolutely riveting, but I generally came away from each of them having learned a thing or two. Below is a list of chapter titles with occasional comments.

1. Seeing Your Life in Data
   Nathan Yau writes about personal data collection, highlighting your.flowingdata which is a Twitter app for gathering personal data. Although I am keenly interested in the data logged by my Garmin 910XT, I don’t think that I have the discipline to tweet every time I go out for a run. Regardless though, it’s a cool idea.
2. The Beautiful People: Keeping Users in Mind When Designing Data Collection Methods
3. Embedded Image Data Processing on Mars
   Instruments on planetary probes operate under significant technological constraints. It’s fascinating to learn the details behind the imaging system on the Phoenix Mars lander.

4. Cloud Storage Design in a PNUTShell

5. Information Platforms and the Rise of the Data Scientist
   Jeff Hammerbacher along with DJ Patil coined the term “Data Scientist”. The chapter starts with a story of how 17 year old Hammerbacher was fired from his job as a cashier in a grocery store and ends with the creation of the role “Data Scientist” at Facebook, reflecting the various and disparate tasks currently undertaken by people working intensively with data.

   By decoupling the requirements of specifying structure from the ability to store data and innovating on APIs for data retrieval, the storage systems of large web properties are starting to look less like databases and more like dataspaces. Beautiful Data, p. 83

6. The Geographic Beauty of a Photographic Archive
7. Data Finds Data
8. Portable Data in Real Time Jud Valeski describes the evolution of APIs for public access to data. Having spent a lot of time recently messing around with the APIs for Twitter and Quandl, this was interesting stuff.
9. Surfacing the Deep Web
10. Building Radiohead’s House of Cards I’ve watched the video for Radiohead’s House of Cards a few times before and thought that it was a cool concept. Now that I know what went into making the video (courtesy of this chapter), my appreciation has gone up a number of notches. The authors explain in appreciable detail how they gathered data using LIDAR systems and used processing to generate the aetherial animations in the video.
    Although at the time of publishing some of the data for the video were released as open source, it appears to have subsequently been withdrawn. That’s a pity. I think I would have enjoyed hacking on that. And it would have been good motivation to learn more about processing.
11. Visualizing Urban Data
12. The Design of Sense.us
13. What Data Doesn’t Do Coco Krumme provides a somewhat dissenting view, writing about the limitations of data. She reminds us that a naive interpretation of statistics can be very misleading; that more data is not always better data; that data alone do not provide explanations; and that even good models have limitations.
14. Natural Language Corpus Data This is probably the most technical chapter in the book. Peter Norvig gives a tutorial on Natural Language Processing (NLP) with sample code in Python. He certainly provides enough information to get you up and running with NLP. He also points out a number of potential gotchas and ways to get around them.
15. Life in Data: The Story of DNA I’m not sure why, but this snippet of code really amused me. Great way of capturing an obscure biological fact in a form which resonates with my inner geek.

    char(3*10^9) human_genome;
    

16. Beautifying Data in the Real World
17. Superficial Data Analysis: Exploring Millions of Social Stereotypes
    The authors write about processing the data gathered at FaceStat.com, providing numerous handy snippets of R code. Although the FaceStat web site has been discontinued, the principles of their analysis will find applications elsewhere.
18. Bay Area Blues: The Effect of the Housing Crisis
19. Beautiful Political Data
20. Connecting Data This chapter addresses the issue of connecting data from disparate sources. Here I found something that is going to be of immediate use to me: Collective Entity Resolution. It appears that this algorithm will solve a problem I have grappled with for a few months. This bit of information alone made the book a worthwhile read.

You’re not going to learn the details of any new technical skills from this book. But you will definitely uncover many inspiring thoughts and ideas. And you’ll probably find a gem or too, just like I did.