One part I’ve been slowly conquering is connecting my SQLite databse to the Processing application.  My current issue is closing the database, the query just runs and runs and runs and…you get the point.  I’m using a database because these counts need to be able to adjust depending on where you click, and I figure that SQL statements are the best way to filter and aggregate the various information.

But today, in my reading from Ben Fry’s Visualizing Data, which I’m approaching the end of, Fry was discussing acquiring data.  Many of the methods I’d come across before in my own research, they focused on how to hook up the database to my interface.  Fry suggested to reconsider whether you need a database, or can use programming to manipulate a flat text file.  With the way I’ve designed this application, I need to count up the number of records where a column meets a certain condition.  I was trying to do that with a count in SQL.  But I can just have the program do a loop, and count it programatically.

Perhaps using a database is a better fit when doing joins or complicated queries, or if one has millions, or even thousands, of records.  But if performing counts, and pulling records that meet certain conditions, is all that’s necessary, right now I’m thinking that a flat text file is best.

Related posts you might enjoy:

• February 15, 2010 -- Visual confections are more than mere presentation
• February 1, 2010 -- Relating zip codes and geography using Processing
• January 10, 2010 -- Which states have been hit hardest by unemployment in the last decade?
• March 3, 2010 -- Persistence of Chicago Art Galleries
• February 28, 2010 -- Django app #2: Conquering forms and Google Maps API
• February 15, 2010 -- Visualizing networking: When it doesn’t work

Or, you know, you could use Group By in SQL.  But no need to import databases just for filtering.

Related posts you might enjoy:

• April 11, 2010 -- Data Delver: Paul Monies, The Oklahoman
• March 25, 2010 -- Self-teaching data and programming skills
• March 8, 2010 -- Data Delver: Jennifer LaFleur, ProPublica
• February 28, 2010 -- Django app #2: Conquering forms and Google Maps API
• February 21, 2010 -- Is a flat text file or a database right for an app?
• February 2, 2010 -- Recognize the importance of journalistic creativity

Enter the new reporter. He can come up with millions of questions in a millisecond, and assimilate information ten times faster than that. He doesn’t take no for an answer, and he doesn’t even need to spend travel time on his interviews. Oh, and he’s the pinnacle of efficiency – doesn’t take lunch or bathroom breaks, doesn’t even stop to sleep.  He’s the new ace reporter on staff: a computer.

Time to be afraid? No, because the computer has a serious Achilles heel — it has no flexibility, no understanding of nuance. It’s either black or white, right or wrong. The best way to work with your new colleague, so to speak, is to let him focus on his strengths, while you focus on yours. I’m talking about using the machine as an analytical tool for details, but leaving the big picture exploration to the humans.  This concept is buzzing around the Internet these days — the idea of parsing “big data” — data sets that are too big for human brains alone to comprehend.

This idea was explored in a large-scale report (66-page-PDF ahead!)  from the Aspen Institute, titled “The Promise and Peril of Big Data.”  I’m not surprised by a title like that, having long been a believer that all technology is amoral, and like much of life, brings its own good and bad aspects.  The computer can examine these large sets, but we have to use our own judgment to bring journalistic and statistical ethics — just because there is a correlation, that doesn’t mean there is a cause and effect, for example.  But just as we go to expert sources to fill in our own knowledge gaps, the computer serves as a certain type of expert that can aid reporting.

Here’s the computer’s specialty:  It helps us analyze information, and then display it so it’s fit for human consumption. This is the essence of journalism, it’s just that the computer can process more stuff than the human brain can.  But we must make sense of it.  He’ll take care of the trees, we have to deal with the forest.

As for statistical analysis, it’s been big in the business community for years. The idea is simple: Parse the information so it can help people make intelligent decisions. The field is progressing rapidly, and the community is talking about using data for storytelling. Roger Magoulas, director of research for O’Reilly Media, recently said, “You need more sophisticated ways of distilling what you know down. A good visualization, if you turn something into a good story, it’s just going to resonate a lot more…than a simple chart that gets delivered to everyone.”

Wait, isn’t that what journalists do? Only difference is that instead of making data accessible to a few decision makers, it’s about making data accessible to a bigger group of people.  Many call it data analysis, I call it opening up democracy.

Related posts you might enjoy:

• January 30, 2010 -- Parallelism: Packing information into visualization