Michelle Minkoff » processing

Is a flat text file or a database right for an app?

Michelle Minkoff — Sun, 21 Feb 2010 18:08:01 +0000

I’ve been busily coding away on my final project for my data visualization independent study. I’m looking at data that I’ve been slowly acquiring on the life of art galleries that were in Chicago in 1990. Back then, the Chicago Artists’ Coalition had a complete listing in a pamphlet (more like a book) they put out for artists. It included galleries within the city, in the metro Chicago area, and even downstate. I’m choosing to focus on ones that were within Chicago’s city limits in 1990, and I’m looking into what happened to them. Ideally, the visualization will use the principle of a bubble chart to show how many galleries, of those 96, existed in 1990, 1995, 2000, 2005 and 2010. You will also be able to filter by zip code, which sheds light on what neighborhood housed galleries that have lasted the longest, and perhaps more interestingly, by medium, showing how these statistics change if you look at galleries only specializing in paintings, or only in sculptures.

One part I’ve been slowly conquering is connecting my SQLite databse to the Processing application. My current issue is closing the database, the query just runs and runs and runs and…you get the point. I’m using a database because these counts need to be able to adjust depending on where you click, and I figure that SQL statements are the best way to filter and aggregate the various information.

But today, in my reading from Ben Fry’s Visualizing Data, which I’m approaching the end of, Fry was discussing acquiring data. Many of the methods I’d come across before in my own research, they focused on how to hook up the database to my interface. Fry suggested to reconsider whether you need a database, or can use programming to manipulate a flat text file. With the way I’ve designed this application, I need to count up the number of records where a column meets a certain condition. I was trying to do that with a count in SQL. But I can just have the program do a loop, and count it programatically.

Perhaps using a database is a better fit when doing joins or complicated queries, or if one has millions, or even thousands, of records. But if performing counts, and pulling records that meet certain conditions, is all that’s necessary, right now I’m thinking that a flat text file is best.

Visualizing networking: When it doesn’t work

Michelle Minkoff — Tue, 16 Feb 2010 03:10:17 +0000

So much in journalism and in life, we strive for perfection in our work. The best story idea, the best presentation. We strive for perfection as individuals, we compete to be the best, get to the scene before someone else, write tighter, edit faster. But sometimes, it’s just as important to recognize when something just didn’t work. I think that’s still an achievement, the real problem is if you think succeeded, when in reality you have failed your audience. That was my lesson of the week.

Social networking analysis is a topic that my independent study advisor Rich Gordon and I have been discussing often during our weekly meetings and through materials he’s pointed me to, as well as my own reading and observation, I think it has some great potential in terms of journalistic applications. Whether it’s about getting a job or pushing your political agenda, so much in this world is about who you know.

That’s why I was so excited that this week’s programming exercise in Ben Fry’s Visualizing Data book, which I’m using to learn Processing, focused on visualizing networks. The chapter focused on showing these connections between adjacent words in the text, and proving how it was not an effective visualization technique. I couldn’t agree more – this was the wrong application for it.

The book mapped the words in Mark Twain’s Huckleberry Finn, but because I don’t like replicating exact projects when I can avoid it, I decided to hop over to Project Gutenberg and find something more interesting. Upon searching for journalism, because why not, I found a book on “Journalism and Women,” written by a man. This piece is worth a post of its own, in trying to supposedly help women. Parts are condescending, and some advice is more relevant today than ever, but for both genders. Today’s subject isn’t the book’s content as a whole, but the connections between words to see if that helped to illuminate certain points. Hint: It didn’t, because this was a failed experiment.

But I learned the skills of how to use a network map to visualize connections, which could be valuable in the future. It’s really difficult to make sense of thousands, and even hundreds of nodes. I think this type of visualization demands a strong interactive component so the user can make sense of what’s happening, and a filtering mechanism would be even better.

As I was writing this, Rich sent me a link to a Slate visualization of how news stories connect to each other — “News Dots”. Great content, and the interface works, using different colors for different categories, having many different levels of filtering and exploration. You can even go back and look at other days. The more you roll over, the more you click, the more you want to explore, but I can understand what’s important at a glance. I just got sidetracked for 20 minutes in deep exploration. Which is the mark of good work, in my eyes.

Back to my visualization. Here’s the final product, which I coded to be output as a PDF, it looks like a jumbled mess. The bigger the bubble, the more time the word appears. And there’s still so many common words with no meaning that the visualization is started to represent how many times “than” was used, utterly unhelpful. (There’s 126 pages worth of connections by the way, when you force the program to output them, and that’s eliminating duplicate nodes.) A user can’t get much out of this, no longer how long he or she stares at it, other than the fact that there are a lot of complicated connections.

The book then instructed us to try to make more sense of the information by bringing it into the open source Graphviz program. It looks at and sorts the nodes hierarchically. The program froze on my 126 pages of connections. When I limited it to 10 pages, the first two chapters of my book, it produced this rather complicated gif file — which has a significantly long load time.

It makes a little more sense, but frankly not much, but at least you can see the individual nodes. And if you zoom in, you can scroll around, which is an interesting way to provide interactivity within a very large static graphic. But this is nowhere near ready for professional publication.

This section has taught me a lot about the importance of optimizing complicated visualizations, probably by coding it so only some parts load at once.

The chapter also took a look at using networking the right way by looking at a Web server and instead of individual nodes letting the data create a shape. The shape depicts a network, but without the same amount of detail. It works because it doesn’t overwhelm the reader. But being as I have no Web server log of my own – I have to dig into the innards of my GoDaddy site, or I might be able to do it with my Django server once I get a project up and running in the coming week or so – I opted to treat it as an interesting reading exercise, but not one to put into practice, for now anyway.

Treemapping Gov. Quinn’s State of the State

Michelle Minkoff — Tue, 09 Feb 2010 16:04:19 +0000

This week in the Processing book, I learned all about trees and hierarchies. There’s a lot of potential here for allowing the user to delve deeper into interactives by providing multiple layers. This is a very cool example of how programming helps support my theory of journalism — the deeper the information you offer, the more the user has to explore. You’re conveying knowledge, and if you’re holding interest, you’re keeping eyeballs on the site. But treemapping is most valuable as a reporting tool, I think, making structured data out of unstructured text.

This example project is a static map of Gov. Quinn’s State of the State address. The bigger the square containing the word, the more times it appeared. I told the program not to include “the,” “an,” “a”, or “of”, because then it’s just a map of large squares of articles. But the more articles you tell it to delete, the more you find more of them. Note to self: Make a list of all articles to exclude from this sort of thing in the future.

I thought it was interesting how few times the word “state” actually came up in the speech. It’s important to remember that even when we see words contained within small boxes, they are still fairly significant. The insignificantly small words are those contained in boxes that are too small to contain letters.

It would be an even better project if you could click on the small boxes to zoom in, in the style of that fabulous NY Times budget interactive from a little while back.

I also made a dynamic treemap that details the structure of a folder on your file system. The dynamic project is of course more interesting, but the way it works, it depends on being loaded on your computer, so I’m holding off from posting it online.

Relating zip codes and geography using Processing

Michelle Minkoff — Mon, 01 Feb 2010 23:54:08 +0000

This week I tackled recreating Ben Fry’s Zipdecode project, which he gives great step-by-step instructions for in his Visualizing Data book that I have been following along with this quarter.

It’s an interesting take on the concept of the scatterplot, even before using its interactive features, it asserts its usefulness as a population density map. Each dot represents a different zip code, and the more zip codes an area requires, the greater its population. This visualization isn’t news per se, but could be a valuable reporting tool, helping journalists to understand how visualizations are geographically related, and helping them find the precise location of where a certain source is located.

I take no creative credit for the project that emerged below. But I did learn a few things about the Processing language and coding that I thought might be useful to note. Probably more useful as a note to self than anything else, but here goes.

If you’re unable to find a class, you may think it’s because you are missing a bracket within that class. But you could be missing a bracket in another class, which could affect other code. It took me much longer than it should have to figure that out.
You can clean a spreadsheet or database efficiently by using a program to adjust small nuances. But sometimes the time it takes to code is longer than it would take for you to do it in a non-programatic way. If the question is making it easier for the computer to parse, then you should let a program make that change. If it’s a question of making it easier on your eyes, there may be times when that isn’t as necessary. Or maybe that’s just me.
Little nuances can take a lot longer than you think. I kept fussing with the placement of textual instructions, wondering if it was clear enough where to click and when. Sometimes simple is better, I think just having the word “Zoom” for people to click on is clearer than a three-level sliding scale in this case.
Embedding Processing applets in WordPress is buggier than it should be. I’m unhappy that the visualization displays as a big gray box. But I couldn’t see a way to put text on that gray box saying “Click in this box to begin,” because that blank box doesn’t appear when I run the visualization directly on my computer, it just appears as it should. It’s a little better in HTML, so I’ll be linking instead of embedding from here forward. You can find today’s visualization here.

Changes in the numbers of students majoring in programming and social sciences

Michelle Minkoff — Tue, 19 Jan 2010 12:13:11 +0000

What is a traditional path to programming nowadays? It’s a question I’ve been thinking about a lot, esp. in the realm of the journalist-programmer. So many people from the older school of journalism came through using databases as tools to help with reporting, not because they took a class in it. That’s certainly encouraging for someone like me.

But that started the wheels turning in my brain — Are people still getting computer science degrees? Is the major more prevalent now than 5 years ago? 20 years ago? How do the number of comp. sci. majors compare to..social science, let’s say, another field that dovetails nicely with CAR (and yet another one I didn’t major in, but that’s beside the point.)

Questions like these are always good candidates for my weekly Processing projects. This week — time-related data! I went to the National Science Foundation, where they track the student getting various degrees in various fields. It’s broken down in lots of interesting ways, across gender, race, state, field of interest in the sciences, etc. A future project is the gender breakdown — where are all the female CAR/data/programmer journalists? I know a few, but still…we’re in my minority. It’s not surprising, I remember my own experiences growing up as the daughter of an Argonne computer scientist, and there weren’t many women around that building in the ’90s either, and there still aren’t as many as I would like to see.

Back to the point….For this project, I decided to look at the number of computer science degrees granted nationwide, across 40 years, from 1966 to 2006 (the most recent data available). I compared this to the same numbers for engineering and social science. The idea being to compare CS to one very technical applied field, and one more theoretical one, but all related to data in some capacity. I was amazed to see just how much more popular social science degrees were across the board, I hadn’t realized the distinction was that stark.

I also took note of the fact that the number of computer science degrees were higher in 2006 than ever before, but not that much higher than the dot-com boom in the late 1990s. But overall, there’s an upward trend. Not everyone’s going the self-taught route, although I imagine those with degrees must still self-teach, because you can’t possibly be taught every language that will be invented during your working life.

In terms of creating the actual project itself, I struggled with getting the data in an acceptable format for Processing. Excel was spitting out a text-delimited file, which is apparently distinct from a text-separated values file. Nuances, nuances. And then, I left the commas in the middle of numbers. It looked right to my eye, but there was no way a computer was going to understand that as a float. Note for next time!

I’m not completely happy with this final product. The code is taken almost verbatim from Ben Fry’s “Visualizing Data” Processing book (which I highly recommend). And for this type of comparison, I would have preferred a way to place all three graphs next to each other, so you could look at the data for engineering and CS degrees side-by-side. The y axis label is also a bit close to the numbers.

But what I am proud of is that I was able to pull off a tabbed graph. This is what the lovely Tribune graphics people did for our joint health care lobbying project, and at the time I remember thinking that it must be really tough to do. Now, I understand how it’s done, and was able to do it with a book holding my hand while I did. Next time, I’ll be even more self-sufficient.

If anyone wants encouragement for embarking on the journey of learning programming, I would argue this is a darn good reason. That feeling of satisfaction when you can make things happen, and truly become master of the machine, is simply amazing. What seems like magic, and impossible, becomes real and tangible. It’s just another tool in your arsenal. And then you use your journalistic creativity to think of how best to use it. How cool is that?

Reflections on Visualization Theory (Data viz readings, week 1)

Michelle Minkoff — Mon, 04 Jan 2010 23:32:33 +0000

In this first set of reading, I learn that the principles of simplicity, accuracy and more are as true in data visualization as they are in a text story.

Edward Tufte, Visual Explanations, “Images and Quantities”

In Tufte’s first paragraph of Visual Explanations, he discusses the importance of readability — a concept I see as having a strong parallel to the usability so often discussed in a more modern era. He breaks down it down into three types of depicting quantities: direct labels, encodings (scales of color) and self-representing scales. I like his breakdown, as I’ve seen examples of all of theses, whether on news sites, or t0 illustrate points in scientific journals when I was doing medical reporting. I attempted to do an encoding on my recent border crossing graph, but was unhappy that using color intensity to express data made everything so light that it was difficult to distinguish between colors. I would add the caution that encodings are best used when the entire graph is just representing various intensities of that one variable.

Tufte also writes that maps are just another type of graph. Statistical graphics are those that don’t just give the data, but spatially arrange the data on a straight one-dimensional line. No comment here, except to say that so far everything makes logical sense. And the main takeaway seems to be a maxim true in written journalism as well, “Keep it simple, stupid.”

At the same time, we should make sure to include all the facts and necessary details, which means including labels when they are not self-explanatory. Tufte critiques a computer visualization for looking pretty, but not putting its data into the proper context. Another maxim: Content is king. Without interesting information, the coolest icons, colors and animations do nothing. It’s nice to hear that it’s a professional point, but before I would have just clicked off of something I didn’t understand. And in the news business, or any business, that’s not something we want.

Ben Fry, Visualizing Data, “The Seven Stages of Visualizing Data”

Here’s Fry’s list:

Acquire
Parse
Filter
Mine
Represent
Refine
Interact

All of these could probably be applied to writing as well. I think it’s interesting that the first six of these elements are equivalent to reporting, and figuring out which quotes to use. It strikes me that in both text and graphic work the actual creation is the end-step, but if you’ve done the previous work well, the piece should just write (or draw, or code) itself. Fascinating.

One part that’s different about interactive data visualizations is the user’s ability to manipulate. This is a major focus of Fry’s introductory chapter. Instead of guiding them through a story by taking them where you want them to go (they read your words in your order), here it’s your job to set up so they can make their own discoveries. Creating visualizations for the Internet is all about putting the power back in the hands of the user. Sometimes with text, I feel it’s a bit too easy to become a bit power-happy, after all, you’re telling the reader what they need to know. Part of what I love about interactivity is that it’s democracy at its finest. User, here’s the data so you can understand it, but there’s a nearly infinite number of ways you can explore it. Have it, and revel in the true sense of democracy. That’s not withstanding the issue of whether the Internet is really democratic because not everyone has access, but that’s a topic for another time. Letting users drive our stories to a greater extent brings us back to why journalism was started, and its relevance goes beyond interactive visualizations, and provides a lesson I take to heart as a journalist, no matter what platform I’m using.

Michelle Minkoff » processing

Is a flat text file or a database right for an app?

Related posts you might enjoy:

Visualizing networking: When it doesn’t work

Related posts you might enjoy:

Treemapping Gov. Quinn’s State of the State

Related posts you might enjoy:

Relating zip codes and geography using Processing

Related posts you might enjoy:

Changes in the numbers of students majoring in programming and social sciences

Related posts you might enjoy:

Reflections on Visualization Theory (Data viz readings, week 1)

Related posts you might enjoy: