Many of the stories we do every day, across beats, could benefit from a data component.  Luckily, a lot of great design and programming folks have created tools to make it easier to organize, clean and display data.  Helpful whether you just want to get a table up quickly, or want to spend months creating an app.  Because we’ve all got a lot to do, and spending more time on creativity is always a good thing. Let the computer deal with the boring parts.

So, here’s a round up of some tools you can use to rapidly produce data pieces without programming knowledge.  There are many others I continue to play with, and new tools are cropping up every day. The best way to get to know all of these is by taking a data set and playing around with the various options.  Each category is organized in my approximate preferential order, which should be taken with many grains of salt.

Prepping tables

1. Tableizer – http://tableizer.journalistopia.com/ – Copy and paste cells from your Excel spreadsheet into this tool, and it’ll spit back a formatted HTML table that you can copy and paste into a CMS of your choice.  Pick the font and header background color.  Make sure your first row consists of headers, as these will be styled differently.  Combine this with the Jquery DataTables plug in listed at the bottom of this info sheet, and you’ve turned your mundane Excel table into a fancy-schmancy searchable, sortable and paginated table — a data news application in a box.
2. TimeFlow – http://wiki.github.com/FlowingMedia/TimeFlow/ – Plot time-related data on a timeline.  Useful for looking at chronology behind a project, or ongoing story on your beat.  Helps you structure your information.  Made for reporting, not presentation.  Resulting timeline is not embeddable.  To use this tool, you have to download it on your computer, either Mac or Windows will work.
3. Freebase Gridworks -  http://www.freebase.com/labs/gridworks – Downloadable tool to help you clean/analyze your data.  Import your data set, and it will look past minor discrepancies, like capitalized and uncapitalized letters, and group your data.  This can help you find patterns that will aid your story-finding abilities.  You can customize how it groups the data, so it isn’t aggregating information that it shouldn’t be aggregating.
4. Mr. Data Converter – http://shancarter.com/data_converter- Enter your data in this web-based tool to convert your Excel file to other data formats.  JSON is especially useful for feeding your data to a Flash developer.  It won’t directly help your presentation, but it will convert your data into a format that’s easier for presentation tools to read.

Interactive viz – no programming

1. Many Eyes – http://manyeyes.alphaworks.ibm.com/manyeyes/ – A fantastic tool for presentation, built by some of the best data visualizers in the business, who now work for Google.  Import your dataset, and get back an embeddable visualizaton.  You can use it sort through and analyze lengthy text, and it’ll give you a graphic-friendly word tree where users can see what ideas pop, and how those ideas are connected in a speech.  Or use it for intensity maps, where certain states are colored by the value of data (how does Calif.compare?).  Or use it for interactive bar/pie/line graphs.  No programming knowledge needed, but while you can customize colors, it can be tough to customize the genral feel of these applications.  More than suitable for a blog entry, though.
2. Google spreadsheets – http://docs.google.com – Feel like everyone knows this baby.  Use it to structure your data into various columns and records.  Share it with your coleagues.  Work with it from a variety of different machines.  Recognize Google may be looking at your data as data, so don’t use it for your super-secret investigative project that must never see the light of day until you say so.  You can use this spreadsheet to feed a mini-version of a Web app, and pipe it into a Web page, like as an HTML table.  More on that in a post TK.
3. Google Fusion Tables  – http://tables.googlelabs.com/Home – A lot like Google spreadsheets, but includes built-in embeddable viz tools.  My favorite has to be an intensity map.  If your data has a US state, or a country, and a value, it will shade those geo areas according to that value.  Sweet.
4. Socrata http://www.socrata.com/try-it-free – Really helpful for organizing data and spitting out charts, that are embeddable.  Limited interactivity.  Good for analyzing complicated data, and tend to take up a large portion of your on screen presentation.
5. Tableau Public – http://www.tableausoftware.com/public/ – Nice, relatively easy way to make charts.  You do it through an app that you download, which is PC only, as of now, I think.
6. DISCONTINUED – Open Flash Charts – http://teethgrinder.co.uk/open-flash-chart/ – This was cool, but no longer exists.  Plus, be forewarned anytime you use a tool that involves Flash, it’s not iPhone/iPad friendly, as of now.
7. EditGrid – http://www.editgrid.com/ – Makes simple graphs.  I found this one to be disappointing, and lack the visual punch I hoped for.
8. DISCONTINUED – Swivel – http://www.swivel.com – This used to be another favorite, and ranked pretty high up, until one day it stopped working.  A discussion on the NICAR list confirmed it wasn’t just me.  I’ll always remember it as how I spent 10 minutes of an arts journalism presentation analyzing data of what kinds of books have been sold over time, and my professor saying, “THAT’S what the independent study is?  That’s. So. Cool.”  It is.
9. XML/SWF charts – http://www.maani.us/xml_charts/ – Another limited and disappointing option.
10. GunnMap – http://www.gunn.co.nz/map/ – Another limited and disappointing option.
11. ChartGo – http://www.chartgo.com/ – Another limited and disappointing option.

Static viz

1. Juice Analytics chart maker – http://www.juiceanalytics.com/chartchooser/ – Nice templates for making better looking graphs in Excel.  Great if you need something pretty quickly, and aren’t completely invested in interactivity.  You’ll upload this as an image.
2. Excel – By default, the spreadsheet program makes some graphs.  Great for analysis, but you’ll want to dig into settings to customize for a better look if you want to use for professional presentation.

Use programming to make custom charts

1. Google Charts (static) – http://code.google.com/apis/chart/ and interactive – Love, love, love.  You can use these as standalone images, or interactive pieces, on a site, or integrate it into a Web app, or anything in between.  Uses Javascript, which makes it accesssible across platforms.  You can make almost any kind of graph you can think of.  Tutorials will walk you through the different types, and you can test your ideas on the fly in the Live Chart Playground, linkable from the left of any documentation page.  More news orgs should be using these puppies early and often.
2. Jquery (Data Tables) – http://jquery.com/; http://www.datatables.net/ – How to bring interactivity to static data-driven webpages.  Think of the various plugins as pre-made apps you can add on.  It does use Javascript, so you need to get comfortable with that.  But Javascript works  across browsers, and on non-Flash-friendly mobile devices.
3. Lovelycharts.com – Not as robust as the other options.  I feel like you can get these sorts of results via the non-programming options, with less time investment.

I’ve been thinking a lot lately about the type of work we do.  If we get a mention on Poynter or Nieman Lab, this is a great thing.  But what’s more important is creating projects that mean a lot to the general public.  So, while much of past data journalism has been based in politics and investigative work, and that remains a cornerstone, there’s this other part of data journ I think we should pursue.  That’s structuring our journalism as data across beats. Creating projects for features as well as hard news. Because news consumers with a variety of interests deserve the interactivity, personalization and cold, hard truth that data can bring to a story.  We CAN do this, and we SHOULD do this.

We have control over how we structure and present our own content, and we can do better with linking it up. Background: Started thinking about this as a simple idea when I read Derek Willis’ Rivers of Data, which takes the concept much further.  The piece is from 2005, and it’s frightening how much remains true. I also wrote about this a bit for Poynter.

So, while I’m proud of all the work I did at the LAT — from a campaign contributions application to an investigative sidecar that explores how YOUR redevelopment agency is spending its money, my redesign and relaunch of the LA Times’ bestseller list as a data project, that’s kind of my favorite accomplishment of the summer.

If you look at the site, you don’t see the back end that takes a complex formula that was being calculated by hand week after week, and translates it into a function that calculates lists automatically. You don’t see the nuanced details of the internal system I created, which wouldn’t be possible without how easy the Django admin makes these sort of things.  I hope it helps reduce the manual calculations for people who enter bestseller lists week after week.

But instead of standalone lists, now we can loop things together.  Bestsellers lists come out every week, with the same darn structure.  We can, and must, take advantage of that fact.  We can, and must, present those lists differently, with better interlinking, on the Web.  It can, and must, be different than how we’ve done these sort of things in print.  This makes exploring bestsellers specific to the SoCal area easier for the readers/users who care about digging deep. That’s what it’s all about.

Show me the data.  It’s a repeatable mantra, and one of my favorites.  Don’t just tell me “The Help” has been on the list 71 weeks.  Show me how it ranked, and when it ranked, each of those 71 weeks.  Let me click on those weeks, and see the ranking in context.

Not because it’s some cool way to tell a story (although, I like to think it is), but because if you’re a lit geek, it matters.  More information, the better.

And this project won’t become irrelevant because of stale data.  The LAT will keep coming out with its bestseller list, and the database will slurp that structured information into an integrated site.  Find the information you want, search for a specific title or author you care about, browse to learn more information about what you never knew existed.

Collaborate with reporters, producers, developers, analysts.  Not one set of them, but anyone who’ll give it a shot.  The data structure is well and good.  But…the person on the books staff who’ll add in links to LAT book reviews associated with each book, that makes it even better.  If the computer can’t get it done, enlist the people who really know the subject.  Structuring our news as data isn’t the final piece. It’s the beginning.  That structure allows us to add pieces to enhance how people can explore information.  We used to just access reviews by flipping through the paper.  Then, we added search engines.  Now, a deep dive page about a given book’s bestseller history can link you to that review.  The more ways users can get to our content, the better.  How effective is it? We’ll see.  But we can’t stick to structuring our news the same way, and failing to give it a shot.

Successful data projects require integration and collaboration across parts of the newsroom, and that’s something we were able to accomplish on this project. Thanks to so many at the LAT for bearing with me, during this and other endeavors, listening to some crazy ideas.  Can’t wait to see how the bestsellers site grows with this new framework in place.

So, data projects, what I spend my life on now.  Yeah, they’re hot these days.  They’re also a way we’ve been doing some parts of journalism for decades.  But here’s my philosophy, as someone who’s been dabbling/obsessing about this for a while: Don’t just do it for investigations, don’t just do it for one type of news.  Don’t just see it as a technique for one type of project.  Data’s not just a specialty, it’s also a way we can structure our sites themselves.  All to help people explore their world.  All to tell the story.

This is a philosophy I’ve been dreaming about for a while.  Haven’t pushed it far enough yet.   But bestsellers is a major step toward what can be, integrating data with the general newsroom process. I know it’s not an entirely new idea, but having the skills to put a vision like that into reality, to create this sort of a framework, for someone doing this for about a year, it’s a great “parting shot” (as Ben called it) for my time at the LAT.

Bet I can do even more at PBS, as the learning continues.  Less than a week till I get out there.  Don’t know exactly what we’re going to put together, but it’s going to be pretty darn cool.  And I hope this philosophy provides a decent jumping off point for some discussions, and some of our new content.

Watch out Washington – I’m coming your way soon!  Start at PBS on Oct. 25.  It’s going to be a roller coaster of a ride!

My “summer” internship at the LA Times is almost over. It’s lasted from April until now, and I could stay longer. I swore you’d have to drag me out of “dream internship” kicking and screaming, as I clung to the massive walls of the still-impressive building. I will never forget the chance I was given at the LAT, and the boost it gave my career. But it’s time to move on.

I’m delighted to report that I’m joining PBS as a Data Producer for the organization’s website. Specifically, this will be a new news and public affairs component of the site. Yes, I’ll work for the national headquarters. Yes, that’s in Washington. We’re going to start something brand spanking new, capitalizing on the engaging, riveting, in-depth style PBS is known for, and translating that to the Web. I’ll get to serve as a generalist, with a specialty. Exercise my entrepreneurial spirit, but within the structure of a strong organization.

We’ll push the data visualization and application side of things, but I’ll also get to use the plethora of skills I’ve picked up along the way outside of data journalism. We’ll go on an adventure together. There’s so much I can’t be specific about, because it’s all new. But I do promise to not only keep you updated, but pick your brains for advice on what you want to see from the digital side of public television.

That being said, my hope is that we can say a few things for certain: We’re not going to be doing the same old journalism the same old way. We WILL push the boundaries of storytelling. We WILL help people stay informed and engaged with the news. We WILL help people make sense of the world, and we will do it with concrete facts. We WILL remember we do this for the users, the public. We WILL combine the best qualities of data journalism, online journalism and public television.

This transition is made easier by the fact that I have high hopes for bosses who “get” the mission I’ve been yammering about for a year. It’s not just about data, but about integration. We’ll be a nimble enough team that we can learn from each other, and not be segmented. The folks already on board have significant experience in this world, whether it’s fellow TribCo alum Tom Davidson or St. Pete Times alum Christine Montgomery — Matt Waite’s boss during the launch of Politifact. They know what data apps are. That means we push harder. Not good enough for me to say, “Hey, you ever heard of data journalism?” Got to come up with something better, something fresh and new. We’ll push each other to create the best product we can. We only fail if we fail to try.

I’ll be moving to Washington DC, in a very few short weeks — planning to land there by the end of October. So excited! After all, about a year ago, I had the pleasure of being introduced to this sort of work. It all started with one class. I discovered more and more people like me, and the community made me better. You can learn a lot from engaging online, but I miss the real-world conversations we had a year ago. I love the people I’ve met in LA with all my heart, but I still can’t help but feel that Washington is my intellectual home.

I’m excited to return to PBS. Last summer, I served as a Web intern for Chicago Tonight at WTTW, Chicago’s PBS affiliate. It was one of the best summers of my life, and I learned so much about broadcast and the Web, and the relationship/integration between the two. But that’s also when the wheels started turning. I wished I knew more about how to get data on a map. I wished we could go more in-depth with Web packages. I wished we could use pure facts, data, not just for investigative pieces, but to enhance people’s knowledge on topics across beats. Now, I have enough knowledge to push further, and to accomplish so many things I just couldn’t a year ago. And I’ve long thought public television sites were a niche ripe for the picking.

And now’s as good a time as any to acknowledge that I wouldn’t be in this position without the mentorship of NICAR folks. You know who you are. And the greatest class I ever took after Medill is the classroom I’m in every day, the LA Times Data Desk. The data analysts show me the thought process, and intense checking, that goes into every data project. My boss, Dan Gaines, listens to new ideas and floats his own, always looking at you with an intensity that shows just how much he cares about journalism, and you as a person. But my greatest teachers have been Ben Welsh and Ken Schwencke. Fellow journalist-programmers, we geek out together and support each other. And no matter how messy my code, how many mistakes I made, how much I’m still in the rough, they make me better, and never with a harsh word. If I get stressed, I only need to look over at them to see it will all be okay. Leaving them will be the hardest part of all. But they’ve prepared me to get to this point, and I’ll find even more mentors and supporters. And guys, you know you’re just a GChat or a tweet away, even if a continent separates us.

On an even more personal note, this next step represents some sort of victory. Yes, I know it’s only the beginning. But at the same time, it was this sort of position that I’ve been working for all along. I didn’t learn programming so I could write some awesome loader, geek out about server loads, etc. It’s all fascinating, sure! But I did it, I do it every day, and I will always do it — for the journalism. It’s a tool, just like a photo journalist using a camera. That doesn’t make him or her a wedding photographer. They practice journalism through photographs.

We see so many posts in the journo-community, asking if journalists should learn programming, if programmers should learn journalism, what do we name people with this hybrid?

But, here’s the question I forgot to ask. Once you start getting all these skills, where can you go to use them? Some of my mentors received quite a few late-night emails, listened to quite a few rambles. Where is the place that understands my tech skills are deeper than uploading my story to a content management system? Where is the place that understands I could add features to your site, and do general tech/IT maintenance all day long, but I have these journalism skills. The tech skills help to tell the story. It’s not one or the other. I consider myself equal parts programmer and journalist, journalist and programmer.

I had a fascinating job search, and managed to never file a single application, without someone from an organization reaching out to me first. That’s not what they tell us will happen at Career Services! If you want to pursue this career change, there isn’t a better time. Still, finding the right organizational match is no easy task. And I can’t tell you how ecstatic I am to have found that match.

In a world where we decry the death of journalism on a daily basis, where we’re afraid we won’t get jobs, I say there IS hope if we hold out. If you believe in what we do, follow your interests and skills, you’ve got a better shot. We must always push onward. And I, well, I never believed that less than a year after graduating Medill, not only would someone pay me to do what I love, but I would get to do it at an organization that instilled my own love for education. It still seems like a dream that’s coming true. I never believed I’d get to a national organization this fast. I never believed I’d get to take my ideas outside of a blog, outside of an academic paper, outside of internship after internship. Doesn’t mean I’m right, I’m probably not. But if the ideas just stay incubated, and are never considered professionally, what good is that?

We don’t have time to waste. The moment is now. We can’t afford to wait, the industry’s already behind where it should be in 2010.

So, I promise anyone reading this, friends I’ve known since elementary school, my dear supportive parents, teachers from all levels of schooling, fellow journalists fighting the fight for truth and knowledge every day, I promise that I will give this everything I’ve got. I recognize the preciousness of this gift. And on behalf of young journos everywhere, PBS, thank you for believing in us, in our potential. Thank you for the opportunity to experiment on a national playground. We only move forward by trying something new.

There will be no resting on laurels, there will be no complacency. Thank you all for your help so far, and I know we’ll be calling on each other in the days, weeks and years to come.

But for now, journalism must push forward. Let’s get started.

I’ve spent much of my “summer” internship at the LA Times (ranging from April through present) working on a campaign contributions searchable application, to help the public explore the money that’s financing various propositions in California. The first step was getting the Prop 8 application out back in late June. I’m really into the idea of templating data apps recently. Make a custom interface, but make it resuable, so it doesn’t have to stop as one application. It’s like how we already use third-party source tools as frameworks (Dipity, ManyEyes, etc.) We should expect the same, if not more, of custom solutions we create ourselves. So, the original idea for the Prop 19 app was to apply Prop 19 data to the Prop 8 code. We did that, but as is my normal mode, more features suddenly found themselves on the list. So, in the end, we have two frameworks for campaign contributions now: the simple version and the more advanced version.

One of the large advancements in the second version is that every single donation is listed separately. In Prop 8, the legacy data passed down to me had aggregate donations, which is helpful, but means users miss out on certain nuances. So, when we were able to pass each donation in separately, I believed it was key to make each one linkable and sharable. Not only does each donation have a separate link, but so does every donor, city, state, zipcode and committee. This means if people find something and want to blog about it, share it with a friend, draw a media organization’s attention to it, whatever, they have a way to point someone else to that exact spot. We don’t do enough of it in the data apps world. Web frameworks make this easier, Flash interactives do not.

Creating permanent links to records and aggregate pages of our data is essential. It’s how we combine our searchable databases with other reporting forms that provide even more context. Did you write a story about someone you know gave a lot of money to a campaign? And there’s a searchable database to go with that campaign? Don’t just tell me they donated a lot of money, show me.

Just an example of one way we can change our mindset as reporters, Web producers, developers or whatever it is we do. Find the minute detail, make it shareable. Because while you may only care about record #1157, someone else cares much more about record #1158. And both of you should be able to find that piece of the story that matters to you and use it. That’s data for personalized news, and yet another lesson I take away from my time at the LAT.

PS — If you want a random Django tip (come on, you know you do) {% ifchanged %} is my new favorite template tag. If you have a big list you’ve pulled out as a query, and it’s sorted, but you want to separate the list into subcategories, you want this. Figure out an attribute that’s going to change, and enter that info as a label, inside the {% ifchanged %} tag. It’ll show up when the code and text inside the tag changes — makes sense. This tip has made an ongoing project a lot easier, which I’ll be able to chat more about in a few weeks once it launches.

Tomorrow, Wed. Sept. 8, we’ll be discussing “Finding the story in the data” at #wjchat, and I have been tapped to host. (One might ask why, they must really be running low on people….I kid, I kid.)

This issue is extremely important, and one that must be addressed by the Web journalism groups. There’s a difference between telling a journalistic story, but lacking the data and displaying data in an aesthetically pleasing way that doesn’t really tell a story. The best data journalism does both. It’s rare, and I believe we all need to work harder at doing more of it. A tall order for someone to do alone, or even a team, but if the community puts its collective head together, we’ve got a better shot.

I couldn’t be more excited to discuss some of the issues I’ve been spending much of the last year internalizing. I have some better ideas about this now than I did back last September, I hope, but it’s also one of my favorite topics to ponder, because there are so many ideas I know I haven’t considered. I hope to report back with a nice compilation of links and thoughts once we tap into the community’s knowledge, but for now, I’m just spreading the word.

If you read this blog, but aren’t familiar with #wjchat, it’s a gathering on Twitter on Wednesday evenings where we discuss various aspects of online journalism. It ranges from social media, to video, to how to get a job, to data. I love all the facets of journalism, so it’s a great way to guarantee yourself some great conversation and learning all wrapped up into one package. At 5 Pacific, 8 Eastern, on Wednesday, that’s tomorrow, just hop on over to Twitter and follow the hashtag #wjchat. I find it easier to follow via tweetchat.com, which gives you a little chatroom that has a nicer interface than twitter.com — the conversation can get fast and furious.

In the meantime, if there are specific topics you’d like to see discussed, or questions you’d like asked, get at me before, or during the chat.

And if you’re one of my dear NICARian mentors, it’d be really fantastic if you could find the time to drop by. Anything I’ve picked up in my career thus far is minimal, and I owe it all to the collective wisdom of those who’ve taught me what I know as a journalist and a programmer.

See you in the virtual space!

So, there’s that. First launch!

My checklist:

• Figure out what part of journalism inspires me. CHECK (Data, programming)
• Find place where I can learn about it from others, but have freedom to try out my ideas and learn from knowledgeable and patient folks. CHECK (LAT)
• Use this opportunity to learn enough about programming to create at least one app. CHECK (See first line of post)

That was a good time.

I’m planning a post on best practices for creating a searchable database application like this, with examples, and technical geekery, and all that, and I’d love your thoughts in the meantime.  Let me know if you have specific questions.

But in a moment of self-reflection, I’d just like to say this.

I was sitting in my apartment this morning, and found that a butterfly had somehow gotten in (blame a broken screen, maybe).  It sat on my windowsill, facing the outside world.  It would feel around the window, trying to find an opening.  After circling the perimeter of the window, it laid back down, almost motionless, just searching.  An hour later, the cycle repeated.  Wanting to help the poor butterfly, after watching it struggle, I tried to capture it in a small box to take it downstairs.  It was too smart for that.  Fine.  What actually worked?  Covering all the windows and opening the balcony door.  You can’t force the butterfly out, even though you’re helping, it had to find the path out on its own.

Why do I bring this up?  Am I slowly going insane after spending the day with Dive Into Python (rec’d)?  Perhaps. But I have a point.

Let’s take the common journalist, yearing to know more about data-driven apps.  (I can’t be the only one, I just can’t!)  I search for the answers, voraciously read, alternate between laying still while observing the space and wandering around the perimeter, feeling it out, trying to dip my toe in the water.  I jump into the pool, head-first, realize it’s not working.  Can’t do it on my own.  Many people who’ve already made the journey try to help.  But they just can’t just give me the knowledge, can’t just take me to my goal, can’t just make it happen.  They must lead me there, give me the same resources they used.  Recommend the books, the sites. But in the end, I’ve got to find my own way out of the house, the prison of not knowing enough coding basics to begin.  Pushes in the right direction, guiding me away from those false starts, this is essential.

I’d just like to take a moment to thank the experts, who guide me away from the false starts, who don’t give up when I keep turning the wrong way.  In particular, in terms of the programming skills I’m learning, it’s Ben Welsh (who I happen to have been working most closely with), who exercises that kind of patience.  Who never tells me not to fly into that closed window, because while it was closed for him, I might discover a way to get out.  Of course, I don’t, and he’s no less interested in helping me get back on course. Thanks!

Still a long way to go.  So, what’s changed?  Now, I’ve gotten outside.  I made my first app.  When someone asks me what steps make up a project, how long it’ll take, I can give a legitimate answer, or know how to find one.

I still remember wondering if I could ever build an application.  And Derek said to me, “There’s only one way to find out.”  Derek, who I used to bug on a daily basis about this stuff.  Derek, who I haven’t asked a Python/Django question of since March.  I’m the LAT’s problem now!

Well, as Derek said, find out we did.  Time to push it further, I say.

There’s a whole world to explore now that I’ve escaped being stuck inside.  LAT folks, prepare yourself, the questions you’ve seen so far are only the beginning.

I’ll just put it this way.  Andy, you’ve been an inspiration.  Watching your work while I was a student at Medill, and how much you enjoyed it, I knew some day I could do anything, if I could just set my mind to it, and find supportive mentors.

It’s a parallel thing.  Andy gets Matt Waite and Jeremy Bowers at the Times of the Southeast (St. Pete), I get Ben Welsh and Ken Schwencke at the Times of the Southwest (LA).  Match good mentors with journalistic enthusiasm and obsession, and you’ll get somewhere!

We both get to bring passion, skill and journalistic knowhow to the table.  We bug the people with the tech knowledge until we have a moment like this.   Maybe someday I’ll report in the field and code like Andy does.  But for now, my reporting consists of investigating the nuances of the still-large ship that is the LA Times, and looking at how it can be even better.  And at this point in time, I wouldn’t change it for the world.

It seems appropriate to post this interview with Andy now (Yes, I’m making excuses for delaying this for months and months….).  I chatted with him in March, just after the launch of his first Django app – MyLawmaker.  And I just launched my first one about two weeks ago.

I started these Data Delver interviews knowing no one in the field other than Derek Willis.  (And if you only got to know one person, he’s as good as it gets.)  And now, I feel like I know so many more.  And for some reason, I get to be a part of their ranks each day.  It’s a privilege, an honor, and an adrenaline rush like no other.

I’ll let Andy take it away in his own words now.

This transcript of my interview with Boyle is a part of my continuing series I’m calling “Data Delvers,” where I pass on transcripts, summaries, quotes and audio clips from conversations with journalists using technology to find, analyze and convey data-driven stories and/or projects to the modern audience.

Extended transcript

Why is having an understanding of data important?
When we have meetings to discuss story ideas with the group I work with we mainly focus on breaking news on the web.  When somebody says something, it’s really nice to be able to go, “Oh, I know how they keep those records so I know exactly how you can ask questions to find out how many total XXX has happened this year.”  It really helps to have a background, it gives you a chance to dig deeper and get more context.

Is St. Pete working to get other reporters to get more up to speed on this?
I think there’s quite a few reporters who do have this type of skill or do have the basics.  They can use Excel; they can use Microsoft Access.  Actually, there’s a reporter who did a really awesome story on leaky underground storage tanks and where they are located.  My compadre, Darla Cameron, does a lot of GIS stuff, so she’s also a data nerd, like me. She was able to help him with that. A lot of people use databases on a pretty average basis and have some CAR training, but not as much huge stuff like the Wetlands project that the St. Pete Times did with Matt Waite and Craig Pittman a few years ago.  But things on a smaller scale are happening.

Going back to the MyLawmaker project, what was the genesis of that?
The genesis is that on Monday the state legislature starts so we print this thing every year in our Perspectives section which runs every Sunday.  We print opinion pieces and stuff like that in a section called ‘For a better Florida’.  One of our politics editors, Amy Hollyfield,  had sent an email out to one of the higher-ups  saying that there’s this project from the New York Times called Represent and another one at Oregon Live –  Your Government, where you type in your address and it shows you your state lawmakers, congressmen, city lawmakers, etc.  She asked if it would be possible for us to set up something to help people find their state representatives and state senators from their address?  This somehow made it to me and I go, “Yeah, yeah, that can be done; it’s possible.”  At that point I had no clue in my mind of how we were going to do that, but I knew it was possible because obviously someone else could do it. So we had the shape files; Darla already had the information.  Somehow we could write a program that would geocode your address and would ask what you wanted to see, and it would print out what you wanted.

So that was basically last Thursday and it is now Friday and we are ready to launch.  Thursday, about 3:30 in the afternoon, it was like, “Hey guys, do you think you can do this?”  I was taught a long time ago that if you think you can, try it. This is something that I really wanted to do.  It was an opportunity to try this stuff   I’ve been working on.  I’ve been working on some Django and Python with Matt Waite and Jeremy Bowers, so it was nice to spearhead a project and have ownership on something and work with a group of awesome people like Darla and Lee.  I’m really jazzed.

So how did you start getting into Django…just by being around Matt and Jeremy?
Yeah, that’s totally it.  He’ll [Matt Waite] browbeat you into believing what he believes.  When I went to the Indianapolis NICAR conference he had a thing, a bootcamp, for three or four of the days on frameworks, so that was Django and the basics of it.  And I remember when he showed it,  I thought, “Man, that’s way easier than making hundreds of  individual web pages and then having to edit the HTML in each one. “ I remember being a kid and having an AOL members’ web page and how horrible it was to have to update every single HTML file and I thought “Wow!  This is something that automatically creates this stuff.   That is sweet.   I need to learn this.” And that was in March of last year.

And then after that it was just me attempting to learn stuff and breaking everything and being really afraid that I’d destroy my computer and not becoming afraid of the terminal. It was a very long process.  There were a couple of projects over the summer that I worked on.  One was FCAT, which is an aptitude test given to students in public schools in Florida. We were able to make a searchable database for that.  I wrote the models on that, and then they did the rest of the heavy lifting.  I guess that was the first project I had any sort of help with.  Then we later had a project for high school sports, called Home Team, which is totally awesome.  I was able to see the inner workings  and it was really cool to see the process of how they were developing it and how it was being set up.  I wanted to build something, so when the opportunity came along for MyLawmaker, I jumped at the chance to do it.

So what did you end up using for the geocoding?
We used Google.    There’s a limit of 2,500 times per IP in a 24 hour period which is an issue if you hope to get a lot of hits on  your website.   So what we do is we actually pass it off to JavaScript on the client side, so the server’s IP address comes from the user’s computer. Not that we expect 2,500 hits a day, but we have no clue as to how many hits it will get.   But we thought it’d be nice just in case.  We don’t want people trying to use it and it’s broken.

Have you had requests from the community for anything like this?  It doesn’t have to be specifically MyLawmaker, but just more interactive stuff?
I think a lot of people really like it.  When our FCAT project launched in the summer, it got a ton of use.   It still kind of does.  A lot of those projects that are usually attached to stories initially get a lot of hits.  With a project like this once we told some of our colleagues about it, they were like  “Wow, that’s a great idea.” With the state senate website, you can only search by zip code.   Then it will tell you the districts that are in that zip code and you have to click on that and see if you are located on a little map of the district, so ours is actually a little better than part of the state legislature’s.  But I guess that’s the goal.  We want to make sure that people can find the information and find whom to call if they have an issue.   A lot of people were quite surprised…they didn’t believe who their state legislators were.  Said, “That can’t be true.”  But it was, and they wouldn’t have known otherwise.

Why do you see this type of work as important?
It’s interactive.  When you pick up a newspaper, you read a story and it kind of ends there.  What’s really cool about this is it helps you find out broader information.  You can go searching a given database online and become more aware of your community. That’s part of what we do, as journalists, and as news organizations, we try to inform the public.  To do it in a way that’s automated, that doesn’t require our constant supervision or our constant writing of stories, helps because it frees up time for us to do other stuff.  It still gives people information and more context when it comes to stories.  I believe that a well-informed populace is much better than an ill-informed populace.  It’s also really cool and lots of fun to do.

Do you enjoy living in both worlds – the reporting and the developing aspects?  What are the advantages and disadvantages?
Yes, I do.  I guess time is a disadvantage in everything we do.  Part of it is because of the reporting, I can come up with projects that we can do.  I think if I was only a code monkey and I was never out on the streets, I wouldn’t find out what data is available or what data we can build on our own. It also helps you keep track of the news.  Like this is being built in response to the fact that the state legislature is meeting on Monday to start a new session. If I was just a reporter, I wouldn’t be aware of different ways to think and inform the public.  And if I was just a developer, I would be a little blind to some of the opportunities that there are to inform the public, and make cool stuff.

Any other advice?
Something I wish I’d been told when I was younger is failure happens, especially with this sort of stuff, so you really need to get used to it, and we all fail.  Your failure is a failure, unless you learn.  Then, your failure becomes a win.  If you’re always trying to learn stuff, as a journalist, if you’re always working to develop different skill sets, whether it’s Web development, normal computer-assisted reporting, GIS, narrative storytelling, it all helps you, it all helps the other things.  Whatever new skill you learn in journalism will add to your big palette of rocking.

Would you recommend all or more reporters get into the coding side of things?
It doesn’t hurt.  It doesn’t hurt to understand the basics of data.  If you’re a city hall reporter, it helps to know basic Excel, it helps to know Access. There’s a ton of cool stuff you can do because of it.  If you are a feature writer, and you write longer narratives, and stuff like that, you can still find stories through databases, you can still find extra people to interview through those methods.  It helps in every aspect I think, just to have a little bit.  Whereas I have made the intense plunge into being a total nerd when it comes to this stuff, if that’s what you want to do, that’s what you want to do.  I don’t think that should be a requirement, but I think you should have at least the bare bones basics of what’s going on, because knowledge is power, and knowing is half the battle.

One of the areas in journalism most ripe for data work, as I see it, is enterprise stories.  At papers with small CAR staffs, there’s often a serious strain on the time of CAR reporters and editors seeking to provide context and fodder for long-term projects, often investigative, and enhance shorter one-off daily pieces, often breaking news.  Spend too much time on one area, you’re neglecting the other.  At the Seattle Times, one editor is called the Data Enterprise Editor, and her time is largely based in project work, that breaks out of the daily story routine.  She works to include interactivity on the website, from searchable databases to Google maps.  And she works with a group of reporters focusing on suburban areas, that are too often undercovered.

It’s her job to organize and foster collaboration across the newsroom, to create the best data-based stories and projects possible.  The woman who holds this title?  The Seattle Times’ Cheryl Phillips.

This interview with Phillips is a part of my continuing series I’m calling “Data Delvers,” where I pass on summaries, quotes, transcripts and audio clips from conversations with journalists using technology to find, analyze and convey data-driven stories and/or projects to the modern audience.

Bolding within Phillips’ answers denote some of the quotes I found most interesting.

Please describe your role as Data Enterprise Editor.

The data enterprise work is basically me and a small team, two other people. We focus on computer-assisted reporting, and then try to take that CAR work a step further and do some interactivity online with it, like a searchable database, whether it’s a Flash piece, interactive map, or database. The stories vary, some are longer projects, some are, “Let’s just create this Google map for a feature piece.” That has nothing to do with investigative journalism, but it creates a more interactive environment for the paper. That takes a big chunk of my time. The other thing that takes an almost equal amount of time would be that I’m an editor for a team of three reporters that cover suburban communities. We try to make them pretty mobile, so they’re out often, but the idea is that they do enterprise. It’s not like a daily suburban story. It’s like, outside the city, what’s going on, what are the issues of note. One of my reporters is doing a big piece about all these revised floodmaps that are coming out, and what the impact is going to be on these rural communities.

When did you start?

I’ve been doing this job a year and a half, since Sept. ’08.

How long has the paper been doing suburban reporting?

We used to have an East Side bureau, but then, when we had cutbacks, we eliminated that bureau. We also had another bureau covering the suburbs to the north. So, this is an effort to make sure that we don’t forget about those suburban communities which still want to be covered, but we can’t cover every city council meeting, so we cover important news that matters. We can give a sense of place for our readers by telling these stories.

Can you tell me about your career path to the Seattle Times?

I’ve been all over the place. I started in the ’80s back in Texas, then I went to Montana, which is my home state, and I worked for a small Gannett paper there [Great Falls Tribune]. I was on loan to USA Today for about half a year. Got back in ‘95. Then I went to the Detroit News and was a CAR projects editor there. I went from there to USA Today, and was a database editor. You probably heard the story about how I left USA Today. I was fired for touching a piece of artwork, along with two other people. I did a project for Dateline on drunken driving, and then I was hired by the Seattle Times.

Can you pinpoint any marked differences between broadcast investigative work for Dateline vs. print reporting?

It was stunning to me, the difference. I hadn’t truly recognized it. I did a lot of work for this Dateline project special. The editor from Dateline did a lot of work, too. They did a lot of filming focusing on one woman who was hit by a drunken driver, and her recovery. I spent months analyzing data from four different suburban metropolitan areas, looking at sentences for manslaughter where someone had also received a DUI for vehicular manslaughter when they were sober. The result we found was that if you also had a DUI, you got the lighter sentence on average, so you got penalized more for being sober than you did for being drunk, which is kind of stunning, and it boiled down to two sentences on air. I could have written an entire newspaper article about that, but it was important for them. They were willing to spend the time to invest in my work, but it was just a piece of this visual narrative, so it was very different.

What is it that drew you to CAR in the first place?

I just felt like there were stories that were going untold, that you could tell if you understood what was happening in bigger swaths of information. You could use these technologies and tools and become a better reporter. One of my very first projects was really simple, it wasn’t a big chunk of data. I used a Lotus spreadsheet to analyze the partnership agreement for the Texas Rangers, back when George W. Bush was one of the owners of the team, to figure out how much he would get if he sold the team, and how much everyone else would get. It was a formula, so I had to use a spreadsheet to do it. It was the first time I’d been exposed to that. I had a CPA friend who helped me with it. And I was like “Oh, wow, I can actually do this,” and it was kind of amazing. It opened a new world for me and it kind of went on from there.

What’s your role in IRE?

I’m chairman of the board at IRE, and I’m a past president.

What’s fueled your involvement in that organization?

I wouldn’t have learned anything had it not been for IRE. I started going to conferences in the mid-‘90s, and I would have gone earlier if I had known about the organization. My first conference was when I was in Montana, and I was on loan to USA Today, and my paper from Montana didn’t really have the money to send me to a conference, and the other database editors at USA Today basically said, “Well, yeah, I’ll share a room with you. My point is, I had no money and a whole slew of people took care of me, from sharing rooms to not letting me buy meals, and it was really stunning. So, I ended up rooming with someone from USA Today, and literally, no one would let me buy anything . It was just great, and I learned a ton, and I saw what was possible.

I went back to my small Montana paper and started to incorporate that into my work, and turned out some pretty good stories that I couldn’t have done otherwise. So, my involvement in IRE has come because I wanted to make sure that other people could have that same experience, and produce some great journalism based on what they learned.

As far as the future of CAR, do you see this as a skill more reporters will be required to have? Do you see it more as the role of a CAR specialist?

There will always be a role for a CAR specialist, now more than ever, really. I hope more reporters will gain basic computer-assisted reporting skills, that they will all know how to use a spreadsheet, they will know how to calculate percentage change, basic stuff, especially as we try to develop searchable applications. But when it comes to dealing with really complex data sets, you’ve got to spend some real time in that world. Because, if you don’t use these skills often, then you lose that. I’m an example of that. It’s been a few years since I even used ArcView mapping programs, I’m pretty slow and clunky now and it’s because I haven’t had a need to do that, to use that map expertise. So I go to my CAR specialist, and I have him do it, because unless I’m going to be doing it all the time, I need him to make sure it doesn’t get screwed up.

I have reporters who don’t do it all the time, they’re not CAR specialists, but they do have computer-assisted reporting skills. I think the balance is that they do the work, which is great, and I want them to keep doing that work, but it gets run by a specialist or by me, to make sure that somebody checks that analysis, just to be doubly sure. Anyway, we always check, but it’s especially important when it’s someone who doesn’t use high-level CAR on an everyday basis.

How do you decide which projects are worthy of interactive and/or searchable database treatment?

I think it depends on a couple of things. It’s similar, I think, to how the graphics department thinks about the graphics that they do for the paper. Do they aid the reader in a better or deeper understanding of the story? That’s one piece of it. Another piece of it is: Would the reader be interested in more information? On the Web site, in a text file or just in the paper? A lot of people are often interested in diving down into the numbers, that’s something which I think can help with greater transparency. If it’s one where you think someone would be interested in those numbers, or it would tell them something useful, create a database for it. If it’s not, if you don’t think people are going to go to it, if they’re not going to be interested, or if it doesn’t illuminate in some way, then we don’t do it.

How is this work broken down at the Seattle Times?

Another part of the Data Enterprise piece of my job is that I work across departments. It may be that the business editor is having a reporter working on a story about home values. So, she’ll come to me and say, “Hey, do this with us. We can create something online,” and she’ll say something about what they’re trying to achieve. Then, I’ll go to the online department, and I’ll say “What resources do we have if we create this searchable interface? Can you do some design with it, or give it some extra treatment?” It’s sort of a collaborative conversation. Also, depending on the level of time involved, I think about if it would take our CAR specialist, or me, x amount of time, because we also have this project coming. It’s really a mix. We’re very collaborative, we talk about everything.

Do you try to bring all the departments in at the outset of a project?

Yes. Pretty much. In ’09, we did an interactive Flash map, it was an environmental graphic. We won the James V. Risser Prize for environmental journalism, Justin Mayo was the CAR specialist on it. We did this really nice thing that took quite a bit of time, it was very complex. We had brought in everyone from the beginning. The photographer was there, two reporters, Justin, and another reporter. And we had the Flash guy involved from the very beginning. We took a look at logging permits, and kind of showed how there was flood damage in particular areas that had been logged heavily. We ended up using these maps, we were creating these maps for our reporting use, to say, “Oh, look, here’s this hazard zone.” We found ourselves using it so much as a reporting tool that it dawned on us, in part because we had the Flash guy involved from the beginning, that we really needed to put this online. If we use it to help us understand a pretty complex subject, then it really would help the reader.

Speaking of readers, what feedback have you gotten from the community to this sort of work?

I don’t get a lot of phone calls, or things like that. They respond to our stories saying, “Thanks for getting this information,” or “Gee, I’m really glad I can look this up.” That’s really about the extent of it. I would say more feedback comes in the way of traffic, and our searchable databases get some pretty good traffic. Even something simple. We did election results, where we created a quick searchable database for the elections, and that got an incredible amount of traffic. You could go to the various counties, and see what those counties were doing, and people seemed to like having it all brought together in one spot.

I’m looking forward to it, yet it seems sort of surreal.  Graduation marks the end of one chapter, the beginning of another.  But I started the new chapter already.

As much as I love connecting with journos, I’ve discovered my underestimation of the year: I’ve got more to learn than I ever could have imagined.  People have said it in countless Web discussions.  Find a project, do it, pick up skills along the way.  So, I’m concentrating on that like I’m cramming for finals, but with a never-ending study period.  Producing an app in a day uses some skills, doing it right over the course of months is a totally different ballgame.

This is where I am now.  But that chat topic begs the question: Was there a point in going to Medill?  Did it mean more than the opportunity to walk across a stage in a purple gown two weeks from today?  Is it just some archaic tradition that has nothing to do with my new coding life?  To these questions, I want to shout at the top of my virtual lungs, “It was essential!  It mattered!  Without j-school, what I am doing would not be possible!”

It’s about much more than curricula

Don’t rely on the curriculum of your j-school.  Fine, Medill doesn’t teach coding, most j-schools don’t.  Maybe your school doesn’t teach this or that, doesn’t have the right specialization.  It doesn’t matter.  The most valuable lessons I learned from j-school didn’t happen in  a structured lecture hall.  They took place off the cuff, with colleagues while working on a story, with professors hounding them during their office hours.  J-school taught me story structure, technical skills, office politics, the philosophy veteran journos bring to the table.  I’ll say it again: We must apply our journalistic curiosity to learning about our craft.

It’s an approach I’ve taken to my work at the Times (Look, I’m all Cali-centric now, and don’t use “the Times” to mean NYT anymore — weird).  Even as I work on a project with the LA Times masthead at the top, it feels like school.  And that’s a good thing.  You couldn’t ask for better teachers.  Yes, so knowledgable, but also patient and kind, always available, and so, so, dedicated.  Whether it’s the data analyst who’s been at this for decades, or my fellow Web devs who’ve both been at the Times for less than 3 years, they have so much to offer.  Would I dare to not ask a question because no one assigned it?  Ha! Would you not ask a question at a press conference because no one told you to?  If you’re in this industry, or even if you just have one curious bone in your body, I hope not.

Life at the Times

So, what’s it like at the Times, for those curious?  I suppose “the best working environment I’ve been fortunate enough to encounter” is not a sufficient answer.  And I’ll add my continual disclaimer.  Should all journalists be programmers? No.  We each have our specialties.  I do wish all journalists knew that this work exists, and understood the basic building blocks of what coding makes possible, and how long it takes.  That being said, I don’t believe every journalist needs to embark on this path, and if you do, it is a career shift.  It’s one I’m frankly in love with, but not everyone needs to/should go this insane.

In the end, my day-to-day life isn’t all that different from how Medill functioned.  Bring your own ideas to the table, incorporate the feedback of others.  Communicate what you’re working on to others.  Instead of working with client papers, keep the rest of the team and the editors up to date.  But mostly, frankly, just get stuff done.  The “Los Angeles Times” may sound glamorous, but that’s not why I’m there.  (And no news organization is safe in this day in age.)  In the end, I walk in every day, and type code on a computer, just like I did when I was practicing.

Each day, I improve a tiny bit (one hopes), and gain a deeper understanding about how far I’ve yet to go.  And once the myriad of files cover up the desktop background image that says, “Los Angeles Times,” I almost forget where I’m working, but always remember why I’m working.  Providing information to the public, taking full advantage of the power of the Internet.  Embrace the future.  Inform the readers.  It’s a mantra.  It is the combination of journalism and computer science, and both tool sets inform each other.  If you just want to code, you can make a lot more money, and have a lot more job opportunities elsewhere.  If you want to write, same thing (although less with the money part.)  But that’s not why I started down this path.

I’m part of a tight-knit group that already feels like a family.  I know many people search their whole lives for that.  And when we work, we mean business.  After all, these are the people who brought us California’s War Dead, and Mapping LA and Homicide Report.  It is an honor to be working alongside them.

So, our team sits, fairly quietly, and you hear some talking, but more of the clicking of keyboards, and the occasional muttering, and the even more occasional cheering when something actually works (okay, that last one’s me).  And while I am fortunate enough to get help to get conceptually unstuck — often when there’s a bug, it’s up to you to fix it.  After all, it’s your application.  Just like in Medill, no one can write your story, no one can tell you what you did to make the CMS give you what was known as “the dreaded Alt error.”  Work with others, but be self-sufficient enough to build products on your own.  (If you’re wondering, no, nothing’s been launched yet, but you’ll see what gets done, all in time.  I’ll keep the blog updated on that score.)

But before you can do this kind of work, and push beyond the curriculum, you must be acutely aware of the basics.  And that’s where j-school comes in.  Teach me AP style, teach me how to write a lede, teach me all that journalism has taught us for the past decades and centuries.  Tell me your experiences, help me form my own experiences, because I’d rather get that knowledge sooner than later.  We  can only move forward once we understand what history has taught us.

J-school provides building blocks

We each make our own experience from what we learn in j-school.  I have colleagues from my cohort who are anchors, TV reporters, radio reporters, social media producers, Web producers and much more.  Our platforms range from print to magazine to online to broadcast (TV and radio).  We all took many of the same classes, but made the overall experience our own.

So, I thank Medill, with all my heart, for all the lessons, all the grounding, all the exposure to new ideas that taught me just how little I know.  I thank Medill for the essential building blocks. I thank Medill for being a practical program, giving me a playground to try some cool new things out.  I’ve still got that playground, it’s just bigger now.  I stumble every day. But if it weren’t for j-school, and specifically Medill, I’d be stumbling a whole lot more.

I use what I learned in j-school every day, both what was in the curriculum, and what wasn’t in the curriculum.  I accept my Medill professors as my teachers, I accept the NICARians as my teachers, I accept my LAT colleagues as my teachers.  And once in a while, my own experiences or what others have told me allow me to pass on some of my knowledge.

The learning never ends. A curriculum can speed it up, but if you don’t learn how to self-teach, someone else will.  It’s scary and invigorating all at the same time.  That’s a big reason why I spend my life this way.  I feel lucky to have found what drives me, and my graduation wish for everyone is to be able to apply your curiosity to discover what drives you.  Oh, and also, the realistic caveat — I hope the job market recognizes the importance of our passions so that we all find a way to make a living doing it.