Luckily, I count many of these colleagues among my friends, and we continually learn from each other.  But for a good roundup of what’s happening, Poynter’s Steve Myers wrote a fantastic roundup of the state of news apps.  I was eager to contribute to Myers’ survey of the field.

Yet in addition, in my push for openness and transparency, I wanted to share my responses with the community as well.

As you read, consider that just today someone reached this blog searching for “how to build journalism news apps”. Surprise, surprise, the answer’s not here — yet.

After you take a quick tour through my day-to-day experience at work, please reach out to me through the comments, email, Twitter, whatever and share what you would consider in this environment if you were starting from scratch to combine data, journalism and programming in 2011.  That could be topics, approaches, hints of what to avoid, ways to maintain a good tech-and-data-friendly culture. I’m hoping we’ve moved forward enough that there are different answers to these questions in 2011 than there were in 2008 or 2010.

We’ll be working to figure out what the future of data, journalism and technology might look like, adjusting our philosophy as we go, and taking feedback from the public both inside and outside of the journalistic community.  I’d love to hear your recommendations, thoughts or cautionary tales!

How is development structured at PBS?

We have more than 30 developers in house at PBS, working on education, general audience, and kids applications.  Django is our core development techology that serves as the backbone for the entire PBS.org website.  But that’s distinct from the group that works to power our upcoming PBSNews.org website.  I run my work off of a separate server, that’s specifically for data applications, and the main developers spun up a development environment for me.  That’s common practice for various PBS projects. But it’s my responsibility to not break it, and fix it if it does break.  So, I try to keep my database queries low – that’s on me.  But creating a program to back up the database, that’s on another team.

How does a data producer spend her time?

I spend 100 percent of my time on news-oriented Web projects. The development is literally a means to practice journalism.  Separating data project development from overall site development is something we’re still addressing.  So, for instance, embedding a document on a template — that’s on me.  Creating an index of all our documents that fits in our overall system is the responsibility of a supporting developer.

Because of how we’ve emerged from the tradition of computer-assisted reporting, posting data online is often a side project to an investigation, or some sort of online directory that doesn’t necessarily tell a newsy story.  That is, they often don’t stand on their own as editorial pieces beyond what I call a “pretty data ghetto,” extending a phrase coined by journalist Matt Waite. They lack substance unto themselves.

How are we approaching the idea of data pieces?

We’re pioneering the concept of DataStories, which combine the visual power of data visualizations with the structured organization traditionally associated with data applications, and add a layer of editorial contextualization to enable Web users to learn something new about their world that is most relevant to them.

I’m also thinking about how to make data projects easier and templated for PBS member stations and show producers, as well as other public media entities.

How does our team work together?

The PBS setup we’re creating is ideal. While I could do server administration from scratch, it would probably have more issues than the setup of someone who deals with dozens and dozens of servers. And developers working on content management system infrastructure will take care of the organization so I can focus on using my editorial and technical skills to find, collect, analyze and present data for news.

Our staff consists of producers with intense journalism experience who’ll be curating content from a variety of public media sources, but there’s strong daily collaboration between me and those producers, as well as our team leaders. I’m considered a member of the team with a specialty in data visualizations and data-driven applications. Those apps are considered as much a journalistic product as an article or a video, so I’m fully integrated into this team.

How do non data geeks react to working with a hacker journo?

There’s definitely a bit of mystique to what I’m doing, but that mystery is met with strong interest and enthusiasm, not trepidation or intimidation.  It’s a very new media savvy newsroom. HTML isn’t intimidating to my new media friendly colleagues, and if people haven’t worked much with data viz/apps before, they’ve certainly heard of them.

My newsroom colleagues know what interactives are capable of, but they rely on me to know the data journalism world inside and out, and to provide my thoughts. There is a solid understanding of the time data pieces take to build — we’re not going to do ten a day.  My opinions are taken very seriously, never dismissed. I’m given a wonderful amount of freedom to “do what it is I do best”, and am encouraged to innovate.

It’s unusual in my experience to see journalists who use programming to tell stories being so thoroughly integrated into a team. All of my colleagues bring immense experience, an educated perspective and creativity to the team, and that helps us all produce the best work possible.

Explore other posts:

• March 18, 2010 -- My next move: LA Times!
• August 28, 2011 -- Thoughts on starting a new adventure
• January 18, 2009 -- How much video is too much video?
• January 4, 2010 -- My new perspective on math – it’s a journalistic tool!
• February 22, 2010 -- “Demos, not memos”: My first Django app
• December 30, 2009 -- Changes in how we travel across U.S. borders

This weekend, I pounded away at the Document Cloud API, experimenting with how we might integrate it into our continually-developing workflow at PBS News.

This taught me a) the DC folks have some of the clearest documentation I’ve ever seen, b) the DC folks are really responsive to requests, as in they added created and modified dates to the API less than two hours after I requested it, c) Don’t make test documents public, esp. now that anyone can search the public docs and d)APIs are an awesome tool!

As data mungers, a lot of the time we deal with completely unstructured data (I’m in the middle of a MASSIVE project dealing with that right now), so it’s always a pleasure when there’s an API, which stands for “Application Programming Interface,” which at least has a fairly consistent pattern.

APIs are showing up all over journalism. My former colleagues at the LAT have one for LA neighborhoods, and both USA Today and the New York Times have several, covering beats from politics (NYT’s Congress API) to literature (USAT’s Bestsellers API). They come in formats that are much easier to parse programatically than a lot of other stuff we have to deal with. And while it’s important that we structure our content as data, we should also pay attention to structuring the information we engage with as data, so we get back to the “reporting” part of database reporting. It’s something I’m trying to spend a lot more time on. And as much aof this as we have,

Fortunately, API skills serve you well whether you’re talking about newsroom content or external information, so it’s a useful skill to have.

Oh, and if you want to avoid the programming part altogether, my mentor Derek Willis can help you bring it directly into Excel. (How many entries do we think I got through without citing Derek? Not many. I’ll try again starting next post. Prediction: It won’t last.)

But even so, the overarching point is that using programming (my language of choice is Python) can help you, yes you, have the flexibility to pick information out of the API and bring it into a CSV, or a database, with the structure you impose. So, hopefully, you can start to imagine the fun I had with pulling in, displaying and remixing the documents we’ll be using to tell stories at PBS News.

Wait, there’s more! You know what’s even better? I’m part of an online journalism group that’s learning Python together. It’s a really great mixture, because I’m learning a lot from people who’ve been doing this much longer and better than I, but I’m also able to help people who are new to Python — where I was a year ago. And I love giving back to the community whenever I am able.

So, when one journalist asked how to get a JSON response from the Twitter API to go into a CSV, I was happy to mock something up, based on the DocCloud code I had put together, and share away. It occurs to me this might help you make the most of structured data from Twitter, Document Cloud, or any of the JSON-ish APIs out there, so I’ll post it here as well. Let me know how to make it better, and if you have any questions.

#To put this into a CSV, csv code adapted from this recipe 
#(http://www.palewire.com/posts/2009/03/03/django-recipe-dump-your-queryset-out-as-a-csv-file/) 
#of Ben Welsh at the LAT, who helped me do my first work with APIs:
 
 
# IMPORTS
 
 
#Make Python understand how to read things on the Internet
import urllib2
 
#Make Python understand the stuff in a page on the Internet is JSON
import json
 
# Make Python understand csvs
import csv
 
# Make Python know how to take a break so we don't hammer API and exceed rate limit
from time import sleep
 
# tell computer where to put CSV
outfile_path='/path/to/my/file.csv'
 
# open it up, the w means we will write to it
writer = csv.writer(open(outfile_path, 'w'))
 
#create a list with headings for our columns
headers = ['user', 'tweet_text']
 
#write the row of headings to our CSV file
writer.writerow(headers)
 
 
# GET JSON AND PARSE IT INTO DICTIONARY
 
# We need a loop because we have to do this for every JSON file we grab
 
#set a counter telling us how many times we've gone through the loop, this is the first time, so we'll set it at 1
i=1
 
#loop through pages of JSON returned, 100 is an arbitrary number
while i<100:
    #print out what number loop we are on, which will make it easier to track down problems when they appear
    print i
    #create the URL of the JSON file we want. We search for 'egypt', want English tweets, 
#and set the number of tweets per JSON file to the max of 100, so we have to do as little looping as possible
    url = urllib2.Request('http://search.twitter.com/search.json?q=egypt&lang=en&rpp=100&page=' + str(i))
    #use the JSON library to turn this file into a Pythonic data structure
    parsed_json = json.load(urllib2.urlopen(url))
    #now you have a giant dictionary.
#Type in parsed_json here to get a better look at this. 
#You'll see the bulk of the content is contained inside the value that goes with the key, or label "results". 
#Refer to results as an index. Just like list[1] refers to the second item in a list, 
#dict['results'] refers to values associated with the key 'results'.
    print parsed_json
 
 
 
    #run through each item in results, and jump to an item in that dictionary, ex: the text of the tweet
    for tweet in parsed_json['results']:
     #initialize the row
     row = []
     #add every 'cell' to the row list, identifying the item just like an index in a list
     row.append(str(tweet['from_user'].encode('utf-8')))
     row.append(str(tweet['created_at'].encode('utf-8')))
     row.append(str(tweet['text'].encode('utf-8')))
     #once you have all the cells in there, write the row to your csv
     writer.writerow(row)
    #increment our loop counter, now we're on the next time through the loop
    i = i +1
    #tell Python to rest for 5 secs, so we don't exceed our rate limit
    sleep(5)

Explore other posts:

• January 12, 2010 -- Data Delver: Ted Mellnik, Charlotte Observer database editor
• January 2, 2011 -- Be the LA Times’ next “Data app producer intern”
• February 27, 2012 -- Teaching Materials from NICAR 2012
• March 28, 2010 -- Data Delver: Mark Schaver, Louisville Courier
• January 29, 2010 -- Data Delver: Perry Swanson, The Gazette
• March 24, 2010 -- Bringing data journalism into curricula

My “summer” internship at the LA Times is almost over. It’s lasted from April until now, and I could stay longer. I swore you’d have to drag me out of “dream internship” kicking and screaming, as I clung to the massive walls of the still-impressive building. I will never forget the chance I was given at the LAT, and the boost it gave my career. But it’s time to move on.

I’m delighted to report that I’m joining PBS as a Data Producer for the organization’s website. Specifically, this will be a new news and public affairs component of the site. Yes, I’ll work for the national headquarters. Yes, that’s in Washington. We’re going to start something brand spanking new, capitalizing on the engaging, riveting, in-depth style PBS is known for, and translating that to the Web. I’ll get to serve as a generalist, with a specialty. Exercise my entrepreneurial spirit, but within the structure of a strong organization.

We’ll push the data visualization and application side of things, but I’ll also get to use the plethora of skills I’ve picked up along the way outside of data journalism. We’ll go on an adventure together. There’s so much I can’t be specific about, because it’s all new. But I do promise to not only keep you updated, but pick your brains for advice on what you want to see from the digital side of public television.

That being said, my hope is that we can say a few things for certain: We’re not going to be doing the same old journalism the same old way. We WILL push the boundaries of storytelling. We WILL help people stay informed and engaged with the news. We WILL help people make sense of the world, and we will do it with concrete facts. We WILL remember we do this for the users, the public. We WILL combine the best qualities of data journalism, online journalism and public television.

This transition is made easier by the fact that I have high hopes for bosses who “get” the mission I’ve been yammering about for a year. It’s not just about data, but about integration. We’ll be a nimble enough team that we can learn from each other, and not be segmented. The folks already on board have significant experience in this world, whether it’s fellow TribCo alum Tom Davidson or St. Pete Times alum Christine Montgomery — Matt Waite’s boss during the launch of Politifact. They know what data apps are. That means we push harder. Not good enough for me to say, “Hey, you ever heard of data journalism?” Got to come up with something better, something fresh and new. We’ll push each other to create the best product we can. We only fail if we fail to try.

I’ll be moving to Washington DC, in a very few short weeks — planning to land there by the end of October. So excited! After all, about a year ago, I had the pleasure of being introduced to this sort of work. It all started with one class. I discovered more and more people like me, and the community made me better. You can learn a lot from engaging online, but I miss the real-world conversations we had a year ago. I love the people I’ve met in LA with all my heart, but I still can’t help but feel that Washington is my intellectual home.

I’m excited to return to PBS. Last summer, I served as a Web intern for Chicago Tonight at WTTW, Chicago’s PBS affiliate. It was one of the best summers of my life, and I learned so much about broadcast and the Web, and the relationship/integration between the two. But that’s also when the wheels started turning. I wished I knew more about how to get data on a map. I wished we could go more in-depth with Web packages. I wished we could use pure facts, data, not just for investigative pieces, but to enhance people’s knowledge on topics across beats. Now, I have enough knowledge to push further, and to accomplish so many things I just couldn’t a year ago. And I’ve long thought public television sites were a niche ripe for the picking.

And now’s as good a time as any to acknowledge that I wouldn’t be in this position without the mentorship of NICAR folks. You know who you are. And the greatest class I ever took after Medill is the classroom I’m in every day, the LA Times Data Desk. The data analysts show me the thought process, and intense checking, that goes into every data project. My boss, Dan Gaines, listens to new ideas and floats his own, always looking at you with an intensity that shows just how much he cares about journalism, and you as a person. But my greatest teachers have been Ben Welsh and Ken Schwencke. Fellow journalist-programmers, we geek out together and support each other. And no matter how messy my code, how many mistakes I made, how much I’m still in the rough, they make me better, and never with a harsh word. If I get stressed, I only need to look over at them to see it will all be okay. Leaving them will be the hardest part of all. But they’ve prepared me to get to this point, and I’ll find even more mentors and supporters. And guys, you know you’re just a GChat or a tweet away, even if a continent separates us.

On an even more personal note, this next step represents some sort of victory. Yes, I know it’s only the beginning. But at the same time, it was this sort of position that I’ve been working for all along. I didn’t learn programming so I could write some awesome loader, geek out about server loads, etc. It’s all fascinating, sure! But I did it, I do it every day, and I will always do it — for the journalism. It’s a tool, just like a photo journalist using a camera. That doesn’t make him or her a wedding photographer. They practice journalism through photographs.

We see so many posts in the journo-community, asking if journalists should learn programming, if programmers should learn journalism, what do we name people with this hybrid?

But, here’s the question I forgot to ask. Once you start getting all these skills, where can you go to use them? Some of my mentors received quite a few late-night emails, listened to quite a few rambles. Where is the place that understands my tech skills are deeper than uploading my story to a content management system? Where is the place that understands I could add features to your site, and do general tech/IT maintenance all day long, but I have these journalism skills. The tech skills help to tell the story. It’s not one or the other. I consider myself equal parts programmer and journalist, journalist and programmer.

I had a fascinating job search, and managed to never file a single application, without someone from an organization reaching out to me first. That’s not what they tell us will happen at Career Services! If you want to pursue this career change, there isn’t a better time. Still, finding the right organizational match is no easy task. And I can’t tell you how ecstatic I am to have found that match.

In a world where we decry the death of journalism on a daily basis, where we’re afraid we won’t get jobs, I say there IS hope if we hold out. If you believe in what we do, follow your interests and skills, you’ve got a better shot. We must always push onward. And I, well, I never believed that less than a year after graduating Medill, not only would someone pay me to do what I love, but I would get to do it at an organization that instilled my own love for education. It still seems like a dream that’s coming true. I never believed I’d get to a national organization this fast. I never believed I’d get to take my ideas outside of a blog, outside of an academic paper, outside of internship after internship. Doesn’t mean I’m right, I’m probably not. But if the ideas just stay incubated, and are never considered professionally, what good is that?

We don’t have time to waste. The moment is now. We can’t afford to wait, the industry’s already behind where it should be in 2010.

So, I promise anyone reading this, friends I’ve known since elementary school, my dear supportive parents, teachers from all levels of schooling, fellow journalists fighting the fight for truth and knowledge every day, I promise that I will give this everything I’ve got. I recognize the preciousness of this gift. And on behalf of young journos everywhere, PBS, thank you for believing in us, in our potential. Thank you for the opportunity to experiment on a national playground. We only move forward by trying something new.

There will be no resting on laurels, there will be no complacency. Thank you all for your help so far, and I know we’ll be calling on each other in the days, weeks and years to come.

But for now, journalism must push forward. Let’s get started.

Explore other posts:

• April 11, 2010 -- Data Delver: Paul Monies, The Oklahoman
• April 7, 2010 -- Columbia’s new joint MS: Good start, but no panacea
• April 5, 2010 -- Women with coding skills are no Passover miracle
• February 21, 2010 -- Is a flat text file or a database right for an app?
• November 25, 2009 -- The limitations of SQL and Access
• January 18, 2009 -- How much video is too much video?

I’ve spent much of my “summer” internship at the LA Times (ranging from April through present) working on a campaign contributions searchable application, to help the public explore the money that’s financing various propositions in California. The first step was getting the Prop 8 application out back in late June. I’m really into the idea of templating data apps recently. Make a custom interface, but make it resuable, so it doesn’t have to stop as one application. It’s like how we already use third-party source tools as frameworks (Dipity, ManyEyes, etc.) We should expect the same, if not more, of custom solutions we create ourselves. So, the original idea for the Prop 19 app was to apply Prop 19 data to the Prop 8 code. We did that, but as is my normal mode, more features suddenly found themselves on the list. So, in the end, we have two frameworks for campaign contributions now: the simple version and the more advanced version.

One of the large advancements in the second version is that every single donation is listed separately. In Prop 8, the legacy data passed down to me had aggregate donations, which is helpful, but means users miss out on certain nuances. So, when we were able to pass each donation in separately, I believed it was key to make each one linkable and sharable. Not only does each donation have a separate link, but so does every donor, city, state, zipcode and committee. This means if people find something and want to blog about it, share it with a friend, draw a media organization’s attention to it, whatever, they have a way to point someone else to that exact spot. We don’t do enough of it in the data apps world. Web frameworks make this easier, Flash interactives do not.

Creating permanent links to records and aggregate pages of our data is essential. It’s how we combine our searchable databases with other reporting forms that provide even more context. Did you write a story about someone you know gave a lot of money to a campaign? And there’s a searchable database to go with that campaign? Don’t just tell me they donated a lot of money, show me.

Just an example of one way we can change our mindset as reporters, Web producers, developers or whatever it is we do. Find the minute detail, make it shareable. Because while you may only care about record #1157, someone else cares much more about record #1158. And both of you should be able to find that piece of the story that matters to you and use it. That’s data for personalized news, and yet another lesson I take away from my time at the LAT.

PS — If you want a random Django tip (come on, you know you do) {% ifchanged %} is my new favorite template tag. If you have a big list you’ve pulled out as a query, and it’s sorted, but you want to separate the list into subcategories, you want this. Figure out an attribute that’s going to change, and enter that info as a label, inside the {% ifchanged %} tag. It’ll show up when the code and text inside the tag changes — makes sense. This tip has made an ongoing project a lot easier, which I’ll be able to chat more about in a few weeks once it launches.

Explore other posts:

• February 9, 2010 -- Treemapping Gov. Quinn’s State of the State
• March 8, 2010 -- Data Delver: Jennifer LaFleur, ProPublica
• January 18, 2009 -- How much video is too much video?
• March 26, 2011 -- My (quickly formed) vision for a journ-prog curriculum
• March 8, 2010 -- Data Delver: Anthony DeBarros, USA Today
• January 11, 2010 -- Importance of combining data analysis with context (reflections on readings from week two)

That’s why I absolutely loved last week’s post by TBD’s Steve Buttry on this isssue. Read it, seriously. I was going to write up my comments, but I’ve got something better.

I started discussing this issue with my father Mike Minkoff, who happened to be in town last week. And he helped me realize the struggle our education finds itself in, well, it’s nothing new.

Mike is a computational scientist at Illinois’ Argonne National Laboratory, using his cs skills to enhance the work of his fellow scientists, and enables them to explore problems whose complexity or sheer magnitude is beyond the reach of mere humans. It’s applying tech skills to another field. And that’s a little bit like what we do as journo-programmers, bring the cs to the journalism.

But when he was going to school, computer science curricula weren’t all that certain. And we can learn from that transition. I’ll let him take it away.

I am a computational scientist with an interest in data in journalism. I find a number of exciting and interesting parallels between the current evolution of journalism as discussed in this post and the early history of the field of computer science when I went to grad school in the 1960s.

I studied computer science at the undergraduate level through Ph.D. between 1963 and 1973 at the University of Wisconsin and Princeton University. During that time, there were relatively few schools that had curricula in computer science (Cal Tech, MIT, the University of Wisconsin and a few others) and the programs were principally at the graduate level. In fact, schools didn’t always call it “Computer Science” — some of the courses were listed as “Programs in Numerical Analysis,” as at Wisconsin.

There are two points I’d like to comment on:

1. In the 1960s, as computer science programs developed there was a fundamental debate regarding what should be taught and where. On the one hand, grad students and faculty felt that learning a programming language was of utmost importance in order to get a job after graduation. On the other hand, others felt that theory is more important so the student can develop and move with the technology rather than become stagnant as technology advanced. Later on, the theory approach justified the placement of computer science in major universities rather than trade schools. There is a parallel here to the “basics” vs. “technology” issue in journalism.

   As this debate developed, the professional societies (notably the Association for Computing Machinery) established a committee to define a core curriculum of an B.S., M.S., or Ph.D. and allow for electives that could be locally defined by the institution. The courses in the core represented the key areas of computer science and are reflected in their decendents today in simulation and modeling, data analysis and operating systems theory:

   • numerical analysis and mathematical optimization
   • systems programming
   • artificial intelligence and automated reasoning

   Of course these are historical topics and just as the technology used in journalism will develop in new ways, so did these areas.

   As to the theory vs. language issue the obvious answer is both. In all fields, as the subject advances there are always new topics to integrate into a curriculum. I would expect that a journalist be as good a writer as in the past, but also he or she must know how to utilize the technology of today (and tomorrow). I would hope that in the rush to move with technology the programs and faculty of traditional journalism is not neglected. There is the danger of being so committed to technology that the core of basic journalism is lost.

2. One of the most exciting parallels to me has lasted my career. The pioneers of computer science were my teachers, mentors and lifelong friends and advisors just as the present digital journalism teachers and mentors are for their students. Graduate school is not just a spoon-fed set of courses, but a place where inquiry and learning can be nurtured. One of the commonalities of computer science of the 1960s and the technology journalism of today is that once in a generation a field provides the opportunity to meet and learn from the people who created the field. This gives the mentor and student to opportunity to rise to the challenge of working in a new and growing field. While students in both areas need to decide for themselves what they want to get out of a career, for me it is exciting to meet and learn from the people who created the field. As a student, I felt that the difference between studying math and computer science was that you could meet and talk with the developers and authors of the textbooks. Of course today it is the blog rather than the textbook, I guess!

I thought this was definitely something to ponder, and I hope that in a generation, I’ll be the one telling my children about the parallels between what we consider modern journalis, and some new transition their field is going through.

Has this gotten you thinking, too? That’s what the comments section is for!

Related posts you might enjoy:

• April 7, 2010 -- Columbia’s new joint MS: Good start, but no panacea
• January 31, 2010 -- Data Delver: Matt Waite, St. Petersburg Times
• November 24, 2009 -- A letter to journo-programmers: Teach me, inspire me

Tomorrow, Wed. Sept. 8, we’ll be discussing “Finding the story in the data” at #wjchat, and I have been tapped to host. (One might ask why, they must really be running low on people….I kid, I kid.)

This issue is extremely important, and one that must be addressed by the Web journalism groups. There’s a difference between telling a journalistic story, but lacking the data and displaying data in an aesthetically pleasing way that doesn’t really tell a story. The best data journalism does both. It’s rare, and I believe we all need to work harder at doing more of it. A tall order for someone to do alone, or even a team, but if the community puts its collective head together, we’ve got a better shot.

I couldn’t be more excited to discuss some of the issues I’ve been spending much of the last year internalizing. I have some better ideas about this now than I did back last September, I hope, but it’s also one of my favorite topics to ponder, because there are so many ideas I know I haven’t considered. I hope to report back with a nice compilation of links and thoughts once we tap into the community’s knowledge, but for now, I’m just spreading the word.

If you read this blog, but aren’t familiar with #wjchat, it’s a gathering on Twitter on Wednesday evenings where we discuss various aspects of online journalism. It ranges from social media, to video, to how to get a job, to data. I love all the facets of journalism, so it’s a great way to guarantee yourself some great conversation and learning all wrapped up into one package. At 5 Pacific, 8 Eastern, on Wednesday, that’s tomorrow, just hop on over to Twitter and follow the hashtag #wjchat. I find it easier to follow via tweetchat.com, which gives you a little chatroom that has a nicer interface than twitter.com — the conversation can get fast and furious.

In the meantime, if there are specific topics you’d like to see discussed, or questions you’d like asked, get at me before, or during the chat.

And if you’re one of my dear NICARian mentors, it’d be really fantastic if you could find the time to drop by. Anything I’ve picked up in my career thus far is minimal, and I owe it all to the collective wisdom of those who’ve taught me what I know as a journalist and a programmer.

See you in the virtual space!

Explore other posts:

• November 22, 2010 -- Using Google spreadsheets as your database
• September 6, 2010 -- What’s regex again? Why should journos care?
• February 28, 2010 -- Django app #2: Conquering forms and Google Maps API
• February 17, 2011 -- An inside look at life at PBS News
• May 24, 2011 -- Oh, hi, blog readers — it’s been a while!
• January 1, 2011 -- Looking back on a year: What now?

You may have heard the term “regex” before — short for regular expressions. And compared to the way we talk and write, they’re anything but regular. But choosing between writing the “regex to rule them all” as I’ve been calling it, in a not-so-veiled reference to Tolkein’s Lord of the Rings geek trilogy, or copying and pasting tens of thousands of records by hand, I’ll take the former any day.

So, if you want to get through a large, unstructured set of data quickly, and make sense of the unsensical, regex is a great way to go. But your pattern had better be consistent. Computers don’t do well with exceptions, after all.

And if you’re a journo who doesn’t want to throw yourself into programming, here’s the one thing you should take away. If you have a document that could be a database, maybe there’s some dollar amounts, and you’d like to know the biggest one and what category it’s associated with, why not go to your programmer and ask if it’s something regex can help solve? Worst case scenario: your programmer says it’s not the right fit.

Some of my favorite explanations of regex are:

• Google’s tutorial in text and video
• Better explanation than I can give is here on Ben Welsh’s site, as per usual
• Simple syntax reference
• Python/PHP regex tester – Plug your text and pattern in, and it’ll tell you if you messed up.  My best friend as of late (which says something about my social life, but I digress).

So, a mini foray into how this works. A poorly-written regex looks like this:

‘[0-9]{1}\)[ ]{0,20}([A-Z0-9\n ]{1}[\w\d\n\(\)\"\'\.\:\,\;\-" /]*[\#]{0,1}[\w\d\n\(\)\"\'\.\;\:\,\-" /]*\.{1}?[\"]{0,1})’

Stop screaming. A less frightening one:
‘\$([0-9]{1,2}[\.]{0,1}[0-9]{0,2})’

Both of those are hand-crafted, newbie coder at work examples.

The second one isn’t any more intelligible.  That’s not better? Let’s break it down.

Some simple tips:

• A regular expression is a string of numbers, letters and special characters. You use your language of choice to parse it (I’m partial to Python, access the parser by importing “re”. Easy enough to remember.
• Define the range of characters you are looking for in brackets. [0-9] means looks for characters that are digits between 0 and 9. Do the same for letters A-Z or a-z, or specify specific letters: [t] means t is the only acceptable character.
• After you define your range in brackets, follow it by saying how many times you are looking for such a character. {2} means it needs to appear 2 times to match, no more, no less. {0,2} means the characters can show up 0, 1, or 2 times. Include 0 in your frequency if you want to be more flexible, and allow this range to be optional.
• Type . to include all characters.
• Type * in your frequency to include the character 0 or more times, + for one or more times.
• If you only have one character in your range or frequency, you can leave your [] or {} out.
• If you want to define a start and end to your pattern, but not include that in what the computer returns, or spits back to you, enclose the part you care about in () and the rest will get junked
• If you are including a special character in your range, such as searching for all ., you need to escape it, or use the \ key in front of it.
• Escaping characters that don’t need to be escaped can’t hurt, at least in my experiments. But forget to escape a character, and you have hours of debugging ahead of you, because of a missing \. Yes, I speak from painful experience.

There’s more, but that’s a good start. So, what does this mean?:

‘\$([0-9]{1,2}[\.]{0,1}[0-9]{0,2})’

Search for strings of characters that start with a $, are followed by one or two numerical digits, followed by 0 or 1 dots, followed by 0, 1, or 2 digits. What does that spell? A price that conforms to AP style. It’ll catch whole numbers ($25) or those with cents attached ($25.50) or prices below $10 with cents attached ($1.25). Pull your budget numbers out of that ghastly text file the city council gave you.

How do I use that regex?

Okay, okay, good point. I’ll tell you how in Python. Let’s say we’re searching for all prices in a file.
Here’s some line by line commented code to help you through it, and save all the prices to a CSV spreadsheet.

#Open the file you're reading from
file = open('raw_file.txt', "r")
#Read the file into a variable, so you can access it
file_to_search = f.read()
#Save your regex pattern into a variable.  Use the r before the opening quote to tell the computer it's a raw string, otherwise it may try to convert it into unicode.  I've noticed Python has a tendency to interpret my unicode strings as raw, and my raw strings as unicode, unless I'm explicit about what I mean.  And as the <a href="http://www.python.org/dev/peps/pep-0020/">Zen of Python</a> says, "Explicit is better than implicit."
price_pattern = r'\$([0-9]{1,2}[\.]{0,1}[0-9]{0,2})'
#Create a list of all pieces of text in your file that match the pattern.  List the pattern as the first item in the parens (otherwise known as an argument), list the file you're search in as the second argument, add the third argument re.DOTALL if you want the . character, which signifies all characters to include newlines -- this tripped me up at first
price_list = re.findall(price_pattern, file_to_search, re.DOTALL)
#Open a CSV object, which you can save list items into, the a means that every time you write a new row, you append the new rows of text, and avoid overwriting what's already in the file
writer = csv.writer(open("prices.csv", "a"))

To figure out the rest of how to write this to a CSV, modify teacher Ben’s recipe — it’s how I figured it out. After all, this is about regex, not CSV writing.

Or, do whatever you want with that list. You could sort it to find the highest or average price…possibilities really are endless.

Oh, and what went wrong with that first, horrificly long example I gave you?  Didn’t know you could use . to signify all characters.  You try specifying every single character that could possibly be used in a string of text.  You’ll always end up forgetting at least one.  The lesson there: the computer is better at catching all instances of something than you are.  Don’t try to compete.  Accept its’ strength and move on.  Your strength: The computer can’t read the human language you’re parsing.  So, point one for us.

Explore other posts:

• February 22, 2010 -- David Carr at Medill
• March 26, 2011 -- My (quickly formed) vision for a journ-prog curriculum
• September 21, 2010 -- The power of linkable data apps (my second LAT app)
• March 8, 2010 -- Data Delver: Anthony DeBarros, USA Today
• March 8, 2010 -- Data Delver: Matt Wynn, Arizona Republic
• August 18, 2011 -- Hope: Recognizing what you can and can’t change

So, there’s that. First launch!

My checklist:

• Figure out what part of journalism inspires me. CHECK (Data, programming)
• Find place where I can learn about it from others, but have freedom to try out my ideas and learn from knowledgeable and patient folks. CHECK (LAT)
• Use this opportunity to learn enough about programming to create at least one app. CHECK (See first line of post)

That was a good time.

I’m planning a post on best practices for creating a searchable database application like this, with examples, and technical geekery, and all that, and I’d love your thoughts in the meantime.  Let me know if you have specific questions.

But in a moment of self-reflection, I’d just like to say this.

I was sitting in my apartment this morning, and found that a butterfly had somehow gotten in (blame a broken screen, maybe).  It sat on my windowsill, facing the outside world.  It would feel around the window, trying to find an opening.  After circling the perimeter of the window, it laid back down, almost motionless, just searching.  An hour later, the cycle repeated.  Wanting to help the poor butterfly, after watching it struggle, I tried to capture it in a small box to take it downstairs.  It was too smart for that.  Fine.  What actually worked?  Covering all the windows and opening the balcony door.  You can’t force the butterfly out, even though you’re helping, it had to find the path out on its own.

Why do I bring this up?  Am I slowly going insane after spending the day with Dive Into Python (rec’d)?  Perhaps. But I have a point.

Let’s take the common journalist, yearing to know more about data-driven apps.  (I can’t be the only one, I just can’t!)  I search for the answers, voraciously read, alternate between laying still while observing the space and wandering around the perimeter, feeling it out, trying to dip my toe in the water.  I jump into the pool, head-first, realize it’s not working.  Can’t do it on my own.  Many people who’ve already made the journey try to help.  But they just can’t just give me the knowledge, can’t just take me to my goal, can’t just make it happen.  They must lead me there, give me the same resources they used.  Recommend the books, the sites. But in the end, I’ve got to find my own way out of the house, the prison of not knowing enough coding basics to begin.  Pushes in the right direction, guiding me away from those false starts, this is essential.

I’d just like to take a moment to thank the experts, who guide me away from the false starts, who don’t give up when I keep turning the wrong way.  In particular, in terms of the programming skills I’m learning, it’s Ben Welsh (who I happen to have been working most closely with), who exercises that kind of patience.  Who never tells me not to fly into that closed window, because while it was closed for him, I might discover a way to get out.  Of course, I don’t, and he’s no less interested in helping me get back on course. Thanks!

Still a long way to go.  So, what’s changed?  Now, I’ve gotten outside.  I made my first app.  When someone asks me what steps make up a project, how long it’ll take, I can give a legitimate answer, or know how to find one.

I still remember wondering if I could ever build an application.  And Derek said to me, “There’s only one way to find out.”  Derek, who I used to bug on a daily basis about this stuff.  Derek, who I haven’t asked a Python/Django question of since March.  I’m the LAT’s problem now!

Well, as Derek said, find out we did.  Time to push it further, I say.

There’s a whole world to explore now that I’ve escaped being stuck inside.  LAT folks, prepare yourself, the questions you’ve seen so far are only the beginning.

Explore other posts:

• January 4, 2010 -- Reflections on Visualization Theory (Data viz readings, week 1)
• January 18, 2009 -- How much video is too much video?
• November 25, 2009 -- The limitations of SQL and Access
• February 21, 2010 -- Is a flat text file or a database right for an app?
• February 28, 2011 -- Why I really love NICAR (and the New York Times)
• January 25, 2009 -- Analysis of Interactive Features In WaPo’s Inauguration Timeline Map

I’ll just put it this way.  Andy, you’ve been an inspiration.  Watching your work while I was a student at Medill, and how much you enjoyed it, I knew some day I could do anything, if I could just set my mind to it, and find supportive mentors.

It’s a parallel thing.  Andy gets Matt Waite and Jeremy Bowers at the Times of the Southeast (St. Pete), I get Ben Welsh and Ken Schwencke at the Times of the Southwest (LA).  Match good mentors with journalistic enthusiasm and obsession, and you’ll get somewhere!

We both get to bring passion, skill and journalistic knowhow to the table.  We bug the people with the tech knowledge until we have a moment like this.   Maybe someday I’ll report in the field and code like Andy does.  But for now, my reporting consists of investigating the nuances of the still-large ship that is the LA Times, and looking at how it can be even better.  And at this point in time, I wouldn’t change it for the world.

It seems appropriate to post this interview with Andy now (Yes, I’m making excuses for delaying this for months and months….).  I chatted with him in March, just after the launch of his first Django app – MyLawmaker.  And I just launched my first one about two weeks ago.

I started these Data Delver interviews knowing no one in the field other than Derek Willis.  (And if you only got to know one person, he’s as good as it gets.)  And now, I feel like I know so many more.  And for some reason, I get to be a part of their ranks each day.  It’s a privilege, an honor, and an adrenaline rush like no other.

I’ll let Andy take it away in his own words now.

This transcript of my interview with Boyle is a part of my continuing series I’m calling “Data Delvers,” where I pass on transcripts, summaries, quotes and audio clips from conversations with journalists using technology to find, analyze and convey data-driven stories and/or projects to the modern audience.

Extended transcript

Why is having an understanding of data important?
When we have meetings to discuss story ideas with the group I work with we mainly focus on breaking news on the web.  When somebody says something, it’s really nice to be able to go, “Oh, I know how they keep those records so I know exactly how you can ask questions to find out how many total XXX has happened this year.”  It really helps to have a background, it gives you a chance to dig deeper and get more context.

Is St. Pete working to get other reporters to get more up to speed on this?
I think there’s quite a few reporters who do have this type of skill or do have the basics.  They can use Excel; they can use Microsoft Access.  Actually, there’s a reporter who did a really awesome story on leaky underground storage tanks and where they are located.  My compadre, Darla Cameron, does a lot of GIS stuff, so she’s also a data nerd, like me. She was able to help him with that. A lot of people use databases on a pretty average basis and have some CAR training, but not as much huge stuff like the Wetlands project that the St. Pete Times did with Matt Waite and Craig Pittman a few years ago.  But things on a smaller scale are happening.

Going back to the MyLawmaker project, what was the genesis of that?
The genesis is that on Monday the state legislature starts so we print this thing every year in our Perspectives section which runs every Sunday.  We print opinion pieces and stuff like that in a section called ‘For a better Florida’.  One of our politics editors, Amy Hollyfield,  had sent an email out to one of the higher-ups  saying that there’s this project from the New York Times called Represent and another one at Oregon Live –  Your Government, where you type in your address and it shows you your state lawmakers, congressmen, city lawmakers, etc.  She asked if it would be possible for us to set up something to help people find their state representatives and state senators from their address?  This somehow made it to me and I go, “Yeah, yeah, that can be done; it’s possible.”  At that point I had no clue in my mind of how we were going to do that, but I knew it was possible because obviously someone else could do it. So we had the shape files; Darla already had the information.  Somehow we could write a program that would geocode your address and would ask what you wanted to see, and it would print out what you wanted.

So that was basically last Thursday and it is now Friday and we are ready to launch.  Thursday, about 3:30 in the afternoon, it was like, “Hey guys, do you think you can do this?”  I was taught a long time ago that if you think you can, try it. This is something that I really wanted to do.  It was an opportunity to try this stuff   I’ve been working on.  I’ve been working on some Django and Python with Matt Waite and Jeremy Bowers, so it was nice to spearhead a project and have ownership on something and work with a group of awesome people like Darla and Lee.  I’m really jazzed.

So how did you start getting into Django…just by being around Matt and Jeremy?
Yeah, that’s totally it.  He’ll [Matt Waite] browbeat you into believing what he believes.  When I went to the Indianapolis NICAR conference he had a thing, a bootcamp, for three or four of the days on frameworks, so that was Django and the basics of it.  And I remember when he showed it,  I thought, “Man, that’s way easier than making hundreds of  individual web pages and then having to edit the HTML in each one. “ I remember being a kid and having an AOL members’ web page and how horrible it was to have to update every single HTML file and I thought “Wow!  This is something that automatically creates this stuff.   That is sweet.   I need to learn this.” And that was in March of last year.

And then after that it was just me attempting to learn stuff and breaking everything and being really afraid that I’d destroy my computer and not becoming afraid of the terminal. It was a very long process.  There were a couple of projects over the summer that I worked on.  One was FCAT, which is an aptitude test given to students in public schools in Florida. We were able to make a searchable database for that.  I wrote the models on that, and then they did the rest of the heavy lifting.  I guess that was the first project I had any sort of help with.  Then we later had a project for high school sports, called Home Team, which is totally awesome.  I was able to see the inner workings  and it was really cool to see the process of how they were developing it and how it was being set up.  I wanted to build something, so when the opportunity came along for MyLawmaker, I jumped at the chance to do it.

So what did you end up using for the geocoding?
We used Google.    There’s a limit of 2,500 times per IP in a 24 hour period which is an issue if you hope to get a lot of hits on  your website.   So what we do is we actually pass it off to JavaScript on the client side, so the server’s IP address comes from the user’s computer. Not that we expect 2,500 hits a day, but we have no clue as to how many hits it will get.   But we thought it’d be nice just in case.  We don’t want people trying to use it and it’s broken.

Have you had requests from the community for anything like this?  It doesn’t have to be specifically MyLawmaker, but just more interactive stuff?
I think a lot of people really like it.  When our FCAT project launched in the summer, it got a ton of use.   It still kind of does.  A lot of those projects that are usually attached to stories initially get a lot of hits.  With a project like this once we told some of our colleagues about it, they were like  “Wow, that’s a great idea.” With the state senate website, you can only search by zip code.   Then it will tell you the districts that are in that zip code and you have to click on that and see if you are located on a little map of the district, so ours is actually a little better than part of the state legislature’s.  But I guess that’s the goal.  We want to make sure that people can find the information and find whom to call if they have an issue.   A lot of people were quite surprised…they didn’t believe who their state legislators were.  Said, “That can’t be true.”  But it was, and they wouldn’t have known otherwise.

Why do you see this type of work as important?
It’s interactive.  When you pick up a newspaper, you read a story and it kind of ends there.  What’s really cool about this is it helps you find out broader information.  You can go searching a given database online and become more aware of your community. That’s part of what we do, as journalists, and as news organizations, we try to inform the public.  To do it in a way that’s automated, that doesn’t require our constant supervision or our constant writing of stories, helps because it frees up time for us to do other stuff.  It still gives people information and more context when it comes to stories.  I believe that a well-informed populace is much better than an ill-informed populace.  It’s also really cool and lots of fun to do.

Do you enjoy living in both worlds – the reporting and the developing aspects?  What are the advantages and disadvantages?
Yes, I do.  I guess time is a disadvantage in everything we do.  Part of it is because of the reporting, I can come up with projects that we can do.  I think if I was only a code monkey and I was never out on the streets, I wouldn’t find out what data is available or what data we can build on our own. It also helps you keep track of the news.  Like this is being built in response to the fact that the state legislature is meeting on Monday to start a new session. If I was just a reporter, I wouldn’t be aware of different ways to think and inform the public.  And if I was just a developer, I would be a little blind to some of the opportunities that there are to inform the public, and make cool stuff.

Any other advice?
Something I wish I’d been told when I was younger is failure happens, especially with this sort of stuff, so you really need to get used to it, and we all fail.  Your failure is a failure, unless you learn.  Then, your failure becomes a win.  If you’re always trying to learn stuff, as a journalist, if you’re always working to develop different skill sets, whether it’s Web development, normal computer-assisted reporting, GIS, narrative storytelling, it all helps you, it all helps the other things.  Whatever new skill you learn in journalism will add to your big palette of rocking.

Would you recommend all or more reporters get into the coding side of things?
It doesn’t hurt.  It doesn’t hurt to understand the basics of data.  If you’re a city hall reporter, it helps to know basic Excel, it helps to know Access. There’s a ton of cool stuff you can do because of it.  If you are a feature writer, and you write longer narratives, and stuff like that, you can still find stories through databases, you can still find extra people to interview through those methods.  It helps in every aspect I think, just to have a little bit.  Whereas I have made the intense plunge into being a total nerd when it comes to this stuff, if that’s what you want to do, that’s what you want to do.  I don’t think that should be a requirement, but I think you should have at least the bare bones basics of what’s going on, because knowledge is power, and knowing is half the battle.

Related posts you might enjoy:

• April 11, 2010 -- Data Delver: Paul Monies, The Oklahoman
• April 5, 2010 -- Women with coding skills are no Passover miracle
• April 5, 2010 -- Data Delver: Phil Meyer
• March 28, 2010 -- Data Delver: Mark Schaver, Louisville Courier
• March 8, 2010 -- Data Delver: Anthony DeBarros, USA Today
• March 8, 2010 -- Data Delver: Jennifer LaFleur, ProPublica

I didn’t retweet it for a few hours, because, for once, an opinion didn’t jump into my mind.  If you were in my classes at Medill, watching me wince in pain every time we stopped short of delving as deep as we needed to, complaining about the pervading fear of code in journalism, if you knew me, you’d think I would jump up and down, and say, “Now, why couldn’t I have that?  It would have been awesome!”  People are saying it helps people who aren’t coders become coders.  While some pointed out that Medill had this first, our program was for programmers to become journalists.  And how does that serve liberal arts majors who want to code for journalism?  Why isn’t it listed as a course option?  Wouldn’t this Columbia program make life so much easier?

Let’s put my biases out front: I’m a recent Medill grad.  I am a programmer-journalist.  I do not hold an interactive concentration from Medill.  I took no programming classes within Medill.  I did not participate in the programmer to journalist program. I didn’t get that tuition-free deal. Medill had this first, but didn’t have a curriculum for turning liberal arts majors into journo-coders.  And, you know what?  I don’t think they should.  You get something from pushing through it, pursuing your own interests, learning how to learn.

Sorry, I’m just not on board with a joint-degree program.  Take a few coding classes, but that can be done by cross-registering outside your department at many schools.  But understand that any coding you’re taught may be replaced by a new language years, weeks or days later.  We shouldn’t be taking too many classes in coding, but learning how to learn quickly from the Internet.  If, let’s say, you need a JQuery library to perform a certain task, know how to adapt tutorials, don’t think, “Now, this wasn’t in that degree I got, what do I do?”

As Poynter’s Dave Stanton pointed out, there’s a lack of statistics in this curriculum.  This is absolutely essential for people who will be concentrating in data mining, or data-driven app creation.  Perhaps this is easily remedied.  But it signals to me that there are probably similar holes elsewhere in the curriculum, holes I don’t understand as well.  Hopefully, these will be patched in future iterations.  I do have some concerns about the first class to go through this program, though.  But we must change curricula, or die.  And there will be some successes, and improvements to be made, with each new attempt.  Kudos to Columbia on taking a chance!

I also hope there’s a way to pick a specialization within the program.  This is a tall order to complete in a year, and you may excel at data analysis, or visualization, or automating news aggregation, or something else.  But they require a distinct set of skills.  I would approach this as I would a general journalism curriculum.  Get familiar with all the different options, figure out what you love, and get really good at it.

And perhaps the biggest problem of all is the misconception that by putting computer science and journalism classes next to each other, even with integration, we’ve created a recipe for programmer-journalists.  You don’t need to struggle to figure out what to learn, we can do it for you.  I can’t help but feel that this encourages complacency.  If I just listen to my advisor, and follow the steps laid out for me, I can do it.  But you must never stop learning.  What do you do when the syllabus runs out?  It’s essential to learn to learn what you need to know, and then do it.  If students don’t search beyond the curriculum, we may stop moving forward.  The world is moving too fast to create a finite cs roadmap at this point.

There’s something to sitting down and hacking something out.  Part of the beauty of coding is that anyone can do it, follow the steps, learn the language, and you WILL get it.  This program may give students a push, but it’s by no means essential.

So, sure, let’s encourage journos to take cs classes, talk to cs professors.  But gaining a masters in both fields, in little more than a year?  It’s a good option, but far from the only one.  It is only the beginning.  If you go to another school, you could make your own program, take classes in both departments.  This is formalizing what some of us are already doing.  If you enter this program, remember to push your faculty resources even further, and don’t believe it’s a panacea, cure-all, complete road map to what we need done.  This is the first rung up the ladder. There are other ways to climb, and even once you get to the first rung, in the end, it’s up to you to go ever higher.

Related posts you might enjoy:

• September 16, 2010 -- Journ curricula’s need for transition isn’t unique
• November 24, 2009 -- A letter to journo-programmers: Teach me, inspire me
• May 3, 2010 -- Note to self: Real world journo-coding lessons
• March 29, 2010 -- Using Javascript for interactive Google charts
• March 28, 2010 -- Data Delver: Mark Schaver, Louisville Courier
• March 25, 2010 -- Self-teaching data and programming skills