Friday, July 11, 2014

Statesman looking for a news applications developer

Edit: This is filed. @adchavez is rockin' it.

We're formalizing our news and data interactives work at the Statesman and we are looking for a developer to join Rob Villalpando and I to form a new News Interactives Team (or some other snazzy name we think up.)

If you read this story, and say, “Yep, that’s me” or “I want to make a difference like that,” then you are the kind of person I’m looking for. If you can buy into this philosophy and help us do some of the same type of work they do, then you are a good candidate. If you understand 80% of the the posts on that blog, then you are probably a really good candidate. If you understand it AND were already a regular reader of that blog, then OMG CALL ME!

I’m looking of someone who loves to create and to share what they know and what they learned today. We’ll soak it up. We also have a lot to offer, so you’ll grow and learn, too.

Here is the full job description: Web Applications Developer. The whole application process runs through that.

If you just learn more about the job and see if you are a good fit, you can reach out to me:

Christian McDonald
Data Editor
Austin American-Statesman
cmcdonald@statesman.com

Monday, June 23, 2014

The immersive story presentation


While the New York Times' Snowfall story may not have been the first immersive story presentation -- as I like to call them -- it was the first that caught all our attention, and that of the Pulitzer committee. Since then, there have been an avalanche of such projects (pun intended) and this past weekend, the Statesman added our own: Drugs follow Eagle Ford energy boom.

Eagle Ford drugs did not break any new ground. In fact, it was a baby step for us built upon the work of others, inspired by the Globe and Mail's Magnetic North feature. We had a desire to get into this space, a good story by one of my UT-Austin students, Michael Marks, and some excellent photography by Jay Janner.

It was a method for us to discover the challenges we might face doing more of this kind of storytelling. It's really just the first step. My aim as the editor for our interactive team is to tell stories in completely different ways than a "long story with a bunch of pictures." I personally hold NPR's T-shirt project as seminal inspiration. We'll get there. (I could really use some help ... come help me build something bad ass.)

So, here is what we used:

  • Foundation: This is the responsive design HTML/CSS framework we used to build Eagle Ford drug (and our XGames package and the Austin Homicide Project and some others. It gives us flexibility to focus more on the content than on how to present it. Our goal is to be mobile-first, and this code base allows us to do that quickly. I started with nothing on Tuesday and had everything finished by Friday, including some new development I hadn't done before.
  • Slick: A javascript plugin that allowed us to embed a swipeable? gallery within the story. I had a BUNCH of great photos that I wanted to include. Slick was one way to do this, but we'll keep experimenting with this an other JS plugins. This brought in quantity, but I didn't do that great a job in giving these multitude of great photos the play they deserved. We need BIGGER. Some kind of lightbox treatment or something. Also, we have some issues with Firefox on a desktop where the usability is a bit buggy.
  • The inset concept I lifted liberally from the aforementioned Globe and Mail presentation. I didn't see anything like this native in Foundation, so I looked at how they accomplished this and tried to build in that functionality to our Foundation framework. It worked for the most part, but there are some issues to work out with IE where our insets don't break the right margin of the text.
I used this project to build a template for future projects, but as you can see it still has some issues. I'm not sophisticated enough as a coder to share all this on Github, but we'll get there eventually. I'm sure there are plenty of other better places to start out there anyway.


Saturday, May 17, 2014

Dude, your KML is too long

My Google Fusion Tables challenge this week was a single shapefile/kml that had about 25 different polygons in the same layer, which when I converted it to KML it put them all within the same tag. Fusion tables would only show some of the shapes on the map, even though they all showed in the KML preview of the data row.

What the Fusion Tables map showed

What the KML preview from the data showed (which is correct)

So I put a question about it into the Fusion Tables API Users Google Group. Folks there explained there is a limitation with Fusion Tables (or maybe the Maps API) that shows only the 10 largest polygons if there are multiple in a Placemark in the KML file. They suggested I use “Singleparts to multipart” tool in QGIS to split the shapes.


Well, I tried that, but I got the following error and couldn't make it work not matter what I tried in that dialog.



So, I peaked at the KML file in a text editor. After all, it's really just an XML file, which is by nature structured, so I hoped I could figure out how it worked and split the file myself. I could see the and tags with very long, multiple tags.

Word wrap is off, but the white line is really, really long.


So I duplicated the tag and then pulled out enough tags to make sure I had no more than 10 within a specific  Luckily there wasn’t too much data to this shape, or that would’ve been a nightmare. The final KML looked like this:


Now there are three tags, each with no more than 10 polygons.




Friday, April 04, 2014

Matt Waite's drone at #ISOJ

Matt Waite of the University of Nebraska-Lincoln is the founder of the Drone Journalism Lab, which specializes in the possible, future use of drones in Journalism (with the capital J).

This is my view from the crowd:



And this is the view from the drone. If you want to sync them, advance the drone video to 1:00 and then start the crowd version.

Saturday, March 22, 2014

Getting Caspio data out to share on Facebook

I've been building an Austin Happy Hours database using Caspio, coming across a not-new challenge where I want to pull data out of Caspio results to use elsewhere on the pages, in this case for Facebook sharing.

It's a special challenge with Facebook, because they want you to use of Open Graph metatags in the of pages to control things like the title, image, share url and description of your content. And while sharing services like AddThis and ShareThis have customization options for these, Facebook doesn't honor them because of this insistence on using og: tags. In other words, those services have the same challenge I do.

I first tried to set javascript variables in my Caspio results to use once the datapage has loaded, which I've used successfully for Google Maps, DocumentCloud and other APIs. While I could append the new metatag to the head using jQuery, Facebook wouldn't honor them. I think that's because Facebook had already processed that part of the page, but I could be wrong.

What ended up working is to include my venue title in the URL of my detail page, so I could then parse it out using PHP and then use to write the metatag when the page was processed. This only worked for me because:

  • We recently got access to publish projects on an PHP server. (We don't have access to any server-side processing from within our CMS.) Without the server-side processing, I'd be out of luck.
  • The page where I wanted this was a Detail report form Caspio, which is drawn when I pass a parameter through the URL into the detail datapage. If I was using a normal Search and Report page, I wouldn't have the content I need in the URL.

There is a downside to this, in that my URL is really long, especially for Twitter. AddThis doesn't use a shortener (at least in their free service.) I started this project using ShareThis, who shortens the url for Twitter, but the ShareThis code was so slow loading I ended up abandoning it for AddThis.

Here are some of the details. I suspect my code is brutish, especially where I search and replace for things like commas, etc. That will break when I get a character I don't expect. I bet there is a PHP command that would do that for me? Please comment if you have a suggestion on that.

Monday, March 03, 2014

Notes from NICAR 2014

This was my first NICAR conference, which has a focus on data-driven reporting and other technical aspects of our craft. The IRE conference usually has similar content, but is also broader in scope.

Here's the full schedule from NICAR 2014 and from IRE 2013 last year. This year's IRE 2014 will be in San Francisco on June 26-29.

But back to NICAR 2014. Here are the goods:


Here is a quick list of the panels I went to and what I got out of them.

Wednesday

Before NICAR in Baltimore, I also went to Tapestry, a data visualization conference sponsored by Tableau. Unlike the 900+ NICAR, Tapestry was limited to about 100 participants, all on the same tract. Speakers included Alberto Cairo, Aron Pilhofer, Fernanda ViĆ©gas and Martin Wattenberg and a host of others. It was good for inspiration, connections and a preview of Alberto's talk at NICAR, allowing me to double up on some content.

Thursday

  • Since I had seen Alberto's talk the day before, I could skip the first part of my D3 News camp to take some MySQL: I actually took two courses on this, which was more any overview of how a database manager can help you vs. Access, and an intro to Navicat, a paid tool to use a MySQL database. It went a little slow for me, but I did enjoy CIR's Chase Davis' take on it.
  • I hoped into the last part of the Intro to D3 talk, which really was a panel walk through of basic code, and I suffered from not being at the beginning. Quickly reviewed notes and hoped I would be OK for afternoon hand-on session.
  • Afternoon D3: In short, this helped me understand the structure of D3 better, but didn't build any proficiency and I would have trouble creating the most simple viz now. But it was better than not going.

Friday

  • Three hour-long sessions on statistics. The intro was really a get-to-know SPSS, a powerful and expensive statistics software package. We didn't get to the most important part of comparing two types of categorical information, so I felt like we missed out a little. The next session on comparing two continuous fields (linear regression) was awesome, and the third on comparing a categorical and continuous information (logistic regression) was also valuable.
  • Since I didn't get much actual SQL on Thursday, I chose the Intro to SQLite class and got the best tip of the conference: you can run SQLite in Firefox using the SQLite Database Manager, with no other software to install (at least on a mac.) This might be how I teach SQL in Data-Driven Reporting next year.
  • Next I helped Scott Klein (ProPublica) and Michelle Minkoff (AP) teach a session on grabbing data from the web.
  • Last was the lightening talks, 10 quick talks. It was a packed room ... well over 800 people. They were all good ... 

Saturday

  • Building a data-journalism course. One of the panelists from Maryland has his students work on the same data set so they can help each other. Amanda Hickman works to get more basic data and viz/charts in earlier courses so students have familiarity so they can get further in class. A couple have groups of students working on the same long-term project or different parts of same subject. Canadian used a donuts to describe databases (then ate them).
  • Clean, clean, clean your mess. Regular expressions class was helpful for this. I had no problem with the code, but had some problems with teaching style. Cleaning data class introduced OpenRefine and class after was helpful in reinforcing those skills. Refine is awesome.
In between all this I made spent good time with folks from the Canadian Broadcasting Company, Orlando Sentinel, Huffington Post, Pixar/Paramount, the Texas Tribune, the AJC, Dayton Daily News, Seattle Times, the Vancouver Sun, small papers in Pennsylvania, South Carolina and god knows where else. And speaking of God, some good bar time with gentleman attending a Catholic youth ministries and a different time with vets getting training at Veteran's Affairs. Also several folks from various schools and many more I'm just not remembering.

My eyes were opened to some database structures, and skills refined for Refine, which I proceeded to crash time after time for four hours straight working on a huge dataset. The statistics classes were well worth it, too ... I know I need to learn more, but that is OK.

Saturday, February 15, 2014

Geocoding and Scooby Doo's Mystery Machine


It's been at least a year ago that my buddy Rob Villalpando was working on an interactive graphic for the Statesman when he came across this street view photo of The Mystery Machine parked in front of a food trailer court at 1106 E. 6th St. in Austin. There are plenty of posts on these interesting sightings, but why I come across them usually is because I'm often geocoding data manually, and I often use Street View as a way to confirm my findings.

Even with the best geocoding services, you almost always get bad addresses and have to fix or at least confirm them. I've written about Texas A&M's geocoding service in the past, and one thing I really like about it is it tells you how accurate it's match was. Anything not parcel based or centerline street, and I know I need to find the lat/long myself.

Before Google updated to their latest maps version online, it was pretty darn easy, though time consuming. Now it's tougher, because they do their Lat/Long coordinates in degrees instead of decimals, which doesn't play well with the tools I use, mostly Fusion Tables, Google Maps API and Tableau.

So here is my semi-pro tip on manually geocoding, using the old maps:


  • Go to https://www.google.com/maps?output=classic
  • Type in the address and hit return and make sure Google Maps takes you to the right place.
  • Right-click on the map at the location and choose “What's here.”
  • That will put something like this in the search bar: “30.258659,-97.744548”
  • Copy and paste the values into your data as needed.


If you have the “new” google maps, you have to do some extra work to get lat/long. It's easier to just use the “classic” link above, but if you insist:


  • Go to the bad new maps: https://www.google.com/maps/preview
  • Type in the address and hit return to find the location
  • X-out the location in the search bar so the pin goes away.
  • Click on the map where the pin was, (and then maybe click again) and a window will come up showing the address and the lat, long, but it will formatted wrong. It will be something like: 30° 22.096', -97° 42.209'. Copy that text.
  • Go to http://dbsgeo.com/latlon/ and paste in the text into Place Name.
  • Make sure it takes you to your location, then from the “Latitude, Longitude” under the map copy your values to put into your data as needed.
Have fun. No one enjoys geocoding, but at least we have fun Street Views to help us pass the time.

Sunday, January 19, 2014

Using Tableau for AISD teacher turnover statistics

There wasn't really anything new with my latest interactive on teacher turnover in AISD, but it did make me realize something: I need to make a clean list of Central Texas schools (or all of Texas) with all their latitude and longitudes for accurate mapping.

In this case, we were mapping all 125 or so AISD schools. We ran a list of Austin school addresses through A&M Geocoding service, which I've written about before, but we had to manually match them with the data we had on turnover rates.

The district data had partial campus IDs, and our address list, which was not limited to AISD, had full campus IDs that didn't match our data, so I couldn't easily join them. We'll fix that, as with this project we come out with a good, manually-checked source list of the AISD addresses. With the help of intern Beth Cortez-Neavel, we'll add district and campus IDs so the next time this comes up, a quick join in Access will get us there.

Cometdocs also saved my bacon again. It does a much better job converting PDF documents to spreadsheets than even Adobe's full Acrobat suite. I can't recommend Cometdocs enough. I fact, I ask that you sign up using this link, so we can both get more free conversion per week.

Saturday, January 11, 2014

How we built the Austin Homicide Project

There has been plenty of great news applications that have focused on homicides, notably HomicideWatch.org and the LA Times' Homicide Report. The HomicideWatch folks have turned their site into a cottage industry, leasing their platform to other publications, like the Chicago Sun-Times, The Trentonion and others.

We wanted to do something similar in Austin ... to write about and follow each case of homicide in our area. Unlike Chicago and L.A., which can top 500 slayings a year, we typically have fewer than 30 each year in Austin. We usually write about each case, but we wanted to follow them better, hence the birth of the Austin Homicide Project.

Our police and courts reporters (Ciara O'Rourke, Julie Chang and Jazmine Ulloa) have kept a Google Spreadsheet with basic information about each homicide for the last couple of years, but to utilize that data in a news application, I wanted to control how the data was entered to keep it clean and consistent. That was a big part of the project ... creating the data management interfaces. But we also wanted to do something special with the display.

Our data interactives "team" is really just a pair at this point: myself and interactives designer/artist Rob Villalpando. We don't have a "developer" nor a model-view-controller infrastructure to build upon, so we pieced together a series of online tools to create our news app.

So, here is how we did it:

A responsive Foundation

Rob has been experimenting with different ways to make our interactive content more mobile-friendly, but we've always struggled to find a solution that works well within our content management system. Probably the best example is the fantastic brewery guide, where he used Foundation. But as we struggled with iframes and wraps, it's been made clear that we need to get out of the CMS to get the visual display we are seeking. This is our first Foundation project on a new (to us) server where I hope we can create some great work. Of course, without the CMS, we have to solve a lot of infrastructure we miss, like metrics, ad serving, access to content, etc.

Caspio for the data entry and display

I used the online database platform Caspio to build the data entry and storage system. We've subscribed to the service for a long time and it works pretty well as a tool to collect structured data and spit it back out, but it is search-engine hostile. Most of their data is invisible to Google. We used their SEO extension for the first time with this project, but it has many limitations and only scratches at problem. But it did allow us to build an application to collect details about victims, suspects and incidents that lead to each homicide, and build datapages to search and display those records.

Google Feeds API for news content

We wanted to utilize all the content we already write about these cases, so we created RSS feeds out of our content management system for each victim and suspect, and stored feeds in the homicide database. We use the Google Feeds API to display those headlines on the detail pages for each person.

Google Maps API for incident maps

As we record each incident in the database, we include a latitude and longitude so we can display an interactive map using the Google Maps API. I've been toying with the idea to make this a static map instead, especially for navigating on a phone. I've tested how to do it Static Maps API, and have considered showing for mobile only. What are your thoughts on that?

DocumentCloud for documents

The free DocumentCloud project is a great research tool for journalists, but the search display widgets also are handy for displaying specific document sets. With each document we upload, we include metadata that we also store with that person's record in our Caspio database, so we can customize the search widget on our display pages. I set the key:value pair as a javascript variable, so I can check for existence before drawing the document search results.

Fusion Tables for overall map

The big map of all the incidents is a standard Google Fusion Tables map. I wish we could publish a JSON output directly from Caspio to feed an online map, but it is simple enough to download the data out of Caspio and then upload it into Fusion Tables, and there isn't that much movement in the data anyway. I did have to tackle a CSS issue where the map tools were obscured, but found the solution on Zurb's site.

Highcharts for stats

Rob put together the statistics using Highcharts, which we already pay for. You could do the same kind of work with the free Google Charts API, which we've also used before, but Highcharts is prettier. This is also a hand-managed data set, and includes APD crime stats in addition to our own data.


Sunday, January 05, 2014

Home affordability in Central Texas


I did some work with Marty Toohey on his story about the "affordability" of Austin, creating a data visualization about home sales using Tableau. It wasn't ground-breaking work (ha!) but it is fairly interesting to see how the sales distribution of home sales have changed over the last decade.

I mostly did this work so that Marty could see trends to write about, and it's an example to show how Tableau is an excellent tool for discovery and exploration of data. Check out how the percentage of homes priced at $100k range and the $200k range have changed over the past decade.

Now for the humbling part: How could this visualization be better? What could I have done to make it more interesting? More useful?

Because I just took at quick peak at Chartbeat to see how folks are interacting with that page, and the average length of stay is 30 seconds. Um ....

Saturday, January 04, 2014

We're hiring a news apps developer

5/17/2914 UPDATE: We are still seeking this position.

We're formalizing our news and data interactives work at the Statesman and we are looking for a developer to join Rob Villalpando and I to form a new News Interactives Team (or some other snazzy name we think up.) Rob has been doing some great work in recent years with news interactives and projects, and I've really enjoyed my work with data-intensive research, analysis and visualizations, but now it's time to "kick it up a notch."

I’m calling this a News Applications Specialist, and you can see the job requirements below.

But let me put it another way: If you read this story, and say, “Yep, that’s me” or “I want to make a difference like that,” then you are the kind of person I’m looking for. If you can buy into this philosophy and help us do some of the same type of work they do, then you are a good candidate. If you understand 80% of the the posts on that blog, then you are probably a really good candidate. If you understand it AND were already a regular reader of that blog, then OMG CALL ME!

I’m looking of someone who loves to create and to share what they know and what they learned today. We’ll soak it up. We also have a lot to offer, so you’ll grow and learn, too.

Resumes and cover letters can be sent to me:

Christian McDonald
Data Editor
Austin American-Statesman
cmcdonald@statesman.com

NEWS APPLICATIONS SPECIALIST:

The Statesman is looking for a news applications specialist to join a new data interactive team. This specialist will work with reporters, editors and other team members to design and build interactive graphics, data visualizations and news applications to support journalism ventures. Because existing members come out of a data analysis and interactive graphics background, candidates with strong programming and user interaction skills are strongly preferred.

JOB DUTIES & TASKS:

  • Communicate with both technical and non-technical colleagues to serve as a bridge between content and digital design of applications and visualizations.
  • Work as part of a newsroom team to bring our most important stories to our readers in a compelling way across all our various digital platforms.
  • Develop, code, test and debug news apps for mobile and wired platforms.
  • Share and expand knowledge with other team members, and to learn from the experience of others.
  • Research new technology and best practices and tools and analyze for best fit, usage, stability and performance.
  • Some reporting and contacting sources for data and information.
  • Communicate with both technical and non-technical colleagues to serve as a bridge between content and digital design of applications and visualizations.
  • Work as part of a newsroom team to bring our most important stories to our readers in a compelling way across all our various digital platforms.
  • Develop, code, test and debug news apps for mobile and wired platforms.
  • Share and expand knowledge with other team members, and to learn from the experience of others.
  • Research new technology and best practices and tools and analyze for best fit, usage, stability and performance.
  • Some reporting and contacting sources for data and information.


SKILLS  & EXPERIENCE:

  • Demonstrated ability to turn concepts into user-focused apps
  • Strong design skills and experience in print and/or digital presentation of data
  • Programming skills to develop apps for HTLM5/CSS3/Javascript environment and others that may emerge
  • Familiarity with web API’s and common data visualization libraries.
  • Experience with a server-side scripting language such as PHP, Ruby or Python and with a server-side web development frameworks such as Rails or Django is preferred.
  • Understanding of data structures and database management, and experience with web service development is preferred.
  • Some reporting experience preferred.





  © Blogger template 'A Click Apart' by Ourblogtemplates.com 2008

Back to TOP