Newsroom tools, data pipelines, and interactive stories.

Interactive · editorial strategy

Sochi 2014 Winter Olympics

Elasticsearch · JSON · highlights over encyclopedic

Project lead. After London's near-encyclopedic results pages, we reviewed analytics data to rethink the approach: highlights, strong photography, and live results woven into the reporting, with detailed results still available for core sports fans. The IOC's ODF data went into Elasticsearch, a JSON search engine that stayed closer to the original feed and was far easier to work with. Readers found more of the coverage this way, instead of having to wade through London's endless separate results pages.

Interactive · first big data games

Vancouver 2010 Winter Olympics

IOC ODF feed · schedule tracker · athlete bios · live medals

Project contributor. The desk's first extensive use of data in Olympics coverage. The IOC's ODF feed was its most comprehensive yet, which let us build a personalised schedule tracker, athlete bios, and live medal awards. It set the pattern for London and Sochi.

Live data at scale

Election results loader

Ruby on Rails · SQL · AP data · JSON API

Contributor, then core team member, for the election results coverage. The system, built originally by Jacob Harris and Ben Koski, pulled Associated Press results every minute, worked out which of tens of thousands of races had changed, and served them to the site through a fast JSON API.

Data partnership

SchoolBook

Ruby on Rails · Solr · Varnish · WordPress

Main engineer on the Times side of a schools data site built with WNYC. I worked with the product lead on the vision and directly with the CAR desk, who calculated z-scores for NYC schools, and built the pipeline that loaded that data into the SchoolBook app. It brought test scores, school reports, and community discussion together in one searchable place for parents.

Editorial analytics tooling

Understanding the audience

Audience segmentation · alerting · early chat bots

Helped set up the newsroom's analytics team (with Aron Pilhofer and James Robinson). After all the work that went into London 2012, we learned that many of the pages we made got very few readers, and we realised we needed far better tools to see how people used, ignored, or missed our coverage. I moved into exploring NYT analytics data and built tools intended to segment readers. I also wrote a primer on the approach for other newsrooms.

Crowdsourcing

A Moment in Time

Global photo crowdsourcing · interactive globe

Built the submission system behind the Times' global photo project, front end and back end: the form builder, the reader submission form, the image upload service, and the moderation dashboard. Readers around the world photographed the same moment and sent in more than thirteen thousand images. Working with the photo desk and the graphics desk, we turned them into an interactive globe, searchable by place and topic. A graphics desk colleague built the globe, and I contributed to it.

Reader callouts

Contribute to Our Reporting

Branching forms · Affordable Care Act coverage

Built the reader callouts for the Times' Affordable Care Act coverage, working with the business and science desks. Custom branching forms took patients, providers, and employers through different questions depending on their earlier answers, so reporters heard from a larger and more varied group. Readers didn't just add to known stories, they pointed us to new ones.

Sensitive data

Haiti earthquake missing persons

UGC platform · PFIF · Google People Finder

Built the Times' missing persons interactive after the January 2010 earthquake. Readers were emailing us to find loved ones, so I published a simple submission form on the user generated content platform I had built, and the first entry arrived within minutes. The State Department had chosen Google's People Finder as the shared resource, so I worked with NPR, Ushahidi, the ICRC, and Google to publish our data in its interchange format (PFIF) and wrote a tool to send regular updates. The interactive stayed current and marked people once they were found safe. CNN and the Miami Herald followed suit.

Open source · Go

Streamtools

Flow based programming · real time data

A visual toolkit from the New York Times R&D Lab for working with streams of data. Instead of writing code, you connect blocks together on a canvas to filter, transform, and route live data in real time. It was released as open source and written in Go. I became involved via working with newsroom analytics data and contributed several features including handling Elasticsearch data and outputting formats for internal charts and dashboards.

Research Engineer · digital storytelling

Article reboot

New forms outside the deadline cycle

A Lab project to rethink what a Times article could be once it was freed from the daily deadline cycle. It explored new digital storytelling forms, from richer structure to more useful ways of surfacing related coverage, using Go and Elasticsearch under the hood. Much of what I learned here fed straight into the Madison and Hive work.

Bots · serendipity

@NYTPlusContext

Ruby · Twitter API · full text search

A playful research bot, built with James Robinson and Brian Abelson. When the parody account @NYTMinusContext stripped a line out of a Times article, our bot found the original story and tweeted the context back. It leaned on the Times full text article search to reunite quotes with their source.

Crowdsourcing · Go

Madison and Hive

Elasticsearch · JSON · open source framework

Madison was a public site built with Jane Friedhoff that asked readers to help identify advertisements from the Times archive. Ads were notoriously hard to read programmatically because of their creative and inconsistent layouts and typefaces, so we gave the audience a few ways to pitch in depending on their interest and spare time.

Madison ran on Hive, an open source platform I created for building crowdsourcing apps in many contexts. Informed by our Streamtools work, Hive used Go for fast JSON parsing and a clean interface to Elasticsearch. Pairing a compiled language with a search engine let it handle a wide variety of reader contributions across very different tasks.

Internal tools · politics

General Election coverage dashboards

UK General Election · party coverage

During the UK General Election I built internal dashboards that helped the newsroom track and balance coverage of the political parties. The idea was simple. Give editors a clear, live picture of who was being covered and how, so the reporting stayed fair and informed rather than guided by hunch.

Illustrated storytelling

Hooked, a graphic novel

Multilingual · digital comic · Focus on Africa

A digital graphic novel following the true story of drug addiction in Guinea-Bissau, made with the studio PositiveNegatives and artist Tayo Fatunla. I produced the interactive build with Naomi Scherbel-Ball, using an open source comic viewer to bring the panels to life on the BBC site.

Elections · automation

@BBCReferendum results bot

Ruby · Redis · Twitter API · image processing

On EU Referendum night, the TV unit generated a result graphic for every counting area. Doing the social posts by hand would have been slow, so I built a bot with Paul Sargeant of Visual Journalism to do it. It watched for each new graphic, cropped and branded it, then composed and tweeted an accurate summary.

It was built carefully, since a wrong tweet on that night would have mattered. If a result flipped between reports, it raised an alarm and waited for a human rather than tweeting. As a small joy, my laptop read each result aloud in a matching regional accent through the night.

Access · censorship

BBC Uzbek newsbot

Telegram · Go services · Ruby on Rails admin

The first newsbot the BBC ever launched, built with Trushar Barot of the World Service. Telegram was the most popular messaging app in Uzbekistan, where the BBC Uzbek website is blocked, so we used it to deliver stories where readers already were.

The bot services were written in Go for speed, reading from the same database as a Ruby on Rails admin that let editors add commands without touching code. Because readers could be at risk, we stored the minimum possible user data and encrypted it, and the bot could localise dates for non Gregorian calendars.

Structured journalism

Structured journalism at News Labs

Reusable story data · research with Paul Rissen

Part of the wider News Labs push to treat the facts inside a story as structured, reusable data rather than plain text. The idea is that once a fact is captured properly it can power many stories, formats, and tools, from graphics to bots.

Head of Interactive Journalism

British Vogue messaging service

Chatbot · conversational delivery

I led the build of a chatbot that delivered British Vogue updates over a messaging app. It was an early experiment in conversational news, and an honest one. The format did not really land, and it taught me a lot about where chat bots do and do not add value for readers. I include it because the lessons were as useful as any success.

Team and strategy · 11 countries

Head of Interactives, Vogue International

Cross-title tooling · editorial technology

As Head of Interactives for Vogue International, I built and led a digital team that worked across eleven countries. My job was to bring interactive and data thinking to Vogue titles and their social channels, helping editorial teams work with data, build interactive features, and pitch new storytelling formats without needing a full engineering team of their own.

News app developer · magazines

Hearst Interactive Studios

New York · 2000 to 2003

My first taste of newsroom technology. From 2000 to 2003 I built interactives and the backend publishing tools behind some of Hearst's best known magazines, including Esquire, Popular Mechanics, and Cosmopolitan.

The projects were as varied as the titles. I built a Drinks Database for Esquire, a content management system for Popular Mechanics that ran features like their famous beer can chicken recipe, and the online quizzes for Cosmopolitan. It was where I learned that the systems editors and producers use behind the scenes matter as much as the thing readers see. That idea, good internal tools make better journalism, turns up again and again through the rest of my career.

Early web and startups · 1995 to 2007

Before the newsroom

New York · San Francisco

I have been building for the web since it was young. I started at Interactive Imaginations in New York while still in high school, working on the riddler.com games site and an early digital ad network. At Macromedia I helped build the first Flash version of their own website and sat on the Dreamweaver advisory board. At vivid studios in San Francisco I built the Webby Awards site and ran Sun servers as a unix sysadmin.

The pattern held as the web grew up. I was tech lead at Oven Digital, then a very early social media hacker at Friendster in Mountain View, where I sped up the Java based site and built Groups, profile skins, and in site messaging with anti spam controls. Later I was a software engineer at StreetEasy, the New York property startup that Zillow went on to buy, and I contributed to Spot.Us, one of the first crowdfunded journalism platforms. All of it fed the work that came next.

Talk

Too Much Information

Hacks/Hackers Media Party · Buenos Aires

A talk on why programmer and journalist skills belong together. The argument was simple. There is no shortage of data, so our job is to give people signals rather than noise, using the New York Times election and Olympics work as examples of making dense data clear.

Talk · Golden Gate Ruby Conference, San Francisco

Thinking Different: using Ruby to fight AIDS

On the Baobab Health work in Malawi

A talk for the Ruby community about using technology to affect change, told through the Baobab Health work in Malawi. It covers the reality of shipping clinical software where the internet is slow, the power cuts out, and there is no app store. The team hacked donated iOpener terminals to take touchscreens and barcode scanners, ran a self healing mesh network, and treated the test suite as something that literally saves lives.

Seminar · Reuters Institute, Oxford

Structured Journalism and Reporting with Data

Reuters Institute for the Study of Journalism

I led a seminar at the Reuters Institute for the Study of Journalism on structured journalism and reporting with data. It drew on the BBC News Labs work, from turning the archive into reusable structured data to using bots to reach readers on the platforms they already use. I have also spoken at NICAR and the Perugia Journalism Festival.