NYC OpenData

#CivicHackNight Meets the Beta NYC Developer Platform

Mon, 13 May 2013 12:00:00 -0400

On Wednesday night, we launched a beta version of NYC’s Developer Platform - a hub for talented civic developers, offering tools, documentation, code samples, peer-to-peer support, and more!

We shared the prototype at the #betaNYC Meetup, a weekly HackNight hosted by Code for America’s NYC Brigade at Blue Ridge Foundation.

After an introduction by DoITT’s Director of R&D, Andrew Nicklin, attendees divided into groups to hack the portal’s UX (user experience), content & documentation, technology, and community-collaboration.

Breakout topics and areas for feedback include:

The Fork and Finish project (in nodejs) – to co-produce an Open311-compatible read-only API from the Socrata dataset, and package that as an API we can deliver through the portal.

Terms & Conditions/Policy – to make sure that we’ve clearly defined the expected behavior of those who register and engage on the developer portal. We’re interested in hearing from the community to see whether these expectations are accurate. For example, would they discourage a developer from registering and taking advantage of the APIs we are offering? Are there considerations that need to carry forward into the terms and conditions for their own customers?

APIs/Site Design & Ease-of-Use – to explore how the site can best help programmers find the resources they need. Suggestions received on Wednesday included: creating a responsive site design; driving end-user contribution through a ranking or reputation system, such as Stack Overflow; building out FAQs, documentation, code-samples, and examples; and enabling collaboration between developers using the platform. What else can we do to improve the experience?

Hack to Source a Real-Time Stream – the City provides feeds that need to be queried once-a-minute for changes. We’re interested in developing a layer which will handle queries automatically and deliver a real-time stream of the data over HTTP. Multiple City data sets – like 311, which has ~40k updates per day – would be even more powerful if provided in real-time. It’s a proof-of-concept with a lot of potential.

We’re so appreciative of the enthusiasm and dedication everyone brought to the meetup. Stay tuned for the next iteration in the coming months. In the meantime, sign up for an account and let us know what we can do to make the site more useful and to improve the experience!

Explore the beta nyc.gov/developer site

View the Beta NYC Developer Platform presentation

Join the #betaNYC Meetup group

As part of the one-year anniversary of the NYC Open Data Law –...

Wed, 08 May 2013 12:37:09 -0400

As part of the one-year anniversary of the NYC Open Data Law – Local Law 11 of 2012 – we’re profiling users of NYC Open Data.

Check out this detailed visualization of NYC’s urban tree canopy by Tom Swanson from ESRI. The map is based on the NYC Parks Department’s high resolution land cover data set, which charts grass and shrubs, bare earth, water, buildings, roads and other paved surfaces.

Urban trees provide environmental, social, and human-health benefits, but these benefits are hard to measure. Using NYC OpenData, the NYC GIS basemap and object-based image analysis, NYC Parks, the University of Vermont Spatial Analysis Lab, and partners developed a comprehensive land-cover map that includes the tree canopy down to the scale of individual trees.

To create the map, the team relied on high-resolution LIDAR to present the height and texture of land cover as well as multispectral imagery and contextual analysis to distinguish trees according to their physical and spectral properties. Converting high-resolution LIDAR and imagery into usable information took significant processing time and labor, but was an effective method for fine-scale canopy mapping in NYC’s complex urban environment. An article on this innovative use of object-based image analysis was published in the Journal of Applied Remote Sensing in 2012 - abstract available at http://dx.doi.org/10.1117/1.JRS.6.063567.

View NYC’s land cover data set on NYC OpenData

View the high-resolution land cover map

Parks & Recreation…and Data

Thu, 02 May 2013 12:08:00 -0400

Photo: Brian Dalessandro presenting at the NYC Parks DataDive

How does data analysis help the Parks department care for the city’s trees? Learn how DataKind Data Ambassador, Brian Dalessandro, analyzed Parks data to find that the Parks’ tree pruning program reduces emergency cleanups by 22%!

Read the blog post

Check out tree census data on NYC OpenData

DoITT’s Albert Webber delivered a presentation on the NYC...

Fri, 26 Apr 2013 11:22:45 -0400

DoITT’s Albert Webber delivered a presentation on the NYC OpenData portal to a full auditorium at the 2013 hackNY Student Hackathon. This event took place on April 6th at the Fu Foundation School of Engineering and Applied Science, Columbia University.

Check out the projects from the student hackathon

Credit: Photo by Matylda Czarnecka from the hackNY flickr

Aaron Schumacher submitted this data visualization of daily...

Mon, 22 Apr 2013 13:52:34 -0400

Aaron Schumacher submitted this data visualization of daily entrances into the MTA subway system. According to Aaron:

“Start with open data, then some processing, and eventually you can make a picture like this. You can also check out the interactive version, where you can see the date and number of entrances for about three years worth of subway traffic. You can clearly see traffic changes around major holidays, and especially the effects around hurricanes Irene and Sandy.”

Check out this visualization designed and created by Eric Schles...

Mon, 22 Apr 2013 13:46:00 -0400

Check out this visualization designed and created by Eric Schles and Thomas Levine. Using MTA turnstyle open data and daily weather observations from NOAA, they show the impact that super storms have on subway ridership.

Staten Island Counts: Using Data to Tell a Story

Wed, 10 Apr 2013 10:39:00 -0400

As part of the one-year anniversary of the NYC Open Data Law – Local Law 11 of 2012 – we’re profiling users of NYC Open Data.

Today we’re highlighting the work of Aileen Gemma Smith, Founder and CEO of Vizalytics Technology, an organization whose mission is to enable change with data. Vizalytics began as an R&D project called Staten Island Counts. Their goal was to use data and visualizations to tell a precise story about the opportunities and challenges for Staten Island communities and businesses.

We condensed the interview we had with Aileen below.

How are you sharing open data with your community?

In Staten Island, information is siloed and not everyone is aware of how much is available, or the potential of merged datasets to give useful insights into needs and opportunities. We’ve shared open data with elected officials and other community stakeholders like the members of Staten Island Chamber of Commerce and members of the Staten Island Economic Development Corporation. People are thrilled to discover how easy it is to access open data.

What type of data does your community find useful?

Hyper-local data is really important. We have gone over 311 data reports of trees, wires, sewer overflows, and crowded drains. After Sandy hit, my partner and I worked from our hotel room (we live in Zone A and evacuated) manipulating data in Google fusion tables from gas station maps and Staten Island business maps. We looked at surge lines and looked at what businesses were within the surge lines. Specifically, we proposed the Staten Island Borough President’s office create maps using Department of Buildings’ red tags, and overlay it with property lot size and values and census info to understand the risk profiles for different neighborhoods.

The map above shows businesses in Staten Island that were within Sandy surge lines.

The map above delineates what parts of Staten Island were within Sandy surge lines.

How can open data help a small business?

Not Just Bagels is a local business that’s looking to be more competitive, especially after being hard hit by Sandy. For small business owners the focus tends to be on what they do best, for example, “How do I make a better bagel?” We help broaden the perspective to, “How do I find more customers? What are influences on my business that might lead to more or less foot traffic?” We help businesses like Not Just Bagels streamline the process about finding more customers by showing open data like bus stops for the express bus who might be coming into the store. Our goal is to show that technology does not have to be scary! It helps you to be more competitive.

How do you help people understand open data and get the information they need from it?

Those who haven’t worked with data don’t know where to begin, so it’s great to make it as simple to understand as possible – making it visual, not just a spreadsheet with thousands of cells. People like to see that information is actually find-able and searchable – I let people know they don’t need a consultant to get to these conclusions. We want to put the user in control, and that user does not have to be a business analyst.

What potential do you see for open data?

I want to see greater use of predictive analysis and trends from open data. It’d be great to understand the risk profile of a neighborhood based on its hyper-local data. I want to look at how data can tell the story of a neighborhood’s ambiance. I’m curious - where people are going, who clusters, where, how?

I’m also interested in information not just in the present, but the big picture over time. For example, I’d like to go from “Here’s the best exit on the R train at this time” to “Here’s the more meaningful picture about mass transit. And why we need more options here on Staten Island.” Or, from “What’s this restaurant’s health grade” to “What’s the health grade of restaurants over time in these areas?”

The photo above is of Aileen Gemma Smith.

To view additional data visualizations provided by Aileen, visit DoITT’s Flickr

Roadify: Using data to empower people on-the-go

Sat, 23 Mar 2013 08:01:00 -0400

In this dense, bustling city, people depend on mass transit every day, sometimes in different modes, and often multiple times daily.

As part of the one-year anniversary of the NYC Open Data Law – Local Law 11 of 2012 – we’re profiling users of NYC Open Data. Our first profile looks at Roadify, a data platform and free iPhone application connecting users with real-time transit info and updates.

Roadify was the Grand Prize winner in the NYC BigApps 2.0 competition in 2010, and alerts users to the latest subway, bus, or driving conditions by using official transit data and real-time updates from commuters. They help answer a basic, vital question posed by millions of commuters daily: “When is my bus/train/subway, etc. coming? And if it’s late, why?”

Data Mashup
Roadify gathers open transit data (including Staten Island Ferry and MTA data) from more than 60 transit agency sources across the United States and Canada, as well as from riders via Twitter and the Smartphone app. This data is:

official/crowdsourced
real-time/static
structured/unstructured

Official structured transit agency data is typically in the Google GTFS (General Transit Feed Specification) format for schedules and real-time arrival information, while unstructured service advisories can be in XML or RSS feeds or via Twitter. Roadify also monitors Twitter to curate comments from riders and agencies about specific transit systems and individual lines, along with user comments provided via the Roadify iPhone app.

Roadify aggregates this information on its own platform and packages the content for hyper-local, real-time distribution to customers via XML feed. Roadify’s digital signage customers can opt to design their own displays for the data feed or use a localized Flash or HTML display in broadcast-ready or interactive mode. Roadify provides transit information on large screens at locations other than transit stations – including Brooklyn’s Barclays Center, “City 24x7” kiosks and the Philadelphia Convention Center.

“All this complexity under the hood is about making it easy for riders to find out what’s going on,” said Roadify CEO Scott Kolber. “If people know when their ride is coming, they’re more likely to use mass transit – and that’s good for riders and cities.”

Watch the NYC Economic Development Corporation’s “Make it Here” profile of Roadify

NYC BigApps 2013 Kicks Off at the New York Tech Meetup

Tue, 19 Mar 2013 19:30:53 -0400

The forth annual NYC BigApps competition launched tonight at the New York Tech Meetup. The annual contest encourages software developers and members of the public to create web or mobile applications using official City data.

More than 1,000 data sets are available on the City’s Open Data portal from 60+ City agencies, commissions, and Business Improvements Districts.

Winning applications will receive cash prizes totaling $150,000 and the opportunity to earn follow-on funding for future app improvements. Over the course of the previous 3 competitions, nearly 240 new apps were created.

This year’s contest focuses on working together to solve specific New York City challenges, known as BigIssues (Jobs and Workforce Mobility, Healthy Living, Lifelong Learning, and Cleanweb: Energy, Environment, and Resilience). The best app in each of the 4 focus areas will win a substantial prize, and be eligible for the NYC BigApps 2013 Grand Prize.

The 2013 judging panel includes:

Dawn Barber Co-founder, New York Tech Meetup;
John Borthwick CEO, Betaworks;
Arianna Huffington, Chair, President, and Editor-in-Chief of the Huffington Post Media Group;
Lawrence Lenihan Founder, CEO and Managing Director, FirstMark Capital;
Ann Li, Managing Director, Center for Economic Transformation, New York City Economic Development Corporation;
Rahul Merchant, New York City Chief Information & Innovation Officer and Commissioner, NYC DoITT;
Ulrich Quay, Managing Director of BMW iVentures;
Danny Schultz Co-founder & Managing Director, DFJ Gotham Ventures;
David Tisch Managing Partner, BoxGroup; and
Fred Wilson, Managing Director, Union Square Ventures.

Entries are due by 5pm on June 7th, 2013.

Learn more: www.NYCBigApps.com

nycedc: NYC BigApps Is Coming Soon Move over Samsung – The Next...

Fri, 22 Feb 2013 12:38:03 -0500

nycedc:

NYC BigApps Is Coming Soon

Move over Samsung – The Next Big Thing is almost here. We are working with CollabFinder to bring the biggest, boldest, and best BigApps ever! While we can’t divulge our secrets yet, yesterday we gave a sneak peek at Social Media Week event: “We Built this City: The State of Civic Technology, with Code for America and IDEO.”

Entering its fourth year, the NYC BigApps competition promotes government transparency and innovative new technologies by challenging mobile and web developers to create cool, free apps for New Yorkers using City data.

Get involved: Sign up now at www.nycbigapps.com to be the first to know about any updates on New York City’s ultimate open data software challenge. Trust us – you don’t want to miss a thing.

Colorful snapshot of New York City’s Twitter languages. This...

Thu, 21 Feb 2013 16:57:07 -0500

Colorful snapshot of New York City’s Twitter languages. This visualization highlights 8.5 million geo-located tweets collected between Jan 2010 and Feb 2013. Read more: http://spatialanalysis.co.uk/2013/02/mapped-twitter-languages-york/

Today’s data visualizations show 24 hours NYC snow plow...

Mon, 11 Feb 2013 13:00:00 -0500

24 hours of PlowNYC data: http://dwtkns.com/plow/

PlowNYC Animated Maps: http://scottwittrock.com/plownyc/

Today’s data visualizations show 24 hours NYC snow plow activity via data pulled from PlowNYC for an incredible look at NYC Department of Sanitation street clearing efforts. Thanks to @fititinpost42 and @dwtkns!

nycedc: Exciting things are underway for NYC BigApps, the...

Fri, 25 Jan 2013 12:14:22 -0500

nycedc:

Exciting things are underway for NYC BigApps, the City’s premier open data software competition!

While we’re not ready to reveal all the surprises in store just yet, we are excited to announce that NYC BigApps 2013 will focus on leveraging technology to help solve NYC’s key challenges, or “BigIssues.”

CitizenConnect, for example, helps tackle a BigIssue with a focus on income mobility, workforce support, and job placement.

Now is your chance to weigh in on what BigIssues the BigApps community should tackle this year. Share your great ideas and get involved by completing our brief survey!

Photo Credit: Karin Beil via Flickr

NYC Historic Neighborhoods Data Mashup

Fri, 18 Jan 2013 15:32:46 -0500

nycgov:

Do you like exploring New York City and discovering its history? Check out nychistoricdistricts.com for a map of New York City’s historic neighborhoods and take a virtual tour of places like Brooklyn Heights, Governor’s Island and DUMBO with information on the neighborhood’s history and photos of historic buildings nearby.

What do 1,551,402 service requests look like? Check out a...

Wed, 02 Jan 2013 16:47:11 -0500

What do 1,551,402 service requests look like? Check out a visualization of NYC 311 requests (received via phone, text, online) in 2012.

The NYC 311 dataset is available at NYC Open Data - www.nyc.gov/data

Interested in the future of NYC’s public payphones? Check out...

Thu, 13 Dec 2012 10:55:00 -0500

Interested in the future of NYC’s public payphones? Check out this animation highlighting their locations across the five boroughs. Imagine how this vast network could be used to boost access and connectivity!

@chris_whong created this video using NYC payphone installations Open Data, Processing and scripts - GeoTrails and Particles vs Text - to generate the moving text.

To learn more about Reinvent Payphones, NYC’s payphone design challenge, visit and register: nyc.gov/reinventpayphones

Many thanks to Chris!

Check out this video of a visualization of NYC’s transit...

Tue, 27 Nov 2012 14:33:00 -0500

Check out this video of a visualization of NYC’s transit system which was compiled through General Transit Feed Specification data and illustrates a day — from 4 A.M. to 4 A.M. — of transit operations.

Source: http://mashable.com/2012/11/27/transit-new-york/

Moneyball for New York City

Wed, 21 Nov 2012 10:18:14 -0500

Michael Flowers, Analytics Director for the Mayor’s Office of Policy and Strategic Planning and Director of the Financial Crime Task Force of the City of New York, shares how data science has played a surprising and effective role in helping city government provide services to over 8 million people, from preventing public safety catastrophes to improving New Yorkers’ quality of life.

- O’Reilly Strata Conference

Check out Canopy, a project that came out of DataKind &...

Tue, 02 Oct 2012 09:59:02 -0400

Check out Canopy, a project that came out of DataKind & DoITT’s DataDive event a few weeks ago.

adamlaiacano writes:

The NYC Parks Department brought full dumps of their databases and a handful of questions. Volunteers brought their modeling, data munging, visualizing, and overall hacking skills.

Our project was provide a good understanding of what the tree diversity is like across the city, and how it is changing over time. The results are above. An interactive map where you can find all of the tree types in the city, the diversity of each census block (“diversity” being the number of unique species seen), some information about each tree type, and more. It was in a near-complete state in just one full day of work from Christopher Reed, Andrew Hill, Brian Abelson, Bennett Andrews, and myself. Chris did all of the front end work and has been updating the project relentlessly, making it better pretty much every day. Andrew set up the cartography database (CartoDB) which exposes an amazing API for querying the data. Bennett pulled in all of the tree information from Encyclopedia of Live. And Brian and I took the raw data provided by the parks department and transformed it into a workable shape.

Huge thanks to Jake Porway and DataKind for putting events like these together. For more information on DataKind, check out Jake’s talk from DataGotham.

Playbook for Gathering, Structuring & Automating Public Data

Mon, 01 Oct 2012 12:29:00 -0400

Last month, we published the Open Data Policy & Technical Standards Manual (TSM), an open data guide for New York City agencies. This publication marks a major milestone in New York City’s open government strategy and the first step for opening all of the City’s data by 2018.

Read the TSM