One, a general interest [...]]]> I’ve spent the last two days at Enterprise Strategies for Location Intelligence.  It was an interesting collection of vendors and enterprise end users.  It was encouraging to hear the enterprises in the audience say the agenda hit the areas where they had questions. Across these conversations some interesting meme’s emerged.

One, a general interest in data external to the enterprise – spanning crowdsourcing and open data to social/mobile. Large players from IBM, ESRI to Oracle all gave the theme attention. A second trend that was oft repeated – the convergence of business intelligence, location intelligence, GIS, data science, and a dozen more labels. As enterprises have begun to open themselves up to new concepts and data sources you get a collision of content.  Yet, the biggest “ask” I heard, when speakers talked about customers, was more simplicity in the myriad of offerings.  Almost every vendor highlighted the customer demand for simplicity.  Yet, the convergence of data that is being highlighted would seem to be the antithesis of simplicity.

Andrew sent me a cool quote last night that I think sums up the end goal well:

“Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.”

- Antoine de Saint Exupery

I think this is one of the single biggest challenges we face as a community. As location becomes ubiquitous how do we make the new and old tools of geographic information more palatable as they need to interconnect and become more accesible to a broader ecosystem.  Traditionally, I’d argue that simplicity has not been the leading priority in either GIS or business intelligence.  This is shifting, in our panel SJ Camarata noted ESRI’s push to prioritize simplicity in the GIS market. This was one of many signals to me that simplicity is a goal the community could all benefit from, but what is the design philosophy for achieving it.

A big chunk of Google’s presentation at ESLI was their focus on simplicity – driven by user testing. Sean Maday quoted that in 30 minutes there will be 21,000 hours of Google Maps usage by the public.  Leveraging that mass of users opens up lots of opportunities for A/B testing, and the crowd quickly let’s you know what is simple and what is not. It has no doubt been massively successful, but having the crowd and resources of a Google can be challenging to come by.

Developing applications at GeoIQ we spend a lot of time thinking about simplicity, and thought it might be useful to share our design method. Over on the GeoIQ developer blog Matt Constantine has written up in depth over view of his design approach, along with a host of useful resources.  Simplicity is a perpetual goal and one of the tenants we can all strive for in our apps. Would love to hear other thoughts on achieving simplicity in applications as our data gets more complex.

“What does mapping technology look like when it’s created distinctly for the web? What was once exclusively the realm of [...]]]> Last night I saw a tweet from Mike Migurski at Stamen about a San Francisco meetup called NoGIS.  Totally intriguing, so I followed the link, and they had the following quick desciption posted up:

“What does mapping technology look like when it’s created distinctly for the web? What was once exclusively the realm of traditional geographers is now accessible by almost any web developer thanks to tools like openlayers, polymaps, mapnik, and tilestache. Conversely, concepts that would be foreign in offline mapping (like map tiles) are the centerpoint of how people think about maps on the web. The nature of the problem has changed and technology is rushing to solve it. We’ve gathered some presenters to show us interesting things happening in our field.”

So, if you are wondering why this is called NoGIS?   It is an homage to NoSQL.  Which begs the question what is NoSQL?  At a high level NoSQL is a movement in database management approaches that ditched the classic relational database approach for new innovations that strived to create distributed datastores (minus ACID).  This included techniques like key-value stores, document databases, and graph databases.  In short – for a lot of emerging data streams, which that are often high volume, dynamic, and persistent – the traditional SQL paradigm did not work.

My interpretation of Mike’s post – we are at a similar cross roads in mapping technology.  Arguably we’ve collectively been there for several years, but I think we are at a point where the broader market is catching up.  For decades, location and geography have been their own special niche, served by GIS technology from a fairly small number of vendors.  As many have pointed out “spatial is no longer special” and as a result location is quickly becoming a feature of many technologies.  As location base apps become ubiquitous the characteristics of geographic data are changing as well.  The data of this new paradigm does not look like the static parcel data, which is stereotypical of much traditional GIS work.  As we saw in the NoSQL characteristics data is now high volume, dynamic and users/developers want to see/query it in real time.  This is something traditional GIS was not built to do, and on multiple levels Mike and company’s analogy to NoSQL is quite apropos.  The list of talks for the meet up reinforces and illuminates the thought:

Michal Migurski from Stamen Design is giving a talk titled “Fast, Cheap, or Dumb: Pick Three,” about the design and publishing of geographic visualizations and interfaces with a bias towards simplicity and reach.

Mike Malone from SimpleGeo is exploring the real world technical challenges faced at SimpleGeo while building a web-scale spatial database on top of Apache Cassandra, as well as some new developments & lessons learned.

Sha Hwang & Zain Memon from Trulia will probably just wing it.

Looks to be just the first meet-up, but it poses an interesting question as to what innovations does a reinvention of geographic information call for.  Visualization and data management are two obvious ones.  What else?  Traditionally GIS analysis is static.  If we are streaming data in real time shouldn’t our analyses also update in real time.  What are the repercussions of dynamically changing analyses.  Does this go beyond developers, or are we just creating a new ivory tower? With all the new data management horsepower what do massive sample sizes mean to spatial statistics.  How do we look at verification and validation in age of crowdsourcing?  How do you deal with error bounds when data is constantly changing?  Is it all a relic?  There have probably been few other times it has been more interesting to be a geo-nerd.

Addendum

Following the Twitter thread on NoGIS I was reminded of our own internal decision to move away from SQL for GeoCommons. We did a presentation on it for Where 2.0 in 2008 and it was interesting to look at where and why we diverged from the traditional path back then.

Travis Pinney, Kate Chapman and I decided to work on a project trying to [...]]]>

Over the President’s Day weekend Code for America held a DataCamp at Big Windows Labs in DC. There was a great diverse group of participants who came up with lots of innovative ideas for remixing government data and creating cool apps.

Travis Pinney, Kate Chapman and I decided to work on a project trying to structure the DC health inspection data and fuse it with Yelp restaurant reviews. We were curious what the relationship was between the health inspection ratings for restaurants and their popularity. DC is appropriately famous being a leading innovator with making government data available to the public. Health inspections for restaurants is no different and there is a clean interface for browsing through the full inspection for each food establishment. The problem is the data is in HTML forms or worse PDF. Travis did some brilliant scripting to scrape the data out. Unfortunately this means we could not get the PDF data, but on the upside the majority was in the HTML forms. I experimented with doing the same with Needlebase which was fun to learn, but I couldn’t keep up with Travis’s skills for some of the edge cases in the data. Next Travis used the Yelp API to grab data on the user ratings for restaurants in DC. The good news is we got a merged list of 515 restaurants with both health inspection and Yelp ratings. The bad news is that there 2500+ health reports and 3700+ Yelp reviewed restaurants. So, we missed a lot, but we are we were also pretty conservative in joining the data.

Without getting into the gory details joining locations can be an ugly affair. You can have multiple locations at the sames address (e.g an airport) and have multiple restaurants with the same name, so a unique ID is quite elusive between disparate data sets. There a good number of tricks you can use but the over all problem is difficult. It will be interesting to see how the various start ups like Factual, SimpleGeo and Cloudmade tackle the problem as they build out large POI repositories and look to cross link data. Pivoting on location (lat/long) seems like a good solution and I’m sure we’ll start seeing location based place graphs in the near future.

Back to our data experiment – while we did not get all the data merged 515 is a pretty good sample to look at potential trends. Once Travis had done all the hard work I uploaded the merged data set to GeoCommons to geocode the data. You can find the raw data here available in a variety of formats to remix yourself. Next I plotted out the points in GeoCommons and ran an analysis correlating the health inspection risk score to the Yelp ratings. I wanted to see if health inspection scores were a good predictors of popularity on Yelp. If they were a good predictor I should see a high negative correlation – i.e. low risk scores are correlated with high Yelp reviews.

The correlation did come back negative but only 2% of the variation in Yelp ratings was explained by health inspections risk scores. You can see the results on the map below. While the correlation was not meaningful if you click the gray triangle in the layers palette for the correlation you will see a scatter plot. Click the circles in the upper right hand quadrant to see restaurants that are popular on Yelp but have high health risk scores. Is your favorite restaurant throwing some serious health violations?

View full map

You can play around clicking on any of the quadrants to see patterns of health risk vs. popularity across DC. If you would like to see the raw data for the correlation analysis it is available here. Feel free to remix the data and come up with another perspective like animating it over time. DataCamp was awesome and the project was a fun one. Many thanks to Code for America for setting it up and Big Window Labs for hosting us.

Among the many reasons we’re out here is to get exposed to new ideas and technologies, so if you’re reading this blog and happen [...]]]> Just thought I’d drop a quick line on the blog to let anyone who’s interested know that several of us are out in San Diego at O’Reilly’s Emerging Technology Conference.

Among the many reasons we’re out here is to get exposed to new ideas and technologies, so if you’re reading this blog and happen to be out here as well and would like to talk shop, or just go out and grab a beer to see how many bad heatmap jokes we can come up with, we’re definitely up for it.

Just drop me a line directly or add a comment to this post and we’d be thrilled to meet you.

Technorati Tags: etech, etech07