The Wikipedia for Data Meme and Federating the Data Commons

Bret Taylor had a great blog post today on the need for a Wikipedia for data. In short Bret feels like the difficulty in accessing data for web development project seriously hinders innovation. Lots of people resonated with the blog post and it is already up to 51 comments and has spawned several related posts – including ReadWriteWeb’s excellent list of web application offering open source data and related services.

We’ve spent a good bit of time working on the geospatial side of producing a large repository of open data. Specifically, dealing with the ability to handled structure quantitative data. I thought this might be a good time to talk a bit about the challenges we ran into and possible solution to the bigger problem posed by Bret in his blog post.

When we launched GeoCommons just short of a year ago we had grand ambitions of creating a community around geospatial data and creating maps. As with most first attempts we got overly ambitious and tried to pull off too many features in the first go, and ran into a serious case of overall mediocrity.

That aside the big technical hurdle that eventually led us to take GeoCommons down was a data scaling issue. The constant reading and writing to the database caused performance to really drag as the overall database got bigger and bigger. In the first go we used MySQL and hit a wall with about one billion data entities and then again with PostGIS when we got up to around five billion.

This problem was only further complicated by the size of the files we were working with, which often were over 10 mb. Geospatial data files trend large and can become quite monstrous. As a result we had upload and download times that far exceeded the patience of most general web users. Also, executing operations (mathematical, query, mining etc.) with data sets that large, and producing visual results in an acceptable time, created yet another challenge.

In short when Bret said, “DataWiki seems like an extremely hard problem” he was correct and we were just tackling one data niche. The good news is that the problems are solvable given enough time and resources, and we were able to find solutions for all the barriers we ran into with the first attempt at GeoCommons.

While this is good news for us I think it illustrates a bigger point – it is unlikely that one company is going to be able to solve the whole DataWiki problem. The list of current open data projects listed in the ReadWriteWeb post illustrates this hunch. Even a project as comprehensive and as well funded as Freebase is “better equipped to handle conceptual rather than statistical information on topics”.

Given the vast amount data out there and the limited resources any one company has, it seems that connecting multiple different repositories through a federation approach would be a clever way to go forward. This of course opens up an entire can of worms on best approaches and possible standards.

There are some important prerequisites like establishing metadata and schemas as well as the use of common formats, but a practical real world implementation I think will get us a long way. We’ve been talking with some folks in the GeoWeb space about doing some federation test beds with GeoCommons and I believe there are some interesting approaches to look at.

I’m curious what other folks out there think about the possibility of federation and who out there is open to doing test bed implementations? I believe that real potential of a DataWiki and the concept of a semantic web driven by data is going to require an interconnection of all the new data resource coming on line instead of the current islands of innovation.