One side of the company felt that by not limiting access to data we were giving away the family jewels. The other side felt that open access was the best way to create a network effect for data by making it as accessible as possible. At the end of the day the open access philosophy prevailed, and from the sound of comments to James Fee’s post after GeoWeb, access to data is still an important facet to both GIS and GeoWeb users.

Now that Finder! has been out for a little while we’ve begun to see a big surge in downloads. I noted last week we hit 18,000 downloads and just a week later we are now over 28,000. This has caused us to take a second look at our access policies. “Knock on wood”, the system has scaled like a champ handling the traffic, but as we get ready to launch Maker! some concerns have come up about potential abuse and its effect on user experience.

The biggest concern is around systematic downloading of data and the potential for that to impact other users experiences on the site. The question is how to make the content available without impinging on the collective user experience. Wikipedia approaches this by making content available as one big tarball and asks users “Please do not use a web crawler to download large numbers of articles. Aggressive crawling of the server can cause a dramatic slow-down of Wikipedia. Our robots.txt blocks many ill-behaved bots.”

I’m not sure a giant tar ball of data is the best way to go for us, especially since the data is available in a variety of formats. A second option is to provide third party access to the data via an API. This API could also work for both download and upload. Andrei had an interesting suggestion in our last post:

“The two-way API will definitely help with the number of uploads. The cool thing to do, would be to add (”Add to Finder!”) a URL request:

…finder.com/add?file=file.kml&type=kml&name…”

If people have other ideas on how they could better access the data in bullk without impinging performance we’d love to hear them. Also thoughts on what the line is between fair use of content and abuse of the commons. It is a bit of gray line in my mind. Is systematic downloading (manually hitting every dataset) abusive? Is scraping datasets with bots abusive? The main goal in my mind is to provide the best service possible without creating a “tragedy of the commons“.

ESRI released their downloadable application, ArcGIS Explorer 480, this week. The upgraded version still offers the same basic function of geoprocessing services, but with major improvements such as KML improvements and increase in performance.

Nokia acquired Plazes, an LBS social mapping network. Plazes provides updates of friends’ activities in a find me/follow me type of service. The social mapping application will take Nokia one step closer to “bringing people and places together” and also acknowledges the rise and demand of LBS.

Microsoft and Virtual Earth launched a program that allows state and local governments to share their imagery, called GoVE. All of the imagery from these governments, municipalities and aerial photographers gets sent to Microsoft, which in turn is published by Virtual Earth.

This looks to be a direct shot at Microsoft’s 3D Virtual Earth that also runs in the browser. The question mark in my mind will be if the Google Earth version has the same performance issues as MSVE. It is also interesting that Google released an API instead of a new version of GE that ran in a browser. Will this be a case of Google testing the waters with the API then releasing a product?

From a personal perspective I’ll be very interested to see how the new Google Earth API handles KML. Frank says the new API will be a, “subset of the Google Earth 3D graphics rendering engine and interfaces with KML support”. The question is will that KML support be robust like Google Earth allowing thousands of geometries to be drawn or less robust like Google Maps where you are limited to the low hundreds. I’m sure we’ll see soon enough, but congrats to Google on porting the technology to a browser, surely not an easy task. Although it begs one last question – does this herald the end of thick client geobrowsers?

There are three factors that constitute file bloat for any vector based geospatial data:

1) The number of attributes (how many columns)
2) The number of features (how many rows)
3) The complexity of the geometry (how much needs to be drawn)

You can do some clever things to manage the first two at a low level – although you still are going to have bloat when you convert to a standard file format. The third factor, geometry complexity, is interesting because you can also do some low level tricks whose savings can be passed along to standard file formats. Reducing the complexity of geometry is often called “map generalization” in academic circles.

The general concept is that you remove details from the map without loosing the message and context of the map. All maps have some form of generalization otherwise it would be a perfect reflection of reality. Academics have used algorithms to heuristically derive a map generalization. This is probably best explained with a few examples. Below is a map of Europe in full detail:

Next is map generalization that removes some of the detail but still keeps the context of Europe and the country boundaries:

Last a more extreme example with even greater detail removed:

To pull off these nifty computational tricks used to require some fairly sophisticated desktop software, but Matt Bloch and Mark Harrower at the University of Wisconsin figured out a clever way to enable enable real-time WYSIWIG map generalization. The resulting application is called MapShaper. You can upload a shapefile and run different generalization routines (with high level of control if you choose) then export the result back out as a shapefile or an EPS file. The shapefile export is down at the moment, but hopefully will back in action soon.

I think these kinds of technologies and mathematics are going to be increasingly important as we need to make ever larger datasets available. Especially when the receiving devices are increasingly mobile with even smaller data handling capabilities.

The good news is that both Virtual Earth and Google Maps support KML, and we are seeing a greater number of applications supporting it and GeoRSS as GeoWeb standards. As the standards get picked up it will be interesting to see how they are supported and how applications differentiate themselves in doing so. Already we can see this beginning between the two titans (Microsoft and Google) expressing how their support of KML has advantages over the other. So, I thought I’d share what our experience was testing with both applications.

Google KML Support

For testing purposes I started off with a polygon data set of the 100 most polluted counties in the United States. The upload process for Google MyMaps was straight forward and my uploaded KML (or GeoRSS) file prepopulated a title and description field. Then after a bit of chugging rendered the KML file on the map. You can see the map I created embedded below:

View Larger Map

If you look closely you’ll notice that there are not 100 counties on the map (only about 44). Google MyMaps will support 200 pushpins on a map, but when you add in complex polygons the number of polygons and associated pushpins it will support goes down significantly. In the MyMaps application it gets around this problem by paginating the KML file into multiple maps each supporting the maximum number of pushpins, lines or polygons. Unfortunately you can only embed one map page at a time, so the map above only shows the first set of polygons.

An interesting observation in the Microsoft blog post about KML support noted that, “on Google Maps the polygons representing the parks didn’t load at all”. Our KML rendered the polygons fine, but we took an extra step in GeoCommons to generate our polygons as multigeometries where a pushpin with the data is included inside the polygon and highlights when you mouse over (at least in Google Earth). So, my hunch is that in order to get polygon KML to render in Google MyMaps you need to structure it as a multigeometry, or they’ve added the functionality since then. It would be great to not to have to add the pushpin to get the data, and enable clickable polygons in both Google Earth and Google Maps.

On the plus side Google MyMaps does a good job handling multi-polygons. A multi-polygon is when you have multiple polygons representing one geographic entity. For instance the United States of America consists several separate polygons, including Alaska, the Hawaiian Islands, and the contiguous states. Several of the counties in our test data set had multi-polygons and you can see those rendered in detail in the embedded map below:

View Larger Map

A second plus for Google MyMaps is balloon support for the data that shows all the attributes in a nicely parsed list. Even when I loaded up a census data set with 74 attributes it listed them all out with a scroll bar. So to recap:

Advantages = prepopulated title and description, quick load, multi-polygon support, full listing of data attributes.

Disadvantages = limited number of polygons rendered on one map, requires multigeometry KML to support clickable polygons, slow rendering of polygons, no ability to export KML or other standard.

Virtual Earth KML Support

Virtual Earth KML support is provided through the “Collections” feature. When you click “Import Collection” you are given the option to add a KML file (or GeoRSS or GPX). I uploaded the same county pollution file and Virtual Earth chugged along for a bit then gave me a message saying, “100 out 100 items uploaded”. I’ve tried this with other files and if the files has more than 200 features it will not upload all of them – just the first 200 then stop. Also if your KML file is over 2mb it will tell you it is too large. Over all this is a nice feature that lets you know the bounds of the system and what will work and what will not.

The second nice part is that all 100 counties made it on one map instead of just 44 as with Google. A second bonus was that Virtual Earth did not need the multigeometries to support the clickable polygons rendered on the map. In fact the multigeometries we included in our KML generation caused both a pushpin to be drawn and and second square that gets highlighted when you mouse over the polygon. You can check out the map here and see the screen shot below:

Sadly Virtual Earth does not support embeds, so just the screen shot and link. Another small ding, a,s you can see in the screen shot, is that Virtual Earth does not support multiple polygons. The spots where you see push pins instead of polygons is indicative of multiple polygons representing a county, like Galveston, that could not be rendered so a push pin was placed there instead. It still gets the job done, but there is still something dissatisfying about America’s or any other political unit’s borders being replaced by a push pin. The last complaint is Virtual Earth only supports a limited number of characters for attributes, so when I tested a census file with 74 attributes I only got the first twenty or so and they were not well formated. So to recap:

Advantages = ability to render more polygons, ability to render polygons faster, ability to support clickable polygons without mulitgeometries, ability to export KML (and other formats)

Disadvantages = inability to support multi-polygons, slow to load KML, limited support of data attributes, no support of balloon styling

Over all I would give a slight edge to Virtual Earth when it comes to KML support from our unique perspective. Specifically the ability to load a larger number of polygons on a map and make those easily clickable allows more of our content to be leveraged at this point. It will be interesting to see how Google, Microsoft and others continue to enhance KML support to make more data available. I believe there is still a long way to go and the vast majority of the datasets in GeoCommons are too large for either to handle at this point. As the GeoWeb and the data it interconnects becomes more sophisticated I think it will be a necessity to greatly increase the amount and complexity of data that can be handled in a browser based map. Hopefully the market pushes Microsoft, Google and others to innovate in that direction.

The first screen shot is of the data details page, which contains the metadata information for the data set. In this case 2000 US Census data at the tract level for Alabama:

Here you can see the major elements we are capturing in a user friendly graphical lay out. One of the cool new bits is the system automatically calculates statistics when you upload the data. Being able to data mine and run statistics on the fly is one of the new developments we are particularly excited about.

All the metadata on the data details page is exposed as Dublin Core elements which should make them machine readable to the rest of the world:

Also there are links to FGDC and ISO 19115 metadata mappings which take you to simple text pages with the indicated information. We probably need another pass to get these completely correct, but the infrastructure is all in place to do so.

FGDC looks like this:

ISO 19115 looks like this:

Hopefully this will help make the data in GeoCommons useful to multiple geospatial work flows. We hope having the ability to get data out in shapefile, KML, and .CSV (spreadsheets) will create more cross fertilization between GeoWeb and GIS users. With some luck it can help get more geospatial data out to the public that has been difficult to access in the past. A couple of examples below.

US Census Tract Data for Alabama

Global Maritime Shipping Lanes

Zillow Neighborhoods and Shipping Lanes (just because it looked kinda cool)

Thanks again for the feedback from folks on the metadata and we’ll keep iterating on getting it spot on.