The major providers all put restrictions around geocoding making it especially difficult to do batch geocoding operations to get large chunks of data geo-enabled. Google, Yahoo and Microsoft’s geocoders are all geared to single address look ups, and not for mass data geo-enablement. There are services like batchgeocode.com that get around some of the limitations but are still restricted by provider’s TOS.

The second big issue with current geocoding is further upstream. All the geocoding API’s are dependent on NAVTEQ, TeleAtlas’s and a few other providers data to geocode against. So, if the street data companies don’t think a country has a big enough market you can’t geocode in these areas. This especailly limits the ability to geocode data in developing countries.

Our thought is the best solution to this problem is an open source geocoder. There have been other open source geocoder projects, some of which have taken criticism as a bad business decisions.

We’ve taken a slightly different approach. One, we enlisted the brilliant help of Schuyler to evolve his work from Geocoder.us to best take advantage of the work and community already existing. Second, we decided to make the Geocoder street data neutral. Meaning that you can plug whatever street data source you want into the geocoder and have it work – sometimes with a bit of tweaking. In the first go we’ve set up the geocoder to work with TIGER data and NAVTEQ. We chose these two mainly because they both use all CAPS for their names.

The hope is that with the community’s help we can extend the geocoder to work with a large number of other data sources. As Andrew mentioned in his post OpenStreetMap is top of the list. Integrating OSM data will be key enabling geocoding in developing countries and other areas overlooked by current commercial providers. I think this is one of many areas where the OSM community is really going to show its power. While the geocoder is currently only accessible to developers through github, stay tuned because we’ll be exposing it as a web application in GeoCommons shortly. We want everyone to be able to geo-enable their data and access it in whatever format meets their needs. Data wants to be free and we all win when the gates are unlocked.

You can read Obama’s technology platform overview here. The plank that really grabbed my attention was the promise to “Open Up Government to its Citizens”. The idea that data about government (Congressional voting records) and created by the government (census data) should be easily available to the public. Specifically:

“Making government data available online in universally accessible formats to allow citizens to make use of that data to comment, derive value, and take action in their own communities. Greater access to environmental data, for example, will help citizens learn about pollution in their communities, provide information about local conditions back to government and empower people to protect themselves.”

The beauty is that we (the collective GeoWeb) have so many of these tools already built. The ability to deliver the data once it is made easily available has great promise. For instance here is EPA data on power plant emissions from GeoCommons:

From the map above you can see which power plants are producing the most poisonous CO2 emissions (click the down carrot on the layers box for the filter) or zoom into your specific neighborhood to see the plant and the type of environment around it. (Still refining the embed capability, but an example of how data can be virally spread).

The report goes on to recommend that the federal government should:

“Establishing pilot programs to open up government decision-making and involve the public in the work of agencies, not simply by soliciting opinions, but by tapping into the vast and distributed expertise of the American citizenry to help government make more informed decisions.”

This strikes again at the heart of the GeoWeb – enabling collaboration of experts and citizens across the country. Several projects and companies have pioneered dynamic collaboration around maps. Below is a Google MyMap with feedback around the GeoCommons power plant data in Florida”

View Larger Map

The blue push pins are the user generated feedback linking to expert opinion and photos from the field. This is just the tip of the iceberg of what is possible with collaboration around maps. These approaches can also be leveraged inside of government agencies, which is another plank in the Obama technology platform:

“Employing technologies, including blogs, wikis and social networking tools, to modernize internal, cross-agency, and public communication and information sharing to improve government decision making.”

We’ve seen a lot of this type of work going on in the intelligence community with Intelink, Intellipedia, and A-Space. There is also data fusion and sharing concepts, like the EPA’s Central Data Exchange. I’d love to hear other projects that fit in with the three planks, and more importantly existing or planned GeoWeb technologies that could help enable the new vision. I’ve really only highlighted two and I know there are tons more out there.

The Catholicgauze reflects on past Geography Awareness Weeks and discusses the trends of three geography surveys from the years 1988, 2002 and 2006. Overall, the past surveys show little or no change since the first study in 1988. The findings indicate that Americans are lacking in geographic literacy. Those that did well on the surveys include people who took and completed a geography course, recent immigrants, frequent travelers and those that keep up with world events.

Mark Doherty of Integraph presented a detailed overview of where the technology of GIS has been, where it is now, and where it is going at the Rocket City Geospatial Conference in Huntsville, AL. He shared four trends ‘converging to create a potentially unique opportunity’ in regards to the new era of computing. The four include SOA and standards, cloud computing, orchestration and thin clients/rich internet applications.

MyGeo Position, an application to help geo-code a single address, was deemed as ‘very accurate’ by Keir Clarke over at Google Maps Mania. The site offers a number of useful facilities for geo-coding a position, including finetuning using a drag and drop marker, geoposition as a float or in degrees, and an iGoogle gadget for quick geocoding from a personal landing page. MyGeoPosition is available in English, German, Dutch and Bulgarian.

O’Reilly’s Where2.0 2009 Call for Participation is now open. Submit your proposal here before December 2^nd.

One of the most useful references for me in this process was a workshop Tom Carden gave at ETech last year on the data rendering capabilities of a variety of approaches. The readers digest version of the workshop went something along these lines:

HTML/Javascript – handles 100-1000 data points – loads in .1 seconds
Flash – handles up to 10,000 data points – loads in 1 second
Java/Processing – handles up to 100,000 points – loads in 10 seconds
OpenGL – handles upwards of 1,000,000 points – loads in 100 seconds

For Maker we wanted to be able to handle 10,000+ points/polygons and there was no way JavaScript was going to be able to handle it. Of course rendering the data was just one of many problems. Not only did we have to render the data but also parse it from the server out to the client while running the mathematical operation enabling you to take advantage of the structured data being sent. The team came with lots of clever tricks to pull it off, but the level of performance afforded by using Flash for rendering the vector data was not available with JavaScript. Processing could be a very cool option as the technology matures. Silverlight could also be a great option if they can get the plug-in universally embedded into browsers as with Flash.

While Flash was a great option for the tiling and vector rendering we did not want to build out the entire application in Flash for a variety of reasons. In GeoCommons everything outside of the map itself is JavaScript/HTML. This is probably rudimentary for many folks, but reading the debate on James’ blog I think sometimes developers lose sight of picking the best tool for the job. Oftentimes it is easy to get wedded to an approach just because it is what you know well. We were complete Flash rookies when we started, but got some great help from Tom with Modest Maps, Axis Maps with the Flash development and cartography, hired some full time resources, and learned a lot on our own. It ended up being a great approach for the specific problems we were facing. As long as you are using standard interfaces in your development, you should be able to fluidly adapt to the technology that makes the most sense for your set of problems.

Creating the map presented us a classic cartographic decision – should we do a data visualization or a data analysis? While this is a very distinct difference, I think it gets largely lost by most GeoWeb users, and hopefully this little example will help illustrate the importance of the difference.

For Laurie’s maps she wanted to show the relative amounts of known natural gas and crude oil around the globe. The data set she had collected from the USGS provided polygons where petroleum and natural gas were located. The most straight forward way to map the data was to create a thematic map shading the polygons based on the amount of oil or natural gas contained in each. In cartographic circles this is called a choropleth map. Below is an example of a choropleth map in Maker! with the oil data from Laurie’s post:

Known Petroleum Reserves

You could also use the centroid of each polygon to make a proportional symbol map – where the size of the symbol is representative of the amount of oil or natural gas in the location. Here is an example of using the natural gas data set but rendered as proportional symbols:

Known Natural Gas Reserves

Proportional or graduated symbols are a great choice when you have small polygons with high values that might get lost in a choropleth map with many large polygons. For instance in the oil reserve map Venezuela has a very high oil reserve, but since the polygon is quite small it is easy not to notice it on the map.

Another technique for getting small places noticed is using extruded polygons, which are 3D. Bjorn Sandvik has done a great job promoting these techniques in Google Earth with his Thematic Mapping Engine. Below is an example of the oil reserve data as extruded polygons on a 3D globe:

3D globes have some shortcomings like accuracy and not being able to see the whole globe at once, but offer a great dynamic way to interact with data. For the geopolitics of oil, we decided to go with creating a thematic map using extruded polygons but on a 2D projection of the earth. That way we could see all the continents and curvature of the earth accuracy would be diminished. While not as cool as Bjorn’s virtual globe maps they got the job done:

A second option we looked at was doing a spatial analysis of the oil data. Instead of visualizing the data values for the polygons, we did an analysis of the spatial distribution of the data. In this case we thought it would be interesting to analyze the spatial density of the oil reserve locations. To do so we needed to convert the polygons to points based on the centroid of each polygon. Then we could run a kernel density analysis of the data. This sounds fancy, but it’s really just placing a grid over the data and tabulating counts for each cell with a bit of fancy math for smoothing to create a continuous surface. While this is not terribly complex, it is much different than just visualizing the data. The results of the density analysis for the oil data can be seen in the map below:

We ended up not using the density analysis because it was not really accurate since the source data had been polygons and not points. If the source data had been well locations weighted by oil production it would have been spot on.

The confusion between geodata visualization and analysis was one of the reasons we deprecated GeoIQ (sometimes called heatmaps for Google maps). Spatial density analysis has popped in lots of applications since the original work but I think it is still commonly misunderstood by users. We found many users that thought the hot spot was the highest single point value on the map, which often it was not. Few realized the hot spot was the cluster of points that were closest together with the highest aggregate value. I think with the right work flow or user interface it can be a great tool, but we are not quite there yet. The difference between the two only becomes apparent when users can have access to geodata visualization tools (i.e. thematic maps) and geodata analysis tools (i.e. density maps).

Photo host Flickr announced a new addition to their existing features: geotagging. Users can now drag their photos to a Yahoo Map through a link provided in their profile. Flickr will still offer Organizr, and highly recommends using that tool for batch geotagging. The new feature will be used to easily tag single photos. Check out this link that has Beijing Flickr photos on top of OpenStreetMap based data. Interestingly, the base Yahoo map for Beijing has hardly any road data and no street names.

Russia-Georgia’s war has been mapped. The CatholicGauze has been keeping track of the battle locations as the war progresses. Both Google Maps and Live Maps are covering the battle areas, even though Google has been called out for not finding the correct data for regions in Georgia.

A post from Sean Gillies asks for distinction between “web GIS” and “Geoweb”. James Fee and Andrew commented on the post, saying that “WebGIS” is tools and technology while “geoweb” is a whole other “social web”.