When we launched GeoCommons three years ago we had high hopes that it would turn into the vibrant community that it is today.  However, I must admit that from a technological standpoint we did not fully predict what the underlying GeoIQ platform would become – a complex, scalable set of distributed services that enable users to ingest and geo-enable data from an ever-increasing number of sources, visualize data in a various forms (maps, charts, images), and analyze data to uncover interesting information.  The GeoCommons of today is very different from what was launched in 2008.

Leading up to the launch of GeoCommons 2.0 our signup, data upload, and map-making rates soared to all-time highs, and as a consequence it became alarmingly clear that the infrastructure that has always been the backbone of GeoCommons would not scale well enough to support GeoCommons 2.0 – nor our ever-evolving plans for it.  It needed a serious overhaul from the bottom up, not due to poor planning or architecture three years ago – in fact the old GeoCommons infrastructure ran largely unmodified from day one without a single major hiccup – but simply because GeoCommons 1.0 was a very different beast from the GeoCommons we launched last week.

So while the GeoIQ engineering team began to build out GeoCommons 2.0′s features we undertook a parallel effort to migrate our infrastructure from a slowly melting half-rack in Washington, D.C. to the magical wonderland known as the Cloud – Amazon AWS, to be specific.

While the GeoIQ platform has been deployed in numerous environments and configurations and is not tied to any specific cloud service, Amazon was an obvious home for GeoCommons for a number of reasons – beyond the fact that it enables us to scale performance and costs as needed.  Firstly, like most startups we’re a very small team and we simply don’t have the resources to stand up, administer, and scale our own physical hardware, nor to manage most of the services that are essential in any scalable infrastructure.   Talent abounds at GeoIQ, but it usually abounds in 10 places at once – we can’t afford to waste our time cutting Cat 5 to length, swapping out bad drives, kicking faulty cooling systems, or calling REDACTED tech support because our overpriced Internet pipe is on the fritz.

Secondly, we’ve been using AWS for GeoIQ Cloud deployments for years and are comfortable with it from an operational standpoint.  Despite the recent outage in the US-East region we trust its reliability, and, perhaps more importantly, we’ve learned through years of trial and error how to effectively utilize AWS to ensure GeoCommons‘ reliability and get the best possible performance out of the GeoIQ stack.  In fact, the US-East outage could not have come at a better time for us.  We were in the midst of testing the new infrastructure and the outage afforded us an opportunity to test our failover plan during a real, unplanned, and uncontrolled situation.  I’m proud to say that (almost) everything went as planned, although some of us did lose more hairs than normal on that day.

Finally, we are consistently impressed by the breadth of Amazon’s services and the rate at which new services are being announced.  While no cloud is truly fire-and-forget, Amazon has come the closest by far.  I trust that they’ll continue to announce stellar services that allow us to further sunset our custom deployment solutions, which can only make our lives easier and allow us to focus on the things we know best – GeoCommons & GeoIQ.

I’ll go ahead and stop before this turns into a love poem to Amazon (too late?), but over the next week or so I’ll be writing posts on our developer blog detailing the architectural and technological changes that necessitated our migration to the cloud.  I will also describe how we’ve used Amazon’s services and outline some key lessons we’ve learned along the way.  Hopefully you’ll find them far more informative and beneficial to your work than this post turned out to be!

Stay tuned!

The story has a nice quote of me saying it was an impossible task to try and control all the geodata out there and who has access to it. The part that did not air is that no one even knows what data is accessible and not accessible to the public. While we do have a good index and census of most of the web pages that exist, we have much less understanding of the databases including geospatial databases connected to the Web (often called the Deep Web). The indexes run by Google and others do a great job finding web pages but databases are a different game. A Cal Berkley study by Bergman found that, “the deep web consists of about 91,000 terabytes. By contrast, the surface web, which is easily reached by search engines, is only about 167 terabytes.” While it is uncertain how much of this data is geospatial in nature it is fair to assume it is a considerable amount of data that we largely have little clue about. Often times government agencies do not even realize what data they have online available to the public, and we definitely do not have a comprehensive way to understand the entire universe of geospatial data. What raised so much alarm with our original research were the authorities realizing that that the data was available open source. Everyone clamored the work should be classified, but the source data is all still out there hidden in myriad local, state, federal and NGO data repositories. This begs the question, how are we going to control a world of data that we have so little comprehension of?

In order to move towards greater security I believe we actually need to open up more so that the entirety of geospatial data can be indexed. We will have no true idea as to what geospatial data available to the public is potentially dangerous until know what is out there. The move towards making KML an OGC standard is a great first step as a standard geospatial data format for the Web. Although KML natively is geared towards providing a geographic framework for text, html, pictures etc., and not structured information like databases. We’ve been working on changing that by ensuring a mechanism exists by which to include feature attribute data in the schema tag of KML . Some of this work has carried over into KML 2.2 as “extended data“.

Once you begin to index the geospatial data out there you are in a much better position to have a logical debate about what data is a security threat and what data contributes to the public good. For instance you may want to know where there have been hazardous pipeline accidents, but not divulge where critical pipeline routing junctures are. By opening up geospatial data, not only do we have a foundation to better insure dangerous data stays out of the hands of bad guys, but we also have the positive externality of a whole wealth of data being made available to the public to solve a wide range of problems.

Potential next steps are even more interesting. Once you have an open and indexable pool of geospatial data you can begin to leverage the power of crowdsourcing as discussed in the last blog post. In that post the discussion centered around crowdsourcing as a tool to improve the accuracy of data, but I believe it also has potential to create greater security through more resilient infrastructures. One of the lessons we learned from our infrastructure research was that by adding a small amount of diversity you could greatly increase resiliency. Take this example from Iraq of routes carrying goods to a series of destinations.

To connect up the locations convoys have to travel 433.5635 miles and they repeatedly use the same roads 33.13% of the time. Each time they repeatedly use the same path the vehicles are further exposed to IEDs and snipers. If we run a little Monte Carlo simulation we can diversify the routes so the same roads are not used repeatedly and we only increase the distance traveled fractionally.

With the new set of routes the distance traveled is 436.8805 miles and the same roads are only used 25.03%; a 8.1% diversity improvement at only a .77% efficiency cost. This works well when you are in a centrally planned military environment, but what happens in the “rent seeking” world of civilian commuter traffic.

Here is where I think there is real potential for “crowdsourcing” to not only enable resilience but greater efficiency. Traffic congestion is typically caused by everyone using the same shortest route repeatedly and trying to maximize their own personal position at others expense (yes the tailgaters). Roughly this is why roads hit a phase transition (congestion) at only 15% of carrying capacity. What if we could add in a little diversity to the routes people take.

So lets take the argument full circle. Once you’ve opened and indexed a good chunk of geodata you can create a common base and road map that users can annotate or automatically ping. Then when traffic becomes congested, a road is closed or an accident happens a user could add data to the map (either automatically or manually) from their car or mobile. All users could then have the option of a requesting a diversified route that avoids the road problems that are being reported by the crowd (including the location of “the crowd” itself . There are technological leaps in this scenario, but is a concept you could easily employ today with existing technologies. Lets take Google Maps as an example. Users are requesting driving directions and also dragging those directions to create new routes. Just as with the Iraq routing map the same roads are being used repeatedly. You could add an option of avoiding the heavily crowd sourced routes which takes that input as another variable in calculating the best route (in addition to speed, distance, number of turns etc.). At the end of the day you have not only a more efficient system but a more resilient system. When a major catastrophe happens the same principles allows destroyed roads and other infrastructure failures to be quickly communicated and routed around. I believe enabling adaptive resiliency through technology and crowdsourcing is infinitely more valuable than our current infrastructure protection mindset where we invest in guns gates and guards. The private sector has never bought into this (and they own 85% of infrastructure), but if you show a means by which they can be more efficient and more resilient then you have a case worth listening to.

Roads and traffic are just one possible area that crowdsourced geospatial data could make a huge differentiation. Once you’ve opened up geospatial data there is the ability to build upon those data sets to not only generate more context, but to also allow that data to respond as situations change, like traffic. I think there is a very solid case that what little security we gain by locking up and trying to control geospatial data is greatly outweighed by the public benefits of opening up geospatial data.

We have teamed with Lockheed Martin to leverage our intelligent mapping services to address these problems in government markets. Lockheed has a long history in the geospatial space and has been very progressive in embracing advanced Web 2.0 technologies such as Intelligent Mapping and Wikis.

I thought it might be helpful to provide an example of the kinds of problems we are addressing with Lockheed. Let’s take a fictitious scenario of a military operation dealing with terrorist attacks in Iraq. Suppose I’m Sergeant Gorman and I’ve uploaded data on a spree of attacks that my patrol collected over the past week.

A GIS analyst at headquarters, in reviewing my data along with historical data from the last three years, notices a pattern of increasing Shia activity around Samarra and sends an alert to field units. The alert prompts me to scan for data on attacks tagged Shia and Samarra, where I find a photo from a previous attack that shows one of the locals we had suspected of being a Shia ring leader.

I post a geo-blog noting that this individual has been suspected of coordinating attacks in my sector. A flurry of responses from other patrol leaders indicates that the same individual has been seen in proximity of other attacks. A GIS analyst at headquarters validates the findings and generates a command report, which results in the order to apprehend the suspect. On our next patrol into Samarra, we locate him and discover a complex cell of terrorist Shia activity in the area.

While the account above is completely fictional, hopefully it conveys the power of democratizing geospatial information throughout an entire organization. The same principles apply to a variety of other environments, such as disaster response, homeland security, and intelligence, where enabling the entire organization to explore, create, and share geospatial information can enhance mission effectiveness. We are excited about the partnership with Lockheed Martin to bring these capabilities to market.

My original hope was to load up a dataset with lat longs for the dangerous bridges across the country, but after Raj spent many hours trying to sort out the geo-coding inaccuracies in the data set it became apparent that would not be possible. The latitude and longitude coordinates simply did not map to reality, the entire state of New Jersey was missing coordinates, and many time the last two digits of resolution were zeroed out. So, Raj aggregated the data to the next best level of resolution – counties. The bad news is you cannot tell where in that county the dangerous bridges are, but you can tell which counties have the riskiest bridges and if it is one you drive through. We are still working on trying to derive a finer grain picture and will definitely post those up if we can come up with something accurate.

Where are the the most dangerous bridges:

//

The top 5 most dangerous counties and the total number of dangerous bridges are:

<p /P

Garfield Oklahoma 78

Attala Mississippi 45

Allegheny Pennsylvania 42

Washington Pennsylvania 37

Montgomery Pennsylvania 36

So, how was this all calculated? First we took the National Bridge Inventory and grabbed the safety ratings for all bridge’s superstructure, substructure, and decks (found here):

“The NBI database contains ratings on the three primary components of a bridge: the deck, superstructure, and substructure. A bridge deck is the primary surface used for transportation. The deck is supported by the superstructure. This transfers the load of the deck and the traffic carried to the supports. Within the superstructure are the girders, stringers, and other structural elements. The substructure is the foundation of the bridge and transfers the loads of the structure to the ground. The superstructure is supported by the substructure elements, such as the abutments and piers.”

For each of the key bridge components, decks, superstructure, and substructure we provided counts by county for the number of bridges by their safety ratings (ranging from “failed” to “excellent”). We also created three indexes: 1) dangerous bridges (the sum of the number of failed, imminent, critical, and serious bridges), 2) risky bridges (the sum of the number of poor, fair, and satisfactory bridges) and 3) safe bridges (the sum of the number of good, very good, and excellent bridges).

The embedded map show the number of dangerous bridges rated by their superstructure. In the Minneapolis bridge collapse the superstructure was rated “poor” and all the bridges mapped are bridges in worse condition by safety rating. This is just one slice of the data. The best way to see how your location rates is to head to GeoCommons and create a map by clicking the “mymaps” tab and adding one of the three datasets. Just search “bridges” or by type “superstructure”, “substructure”, “decks”. Once you’ve made the map you can also click on the “about this dataset” link and create top ten lists for whatever data attribute you think is most critical. We hope this allows a mechanism for the public to discover if there are dangerous bridges in their backyard.