There have been TechCamps all around the world with great success in sharing experiences with the use of low-cost and easy to use technology by NGOs. At TechCamp Tel Aviv I will be walking through just a few of the examples of organizations that are using GeoIQ and GeoCommons to investigate data and tell compelling stories with the world. Groups such as the World Bank’s Mapping for Results and IADB’s MapAmericas, and even local groups such as the Bicycle Coalition of Philadelphia.

We will also be working over two days directly with NGOs on specific problems and objectives they have with making maps. I’d love to know if you have any open data for Israel or Palestine that we can also highlight to the local groups. We will make sure to post the videos and notes from the TechCamp when they’re online.

While it can be very valuable to work with private data we don’t always want to do that alone.  Often times we want to collaborate in groups with our private data.  Collaboration is a linchpin to GeoCommons, and we wanted to make sure this was a focus for private data.  So, we’ve added a groups functionality, which allows you to decide the GeoCommons’ users you would like to have access to your data.  You can collaboratively edit the map and work together to create awesome visualizations and analysis of data.

To provide a sense of how this all works I’ll do a quick run through of the new features.  A new user “Peter Fox” joins GeoCommons and wants to map out where he is most popular with fans.  After registering he navigates to “manage groups” on the user tab seen below:

By clinking manage groups Peter is taken to a group administration screen to set up his new group by giving it a name:

Now that Peter has given his new group a name he can start populating it with the users on GeoCommons who he would like to collaborate with.  Peter taps some of his most trusted fans through a  simple search interface and adds them to the group:

Once Peter has all his collaborators added he can continue to manage the group from the admin screen deleting and adding new users as needed:

If Peter would like to have more than one group he can sign up additional collaborators, and assign them to different topics or teams.  Adding and managing additional groups also has a simple interface for users:

Now that Peter has his new group all set he can keep his data private for just himself, or share it with one of his new groups.  He uploads some data comparing his popularity to Savage Garden from “Google Trends” and has the “option” right on upload to set his sharing permissions through a few simple check boxes:

Once Peter makes a map showing where he is most popular on Google Trends he can them also set permissions on who can access his map or data analysis.

The great thing is this secure permission system for private data and collaboration permeates all of GeoCommons.  Say you want to keep your point data private, but want to share an aggregation of the data to zip codes.  No problem!  Set the permission for the source data to private and the analysis as public.  Alternatively you did a correlation and you want to share the results with a small number of analysts to check your work before you publish.  Create a quick group on the fly – publish your results and collaborate on the findings or alternative analyses.  This is the great thing about doing everything in a pure Web platform.  Collaboration and sharing is seamless allowing you to track your work and collaborations across multiple disparate colleagues and data sources.   We are excited to see how the GeoCommons community leverages all the new opportunities and get feedback on how we can make it even better.

Background

Google’s Fusion Tables is a tool that allows users to quickly visualize tabular data. Within it there’s a lot of potential for exposing open data on the web in a free, public, and simple way. We here at GeoIQ are ALL about open, both in software development and geo-data, we all share similar ideas about how critical open data and software are to the web and particularly the “geo/location” tech industry.

So I thought I would do some experimenting this weekend. To be honest I haven’t played with Fusion Tables as much as I should have by now, but this weekend I thought, “why not?”. First off, I decided to try messing around and get some Fusion Tables data mapped in GeoCommons. As it turns out GeoCommons was fully ready for this, as was Fusion Tables, so doing this way all REALLY EASY. Since it was so easy I started, and nearly finished another , related hack, but more on that another day…

Let’s get down to it!

Fusion Tables to GeoCommons

I thought about starting off with a list of the similarities and differences between Fusion Tables and GeoCommons, but I decided I’d rather not. So in short I’m treating Fusion Tables like a simple database in the cloud, essentially just assuming its a source for data with an awesome HTTP based API. Simple enough, and its really not that far off from what Fusion Tables is in reality. Granted there are some solid built in tools for visualizing and filtering data but in many ways its a great source of public data.

GeoCommons is a web platform for collaborative data management, sharing, analysis and visualization. We really enjoy the idea of mapping data dynamically from the web, and we feel that we’ve built a tool that really kicks ass at doing it.

So for the rest of this demo/post I’m going to skip over the details of how Fusion Tables works, and simply treat it like a public source for open data. And Fusion Tables turned out to be my friend in this since it makes it easy to find public tables by providing a clear link to several public tables in left hand navigation list (see below), and searching Fusion Tables provides access to a bunch more tables as well.

http://www.google.com/fusiontables/Home

I’m just going to pick one at random (well almost ) for my demo:

The Global Historical Climate Network Station Inventory

I wanted data with a good number of data points, and this one has ~7k which is good enough to show off the speed of the new GeoCommons 2.0 map rendering. Now that we have some data we need to figure out how to get it out of Fusion Tables and into GeoCommons. We could just export the data from Fusion Tables and upload it into GeoCommons, but that’s not really what we want. We want to keep the data in Fusion Tables, but map it in GeoCommons. This way we can go and edit the data in Fusion Tables and our maps in GeoCommons can be updated without re-dumping and loading the data.

Fusion Tables makes it simple to export data as CSV via its user interface:

Its not exactly a simple task to figure out the correct export URL that they’re using to get the data, format and return it all as CSV, but that’ is the URL we want! GeoCommons can upload any URL pointing to CSV data and maintain memory of how to retrieve the data.  In GeoCommons this means the data source remains a link back to the original data. This makes it possible for us to keep data elsewhere but still access, edit, share, and visualize the data.

To get the URL for a CSV export of a Fusion Tables dataset I simply opened up Charles, the extremely awesome web debugging proxy. Charles logs all network traffic while its open and makes it really easy to see actions and responses across the web. From Charles we can see the “Fusion Tables -> File -> Export” command is simply calling the server with this:

http://www.google.com/fusiontables/exporttable?query=select+col0,+col1,+col2,+col3,+col4,+col5,+col6,+col7,+col8,+col9,+col10,+col11,+col12,+col13,+col14,+col15+from+267862

Awesome! Fusion Tables is just doing some SQL via the URL query string. That’s really easy for us, and there’s certainly no magic involved. This is exactly what we need for our GeoCommons experiment. However, using Charles, and all its awesomeness, might be a tricky thing for some people. There’s got to another way right? Yup. Google of course has a Fusion Tables API, glorious!

The API doesn’t mention the “exporttable” endpoint above though? So Charles still remains a special tool However from the API we can see all of the supported SQL calls for accessing Fusion Tables via the browser. So our export URL to dump the whole table into CSV can now look a little cleaner:

http://www.google.com/fusiontables/exporttable?query=select+*+from+267862

Now let’s shift over to GeoCommons and load this data into a map!

In GeoCommons we sign in, then go to the upload page at: http://geocommons.com/home – from here we have two options for uploading data: either as a file or a URL.

We want to select the URL option. In the URL input space we’re going to copy and paste in our Fusion Tables URL from above, and we’re going to select “Spreadsheet (csv)” from the list of format options. Next click “Continue”.

Since this dataset already has a latitude and longitude GeoCommons needs to do very little to get it ready. Thus we’re placed directly into GeoCommons’ data edit page where we fill out all the crucial metadata for this dataset. It’s important to note that if the Fusion Tables dataset doesn’t have lat/lon columns GeoCommons can still work with it by attempting to either perform a Geo-Join or a Geocode on the data.

Once we’re finished we can click “Save” at the bottom. Now our data are ready to be mapped. Again, in GeoCommons this is simple: we click the “Map Data” button in the upper right corner of the screen in newly saved overlay page. Our data will be loaded into a map and we can begin to style it. Once we save the map we can share it! Here’s the map I made:

http://geocommons.com/maps/76030

This proved to be really easy to do. It took a lot more time to simply write this post than actually load in the data and make a sweet map. Also, I was impressed with the speed on both ends of the chain. In terms of ways we can improve this work, GeoCommons could provide complete access to all of the Fusion Tables data simply by integrating our data searches into the Fusion Tables API. This would be way cool, and would open up access to more data which would be a good thing!

In Washington, DC there is a lot of data provided by the District government through their data catalog. You can download all the 311 requests from 2010 for example, there is even an API for 311 requests so developers can build additional 311 tools. In downloading the requests from 2010 I noticed that there is a field titled “SERVICEORDERSTATUS” and the values are Open/Closed/Overdue. I became interested in looking geographically at where the overdue tickets were versus the total number. I decided to focus on abandoned vehicles for my analysis. I created a subset spreadsheet of abandoned vehicle requests and then a subset of those requests that were specifically overdue. I then uploaded those into GeoCommons as datasets. The full abandoned vehicles 311 request dataset is available here and the specifically overdue ones are here. Once these were uploaded I aggregated the data by Ward, this way I could see where the most 311 requests were coming from as well as the most that were overdue. To further my analysis I calculated the percentage of overdue requests out of the total requests. The resulting map shows significantly more overdue requests by percentage in Ward 7 and Ward 8. This allows me to start asking further questions, if I actually was a journalist maybe I would approach the DC government with this information and see what they said. Perhaps I would interview residents in those Wards for further information as well. Finally when I’ve developed the full story I can include the map I’ve used for my analysis with my resulting story.

I include the map by clicking on the “Details” button.

Next I click the “Embed this map in your website” button, which brings up the map embed code. I can now copy and paste this code into my news site’s content management system.

Many news organizations are already using GeoCommons to help enhance their stories. Ones we’ve noticed have included:

LA Times: California’s Unemployment Rate is Second Highest in the Nation

Calgary Herald: Explore Dog Data in your Calgary Neighbourhood

Huffington Post: How The Economic Impact of the Oil Spill Is Spreading Beyond the Gulf

Atlantic Monthly: WikiLeaks Data on Afghanistan Deaths Visualized

The Roanoke Times: Roanoke’s Population Expands Over Decade

The Vancouver Sun: Worst Intersections: Vancouver

ABC World News Blog: Live Feed of Tweets Mentioning #Egypt #Jan25 #Tahrir

Richmond Times-Dispatch: Census 2010: Racial Breakdown by Geographic Area

Edmonton Journal: Edmonton Public School Attendance Maps

Ottawa Citizen: High Flying Prime Minister Harper Racks up Air Miles: Analysis

Are you a journalist already using GeoCommons and you’d like to be included on the list above? Let us know in the comments.
Did my DC analysis peak your interest and you might want to do a story? Checkout the map below:

View full map

I’m proud to present the GeoCommons Chrome App.  With Chrome if you go here and click ‘Install” the GeoCommons App is added to the front page of Chrome.

Don’t worry if you weren’t lucky enough to get a Cr-48, you can still check it out using your Chrome Browser as well.  If you were lucky enough to get a Cr-48 make your friends jealous by showing them the GeoCommons App on it.  Look for more Chrome fun from us soon, next stop?  Chrome Themes.

In using Alfred, I noticed a very nice feature to setup a web search. You can specify a shortcut command that will take the rest of what you enter and open up a search. The defaults include a Google search where you type “google bicycle” to do a search for bicycles on google.

I wanted to add a custom GeoCommons search. So I went into Alfred’s preferences, to “Custom Searches” and put “http://geocommons.com/search?query={query}” into the Search URL. I then specified the Keyword “gc” and saved the search. Now, I can press <command>-<space>, type in “gc environment” <enter> and immediately get a page of search results for environment data and maps from GeoCommons!

Almost exactly a year ago today Kate blogged about OpenSearch feature in GeoCommons where you can make GeoCommons a search-bar option. With Alfred you can now search GeoCommons directly from your desktop without even having to open your browser.