After you’ve got your spreadsheet ready, Click on the ‘File’ tab, then go down to click the ‘Publish to the Web…’ button.

Next you’ll have to click the ‘Start Publishing’ Button. I would also recommend that you click the box that says ‘Automatically re-publish when changes are made’, this will allow you to update your maps on GeoCommons directly from Google Spreadsheets. After you’ve clicked these buttons, you will be allowed to get a link to the published data. Here you should change from the default ‘Web-page’ to ‘CSV’. This will give you a link to your published data below.Copy the Link. It should look something like this:

Now you can head over to GeoCommons.com and after you’ve signed in, click the button the “Add a URL link” in the upload section. Paste the link from Google spreadsheets where it tells you to enter the URL, then specify CSV in the format section.

You will need to geocode your dataset, this should be pretty self explainatory if you’ve ever used GeoCommons, so I wont go into the details, but just know that once you’ve geocoded this URL once, we will continue using the attributes you specified to geocode any additional data you add into your spreadsheet. Your final dataset will look something like this:

Because the raw data is linked to a live website, you’ll see the ‘Fetch Latest” button above the title. Clicking on this button will update the GeoCommons dataset based on any changes that may have happened to your spreadsheet, and by updating the dataset, you’ll also be updating all of the maps using this as a layer. This means that when you’re working with a dataset that is constantly changing and being updated you’ll be able to update your maps by just clicking one button.

I’ve added a short video to show you the entire process from start to finish in less than 2 minutes.

Starting with the Oscar dashboard, we collected all the Tweets that mentioned the nominees for best movie, best actor and best actress. From this collection of data we populated the dashboard with a broad array of analyses. For each Tweet we assigned it to a major market based on its geography and also calculated the sentiment for each Tweet. We took a quick pass at putting together the highlights of the analysis in the slides below:

There is a lot going on under the hood here, but to keep it simple we are collecting a set of tags from the Twitter streaming API then performing a variety of analysis against the Tweets we pull. The two main analytical tasks are determining geographic origin followed by analyzing the sentiment of the Tweet. For geography we perform it at three different levels: 1) grabbing the coordinates for Tweets from GPS enabled phones 2) taking the bounding box for Tweets from Geo-IP and user designations and 3) using the location from the users profile. We work progressively from 1) to 3) and typically get locations for between 30-60%. For sentiment we tested out a variety of API’s and ended up using Repustate for this project and it held up well to the load.

This approach is not without its challenges. Profile location is notoriously ambiguous as research studies have elucidated. While there are good mitigation strategies for the profile issue we’ve seen a larger issue plague geographic analysis of Twitter. In the vast majority of Twitter geographic visualizations the data is geocoded and represented as a point on the map. The problem is the accuracy of this point versus where the Tweet actually came from varies wildly. In the case of lat long coordinates from a GPS enabled mobile phone this can be accurate within a few feet, but in the case of a profile city geocode it can be hundreds of miles off. Despite large variances in accuracy these points are typically shown as the same, which can cause misleading results.

To solve this problem we aggregate all the Tweets to polygons – in this case major market areas. The key is that the polygons you are aggregating to are larger than your accuracy error bound from geocoding. The cool thing the team did with the Oscar dashboard was make it so these aggregations happen dynamically. As Tweets come in they are intersected with major market polygons and the summary statistics are calculated for each major market. For any of the markets just click the graduated circle for it to get the aggregated statistics. Also you can click multiple movies or actors/actresses and it will calculate the aggregate summary statistics for any of the clicked items. You can also see Tweets from mobile devices by clicking “Current Tweets” to see exact locations and animate them over time by clicking “play”.

A second challenge with location based sentiment analysis is how meaningful are the results. I think one of the things we miss are margin of error calculations for sentiment analysis. Once we’ve aggregated data we have a sample size for that geography that we can calculate a margine of error against. In the summary statistics for each major market you can find a margin of error calculation for the sample size. This allows the viewer to know the confidence level for any analysis by geography.

The last nuance we’ve added to the dashboard is the ability the bring in demographics by major market to overlay a variety of income, ethnicity, age and gender beneath the Twitter sentiment. This allows users to see how a variety of demographics trends intersect the sentiment data. There is lots to play with a we look forward to feedback on how it can be approved and applied to other use cases.

A combination of talks and break-out sessions the event was interesting.  The mix of students, academics and professionals meant there were different views on the suitability of crowd-sourced information.  Discussion within my break-out group ranged from how can we verify the crowd to trust the information to how can we incentivize people to provide more structured information.  There was also the typical concern of how can responders know if they can trust information.  This had already been brought up in Michael Goodchild‘s talk “It’s About Time: The Temporal Dimension in VGI,” the idea being having some unverified data is better than having no data.  I think wider acceptance of crowd-sourced information is just a matter of better analysis tools to determine what data is better and encouraging the crowd to submit data so there is more information available.

I gave a talk about the new collaborative analytics tools we’ve been adding to GeoCommons.  Specifically my talk was “Enabling Collaborative Analytics for Faster Answers in a Crisis,” the idea is that the next step in a crisis is enabling the crowd to perform analysis.  Traditionally analysts create reports which then go to decision makers.  If changes need to be made to the end report tasks go back to the analyst.

The next step in analysis is to enable everyone to perform analysis. There are key things that need to happen in a system for this to be effective though. The first is to make analytics easy, this allows the user to make good decisions when they perform their analysis. Within GeoCommons we have aimed to do this and you see results of it through the application, such as when making a thematic map. When deciding on classifications of data we allow the user to match the type of data by matching their histogram versus the available classification schemes.

By making decisions as easy as matching pictures but allowing the user to go into the details it means that both experts and beginners can perform useful analysis. The next steps in encouraging collaborative analytics is making analysis traceable and making results extend-able. That changes the flow of analysis from an analysis making a report to allow analysis to be branched from and putting analysis tools in the hands of decision makers. This eliminates the bottle neck of only have a small group of individuals that can perform analysis and allows for faster response.

See my full presentation below:

Bring along your open data, and Todd Huffman and I will show you how you can share it with the world. In exchange, we’ll buy you a beer (or other drink of your choice). Of course, the quality of the data will be representative of the quality of the beer.

Hope to see you there. If you can’t make it, look for future programs nearby.

Update: Want to participate remotely? Upload data to GeoCommons and tag it “beer4data”. Next time we’re in your town, or you’re in ours, we’ll buy the first round. Bonus points if you also make a map and tag it.

The major providers all put restrictions around geocoding making it especially difficult to do batch geocoding operations to get large chunks of data geo-enabled. Google, Yahoo and Microsoft’s geocoders are all geared to single address look ups, and not for mass data geo-enablement. There are services like batchgeocode.com that get around some of the limitations but are still restricted by provider’s TOS.

The second big issue with current geocoding is further upstream. All the geocoding API’s are dependent on NAVTEQ, TeleAtlas’s and a few other providers data to geocode against. So, if the street data companies don’t think a country has a big enough market you can’t geocode in these areas. This especailly limits the ability to geocode data in developing countries.

Our thought is the best solution to this problem is an open source geocoder. There have been other open source geocoder projects, some of which have taken criticism as a bad business decisions.

We’ve taken a slightly different approach. One, we enlisted the brilliant help of Schuyler to evolve his work from Geocoder.us to best take advantage of the work and community already existing. Second, we decided to make the Geocoder street data neutral. Meaning that you can plug whatever street data source you want into the geocoder and have it work – sometimes with a bit of tweaking. In the first go we’ve set up the geocoder to work with TIGER data and NAVTEQ. We chose these two mainly because they both use all CAPS for their names.

The hope is that with the community’s help we can extend the geocoder to work with a large number of other data sources. As Andrew mentioned in his post OpenStreetMap is top of the list. Integrating OSM data will be key enabling geocoding in developing countries and other areas overlooked by current commercial providers. I think this is one of many areas where the OSM community is really going to show its power. While the geocoder is currently only accessible to developers through github, stay tuned because we’ll be exposing it as a web application in GeoCommons shortly. We want everyone to be able to geo-enable their data and access it in whatever format meets their needs. Data wants to be free and we all win when the gates are unlocked.

It would never make it out of the labyrinth of classification schemas in the US government, but it would be fascinating to see what a crowdsourced search for Bin Laden would turn up if better data was made available from the intel/defense community. Since the government data will never be released we thought we could at least help make the open source data easily accesible. So, we took the available data in the MIT article plus relevant data on Afghanistan and pushed it into GeoCommons. We’ve embedded a map with our own take below.

Maker.maker_host=’http://maker.demo.geocommons.com’;Maker.finder_host=’http://finder.demo.geocommons.com’;Maker.core_host=’http://core.demo.geocommons.com’;
Maker.load_map(“maker_map_28″, “28″);

To view this map in GeoCommons Maker! click here.

In addition to the UCLA data we’ve added gridded population data for the area. A big part of the UCLA thesis was Osama would be, “in a larger town rather than a smaller and more isolated town where extinction rate would be higher”. So, the gridded data gives a rough view of population densities in the remote Tora Bora region.

Source data for the maps is here:

Structure Locations of Possible Hiding Spots of Osama Bin Laden, Parachinar, Pakistan, 2009
Tora Bora 10 KM Buffer Rings
Gridded Population Data, Afghanistan and Pakistan border near Tora Bora

Would be great to see what other folks can do with the data to promote other perspectives. Also a nice opportunity to show the power of opening data up for better analysis, QA and alternative perspectives. Kudos to the UCLA team – great to see geographers in the news for doing what they do best.

You can read Obama’s technology platform overview here. The plank that really grabbed my attention was the promise to “Open Up Government to its Citizens”. The idea that data about government (Congressional voting records) and created by the government (census data) should be easily available to the public. Specifically:

“Making government data available online in universally accessible formats to allow citizens to make use of that data to comment, derive value, and take action in their own communities. Greater access to environmental data, for example, will help citizens learn about pollution in their communities, provide information about local conditions back to government and empower people to protect themselves.”

The beauty is that we (the collective GeoWeb) have so many of these tools already built. The ability to deliver the data once it is made easily available has great promise. For instance here is EPA data on power plant emissions from GeoCommons:

From the map above you can see which power plants are producing the most poisonous CO2 emissions (click the down carrot on the layers box for the filter) or zoom into your specific neighborhood to see the plant and the type of environment around it. (Still refining the embed capability, but an example of how data can be virally spread).

The report goes on to recommend that the federal government should:

“Establishing pilot programs to open up government decision-making and involve the public in the work of agencies, not simply by soliciting opinions, but by tapping into the vast and distributed expertise of the American citizenry to help government make more informed decisions.”

This strikes again at the heart of the GeoWeb – enabling collaboration of experts and citizens across the country. Several projects and companies have pioneered dynamic collaboration around maps. Below is a Google MyMap with feedback around the GeoCommons power plant data in Florida”

View Larger Map

The blue push pins are the user generated feedback linking to expert opinion and photos from the field. This is just the tip of the iceberg of what is possible with collaboration around maps. These approaches can also be leveraged inside of government agencies, which is another plank in the Obama technology platform:

“Employing technologies, including blogs, wikis and social networking tools, to modernize internal, cross-agency, and public communication and information sharing to improve government decision making.”

We’ve seen a lot of this type of work going on in the intelligence community with Intelink, Intellipedia, and A-Space. There is also data fusion and sharing concepts, like the EPA’s Central Data Exchange. I’d love to hear other projects that fit in with the three planks, and more importantly existing or planned GeoWeb technologies that could help enable the new vision. I’ve really only highlighted two and I know there are tons more out there.

There seem to be two fundamental problems, of many, worsening our current economic quagmire. 1) The housing bubble pushed home prices to levels most working Americans could not afford and to keep the bubble going the financial community became very creative with mortgages and how the risk associated with them was calculated. The end result was lots of people in houses they could not really afford and very little transparency in the risk this created in the financial markets. There is a lot more to the story but for the sake of brevity we’ll leave it at that. 2) Credit liquidity in the current market has almost ossified causing our collective economic gears to come to a rattling halt. Wall Street freaks…the media freaks…the consumer freaks (no spending)…sales of goods plummet…Wall Street freaks again…media fuels more freaking…rinse and repeat.

To break the cycle it would seem logical that liquidity needs to be injected into the market. A lot of pundits have looked at this being solved by the government buying up the bad assets, giving capital to the banks in return for equity stakes, and several other derivative plans. While all these ideas have their merits and risks the idea I exposed on the plane was slightly different. Back to the core issues – I saw the biggest failing being lack of market transparency and a fundamental mismatch between supply and demand in the housing market. So how could we restore transparency to the market while getting people in homes they can actually afford thus freeing capital for consumer spending and financial investment.

My answer was a foreclosure clearing house. This may be Polly Anna and not feasible, but it made for a fun intellectual exercise. There has been lots of talk around providing bail outs to people whose homes are foreclosing, but even this will be short term and will not solve the fundamental problem that they are in a home they cannot afford. The only real solution is to put these individuals and families into homes they can afford. The easy credit and risk shell game that banks ran has created a basic mismatch of people buying supply with demand they did not really have.

The clearing house is a simple idea of providing a transparent market place where people can trade down to houses they can afford and have new loans guaranteed to do so. The loans could be guaranteed by the government but competed for by the banks. Banks that already have the mortgages on existing properties could have the choice of refinancing the house so the owner could afford the payments (that would be their own risk calculation) or entering the home into the clearing house. Also the home owner could have the choice to enter their home into the clearing house if they would like to trade down voluntarily.

The clearing house itself could run like many of the existing home real estate market places matching buyers and sellers (Zillow, Trullia. RedFin etc.). In fact the government could probably contract with one of the sites to run the technology side of the clearing house at a reasonable cost. Once a person’s home was identified for purchase they would then be free to look for a new home in the clearinghouse they could afford. The government backing would allow loans to be made so the individual, now free of the foreclosed home, could buy a new home they could afford. Banks would still compete to provide the best rate and terms to new owner, but the risk would all be transparent to the government since they would be providing financial backing and to the owners so they were not mislead into buying more house than they could afford (again).

In theory this should introduce liquidity back into the market and with a little time put liquidity back into the consumer market since the majority of a person’s paycheck would no longer be going to a mortgage. The market would be transparent again but not run or partially owned by the government. I would argue that it was not capitalism or the market economy that broke during this financial crisis, but a loss of transparency and a resulting hiding of risk. In fixing the crisis the government’s role should be ensuring transparency in the market place so that it can function effectively. My idea is most likely off the deep end, but I do hope government action is centered around restoring transparency and restoring liquidity to the market. If you were Sec. Paulson for a day what would you do with $700 billion? There are no shortage of smart people around the globe. Can we crowdsource an answer?