A combination of talks and break-out sessions the event was interesting.  The mix of students, academics and professionals meant there were different views on the suitability of crowd-sourced information.  Discussion within my break-out group ranged from how can we verify the crowd to trust the information to how can we incentivize people to provide more structured information.  There was also the typical concern of how can responders know if they can trust information.  This had already been brought up in Michael Goodchild‘s talk “It’s About Time: The Temporal Dimension in VGI,” the idea being having some unverified data is better than having no data.  I think wider acceptance of crowd-sourced information is just a matter of better analysis tools to determine what data is better and encouraging the crowd to submit data so there is more information available.

I gave a talk about the new collaborative analytics tools we’ve been adding to GeoCommons.  Specifically my talk was “Enabling Collaborative Analytics for Faster Answers in a Crisis,” the idea is that the next step in a crisis is enabling the crowd to perform analysis.  Traditionally analysts create reports which then go to decision makers.  If changes need to be made to the end report tasks go back to the analyst.

The next step in analysis is to enable everyone to perform analysis. There are key things that need to happen in a system for this to be effective though. The first is to make analytics easy, this allows the user to make good decisions when they perform their analysis. Within GeoCommons we have aimed to do this and you see results of it through the application, such as when making a thematic map. When deciding on classifications of data we allow the user to match the type of data by matching their histogram versus the available classification schemes.

By making decisions as easy as matching pictures but allowing the user to go into the details it means that both experts and beginners can perform useful analysis. The next steps in encouraging collaborative analytics is making analysis traceable and making results extend-able. That changes the flow of analysis from an analysis making a report to allow analysis to be branched from and putting analysis tools in the hands of decision makers. This eliminates the bottle neck of only have a small group of individuals that can perform analysis and allows for faster response.

See my full presentation below:

You can read Obama’s technology platform overview here. The plank that really grabbed my attention was the promise to “Open Up Government to its Citizens”. The idea that data about government (Congressional voting records) and created by the government (census data) should be easily available to the public. Specifically:

“Making government data available online in universally accessible formats to allow citizens to make use of that data to comment, derive value, and take action in their own communities. Greater access to environmental data, for example, will help citizens learn about pollution in their communities, provide information about local conditions back to government and empower people to protect themselves.”

The beauty is that we (the collective GeoWeb) have so many of these tools already built. The ability to deliver the data once it is made easily available has great promise. For instance here is EPA data on power plant emissions from GeoCommons:

From the map above you can see which power plants are producing the most poisonous CO2 emissions (click the down carrot on the layers box for the filter) or zoom into your specific neighborhood to see the plant and the type of environment around it. (Still refining the embed capability, but an example of how data can be virally spread).

The report goes on to recommend that the federal government should:

“Establishing pilot programs to open up government decision-making and involve the public in the work of agencies, not simply by soliciting opinions, but by tapping into the vast and distributed expertise of the American citizenry to help government make more informed decisions.”

This strikes again at the heart of the GeoWeb – enabling collaboration of experts and citizens across the country. Several projects and companies have pioneered dynamic collaboration around maps. Below is a Google MyMap with feedback around the GeoCommons power plant data in Florida”

View Larger Map

The blue push pins are the user generated feedback linking to expert opinion and photos from the field. This is just the tip of the iceberg of what is possible with collaboration around maps. These approaches can also be leveraged inside of government agencies, which is another plank in the Obama technology platform:

“Employing technologies, including blogs, wikis and social networking tools, to modernize internal, cross-agency, and public communication and information sharing to improve government decision making.”

We’ve seen a lot of this type of work going on in the intelligence community with Intelink, Intellipedia, and A-Space. There is also data fusion and sharing concepts, like the EPA’s Central Data Exchange. I’d love to hear other projects that fit in with the three planks, and more importantly existing or planned GeoWeb technologies that could help enable the new vision. I’ve really only highlighted two and I know there are tons more out there.

There seem to be two fundamental problems, of many, worsening our current economic quagmire. 1) The housing bubble pushed home prices to levels most working Americans could not afford and to keep the bubble going the financial community became very creative with mortgages and how the risk associated with them was calculated. The end result was lots of people in houses they could not really afford and very little transparency in the risk this created in the financial markets. There is a lot more to the story but for the sake of brevity we’ll leave it at that. 2) Credit liquidity in the current market has almost ossified causing our collective economic gears to come to a rattling halt. Wall Street freaks…the media freaks…the consumer freaks (no spending)…sales of goods plummet…Wall Street freaks again…media fuels more freaking…rinse and repeat.

To break the cycle it would seem logical that liquidity needs to be injected into the market. A lot of pundits have looked at this being solved by the government buying up the bad assets, giving capital to the banks in return for equity stakes, and several other derivative plans. While all these ideas have their merits and risks the idea I exposed on the plane was slightly different. Back to the core issues – I saw the biggest failing being lack of market transparency and a fundamental mismatch between supply and demand in the housing market. So how could we restore transparency to the market while getting people in homes they can actually afford thus freeing capital for consumer spending and financial investment.

My answer was a foreclosure clearing house. This may be Polly Anna and not feasible, but it made for a fun intellectual exercise. There has been lots of talk around providing bail outs to people whose homes are foreclosing, but even this will be short term and will not solve the fundamental problem that they are in a home they cannot afford. The only real solution is to put these individuals and families into homes they can afford. The easy credit and risk shell game that banks ran has created a basic mismatch of people buying supply with demand they did not really have.

The clearing house is a simple idea of providing a transparent market place where people can trade down to houses they can afford and have new loans guaranteed to do so. The loans could be guaranteed by the government but competed for by the banks. Banks that already have the mortgages on existing properties could have the choice of refinancing the house so the owner could afford the payments (that would be their own risk calculation) or entering the home into the clearing house. Also the home owner could have the choice to enter their home into the clearing house if they would like to trade down voluntarily.

The clearing house itself could run like many of the existing home real estate market places matching buyers and sellers (Zillow, Trullia. RedFin etc.). In fact the government could probably contract with one of the sites to run the technology side of the clearing house at a reasonable cost. Once a person’s home was identified for purchase they would then be free to look for a new home in the clearinghouse they could afford. The government backing would allow loans to be made so the individual, now free of the foreclosed home, could buy a new home they could afford. Banks would still compete to provide the best rate and terms to new owner, but the risk would all be transparent to the government since they would be providing financial backing and to the owners so they were not mislead into buying more house than they could afford (again).

In theory this should introduce liquidity back into the market and with a little time put liquidity back into the consumer market since the majority of a person’s paycheck would no longer be going to a mortgage. The market would be transparent again but not run or partially owned by the government. I would argue that it was not capitalism or the market economy that broke during this financial crisis, but a loss of transparency and a resulting hiding of risk. In fixing the crisis the government’s role should be ensuring transparency in the market place so that it can function effectively. My idea is most likely off the deep end, but I do hope government action is centered around restoring transparency and restoring liquidity to the market. If you were Sec. Paulson for a day what would you do with $700 billion? There are no shortage of smart people around the globe. Can we crowdsource an answer?

One side of the company felt that by not limiting access to data we were giving away the family jewels. The other side felt that open access was the best way to create a network effect for data by making it as accessible as possible. At the end of the day the open access philosophy prevailed, and from the sound of comments to James Fee’s post after GeoWeb, access to data is still an important facet to both GIS and GeoWeb users.

Now that Finder! has been out for a little while we’ve begun to see a big surge in downloads. I noted last week we hit 18,000 downloads and just a week later we are now over 28,000. This has caused us to take a second look at our access policies. “Knock on wood”, the system has scaled like a champ handling the traffic, but as we get ready to launch Maker! some concerns have come up about potential abuse and its effect on user experience.

The biggest concern is around systematic downloading of data and the potential for that to impact other users experiences on the site. The question is how to make the content available without impinging on the collective user experience. Wikipedia approaches this by making content available as one big tarball and asks users “Please do not use a web crawler to download large numbers of articles. Aggressive crawling of the server can cause a dramatic slow-down of Wikipedia. Our robots.txt blocks many ill-behaved bots.”

I’m not sure a giant tar ball of data is the best way to go for us, especially since the data is available in a variety of formats. A second option is to provide third party access to the data via an API. This API could also work for both download and upload. Andrei had an interesting suggestion in our last post:

“The two-way API will definitely help with the number of uploads. The cool thing to do, would be to add (”Add to Finder!”) a URL request:

…finder.com/add?file=file.kml&type=kml&name…”

If people have other ideas on how they could better access the data in bullk without impinging performance we’d love to hear them. Also thoughts on what the line is between fair use of content and abuse of the commons. It is a bit of gray line in my mind. Is systematic downloading (manually hitting every dataset) abusive? Is scraping datasets with bots abusive? The main goal in my mind is to provide the best service possible without creating a “tragedy of the commons“.

The first screen shot is of the data details page, which contains the metadata information for the data set. In this case 2000 US Census data at the tract level for Alabama:

Here you can see the major elements we are capturing in a user friendly graphical lay out. One of the cool new bits is the system automatically calculates statistics when you upload the data. Being able to data mine and run statistics on the fly is one of the new developments we are particularly excited about.

All the metadata on the data details page is exposed as Dublin Core elements which should make them machine readable to the rest of the world:

Also there are links to FGDC and ISO 19115 metadata mappings which take you to simple text pages with the indicated information. We probably need another pass to get these completely correct, but the infrastructure is all in place to do so.

FGDC looks like this:

ISO 19115 looks like this:

Hopefully this will help make the data in GeoCommons useful to multiple geospatial work flows. We hope having the ability to get data out in shapefile, KML, and .CSV (spreadsheets) will create more cross fertilization between GeoWeb and GIS users. With some luck it can help get more geospatial data out to the public that has been difficult to access in the past. A couple of examples below.

US Census Tract Data for Alabama

Global Maritime Shipping Lanes

Zillow Neighborhoods and Shipping Lanes (just because it looked kinda cool)

Thanks again for the feedback from folks on the metadata and we’ll keep iterating on getting it spot on.

1) people’s tags may be difficult for others to understand,
2) people may have tagged items inappropriately for others’ needs.

In short your users will not always implement tags in ways that are productive for the community – in the extreme resulting in Flickr’s 20 million unique tags. How many of those 20 million tags are misspelled words or so off the path they never get found.

In addition to the problems you encounter with folksonomies in general you have the further complications of geopspatial data. All geospatial data sets have location tags, but adding them in an unstructured way creates enough chaos that it is very difficult to leverage location tags in a thorough way. Secondly many potential users do not know the variety of geodata available. Put more simply they do not know what to search for, and having the ability to browse through data by topics is appealing.

Despite the downsides of folksonomies they are incredibly powerful and have been hugely effective in organizing vast amount of data on the web. So, as we worked on the next iteration of GeoCommons we started looking at possible hybrid approaches to folksonomies and hierarchies.

Specifically we looked at the two problems specific to geospatial data listed above 1) place tags and 2) organizing data for browsing. Solving the problems required both short term and long term solutions.

Fortunately we had a small advantage over many crowd sourced project in that we have a full time data team. They are a great group of folks that spend their day finding cool geodata and coming up with clever ways to organize it.

Through the data team and the other community members that contributed data to the first iteration of GeoCommons we had a big pool of data with a wide variety of tags to examine. What we found were some distinct trends in the tagging and titling of data. Across the data there were a commons set of tags that broke the data up into a useful set of distinct categories, but there were also many data sets that were tagged with elements that made them often indiscoverable. After the analysis we started to look at structures we could establish to help create self similarity in tagging that still had the flexibility to be adaptive.

The result was the creation of a location and topical taxonomy based on our existing corpus of data that has the intelligence to adapt as the content grows and evolves. I can’t go into the technical details in depth, but fundamentally the concept is to intelligently leverage the taxonomies and structures to provide suggestions to users to tag their data better.

In many cases this can be very simple – like providing tips on how to tag and title effectively to make your data more valuable to the community. For instance with titles we found across GeoCommons there were four key pieces of information used for datasets in the past.

1) Source name, 2) Original Name of Dataset from Source (or short description of dataset) 3) Geographic Area, 4) Time period of data

Examples:

Communicating this effectively to users is a great way to get better consistency across data contributions, while still allowing flexibility for users to be creative and bring in information that does fit the rigid mold of a hierarchy. Of course this is the most simple and you can get far more clever.

Del.icio.us for instance has a great feature that notifies a user they are putting in a new tag no one has used before and asking if that is what they meant to do. You can also suggest tags from your taxonomy that are semantically related to the data the user is contributing. This creates a consistency across tags that makes data easier to find as the system scales to larger volumes.

The nice thing about taxonomies as opposed to folksonomies is that they can be structured as trees, which means you can compute across them quite easily. With a solid and adaptive taxonomy in place you can go a long ways in intelligently guiding users towards creating better and more consistent tags. At least that is what we think and it will be fun to see how it works out after the launch.

From that standpoint the proposal is for an implementation not a standard. We have just about 5,000 unique datasets and about 70,000 data layers, and it would be great to expose useful metadata for the data. The data covers the gambit, from EPA toxic release sites to the number of Facebook users by city. The system and metadata requirements needs to be flexible enough to accommodate both a user uploading Facebook data and one uploading EPA data.

While GIS users might not be intimidated by a metadata form with 75 or even 335 elements your average Web/GeoWeb user definitely will be. The goal with GeoCommons is to provide a destination where both communities can consume and share data, and I think both communities will find tools and data that are useful.

In regard to the metadata elements we proposed to map to in the last post, we were looking for those that both technical and non-technical users would understand, and also automatically trap as many additional elements as possible. To cover technical users, that have a full compliment of metadata, the plan is to have an element where you can you can provide a link to a full metadata specification.

The comments directing us to the ISO 19115 standard were very useful and we are looking to see what elements we are missing to map to that standard as we evolve. The thing we want to make sure we get right is finding to best set of metadata elements to request from users. Balancing the fact that if we have a huge number of elements, most people are going to go running for the hills.

Right now it looks like we’ll have 17-20 elements that will map to Dublin Core, FGDC, and in a next release ISO 19115. So, for each data set in Geocommons you’ll have a page that lists those 17-20 elements in the metadata format technical folks are used to seeing. This should also provide a means by which to explore federating the data with other applications and search approaches.

The goal here is to create a bridge between content being created for the GeoWeb and content created for the GIS world and make both usable and remixable by the web community as a whole. I fully respect the motivations and requirements for the GIS metadata specifications out there, and I hope we can leverage them to create an implementation that will see a high level of adoption.

Without adoption standards are pretty hollow as we’ve seen with all the work that went into GML versus the much lighter specifications for KML and GeoRSS. While both have their place it is clear what the market is supporting. As more geospatial data is created outside of the government we are not going to have the government mandate to force metadata creation and what the market accepts is going to become increasing critical – IMHO. Look forward to getting more feedback as we get ready to launch.

Unfortunately when the wagons circle everything starts picking up the label – in this case semantic. This is especially dangerous when you have a word like “semantics” that can be defined, so many different ways. Just look at the definition tree created by Wikipedia:

*Semantics is the study of meaning in communication.
*In computer science semantics reflects the meaning of programs or functions.
*The Semantic Web refers to the extension of the World Wide Web through the embedding of additional semantic metadata

More often I see folks labeling things semantic that are really syntax. “Syntax” being the rules to construct and define something like a sentence or line of code and “semantics” the meaning of those rules or definitions. Syntax is fairly easy and semantics are fairly hard, as most folks in artificial intelligence would argue. Even going so far as saying all programming languages other than LISP are syntax and not semantic.

This is a bit more clear with an example. Lets take the Textwise announcement – a technology that will parse plain text on a website or elsewhere and categorizes it to predefined topics. One example in the Techcrunch comments was the following:

input text:
Call us crazy, but we think there are some brilliant minds out there that can find some really amazing uses for this incredibly powerful and scalable technology. Think you’re up to the Challenge? We think you are!

categories (ranked from 0 (worst) to 100 (best)):
Shopping/Health/Alternative/Hypnotherapy/Audio_and_Video 43 Business/Telecommunications/Services/Wireless/Software 33 Arts/Music/Bands_and_Artists/311/Tablature 28
Computers/Internet/Consultants/Research 26 Shopping/Health/Alternative/Meditation/Audio_and_Video 25

The output is really not telling me anything about the meaning of the text just setting up rules to provide categorization. So I would definitely put this in the syntax and not semantic category. I would also say what Yahoo! is doing is really more syntax than semantics although there is the possibility of building truly semantic technologies on top of what they are enabling. They’ve created a set of rules based on rich standards to allow applications to be built. Remains to be seen what will come of it, but in rush of market buzz I think it is easy to miss that building truly semantic technologies is quite hard. Some folks in AI (the Chinese room) would argue machines are not even capable of semantic meaning or understanding.

From this perspective I think we’ll see a lot of people building applications based on syntax that reorganize and categorize content by giving the “page web” a bit of structure. Oddly its like we’ve gone full circle back to DMOZ. While these technologies may be clever and useful I do not think they will fundamentally change the Web. In the other category I think we’ll see a few companies pushing towards something more sophisticated (call it a semantic, implicit, computational web) where new data and services are mixed with existing web content to provide answers to users questions.

The heat maps that follow show what parts of the country the top performers for each position come from; in more technical terms, the maps are players’ hometowns weighted by fantasy football points. For football trivia buffs, some summary statistics are provided below each map. Top performing team for each position is based on average fantasy football points and top players on total points.

The complete dataset with other player attributes: college/university attended, number of years pro, division and conference, average fantasy football points for the regular season and number of games played can be found at www.geocommons.com.

Wide Receivers

Top Performers: Randy Moss (New England Patriots); Braylon Edwards (Cleveland Browns); Terrell Owens (Dallas Cowboys)

Top Performing Team: New England Patriots

Most Common Hometown State: California

Average Years Pro: 4.9

Running Backs

Top Performers: LaDainian Thomlinson (San Diego Chargers); Brian Westbrook (Philadelphia Eagles); Adrian Peterson (Minnesota Vikings)

Top Performing Team: Pittsburgh Steelers

Most Common Hometown State: Florida

Average Years Pro: 4.33

Tight Ends

Top Performers: Jason Whitten (Dallas Cowboys); Antononio Gates (San Diego Chargers); Tony Gonzalez (Kansas City Chiefs)

Top Performing Team: Dallas Cowboys

Most Common Hometown State: California

Average Years Pro: 4.6

Quarterbacks

Top Performers: Tom Brady (New England Patriots); Tony Romo (Dallas Cowboys); Drew Brees (New Orleans Saints)

Top Performing Team: New England Patriots

Most Common Hometown State: California

Average Years Pro: 6.03

Tony Romo and Jessica Simpson

Tony Romo was clearly a superstar this season but a look at his weekly fantasy football points shows a precipitous drop towards the end of the 17 weeks. Have anything to do with Jessica Simpson? Weigh in below.