When you upload or link to data from GeoCommons the option to add tags shows up on the final edit page of the process.  For this example I decided to map Flickr tags.  This week in D.C. filming of the Transformers 3 movie is going on, so the tag I searched for was “transformers.”  To get the URL feed to add to GeoCommons I went to Flickr and clicked “Explorer” then “Popular Tags” and searched for “transformers”.  At the bottom of the search results page you can grab the link for either a GeoRSS Feed or KML feed of the results.

I then copied that link and went to GeoCommons and clicks “Add a URL Link.”   Next I pasted the link into the “Enter a URL:” text box.

After this I began to edit edit the metadata for my feed.  You’ll notice the second text box on the “Describe and Share your Data” page is “Organize with tags:” you can type your tags here.

You’ll notice that in my screenshot there are many tags below the textbox.  Tags you’ve recently used appear here to add those tags to your dataset just click on them and they will be automatically added to the textbox.  If you are uploading multiple datasets and want to tag them similarly this can be useful to avoid typos and misspellings.

Once I’ve appropriately tagged, filled out the metadata and saved my dataset then I’m ready to make a map.  I click “Make a Map” from the dataset and zoom to Washington D.C.  Next I label the map and hit save.  Once you hit save the “Details” window of the map pops up, this is where you enter a description and tags of your map.  I labeled my dataset “transformers and flickr” but I also add the tag “DC” to the map since I’ve zoomed in on the D.C. area.

Once I’ve saved my map both it and my dataset will show up when I search by tag in GeoCommons.  In order to search by tag you preface your search string with the word tags (for example tags:”transformers”).  By searching the tag “transformers” you can see that there is an additional dataset I tagged yesterday as well.

So to make your data more easily searchable,discoverable and organized, tag your data and maps today!

]]> http://blog.geoiq.com/2010/10/14/better-know-a-geocommons-feature-tags/feed/ 0 http://blog.geoiq.com/2008/09/18/andrews-talk-at-web-20-expo-trends-and-technology-in-where-20/ http://blog.geoiq.com/2008/09/18/andrews-talk-at-web-20-expo-trends-and-technology-in-where-20/#comments Thu, 18 Sep 2008 21:30:19 +0000 bill greer http://blog.fortiusone.com/?p=639 Our own Andrew Turner and Mikel Maron presented at Web 2.0 Expo today on trends and technology in Where 2.0.

Cutting Edge Where 2.0 Trends

To kick things off Brady Forrest of O’Reilly and Andrew just published a report on the GeoWeb space, it’s available now.

For those not familiar [...]]]> Our own Andrew Turner and Mikel Maron presented at Web 2.0 Expo today on trends and technology in Where 2.0.

Cutting Edge Where 2.0 Trends

To kick things off Brady Forrest of O’Reilly and Andrew just published a report on the GeoWeb space, it’s available now.

For those not familiar with the Geo-geek world they started off defining the Geoweb as an interlinked set of people and places around the world that is finally web aligned. These people and places are linked together through open standards that can be searched and indexed online. This gives us a huge trove of information and data from numerous sources.

A new driver of geodata is the trend towards mass adoption of location aware mobile devices. Users are bringing mobile devices into social environments, business, and even global settings. Socialight provides users collaboration capabilities leveraging online and mobile technologies to provide reviews and information on top restaurants and other locations. Large established companies such as TeleAtlas and NAVTEQ are actively expanding into mobile as well. Users can also gather data on their own, with massive success of OpenStreetMap being one prominent example.

The majority web 2.0 services are starting to add geography. Wikipedia, Flickr, even YouTube. This information can now be mined. Check out geocodr which create geotags based on Flickr photos.

One issue that’s coming up with all this open data is metadata and providing source authentication. Can the data be trusted? The ability to crowdsource information for a specific crowd or business is exciting, but having authoritatively knowing the source of where the information is coming from is even more important. Privacy is another pressing issue with open data. Flickr is offering geo-privacy to their photo uploads – this introduces the idea of “casual privacy”. Trusted locations also have a strong impact on how data is perceived by users.

Omnifocus has an iPhone tool that adds geolocation to your “to-do” list. It automatically geo-locates you, so you can discover the closest option to complete the next task on your list. So, if you’re out at CVS and need to go pick up something at a grocery store, Omnifocus can tell you where the closest store is to check that task off your list.

Mapvertising is another interesting concept. Coupons and other location based advertising have to be able to understand what users are searching for in context. For example, searching for a “romantic restaurant” should NOT result in an advertisement for Hooters.

FAIL:

Andrew also discussed advances in traditional GIS. ESRI is one big GIS solution provider that is opening their data via KML, javascript and flash.

Many new users are looking to do more complex analysis than just mapping push pins and would like to map more sophisticated open source and private data. For example, with Maker! you can take a look at how average rent in Manhattan can be thematically mapped to discover price trends.

Burning Man 2008 GeoHacks Technology (The future)

At this point Andrew passed the presentation over the Mikel to provide more of a geo-hacking perspective perspective on the GeoWeb. He got things into gear by discussing what he learned in the Black rock Desert of Nevada. Burning Man is a blank canvas on which to dream and create anything. It’s a single week long social experiment and a petri dish of urban development. It’s almost a laboratory that we can use to begin to examine all the geodata and tools we have. We can look at urban development, how social groups form in cities, etc.

Burning Man Earth had a lot of “geogeekery”. Over 100 GB of data were collected over the week.

Why is this important for Web 2.0 Expo? This is a prime example of Web 2.0 and Where 2.0 and what these technologies can enable. Open platforms, open standards, open data, and collaboration that is easy and cheap.

Amateur Remote Sensing

• pictearth.com
• diydrones.ning.com
• openaerialmap.org

We took remote sensing data every day. You don’t need expensive gear or a satellite. Small planes are used to get imagery with under $500. A plane is used with continuous shots and a camera hanging out the window with a clamp. The pilot has to manage the camera out the window, but there were no problems finding volunteers. “There are 3 F’s that no pilot will refuse….Food, Fuel, and something else…” Not only did we get aerial pics from planes, but also from kites.

Some great shots were taken as burning man convened. You can see how people created their spaces based on where the ones before them placed their tents.

We then processed the photo using ERMapper, ESRI, Photoshop, and some blood, sweat and tears.

The GeoDjango platform was used to collect even more visual data. If you have geographic items in your models, it can map to nicer views. A camp layout was placed in CAD for even more visualization, which we received in a PDF. To georectify, we used ESRI. We also had to use WMS and tiling (TileCache). These OpenLayers provided vectors from the PDF.

We then wanted to extend this to a social networking platform and get media artifacts from the information. We used “pinax” for networking. This data can be used for future city layout and camp planning tools.

Flickr took these tiles so people could geotag their photos from Burning Man based on the location of their maps.

Some of the other technologies that were used during burning man was Garmin radio for friend location, GPS tracking of vehicles, digipeater (which rebroadcasts to the Internet down the line for free).

Parting words:

“The Internet brings us together, but what if the single link holding us in place breaks?
What if what we learn in the harsh environment of the playa could be reapplied to those in crisis, instead of artistic indulgence?”

FortiusOne: Where do you see GIS going in the next 10 years?

James Fee: I think data and collaboration will be huge in the next 10 years.  The explosion of Neogeography and projects such as OpenStreetMap have brought many new faces into GIS.  Not only are we seeing spatial data being pushed out to the public at large, but this data is beginning to be integrated into GIS workflows.  We’ll begin to see metadata and documentation of these datasets as well making them very valuable.  

While freely sharing data has been great, the next logical step is allowing companies to monetize their datasets, share as easily.  The ability to pay and use data services should revolutionize the industry.  The price of data hasn’t really been a limiting factor yet, but the difficulty of integrating these datasets into online mapping or even in desktop GIS has hurt adoption.  In addition the speed of geospatial data services has been poor so moving these services into the Cloud should improve performance and increase profitability given that there is no need for large overhead (such as servers, or bandwidth).

FortiusOne: Will there be convergence between GIS and the GeoWeb to the point that they become indistinguishable?

James Fee: Possibly, I think this have been the holy grail that everyone has been trying to attain.  GIS by its nature is complex and you generally need complex solutions to complex problems.  That said I think we’ll see many operations that were the domain of GIS begin to be part of the GeoWeb.  Basic geoprocessing over the web via an easy-to-use interface can satisfy a vast number of use cases of general users without hitting them over the head with a steep learning curve.  

Usually moving GIS to general uses has been by giving them the kitchen sink and expecting them to figure it out.  Simple solutions to their “simple” problems is how we’ll see GIS and the GeoWeb converge.  Over time more and more “complex” analysis will be available to use by just about anyone with a computer, but I’m not sure we’ll see that in the next 10 years.

FortiusOne: Do datasharing and crowdsourcing have a place in GIS?

James Fee: Yes, but the problem is how do you give GIS professionals the ability to use the data and make decisions about its accuracy.  I guess it brings up the question, do you trust a Biologist in the field with at GPS more than a hobbyist?  I’d guess most GIS professionals would pick the Biologist, but a degree in Biology doesn’t mean the data is necessarily good.  

Datasharing and crowdsourcing are great ideas but for GIS to use them, they need metadata, documentation, and possibly a rating system.  A “marketplace” should allow users to rate the quality and accuracy of the data which both helps others make decisions about the data and gives feedback to the creator on how they can improve their dataset.  OpenStreetMap has been a great example on how “experts” can help “novices” grow to be experts in data collection.

FortiusOne: Should there be a marketplace for online geodata?  

James Fee: Totally, I think there has to be.  First off, you need some place users can feel comfortable buying data.  Second you need a place where data can be rated and reviewed.  Third you need a place where data providers can put their information in the cloud for quick and easy access buy everyone.  If someone is investing time and energy into creating their data, I don’t see any reason they can’t be rewarded for this.  

I think some data will be available via micropayments and other data will be very expensive (or the ability to pay for read only data vs editable data).  Having some place where users can go to both sell and buy data, search for data, and review data is critical today.  Sure Google will index spatial data, but being able to go to a focused marketplace will put buyers and sellers together quickly.  And at least today, any site that sells data should be compatible with ESRI software.  Offering up data types that aren’t compatible with ESRI will limit any marketplace.

FortiusOne: What emerging technology trend will have the biggest impact on GIS?

James Fee: I think putting a GPS in so many “ordinary” things is going to impact GIS immensely.  Walking around with a GPS in your phone should give you access to many GIS applications, digital cameras and video cameras with GPS will spatially enable tons of datasets.  

FortiusOne: What is your reaction to the Google – ESRI announcement?

James Fee: We’ll have to see what impact this really has.   It isn’t revolutionary the idea that Google might index GIS servers, the hard part is getting all these traditional ESRI clients to open up their data is the challenge.  They’ll need to see the benefit to allowing users to view their data in any way they choose rather than the traditional hard to use ESRI web mapping front end.  

FortiusOne: What impact will Google have on GIS?

James Fee: Google has already had a huge impact on GIS.  At a minimal level, it has already allowed GIS users to search for data sets.  Google Maps has totally changed how web mapping is used and displayed on the internet, Google Earth has pushed 3D GIS to the mainstream and now their geo search 2.0 and geo sitemaps they pushing spatial searching.  Google has been really good about getting spatial data in front of everyone in a way everyone can use it.  GIS has learned much from this and the new tools coming out by ESRI, Autodesk, etc. all are very “Google-like”. 

The first screen shot is of the data details page, which contains the metadata information for the data set. In this case 2000 US Census data at the tract level for Alabama:

Here you can see the major elements we are capturing in a user friendly graphical lay out. One of the cool new bits is the system automatically calculates statistics when you upload the data. Being able to data mine and run statistics on the fly is one of the new developments we are particularly excited about.

All the metadata on the data details page is exposed as Dublin Core elements which should make them machine readable to the rest of the world:

Also there are links to FGDC and ISO 19115 metadata mappings which take you to simple text pages with the indicated information. We probably need another pass to get these completely correct, but the infrastructure is all in place to do so.

FGDC looks like this:

ISO 19115 looks like this:

Hopefully this will help make the data in GeoCommons useful to multiple geospatial work flows. We hope having the ability to get data out in shapefile, KML, and .CSV (spreadsheets) will create more cross fertilization between GeoWeb and GIS users. With some luck it can help get more geospatial data out to the public that has been difficult to access in the past. A couple of examples below.

US Census Tract Data for Alabama

Global Maritime Shipping Lanes

Zillow Neighborhoods and Shipping Lanes (just because it looked kinda cool)

Thanks again for the feedback from folks on the metadata and we’ll keep iterating on getting it spot on.

1) people’s tags may be difficult for others to understand,
2) people may have tagged items inappropriately for others’ needs.

In short your users will not always implement tags in ways that are productive for the community – in the extreme resulting in Flickr’s 20 million unique tags. How many of those 20 million tags are misspelled words or so off the path they never get found.

In addition to the problems you encounter with folksonomies in general you have the further complications of geopspatial data. All geospatial data sets have location tags, but adding them in an unstructured way creates enough chaos that it is very difficult to leverage location tags in a thorough way. Secondly many potential users do not know the variety of geodata available. Put more simply they do not know what to search for, and having the ability to browse through data by topics is appealing.

Despite the downsides of folksonomies they are incredibly powerful and have been hugely effective in organizing vast amount of data on the web. So, as we worked on the next iteration of GeoCommons we started looking at possible hybrid approaches to folksonomies and hierarchies.

Specifically we looked at the two problems specific to geospatial data listed above 1) place tags and 2) organizing data for browsing. Solving the problems required both short term and long term solutions.

Fortunately we had a small advantage over many crowd sourced project in that we have a full time data team. They are a great group of folks that spend their day finding cool geodata and coming up with clever ways to organize it.

Through the data team and the other community members that contributed data to the first iteration of GeoCommons we had a big pool of data with a wide variety of tags to examine. What we found were some distinct trends in the tagging and titling of data. Across the data there were a commons set of tags that broke the data up into a useful set of distinct categories, but there were also many data sets that were tagged with elements that made them often indiscoverable. After the analysis we started to look at structures we could establish to help create self similarity in tagging that still had the flexibility to be adaptive.

The result was the creation of a location and topical taxonomy based on our existing corpus of data that has the intelligence to adapt as the content grows and evolves. I can’t go into the technical details in depth, but fundamentally the concept is to intelligently leverage the taxonomies and structures to provide suggestions to users to tag their data better.

In many cases this can be very simple – like providing tips on how to tag and title effectively to make your data more valuable to the community. For instance with titles we found across GeoCommons there were four key pieces of information used for datasets in the past.

1) Source name, 2) Original Name of Dataset from Source (or short description of dataset) 3) Geographic Area, 4) Time period of data

Examples:

Communicating this effectively to users is a great way to get better consistency across data contributions, while still allowing flexibility for users to be creative and bring in information that does fit the rigid mold of a hierarchy. Of course this is the most simple and you can get far more clever.

Del.icio.us for instance has a great feature that notifies a user they are putting in a new tag no one has used before and asking if that is what they meant to do. You can also suggest tags from your taxonomy that are semantically related to the data the user is contributing. This creates a consistency across tags that makes data easier to find as the system scales to larger volumes.

The nice thing about taxonomies as opposed to folksonomies is that they can be structured as trees, which means you can compute across them quite easily. With a solid and adaptive taxonomy in place you can go a long ways in intelligently guiding users towards creating better and more consistent tags. At least that is what we think and it will be fun to see how it works out after the launch.

From that standpoint the proposal is for an implementation not a standard. We have just about 5,000 unique datasets and about 70,000 data layers, and it would be great to expose useful metadata for the data. The data covers the gambit, from EPA toxic release sites to the number of Facebook users by city. The system and metadata requirements needs to be flexible enough to accommodate both a user uploading Facebook data and one uploading EPA data.

While GIS users might not be intimidated by a metadata form with 75 or even 335 elements your average Web/GeoWeb user definitely will be. The goal with GeoCommons is to provide a destination where both communities can consume and share data, and I think both communities will find tools and data that are useful.

In regard to the metadata elements we proposed to map to in the last post, we were looking for those that both technical and non-technical users would understand, and also automatically trap as many additional elements as possible. To cover technical users, that have a full compliment of metadata, the plan is to have an element where you can you can provide a link to a full metadata specification.

The comments directing us to the ISO 19115 standard were very useful and we are looking to see what elements we are missing to map to that standard as we evolve. The thing we want to make sure we get right is finding to best set of metadata elements to request from users. Balancing the fact that if we have a huge number of elements, most people are going to go running for the hills.

Right now it looks like we’ll have 17-20 elements that will map to Dublin Core, FGDC, and in a next release ISO 19115. So, for each data set in Geocommons you’ll have a page that lists those 17-20 elements in the metadata format technical folks are used to seeing. This should also provide a means by which to explore federating the data with other applications and search approaches.

The goal here is to create a bridge between content being created for the GeoWeb and content created for the GIS world and make both usable and remixable by the web community as a whole. I fully respect the motivations and requirements for the GIS metadata specifications out there, and I hope we can leverage them to create an implementation that will see a high level of adoption.

Without adoption standards are pretty hollow as we’ve seen with all the work that went into GML versus the much lighter specifications for KML and GeoRSS. While both have their place it is clear what the market is supporting. As more geospatial data is created outside of the government we are not going to have the government mandate to force metadata creation and what the market accepts is going to become increasing critical – IMHO. Look forward to getting more feedback as we get ready to launch.

I’ll assume that some day ESRI will nail the technical side of Explorer, and look a little further down the road – to the content available for Google and ESRI’s thick client applications and how easy it is to access. As the GeoWeb evolves and we’ve rendered ever more amazing three dimensional worlds I think an increasing premium is going to be on the scope of data and content that can be delivered to these applications.

So where do the two ends of the GeoWeb spectrum sit on the topic? From my observations pretty far apart. For practical purposes I’ll break the comparison into four topics 1) data formats 2) data sources 3) data search and 4) data packaging

Data Formats

This is one category where ArcGIS Explorer ups Google Earth cleanly. Explorer allows you to load ArcGIS Explorer files (.nmf why they felt the need to create another proprietary file format that works with nothing else is beyond me), servers (WMS, IMS, file server), geodatabases, shapefile, raster, and KML. This is an impressive list, although a little bit less impressive when you consider that half are open standards KML, WMS and raster and the others are really ESRI proprietary formats at the end of the day. On the upside the raster support is quite extensive (30 or so different formats), although they require a spatial reference file.

On the Google side you have KML and KMZ (once proprietary now turned over to the OGC – jury is still out on how open it will be) for the free version of Google Earth, and within the free Google Earth 3-D warehouse support for SketchUp “.skp” and COLLADA “.dae”. In Google Earth Pro and Enterprise versions there is vector support for ESRI shapefile and MapInfo .tab, and for imagery support for TIFF (.tif), including GeoTiff, National Imagery Transmission Format (.ntf), Erdas Imagine Images (.img), Atlantis MFF Raster (.hdr), PCIDSK Database File (.pix), Portable Pixmap Format (.pnm), Device Independent Bitmap (.bmp). While this is an extensive list you have to pay to get the file support, in ArcGIS Explorer the support is currently free, so ESRI does have a distinct edge in the category.

Data Sources

In addition to supporting different file formats there is also baked in content for both applications. For Explorer you can access ArcGIS Online directly through the file server option, which allows you to access a file directory of cotnent. A second option is you can access the “Resource Center” website through the help tab, where you can download content in the .nmf file format. The “Resource Center” is definitely the more user friendly of the two with a nice user interface categorizing content into useful categories like, “imagery, street, physical features”.

On the downside the content is very limited – twenty four layers supplied by ESRI and four contributed by the community. ArcGIS Online has more content, but was a pain in the ass to access. You have to get a user name and password from ESRI, read through the incorrect direction to access it, then you get a list of UNIX style titles with abbreviations and underscores, like UNEP_WCMC_WDPA2006_2D. Not exactly user friendly, but you do get close to fifty additional layers of data once you jump through the hoops.

In Google Earth there are two sources of baked in data from the application, “Layers” including (terrain, geoweb, roads, traffic, 3-building, borders and labels, gallery, global awareness, and places of interest) and the ability to search for businesses. You can also click the “help” tab and be taken to the Google Earth Community web page. On the Google Earth Community page site alone there are 638,213 KML or KMZ files.

While finding this content (they are all file attachments to bulletin board posts) is pretty clunky and often frustrating, it is a LOT of content. Especially when you compare it to the four user generated files on the ESRI equivalent. The quality and source of this content/data varies wildly, and it is difficult to tell what is good and what is bad, but the potential is there. Actually the metadata support for both is pretty sparse. This is ESRI’s metadata for a .nmf imagery layer from the “Resource Center”:

“Displays satellite and aerial imagery at a 15m minimum resolution worldwide, and 1m resolution for the U.S. World boundaries, place names, and transportation layers are also included. Use this map to view man-made and natural features, or as a base map for overlaying associated data layers.”

For an ArcGIS Online layer this:

Layer Name: Imagery
Layer Source: ArcGIS Globe Service Layer
Layer Type: Draped
URL: http://services.arcgisonline.com/v92
Service: I3_Imagery_Prime_World
Sub Service: Imagery
LOD Tile Fetch: True
Hidden: False

Neither super useful.

Data Search

Neither application directly supports data search, but both have communities or services built around them that do. On the ESRI side there is a huge number of geospatial data repositories that have shapefiles in them and the search capabilities of them vary widely. Probably the largest is the Geospatial One-Stop that has a connection to most Federal geospatial data. Although in reality it is more often access to the metadata than the data itself. Still a large amount of content that could be conceivably viewed in ArcGIS Explorer, although largely disconnected in a large number of different repositories.

Google Earth has not only a good number of community aggregators like the official GE Community site above and unofficial like Mapufacture, GE Library and GLayers, but also the ability to search all KML files indexed on the web. I’ve heard numbers north of 10 million KML files indexed by this approach, but have nothing official. One way to search this content base is to type filetype:kml into the standard Google search box. You get a good amount of content in the results but figuring out what that content is and means is pretty sketchy. Here is the result for a search for “sharks”:

Till you download it and open it in Google Earth you really have no clue, and even then you still might have no clue.

So with GIS/ESRI data you get great metadata and context, but no unified search. On the Google front you get great unified search and community content but no metadata or context for the data.

Data Packaging

The last topic is short and sweet. ArcGIS Explorer gives you a blank globe with just one layer of base imagery (looks like blue marble), then it is up to you to populate the globe with data to fit your needs. Google Earth on the other hand comes packaged with a wide variety of layers already populated on the map. One is geared towards a professional audience and the other mass consumer. Although I would argue that if ESRI truly wants to create GIS for everyone they are going to need to package up content and GIS data, so anyone can hit the ground running.

Even as a GIS geek it took me way to long to get the whole rig going to create something useful. Having all the options to bring in a wide variety of content was great, but I think there is still a lot to be learned from Google about how to package up content to appeal to a much broader audience. End of the day I’d say ESRI wins the content variety category and Google wins the content volume and packaging category. Lot of good things being done by both from opposite directions, but I believe they inevitably run up against each other. How and when this happens will be interesting to watch.