GeoIQ Blog » collaboration

Dynamically Map your Google Spreadsheets with GeoCommons

bill greer — Tue, 29 Nov 2011 19:00:02 +0000

There is a good chance that you’ve been using Google Docs. Its a great way to share spreadsheets, documents, and information throughout your organization, or with friends and family. Ever wanted to plot you data on map, create thematics or animate it over time? This tutorial will focus on how to build Maps in GeoCommons that link with Google Spreadsheets, for even more data goodness. We will build up a spreadsheet and create a map using the spreadsheet data in GeoCommons, then we will be able to update the map from Google Spreadsheets. The best part is we can do this without writing any code, anyone can do this! The first step to mapping your Google Spreadsheet data in GeoCommons is to build up a spreadsheet. For this example we’ll be using this list of US ski resorts with latitude and longitude information, as well as a few other variables.

After you’ve got your spreadsheet ready, Click on the ‘File’ tab, then go down to click the ‘Publish to the Web…’ button.

Next you’ll have to click the ‘Start Publishing’ Button. I would also recommend that you click the box that says ‘Automatically re-publish when changes are made’, this will allow you to update your maps on GeoCommons directly from Google Spreadsheets. After you’ve clicked these buttons, you will be allowed to get a link to the published data. Here you should change from the default ‘Web-page’ to ‘CSV’. This will give you a link to your published data below.Copy the Link. It should look something like this:

Now you can head over to GeoCommons.com and after you’ve signed in, click the button the “Add a URL link” in the upload section. Paste the link from Google spreadsheets where it tells you to enter the URL, then specify CSV in the format section.

You will need to geocode your dataset, this should be pretty self explainatory if you’ve ever used GeoCommons, so I wont go into the details, but just know that once you’ve geocoded this URL once, we will continue using the attributes you specified to geocode any additional data you add into your spreadsheet. Your final dataset will look something like this:

Because the raw data is linked to a live website, you’ll see the ‘Fetch Latest” button above the title. Clicking on this button will update the GeoCommons dataset based on any changes that may have happened to your spreadsheet, and by updating the dataset, you’ll also be updating all of the maps using this as a layer. This means that when you’re working with a dataset that is constantly changing and being updated you’ll be able to update your maps by just clicking one button.

I’ve added a short video to show you the entire process from start to finish in less than 2 minutes.

SxSW and Interoperable Location Data

admin — Fri, 18 Mar 2011 18:50:46 +0000

This week I was at SxSW Interactive, for those that aren’t familiar it is a 5 day conference about new technology. I was there to be on a panel about “Interoperable Location Data” with Josh Babetski from Mapquest, Adam DuVander of Programmable Web, Scott Raymond from Gowalla and Tyler Bell of Factual. The premise of the panel was that many organizations are currently creating point of interest databases, but how do you combine them or allow them to interact?

What was interesting was the day before Foursquare made an announcement that fit directly into this premise. They began partnering with other organizations to link information between checkins and other services. You can see this in action on the the Thrillist section of Foursquare. If you notice there is a “More Info” link next to each of the displayed venues. The permise of this is you can then go read the Thrillist review of the venue and easily add it to your Foursquare to-do list. I think the idea of “pre-exploring” an area is interesting, meaning when I normally use Foursquare I tend to only use it when I am out and about. Sometimes I look to see where my friends are, other times I look for popular places to go. Combining the Foursquare information with other things allows me to explore from my desktop and plan to do things later. Other partners for this launch were The New York Times, New York Magazine and MenuPages. For the full Foursquare blog post look here.

So Foursquare has starting doing these integrations how can everyone do it? Services are emerging to crawl and collecting this data such as Factual and others are attempting to solve the problem on their own. The most idealist suggestion was to create a foundation to hold specific location information everyone would link to. This was suggested as a possibility within our SxSW panel, but I think everyone agrees it would likely be impractical. I’m excited to see how GeoIQ analysis modules can further aggregate data from a variety of sources to help others create these links and potentially serve as an outside link as well. This can either happen through our APIs or straight downloads as well. I’m excited because I think it means more cross data analysis similar to Sean’s recent analysis into San Francisco start-ups and his other analysis on DC Restaurant Inspections and Yelp Reviews.

As combining data continues to become easier I can wait to see what other insight is possible!

FoodSheds and Community Analytics

Andrew Turner — Mon, 07 Mar 2011 13:30:13 +0000

A few weeks ago, during US Presidents Day, Big Window Labs and Code For America held a Data Camp hackathon. Over the past two years there has been an increasing, and interesting, interaction at the convergence of technologists and government policy. Driven by, and rewarded with, open data and communications, developers and users are able to quickly create intriguing discoveries and useful applications.

One particular application that was created in that one day of development was a DC FoodShed site where residents of Washington, DC, as well as local government and organizations, can investigate the availability of grocery stores and distances.

A Food Desert is a region with limited or no access to healthy foods. Some DC neighborhoods have limited shopping options for residents to buy fresh groceries and produce. This map illustrates the disproportionate availability of grocery stores across the city.

The team used GeoCommons to create a map and our new analysis tools such as buffer, which calculates the distance from locations. In this case, showing the food options in DC show the quarter-, half-, and full-mile areas that are covered by these stores. The result is a simple visualization to indicate the easy availability of healthy buying options and “food deserts” where reduced access may result in poor food purchasing options.

This morning, a discussion emerged on twitter between Alex Howard and Clay Johnson on the utility of such an application. @digiphile questioned about the problem it may solve and who may use it and@cjoh pondered if there was an inverse correlation with liquor stores.

The simple answer is, thanks to open data, and collaborative tools such as GeoCommons, anyone can use this and ask their own questions. The source of all the data and the map itself is available with a simple search. In order for Clay to answer his question he can make his own map with a click and add DC Liquor stores. Perhaps he could even add Farmer’s Markets that may offer alternatives to grocery stores.

The capability here is to ask pose a question or hypothesis and immediately share that with a community. That community can then respond to provide their own perspective, ask additional questions, or suggest solutions. Data for data’s sake doesn’t solve any problems – but in access the data and tools, we can effectively collaborate in specific and demonstrable ways that help lead us all to better understanding and cooperation.

Our Trip to Redlands GIS Week

admin — Mon, 14 Feb 2011 17:47:12 +0000

Last week Andrew and I went to Redlands GIS Week. Hosted by Esri it was a conference of students, academics and professionals. Each year there is a different topic and the one covered this week was Volunteered Geographic Information (VGI): Real-Time and Emergency Applications. Essentially this is how can crowd-sourced information be utilized, created, enabled for crisis response, especially with a focus on real-time data.

A combination of talks and break-out sessions the event was interesting. The mix of students, academics and professionals meant there were different views on the suitability of crowd-sourced information. Discussion within my break-out group ranged from how can we verify the crowd to trust the information to how can we incentivize people to provide more structured information. There was also the typical concern of how can responders know if they can trust information. This had already been brought up in Michael Goodchild‘s talk “It’s About Time: The Temporal Dimension in VGI,” the idea being having some unverified data is better than having no data. I think wider acceptance of crowd-sourced information is just a matter of better analysis tools to determine what data is better and encouraging the crowd to submit data so there is more information available.

I gave a talk about the new collaborative analytics tools we’ve been adding to GeoCommons. Specifically my talk was “Enabling Collaborative Analytics for Faster Answers in a Crisis,” the idea is that the next step in a crisis is enabling the crowd to perform analysis. Traditionally analysts create reports which then go to decision makers. If changes need to be made to the end report tasks go back to the analyst.

The next step in analysis is to enable everyone to perform analysis. There are key things that need to happen in a system for this to be effective though. The first is to make analytics easy, this allows the user to make good decisions when they perform their analysis. Within GeoCommons we have aimed to do this and you see results of it through the application, such as when making a thematic map. When deciding on classifications of data we allow the user to match the type of data by matching their histogram versus the available classification schemes.

By making decisions as easy as matching pictures but allowing the user to go into the details it means that both experts and beginners can perform useful analysis. The next steps in encouraging collaborative analytics is making analysis traceable and making results extend-able. That changes the flow of analysis from an analysis making a report to allow analysis to be branched from and putting analysis tools in the hands of decision makers. This eliminates the bottle neck of only have a small group of individuals that can perform analysis and allows for faster response.

See my full presentation below:

Enabling Collaborative Analytics for Faster Answers in Crisis

View more presentations from Kate Chapman.

As analysis becomes more and more accessible the ability of the crowd to perform analysis quickly will continue to grow. In turn this will lessen response times and potentially save lives.

Off to Camp Roberts this Week!

admin — Wed, 04 Aug 2010 02:20:09 +0000

This week I’m off to Camp Roberts again for this quarter’s RELIEF exercise. FortiusOne has been participating for the past year or so in RELIEF, which is a crisis response integration exercise. Essentially different groups get together to hack on things and make them work together.

The first of these exercises that FortiusOne participated in was last August, but only remotely. That was the start of our involvement running an offline geo-stack for humanitarian relief. Since then Todd Huffman has taken the prototype to Afghanistan and the tools worked on at that exercise have continued to be developed. Of course these events have been minor compared to the effect of the earthquake in Haiti. The importance of collaboration in crisis response has been long stated as a necessity, but Haiti really showed how powerful it could be.

One really minor feature I’m really excited about t is no longer using subdomains in our Mac Mini deployments. This means that if the offline instance URL is http://geoiq.local for example there is not extra configuration to access Finder and Maker. Previously the URLs looked like http://finder.geoiq.local and http://maker.geoiq.local, in the absence of DNS this necessitated adding 3 hostfile entries. Now the URLs look like http://geoiq.local/finder and http://geoiq.local/maker for example. This means just an IP address can be used or a single hostfile entry is required.

One addition to RELIEF that I want to work with is Stewart Long and his balloon photography. Two weeks ago at WhereCampSoCal he made this photo using an inexpensive camera and a weather balloon, then careful stitching. After processing imagery this week I plan to tile the imagery and put it into the offline GeoIQ instance I built and derive vector data within an OpenStreetMap instance set-up for the exercise. Working through the whole process is important to work out integrations before there is a crisis. Look for more updates throughout the week, I promise to not get so consumed by the hacking and collaboration that I forget to post.

What Could the Obama Administration Mean for the GeoWeb?

Sean Gorman — Mon, 24 Nov 2008 21:52:28 +0000

With the elections over I’ve had a little time to think about what the new administration could mean for the GeoWeb. For those who follow the GeoWanking list serv there has been a raging debate on neogeography versus paleogeography. Some of the rhetoric reminds me of the just finished election and how we strive to create a binary world – blue state/red state or neo/paleo. In the spirit of moving beyond stereotypes and on to solving problems; I thought a closer look at what the potential impact of Obama’s technology platform on the GeoWeb could be. Might be a good diversion from our own self reflection – despite the fact I’ve added plenty of fuel to that fire

You can read Obama’s technology platform overview here. The plank that really grabbed my attention was the promise to “Open Up Government to its Citizens”. The idea that data about government (Congressional voting records) and created by the government (census data) should be easily available to the public. Specifically:

“Making government data available online in universally accessible formats to allow citizens to make use of that data to comment, derive value, and take action in their own communities. Greater access to environmental data, for example, will help citizens learn about pollution in their communities, provide information about local conditions back to government and empower people to protect themselves.”

The beauty is that we (the collective GeoWeb) have so many of these tools already built. The ability to deliver the data once it is made easily available has great promise. For instance here is EPA data on power plant emissions from GeoCommons:

From the map above you can see which power plants are producing the most poisonous CO2 emissions (click the down carrot on the layers box for the filter) or zoom into your specific neighborhood to see the plant and the type of environment around it. (Still refining the embed capability, but an example of how data can be virally spread).

The report goes on to recommend that the federal government should:

“Establishing pilot programs to open up government decision-making and involve the public in the work of agencies, not simply by soliciting opinions, but by tapping into the vast and distributed expertise of the American citizenry to help government make more informed decisions.”

This strikes again at the heart of the GeoWeb – enabling collaboration of experts and citizens across the country. Several projects and companies have pioneered dynamic collaboration around maps. Below is a Google MyMap with feedback around the GeoCommons power plant data in Florida”

View Larger Map

The blue push pins are the user generated feedback linking to expert opinion and photos from the field. This is just the tip of the iceberg of what is possible with collaboration around maps. These approaches can also be leveraged inside of government agencies, which is another plank in the Obama technology platform:

“Employing technologies, including blogs, wikis and social networking tools, to modernize internal, cross-agency, and public communication and information sharing to improve government decision making.”

We’ve seen a lot of this type of work going on in the intelligence community with Intelink, Intellipedia, and A-Space. There is also data fusion and sharing concepts, like the EPA’s Central Data Exchange. I’d love to hear other projects that fit in with the three planks, and more importantly existing or planned GeoWeb technologies that could help enable the new vision. I’ve really only highlighted two and I know there are tons more out there.

James Fee Gives His Two Cents on GIS and GeoWeb

Sean Gorman — Wed, 04 Jun 2008 13:15:57 +0000

We had the opportunity to catch up with James Fee at Spatially Adjusted and Planet Geospatial to get his opinion on the current and future state of GIS and the GeoWeb. James is a certified GIS Professional (GISP) and GIS developer, analyst, and consultant and has spent the last decade implementing, developing and consulting on GIS projects. He has experience with almost all of the large players in the geospatial field such as ESRI, MapInfo, Manifold, OSGeo (MapServer, GDAL, QGIS, OpenLayers), MapDotNet, Oracle, Microsoft and Google.

FortiusOne: Where do you see GIS going in the next 10 years?

James Fee: I think data and collaboration will be huge in the next 10 years. The explosion of Neogeography and projects such as OpenStreetMap have brought many new faces into GIS. Not only are we seeing spatial data being pushed out to the public at large, but this data is beginning to be integrated into GIS workflows. We’ll begin to see metadata and documentation of these datasets as well making them very valuable.

While freely sharing data has been great, the next logical step is allowing companies to monetize their datasets, share as easily. The ability to pay and use data services should revolutionize the industry. The price of data hasn’t really been a limiting factor yet, but the difficulty of integrating these datasets into online mapping or even in desktop GIS has hurt adoption. In addition the speed of geospatial data services has been poor so moving these services into the Cloud should improve performance and increase profitability given that there is no need for large overhead (such as servers, or bandwidth).

FortiusOne: Will there be convergence between GIS and the GeoWeb to the point that they become indistinguishable?

James Fee: Possibly, I think this have been the holy grail that everyone has been trying to attain. GIS by its nature is complex and you generally need complex solutions to complex problems. That said I think we’ll see many operations that were the domain of GIS begin to be part of the GeoWeb. Basic geoprocessing over the web via an easy-to-use interface can satisfy a vast number of use cases of general users without hitting them over the head with a steep learning curve.

Usually moving GIS to general uses has been by giving them the kitchen sink and expecting them to figure it out. Simple solutions to their “simple” problems is how we’ll see GIS and the GeoWeb converge. Over time more and more “complex” analysis will be available to use by just about anyone with a computer, but I’m not sure we’ll see that in the next 10 years.

FortiusOne: Do datasharing and crowdsourcing have a place in GIS?

James Fee: Yes, but the problem is how do you give GIS professionals the ability to use the data and make decisions about its accuracy. I guess it brings up the question, do you trust a Biologist in the field with at GPS more than a hobbyist? I’d guess most GIS professionals would pick the Biologist, but a degree in Biology doesn’t mean the data is necessarily good.

Datasharing and crowdsourcing are great ideas but for GIS to use them, they need metadata, documentation, and possibly a rating system. A “marketplace” should allow users to rate the quality and accuracy of the data which both helps others make decisions about the data and gives feedback to the creator on how they can improve their dataset. OpenStreetMap has been a great example on how “experts” can help “novices” grow to be experts in data collection.

FortiusOne: Should there be a marketplace for online geodata?

James Fee: Totally, I think there has to be. First off, you need some place users can feel comfortable buying data. Second you need a place where data can be rated and reviewed. Third you need a place where data providers can put their information in the cloud for quick and easy access buy everyone. If someone is investing time and energy into creating their data, I don’t see any reason they can’t be rewarded for this.

I think some data will be available via micropayments and other data will be very expensive (or the ability to pay for read only data vs editable data). Having some place where users can go to both sell and buy data, search for data, and review data is critical today. Sure Google will index spatial data, but being able to go to a focused marketplace will put buyers and sellers together quickly. And at least today, any site that sells data should be compatible with ESRI software. Offering up data types that aren’t compatible with ESRI will limit any marketplace.

FortiusOne: What emerging technology trend will have the biggest impact on GIS?

James Fee: I think putting a GPS in so many “ordinary” things is going to impact GIS immensely. Walking around with a GPS in your phone should give you access to many GIS applications, digital cameras and video cameras with GPS will spatially enable tons of datasets.

FortiusOne: What is your reaction to the Google – ESRI announcement?

James Fee: We’ll have to see what impact this really has. It isn’t revolutionary the idea that Google might index GIS servers, the hard part is getting all these traditional ESRI clients to open up their data is the challenge. They’ll need to see the benefit to allowing users to view their data in any way they choose rather than the traditional hard to use ESRI web mapping front end.

FortiusOne: What impact will Google have on GIS?

James Fee: Google has already had a huge impact on GIS. At a minimal level, it has already allowed GIS users to search for data sets. Google Maps has totally changed how web mapping is used and displayed on the internet, Google Earth has pushed 3D GIS to the mainstream and now their geo search 2.0 and geo sitemaps they pushing spatial searching. Google has been really good about getting spatial data in front of everyone in a way everyone can use it. GIS has learned much from this and the new tools coming out by ESRI, Autodesk, etc. all are very “Google-like”.