GeoIQ Blog

TechCamp

Andrew Turner — Mon, 30 Apr 2012 09:01:51 +0000

This week I am leading tech training at the TechCamp events in Tel Aviv and Ramallah discussing open mapping platforms for sharing and visualizing data. TechCamp is part of Secretary of State Hillary Clinton’s Civil Society 2.0 initiative that aims to empower local citizens and organizations to gain access to technology and information in order to actively engage with their government and communities. Technology can produce change in our society and in the hands of ordinary people are very powerful. This is something we believe in very passionately at GeoIQ and drives our design and innovation to provide cutting-edge, open mapping tools that are easy to use by everyone.

There have been TechCamps all around the world with great success in sharing experiences with the use of low-cost and easy to use technology by NGOs. At TechCamp Tel Aviv I will be walking through just a few of the examples of organizations that are using GeoIQ and GeoCommons to investigate data and tell compelling stories with the world. Groups such as the World Bank’s Mapping for Results and IADB’s MapAmericas, and even local groups such as the Bicycle Coalition of Philadelphia.

We will also be working over two days directly with NGOs on specific problems and objectives they have with making maps. I’d love to know if you have any open data for Israel or Palestine that we can also highlight to the local groups. We will make sure to post the videos and notes from the TechCamp when they’re online.

Visualizing our Changing Climate with Climascope

Andrew Turner — Fri, 27 Apr 2012 15:12:21 +0000

Last year we worked closely with the US World Wildlife Fund to develop the capability to upload and share climate modeling data through GeoIQ. Today you can explore the changing world at Climascope. Partnerships with the Tyndall Centre for Climate Change Research, University of East Anglia, and Zekiah collaborated to build the user interface to explore through the complex models and data. Users can explore temperature, precipitation, cloud cover, wetness days and more extremely valuable data through an easy map interface. You can even filter the data by ranges and click to dive into the data at individual locations and times.

The intended results are to give planners and practioners access to the data they need to prepare for climate change (adaptation in particular) and to provide the range of information, from the [different climate] scenarios to show policymakers spatially explicit climate implications for different emission scenario decisions.

ClimaScope provides spatially explicit climate change data for 18 climate model patterns, for the new IPCC RCP scenarios, the old SRES scenarios, and specific adaptation scenarios for 2C, 3C and 4C warming. There are 8 climate varibales – maximum, minimum and average temperatue, sea surface temperature, precipitation, wet day frequency, cloud cover and vapor pressure. All for annual, seasonal and monthly time periods for a range of 30-year periods. This is done by turning the complex climate data into spatially explicit georeferenced maps. These files can then easily be layered to allow users to look, for example, at all emission scenarios for a given climate model or all climate models for a given emission scenario for a given point.

By working to develop this with GeoIQ (grant funded) the overall goal was to easily be able to federate this data with other data. As World Bank’s Mapping for Results uses the same platform and open standards, then this federation is easily accomplished such that all World Bank Projects currently in Mapping for Results can be overlaid on ClimaScope data (see examples in the Featured Maps section of Climascope). Thus, users could click on the World Bank data and get information on that project, and click on arrow keys to see how the climate was projected to change at that site. Similarly propposed solar projects could be overlaid on the cloud frequency map to see if projected changes in cloudiness would help or hinder the project, precipitation and wet day frequency could be used to aid in assessing any water project.

Climascope has been submitted to the World Bank’s Apps for Climate and you can read more and vote for Climascope.

World Bank Annual Meetings

Andrew Turner — Mon, 23 Apr 2012 13:35:51 +0000

This weekend the World Bank hosted their Spring Meetings. As part of the event, the World Bank Institute extended the Mapping for Results to move beyond the in-depth maps and data to tell stories around development objectives and measured outputs. You can see the new homepage and stories by visiting maps.worldbank.org.

In addition, the World Bank is actively working to distribute the open data and maps across their numerous and growing open data portals. You can now find country project maps on each countries page of the World Bank Data catalog. These provide quick and up to date views of where the World Bank is actively working in countries with the abiity to drill down by region, sector and compare with numerous indicators to measure potential impact.

Beyond the 144 countries now hosted and published through the World Bank’s maps, there will be more individual projects and more local stories and perspectives beginning to be published. You can subscribe to updates at Mapping for Results or explore your own ability to combine the World Bank data with other indicators on our free GeoCommons community site.

Just in Time Analytics – Kanban for Big Data

Sean Gorman — Thu, 05 Apr 2012 16:58:23 +0000

Where is wrapping up this year, and thought I’d post up our presentation on streaming analysis. Since the slides themselves are mostly images I figure a little narrative might be helpful.

Just in Time Analytics – Where Conference

View more presentations from seagor.

The presentation was on the concept of how analysis can evolve to better take advantage of real time data streams. The community currently does lots of fascinating analysis of real time data from Twitter, mobiles devices, sensors etc., but it is inevitably a post mortem. By that I mean we do the analysis well after the event itself is over. If we think of a data stream as a living organism that is constantly changing we focus our analysis on the history that has already past.

Partially this is due to technology. Until recently most of what we use to munge large data streams, like Hadoop, run as batch operation. Lots of great presentations at Where went something like: I did a collect from Twitter and pulled 5 million Tweets over 2 days, pushed that into EC2 ran some map/reduce queries, ran analytics and produce a really nice visualization. Lots of fascinating results from these approaches, but the downside, it all comes well after the event itself happens. While this is useful for lots of purposes it misses the opportunity inherent in the data persistently updating. As the data changes so do the answers to our queries and equations. So, the premise of the presentation was how can we make our analysis as dynamic as our data.

The analogy at the beginning of the presentation was to “just in time” manfucaturing, originated by Toyota in the 1950′s as the far cooler sounding term – kanban. In short a kanban approach treats manufacturing like a super market, as you see demand from consumers depleting the store shelves you restock (just in time). Previously you’d do historical analysis to try and predict demand, and then schedule your inventories accordingly. Demand can be hard to predict in many industries and kanban ended up being a great innovation – just look at Toyota’s global growth after the 1950′s.

On the Web predicting global demand can be quite tricky as well. Yet a good chunk of the big data analysis we do currently takes this approach. Get a big hunk o’ data look for patterns and try to forecast that forward to gain insight. Could it be possible to use a kanban type approach, where we look at real time demand and update our analyses as the data streams to optimize response.

This is by no means a unique idea on our part. Lots of companies have been building real time analysis platforms like Backtype (now Twitter Storm), EsperTech, StreamBase, Yahoo S4, Hstreaming etc. Currently, I’ve not seen anyone looking at it from a geographic perspective or using non-reductionist approaches to the data analysis. This looked like a good opportunity to try and add to the conversation. In our Where presentation we also talked a good bit about the bias and short comings of reductionist approaches to data science, and alternatives like entropy calculations, but we’ll save that for a follow on post so this does not get too bloated.

To test the concept we took data from a mobile app used during the NYC Marathon to track runners. Then we treated it like a live feed and as the data streamed out, we intersected each event from the device with a grid over NYC. As each event intersected a grid cell we updated the equation result to the map. Calculating a sum for each grid cell is just one of many possible equations you could run against the data. We also calculated spatial entropy, to try and detect certainty/deviations in the data, but we’ll save that for the follow on post.

While the visualization of the data in fun, it really doesn’t get to the true potential of having perpetually updating analysis results. The real potential we see is setting a threshold and having the analysis alert you when a new result has been achieved that you want to know about. Let’s say for the NYC marathon data above we took the aggregation count for each cell and looked at the density of people per square meter for a time period. If density gets to threshold X send a promotion to each mobile device, and if it gets to why Y send a text message to safety officials that there be a crowd control problem. The ability to make analysis immediately actionable opens up whole new opportunities that create value for users. Instead of having a post mortem on what happened during the event, so we can better prepared next year, we can immediately respond to make it better right now.

GeoIQ team at Where2.0, JSConf, FOSS4G-NA

Andrew Turner — Thu, 22 Mar 2012 12:44:16 +0000

As is typical of Springtime, the GeoIQ team will be out and about in the world presenting and meeting at a number of upcoming events.

The first, and personally the one I’m always most excited about, is WhereCamp in San Francisco. WhereCamp is the best open venue to meet, brainstorm and hack with other geo-geeks on new technology. We’ll be bringing a number of tools and demos we’ve been building at GeoIQ to share with everyone.

Immediately after is the inexorable Where2.0 conference. Sean is speaking about “Just in Time Analytics: Making Answers Keep up with Data” sharing his thoughts on next generation real-time analysis for new sources of data. And I am giving a deep dive through the GeoIQ technology stack. Through the magic of scheduling, we’re both at the same time so you’ll likely catch one of us, but not both. We’ll make sure to share the slides afterwards.

And through similar scheduling craziness, Chris will be at JSConf at the same time in Phoenix, Arizona, showing off some of the crazy large-scale Javascript processing and visualization we’re doing with Node, Polymaps and a slew of other tools to realize the vision behind Sean’s talk.

Lastly, the second week of April I’ll be speaking at the local FOSS4G-NA on our use of open-source tools for the scalable, collaborative analysis and visualization that enables the public community of GeoCommons to build and share maps easily and how it’s used to open up data and knowledge sharing at organizations like the World Bank, IADB, and the United Nations.

If you’re at any of the conferences, make sure and connect – we’d love to meet and talk with you.

IADB Launches MapAmericas

Andrew Turner — Mon, 19 Mar 2012 15:13:18 +0000

This weekend the Inter-American Development Bank is launching their new MapAmericas platform at their annual meetings in Montevideo. The new platform features interactive maps of the region’s countries that provide detailed information on active IDB projects, ranging from data showing where and how money is being invested, to multimedia stories describing the development impact on communities. You can read more about the project or dive in and start exploring.

We are excited the IADB built their MapAmericas platform with GeoIQ. By using GeoIQ, they were able to quicky and easily begin organizing and georeferencing their data from the very kick-off of the project. Creating maps in meetings to share ideas and prototype interfaces and explore data quickly demonstrated effective ways to share the stories and insights of the IADB’s work. Using our API they then created a custom experience for visitors to explore the projects, results, documents, videos and more information effectively.

Like the World Bank’s Mapping for Results, the IADB is opening up their invaluable project data and result output data. Using the dashboard you can dive into any project to see where it is building schools, improving irrigation, or boosting crops through technology and training. The platform is initially with 141 projects in five of the poorest countries in the region: Bolivia, Haiti, Guyana, Nicaragua, Honduras, and Urugay. The remaining 21 countries will be continually added.

The new platform enables the IADB to embed images, videos, news stories, statistics, project indicators, and other related documents, providing a fuller understanding of the progress being made on the hundreds of projects that receive IDB financing throughout the region. By making so much information so readily available to governments and to the general public, the Bank hopes to enrich dialogue with donors, beneficiaries, and civil society.

Stay tuned for more information on features we built into GeoIQ for IADB’s MapAmericas that will be available on GeoCommons. In the meantime, check out the video they made to highlight the project.

White House Maps and the Federal Cloud

Andrew Turner — Thu, 16 Feb 2012 13:41:48 +0000

The current administration of the US Government has been increasingly looking at ways to both reduce costs as well as leverage more appropriate technology in order to provide services. The “Federal Cloud” is an initiative to adopt more cost-effective solutions that don’t require huge infrastructure costs for the government to build and maintain.

Steven VanRoekel, the new US Federal Chief Information Officer, published an article on the White House blog discussing the US Governments efforts to optimize their data center overhead and move more services to hosted services.

“agencies are on track to save taxpayers billions of dollars by cutting spending on wasteful, underutilized hardware, software and operations as well as enhance our cybersecurity; shrink our energy and real estate footprints; and take advantage of transformational technologies like cloud computing to make government work better for our nation’s families.”

A cloud hosted GeoCommons map is used to geographically visualize the national extent of the data centers that are being consolidated or closed. An elegant move by the US Government to demonstrate moving to the cloud by using the cloud in the discussion itself. You can also view the source map to access the underlying data in a variety of open formats.

We’re proud that GeoIQ is used across several government agencies and NGO’s to make it easier than ever to share data and create informative analyses. If you have more examples of the government using GeoCommons or any other great cloud solutions, we’d love to hear about them.

UNEP/GRID-Arendal launches OCEANIDS

Andrew Turner — Wed, 15 Feb 2012 14:44:47 +0000

Last week the tremendous team at the United Nations Environment Programme GRID-Arendal launched their OCEANIDS tool. Focused on sharing and open access to rich scientific data, the team is using GeoIQ to engage with citizens, governments, and researchers around the world to collaborate on important issues in the marine environment.

“OCEANIDS [is] a new tool to find and examine public marine-related scientific datasets. A main aim in the development of OCEANIDS is to provide end users with an interdisciplinary and multi-thematic geospatial and metadata portal of public data and information – but with the non-GIS expert end user in mind.”

The OCEANIDS portal is a custom designed version of the same platform behind GeoCommons, providing a focused experience around ocean data. However, users can still leverage the almost 100,000 public datasets in GeoCommons through our open data federation. Searching in OCEANIDS, users can optionally choose to import and link data from GeoCommons to compare it with the UNEP/GRID-Arendal scientific data.

Through our work with UNEP/GRID-Arendal, we also extended GeoIQ to include some interesting new features. Particular to oceanographic is support for North and South Polar projections. Scientific data that analyzes ice shelves, shipping routes, or Antarctica can be appropriately visualized.

As part of the launch, UNEP/GRID-Arendal has made an open call for anyone to create maps with the public scientific datas. The first deadline for submission of maps/stories is the 5 March 2012.

We’re excited to be an integral part of not just opening up access to the incredibly valuable, credible scientific data, but also providing tools that anyone can use to work with the data, ask questions, and share their insights.

Using the Google Translate Function to Make Multilingual Maps in GeoCommons

Sean Gorman — Thu, 09 Feb 2012 23:31:42 +0000

As I’ve been tracking the ongoing crisis in Syria I often felt I was only getting half the story not being able to read Arabic. This was especially frustrating when I’d check out all the rich data being generated by SyriaTracker. Citizens in Syria are reporting on the violence through social media and text messages creating a unique perspective. Ushahidi via Crowdmap does brilliant work powering the platform, but my lack of language skills limited what I could understand.

Fortunately Crowdmap has a great data download button – one of our favorite features of any platform. So, I downloaded the data and pushed it up into Google Spreadsheets. Turns out Google Spreadsheets has this brilliant feature that allows you to insert a translation function to a data cell. You just plug in:

=GoogleTranslate(“text”, “source language”,”target language”)

For the language you use the two letter ISO code for the language, and the Goog supports over 40. Since I wanted to translate from Arabic to English I plugged in:

=GoogleTranslate(A1, “ar”, “en”)

Then I just cut and paste the equation to all the cells I wanted translated. You can find the results at this URL. Then I grabbed the URL for the CSV link from the Google Spreadheet and loaded it into GeoCommons. For specifics on linking Google Spreadsheets to GeoCommons check out the post here. The end result was a dynamically linked dataset on GeoCommons that you can grab in a half dozen format or create maps/analysis with.

Can’t finish off without a custom map:

View map on GeoCommons

Tech@State Panel – Realtime Awareness

Andrew Turner — Thu, 02 Feb 2012 20:46:45 +0000

Tomorrow is the next Tech@State focusing on Realtime Awareness. In the afternoon I’m charing a panel on “Conveying Real-time Information”. You can still sign up for the conference tomorrow as well as the Unconference on Saturday.

Real-Time Awareness will focus on how to use the vast amount of data generated by social media and other internet technologies to deliver real-time awareness. Specifically, we will be examining the range of challenges that exist when trying to create real-time awareness, especially in the context of breaking news or events, crisis response and citizen engagement. We’ll discuss questions like: Which data feeds are the most reliable? What is the best way to filter the data? How should the data be analyzed? What’s the best way to convey the information to the user? What do citizens expect when they use social media to communicate with governments?

In our panel, we’ll be discussing the various issues and possibilities to visualize and utilize real-time information. Social media feeds, traffic cameras, mobile devices, satellite sensor feeds, crowd-sourced data, and more are quickly becoming valuable sources of information but the capability to quickly and effectively harness them are an open question.

My fellow panelist are Nancy Choi from Development Gateway, Ahmed Shihab-Eldin from The Stream Al Jazeera, Chris Damsen from NetVibes, and Paul Shinkman from WTOP. It should be a fascinating conversation!