GeoIQ Blog » geoiq

Just in Time Analytics – Kanban for Big Data

Sean Gorman — Thu, 05 Apr 2012 16:58:23 +0000

Where is wrapping up this year, and thought I’d post up our presentation on streaming analysis. Since the slides themselves are mostly images I figure a little narrative might be helpful.

Just in Time Analytics – Where Conference

View more presentations from seagor.

The presentation was on the concept of how analysis can evolve to better take advantage of real time data streams. The community currently does lots of fascinating analysis of real time data from Twitter, mobiles devices, sensors etc., but it is inevitably a post mortem. By that I mean we do the analysis well after the event itself is over. If we think of a data stream as a living organism that is constantly changing we focus our analysis on the history that has already past.

Partially this is due to technology. Until recently most of what we use to munge large data streams, like Hadoop, run as batch operation. Lots of great presentations at Where went something like: I did a collect from Twitter and pulled 5 million Tweets over 2 days, pushed that into EC2 ran some map/reduce queries, ran analytics and produce a really nice visualization. Lots of fascinating results from these approaches, but the downside, it all comes well after the event itself happens. While this is useful for lots of purposes it misses the opportunity inherent in the data persistently updating. As the data changes so do the answers to our queries and equations. So, the premise of the presentation was how can we make our analysis as dynamic as our data.

The analogy at the beginning of the presentation was to “just in time” manfucaturing, originated by Toyota in the 1950′s as the far cooler sounding term – kanban. In short a kanban approach treats manufacturing like a super market, as you see demand from consumers depleting the store shelves you restock (just in time). Previously you’d do historical analysis to try and predict demand, and then schedule your inventories accordingly. Demand can be hard to predict in many industries and kanban ended up being a great innovation – just look at Toyota’s global growth after the 1950′s.

On the Web predicting global demand can be quite tricky as well. Yet a good chunk of the big data analysis we do currently takes this approach. Get a big hunk o’ data look for patterns and try to forecast that forward to gain insight. Could it be possible to use a kanban type approach, where we look at real time demand and update our analyses as the data streams to optimize response.

This is by no means a unique idea on our part. Lots of companies have been building real time analysis platforms like Backtype (now Twitter Storm), EsperTech, StreamBase, Yahoo S4, Hstreaming etc. Currently, I’ve not seen anyone looking at it from a geographic perspective or using non-reductionist approaches to the data analysis. This looked like a good opportunity to try and add to the conversation. In our Where presentation we also talked a good bit about the bias and short comings of reductionist approaches to data science, and alternatives like entropy calculations, but we’ll save that for a follow on post so this does not get too bloated.

To test the concept we took data from a mobile app used during the NYC Marathon to track runners. Then we treated it like a live feed and as the data streamed out, we intersected each event from the device with a grid over NYC. As each event intersected a grid cell we updated the equation result to the map. Calculating a sum for each grid cell is just one of many possible equations you could run against the data. We also calculated spatial entropy, to try and detect certainty/deviations in the data, but we’ll save that for the follow on post.

While the visualization of the data in fun, it really doesn’t get to the true potential of having perpetually updating analysis results. The real potential we see is setting a threshold and having the analysis alert you when a new result has been achieved that you want to know about. Let’s say for the NYC marathon data above we took the aggregation count for each cell and looked at the density of people per square meter for a time period. If density gets to threshold X send a promotion to each mobile device, and if it gets to why Y send a text message to safety officials that there be a crowd control problem. The ability to make analysis immediately actionable opens up whole new opportunities that create value for users. Instead of having a post mortem on what happened during the event, so we can better prepared next year, we can immediately respond to make it better right now.

UNEP/GRID-Arendal launches OCEANIDS

Andrew Turner — Wed, 15 Feb 2012 14:44:47 +0000

Last week the tremendous team at the United Nations Environment Programme GRID-Arendal launched their OCEANIDS tool. Focused on sharing and open access to rich scientific data, the team is using GeoIQ to engage with citizens, governments, and researchers around the world to collaborate on important issues in the marine environment.

“OCEANIDS [is] a new tool to find and examine public marine-related scientific datasets. A main aim in the development of OCEANIDS is to provide end users with an interdisciplinary and multi-thematic geospatial and metadata portal of public data and information – but with the non-GIS expert end user in mind.”

The OCEANIDS portal is a custom designed version of the same platform behind GeoCommons, providing a focused experience around ocean data. However, users can still leverage the almost 100,000 public datasets in GeoCommons through our open data federation. Searching in OCEANIDS, users can optionally choose to import and link data from GeoCommons to compare it with the UNEP/GRID-Arendal scientific data.

Through our work with UNEP/GRID-Arendal, we also extended GeoIQ to include some interesting new features. Particular to oceanographic is support for North and South Polar projections. Scientific data that analyzes ice shelves, shipping routes, or Antarctica can be appropriately visualized.

As part of the launch, UNEP/GRID-Arendal has made an open call for anyone to create maps with the public scientific datas. The first deadline for submission of maps/stories is the 5 March 2012.

We’re excited to be an integral part of not just opening up access to the incredibly valuable, credible scientific data, but also providing tools that anyone can use to work with the data, ask questions, and share their insights.

Using the Google Translate Function to Make Multilingual Maps in GeoCommons

Sean Gorman — Thu, 09 Feb 2012 23:31:42 +0000

As I’ve been tracking the ongoing crisis in Syria I often felt I was only getting half the story not being able to read Arabic. This was especially frustrating when I’d check out all the rich data being generated by SyriaTracker. Citizens in Syria are reporting on the violence through social media and text messages creating a unique perspective. Ushahidi via Crowdmap does brilliant work powering the platform, but my lack of language skills limited what I could understand.

Fortunately Crowdmap has a great data download button – one of our favorite features of any platform. So, I downloaded the data and pushed it up into Google Spreadsheets. Turns out Google Spreadsheets has this brilliant feature that allows you to insert a translation function to a data cell. You just plug in:

=GoogleTranslate(“text”, “source language”,”target language”)

For the language you use the two letter ISO code for the language, and the Goog supports over 40. Since I wanted to translate from Arabic to English I plugged in:

=GoogleTranslate(A1, “ar”, “en”)

Then I just cut and paste the equation to all the cells I wanted translated. You can find the results at this URL. Then I grabbed the URL for the CSV link from the Google Spreadheet and loaded it into GeoCommons. For specifics on linking Google Spreadsheets to GeoCommons check out the post here. The end result was a dynamically linked dataset on GeoCommons that you can grab in a half dozen format or create maps/analysis with.

Can’t finish off without a custom map:

View map on GeoCommons

Tech@State Panel – Realtime Awareness

Andrew Turner — Thu, 02 Feb 2012 20:46:45 +0000

Tomorrow is the next Tech@State focusing on Realtime Awareness. In the afternoon I’m charing a panel on “Conveying Real-time Information”. You can still sign up for the conference tomorrow as well as the Unconference on Saturday.

Real-Time Awareness will focus on how to use the vast amount of data generated by social media and other internet technologies to deliver real-time awareness. Specifically, we will be examining the range of challenges that exist when trying to create real-time awareness, especially in the context of breaking news or events, crisis response and citizen engagement. We’ll discuss questions like: Which data feeds are the most reliable? What is the best way to filter the data? How should the data be analyzed? What’s the best way to convey the information to the user? What do citizens expect when they use social media to communicate with governments?

In our panel, we’ll be discussing the various issues and possibilities to visualize and utilize real-time information. Social media feeds, traffic cameras, mobile devices, satellite sensor feeds, crowd-sourced data, and more are quickly becoming valuable sources of information but the capability to quickly and effectively harness them are an open question.

My fellow panelist are Nancy Choi from Development Gateway, Ahmed Shihab-Eldin from The Stream Al Jazeera, Chris Damsen from NetVibes, and Paul Shinkman from WTOP. It should be a fascinating conversation!

ONR Social Media Conference

Sean Gorman — Tue, 17 Jan 2012 14:58:37 +0000

Right before Christmas I headed out to San Jose for a conference on social media hosted by the Office of Naval Research. It was an interesting cross section of government research on social media and comercial social media companies that provided either social media apps (Facebook, Google+, Yahoo) or analytics (Topsy, Radian6, SAS). There was some fascinating research being done and some good debates on privacy implications. It was encouraging to see the open dialogue happening – although I would not say that everyone was in agreement.

We gave a talk on how the “geo” attributes of social media streams can be leveraged to help analysis and provide better context. The general theme was how we use location as a common thread to tie together multiple disparate data sets. One of the toughest things with social media is putting it into a broader context. The presentation below covers some of the examples we gave of how this can be helpful in a variety of scenarios.

ONR Social Media Conference – GeoIQ

View more presentations from seagor

Lots of great presentations although the coolest demonstration of the day did not involve social media. Affectiva showed off a tool for detecting the heart rate and breathing for users looking at a Webcam. The heart rate was live updating on the screen as the users in Webcam responded to what was on their screen. Minority report is closer than we think…

Dynamically Map your Google Spreadsheets with GeoCommons

bill greer — Tue, 29 Nov 2011 19:00:02 +0000

There is a good chance that you’ve been using Google Docs. Its a great way to share spreadsheets, documents, and information throughout your organization, or with friends and family. Ever wanted to plot you data on map, create thematics or animate it over time? This tutorial will focus on how to build Maps in GeoCommons that link with Google Spreadsheets, for even more data goodness. We will build up a spreadsheet and create a map using the spreadsheet data in GeoCommons, then we will be able to update the map from Google Spreadsheets. The best part is we can do this without writing any code, anyone can do this! The first step to mapping your Google Spreadsheet data in GeoCommons is to build up a spreadsheet. For this example we’ll be using this list of US ski resorts with latitude and longitude information, as well as a few other variables.

After you’ve got your spreadsheet ready, Click on the ‘File’ tab, then go down to click the ‘Publish to the Web…’ button.

Next you’ll have to click the ‘Start Publishing’ Button. I would also recommend that you click the box that says ‘Automatically re-publish when changes are made’, this will allow you to update your maps on GeoCommons directly from Google Spreadsheets. After you’ve clicked these buttons, you will be allowed to get a link to the published data. Here you should change from the default ‘Web-page’ to ‘CSV’. This will give you a link to your published data below.Copy the Link. It should look something like this:

Now you can head over to GeoCommons.com and after you’ve signed in, click the button the “Add a URL link” in the upload section. Paste the link from Google spreadsheets where it tells you to enter the URL, then specify CSV in the format section.

You will need to geocode your dataset, this should be pretty self explainatory if you’ve ever used GeoCommons, so I wont go into the details, but just know that once you’ve geocoded this URL once, we will continue using the attributes you specified to geocode any additional data you add into your spreadsheet. Your final dataset will look something like this:

Because the raw data is linked to a live website, you’ll see the ‘Fetch Latest” button above the title. Clicking on this button will update the GeoCommons dataset based on any changes that may have happened to your spreadsheet, and by updating the dataset, you’ll also be updating all of the maps using this as a layer. This means that when you’re working with a dataset that is constantly changing and being updated you’ll be able to update your maps by just clicking one button.

I’ve added a short video to show you the entire process from start to finish in less than 2 minutes.

More Ways to Visualize Data: Charts

Brendan Heberton — Tue, 15 Nov 2011 10:25:22 +0000

Maps are awesome. Adding charts to a map is even more awesome. In addition to mapping data at Geocommons, users can now visualize the same data by utilizing our newly introduced charts. The backbone of these charts was created using g.Raphael, which is based on Raphael‘s JavaScript graphics library. RaphaelJS offers the unique ability to draw Scalable Vector Graphics (SVG) across browsers and platforms by using SVG to draw vectors and implementing VML logic for older IE browsers. Point, line, area, column and pie charts are all included now in Geocommons.

Creating a chart is easy! After making a map, users can now select the option to include a chart (or many), then build it in two quick steps. First, select a chart type, then feed it the data one wishes to chart. The chart details pane allows users to easily change the chart’s title, x and y axis, attributes label, and how the chart is to be sorted.

The chart then displays in a dashboard at the bottom of the map together with the layer’s data grid. One can easily save this chart, add another, or edit the chart data feeds and type to better represent your data. The chart below looks at base elevation compared to total acreage for Colorado ski resorts. The chart clearly that resorts with higher base elevations are generally smaller in acreage in comparison to those with lower base elevations. Simple as this information may be, and even logical in this case, it offers more insight into the mapped data and provides new tools for analysis for the user.

Additional Tools and Functionality

Functionality for the charts includes sorting data (highest/lowest), charting numeric and non-numeric data, and creating multiple charts for the same layer. The charts are also interactive. As in the image below, selecting a feature on the bar chart will highlight the matching feature on the map. By mousing over the values in the charts, users are alerted to the associated values via overlay tooltips.

Charting non-numeric data is handled with our pie charts. For example, if you wish to view how many of your data points share the same description for a specified data set, you would select pie chart and choose “description” as the selected attribute.

More announcements to be forthcoming as we continue to release new capabilities in Geocommons!

More Fun at Camp Roberts – Mobile Interoperability

mattmadigan — Fri, 11 Nov 2011 19:15:28 +0000

Last week I had the opportunity to go to Camp Roberts and do experiments with the Naval Postgraduate’s School RELIEF exercise at Camp Roberts, CA. GeoIQ has partnered with the QuickNets program including Ushahidi to provide dynamic visualization and analytics for situational awareness during disaster response. The 42,000 acres of rolling hills in Central California were the perfect backdrop to field our first mobile application.

The exercise was to test new technologies for identifying victims and dispatching responders more efficiently during disasters. For the purposes of the exercise victims were played by ribbons staked in the ground to act as a proxy of their location. To implement the test response system the Quicknets team used Ushahidi based microtasking, led by Cat Graham of HumanityRoad, to direct a volunteer core of responders. Microtasking is a brokerage function that facilitates voluntary selection of tasks by responders, monitoring status of those responses, and tracking them to closure.

At each location the victims were documented in a notebook. Responders recorded the victims condition and their location. This information was then texted with the identified victims description and location to the operations center. Additionally at each victim location we used the GeoIQ mobile application to mark the precise GPS coordinates, category and a picture of the victim.

During the response day, two teams of Israeli Hatzalah responders were dispatched by Quicknets working with NowForce and Camp Roberts base personnel to retrieve the ribbons as described by the dispatcher. At each location the responder collected the ribbon, but also closed the ticket by changing the category on the GeoIQ mobile application. The operation center was a flurry of activity. Dispatchers actively processed the incoming texts and the GeoIQ mobile app was updated in real-time as victims were discovered and collected.

As the afternoon wound down, there were several identified victims that had not been responded that we could see on the map. The Israeli team needed to head to the airport so John Crowley, Galit Sorokin and myself headed out with the GeoIQ mobile application and only using the map were quickly able to clear the field of the final eight ribbons and updated the status for operators back at base.

Operations Center view of updates via mobile device

The bonus of pushing the field data back to the GeoIQ server – it was instantly discoverable by the Camp Roberts team and portable in over a half dozen formats. Also the data could then be combined with other field collected data like Ushahidi, Geocent, Nowforce, and traditional GIS data from various sources. It is critical that during disasters data is accessible and portable, and does not get trapped in any single system. This will be all the more true when data is coming from mobile applications. To this end we’ve been working with a variety of third party mobile apps so their data can easily be published for quick discovery and format translation.

For the Quicknets team this meant they could start running analysis on the data right away. In this case we took the mobile data from the field and aggregated to the five rescue regions to quantify the activity by area. We also were able to use intersection, filtering, merging and temporal visualization to analyze the entire exercise.

The exciting next steps is combining GeoIQ’s new streaming analysis into a Camp Roberts exercise. This would allow a responder to kick off a search of people using Twitter to request help, and stream that data live to the mobile application. The responder can then verify the social media on the ground through annotations from the mobile application. These annotations stream back in real time to the command center allowing dynamic collaboration between the two. The potential to leverage the human sensor network of mobile devices during a disaster is something we believe will be transformative. We’ve already seen its impact during Hurricane Irene, where social media directed responders to flooded areas well ahead of satellite imagery availability. We’ve also gotten great feedback from urban search and rescue teams on the combination. We’ll be blogging more about cool ways to interconnect streaming data and mobile applications soon – lots of cool projects under way. Camp Roberts was an awesome experiences and we look forward to hooking up with more cool technologies and people the next time around.

Analysis of Response Active Across 5 Teams

New Tools for Exploring Data in GeoCommons!

Sean Gorman — Thu, 10 Nov 2011 02:05:25 +0000

As the number of data sets on GeoCommons continues to grow we are constantly looking for new ways to allow users to explore the data. A variety of recent projects have pushed us develop new tools for interacting with emerging multidimensional data streams. Today we are pushing out a new release that gives you three new tools for diving into the data in GeoCommons:

Charts – as much as we love map visualizations often times it is useful to view your data non-geographically. To get a new perspective on data in GeoCommons you can now, in a few clicks, create pie charts, bar charts, line graphs, area graphs and scatter plots.

Data grids - view and edit the raw tabular data while you are in the map view.

Multidimensional time slider - we’ve revamped out time slider to give you a new range of options for exploring the temporal dimensions of your data. The new time slider is two tiered so you can zoom into any time slice for more details. You also have the choice to aggregate and disaggregate your data at different temporal scales (seconds, minutes, hours, days, months or years). Last but not least you can choose between animating your data cumulatively or as a range.

We hope these new tools will let you plum the depths of your data for new discovery. As we see a larger variety of data coming on line the need to diversify the techniques by which you can explore new content will continue to be critical. The most exciting part – this is just the tip of the iceberg of new capabilities we’ve built into the GeoIQ platform. Stay tuned for additional releases as well as new and improved back-end API support to repurpose any of it for your own projects.

Streaming Data and GeoIQ

Chris Helm — Wed, 02 Nov 2011 12:00:09 +0000

Last week we announced GeoIQ Social where anyone can immediately visualize and analyze social media data with their internal and external data. But there is more than just connecting social data, it’s is necessary to make this data available on-demand in realtime.

We’ve been focusing on connectivity across data sources for a while, and this has been a driving force of GeoIQ itself: disparate datasets converging in easy-to-use tools that add value through data centralization and powerful visualization and deep analytics. However, for a while now the idea of centralizing data (in the cloud or anywhere else) has meant uploading data to a new location or a new service that then makes them available in new ways. This mold we’ve now broken.

A few months back we announced GeoIQ Connect. The idea was that within our platform users can tap into existing databases, run our analytics, explore data in new ways, and visualize data next to any dataset in the platform. This also meant that no longer would users have to export their data only to reimport it all into our software, but this also flipped a switched within the platform itself. GeoIQ had become dynamic. Data were dynamic. Maps and analyses were dynamic. We created “adapters” for all sorts of databases including PostgreSQL, MySQL, Oracle, HBase, and MongoDB as well as an even newer types of databases and APIs like Google Fusion Tables.

The real power in our dynamic data-stores is our ability to map data as they change. As data in a database are updated or changed in anyway, maps in GeoIQ can be refreshed and the new data will appear. This is great, but its not great enough. We wanted to go further, and now we have. We now have the ability to stream real-time data directly into maps. No more need to refresh!

To see how the new streaming feature works checkout this video that streams tweets mentioning rain, snow, or weather in real-time:

Streaming data layers allow for instant feedback, rapid decision making, and all an around cool experience. At first opening this streaming up for the Twitter Streaming API. Twitter has paved the way in its implementation of a streaming API and is also great because of the quantity of geo-spatial data that it streams. Of course there are more streaming APIs that just Twitter’s, and one that is very interesting to us is pachube.com. Pachube lets users relay streams of real time data and make use of its API to let others get access. These types of streams of data and APIs are becoming more common and now we’ve got the tools to use them.

Talking about the technical pieces of this new capability is the fun part of what we’ve done. We’ve employed a variety of tools to make it possible to create new streams of data that pour directly in our maps. We’ve also enhanced our API to make it possible for others to create a real-time feel to their maps embedded in other places on the Web. As an introduction I’ve briefly describe the technologies we’re using for streaming data:

Node.js

Node.js is starting to pop up all over the place. This is because its very simple to build fast and scalable web applications that support high numbers of concurrent requests. We’ve used Node.js to built a service that allows us to tap into an external streaming API. In this service we maintain all the logic for connecting to streaming data sources, processing the data and routing data through a series of messaging queues. Inside the GeoIQ platform we then stream the data in and route them to the correct maps via web-sockets and the Node.js library Socket.IO.

AMQP

We use use AMQP as a messaging system to communicate and transmit data and results across the web. AMQP provides scalability for our system in that we can bind to the message queue from various places and spawn new worker applications at will. The AQMP server acts a primary hinging point for routing data from the streaming service to the various GeoIQ servers that send data to maps.

Services

One of the most powerful aspects of this capability is our ability to pass data through a series of external services. This means that we can process the data in different ways as we receive it, and we can custom tailor the processing based on user demands, needs or intentions. For instance we have a set of three services that tweets can be passed through: geocoding tweet profile locations, a tweet sentiment engine (Repustate), and klout. This list will grow in the future to include various other services that can help users add more information to streams of data.

MongoDB

All incoming data gets stored in a MongoDB document store. This provides a fast and flexible way to store loosely structured data, and gives us some geo-spatial indexing as well. As part of GeoIQ Connect we built an adapter for MongoDB that allows us to connect to any MongoDB database and pull in the data directly to GeoIQ. For the streaming datasets we’ve re-used this adapter and take advantage of MongoDB’s simple query structure for limiting data to certain spatial extents, filtering the data, and extracting data in a variety ways.

Web-Sockets

One of the most important pieces of the new streaming in GeoIQ is the mechanism of delivery: Web-Sockets. Within all maps we now open a web-socket to a local socket server running Node.js and using Socket.IO. The socket server binds to the remote AMQP instance, routes the data to the correct MongoDB collection, and emits data to our maps. The web socket connection made by each map routes data pertaining to the correct data layers in the map and the Javascript handles the rest.

AddFeatures API

The ability for the web-sockets to pass data to a map is huge, but previously we had no way to change the data in a map once it had been mapped. This all changed when we developed a new API method called addFeatures. The method lets us dynamically append data to any layer in a map. So as new data points are received from the server we add it the correct layer in the map using “addFeatures”. Its very handy, and it allows any one to easily alter the data in their GeoCommons and GeoIQ maps and create their own realtime applications.

Our Next Steps

Its easy to think about all the possibilities that streaming opens up in our applications, and we’re not stopping any time soon. Its probably safe to expect us to be taking the idea of dynamic maps and data pretty far. Without giving it all away we’re thinking along the lines of realtime analytics, dynamic event alerting and more tools for easy collaboration. What we’ve just opens the door to bright, and dynamic future at GeoIQ.