We’re fast into a new calendar year, predictions have been lain and we’re already beginning to see the emergence of some new and exciting technologies that will change future markets.

Last year was an incredibly exciting [...]]]>

We’re fast into a new calendar year, predictions have been lain and we’re already beginning to see the emergence of some new and exciting technologies that will change future markets.

Last year was an incredibly exciting year for GeoIQ. We were fortunate to work with a number of amazing users that are solving hard and meaningful problems. We launched products that have had an indelible mark on changing the mapping and geospatial web and a few new capabilities that we’re just getting started with.

A New Kind of Basemap

Nearly a year ago today we rethought the basemap when we launched Acetate. Our goal was to move beyond the cookie cutter standard maps that are confusing when visualizing thematic data and instead build a clean context where the data are more easily read and understood. And beyond just a simple “basemap”, Acetate peeled apart the map where data fits within the other layers of roads and placenames for a beautiful composite map.

We made Acetate the default basemap on GeoCommons as well as provided a terrain version. And because Acetate was built with open data and open tools we have distributed it to our users both online and offline in the field and behind firewalls – providing a simple mapping experience to everyone.

What we were hoping for, and happily saw occur, was the adoption of Acetate and the concept across the community.

GeoCommons 2.0 & Collaboration

Last Spring we revamped our user experience and launched GeoCommons 2.0. With the new capabilities, easier to create maps for everyone we felt that we truly opened up mapping and analysis to the world. To date, the community has grown to over 80,000 users per month and 4 million maps – considerably large for what used to be considered a niche domain.

For the first time, we provided free and open access to powerful geospatial analysis. At Where2.0 I spoke about Collaborative Analytics – enabling groups and organizations to quickly and easily share insight and make decisions together. Through GeoCommons anyone is able to ask a question and see their answers within a few minutes. But the true power occurs when they share this with their colleagues or friends – whether that’s within their team’s wiki or posting it to Facebook.

Beyond just a better user experience, we also did a lot under the hood to leverage the scaling of the Cloud. We can now dynamically increase or decrease GeoIQ on demand based on usage, ensuring that data is globally and immediately available. We also made this a core capability to the GeoIQ platform so that organizations deploying the GeoIQ platform behind their firewalls can do it quickly and easily.

The point is that only through collaboration of analysis can we reach concensus. Last year we shared with the world that idea, and looking forward we have a lot of concepts on how to make this more capable for monitoring and alerting on new information.

Graceful Degradation & HTML5

Web Browser innovation has evolved to a new era of web technology that we’re now starting to utilize. Historically Adobe Flash was the only way to provide truly performant and highly interactive visualizations. It still is the most powerful technology with nearly ubiquitous adoption. However, we’re finally seeing the advent of web native formats that are open and provide similar capabilities. Last year we introduced the capability for fully Javascript HTML5 maps when a user didn’t have Flash available. This ‘graceful degradation’ allowed users with Flash available to use the best performance, but also users the viewed maps on iPads or other devices to use the HTML5 maps seamlessly.

Looking forward this concept of appropriate interfaces for the user is being extended to more mobile screens, and even smaller and offline tablets. You can’t presuppose how or where a user will want to see and use their data so we’ll be pushing more into making GeoIQ maps and data available wherever, and however, you want.

Realtime Maps

Never content to merely improve capability we sought to introduce new ways to access and analyze realtime streaming data. Just as much as users are moving from desktops to mobile, data are moving from static captures to continous and dynamic. GeoIQ provides simple and easy access to these new streams of data that can be combined with static and organizational data to not just visualize, but monitor emerging information and see the impact of events.

The new technology we launched powers GeoIQ Social which has been used from monitoring events to helping search and rescue teams respond to cries for help. Like everything we do at GeoIQ our technology crosses domains to the common elements of time and space to help find solutions and measure their impact.

So those are just some highlights of where we’ve been and what we’ve done. It only begins to convey how busy, and excited, we’ve been the last 12 months alone in developing new and innovative technology that have helped our users and improved our customers. We’re looking forward to the upcoming year planning to develop and share with the word just as much – and even more – in pushing the boundaries of collaborative mapping.

We’re devoted to improving open-source geospatial technology [...]]]> If you’re not aware of OpenStreetMap – it’s a global community of volunteers that are gathering, measuring, and improving a map of the world. We have been long time members and supporters of the OpenStreetMap community and utilize the data and technology throughout GeoIQ and GeoCommons.

We’re devoted to improving open-source geospatial technology and so we’re excited to be hosting the first US-based OpenStreetMap Hack Weekend at our offices near Washington, DC next month on February 18 & 19, 2011. The last hack weekend was held in London last year

The meetup will be very technical and focus on building core applications and technology for the OpenStreetMap project as well as projects that support OpenStreetMap. For example, I’m interested in improving the export and sharing of OpenStreetMap data and want to see how to provide more up to date extracts of regions and types of data. Other projects include extending the core website platform, Mapnik and stylesheet improvements, data editor.

So if you are a member of OpenStreetMap and want to meet other developers working on the project you can sign up and join in on the planning. Or if you just want to learn more about the project and join the local community of people helping improve the map through data and mapping parties then check out MappingDC.

Last week I had the opportunity to go to Camp Roberts and do experiments with the Naval Postgraduate’s School RELIEF exercise at Camp Roberts, CA.  GeoIQ has partnered with the QuickNets program including Ushahidi to provide dynamic visualization and analytics for situational awareness during disaster response.   The 42,000 acres [...]]]>

Last week I had the opportunity to go to Camp Roberts and do experiments with the Naval Postgraduate’s School RELIEF exercise at Camp Roberts, CA.  GeoIQ has partnered with the QuickNets program including Ushahidi to provide dynamic visualization and analytics for situational awareness during disaster response.   The 42,000 acres of  rolling hills in Central California were the perfect backdrop to field our first mobile application.

The exercise was to test new technologies for identifying victims and dispatching responders more efficiently during disasters. For the purposes of the exercise victims were played by ribbons staked in the ground to act as a proxy of their location.  To implement the test response system the Quicknets team used Ushahidi based microtasking, led by Cat Graham of HumanityRoad,  to direct a volunteer core of responders.  Microtasking is a brokerage function that facilitates voluntary selection of tasks by responders, monitoring status of those responses, and tracking them to closure.

At each location the victims were documented in a notebook.  Responders recorded the victims condition and their location.  This information was then texted with the identified victims description and location to the operations center.  Additionally at each victim location we used the GeoIQ mobile application to mark the precise GPS coordinates, category and a picture of the victim.

During the response day, two teams of Israeli Hatzalah responders were dispatched by Quicknets working with NowForce and Camp Roberts base personnel to retrieve the ribbons as described by the dispatcher.  At each location the responder collected the ribbon, but also closed the ticket by changing the category on the GeoIQ mobile application.  The operation center was a flurry of activity.  Dispatchers actively processed the incoming texts and the GeoIQ mobile app was updated in real-time  as victims were discovered and collected.

As the afternoon wound down, there were several identified victims that had not been responded that we could see on the map.   The Israeli team needed to head to the airport so John Crowley, Galit Sorokin and myself headed out with the GeoIQ mobile application and only using the map were quickly able to clear the field of the final eight ribbons and updated the status for operators back at base.

Operations Center view of updates via mobile device

The bonus of pushing the field data back to the GeoIQ server –  it was instantly discoverable by the Camp Roberts team and portable in over  a half dozen formats.  Also the data could then be combined with other field collected data like Ushahidi, Geocent, Nowforce, and traditional GIS data from various sources.   It is critical that during disasters data is accessible and portable, and does not get trapped in any single system.  This will be all the more true when data is coming from mobile applications.  To this end we’ve been working with a variety of third party mobile apps so their data can easily be published for quick discovery and format translation.

For the Quicknets team this meant they could start running analysis on the data right away.  In this case we took the mobile data from the field and aggregated to the five rescue regions to quantify the activity by area.  We also were able to use intersection, filtering, merging and temporal visualization to analyze the entire exercise.

The exciting next steps is combining GeoIQ’s new streaming analysis into a Camp Roberts exercise.  This would allow a responder to kick off a search of people using Twitter to request help, and stream that data live to the mobile application.  The responder can then verify the social media on the ground through annotations from the mobile application.  These annotations stream back in real time to the command center allowing dynamic collaboration between the two.  The potential to leverage the human sensor network of mobile devices during a disaster is something we believe will be transformative.  We’ve already seen its impact during Hurricane Irene, where social media  directed responders to flooded areas well ahead of satellite imagery availability.  We’ve also gotten great feedback from urban search and rescue teams on the combination.  We’ll be blogging more about cool ways to interconnect streaming data and mobile applications soon – lots of cool projects under way.  Camp Roberts was an awesome experiences and we look forward to hooking up with more cool technologies and people the next time around.

Analysis of Response Active Across 5 Teams

We’ve been focusing on connectivity across [...]]]>
Last week we announced GeoIQ Social where anyone can immediately visualize and analyze social media data with their internal and external data. But there is more than just connecting social data, it’s is necessary to make this data available on-demand in realtime.

We’ve been focusing on connectivity across data sources for a while, and this has been a driving force of GeoIQ itself: disparate datasets converging in easy-to-use tools that add value through data centralization and powerful visualization and deep analytics. However, for a while now the idea of centralizing data (in the cloud or anywhere else) has meant uploading data to a new location or a new service that then makes them available in new ways. This mold we’ve now broken.

A few months back we announced GeoIQ Connect. The idea was that within our platform users can tap into existing databases, run our analytics, explore data in new ways, and visualize data next to any dataset in the platform. This also meant that no longer would users have to export their data only to reimport it all into our software, but this also flipped a switched within the platform itself. GeoIQ had become dynamic. Data were dynamic. Maps and analyses were dynamic. We created “adapters” for all sorts of databases including PostgreSQL, MySQL, Oracle, HBase, and MongoDB as well as an even newer types of databases and APIs like Google Fusion Tables.

The real power in our dynamic data-stores is our ability to map data as they change. As data in a database are updated or changed in anyway, maps in GeoIQ can be refreshed and the new data will appear. This is great, but its not great enough. We wanted to go further, and now we have. We now have the ability to stream real-time data directly into maps. No more need to refresh!

To see how the new streaming feature works checkout this video that streams tweets mentioning rain, snow, or weather in real-time:

Streaming data layers allow for instant feedback, rapid decision making, and all an around cool experience. At first opening this streaming up for the Twitter Streaming API. Twitter has paved the way in its implementation of a streaming API and is also great because of the quantity of geo-spatial data that it streams. Of course there are more streaming APIs that just Twitter’s, and one that is very interesting to us is pachube.com. Pachube lets users relay streams of real time data and make use of its API to let others get access. These types of streams of data and APIs are becoming more common and now we’ve got the tools to use them.

Talking about the technical pieces of this new capability is the fun part of what we’ve done. We’ve employed a variety of tools to make it possible to create new streams of data that pour directly in our maps. We’ve also enhanced our API to make it possible for others to create a real-time feel to their maps embedded in other places on the Web. As an introduction I’ve briefly describe the technologies we’re using for streaming data:

Node.js

Node.js is starting to pop up all over the place. This is because its very simple to build fast and scalable web applications that support high numbers of concurrent requests. We’ve used Node.js to built a service that allows us to tap into an external streaming API. In this service we maintain all the logic for connecting to streaming data sources, processing the data and routing data through a series of messaging queues. Inside the GeoIQ platform we then stream the data in and route them to the correct maps via web-sockets and the Node.js library Socket.IO.

AMQP

We use use AMQP as a messaging system to communicate and transmit data and results across the web. AMQP provides scalability for our system in that we can bind to the message queue from various places and spawn new worker applications at will. The AQMP server acts a primary hinging point for routing data from the streaming service to the various GeoIQ servers that send data to maps.

Services

One of the most powerful aspects of this capability is our ability to pass data through a series of external services. This means that we can process the data in different ways as we receive it, and we can custom tailor the processing based on user demands, needs or intentions. For instance we have a set of three services that tweets can be passed through: geocoding tweet profile locations, a tweet sentiment engine (Repustate), and klout. This list will grow in the future to include various other services that can help users add more information to streams of data.

MongoDB

All incoming data gets stored in a MongoDB document store. This provides a fast and flexible way to store loosely structured data, and gives us some geo-spatial indexing as well. As part of GeoIQ Connect we built an adapter for MongoDB that allows us to connect to any MongoDB database and pull in the data directly to GeoIQ. For the streaming datasets we’ve re-used this adapter and take advantage of MongoDB’s simple query structure for limiting data to certain spatial extents, filtering the data, and extracting data in a variety ways.

Web-Sockets

One of the most important pieces of the new streaming in GeoIQ is the mechanism of delivery: Web-Sockets. Within all maps we now open a web-socket to a local socket server running Node.js and using Socket.IO. The socket server binds to the remote AMQP instance, routes the data to the correct MongoDB collection, and emits data to our maps. The web socket connection made by each map routes data pertaining to the correct data layers in the map and the Javascript handles the rest.

AddFeatures API

The ability for the web-sockets to pass data to a map is huge, but previously we had no way to change the data in a map once it had been mapped. This all changed when we developed a new API method called addFeatures. The method lets us dynamically append data to any layer in a map. So as new data points are received from the server we add it the correct layer in the map using “addFeatures”. Its very handy, and it allows any one to easily alter the data in their GeoCommons and GeoIQ maps and create their own realtime applications.

Our Next Steps

Its easy to think about all the possibilities that streaming opens up in our applications, and we’re not stopping any time soon. Its probably safe to expect us to be taking the idea of dynamic maps and data pretty far. Without giving it all away we’re thinking along the lines of realtime analytics, dynamic event alerting and more tools for easy collaboration. What we’ve just opens the door to bright, and dynamic future at GeoIQ.