GeoIQ Blog » Engineering

OpenStreetMap code hacking weekend

Andrew Turner — Wed, 18 Jan 2012 18:28:58 +0000

If you’re not aware of OpenStreetMap – it’s a global community of volunteers that are gathering, measuring, and improving a map of the world. We have been long time members and supporters of the OpenStreetMap community and utilize the data and technology throughout GeoIQ and GeoCommons.

We’re devoted to improving open-source geospatial technology and so we’re excited to be hosting the first US-based OpenStreetMap Hack Weekend at our offices near Washington, DC next month on February 18 & 19, 2011. The last hack weekend was held in London last year

The meetup will be very technical and focus on building core applications and technology for the OpenStreetMap project as well as projects that support OpenStreetMap. For example, I’m interested in improving the export and sharing of OpenStreetMap data and want to see how to provide more up to date extracts of regions and types of data. Other projects include extending the core website platform, Mapnik and stylesheet improvements, data editor.

So if you are a member of OpenStreetMap and want to meet other developers working on the project you can sign up and join in on the planning. Or if you just want to learn more about the project and join the local community of people helping improve the map through data and mapping parties then check out MappingDC.

Streaming Data and GeoIQ

Chris Helm — Wed, 02 Nov 2011 12:00:09 +0000

Last week we announced GeoIQ Social where anyone can immediately visualize and analyze social media data with their internal and external data. But there is more than just connecting social data, it’s is necessary to make this data available on-demand in realtime.

We’ve been focusing on connectivity across data sources for a while, and this has been a driving force of GeoIQ itself: disparate datasets converging in easy-to-use tools that add value through data centralization and powerful visualization and deep analytics. However, for a while now the idea of centralizing data (in the cloud or anywhere else) has meant uploading data to a new location or a new service that then makes them available in new ways. This mold we’ve now broken.

A few months back we announced GeoIQ Connect. The idea was that within our platform users can tap into existing databases, run our analytics, explore data in new ways, and visualize data next to any dataset in the platform. This also meant that no longer would users have to export their data only to reimport it all into our software, but this also flipped a switched within the platform itself. GeoIQ had become dynamic. Data were dynamic. Maps and analyses were dynamic. We created “adapters” for all sorts of databases including PostgreSQL, MySQL, Oracle, HBase, and MongoDB as well as an even newer types of databases and APIs like Google Fusion Tables.

The real power in our dynamic data-stores is our ability to map data as they change. As data in a database are updated or changed in anyway, maps in GeoIQ can be refreshed and the new data will appear. This is great, but its not great enough. We wanted to go further, and now we have. We now have the ability to stream real-time data directly into maps. No more need to refresh!

To see how the new streaming feature works checkout this video that streams tweets mentioning rain, snow, or weather in real-time:

Streaming data layers allow for instant feedback, rapid decision making, and all an around cool experience. At first opening this streaming up for the Twitter Streaming API. Twitter has paved the way in its implementation of a streaming API and is also great because of the quantity of geo-spatial data that it streams. Of course there are more streaming APIs that just Twitter’s, and one that is very interesting to us is pachube.com. Pachube lets users relay streams of real time data and make use of its API to let others get access. These types of streams of data and APIs are becoming more common and now we’ve got the tools to use them.

Talking about the technical pieces of this new capability is the fun part of what we’ve done. We’ve employed a variety of tools to make it possible to create new streams of data that pour directly in our maps. We’ve also enhanced our API to make it possible for others to create a real-time feel to their maps embedded in other places on the Web. As an introduction I’ve briefly describe the technologies we’re using for streaming data:

Node.js

Node.js is starting to pop up all over the place. This is because its very simple to build fast and scalable web applications that support high numbers of concurrent requests. We’ve used Node.js to built a service that allows us to tap into an external streaming API. In this service we maintain all the logic for connecting to streaming data sources, processing the data and routing data through a series of messaging queues. Inside the GeoIQ platform we then stream the data in and route them to the correct maps via web-sockets and the Node.js library Socket.IO.

AMQP

We use use AMQP as a messaging system to communicate and transmit data and results across the web. AMQP provides scalability for our system in that we can bind to the message queue from various places and spawn new worker applications at will. The AQMP server acts a primary hinging point for routing data from the streaming service to the various GeoIQ servers that send data to maps.

Services

One of the most powerful aspects of this capability is our ability to pass data through a series of external services. This means that we can process the data in different ways as we receive it, and we can custom tailor the processing based on user demands, needs or intentions. For instance we have a set of three services that tweets can be passed through: geocoding tweet profile locations, a tweet sentiment engine (Repustate), and klout. This list will grow in the future to include various other services that can help users add more information to streams of data.

MongoDB

All incoming data gets stored in a MongoDB document store. This provides a fast and flexible way to store loosely structured data, and gives us some geo-spatial indexing as well. As part of GeoIQ Connect we built an adapter for MongoDB that allows us to connect to any MongoDB database and pull in the data directly to GeoIQ. For the streaming datasets we’ve re-used this adapter and take advantage of MongoDB’s simple query structure for limiting data to certain spatial extents, filtering the data, and extracting data in a variety ways.

Web-Sockets

One of the most important pieces of the new streaming in GeoIQ is the mechanism of delivery: Web-Sockets. Within all maps we now open a web-socket to a local socket server running Node.js and using Socket.IO. The socket server binds to the remote AMQP instance, routes the data to the correct MongoDB collection, and emits data to our maps. The web socket connection made by each map routes data pertaining to the correct data layers in the map and the Javascript handles the rest.

AddFeatures API

The ability for the web-sockets to pass data to a map is huge, but previously we had no way to change the data in a map once it had been mapped. This all changed when we developed a new API method called addFeatures. The method lets us dynamically append data to any layer in a map. So as new data points are received from the server we add it the correct layer in the map using “addFeatures”. Its very handy, and it allows any one to easily alter the data in their GeoCommons and GeoIQ maps and create their own realtime applications.

Our Next Steps

Its easy to think about all the possibilities that streaming opens up in our applications, and we’re not stopping any time soon. Its probably safe to expect us to be taking the idea of dynamic maps and data pretty far. Without giving it all away we’re thinking along the lines of realtime analytics, dynamic event alerting and more tools for easy collaboration. What we’ve just opens the door to bright, and dynamic future at GeoIQ.

GeoCommons 2.0: It’s Not Just a Pretty Face

admin — Thu, 09 Jun 2011 13:00:31 +0000

When we launched GeoCommons three years ago we had high hopes that it would turn into the vibrant community that it is today. However, I must admit that from a technological standpoint we did not fully predict what the underlying GeoIQ platform would become – a complex, scalable set of distributed services that enable users to ingest and geo-enable data from an ever-increasing number of sources, visualize data in a various forms (maps, charts, images), and analyze data to uncover interesting information. The GeoCommons of today is very different from what was launched in 2008.

Leading up to the launch of GeoCommons 2.0 our signup, data upload, and map-making rates soared to all-time highs, and as a consequence it became alarmingly clear that the infrastructure that has always been the backbone of GeoCommons would not scale well enough to support GeoCommons 2.0 – nor our ever-evolving plans for it. It needed a serious overhaul from the bottom up, not due to poor planning or architecture three years ago – in fact the old GeoCommons infrastructure ran largely unmodified from day one without a single major hiccup – but simply because GeoCommons 1.0 was a very different beast from the GeoCommons we launched last week.

So while the GeoIQ engineering team began to build out GeoCommons 2.0′s features we undertook a parallel effort to migrate our infrastructure from a slowly melting half-rack in Washington, D.C. to the magical wonderland known as the Cloud – Amazon AWS, to be specific.

While the GeoIQ platform has been deployed in numerous environments and configurations and is not tied to any specific cloud service, Amazon was an obvious home for GeoCommons for a number of reasons – beyond the fact that it enables us to scale performance and costs as needed. Firstly, like most startups we’re a very small team and we simply don’t have the resources to stand up, administer, and scale our own physical hardware, nor to manage most of the services that are essential in any scalable infrastructure. Talent abounds at GeoIQ, but it usually abounds in 10 places at once – we can’t afford to waste our time cutting Cat 5 to length, swapping out bad drives, kicking faulty cooling systems, or calling REDACTED tech support because our overpriced Internet pipe is on the fritz.

Secondly, we’ve been using AWS for GeoIQ Cloud deployments for years and are comfortable with it from an operational standpoint. Despite the recent outage in the US-East region we trust its reliability, and, perhaps more importantly, we’ve learned through years of trial and error how to effectively utilize AWS to ensure GeoCommons‘ reliability and get the best possible performance out of the GeoIQ stack. In fact, the US-East outage could not have come at a better time for us. We were in the midst of testing the new infrastructure and the outage afforded us an opportunity to test our failover plan during a real, unplanned, and uncontrolled situation. I’m proud to say that (almost) everything went as planned, although some of us did lose more hairs than normal on that day.

Finally, we are consistently impressed by the breadth of Amazon’s services and the rate at which new services are being announced. While no cloud is truly fire-and-forget, Amazon has come the closest by far. I trust that they’ll continue to announce stellar services that allow us to further sunset our custom deployment solutions, which can only make our lives easier and allow us to focus on the things we know best – GeoCommons & GeoIQ.

I’ll go ahead and stop before this turns into a love poem to Amazon (too late?), but over the next week or so I’ll be writing posts on our developer blog detailing the architectural and technological changes that necessitated our migration to the cloud. I will also describe how we’ve used Amazon’s services and outline some key lessons we’ve learned along the way. Hopefully you’ll find them far more informative and beneficial to your work than this post turned out to be!

Stay tuned!

Zachary Johnson joins GeoIQ

Andrew Turner — Mon, 06 Jun 2011 15:28:29 +0000

I’m excited to announce that GeoIQ is adding a talented cartographer and web developer to our engineering team – Zachary Forest Johnson. I’ll let him introduce himself, but let me just say, he’s impressed me since the first time I came across his work three years ago, and it never hurts to flatter us with blogging about some cool hacks using our API.

I’m very excited to be joining the GeoIQ development team as a Visualization Engineer. The GeoCommons 2.0 mapping platform is quite impressive, and I can’t wait to dig in and help the team develop new symbolizations, tools, and workflows that help our customers and the GeoCommons public make better sense of their huge geographic datasets.

My formal training is in cartography, having earned a M.S. from the University of Wisconsin in 2008. Along the way, I taught interactive cartography labs and led development on web mapping projects through the UW Cartography Lab. After graduating I worked on the SpatialKey mapping platform with Universal Mind, and then led development on the Indiemapper online cartography tool as a partner with Axis Maps. Though my background includes a lot of Flash programming, for the last year or so I’ve become quite comfortable in JavaScript, and can’t wait to explore and advance the Polymaps side of the GeoCommons visualization engine.

At GeoIQ I’ll be pushing for more charting options, more ways to visualize your data, and richer temporal animation tools. I think we have a great engineering team, and I can’t wait to add my cartographic background to the mix.

Just imagine what we’ll be able to create now with another amazing visualization engineer in our mix! We’re still looking for more world-class engineers on Javascript front-end, visualization, large-scale analytics and generally people we love working with. So check out our current postings or just drop us a line!

Better Know a GeoCommons Feature – Atom Feeds

admin — Tue, 23 Nov 2010 20:53:55 +0000

One of the great parts of working GeoCommons is seeing all the interesting datasets people upload. We have everything from Afghanistan Polling Stations to Collegiate Quidditch Teams in Texas. Daily I look through for interesting information to post on either our Facebook Group or on the GeoCommons Twitter account.

One feature that makes it easier for me to track what new maps and datasets are uploaded is the Atom Feeds feature. When you perform a search there is a link on the right hand side to link to the Atom feed. Since I want to see every dataset and every map that is made I search on ‘%’. You might have a specific interest though such as “water” or a country such as “Haiti“. Once you performed a search you like copy the Atom Feed link. You can now past this into your favorite RSS reader.

You can for example add this to Google Reader.

You can also add the link directly to your browser for example in Firefox as a Live Bookmark.

Atom feeds makes it easy to monitor new data you are interested in as it is upload into GeoCommons.

Toolbars for GeoCommons and GeoIQ

admin — Mon, 22 Nov 2010 21:58:38 +0000

I first talked about the ArcGIS version of our toolbars back at the DC Dev Meetup in September. Since then we’ve launched our developer site where you can download the toolbars, but you probably haven’t heard very much else about them. Designed to make sharing your analysis easier as well as find additional data we’ve added toolbars for both Excel and ArcGIS. Based on our .Net library wrapper they both allow direct upload into GeoCommons or your own GeoIQ instance, as well as searching and download of data.

With the addition of GeoJoin earlier this year we made it easier to turn your spreadsheet into a map. Still many of you spend a lot of your time in Excel preparing that spreadsheet. Now we’ve streamlined the process even further with the addition of our Excel toolbar. You can download the toolbar from here and once you’ve installed it there is a “GeoCommons” dropdown available from the menu. Currently the toolbar has been tested on version 2003-2007 of Excel and versions of Windows XP & 7.

Now with the addition of the ArcGIS Toolbar you can both search GeoCommons/GeoIQ directly from ArcMap and pull additional datasets into your analysis. One the analysis is finished publish back to GeoCommons or your own GeoIQ instance. This eliminates the step of saving your files from ArcGIS into a Shapefile, then uploading now you can publish directly. You can download the toolbar here and it works on version 9.2-9.3.1 of ArcGIS.

This is our first release of our toolbars and we’d love to hear what you think. Look for additional features as well as increased version support in the future!

Introducing Polymaps to GeoCommons

Chris Helm — Wed, 17 Nov 2010 19:03:40 +0000

A core component of GeoCommons is visualizing data through maps – allowing anyone to be a cartographer. We’ve worked with a number of amazing organizations to help us make GeoCommons a leading map visualization platform – such as AxisMaps and Stamen Design. In particular, a core piece of this visualization is built on Stamen’s work with the ModestMaps flash library. While Flash has recently received a lot of difficult criticism, we have sought to harness its power by integrating it as part of a larger web-based architecture and interface. In testing there has not, to date, been a nearly ubiquitously available technology across all of the Web that also provides the capability for high-end client side processing.

But that’s changing. Over the past few weeks the engineering team at FortiusOne has been hammering out some great new capabilities. One of those new capabilities has been the integration of Polymaps into our mapping platform.

We’re really excited to announce the integration of Polymaps into our geospatial visualization platform. Polymaps is a fast and light-weight javascript mapping framework specifically for rendering tiled imagery and vector data quickly and efficiently (using SVG for vectors).

The goal of this work has been to provide GeoCommons users with new ways of making maps, and to make those maps easily accessible on a larger number of platforms and devices. The integration of Polymaps creates a lot of new possibilities for our users that didn’t exist before, and will ensure that GeoCommons remains an industry leader and one of the most cutting edge tools available. Over the coming weeks and months users can expect to see a lot of additional capabilities being added to our Polymaps platform. We hope to deliver a complete and fully functional alternative to our Flash based maps in open and accessible formats.

We’re releasing our integration with Polymaps as a “beta” version and primarily for developers at this point. Many components that users are used to in regular GeoCommons maps aren’t quite ready yet, but will be soon. And of course we’d love to get feedback, so please let us know what you think!

Ground Water Withdrawal in millions of gallons vs. Generation of Energy from Hydroelectric Power in thousands of megawatthours

How to use Polymaps inside GeoCommons:

Using Polymaps within GeoCommons is simple. Any map, new or old, can be viewed as a Polymaps map. Below are a few examples of how to create/access a map with Polymaps from within the GeoCommons platform.

Geocommons: To access polymaps from any geocommons map simply append “?view=javascript” to the end of the map URL.

regular map url: http://geocommons.com/maps/29087
polymaps map url: http://geocommons.com/maps/29087?view=javascript

Embeds: GeoCommons maps can be viewed in “embed mode”, useful for embedding maps in other websites. Embedded maps can also take advantage of our Polymaps integration by also simply adding “?view=javascript” to the end of your map embed URL:

an embeddable polymaps url: http://geocommons.com/maps/33191/embed?view=javascript

API: Accessing Geocommons maps can also be done via our new API. Here again we’ve added the “view” parameter to the embeddable maps API.

see: http://geocommons.com/help/Javascript_API#Embed-map

Its important to note that this is still the beginning of our work with Polymaps. We wanted to release this to the public early to get your feedback and demonstrate to developers how to use the new GeoCommons API’s. We’re committed to regularly releasing updates to this new capability.

GeoIQ Developer Site

Andrew Turner — Fri, 12 Nov 2010 14:35:11 +0000

A few weeks ago we updated GeoCommons with the new major version of our platform. Amongst a number of user interface improvements we also completely re-architected our application programming interface (API) – providing a full suite of capabilities for developers to use in building applications on top of the GeoCommons and underlying GeoIQ platform.

This API integrates with all aspects of the data management, creation, visualization and analysis components both at the server level via REST and open-standards, as well as an interactive JavaScript interface for building web and mobile applications with our advanced mapping and charting capabilities. You can use it as just a place to put data – create some amazing interactive, embeddable visualizations, or as a fully integrated stack combined with your existing infrastructure of databases and collaboration tools.

To support developers and integrators that want to use our platform, we’re launching GeoIQ Developer – available at http://developer.geoiq.com. This landing site will provide a number of tools, documentation, and examples of applications that utilize the GeoIQ API in order to provide you with quick and easy suggestions for getting up and running.

We’re excited about the current applications that people have built using GeoIQ – which you can see in the gallery. Organizations such as the World Bank, Transnational Crisis Project, and the Vancouver Sun are just a few examples that you can currently check out for ideas, inspiration and code examples. We’ll be rolling out new examples over the next few weeks as more of our partners and users launch their applications. And if you would like us to highlight your GeoIQ integration – just let us know!

Federated GeoData powering the future of GIS

Andrew Turner — Fri, 08 Oct 2010 15:34:20 +0000

We’ve come a long way in the development and usage of data standards. I’ve personally been working over the last few years to push the adoption of formats like GeoRSS and OpenSearch-Geo. These have been baked deeply into any project or product that we’ve developed. Mapufacture’s original purpose was to promote GeoRSS by providing a catalog and tools to use the format that was meaningful to users. GeoPress adds GeoRSS and OpenSearch to WordPress, and Mapstraction added GeoRSS to the various mapping API’s before they supported it.

Our GeoIQ platform has these built in from the ground-up. Every dataset uploaded into GeoCommons is available as GeoRSS, KML, CSV, Shapefile, Atom, SQLite and more. Users are able to easily publish data in a variety of formats that make it consumable and shareable across all types of applications and integrations. In addition, you can search any GeoIQ server using OpenSearch-Geo and OpenSearch-Time and get results as KML, Atom, JSON, or HTML. Again, putting power into the user without them having to worry about the complexities of the format.

Kate featured how users can use OpenSearch in their web browser for quick searching. You can even use OpenSearch in Windows 7. We’ll be releasing some tools soon to connect GeoCommons and GeoIQ into even more of your tools.

GeoIQ uses OpenSearch to federate data between any server. All of our clients can easily perform a single search to get results back from any of their GeoIQ nodes or from GeoCommons. In the next few weeks we’ll be highlighting a few of our clients that will be sharing out some amazing data with their communities and everyone using this capability.

Just recently, Esri built in the ability to search GeoCommons from their GeoPortal tools. You can try it out. Many of the Esri portals can now leverage the tens of thousansds of datasets the GeoCommons community has helped find and share with the world. Similarly, like GeoIQ federates data between each other, they can search and pull in data from OpenSearch servers such as Esri, GeoNetwork and others.

It’s great to see the world of GIS leveraging these formats to connect specialized users into global communities. Spatial analysts can leverage business data and individuals and organizations can similarly use powerful and easy to use tools such as GeoIQ to pull in GIS data for their intelligence. In GIS, this is often referred to as an SDI (Spatial Data Infrastructure) – which like many acronyms and titles has brought to mean many different things to people. But really what it means is the ability to connect data and leverage distributed data and capabilities across many platforms.

Ping us if you have OpenSearch enabled services you would like to connect into GeoCommons or use GeoCommons in your search services. We’re excited to be a major connector on the growing GeoWeb.

Chris Helm joins FortiusOne

Andrew Turner — Wed, 22 Sep 2010 11:00:00 +0000

FortiusOne is rapidly growing and we have a very dedicated core engineering team and we want to work with other like minded, excited, intelligent and talented people that we respect. It’s difficult to find that right mix of a person that fits so well with our vision and team. So we’re excited to announce we’ve recently brought on an incredibly powerful engine in Chris Helm. Given his background in spatial analytics he’ll be taking a lead role in building out our next generation GeoIQ analysis that we’ve been rolling out over the past few months – and you can expect even more very soon.

I’ll let Chris introduce himself:

I’m really, really happy to be joining FortiusOne, and partaking in the complete awesomeness that is their platform. I hope to bring an extra bit of analytical punch to the team, and I’m sure we’re going continue to develop a great line of tools and services.

By way of introduction I’m a geo-spatial programmer (aka geo-geek supreme) residing on Colorado’s Front Range (Denver, CO) where I spend my life fishing, painting, and programming. I have a history of developing applications with open source techs that goes back 10 years and includes work with MapServer, OpenLayers, and PostGIS. Most recently I’ve been causing serious amounts of geo-trouble at the National Renewable Energy Lab (NREL) in Golden, CO where I was lead developer of the In My Backyard Tool (or IMBY), the Open PV Project and others.

At FortiusOne I plan on causing a lot more geo-trouble than ever before, pushing for more Javascript/HTML5, python, and lots of open-source tools. I plan playing a pivotal role in the next generation of GeoCommons, and I’m sure that the news from us will continue to pour out over the coming weeks.