We worked with the World Bank starting about a year ago to launch Mapping for Results, a revolutionary initiative to geolocate and openly share every single location the World Bank was supporting activities. This past spring they launched with the 79 IDA countries including project activity locations and social indicators.

Today, they are releasing data for nearly their entire portfolio covering 136 countries, including the newest country of South Sudan where the World Bank has project activities but no reported financing yet. The data updates daily from the World Bank’s project API and is easily accessible through the download links on each page.

The World Bank are also leveraging the open-source Acetate maps to include terrain hillshading and placename labels to provide better context and meaning to the projects and activities in assessing their impact. See Kenya and Afghanistan.

What was particularly exciting about this release was that we were not involved at all in the development and updating of the newly released country data. Through the easy to use GeoIQ platform, the World Bank team was able to indepedently manage, visualize, and publish their own maps. Data and Mapping tools are made to be used by the domain and development experts, not by technologists and integrators just to share new data. The easier it is to share and create compelling and informative analysis the more timely and more effective that analysis will be.

There are a number of upcoming additions to Mapping for Results that we’ll be working with the World Bank team in adding some new capabilities. Now that the project has been gathering data they will be able to visualize changes over time in order to better share where development efforts are effective – and where they could be better. Enjoy the livestream and join the conversation.

Starting with the Oscar dashboard, we collected all the Tweets that mentioned the nominees for best movie, best actor and best actress. From this collection of data we populated the dashboard with a broad array of analyses. For each Tweet we assigned it to a major market based on its geography and also calculated the sentiment for each Tweet. We took a quick pass at putting together the highlights of the analysis in the slides below:

There is a lot going on under the hood here, but to keep it simple we are collecting a set of tags from the Twitter streaming API then performing a variety of analysis against the Tweets we pull. The two main analytical tasks are determining geographic origin followed by analyzing the sentiment of the Tweet. For geography we perform it at three different levels: 1) grabbing the coordinates for Tweets from GPS enabled phones 2) taking the bounding box for Tweets from Geo-IP and user designations and 3) using the location from the users profile. We work progressively from 1) to 3) and typically get locations for between 30-60%. For sentiment we tested out a variety of API’s and ended up using Repustate for this project and it held up well to the load.

This approach is not without its challenges. Profile location is notoriously ambiguous as research studies have elucidated. While there are good mitigation strategies for the profile issue we’ve seen a larger issue plague geographic analysis of Twitter. In the vast majority of Twitter geographic visualizations the data is geocoded and represented as a point on the map. The problem is the accuracy of this point versus where the Tweet actually came from varies wildly. In the case of lat long coordinates from a GPS enabled mobile phone this can be accurate within a few feet, but in the case of a profile city geocode it can be hundreds of miles off. Despite large variances in accuracy these points are typically shown as the same, which can cause misleading results.

To solve this problem we aggregate all the Tweets to polygons – in this case major market areas. The key is that the polygons you are aggregating to are larger than your accuracy error bound from geocoding. The cool thing the team did with the Oscar dashboard was make it so these aggregations happen dynamically. As Tweets come in they are intersected with major market polygons and the summary statistics are calculated for each major market. For any of the markets just click the graduated circle for it to get the aggregated statistics. Also you can click multiple movies or actors/actresses and it will calculate the aggregate summary statistics for any of the clicked items. You can also see Tweets from mobile devices by clicking “Current Tweets” to see exact locations and animate them over time by clicking “play”.

A second challenge with location based sentiment analysis is how meaningful are the results. I think one of the things we miss are margin of error calculations for sentiment analysis. Once we’ve aggregated data we have a sample size for that geography that we can calculate a margine of error against. In the summary statistics for each major market you can find a margin of error calculation for the sample size. This allows the viewer to know the confidence level for any analysis by geography.

The last nuance we’ve added to the dashboard is the ability the bring in demographics by major market to overlay a variety of income, ethnicity, age and gender beneath the Twitter sentiment. This allows users to see how a variety of demographics trends intersect the sentiment data. There is lots to play with a we look forward to feedback on how it can be approved and applied to other use cases.

A combination of talks and break-out sessions the event was interesting.  The mix of students, academics and professionals meant there were different views on the suitability of crowd-sourced information.  Discussion within my break-out group ranged from how can we verify the crowd to trust the information to how can we incentivize people to provide more structured information.  There was also the typical concern of how can responders know if they can trust information.  This had already been brought up in Michael Goodchild‘s talk “It’s About Time: The Temporal Dimension in VGI,” the idea being having some unverified data is better than having no data.  I think wider acceptance of crowd-sourced information is just a matter of better analysis tools to determine what data is better and encouraging the crowd to submit data so there is more information available.

I gave a talk about the new collaborative analytics tools we’ve been adding to GeoCommons.  Specifically my talk was “Enabling Collaborative Analytics for Faster Answers in a Crisis,” the idea is that the next step in a crisis is enabling the crowd to perform analysis.  Traditionally analysts create reports which then go to decision makers.  If changes need to be made to the end report tasks go back to the analyst.

The next step in analysis is to enable everyone to perform analysis. There are key things that need to happen in a system for this to be effective though. The first is to make analytics easy, this allows the user to make good decisions when they perform their analysis. Within GeoCommons we have aimed to do this and you see results of it through the application, such as when making a thematic map. When deciding on classifications of data we allow the user to match the type of data by matching their histogram versus the available classification schemes.

By making decisions as easy as matching pictures but allowing the user to go into the details it means that both experts and beginners can perform useful analysis. The next steps in encouraging collaborative analytics is making analysis traceable and making results extend-able. That changes the flow of analysis from an analysis making a report to allow analysis to be branched from and putting analysis tools in the hands of decision makers. This eliminates the bottle neck of only have a small group of individuals that can perform analysis and allows for faster response.

See my full presentation below:

In keeping with our final days of analytics I’m going to show how you can use GeoIQ to calculate the apportionment yourself. The University of Alabama has a decent breakdown of the equations needed for apportionment on their Math 103 website. We are using the Huntington-Hill Method for this since it is the method currently in use by the Census.

To get started I uploaded a spreadsheet of state names and populations. I then joined it to our state boundary dataset based on state name.

The first thing we need to do is calculate the Standard Divisor and the Standard Quota. I combined these into one custom equation. So I created an expression of “State Population/(Total National Population/Number of House Seats).” I then run an analysis using my custom equation and my newly uploaded state population dataset.

Next we need to calculate the geometric mean of the numbers the Standard Quota is between for each of the states. The geometric mean is the square root of the multiplication of 2 numbers. So if a state’s Standard Quota is 2.4 we need to calculate the geometric mean of 2 & 3. In order to do this I create an analysis widget that calculates the square root of the Standard Quota multiplied by the Standard Quota plus 1. Next I compare the Standard Quota to the Geometric Mean, if the Standard Quota is greater I round up if it is lower I round down this equation looks like this “if([standard_quota > [geometric_mean]) then [standard_quota].to_i else [standard_quota].to_i + 1 end”.

So now we have the predicted apportionment by state. In actuality the Census didn’t just release the population information by state today they also released the actual apportionments. We can also use GeoIQ to compare our results to those of the Census. We are going to do this with another analysis function, subtraction. Here I subtract the GeoIQ Apportionment result from the Official Apportionment.

After finishing the analysis and saving the result I check the statistics. The statistics of subtracting the two attributes should all be 0 if everything matches up. As you can see below that is the case.

One feature of the next analysis tools we have not highlighted so far is the ability to share these custom widgets. There are other ways of performing apportionment that have been used in the past. With many of them you still need to calculate the Standard Quota, by creating an analysis widget someone else can search and use this against their own data. This can be useful to collaborate on additional analysis or to run the same analysis on new data when it becomes available. It also provides lineage to how your data was created and so others can build off of your work or verify how your data was created.

Naturally I also uploaded the actual apportionment information into GeoCommons, and it is now available for download here and you can see the resulting map below.

#maker_map_42500 {width: 100%; height: 400px;}

View full map

maker_map_42500 = new F1.Maker.Map({map_id: “42500″, dom_id: “maker_map_42500″});

So that wraps up the 12 Days of Analytics here at FortiusOne. You saw how you can do very simple analysis such as addition and very complex such as the U.S. House of Representatives Apportionment.

One way you can use it is to measure Twitter influence. I subtracted Twitter followers from number of people a user followed. Then I created a map with two layers, one of people had more followers than people they follow and the other of people who follow more people.

The addition feature can be especially useful for adding together populations to create a total. In this map I took number of females under 16 and number of males under 16 and added them together. By using addition you can create targeted population groups for your analysis.

So remember that “column A + B = analysis,” not to be confused with the 2Gether video “U + Me = Us.”

There are three factors that constitute file bloat for any vector based geospatial data:

1) The number of attributes (how many columns)
2) The number of features (how many rows)
3) The complexity of the geometry (how much needs to be drawn)

You can do some clever things to manage the first two at a low level – although you still are going to have bloat when you convert to a standard file format. The third factor, geometry complexity, is interesting because you can also do some low level tricks whose savings can be passed along to standard file formats. Reducing the complexity of geometry is often called “map generalization” in academic circles.

The general concept is that you remove details from the map without loosing the message and context of the map. All maps have some form of generalization otherwise it would be a perfect reflection of reality. Academics have used algorithms to heuristically derive a map generalization. This is probably best explained with a few examples. Below is a map of Europe in full detail:

Next is map generalization that removes some of the detail but still keeps the context of Europe and the country boundaries:

Last a more extreme example with even greater detail removed:

To pull off these nifty computational tricks used to require some fairly sophisticated desktop software, but Matt Bloch and Mark Harrower at the University of Wisconsin figured out a clever way to enable enable real-time WYSIWIG map generalization. The resulting application is called MapShaper. You can upload a shapefile and run different generalization routines (with high level of control if you choose) then export the result back out as a shapefile or an EPS file. The shapefile export is down at the moment, but hopefully will back in action soon.

I think these kinds of technologies and mathematics are going to be increasingly important as we need to make ever larger datasets available. Especially when the receiving devices are increasingly mobile with even smaller data handling capabilities.