However, we turned to Latent Dirichlet Allocation for extracting out structure from these tweets. LDA is a generative model that stipulates how documents are comprised of a mixture of topics and how each topic has a unique distribution over vocabulary. Since these topics and their vocabulary distributions are not directly observable, we can use statistical inference to infer those distributions given a corpus of text documents. As a result, we obtain a distribution over topics for each text document along with a distribution over vocabulary for each topic that can be used to gain insight into structuring of those documents.

LDA has been used before to solve a variety of problems, but one of the questions we’re interested in addressing is what happens when we perform LDA on a large scale discussion that evolves rapidly over time? We observe that topics exhibit ordering over time, suggesting that the topics extracted by LDA correlate to topics of discussion surrounding a sequence of events:

Upon inspection of the vocabulary distributions for each topics, we can reconstruct the series of events that drove discussions. We can see that initial discussions were focused around initial reactions and observations (“Two explosions reported near Boston Marathon finish line”). The discussion then transitions to people sympathizing with the victims (“How can I donate?” and “Our prayers go out to the victims”). The forth and fifth topics focus around the discovery of additional explosives that were dismantled along with the increased participation with news and media organizations. Finally, the last two topics focus around Google People Finder and Obama’s speech regarding the events. The top vocabulary associated with each topic is given as:

So, in short, we were able to demonstrate how there can be a temporal ordering of topics from topic modeling approaches that can help rebuild a story for a series of complex events. Moving forward, there are several questions that we’re interested in answering:

• How does the evolution of discussion differ depending on geography?
• What additional insight can we gain when using correlated topic models to extract out significant changes in these discussions?
• Are there any other topic model extensions that can be modified for taking into account spatio-temporal aspects?

What are the next steps for the Esri Dev Center team? Well, stay tuned as we extend these methodologies to look at how discussion evolve over geospatial regions!

Update (Friday, 3:50PM):

After hearing about the series of events the occurred on MIT campus last night, we collected tweets containing the keywords associated with “MIT” and “shooting”. Using the same methodologies, we see that the extracted topics exhibit intuitive ordering over time.

The corresponding vocabulary distribution for each topic is given by:

We believe stream analysis has the ability to be a paradigm shift in GIS and also broaden its applicability to new audiences.  The shift to streaming means that we don’t have to wait to collect our data before we do analysis.  This opens the door to generating analysis that detects patterns in data as events are occurring.  We can be proactive in our response vs. reactive after events occur.

A great example of this concept is disaster response.  Social media provides a direct communication channel to a subset of citizens.  We can tap into conversation about how citizens are reacting to disasters and the issues emerging in their streams of dialog.  The majority of social media usage in disasters is currently qualitative.  We visualize, map and ultimately read social media messages.  Exceptional project like TweakTheTweet and Ushahidi help us categorize and organize social data during crises.  These are immensely valuable, but we ultimately run into scaling issues with qualitative approaches.  This is where we believe the addition of stream analysis opens up the door to more quantitative approaches with social data.   

Not only can we analyze data as it emits, we can also look for changes in the stream that are indicative of significant events.  In which case we can alert users there is a change of interest to them.  Like streaming data, analysis can be perpetual as well, constantly keeping us updated and alerting us about significant activity.  This concept is best seen in practice.  Below is a video that walks us through tweets on a map, to simple streaming analysis and culminating in alert triggers based on event detection in the stream.

Social Media Streaming Analytics from GeoIQ on Vimeo.

We are excited to be moving these capabilities into the Esri platform and stay tuned for more on that front soon!

Approach

In order to successfully quantify sentiment within tweets, we wish to construct a binary classifier to differentiate between “positive” and “negative” tweets, otherwise known as polarity. However, constructing a relevant training data set can be considerably arduous (yet feasible), so we are going to rely on publicly available data sets to help train our models to differentiate between positive and negative tweets.

Selected models

Before jumping into the results, let’s take a moment to review the different classification models we’re going to be working with:

• Naive Bayes: a simple probabilistic model that we all love.
• Maximum entropy (via MEGAM): a general technique that estimates the conditional distribution of the class variable given a document. Note that for those who are familiar with multinomial logistic regression, it satisfies the characteristics of a maximum entropy classifier. To quote Nigam et. al.:
  
  “The underlying principle of maximum entropy is that without external knowledge, one should prefer distributions that are uniform. Constraints on the distribution, derived from labeled training data, inform the technique where to be minimally non-uniform. The maximum entropy formulation has a unique solution which can be found by the improved iterative scaling algorithm.”
  
• Support vector machine (via SVMlight): a non-probabilistic linear classifier that takes advantage of kernel functions to construct an optimally separating hyperplane between the two classes.
• Sentiment140.com: A publicly available API that we can use as a baseline. Their implementation is closed-source, however their algorithm can be read here.

Available data

We’re going to focus on training each of the three models on three data sets:

• B. Pang’s “Movie Review Data” (2004): 1k positive and 1k negative movie reviews taken from IMDb’s archive of the rec.arts.movies.reviews newsgroup.
• A. Maas’s “Large Movie Review Dataset” (2011): 12.5k positive and 12.5k negative movie reviews taken directly from IMDb. Note that we are using just the provided training set to conduct training and testing on.
• Sentiment140.com’s tweet corpus, a collection of tweets that have been classified via emoticons.

Note that any models that train using the first two data sets would be biased towards features (vocabulary) that is prevalent in movies reviews that extend beyond the 140-character limitation of tweets. This is a subtle caveat that may contribute to inaccurate predictions, however we will talk about how this limitation can be mitigated by the use of ensemble classifiers.

Training and validation: Building our models

Using NLTK (and the help of nltk-trainer), each model is trained and tested using a random 80%/20% split on each available data set. Additionally, we use perf to conduct analysis on the performance of the classifier.

The overall accuracy of each model configuration, based on the evaluation of the test data set, can be given as:

*At a threshold of 0.50

We can assess the performance of each model by analyzing predictions from the test data set. Once a prediction score is generated for a given piece of text, we use a “threshold” to decide whether or not a given text is positive or negative (thus, collapsing the prediction score to a class designation). If the prediction score exceeds the threshold, the text is decided to be of positive sentiment, otherwise the text is assumed to be negative. In the extreme case at which we specify the threshold to be 1.0, all tweets would be assumed to be negative and vice versa if we set the threshold to be 0.0. By adjusting the threshold, this allows us to evaluate Type I and Type II errors, which can be represented by the notions of precision and recall. This gives us a more insightful understanding of the performance of our model.

In the example case of the Naive Bayes model which was trained using Pang’s “Movie Review Data”, we can visualize different performance characteristic of the classifier. Additionally, we can test across different data sets to compare prediction consistency:

In this case, the model that was trained on Maas’s data set performed just as well when testing against Pang’s. (This the increased accuracy in the top-left graph.)

For a better description of these measures, I encourage you to take a look at the source code for perf as it includes a lot of fantastic documentation on how to interpret these values.

Training and validation: Validating predictions

Throughout the course of weekend of the Oscars, The Esri R&D Center team were able to collect 3 million tweets related to films highlighted during the Oscars. Generating prediction scores for each model configuration yields interesting interesting distributions:

Note that the “Default” and “Movies” datasets correspond to the “default” and “movie” topics available via the Sentiment140 API. We can see that the Naive Bayes model, trained with Pang’s data set, exhibits a natural distribution over the 3M tweets collected.

Furthermore, we can visualization the correlation of prediction scores across models:

The most notable observation is how both SVM and Sentiment 140 predictions exhibit negative correlation with other models. Keep in mind that the data used to train the Sentiment140 model was derived from tweets containing emoticons, thus the negative correlation with the Naive Bayes and maximum entropy model suggests that there is a poor intersection of common features among the two data sets. One possible way to mitigate this is by combining predictions across models / data sets via an ensemble classifier.

For the remainder of the analysis, we chose the Naive Bayes model trained using Pang’s data set.

Aggregating predictions to the county

Taking advantage of Esri’s geocoding services, we are able to geocode approximately 400k of the tweets. Aggregating these tweets to the county level highlights that a significant amount of them (unsurprisingly) originated in the Los Angeles area:

However, how can we go one step further. After aggregating tweets to county geometries, we can then reference the Tapestry data set to extract out market segment composition per county. Once that is done, a linear regression model is fit to each segment-film combination so that residual scores can be extracted for each county. As a result, we can identify counties which exhibit relatively high sentiment for the Laptops and Lattes segment and the film Argo. Specifically, when we zoom into the Northeast corridor, we observe high sentiment around the Washington, D.C. area:

Additionally, by looking at the residual scores for High Rise Renters and Lincoln, we can see that Los Angeles had a stronger negative sentiment compared to portions of the Northeast:

Please feel free to play around with the publicly available data set to see if you’re able to uncover any interesting trends! Note that you can filter on market segments via the Tapestry Households code, which is described here.

Conclusion

One of the significant challenges in this analysis was trying to identify a training data set that represents a natural separation in vocabulary which parallels what was observed within the production data set. We explained how different models can be trained on various data sets and further combined by the use of ensemble classifiers. Moving forward, additional geospatial analysis can be conducted on the residuals generated from the regression model to extract out additional insight from the data.

Keep in mind that the techniques illustrated here can be easily translated to solving other NLP-related problems centered around classification. Whether you’re trying to classify documents as threat/non-threat in the security domain, attempting to document insurance claims as fraud/non-fraud, or trying to determine social media posts as emergency/non-emergency, the methods here can be easily adapted to those separate domains.

Please feel free to play with the data and let us know if you’re able to uncover any interesting patterns! In the mean time, stay tuned for additional discussions related to further geospatial analysis on this data!

Additional resources

• To learn about how to assess the performance of a binary classifier, take a look at the source code for perf. There is excellent documentation that provides insightful descriptions on how to interpret binary classification predictions.
• There are a slew of NLP toolkits out there that feature rich pre-processing tools.
• For more information on statistical techniques used for solving classification, The Elements of Statistical Learning is available for free online.

Off the bat we got a fresh reminder of how few people include a GPS location for their Tweets – only about 2,500 in the case of the wildfires. Unfortunately the Twitter place ids were not included in the data, which will give you a city to neighborhood level of accuracy for 15-30% of Tweets roughly. It was still a very interesting sample of data and made for some useful analysis. Since we did a variety experiments with the data that would get overly long and boring I boiled it down to a list of challenges and potential best practices we discovered for working with the data.

CHALLENGES

• The volume of data with precise geographic coordinates tends to be quite low as a percentage of the over al data (1-2%).
  • This allows for the tactical extraction of specific Tweets, but makes for bad samples or indicators of the over all population.
• In addition to the demographic bias of Twitter users, there is also a bias in the volume of Tweets coming from individual users.
  • Current analysis techniques treat all Tweet equally whether it is a single Tweet from a user about the disaster or a user who Tweets 120 times in a day.
  • Aggregate analysis and correlation can be unduly influenced by deviations in the volume of Tweets from individual users.
  • Specifically we called this the Racerboi8 problem who was a user in the wildfire data who Tweeted 10x more than the next closest user and was far removed from the threat area of the wildfires.
  • Without normalizing the Tweet volume per users this can create false positives in the data.
• The need to have a specific keyword to identify disaster related Tweets means social media can’t be used until the disaster is well underway and the community has established keywords or hashtags.
  • Tweak-the-Tweet does a good job of curating data with the community but requires a taxonomy emerging from the community before it can start generating data.
  • This causes the loss of the capacity to tap Twitter as an early warning and indicator of an emerging disaster.

The Raceboi8 problem was a particulary thorny issue and I think a picture is worth a thousand words in this case:

That is one tenacious and pacing Tweeter!

BEST PRACTICES

While there are a good number of challenges with using social media streams like Twitter, I think there is also huge opportunity.  So, we wanted to share some of the approaches we took to tackling the problems we’ve encountered using social media during disasters.

• Filter Tweets by proximity to the disaster through intersection or buffer to remove Tweets that are reacting to news or outside observers, but not actually involved in the disaster.
• For aggregate statistics and analysis normalize Tweets by the volume of activity by users to remove sample bias.
• When doing macroscopic analysis like detection of patterns use the entire Twitter dataset for calculations then infer geography after the analysis has been run to do mapping and geoprocessing.
• Use pattern detection techniques to identify emergent hashtags and keywords that can be used to kick of targeted searches for tactical use.
• Use spatial/temporal regressions to identify social media voids by identifying geographies with more or less activity than the population would predict.  Entropy could be another good indicator if users wanted to make this a dynamically updating analysis.

A few visual examples are always nice to put these techniques in context:

In the analysis we filtered Tweet by proximity to wildfires in Colorado Springs and their proximity to critical infrastructure.  The gridded thematic is showing night time population courtesy of the Landscan project at ORNL.

To detect hot spots and voids we ran a simple linear correlation between the population in each grid cell (independent) and the number of Tweets (dependent).  The resulting map highlights areas with more Twitter activity than the population would predict.  The dark green square in the lower left hand corner ended up being the location of the local TV station that was pushing out a deluge of Tweets.  This and the Raceboi8 example drove home the need to normalize the data by Tweet volume for statistical analysis.

Last we hooked the data up to Twitch and did some streaming analysis to see what patterns we saw emerging in the data aggregated over time.  Generally Twitch is better for real time events but was still useful for some post event pattern analysis.

In the course of the experiment we had some great conversations and help from Nat Wolpert and Katie Baucom of NGA’s disaster response team, Chris Vaughn and Michael Gresalfi of FEMA, and data help from Chris Mayfield of NORTHCOM.  Their ideas and feedback have given us new set of motivations for furthering developing the ideas here.  Hope to have more to show to that end soon.  Thanks to Crowley & Co. for hosting us at another great event and thanks to Brendan Heberton for the heavy lifting!

Our goal at GeoIQ has been to create new mapping interfaces and hopefully change the geospatial market in the process. Esri opens the door for us to operate within the ArcGIS platform and to directly work with their millions of users. We are excited about the opportunity to collaborate with Esri to create the next generation of GeoWeb technologies. To accomplish this we’ll be working with Esri to establish a new research and development center in the Washington DC area. The District emerged as a hot spot for the clustering of GeoNerds, and we are looking forward to using the new office as a venue to support and collaborate with the community.

The team in the development center will focus on engineering core technologies for Esri, leveraging GeoIQ’s expertise in working with open data and technology communities, as well as real time and large data analytics. The team will build projects that push the bounds of data handling and analysis of emerging dynamic data sources. For example, over the past few months we have been working on an R&D project called “Twitch” that handles dynamic aggregation and visualization of millions of points from social media streams with in-browser HTML5 support. It is just an example of what is possible and we will be sharing more with the community as we integrate this technology with the ArcGIS platform.

Real time crowd patterns for the NYC marathon from GeoIQ on Vimeo

GeoCommons and GeoIQ customers will continue to be supported as we integrate the capabilities of ArcGIS Online and GeoIQ into a next generation platform. GeoCommons data will continue to stay open and available and we’re planning to make it even more accessible. If you have any questions for the team, we’ll be answering on the blog and Twitter @geoiq. Some of our team will also be at the International Open Government Data Conference in DC this week and the Esri User Conference in San Diego July 23rd to July 27th to meet the community and discuss our plans.

We look forward to continuing to work with everyone that has supported us and in becoming part of the Esri family.

The GeoIQ & GeoCommons team

Because GeoCommons is a open web platform every day we discover new uses that we didn’t expect or weren’t involved in creating. They’re constant sources of delight, interest, and a generate new ideas for how we can improve the tools we make for you to answer your questions and share your stories.

A few weeks ago the Washington Post’s Mike Debonis made maps of the DC Ward 5 special election using GeoCommons. And there are a lot more election maps that user have made. In other journalism the Bicycle Coalition of Greater Philadelphia built various maps that display commuting, health and crash data.

Even our partners see unexpected benefits. We shared the launch of UNEP/GRID-Arendal using GeoIQ to share ocean data and climate information. Using the common platform the World Wildlife Fund built an atlas to to compare sea ice levels over time with today’s industrial uses and protected areas, and see where WWF is working.

Similarly, the World Bank uses the GeoIQ platform to host their project location data and analysis. Within the Bank another group used the same World Bank GeoIQ platform to build an interactive benchmarking tool to help countries identify the challenges and opportunities they face in their performance on trade logistics and what they can do to improve their performance.

Schools and universities have independently started developing their own curricula and changing the concept of what GIS and mapping mean to students. University of Virginia, Harvard, Rowan Univeristy, Duke, and even the University of Redlands are just a few of many schools that have been using GeoCommons in the classroom.

Fortunately because GeoIQ is an entirely web based platform we don’t require users to download or install anything. You just need a web browser and you’re able to access a suite of online and easy to use tools for whatever visualization and analysis you can dream up. Then you can embed these into your newspaper, site, blog, or web application to effectively share important stories.

You can track additional sites where GeoCommons has been used on my pinboard bookmarks. And let us know where else you are using the GeoIQ platform or GeoCommons.

Also, if you are around DC I’ll also be presenting tonight at GeoDC about these and even some new upcoming features and uses of GeoCommons.

There have been TechCamps all around the world with great success in sharing experiences with the use of low-cost and easy to use technology by NGOs. At TechCamp Tel Aviv I will be walking through just a few of the examples of organizations that are using GeoIQ and GeoCommons to investigate data and tell compelling stories with the world. Groups such as the World Bank’s Mapping for Results and IADB’s MapAmericas, and even local groups such as the Bicycle Coalition of Philadelphia.

We will also be working over two days directly with NGOs on specific problems and objectives they have with making maps. I’d love to know if you have any open data for Israel or Palestine that we can also highlight to the local groups. We will make sure to post the videos and notes from the TechCamp when they’re online.