GeoDigger
Data Mining Location Information from Social Networking Services for Human Mobility Applications
Abstract
Recent developments in mobile computing and communication technologies have made real-time tracking of individuals a reality. Mobile devices such as smartphones and tablets, carried by millions of people around the world, create global-scale human sensor networks. People's social relations and whereabouts can now be learned by monitoring these networks through social networking services (SNS), such as Twitter, Facebook, and Foursquare. SNS generate massive amounts of near real-time data that are being actively explored in human mobility research, with a focus on areas such as urban planning, emergency response, and epidemic simulation.
GeoDigger is a tool developed at the Colorado School of Mines which can be used to collect and filter location information from SNS.
GeoDigger filtering capabilities include geographical scope, number of users, content, and time period. Although our tool currently only supports Twitter, we plan to include the ability to collect data from other SNS in the near future.
GeoDigger has successfully collected location information from more than 50 million Twitter users since its deployment in Fall 2013. As an open source initiative,
GeoDigger is available to the research community, so others with similar needs can use it to collect data themselves in an easy and productive way.
Requirements
GeoDigger has only been tested on Linux. It may or may not work on Windows or Mac OSX systems.
Python 2.7
MongoDB Server >= 2.4.6
Python Modules
Tweepy >= 2.1
PyCrypto >= 2.6.1
PyMongo >= 2.6.3
Installation
To install the
GeoDigger script and libraries, use:
tar xvfz geodigger.gz
cd geodigger
sudo python setup.py install
Usage
Make sure you have a MongoDB server installed and running. Running
GeoDigger on the same machine as the database server is easiest, and is recommended.
Obtain all the developer access secrets for the social networks you want to use. (Currently, only Twitter is supported.)
Create a configuration file. See examples/conf/geodigger.conf for an example.
Run the
GeoDigger script, passing in the paths to your configuration and log files.
GeoDigger will try to create the log file if it does not already exist.
geodigger --conf MY_CONFIG_FILE --log MY_LOG_FILE
Code Availability
If you would like the code for
GeoDigger, please complete the form below. This form will help us keep track of interest (for publications, etc); no personal information will be made available to anyone outside of the Toilers. Once you complete this form, you will receive an email from us with information on downloading the code.
