Pulling data from twitter26 February 2016
I’ve been doing research with data from twitter for a couple years and one of the things people want to know after my talks about it is how I collected the data.
There are various ways of getting tweets – some more complicated than others – but I have tended to use an online google document available at https://tags.hawksey.info/get-tags/ as it allows me to get what I want and is not too difficult to set up.
You’ll need a google account and a twitter account.
If you follow the link you’ll be able to save a version of the following page (this is where you’ll need a google account):
Once it’s set up (it explains what you need to do) you can put in the terms you want to be searched, set it up to run every hour and so on. [The only thing that is slightly more complicated is setting up twitter access the first time (you’ll need to set it up here in part using your twitter account: https://apps.twitter.com/, but the instructions are pretty clear and you only need to do it on your first file and any others you set up will already have the data]
For hourly searches of words that are very frequent, I’ve noticed that the program starts to struggle somewhere between 60 and 80 000 tweets, so you may need to set up a new search and stop the hourly searches on that particular file.
There’s a second tab on the page which has all the tweets collected (user names obscured):
I tend to copy the columns that I need into a new excel file then and work from there (the program sometimes collects double tweets, but excel lets you search for duplicates and delete them).