The Web Data Research Assistant (aka WebDataRA)

WebDataRA is a tool has been developed by Prof Leslie Carr of the Web and Internet Science research group at the University of Southampton to support researchers using Web sourced data. Its functionality focuses on Twitter (the most commonly-researched Internet data source), but it also works for Facebook, Instagram, Tiktok, Google, Google News, Google Scholar and other database-like sites such as SSRN, Github and Quora. The ultimate aim is to make advanced network analysis and textual analysis methods accessible to social science and non-programming researchers, especially those who work in an interdisciplinary context.

>>>is transformed from a web page into a spreadsheet>>>

Installation and Support: To install the software, please go to the Chrome Web Store in your Chrome browser. The software is also compatible with the Microsoft Edge browser. For support please contact Leslie Carr. If you use it in your research, please acknowledge the Web Science Institute of the University of Southampton in any publications.

Operation

This Chrome extension takes information from search results pages and makes them available in an accessible spreadsheet form, summarising key components (title, contents, date, author etc.) The software understands the layout of web platform pages, and extracts information as appropriate. The information is saved as an HTML file, which can be subsequently opened directly as a spreadsheet in Excel.

On Twitter and Facebook pages, the software will continuously scroll to the bottom of the page, triggering the server to send more data and to allow all available results to be gathered. On Instagram, successive display pages of a user or a set of search results will be opened one after the other and the results saved. On Google, the current page's results will be extracted and saved.

The software requires keypresses in the main browser window to trigger its operation. Press Shift-Ctrl-A to start collecting data from the page. If you are in Twitter or Facebook then this keypress starts the browser scrolling downwards to trigger more data collection as normal. Press Shift-Ctrl-H to halt the data collection and save the data to a file. Press Shift-Ctrl-Q to check progress.

Documentation

The following training resources are available (but currently need some updating [June 2023]):

A Tutorial document that explains how to use this software to investigate simple research qyuestions, covering the use of Excel, Gephi and Voyant Tools to analyse the Twitter data.
A set of summary slides that distils the information from the tutorial.

Misc Notes

More information is available about forthcoming Twitter capabilities.