Event detection in finance using hierarchical clustering algorithms on news and tweets

• Published in: PeerJ. Computer science Vol. 7; p. e438
• Main Authors: Carta, Salvatore, Consoli, Sergio, Piras, Luca, Podda, Alessandro Sebastian, Reforgiato Recupero, Diego
• Format: Journal Article
• Language: English
• Published: United States PeerJ. Ltd 10-05-2021; PeerJ, Inc; PeerJ Inc
• Subjects: Algorithms; Cluster analysis; Clustering; Computational Linguistics; COVID-19; Data analysis; Data Mining and Machine Learning; Data Science; Datasets; Decision analysis; Decision making; Event detection; Finance; Hierarchical clustering; Information management; Inspection; Language processing; Natural Language and Speech; Natural language interfaces; Natural language processing; News; News analysis; Newspaper publishing; Qualitative analysis; Securities markets; Social Computing; Social media; Social networks; Stock exchanges; Web sites; New York; United States; China; Text mining; Stocktwits; Event detection; News analysis; Finance; Social media; Big data; Natural language processing; Hierarchical clustering
• ISSN: 2376-5992; 2376-5992
• DOI: 10.7717/PEERJ-CS.438

In the current age of overwhelming information and massive production of textual data on the Web, Event Detection has become an increasingly important task in various application domains. Several research branches have been developed to tackle the problem from different perspectives, including Natural Language Processing and Big Data analysis, with the goal of providing valuable resources to support decision-making in a wide variety of fields. In this paper, we propose a real-time domain-specific clustering-based event-detection approach that integrates textual information coming, on one hand, from traditional newswires and, on the other hand, from microblogging platforms. The goal of the implemented pipeline is twofold: (i) providing insights to the user about the relevant events that are reported in the press on a daily basis; (ii) alerting the user about potentially important and impactful events, referred to as hot events, for some specific tasks or domains of interest. The algorithm identifies clusters of related news stories published by globally renowned press sources, which guarantee authoritative, noise-free information about current affairs; subsequently, the content extracted from microblogs is associated to the clusters in order to gain an assessment of the relevance of the event in the public opinion. To identify the events of a day d we create the lexicon by looking at news articles and stock data of previous days up to d
Although the approach can be extended to a variety of domains (e.g. politics, economy, sports), we hereby present a specific implementation in the financial sector. We validated our solution through a qualitative and quantitative evaluation, performed on the Dow Jones'
dataset, on a stream of messages extracted from the microblogging platform Stocktwits, and on the
index time-series. The experiments demonstrate the effectiveness of our proposal in extracting meaningful information from real-world events and in spotting
events in the financial sphere. An added value of the evaluation is given by the visual inspection of a selected number of significant real-world events, starting from the Brexit Referendum and reaching until the recent outbreak of the Covid-19 pandemic in early 2020.