Social media monitoring is one of several ways for health authorities to capture specific insights into population perceptions about vaccine safety [1]. Commercial and open-source tools are helpful for gathering data on socio-cultural, religious, and political trends, but also for detecting what is being said about vaccine safety and why populations are delaying or refusing vaccination [2]. Monitoring and tracking digital forums for target audiences and influencers and identifying misinformation provides additional understanding. However, despite the clear need, public health authorities globally are severely constrained in their capacity to effectively address the overwhelming volume and complexity of misinformation [3]. To complicate matters, international, national, and corporate infodemic management policies are imposing information users to change the way they share information online and on social media platforms. Recent examples of how information users are adapting include the use of memes and increasing the amount and speed of information disseminated between platforms. This rapid evolution of misinformation tactics and limited public health resources including insufficient staffing, and a lack of specialized digital capacity within many public health authorities, renders comprehensive oversight incredibly challenging. As digital information environments become more complex, existing tools for social media monitoring need to adapt to meet the needs of public health authorities that may not have the resources to undertake comprehensive social media listening [4]. For health authorities to truly benefit from better quality social media intelligence, innovations must be developed that are not only required but also accessible, adaptable and feasible to implement [5].

The Vaccine Safety Net (VSN), the World Health Organization’s (WHO) global network that facilitates access to trustworthy, science-based vaccine safety information, identified this challenge as an opportunity to contribute to a rapidly expanding area. The VSN consists of member websites seeking to achieve more effective ways for communicating through digital and social media analytics research [6]. The latter involves identifying high impact vaccine safety related issues on social media for predicting and pre-bunking misinformation, developing and testing social media messages, as well as assessing their relevance and impact using a commercial platform with a social listening tool. Leveraging the VSN’s expertise in social medial listening, we developed and tested a custom keyword filter designed for adaptable global implementation by public health authorities. This filter aims to address the significant challenge posed by the sheer volume and evolving nature of misinformation, which often overwhelm existing commercial and online generic filters and their ability to provide precise results. For example, many generic keyword filters rely on estimations and are capable of over-filtering (or under-filtering) valuable social media content and fail to capture relevant information. Social media content is highly contextual and generic keyword filters do not interpret the nuances of language (e.g., comedy, sarcasm, anger, etc.)

We sought to understand how to optimize social media searches on vaccine safety using a custom keyword filter for better quality search results. A proof-of-concept project whereby a custom keyword filter was designed using Kim et al.’s [7] conceptual framework and tested using a commercial social listening platform and open source artificial intelligence (AI) tools with the intent of analyzing the quantity of irrelevant relevant mentions retrieved from vaccine safety searches on X^® in Canada, United States, Italy and United Kingdom.

Unfiltered keywords can yield large amount of irrelevant data [8]. Therefore, custom keyword filters are meaningful methods for improving digital and social media monitoring practices in response to constantly evolving information environments. For additional accuracy, we chose to first create and test a keyword filter with vaccine -related keywords. A vaccine safety keyword filter was subsequently created and tested to distill the information. We added artificial intelligence (AI) derived keywords to the filter, which expanded the social media datasets.

The custom keyword filter involves three steps: 1) frequency screening; 2) sampling; and 3) search implementation.

Thirty (30) vaccine and vaccine safety-related keywords were extracted for each country (see Figures 1, 2) using manual and AI methods. Vaccine-related keywords were used in three thousand one hundred and seventy-eight (3178) posts. While vaccine safety-related keywords were used in eight-hundred and sixty-nine (869) posts (see Table 1). Bert and GPT identified additional keywords including combined terms not previously identified. Themes extracted and analysed from vaccine safety-related mentions include public skepticism about vaccine safety, particularly COVID-19 vaccines, polarization between vaccination perspectives, concerns about misinformation, mistrust in government, influencers, and pharmaceutical companies.