A Mixed-Method Analysis of People’s Perception and Behaviour on Vaccination Program in Online Social Media

Soelar Shahrul Aiman*, Mustaffa Nurakmal Ahmad


School of Quantitative Sciences, Universiti Utara Malaysia, 06010 Sintok, Kedah, Malaysia.


*Corresponding Author’s Email: soelar.sa@gmail.com


ABSTRACT


Introduction: Social media has been a major source of various parties obtaining misinformation about vaccination programs and it has raised doubts about the vaccine, especially among parents. Vaccine hesitancy was listed as one of the top ten issues that threaten global health by the World Health Organization (WHO) in 2019. Therefore, this study aimed to examine sentiment on vaccines from Facebook pages, focusing on Facebook users’ views on vaccination programs through a mixed-method approach, which is a qualitative and quantitative combination analysis. Methods: This analysis uses all comments posted on Facebook pages downloaded using Facepager software from 01 December 2020 to 31 December 2020. R software is used to clean the comments and performed the sentiment analysis, while Gephi software is used to perform the semantic network analysis. Results: Out of 5,055 comments, the majority of the comments are negative sentiment (49.33%), followed by positive (32.58%) and neutral (18.08%) sentiment. For the negative sentiment, tolak, mati, and cucuk were the most frequent words ever used. For the positive sentiment, terima, setuju, and selamat were the most frequent words ever used. The positive network centered around the word dajjal focused on the word lab and FDA. In contrast, the negative network centered around the word vaksin focused on the word semulajadi and kerajaan. Conclusion: These findings are useful to understand the problem of parents who are skeptical of the vaccination program and also can be a guide for public health to communicate with the public through social media.


Keywords: Vaccine Hesitancy; Sentiment Analysis; Semantic Network Analysis; Social Media


INTRODUCTION

Children vaccination is one of the best strategies and cost-effectiveness in public health to prevent child mortality caused by infectious diseases (Ahmad et al., 2017). To date, two initiatives on children's vaccination have been introduced globally, which are the Decade of Vaccines and the Global Vaccine Action Plan (Macdonald et al., 2014), and locally in Malaysia, children's vaccination is known as the National Immunization Program, overseen by the Ministry of Health (MOH), who is responsible to provide free-of-charge vaccination services to all Malaysian children in all MOH facilities. Under this program, parents are provided with a schedule of vaccination as a guideline that the parents must follow. To date, the vaccination coverage in Malaysia has achieved more than 95% among the infants and young children who have completed the vaccination program is about 86.4% (Azizi et al., 2017; Lim et al., 2017).


Nonetheless, there are loopholes in the law to make parents vaccinating their children compulsory (Asari et al., 2018) and most likely became an attribute to the failure of the parents to comply with the childhood vaccination schedule (Abdullah, 2016). Malaysia recorded 18 cases of diphtheria two deaths in Kedah and Malacca in June 2016 (Abas, 2016). These adverse events have not met the main goal of the Global Vaccine Action Plan, which is targeted to reduce child mortality world widely (Macdonald et al., 2014). Furthermore, the number of parents who refused to vaccinate their children has increased day by day in Malaysia (Rumetta et al., 2020).


Vaccine hesitancy is a major cause of the worsening risk of disease outbreaks in the United States where vaccination coverage does not reach optimal levels (Zipprich et al., 2015). In addition, vaccine hesitancy was listed as one of the ten issues that threaten global health by the World Health Organization (WHO) in 2019 (Deiner et al., 2019). Vaccine hesitancy has three components that influence an individual’s refusal, delay, or reluctance to vaccinate even though having active concerns; these are behaviors, beliefs, and attitudes (Kang, Culp, et al., 2017; Peretti-Watel et al., 2015). The intervention program for vaccination refusal is the best strategy to find out the reasons for parents not to vaccinate their children and at the same time try to advise parents to return to vaccinate their children (Chan et al., 2018), however, the study was only conducted in Kedah and the evidence of the effectiveness that intervention program remains limited. A review on strategies intended to address vaccine hesitancy showed that the meta-analysis of 15 published literature reviews was no strong evidence of efficacy for any specific intervention (Dubé et al., 2015). In addition, interventions that are against anti-vaccine groups may be unsustainable, ineffective, and potentially more harmful than no intervention at all (Betsch et al., 2015; Betsch & Böhm, 2016; Dubé et al., 2015).


Four factors that cause vaccine hesitancy, which is a lack of understanding regarding the risk of the vaccine and the vaccine itself, having experience with vaccines, involving religious or political issues, and socioeconomic status (Anthony Appiah, 2021; Larson et al., 2011). The main cause of vaccine hesitancy does not necessarily involve individual-level issues such as knowledge and awareness, it also needs more attention at the community and social network level (Jarrett et al., 2015). This research focuses on current vaccine sentiment and differing vaccine sentiment among the general public at the community and social network level from Facebook pages.


Sentiment analysis is a computational study that has been conducted as a Natural Language Processing (NLP) that analyzes the people's sentiment, opinion, appraisal, evaluation, emotions, and attitudes regarding service, product, individual, organization, politic, topic, issue or their attributes (Ainin et al., 2020; Bakar et al., 2020; Budiharto & Meiliana, 2018; Kabir et al., 2020; Poecze et al., 2019). Usually, most researchers use the term sentiment analysis (Nasukawa & Yi, 2003) the original term is opinion mining (Dave et al., 2003). In the medical field, sentiment analysis was used to examine current vaccine sentiment on social media (Kang, Ewing-Nelson, et al., 2017; Klimiuk et al., 2021; Lyu et al., 2021, p.; On et al., 2019; Piedrahita-Valdés et al., 2021). It also was beneficial for public health to communicate closely with the community at the community and social network level, like Facebook pages.


Semantic network analysis is a graphical study that represents knowledge of relationships between words or written text that is structured as a network (Drieger, 2013; Eklund & Haemmerlé, 2008; Gloor & Diesner, 2014). Semantic network analysis consists of two terms that are nodes and edges. Nodes represent words that can be found in comments or text, classes of things, individuals, instances, or objects. While edges represent the connections or relationships between nodes. Semantic network analysis can extract meaningful ideas in terms of identifying emerging clusters in the form of a graph rather than analyzing the frequency of words itself (Doerfel, 1998). Based on previous studies, semantic network analysis was also closely related to Big Data and most researchers use the term Big Data in their title of the study (Ban & Kim, 2019; H.-S. Kim, 2017). In addition, semantic network analysis also was used regarding financial markets, fashion design trends, emergency management, political issues, service, and experience (An & Park, 2020; Colladon et al., 2020; Eddington, 2020; M. Kim et al., 2017; Liu et al., 2018; Oh & Kim, 2020; Xiong et al., 2019). In the medical field, semantic network analysis was used to examine vaccine hesitancy in online social media (DeDominicis et al., 2020;Ruiz, et al., 2020; Kang, Ewing-Nelson, et al., 2017; Ruiz et al., 2021; Wang et al., 2020) and also used to understand the user experiences of mental disorders (Han et al., 2017; Yoo et al., 2019). With semantic network analysis, analyzing the current vaccine sentiment from Facebook pages related to the vaccination program can enhance understanding of complex health behavior, particularly for vaccine hesitancy.


Despite the efforts by MOH to strengthen the existing strategies and maximize the vaccination coverage, especially in urban areas, children vaccination refusal cases are still occurring. Two main problems have been identified in this research that causes the occurrence of children vaccination refusal cases in Malaysia.


Firstly, there has been reported that parents always received wrong information about vaccination programs from unverified sources, especially on social media platforms (Himelboim et al., 2020; Kang, et al., 2017). They tend to believe what has been written and posted on social media instead of information from reliable sources like medical practitioners. The fact is a survey study conducted in Germany shows that 95.0% of participants agreed that the most reliable source to get information regarding children's vaccination is a pediatrician (Heininger, 2006).


Secondly, this problem has negatively impacted parents who are not up to date (e.g.: lower knowledge and practices on children vaccination), compared to those parents who are up to date (Awadh et al., 2014). According to Awadh et al (2014), a possible cause of this problem is the education and counseling program are not well routed among parents who refused the vaccination in Malaysia. Furthermore, there is a limited extensive study on factors and reasons for these parents who refused and return to the vaccination program. To date, the existing qualitative studies worldwide mainly explore the understanding of children's vaccination (Sobo, 2015; Tombs-Heirman, 2009), sources from which parents receive the information on vaccination (Leask et al., 2006), reasons for the children's vaccination refusal (Gross et al., 2015; Harmsen et al., 2013; Rumetta et al., 2020) and recommendations for children vaccination among parents who refused vaccination (Rumetta et al., 2020). The previous study has found an interesting finding whereby the number of parents who refused their children's vaccination decreased and return to accept the vaccination after the intervention (Chan et al., 2018). There is an urge to understand the pattern or combination of words that has arisen in the children's vaccination that can represent the negative, positive and neutral sentiment among parents (Callaghan et al., 2019).


Recently, a qualitative study is conducted by Rumetta et al (2020) in Malaysia on reasons and recommendations for children vaccination among parents who do not vaccinate their children with any type of vaccination. However, the study is conducted among parents in Klang Valley only, which is an urban area. Besides, due to the small number of samples, the study is not generalized to the whole population itself.


Hence, due to these gaps, this current study is aimed to examine current vaccine sentiment from Facebook pages related to the vaccination program and analyze the semantic network of vaccine sentiment. This research can be essential for MOH, as it has a great opportunity to maximize the vaccination coverage.


METHODOLOGY

This research employed the approaches of quantitative and qualitative which are addressed to sentiment analysis and semantic network analysis, respectively, commonly known as a mixed-method approach. The mixed-method approach type of analysis adopted in this research is semantic social networks (SSN), which are used to digitalize and visualize the ethnographic data, whether it is the online conversation format or social interaction where the codes are made by researchers in network form (Cottica et al., 2020). Firstly, a quantitative approach was conducted by using sentiment analysis based on the number of negative, positive, and neutral words regarding the vaccination program from the Facebook pages related to the vaccination program. Secondly, a qualitative approach uses semantic network analysis for analysing the vaccine sentiment.

Sentiment Analysis

The data analyses were performed using RStudio software version 4.1.0 (Allaire, 2012). Recently, R software has provided many free packages to quickly calculate the sentiment of textual contents at the sentence level and deliberately cumulate by grouping or row variables such as sentimentr and Sentiment Analysis (Kao & Poteet, 2007). Usually, the Open Authorization (OAuth) package of R language was used to access social media such as Facebook via APIs (Ravindran & Garg, 2015). First, an account on Facebook API was generated and linked to the Facebook account. Second, the Facebook API Authentication process was executed in Facepager software version 4.3 (Jünger & Keyling, 2019). The data was extracted raw from the well-known social media networking site in Malaysia – Facebook. The search strategy keywords applied, which refer to the words used in Facebook comments are tolak vaksin malaysia, tolak imunisasi malaysia, terima vaksin malaysia, terima imunisasi malaysia, penolakan vaksin malaysia, penolakan imunisasi malaysia, penerimaan vaksin malaysia and penerimaan imunisasi Malaysia. A total of 7,107 comments were collected between 01 December 2020 to 31 December 2020. Third, the raw data were screened to remove the unnecessary data such as emoticons, punctuations ‘@’, symbols, numbers, special characters, spaces, or stopwords such as ‘yang’, ‘di’, ‘pada’, and so on. The duplicate comments on a Facebook page were also have been removed in this step (Al-Saffar et al., 2018) as well as any comment on a Facebook page that contained “www” or “https://”, “@username”, “URL”, and “AT_USER” were removed (Ibrahim & Yusoff, 2015). Then lastly, the data was converted from uppercase to lowercase so that the data keep important information only for the next analysis (Rashid et al., 2013).


Sentiment analysis was used to analyze the sentiment, emotion, attitude, people’s opinion, evaluation, and appraisal toward some entities such as topics, events, issues, products, service, individual, organization, or their attributes (Budiharto & Meiliana, 2018). Generally, the sentiment analysis was generated to set up or search a list of words in the SentiLexM, which is a sentiment lexicon for the Malay language built using multiple sources of information such as AFINN-111 (Nielsen, 2017) and Kamus Dewan Inggeris Melayu – An English-Malay Dictionary (Bakar et al., 2020; Tan et al., 2016). The lexicon is constituted of around 26,004 polarized words with more specific 2,477 for English words and 23,527 for Malay words, classified by their morphological category annotated with polarities positive, negative, and neutral. In constructing a classifier model, the pre-processed data was scored and grouped into two groups that are positive and negative by using the Syuzhet package. The sentiment score was calculated using the score formula below:


image

The Npositive is denoted as the number of positive words, while Nnegative is denoted as the number of negative words. The sentiment score for each sentence is categorized into:

  1. Sentiment score of more than 0 is considered to be an overall positive opinion.

  2. A sentiment score of less than 0 is considered to be an overall negative opinion and,

  3. A sentiment score equal to 0 is considered to be a neutral opinion.


Semantic Network Analysis

The semantic network analysis was used for the selected network metrics, which serves the purpose to reduce bias in the interpretation (Thovex & Trichet, 2013). Besides, semantic network analysis is also described as a process of analyzing the semantic network structure among nodes or agents (Drieger, 2013). The positive and negative sentiment networks analysis was concentrated on the best-connected component or subgraph. There are four most common centrality measures (Nieminen, 1973; Shaw, 1954), which are the degree of betweenness (Freeman, 1977), and closeness (Beauchamp, 1965; Sabidussi, 1966), and eigenvector (Bonacich, 1972). In a centrality degree, the centrality is denoted as CD, which measures the number of links upon a node. In betweenness centrality, the centrality is denoted as CB, which is a measure of the shortest paths between the other two nodes in the network. In closeness centrality, the centrality is denoted as CC, which measures how fast the information from a given node to every other node in the network. Finally, the eigenvector centrality, which is denoted as CE, measures the importance of a node to its neighbors. The WORDij software (Danowski, 2013) was used to construct and analyze the network, while visualizations were created in Gephi software version 0.9.2 (Bastian et al., 2009).


RESULTS AND DISCUSSION

Out of 5,055 comments, the majority of the comments are negative sentiment (49.33%), followed by positive (32.58%) and neutral (18.08%) sentiment.


Sentiment Analysis

Overall, out of 19,974 words, Table 1 showed that the most frequent words used were tolak (9.71%) and vaksin (8.25%) where both words express more the negative sentiment, which represents 91.59% and 48.76% of all words, respectively, followed by terima (3.90%), which express more towards the positive sentiment (79.82%). Words listed in Table 1 can also be presented in a graphic format. The Wordcloud can show the most popular words graphically and it is a quick way to summarize the content of comments on a Facebook page. In the Wordcloud, the more often the word is used, the bigger the word appears. As can be seen in Figure 1, tolak is the most used word in this study, followed by terima, tidak, setuju dan selamat. This result is supported by another study regarding the COVID-19 vaccine on Twitter (Lyu et al., 2021) which is the most tweeted topic was opinions about vaccination that is 15.2%. According to Lyu et al., (2021), out of 1,499,421 tweets posted on Twitter, the highest percentage of the topic are opinions about vaccination, which contribute 15.2% of all words studied. The words that contributed to this topic are people, get, take, go, want, make, like and think. In addition, other topics are almost similar to this study, which is related to the function of vaccines (5.43%). The words that contributed to this topic are people, effect, test, risk, side, immune, and death.


Semantic Network Analysis

Table 2 presents the outcome of the Overall Sentiment for Words from the Semantic Network Analysis. From this table, vaksin, tolak and terima have the highest degree (CD), betweenness (CB), closeness (CC) and eigenvector centrality (CE) with the values of CD=1.00; CB=1.00; CC=0.81; CE=1.00, CD=0.78; CB=0.25; CC=0.59; CE=0.72 and CD=0.73; CB=0.12; CC=0.56; CE=0.63, respectively. These three words show the existence of very high connectivity, very importance in information flow, very easy access to all nodes, and very high neighbor connectivity in the analysis. The Negative Sentiment analysis is presented in Table 3 and the explanation of the findings is as follows. Only three words scored a highest degree, betweenness, closeness and eigenvector centrality, which are vaksin (CD=1.00, CB=1.00, CC=0.81, CE=1.00), semulajadi (CD=0.78, CB=0.25, CC=0.59, CE=0.72) and kerajaan (CD=0.73, CB=0.12, CC=0.56, CE=0.63). Meanwhile, for the Positive Sentiment, similar to the Overall Sentiment analysis, three words have been determined to score the highest based on the values of degree, betweenness, closeness, and eigenvector centrality, as shown in Table 4. The three words are dajjal (CD=1.00, CB=1.00, CC=0.81, CE=1.00), lab (CD=0.78, CB=0.25, CC=0.59, CE=0.72) and FDA (CD=0.73, CB=0.12, CC=0.56, CE=0.63). Figure 2 provides a semantic network view of the connectivity between these words. This semantic network view is also helpful in visualizing how the various words are related. Overall, the word vaksin which has a high centrality value was directly connected to the words tolak, terima, cucuk, suntik, mati, and kesan indicating that accept vaccination if no side effects, however, most of them reject vaccination mainly due to side effects that can lead to death. For the negative sentiment, the word vaksin which has a high centrality value was directly connected to the words semulajadi, kerajaan, and tolak indicating that rejection of vaccination is mainly recommended to the government to use nature as the main ingredient in producing vaccines. For the positive sentiment, the word dajjal which has a high centrality value was directly connected to the words lab and FDA indicating that believing that these vaccines have a specific agenda that is closely related to the Food and Drug Administration (FDA). This result is supported by Kang et al (2017) regarding vaccine sentiment in online social media. According to Kang et al (2017), vaccines and vaccination were the most central nodes and centrality measures for the negative sentiment, neutral sentiment, and positive sentiment.


CONCLUSION

Findings from this research contributed to two new discoveries. In terms of the methodology of research, this is the first study that has been conducted in Malaysia whereas most of the previous studies were handled in the United States, Poland, Spain, and the Republic of Korea. Hence, this research finding enriched the body of knowledge in terms of cross-sectional research design. From a practical perspective, by understanding parents’ behavior and opinion about children’s vaccinations on social media, this research is beneficial for practitioners, especially the Malaysia Ministry of Health (MOH). It can assist the MOH, specifically the public health communication department to identify important keywords that are usually being used by the public in order to study the parents’ sentiment on the vaccination program. The keywords determined in this research can be used to study the vaccine hesitancy issues, such as vaccination campaigns or consultations on social media in the MOH strategies of the vaccination program.


Conflict of Interest

The authors declare that they have no competing interests in writing this article.


ACKNOWLEDGEMENT

The authors are thankful to the institutional authority for completion of the work.


REFERENCES


Abas, A. (2016), Five diphtheria deaths in Malaysia so far. NST Online: https://www.nst.com.my/news/2016/06/155591/five-diphtheria-deaths-malaysia-so-far

Abdullah, N. H. (2016). Kenyataan Akhbar KPK 26 Ogos 2016 – Status Kes Difteria di Malaysia – From the Desk of the Director-General of Health Malaysia. https://kpkesihatan.com/2016/08/26/kenyataan-akhbar-kpk-26-ogos-2016-status-kes-difteria-di- malaysia/

Ahmad, N. A., Jahis, R., Kuay, L. K., Jamaluddin, R., & Aris, T. (2017). Primary immunization among children in malaysia: Reasons for incomplete vaccination. J Vaccines Vaccin, 8(3), 2. https://doi.org/10.4172/2157-7560.1000358

Ainin, S., Feizollah, A., Anuar, N. B., & Abdullah, N. A. (2020). Sentiment analyses of multilingual tweets on halal tourism. Tourism Management Perspectives, 34, 100658. https://doi.org/10.1016/j.tmp.2020.100658

Allaire, J. (2012). RStudio: Integrated development environment for R. Boston, MA, 770, 394.

Al-Saffar, A., Awang, S., Tao, H., Omar, N., Al-Saiagh, W., & Al-Bared, M. (2018). Malay sentiment analysis based on combined classification approaches and Senti-lexicon algorithm. PloS One, 13(4), e0194852. https://doi.org/10.1371/journal.pone.0194852

An, H., & Park, M. (2020). Approaching fashion design trend applications using text mining and semantic network analysis. Fashion and Textiles, 7(1), 1–15. https://doi.org/10.1186/s40691-020- 00221-w

Anthony Appiah, K. (2021). As a Doctor, May I Refuse to See Unvaccinated Patients? - The New York Times. The Ethicist, The New York Times Magazine. https://www.nytimes.com/2021/07/13/magazine/vaccination-patients- ethics.html?smtyp=cur&smid=fb-nytimes

Asari, K. N., Makhtar, M., Asuhaimi, F. A., & Pauzai, N. A. (2018). Compulsory Childhood Vaccination in Malaysia: Public Health versus Parental Autonomy. International Journal Of Academic Research In Business And Social Sciences, 8(12). http://dx.doi.org/10.6007/IJARBSS/v8-i12/5256


Awadh, A. I., Hassali, M. A., Al-lela, O. Q., Bux, S. H., Elkalmi, R. M., & Hadi, H. (2014). Immunization knowledge and practice among Malaysian parents: a questionnaire development and pilot-testing. BMC public health, 14, 1107. https://doi.org/10.1186/1471-2458-14-1107

Mohd Azizi, F. S., Kew, Y., & Moy, F. M. (2017). Vaccine hesitancy among parents in a multi-ethnic country, Malaysia. Vaccine, 35(22), 2955–2961. https://doi.org/10.1016/j.vaccine.2017.04.010

Abu Bakar, M.F., Idris, N., Shuib, L., & Khamis, N. (2020). Sentiment Analysis of Noisy Malay Text: State of Art, Challenges and Future Work. IEEE Access, 8, 24687-24696.

Ban, H.-J., & Kim, H.-S. (2019). Semantic network analysis of hotel package through the big data. Culinary Science & Hospitality Research, 25(2), 110–119. https://doi.org/10.20878/cshr.2019.25.2.014


Bastian, M., Heymann, S., & Jacomy, M. (2009). Gephi: An open source software for exploring and manipulating networks. Proceedings of the International AAAI Conference on Web and Social Media, 3(1). https://doi.org/10.13140/2.1.1341.1520


Beauchamp M. A. (1965). AN IMPROVED INDEX OF CENTRALITY. Behavioral science, 10, 161– 163. https://doi.org/10.1002/bs.3830100205

Betsch, C., & Böhm, R. (2016). Detrimental effects of introducing partial compulsory vaccination: experimental evidence. European journal of public health, 26(3), 378–381. https://doi.org/10.1093/eurpub/ckv154

Betsch, C., Böhm, R., & Chapman, G. B. (2015). Using behavioral insights to increase vaccination policy effectiveness. Policy Insights from the Behavioral and Brain Sciences, 2(1), 61–73. https://doi.org/10.1177/2372732215600716

Bonacich, P. (1972). Factoring and weighting approaches to status scores and clique identification. Journal of Mathematical Sociology, 2(1), 113–120. https://doi.org/10.1080/0022250X.1972.9989806

Budiharto, W., & Meiliana, M. (2018). Prediction and analysis of Indonesia Presidential election from Twitter using sentiment analysis. Journal of Big Data, 5(1), 1–10.

Callaghan, T., Motta, M., Sylvester, S., Lunz Trujillo, K., & Blackburn, C. C. (2019). Parent psychology and the decision to delay childhood vaccination. Social science & medicine (1982), 238, 112407. https://doi.org/10.1016/j.socscimed.2019.112407

Chan, H. K., Soelar, S. A., Md Ali, S. M., Ahmad, F., & Abu Hassan, M. R. (2018). Trends in Vaccination Refusal in Children Under 2 Years of Age in Kedah, Malaysia: A 4-Year Review From 2013 to 2016. Asia-Pacific journal of public health, 30(2), 137–146. https://doi.org/10.1177/1010539517751312

Colladon, A. F., Grassi, S., Ravazzolo, F., & Violante, F. (2020). Forecasting financial markets with semantic network analysis in the COVID-19 crisis.

Cottica, A., Hassoun, A., Manca, M., Vallet, J., & Melançon, G. (2020). Semantic Social Networks: A Mixed Methods Approach to Digital Ethnography. Field Methods, 32(3), 274–290. https://doi.org/10.1177/1525822X20908236

Danowski, J. A. (2013). WORDij version 3.0: Semantic network analysis software. Chicago: University of Illinois at Chicago.

Dave, K., Lawrence, S., & Pennock, D.M. (2003). Mining the peanut gallery: opinion extraction and semantic classification of product reviews. WWW '03. https://doi.org/10.1145/775152.775226

DeDominicis, K., Buttenheim, A. M., Howa, A. C., Delamater, P. L., Salmon, D., Omer, S. B., & Klein, N. P. (2020). Shouting at each other into the void: A linguistic network analysis of vaccine hesitance and support in online discourse regarding California law SB277. Social Science & Medicine, 266, 113216. https://doi.org/10.1016/j.socscimed.2020.113216

Deiner, M. S., Fathy, C., Kim, J., Niemeyer, K., Ramirez, D., Ackley, S. F., Liu, F., Lietman, T. M., & Porco, T. C. (2019). Facebook and Twitter vaccine sentiment in response to measles outbreaks. Health informatics journal, 25(3), 1116–1132. https://doi.org/10.1177/1460458217740723

Doerfel, M. L. (1998). What constitutes semantic network analysis? A comparison of research and methodologies. Connections, 21(2), 16–26.

Drieger, P. (2013). Semantic network analysis as a method for visual text analytics. Procedia-Social and Behavioral Sciences, 79, 4–17. https://doi.org/10.1016/j.sbspro.2013.05.053


Dubé, E., Gagnon, D., MacDonald, N. E., & SAGE Working Group on Vaccine Hesitancy (2015). Strategies intended to address vaccine hesitancy: Review of published reviews. Vaccine, 33(34), 4191– 4203. https://doi.org/10.1016/j.vaccine.2015.04.041


Eddington, S. M. (2020). Alt-resilience: A semantic network analysis of identity (re) construction in an online men’s rights community. Journal of Applied Communication Research, 48(1), 114–135. https://doi.org/10.1080/00909882.2019.1706099

Eklund, P., & Haemmerlé, O. (2008). Conceptual Structures: Knowledge Visualization and Reasoning: 16th International Conference on Conceptual Structures, ICCS 2008 Toulouse, France, July 7-11, 2008 Proceedings (Vol. 5113). Springer.

Freeman, L. C. (1977). A set of measures of centrality based on betweenness. Sociometry, 35–41. https://doi.org/10.2307/3033543

Gloor, P., & Diesner, J. (2014). Semantic Social Networks. Encyclopedia of Social Network Analysis and Mining. Springer.

Gross, K., Hartmann, K., Zemp, E., & Merten, S. (2015). ‘I know it has worked for millions of years’: The role of the ‘natural’in parental reasoning against child immunization in a qualitative study in Switzerland. BMC Public Health, 15(1), 1–7. https://doi.org/10.1186/s12889-015-1716-3

Han, S., Kim, K. J., & Kim, J. H. (2017). Understanding Nomophobia: Structural Equation Modeling and Semantic Network Analysis of Smartphone Separation Anxiety. Cyberpsychology, behavior and social networking, 20(7), 419–427. https://doi.org/10.1089/cyber.2017.0113

Harmsen, I. A., Mollema, L., Ruiter, R. A., Paulussen, T. G., de Melker, H. E., & Kok, G. (2013). Why parents refuse childhood vaccination: a qualitative study using online focus groups. BMC public health, 13, 1183. https://doi.org/10.1186/1471-2458-13-1183

Heininger U. (2006). An internet-based survey on parental attitudes towards immunization. Vaccine, 24(37-39), 6351–6355. https://doi.org/10.1016/j.vaccine.2006.05.029

Himelboim, I., Xiao, X., Lee, D., Wang, M. Y., & Borah, P. (2020). A Social Networks Approach to Understanding Vaccine Conversations on Twitter: Network Clusters, Sentiment, and Certainty in HPV Social Networks. Health communication, 35(5), 607–615. https://doi.org/10.1080/10410236.2019.1573446

Ibrahim, M. N. M., & Yusoff, M. Z. M. (2015). Twitter sentiment classification using Naive Bayes based on trainer perception. 2015 IEEE Conference on E-Learning, e-Management and e-Services (IC3e), 187–189. https://doi.org/10.1109/IC3e.2015.7403510


Jarrett, C., Wilson, R., O'Leary, M., Eckersberger, E., Larson, H. J., & SAGE Working Group on Vaccine Hesitancy (2015). Strategies for addressing vaccine hesitancy - A systematic review. Vaccine, 33(34), 4180–4190. https://doi.org/10.1016/j.vaccine.2015.04.040

Jünger, J., & Keyling, T. (2019). Facepager. An application for automated data retrieval on the web.

Kabir, A. I., Ahmed, K., & Karim, R. (2020). Word Cloud and Sentiment Analysis of Amazon Earphones Reviews with R Programming Language. Informatica Economica, 24(4), 55–71. https://doi.org/10.24818/issn14531305/24.4.2020.05

Kang, G. J., Culp, R. K., & Abbas, K. M. (2017). Facilitators and barriers of parental attitudes and beliefs toward school-located influenza vaccination in the United States: Systematic review. Vaccine, 35(16), 1987–1995. https://doi.org/10.1016/j.vaccine.2017.03.014

Kang, G. J., Ewing-Nelson, S. R., Mackey, L., Schlitt, J. T., Marathe, A., Abbas, K. M., & Swarup, S. (2017). Semantic network analysis of vaccine sentiment in online social media. Vaccine, 35(29), 3621– 3638. https://doi.org/10.1016/j.vaccine.2017.05.052


Kao, A., & Poteet, S. R. (2007). Natural language processing and text mining. Springer Science & Business Media.

Kim, H.-S. (2017). A semantic network analysis of big data regarding food exhibition at convention center. Culinary Science and Hospitality Research, 23(3), 257–270.https://doi.org/10.20878/cshr.2019.25.2.014

Kim, M., Choi, M., & Youm, Y. (2017). Journal of Korean Academy of Nursing, 47(6), 806–816. https://doi.org/10.4040/jkan.2017.47.6.806

Klimiuk, K., Czoska, A., Biernacka, K., & Balwicki, Ł. (2021). Vaccine misinformation on social media - topic-based content and sentiment analysis of Polish vaccine-deniers' comments on Facebook. Human vaccines & immunotherapeutics, 17(7), 2026–2035. https://doi.org/10.1080/21645515.2020.1850072

Larson, H. J., Cooper, L. Z., Eskola, J., Katz, S. L., & Ratzan, S. (2011). Addressing the vaccine confidence gap. Lancet (London, England), 378(9790), 526–535. https://doi.org/10.1016/S0140- 6736(11)60678-8

Leask, J., Chapman, S., Hawe, P., & Burgess, M. (2006). What maintains parental support for vaccination when challenged by anti-vaccination messages? A qualitative study. Vaccine, 24(49-50), 7238–7245. https://doi.org/10.1016/j.vaccine.2006.05.010

Lim, K. K., Chan, Y. Y., Noor Ani, A., Rohani, J., Siti Norfadhilah, Z. A., & Santhi, M. R. (2017). Complete immunization coverage and its determinants among children in Malaysia: findings from the National Health and Morbidity Survey (NHMS) 2016. Public health, 153, 52–57.

https://doi.org/10.1016/j.puhe.2017.08.001

Liu, W., Lai, C.-H., & Xu, W. W. (2018). Tweeting about emergency: A semantic network analysis of government organizations’ social media messaging during hurricane Harvey. Public Relations Review, 44(5), 807–819. https://doi.org/10.1016/j.pubrev.2018.10.009

Lyu, J. C., Han, E. L., & Luli, G. K. (2021). COVID-19 Vaccine-Related Discussion on Twitter: Topic Modeling and Sentiment Analysis. Journal of medical Internet research, 23(6), e24435. https://doi.org/10.2196/24435

Macdonald, G. J., McIntyre, M. A., & Barry, M. A. (2014). Immunizing Children: Current Canadian Health Professional Competencies. SAGE Open. https://doi.org/10.1177/2158244014559510

Nasukawa, T., & Yi, J. (2003). Sentiment analysis: Capturing favorability using natural language processing. Proceedings of the 2nd International Conference on Knowledge Capture, 70–77. https://doi.org/10.1145/945645.945658

Nielsen, F. \AArup. (2017). Afinn project.

Nieminen, U. J. (1973). On the centrality in a directed graph. Social Science Research, 2(4), 371–378. https://doi.org/10.1016/0049-089X(73)90010-0

Oh, M., & Kim, S. (2020). Dimensionality of ethnic food fine dining experience: An application of semantic network analysis. Tourism Management Perspectives, 35, image100719]. https://doi.org/10.1016/j.tmp.2020.100719

On, J., Park, H. A., & Song, T. M. (2019). Sentiment Analysis of Social Media on Childhood Vaccination: Development of an Ontology. Journal of medical Internet research, 21(6), e13456. https://doi.org/10.2196/13456

Peretti-Watel, P., Larson, H. J., Ward, J. K., Schulz, W. S., & Verger, P. (2015). Vaccine hesitancy: clarifying a theoretical framework for an ambiguous notion. PLoS currents, 7, ecurrents.outbreaks.6844c80ff9f5b273f34c91f71b7fc289. https://doi.org/10.1371/currents.outbreaks.6844c80ff9f5b273f34c91f71b7fc289

Piedrahita-Valdés, H., Piedrahita-Castillo, D., Bermejo-Higuera, J., Guillem-Saiz, P., Bermejo- Higuera, J. R., Guillem-Saiz, J., Sicilia-Montalvo, J. A., & Machío-Regidor, F. (2021). Vaccine Hesitancy on Social Media: Sentiment Analysis from June 2011 to April 2019. Vaccines, 9(1), 28. https://doi.org/10.3390/vaccines9010028

Poecze, F., Ebster, C., & Strauss, C. (2022). Let’s play on Facebook: Using sentiment analysis and social media metrics to measure the success of YouTube gamers’ post types. Personal and Ubiquitous Computing,26, 901–910. https://doi.org/10.1007/s00779-019-01361-7

Rashid, A., Anwer, N., Iqbal, M., & Sher, M. (2013). A survey paper: Areas, techniques and challenges of opinion mining. International Journal of Computer Science Issues (IJCSI), 10(6), 18.

Ravindran, S. K., & Garg, V. (2015). Mastering social media mining with R. Packt Publishing Ltd.

Ruiz, J., Featherstone, J. D., & Barnett, G. A. (2021). Identifying Vaccine Hesitant Communities on Twitter and their Geolocations: A Network Approach. Proceedings of the 54th Hawaii International Conference on System Sciences, 3964.

Rumetta, J., Abdul-Hadi, H., & Lee, Y. K. (2020). A qualitative study on parents' reasons and recommendations for childhood vaccination refusal in Malaysia. Journal of infection and public health, 13(2), 199–203. https://doi.org/10.1016/j.jiph.2019.07.027

Sabidussi, G. (1966). The centrality index of a graph. Psychometrika, 31(4), 581–603. https://doi.org/10.1007/BF02289527

Shaw, M. E. (1954). Group structure and the behavior of individuals in small groups. The Journal of Psychology, 38(1), 139–149. https://doi.org/10.1080/00223980.1954.9712925

Sobo, E. J. (2015). Social cultivation of vaccine refusal and delay among Waldorf (Steiner) school parents. Medical Anthropology Quarterly, 29(3), 381–399. https://doi.org/10.1111/maq.12214

Tan, Y., Lam, H., Azlan, A., & Soo, W. (2016). Sentiment Analysis for Telco Popularity on Twitter Big Data Using a Novel Malaysian Dictionary. ICADIWT.

Thovex, C., & Trichet, F. (2013). Semantic social networks analysis. Social Network Analysis and Mining, 3(1), 35–49. https://doi.org/10.1007/s13278-012-0055-y

Tombs-Heirman, E. (2009). Understanding vaccination refusal: a qualitative study of parents' health beliefs and practices.

Wang, Q., Zhang, W., Cai, H., & Cao, Y. (2020). Understanding the perceptions of Chinese women of the commercially available domestic and imported HPV vaccine: A semantic network analysis. Vaccine, 38(52), 8334–8342. https://doi.org/10.1016/j.vaccine.2020.11.016

Xiong, Y., Cho, M., & Boatwright, B. (2019). Hashtag activism and message frames among social movement organizations: Semantic network analysis and thematic analysis of Twitter during the# MeToo movement. Public Relations Review, 45(1), 10–23. https://doi.org/10.1016/j.pubrev.2018.10.014

Yoo, M., Lee, S., & Ha, T. (2019). Semantic network analysis for understanding user experiences of bipolar and depressive disorders on Reddit. Information Processing & Management, 56(4), 1565–1575. https://doi.org/10.1016/j.ipm.2018.10.001

Zipprich, J., Winter, K., Hacker, J., Xia, D., Watt, J., & Harriman, K. (2015). Measles outbreak— California, december 2014–february 2015. MMWR. Morbidity and Mortality Weekly Report, 64(6), 153.