A DOMAIN-SPECIFIC EVALUATION OF THE PERFORMANCE OF SELECTED WEB-BASED SENTIMENT ANALYSIS PLATFORMS
DOI:
https://doi.org/10.15282/ijsecs.9.1.2023.1.0105Keywords:
Sentiment analysis, Sentiment polarity, Polarity identification, Analysis-as-a-service, Text classificationAbstract
There is now an increasing number of sentiment analysis software-as-a-service (SA-SaaS) offerings in the market. Approaches to sentiment analysis and their implementation as SA-SaaS vary, and there really is no sure way of knowing what SA-SaaS uses which approach. For potential users, SA-SaaS products are black boxes. Black boxes, however, can be evaluated using a set of standard input and a comparison of the output. Using a test data set drawn from human annotated samples in existing studies covering sentiment polarity of news headlines, this study compares the performance of selected popular and free (or at least free-to-try) SA-SaaS in terms of the accuracy, precision, recall and specificity of the sentiment classification using the black box testing methodology. SentiStrength, developed at the University of Wolverhampton in the UK, emerged as consistent performer across all metrics.
References
S. Mohammad, B. Dorr, and C. Dunne, “Generating high-coverage semantic orientation lexicons from overtly marked words
and a thesaurus,” Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP-2009):
Singapore; pp. 599-608, 2009, https://aclanthology.org/D09-1063.pdf.
B. Liu, “Sentiment Analysis and Subjectivity,” Handbook of Natural Language Processing 5, 1–38, 2010,
https://www.cs.uic.edu/~liub/FBS/NLP-handbook-sentiment-analysis.pdf.
A. Esuli and F. Sebastiani, “SentiWordNet: A publicly available lexical resource for opinion mining,” Proceedings of 5th
International Conference on Language Resources and Evaluation (LREC): Genoa, Italy; pp. 417-22, 2006.
Y. Mejova, “Sentiment Analysis: An Overview,” unpublished comprehensive exam paper at the Computer Science
Department, University of Iowa, 2009, https://www.academia.edu/291678/Sentiment_Analysis_An_Overview.
F. Benamara, C. Cesarano, A. Picariello, D. Reforgiato, and V. Subrahmanian, “Sentiment analysis: Adjectives and adverbs
are better than adjectives alone,” Proceedings of International Conference on Weblogs and Social Media, ICWSM. Boulder,
CO, 2007.
Y. Lu, X. Kong, X. Quan, W. Liu and Y. Xu, “Exploring the Sentiment Strength of User Reviews,” L. Chen, C. Tang, J. Yang,
and Y. Gao (eds), Web-Age Information Management, WAIM 2010, Lecture Notes in Computer Science, Vol. 6184, 2010,
Springer, Berlin, Heidelberg, doi:10.1007/978-3-642-14246-8_46.
M. Z. Ansari, M. B. Aziz, M. O. Siddiqui, H. Mehra, and K. P. Singh, “Analysis of Political Sentiment Orientations on
Twitter,” Procedia Computer Science, 167: 1821-1828, 2020, doi: 10.1016/j.procs.2020.03.201.
M. Bansal, C. Cardie, and L. Lee, “The power of negative thinking: Exploiting label disagreement in the min-cut classification
framework,” Proceedings of the International Conference in Computational Linguistics (COLING): Manchester UK; pp. 15-
, 2008, https://aclanthology.org/C08-2004.pdf.
N. Diakopoulos, “Algorithmic accountability reporting: On the investigation of black boxes,” A Tow/Knight Brief. Tow Center
for Digital Journalism, Columbia Journalism School, Columbia University, 2014, doi: 10.7916/D8ZK5TW2
J. Mihaljević, J., “Analysis and Creation of Free SentimentAnalysis Programs,” Croatian Journal for Journalism and the
Media, 25(1): 83-104, 2019, doi: 10.22572/mi.25.1.4.
J. Serrano-Guerrero, J. A. Olivas, F. P. Romero, and E. Herrera-Viedma, “Sentiment analysis: A review and comparative
analysis of web services,” Information Sciences, 311: pp. 18-38, 2015, doi: 10.1016/j.ins.2015.03.040.
M. Thelwall, K. Buckley, G. Paltoglou, and D. Cai, “Sentiment strength detection in short informal text,” Journal of the
American Society for Information Science and Technology, 61(12), 2544–2558, 2010,
http://www.scit.wlv.ac.uk/~cm1993/papers/SentiStrengthPreprint.doc. [14] J. Reis, F. Benevenuto, P. O. S. Vaz de
Melo, R. Prates, H. Kwak, and J. An, “Breaking the News: First Impressions Matter on Online News,” The International AAAI
Conference on Web and Social Media (ICWSM), 2015,
S. Lawrie, “Newspaper coverage of psychiatric and physical illness,” The Psychiatrist, 24:104-106, 2000,
J. Reis, F. Benevenuto, P. O. S. Vaz de Melo, R. Prates, H. Kwak, and J. An, “Breaking the News: First Impressions Matter
on Online News,” The International AAAI Conference on Web and Social Media (ICWSM), 2015,
H. Nisar and E. Bleich, “Group status, geographic location, and the tone of media coverage: Jews and Muslims in New York
Times and Guardian Headlines, 1985–2014,” Comparative Migration Study, 8(3), 2020, doi:10.1186/s40878-019-0153-3.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2022 Manuel O. Diaz Jr.
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
