COMPARATIVE ANALYSIS OF BACK-TRANSLATION MODELS FOR NORMALIZATION MOBILE APP USER REVIEWS

Authors

  • Amran Salleh Faculty of Computer Science and Information Technology, Universiti Putra Malaysia, Jalan Universiti 1, 43400 Serdang, Selangor, Malaysia
    • Mohd Hafeez Osman Faculty of Computer Science and Information Technology, Universiti Putra Malaysia, Jalan Universiti 1, 43400 Serdang, Selangor, Malaysia
      • Sa’adah Hassan Faculty of Computer Science and Information Technology, Universiti Putra Malaysia, Jalan Universiti 1, 43400 Serdang, Selangor, Malaysia
        • Mar Yah Said Faculty of Computer Science and Information Technology, Universiti Putra Malaysia, Jalan Universiti 1, 43400 Serdang, Selangor, Malaysia

          DOI:

          https://doi.org/10.15282/ijsecs.11.2.2025.10.0142

          Keywords:

          Back-translation, Mobile app reviews, Machine learning, Natural language processing, Evaluation metrics

          Abstract

          The increase in mobile apps has led to an exponential growth of user-generated reviews, which are often noisy, informal, and linguistically diverse, thereby posing significant challenges for automated analysis in requirements engineering. This study evaluates whether back-translation (BT) can normalize informal reviews while preserving meaning, and which model (Google Translate vs Facebook M2M100_418M) offers better semantic preservation, grammatical quality, and lexical alignment. We collected 323 Google Play reviews (667 sentences) from three Malaysian government apps. Texts were cleaned, expanded for colloquial forms, and then BT was applied using Malay as an intermediate language. Evaluation used four metrics which are semantic similarity (Sentence-BERT), grammar error count (LanguageTool), BLEU (NLTK), and perplexity (GPT-2). Model differences were tested with paired t-tests and Wilcoxon signed- rank tests, while paired scatterplots showed distributional patterns. Google was significantly better on semantic similarity (t(322)=5.38, p<.001), grammar errors (t(322)=3.66, p<.001), and BLEU (t(322)=2.99, p=.003); effect sizes were small to moderate. Perplexity differences were not significant, indicating comparable sentence-level fluency. Visualizations confirmed Google’s steadier performance with fewer extreme outliers. BT is a practical normalization step for noisy reviews. For the English–Malay pipeline studied here, Google provides more reliable semantic preservation and grammatical quality, while both systems are similar in fluency. However, the generalizability of these results is constrained by the relatively modest sample size (323 reviews, 667 sentences), and future work should validate results on large datasets and explore hybrid strategies combining strengths of both models.

          Downloads

          Download data is not yet available.

          References

          [1] U. Naseem, S. K. Khan, I. Razzak, and I. A. Hameed, “Hybrid words representation for airlines sentiment analysis,” in Proceedings of the Australasian Joint Conference on Artificial Intelligence, Cham, Switzerland: Springer International Publishing, 2019, pp. 381–392.

          [2] N. Jha and A. Mahmoud, “Using frame semantics for classifying and summarizing application store reviews,” Empirical Software Engineering, vol. 23, no. 6, pp. 3734–3767, 2018, https://doi.org/10.1007/s10664-018-9605-x.

          [3] G. B. Shermamatova, Z. A. Shermamatova, A. B. Baxronov, and O. V. Roziqov, “Usage of colloquial style,” Mental Enlightenment Scientific-Methodological Journal, vol. 5, no. 5, pp. 278–286, 2024.

          [4] A. Y. Al-Obaidi and V. W. Samawi, “Opinion mining: Analysis of comments written in Arabic colloquial,” in Proceedings of the World Congress on Engineering and Computer Science, vol. 1, 2016.

          [5] N. Genc-Nayebi and A. Abran, “A systematic literature review: Opinion mining studies from mobile app store user reviews,” Journal of Systems and Software, vol. 125, pp. 207–219, 2017, https://doi.org/10.1016/j.jss.2016.11.027.

          [6] S. Y. Eftee, M. Y. Khan, R. Noor, H. Mahmud, and M. K. Hasan, “Extraction of app problems and its corresponding user action from user review of apps using few-shot learning,” SSRN, [Online]. Available: https://ssrn.com/abstract=5093717.

          [7] M. D. Szeto, C. Barber, V. K. Ranpariya, et al., “Emojis and emoticons in health care and dermatology communication: Narrative review,” JMIR Dermatology, vol. 5, no. 3, Art. no. e33851, 2022, https://doi.org/10.2196/33851.

          [8] A. Aslam and B. Hussian, “Emotion recognition techniques with rule-based and machine learning approaches,” arXiv, arXiv:2103.00658, 2021. [Online]. Available: https://arxiv.org/abs/2103.00658.

          [9] B. Noori, “Classification of customer reviews using machine learning algorithms,” Applied Artificial Intelligence, vol. 35, no. 8, pp. 567–588, 2021.

          [10] J. Sangeetha and U. Kumaran, “Sentiment analysis of Amazon user reviews using a hybrid approach,” Measurement: Sensors, vol. 27, Art. no. 100790, 2023.

          [11] M. Ahmad, S. Aftab, I. Ali, and N. Hameed, “Hybrid tools and techniques for sentiment analysis: A review,” International Journal of Multidisciplinary Science and Engineering, vol. 8, no. 3, pp. 29–33, 2017.

          [12] I. M. Subedi, M. Singh, V. Ramasamy, and G. S. Walia, “Application of back-translation: A transfer learning approach to identify ambiguous software requirements,” in Proceedings of the 2021 ACM Southeast Conference, 2021, pp. 130–137.

          [13] A. Fan, S. Bhosale, H. Schwenk, et al., “Beyond English-centric multilingual machine translation,” Journal of Machine Learning Research, vol. 22, no. 107, pp. 1–48, 2021.

          [14] E. Chew, M. Chakraborti, W. Weisman, and S. Frey, “Machine translation for accessible multi-language text analysis,” Computational Communication Research, vol. 7, no. 1, p. 1, 2025.

          [15] A. Mukherjee and M. Shrivastava, “Lost in translation? Found in evaluation: A comprehensive survey on sentence-level translation evaluation,” ACM Computing Surveys, 2025.

          [16] N. Mathur, T. Baldwin, and T. Cohn, “Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics,” arXiv, arXiv:2006.06264, 2020. [Online]. Available: https://arxiv.org/abs/2006.06264.

          [17] A. C. Klotz, B. W. Swider, and S. H. Kwon, “Back-translation practices in organizational research: Avoiding loss in translation,” Journal of Applied Psychology, vol. 108, no. 5, p. 699, 2023.

          [18] S. Poh, S. J. Yang, J. Tan, et al., “MalayMMLU: A multitask benchmark for the low-resource Malay language,” in Findings of the Association for Computational Linguistics: EMNLP 2024, 2024, pp. 650–669.

          [19] J. Ma and L. Li, “Data augmentation for Chinese text classification using back-translation,” Journal of Physics: Conference Series, vol. 1651, no. 1, Art. no. 012039, 2020.

          [20] K. Papineni, S. Roukos, T. Ward, and W. J. Zhu, “BLEU: A method for automatic evaluation of machine translation,” in Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, PA, USA, 2002, pp. 311–318, https://doi.org/10.3115/1073083.1073135.

          [21] J. Lee, J. Kim, and P. Kang, “Back-translated task adaptive pretraining: Improving accuracy and robustness on text classification,” arXiv, arXiv:2107.10474, 2021. [Online]. Available: https://arxiv.org/abs/2107.10474.

          [22] S. Edunov, M. Ott, M. Auli, and D. Grangier, “Understanding back-translation at scale,” arXiv, arXiv:1808.09381, 2018. [Online]. Available: https://arxiv.org/abs/1808.09381.

          [23] Y. Tong, Y. Chen, G. Zhang, J. Zheng, H. Zhu, and X. Shi, “Generating diverse back-translations via constraint random decoding,” in Proceedings of the China Conference on Machine Translation, Singapore: Springer, 2021, pp. 92–104.

          [24] B. Chen, J. Golebiowski, and Z. Abedjan, “Data augmentation for supervised code translation learning,” in Proceedings of the 21st International Conference on Mining Software Repositories, 2024, pp. 444–456.

          [25] J. A. Prenner and R. Robbes, “Making the most of small software engineering datasets with modern machine learning,” IEEE Transactions on Software Engineering, vol. 48, no. 12, pp. 5050–5067, 2022.

          [26] L. Brigato and L. Iocchi, “A close look at deep learning with small data,” in Proceedings of the 25th International Conference on Pattern Recognition, IEEE, 2021, pp. 2490–2497.

          [27] R. Shadman, M. S. Murshed, E. Verenich, A. Velasquez, and F. Hussain, “The utility of feature reuse: Transfer learning in data-starved regimes,” in Proceedings of the 2023 International Conference on Computational Science and Computational Intelligence (CSCI), IEEE, 2023, pp. 37–42.

          [28] I. Feldman and R. Coto-Solano, “Neural machine translation models with back-translation for the extremely low-resource indigenous language Bribri,” in Proceedings of the 28th International Conference on Computational Linguistics, 2020, pp. 3965–3976.

          [29] P. McNamee and K. Duh, “An extensive exploration of back-translation in 60 languages,” in Findings of the Association for Computational Linguistics: ACL 2023, 2023, pp. 8166–8183.

          [30] S. Liu and W. Zhu, “An analysis of the evaluation of the translation quality of neural machine translation application systems,” Applied Artificial Intelligence, vol. 37, no. 1, Art. no. 2214460, 2023.

          [31] J. Song, H. Zan, T. Liu, K. Zhang, X. Ji, and T. Cui, “Text classification based on multilingual back-translation and model ensemble,” in Proceedings of the China Health Information Processing Conference, Singapore: Springer, 2023, pp. 231–241.

          [32] A. Yadav, A. Patel, and M. Shah, “A comprehensive review on resolving ambiguities in natural language processing,” AI Open, vol. 2, pp. 85–92, 2021.

          [33] M. K. M. Boussougou, P. Hamandawana, and D. J. Park, “Enhancing voice phishing detection using multilingual back-translation and SMOTE: An empirical study,” IEEE Access, 2025.

          [34] Q. M. Do, K. Zeng, and I. Paik, “Resolving lexical ambiguity in English-Japanese neural machine translation,” in Proceedings of the 3rd Artificial Intelligence and Cloud Computing Conference, 2020, pp. 46–51.

          [36] H.-Q. Nguyen-Son, T. T. Phuong, S. Hidano, and S. Kiyomoto, “Detecting machine-translated text using back translation,” in Proceedings of the 12th International Conference on Natural Language Generation, Tokyo, Japan, 2019, pp. 189–197, https://doi.org/10.18653/v1/W19-8626.

          [37] M. Benmansour and Y. Hdouch, “The role of the latest technologies in the translation industry,” Emirati Journal of Education and Literatures, vol. 1, no. 2, pp. 31–36, 2023.

          [38] S. Castilho, J. Moorkens, F. Gaspari, I. Calixto, J. Tinsley, and A. Way, “Is neural machine translation the new state of the art?” The Prague Bulletin of Mathematical Linguistics, no. 108, 2017.

          Published

          2025-12-18

          How to Cite

          [1]
          amran salleh, Mohd Hafeez Osman, Sa’adah Hassan, and Mar Yah Said, “COMPARATIVE ANALYSIS OF BACK-TRANSLATION MODELS FOR NORMALIZATION MOBILE APP USER REVIEWS”, IJSECS, vol. 11, no. 2, pp. 124–136, Dec. 2025, doi: 10.15282/ijsecs.11.2.2025.10.0142.

          Share