Computers, Materials & Continua DOI:10.32604/cmc.2022.029507 | |
Article |
A Structural Topic Model for Exploring User Satisfaction with Mobile Payments
1Department of Human-Artificial Intelligence Interaction, Sungkyunkwan University, Seoul, 03063, Korea
2Department of Interaction Science, Sungkyunkwan University, Seoul, 03063, Korea
3Department of Applied Artificial Intelligence, Sungkyunkwan University, Seoul, 03063, Korea
4NAVER Green Factory, Seongnam, 13561, Korea
*Corresponding Author: Dongyan Nan. Email: ndyzxy0926@skku.edu
Received: 05 March 2022; Accepted: 28 April 2022
Abstract: This study explored user satisfaction with mobile payments by applying a novel structural topic model. Specifically, we collected 17,927 online reviews of a specific mobile payment (i.e., PayPal). Then, we employed a structural topic model to investigate the relationship between the attributes extracted from online reviews and user satisfaction with mobile payment. Consequently, we discovered that “lack of reliability” and “poor customer service” tend to appear in negative reviews. Whereas, the terms “convenience,” “user-friendly interface,” “simple process,” and “secure system” tend to appear in positive reviews. On the basis of information system success theory, we categorized the topics “convenience,” “user-friendly interface,” and “simple process,” as system quality. In addition, “poor customer service” was categorized as service quality. Furthermore, based on the previous studies of trust and security, “lack of reliability” and “secure system” were categorized as trust and security, respectively. These outcomes indicate that users are satisfied when they perceive that system quality and security of specific mobile payments are great. On the contrary, users are dissatisfied when they feel that service quality and reliability of specific mobile payments is lacking. Overall, our research implies that a novel structural topic model is an effective method to explore mobile payment user experience.
Keywords: Mobile payment; user satisfaction; online review; structural topic model
Owing to the growing availability of high-speed mobile data networks and smartphones, the use of mobile payments (m-payments) is increasing. According to a report from Allied Market Research [1], the global m-payment industry was valued at $1.48 trillion in 2019 and is expected to expand at a compound annual growth rate of 30.1 percent from 2020 to 2027, reaching $12.06 trillion by 2027. As of 2021, the m-payment market took a huge portion in the consumer market. According to a study from Cornerstone Advisors, over eight out of ten smartphone owners have at least one m-payment app on their smartphone, and the PayPal app has been installed on roughly two-thirds of all smartphones [2].
Furthermore, the COVID-19 pandemic has also spurred the global growth of the m-payment market . Contact-free society services recommended by the World Health Organization positively impacted the m-payment market [3]. The year 2020 was the first year many consumers had utilized contactless payment methods, and m-payments surpassed cash payments globally [4]. Furthermore, in April 2020, Organization for Economic Co-operation and Development published a document for taxpayers, enhancing services in the m-payment system of taxes and government bills [5].
User satisfaction is one of the primary elements contributing to marketing success in competitive markets [6–8]. Several studies have explored which factors lead to higher user satisfaction with m-payments [9–17]. They investigated user evaluations that may link to higher user satisfaction with m-payments via analyzing a limited number of samples (less than 1000). Our research addressed this limitation of existing studies by investigating the factors affecting m-payment user satisfaction by employing large-scale online review data.
We applied the topic model approach to analyze the data; in particular, we used the structural topic model (STM) [18]. STM is an extension of Latent Dirichlet Allocation (LDA), the most prominent topic model [19]. We applied STM in our study despite LDA [20] being one of the most widely employed text analysis techniques for analyzing large-scale online reviews of specific services or products to investigate individual experiences [21,22]. This was due to the following reasons. Roberts et al. [23] and Kim et al. [24] indicated that it is difficult to investigate the association between document metadata and document content utilizing LDA because LDA lacks additional document-level information. In other words, analyzing the association between review content and review rating (i.e., user satisfaction) with LDA is challenging. Countering the limitation of LDA, Hu et al. [25] suggested that STM can be utilized to effectively explore the connection between comment content and satisfaction. Although few studies have utilized STM to analyze online reviews in the context of m-payment user experience, the method has been applied to analyze user-generated text data (e.g., online review) in the field of hospitality [19,25,26]. Hence, we propose that using STM to analyze large-scale online reviews is useful for exploring m-payment user experience and satisfaction.
Overall, our research is one of the first studies to explore user assessments of m-payments by computing a large-scale dataset with STM. We believe that our results will provide a deeper understanding of user experience and satisfaction in the context of m-payments.
2.1 User Experience, Online Review, and STM
Goodman et al. [27] pointed out that exploring user experience is essential to offering successful services. Moreover, Kim et al. [6] suggested that computing such user-generated data as online reviews is an effective way to investigate the user experience of specific services. This is because online reviews are relatively easy to access and contain a variety of user perceptions and emotions regarding a particular service [6,28].
Some scholars have also utilized topic model approaches such as STM to explore user experience and enhance user satisfaction. Hu et al. [25] performed STM analysis on 27,864 New York hotel reviews and discovered that “severe service failure” plays a notable role in user dissatisfaction. Following the STM results of 242,020 Malaysia Airbnb reviews, Ding et al. [19] found that international users care more about the possibility of “group stay” in hotels than Malaysian users. He et al. [29] utilized STM to analyze 19,054 online drug reviews from JD.com and found that “expiration date,” “mailing service,” and “after-sales service” play notable roles in influencing user satisfaction.
These studies imply that performing STM on a large-scale of online reviews is an effective method to track user experience and satisfaction with specific products and services.
2.2 Information System Success Theory (ISST)
ISST, developed by DeLone et al. [30], is one of the representative theories for investigating user satisfaction with specific information systems and services [31–33]. As indicated in ISST, user satisfaction is notably influenced by service, system, and information qualities [30].
ISST has also been validated in the field of m-payments. For instance, based on the results from 195 samples, Zhou [9] concluded that there is a significant connection between the system quality and satisfaction with m-payment. Yuan et al. [17] examined the data from 343 Chinese consumers and demonstrated that m-payment satisfaction is notably affected by service, system, and information qualities. Lin et al. [11] proved that both information quality and system quality positively influence Chinese and Korean user satisfaction with m-payments.
According to the abovementioned studies, we find that user satisfaction with m-payments is closely related to ISST constructs. Furthermore, Teng et al. [34] demonstrated that ISST-related attributes can be extracted via the text analysis of online reviews of specific m-payments. Hence, we infer that ISST-related dimensions can be extracted via STM analysis of m-payment online reviews.
As demonstrated by previous studies [35–37], perceived security and trust notably influence individual assessments of certain technological services. In particular, with respect to financial technology (fintech)-based services, trust and security dimensions should be considered when tracking user experience or satisfaction [38,39]. This is because fintech services are highly related to personal assets [36], and fintech service users need to provide the service providers with more personal information, compared to users of other services [40].
Therefore, prior studies have demonstrated that perceived security and trust impact m-payment user satisfaction. For instance, by computing survey-based data from 205 Koreans, Nan et al. [14] concluded that perceived security leads to greater user satisfaction with m-payments. Based on the outcomes from 243 samples, Chen et al. [15] reported that Chinese users are satisfied with m-payment services when they perceive the service providers to be trustworthy. Gupta et al. [41] examined 716 samples in India and concluded that higher perceived security results in greater m-payment user satisfaction. Cao et al. [10] indicated that m-payment user satisfaction can be increased by perceived trust based on results from 219 samples. These studies also implied that individuals are dissatisfied with specific m-payment services when they feel that the services are not secure, and the service providers are not trustworthy.
Overall, based on abovementioned arguments, we indicated that both trust and security have robust connections with m-payment user satisfaction and experience. Consequently, we deduced that attributes related to trust and security can also be extracted by analyzing m-payment user reviews.
Our study focused on PayPal, one of the most popular m-payment systems, with a total of 403 million active user accounts from Q1 2010 to Q2 2021 [42]. We collected all the “most relevant” Google Play Store reviews of the “PayPal” mobile application. The corresponding data contain 17,927 comments with star ratings from October 10, 2018 to June 9, 2021.
Based on the guidelines of Ho-Dac et al. [43], Babić Rosario et al. [44], and Hu et al. [25], we considered ratings of 1 and 2 to be negative reviews (37.39%) and ratings of 4 and 5 to be positive reviews (56.52%); only the positive and negative reviews were used for the analysis. Data preprocessing started with deleting all punctuations and numbers, word normalization (i.e., transforming all letters to lower case), and tokenizing sentences into each individual words. Then we removed stop words, such as “is,” “at,” and “and,” and the custom user-defined stop words (e.g., PayPal). Next, we did lemmatization using the spaCy package, which has been widely employed in previous studies [45,46]. The word lemmatization takes into consideration the morphological analysis of the words (e.g., “studying,” “studies,” and “studied” all automates into study). Finally, we built bigram and trigram models to combine words like “Western” and “Union” into “Western_Union.” This has been done to prevent every word from being considered individually (e.g., “Western_Union” is considered to be one word).
Our study aimed to identify the themes of both the positive and negative reviews as well as to discover their proportions and which topics are shown from each side. Therefore, our modeling was based on the extremity of reviews and its influence on the topical parameter μ of STM. We used positive negative ratings where 1-and 2-star ratings are considered as Negative and 4-and 5-star ratings are considered as Positive. The ratings are the metadata for our model. Finally, this parameter affects the proportions of the document and the topic θ. The introduction of STM is shown in Fig. 1.
STM draws document-level attention to each topic from a logistic-normal generalized linear model based on a vector of document covariates Xd [48]. It is a unique function of the STM model; in our research, we defined one covariate, the document metadata, and pos_neg, which indicate the Positive and Negative tags for each document in the data set. Positive reviews are tagged as Positive, and Negative reviews are tagged as Negative.
3.4 Estimation of Topic Number
According to the suggestions of Kuhn [49], Hu et al. [25], and Ding et al. [19], our estimation of topic number was based on the semantic coherence and exclusivity of the topic model. A relatively larger value of semantic coherence indicates that the most frequently used words in a given topic are likely to frequently appear together [19]. A higher value of exclusivity reveals that the high-probability words in a topic are less likely to appear in other topics [50]. Generally, as the topic number increases, semantic coherence decreases, but exclusivity is enhanced [49]. Furthermore, given that models with good statistical outcomes are likely to have poor interpretability [51], Ding et al. [19] suggested that in addition to a statistical approach, the interpretability of the top words of each topic should be qualitatively verified. Therefore, by comparing the coherence and exclusivity of a series of models with a different number of topics, and by qualitatively verifying the relevance of the topic content, a 6-topic model was chosen for our research.
The STM outcomes are reported in Tab. 1. The frequency-exclusivity (FREX) words were employed for topic labeling because FREX statistic generates words by the overall frequency of words and the exclusivity level to the topic; this can offer a more semantically intuitive representation of the topic [18,19]. Each topic was manually labeled based on the suggestions of two m-payment experts. Afterwards, we checked a series of reviews in each topic to validate the fit of suggested topic names [19].
Based on the measurements, which have been validated in representative user-oriented studies, the six topics were assigned as follows.
According to Fang et al. [52], Iivari [53], Teng et al. [34], and DeLone et al. [30], system quality can be measured by convenience, user-friendly interface, and simplicity. Thus, the topics “convenience of use,” “simple process,” and “user-friendly interface” are categorized as system quality. Following DeLone et al. [54] and Teng et al. [34], “poor customer service” is assigned to service quality. Moreover, based on Roca et al. [35] and Nan et al. [14], “secure system” is classified as perceived security. Furthermore, given that the reliability of a system can be employed to measure perceived trust [55], we categorize “lack of reliability” as perceived trust.
Fig. 2 visualizes the estimated changes in the theme proportions when the number of positive reviews shifts from the number of negative reviews and vice versa. Each point indicates the mean values of the estimated differences, and the bars are 95% confidence intervals for the difference. For example, the proportion of “poor customer service” in the negative reviews is 5.79% higher than in the positive reviews, while the 95% confidence interval of this difference is [–5.58%, −6.00%]. Hence, we have identified “poor customer service” as a “negative” theme.
Our study aims to track user satisfaction with a specific m-payment (i.e., PayPal) by applying a novel STM analysis to large-scale online reviews, i.e., our research focused on exploring the relationships among the attributes extracted from m-payment online reviews and user satisfaction. Considering that online reviews are easy to acquire and STM can process such a large amount of text data, the approach employed in our research reduces financial and time costs [6] compared to survey methodologies that have been widely utilized to investigate user experience [56–58]. Based on the outcomes, several theoretical and practical implications are reported as follows.
5.1 Theoretical Interpretations
First, by employing STM, our study successfully extracted the attributes of service quality and system quality from m-payment online reviews. Notably, information quality attributes were not extracted. These outcomes may mean that system and service qualities are more significant than information quality with respect to m-payment user experience. This finding is also supported by Cui et al. [58]. They demonstrated a phenomenon that causes users to not care about information quality when utilizing a specific service.
Second, security and trust attributes were also identified from the STM results. These results are in line with those of Khalilzadeh et al. [38]; they argued that trust and security are significant elements in influencing user perception of m-payments. In addition, these outcomes imply that perceived trust and security should be considered when exploring m-payment user experience.
Third, it was found that “lack of reliability” is more likely to appear in the negative reviews. It indicates that users negatively assess m-payments when they consider m-payment service providers as untrustworthy. This outcome is in line with the findings of Chen et al. [15] and Cao et al. [10]; they demonstrated the notable association between trust and satisfaction in m-payment services.
Fourth, we find that “poor customer service” is more likely to appear in negative reviews. It means that users are dissatisfied when they feel that the service quality of m-payments is poor. This finding is supported by ISST and related studies [17,30,59], which concluded that service quality induces greater user satisfaction for particular services and systems.
Fifth, system quality attributes such as “convenience of use,” “user-friendly interface,” and “simple process” tend to appear more in positive reviews. It implies that individuals tend to positively evaluate a m-payment system when they think that it has a superior system quality. This viewpoint is supported by ISST and related research [9,30,60], which indicated that system quality plays a positive role in enhancing user satisfaction with specific technologies.
Sixth, our research found that “secure system” tends to appear more in positive reviews. This means that when users perceive that using m-payments is secure, their satisfaction with m-payments tends to increase. This outcome is supported by the viewpoints of Nan et al. [14] and Gupta et al. [41], who proved the positive influence of perceived security on m-payment user satisfaction.
Following the STM results, we offer some marketing strategies to enhance user satisfaction with m-payments. Particularly, m-payment service providers should focus on enhancing users’ perceived service quality and perceived trust with respect to m-payments. Service providers should not only keep their promises to users but also provide data privacy norms to provide users with high levels of privacy and asset safety; this will help users increase their trust in service providers [61]. In terms of enhancing service quality, customer service centers need to respond quickly and clearly to user complaints.
As user demand changes dynamically over time [62–64], service providers should constantly track user experience by examining online reviews of their services. Hence, we suggest that m-payment service providers develop online review analysis platforms for their services by applying text-mining methodologies such as STM. By doing this, service providers can effectively establish strategies to improve user satisfaction as well as gain an edge in the competitive market [25,65].
Major limitations of our research are as following. Firstly, although PayPal is a well-known mobile payment, the outcomes of this research need to be verified in the context of other brands of m-payments (e.g., KakaoPay, Alipay, WeChat Pay, and Venmo). Secondly, although the notable effects of demographic characteristics (e.g., age, gender, and income) on the user experience of certain services have been demonstrated in earlier studies [38,66,67], we did not consider such impacts when exploring PayPal user experience.
Acknowledgement: This study is dedicated to our lab member, Sang Kyu Kang, who passed away in February 2022. He was an outstanding researcher and a real gentleman. All our lab members will remember him forever.
Funding Statement: This work was supported by a National Research Foundation of Korea (NRF) grant funded by the Korean government (NRF-2020R1A2C1014957).
Conflicts of Interest: The authors declare that they have no conflicts of interest to report regarding the present study.
This work is licensed under a Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. |