DOI:10.32604/csse.2022.022637
| Computer Systems Science & Engineering DOI:10.32604/csse.2022.022637 | |
| Article |
An Intelligent Recommendation System for Real Estate Commodity
1Department of Marketing and Distribution Management, National Kaohsiung University of Science and Technology, Kaohsiung, 824, Taiwan
2College of Management, National Kaohsiung University of Science and Technology, Kaohsiung, 824, Taiwan
3Department of Information Management, Asia Eastern University of Science and Technology, New Taipei, 220, Taiwan
*Corresponding Author: Wen-Lung Tsai. Email: tswelu@gmail.com
Received: 13 August 2021; Accepted: 30 September 2021
Abstract: Most real estate agents develop new objects by visiting unfamiliar clients, distributing leaflets, or browsing other real estate trading website platforms, whereas consumers often rely on websites to search and compare prices when purchasing real property. In addition to being time consuming, this search process renders it difficult for agents and consumers to understand the status changes of objects. In this study, Python is used to write web crawler and image recognition programs to capture object information from the web pages of real estate agents; perform data screening, arranging, and cleaning; compare the text of real estate object information; as well as integrate and use the convolutional neural network of a deep learning algorithm to implement image recognition. In this study, data are acquired from two business-to-consumer real estate agency networks, i.e., the Sinyi real estate agent and the Yungching real estate agent, and one consumer-to-consumer real estate agency platform, i.e., the, FiveNineOne real estate agent. The results indicate that text mining can reveal the similarities and differences between the objects, list the number of days that the object has been available for sale on the website, and provide the price fluctuations and fluctuation times during the sales period. In addition, 213,325 object amplification images are used as a database for training using deep learning algorithms, and the maximum image recognition accuracy achieved is 95%. The dynamic recommendation system for real estate objects constructed by combining text mining and image recognition systems enables developers in the real estate industry to understand the differences between their commodities and other businesses in approximately 2 min, as well as rapidly determine developable objects via comparison results provided by the system. Meanwhile, consumers require less time in searching and comparing prices after they have understood the commodity dynamic information, thereby allowing them to use the most efficient approach to purchase real estate objects of their interest.
Keywords: Real estate agency; web crawler; image comparison; text mining; deep learning; real estate object dynamic recommendation system
In the current digital age, consumers typically rely on the search function provided by the Internet when purchasing products. When purchasing high-involvement commodities such as houses, cars, or other high-priced products, consumers often spend a significant amount of time browsing websites and perform comparisons. Because of the diversity of information provided over the Internet, it is difficult for consumers to gauge any status changes to the information presented on a commodity website. Even if a significant amount of website data can be crawled using crawler technology, the information value cannot be gauged readily because the information has not been cleaned, compared, sorted, and analyzed. In the study, real estate means the land along with any permanent improvements attached to the land, whether natural or man-made. Methods of obtaining target information in the past, including the recommendation of relatives and friends, physical stores of real estate agents, and home sales advertisements remain; however, they are gradually being replaced by object information search through real estate agency networks, real estate transaction information platforms, real estate agency applications, social group media, and other methods. Owing to the different sizes of various trading platforms, the objects and number of objects are different, and the quality of objects varies. These differences render the purchasing process difficult for consumers.
A recommendation system displays services or products [1] that would most likely be of interest to consumers. Recommendation systems are prevalent in the current age; they are software tools and technologies that provide suggestions based on consumers’ requirements and interests, as well as identify new and suitable products by filtering user-preferred personalized items (e.g., information, services, and products) from a significant amount of information [2]. Park et al. [3] compiled 210 articles pertaining to the application and use of recommendation systems, among which 53 were associated with movies (25.2%) and 42 were associated with shopping (20.0%). Seven related recommendation systems exist in the image field (3.3%), and nine exist in each field of music and TV programs (4.2%).
Searching and comparing prices from numerous websites is time consuming and labor intensive for both real estate agents and consumers. As it is difficult for agents and consumers to detect any status change on the target object, concerns arise regarding losses due to information asymmetry. Hence, a dynamic commodity recommendation system with deep learning and text mining was developed in this study. The system (1) uses Python to develop a crawler program to extract web page information from two business-to-consumer (B2C)-type real estate agency websites in Taiwan, i.e., the Sinyi real estate agent (S-REA) and Yungching real estate agent (Y-REA), and a consumer-to-consumer (C2C)-type real estate trading platform, i.e., the FiveNineOne real estate agent (F-REA), including text descriptions and images of real estate objects; (2) conducts object difference analysis via text mining and deep learning image comparison; (3) provides comparison results to allow real estate agents to rapidly develop new objects; (4) recommends text or image information to consumers after performing a comparison to allow them to obtain real-time and dynamic information regarding objects, as well as reduce the time and cost of searching.
Currently, real estate agents often use websites to reduce information costs and increase sales [4,5]. The real estate industry is an information-intensive and information-driven industry in the current age. In fact, advances in information technology are rapidly driving the development of real estate [6], whereas the rapid development of network technology and digital media has enabled real estate companies to establish contact and interact with consumers via different methods [7]. Most real estate agents develop websites to provide object-related information such that consumers can understand objects more effectively. However, advanced data analysis, deep learning, and web crawling methods that enable real estate agents or consumers to identify objects rapidly are seldom used, and research pertaining to such methods is insufficient. Therefore, web crawling technology was adopted in this study to extract information regarding objects on real estate agency networks, conduct image and text mining, and develop a dynamic object recommendation system.
A web crawler, which is vital to search engines, is a program that can automatically crawl information on the Internet based on certain logic and computation rules [8]. Web crawlers are typically based on one or several default initial web page URLs; the rules of web page crawling for each website are adhered to, and a list of URLs is obtained from the initial web page. After each web page is crawled, the crawler extracts the new URL from the web page and recursively searches for the URLs from which the information has not been acquired to commence the next stage of the web crawler. The process continues and repeats until all web page data have been crawled or the crawler stops when the predetermined conditions are fulfilled [9]. Bangwayo–Skeete and Skeete [10] reported that data obtained through Google Trends can effectively improve the accuracy of predictions. To confirm that the keywords for a search are the primary actions for performing web crawlers on search engines, researchers can capture keywords using specific domain knowledge or web crawling and text mining methods. In this study, the real estate industry was considered a specific field. By referring to Google’s approach, web crawlers were adopted in this study to store data from related websites and identify the value of generating data through text mining and impact identification.
Web crawlers are currently widely used in the business field, primarily to acquire product information such as prices, news, and other information from the website regularly through automation to understand the price fluctuations and dynamics of competing products. In addition, web crawlers can be used to compare prices of similar commodities to identify preferential commodities rapidly [9,11,12]. In summary, although web crawlers are widely used in the commercial field, particularly to compare product prices or display similar products from different websites on the same page, only a few real estate agents use web crawlers to acquire information. Both customers and real estate agents spend a significant amount of time browsing housing websites; furthermore, they would not be able to immediately gauge the status changes on different objects and object information.
2.2 Deep Learning and Image Recognition
The most common applications of deep learning are image recognition, speech recognition, and natural language processing. In particular, the field of image recognition (which constitutes the area of deep learning) has progressed rapidly in recent years, such as in tourism and medical imaging applications. Because images can present issues such as poor quality, complex backgrounds, and different shooting angles, image preprocessing is performed to improve image quality by removing damaged images and eliminating noise effects [13]. The differences in background color and image size can be reduced, the calculation of network training can be reduced. Furthermore, the efficiency of the algorithm can be improved through preprocessing. For image-related issues, researchers suggest the use of batch processing, which involves four image preprocessing approaches, including image cropping, image color adjustment, blur removal, and image noise reduction [14–16].
CNNs are also known as CNN or ConvNets. Since 2000, CNN has been successfully applied to the detection, segmentation, and recognition of objects and regions in images [17]. A CNN is composed of several neuron layers, instead of a simple input layer, a hidden layer, and an output layer. In particular, it is composed of convolution, pooling, flattening, hidden, and output layers [18]. The convolution layer performs the convolution operation of the original image and filter, and it can be regarded as the layer that processes specific effects from the original image filter. The number of filters is set by “filters,” whereas the filter size is set by “kernel_size.” Each filter will generate different convolution operations in the form of random numbers; therefore, different filter effects can be obtained. The features extracted by the CNN convolutional layer were used as the input of the next layer. Max pooling is the most often used operation in the pooling layer. In max pooling, the maximum value is obtained from the target area, and this is equivalent to selecting the most conspicuous feature in the image, thereby reducing the number of convolution operation images generated by the convolution layer.
To provide a network of deeper and more stable results, the Visual Geometry Group of Oxford University proposed the VGGNet [19], which proved that using a small convolution layer (3 × 3) to increase the network depth can effectively improve the effect of the model. Furthermore, the VGGNet [20] offers favorable general capabilities for other datasets. In this study, the VGG16 model [21] was used as an image recognition model for deep learning. VGG16 exhibits a 16-layer structure in Fig. 1, including 13 convolutional layers and three fully connected layers.
Figure 1: Structure of CNN VGG16
Recommendation systems are primarily used to recommend commodities that may be of interest to users; they benefit not only users, but also businesses by increasing sales [22]. The services or products that are mostly likely to be of interest to consumers are displayed through the recommendation system [23]. The recommendation system provides suggestions based on the requirements and interests of consumers and filters the personalized information, services, and commodities of the user’s preferences from a significant amount of information [2]. Park et al. [3] categorized 210 research papers based on application areas, most of which were associated with movies (53 papers, 25.2%) and shopping (42 papers, 20.0%); seven papers were associated with the image field (constituting 3.3%), and nine papers each were associated with music and TV programs (constituting 4.2%). The recommendation system has been successfully applied in many fields, including e-commerce, information search, social media, and news recommendations.
Recommendation systems are primarily classified into two categories [24,25], i.e., content-based (CB) and collaborative filtering (CF) recommendations. CB recommendations filter and sort the recommended commodities based on matching or the relevance between product and user characteristics. The characteristics of the recommended commodities are extracted and described, the natural language processing technology is used for the text, and the features of the screen and brief information are extracted for images and videos. The algorithms of CB recommendations primarily depend on the user’s behaviors, including the related descriptions, information, and operational behaviors [26]. The main functions of CF are prediction and recommendation. It can be classified based on the different behaviors of target users and recommends commodities at the corresponding level. CF is primarily used to mine the relationship between users to perform classification and provide recommendations based on the similarity between users. The recommendation algorithm based on deep learning primarily uses CNNs, cyclic neural networks, and restricted Boltzmann machines. Compared with traditional recommendation systems, deep learning recommendation systems offer greater advantages. In summary, deep learning benefits recommendation systems significantly; however, this method is rarely used in products related to real estate agents. In this study, the advantages of CB and CF were combined; furthermore, both text mining and deep learning were used to construct a dynamic recommendation system for real estate objects to provide an innovative application model that aids real estate agents and consumers.
3 Construction of Dynamic Recommendation System for Real Estate Objects
The webpages and real estate trading platforms of real estate agents were used in this study. Web crawlers were used to acquire object information to improve the efficiency of real estate agents in acquiring object information. Subsequently, text mining and CNN images were compared to rapidly analyze the object information on webpages, and a dynamic recommendation system was established for real estate objects, thereby allowing real estate agents to rapidly identify objects different from their own on other web pages. Next, the real estate owner was consulted to obtain agency sales contracts, which can provide dynamic changes in the object information, thereby enabling real estate agents and consumers to understand the dynamic changes of object information as well as the latest information pertaining to the object. The process encompasses four steps in Fig. 2, as follows:
Figure 2: Process flow for developing dynamic recommendation system for real estate objects
Step 1: Write a crawler program for data acquisition
Python3 of the Anaconda Navigator Jupyter Notebook was utilized to write the crawler program, whereas the Requests and BeautifulSoup package was used to capture the object information of the location in Nanzih district (Kaohsiung City) of the S-REA, Y-REA, and F-REA.
Step 2: Data preprocessing and image amplification
Python was adopted for data cleaning and integration. Subsequently, the image angle (15°, 30°, –15°, –30°), color, linearity, and size were converted using OpenCV to generate more information in the image database such that the success rate of searching is increased when input images are input.
Step 3: Text and image comparison
A text mining program was written in Python to compare the text information of three different source websites with the object name as the main comparison field. An image comparison was performed using the CNN–VGG16 algorithm. Image comparison was perform to extract images for which the users intended to search, as well as the image characteristics of the image database. Subsequently, a similarity matrix was calculated to obtain similar images based on the similarity of the images.
Step 4: Recommend real estate commodities
The results of image comparison and text mining can assist real estate agents and consumers in searching for similar objects to purchase or bargain; this is accomplished by inputting the image of the object of interest. Based on the text comparison results, the number of days for the sale of the object as well as dynamic price fluctuation trends can be understood easily, thereby allowing recommendations to be provided to real estate agents and consumers.
4 Construct Recommendation System via Web Crawling and Deep Learning
4.1 Web Crawling Result Acquisition and Image Amplification Processing
4.1.1 S-REA, Y-REA, and F-REA Text Data Crawling Results
For the website crawling process in Fig. 3, the URL is regular because the page number is +1 when it arrives at the next page. The “for” loop was used to automatically turn the page and capture the object information of each page, and “requests.get” was used to request for the website link and obtain all information pertaining to “div class LongInfoCard_Type.”
Figure 3: S-REA, Y-REA, and F-REA object text information crawler program code
The objects in Nanzih district (Kaohsiung City) were acquired from the websites of the S-REA, Y-REA, and F-REA through the crawler program, see Fig. 3, and those websites contained 707, 1,626 and 1,574 objects respectively. The contents, see Tab. 1, included the object name, add, age, size, room, floor, officer, and other text information.
4.1.2 S-REA, Y-REA, and F-REA Image Data Acquisition Results
For the S-REA, Y-REA, and F-REA object image crawlers in Fig. 4, “requests.get” was used to request for the website link and obtain all “div class LongInfoCard_Images” images to identify their images.
Figure 4: S-REA, Y-REA, and F-REA object image crawler program code
During data acquisition, the object image data from different websites were captured using the crawler program. The total number of objects on the three websites was 3,907, and each object contained an average of 7.8 images; the information of 30,475 object images was obtained in Fig. 5.
Figure 5: S-REA, Y-REA, and F-REA object image crawling results (example)
4.1.3 S-REA, Y-REA, and F-REA Website Image Amplification Results
Crawlers were used to obtain 3,907 object images in Nanzih district (Kaohsiung City) from the S-REA, Y-REA, and F-REA. Seven linear amplification methods were adopted for the images, including grayscale, resizing, and angle adjustment (–15°, –30°, 15°, and 30°). The amplification image database contained 213,325 images (30,475 × 7 = 213,325).
4.2 Results of Comparing Website Text Information
4.2.1 Cross Comparison of Similarity Among S-REA, Y-REA, and F-REA Objects
In the text information program to compare the S-REA, Y-REA, and F-REA website objects in Fig. 6, the objects owned exclusively by each real estate agent’s network, those jointly owned by two real estate agents’ networks, and those jointly owned by three real estate agents’ networks can be output to Excel using three worksheets, i.e., diff, dropped, and added.
Figure 6: Text information comparison program code to compare among S-REA, Y-REA, and F-REA
Real estate agents can query different worksheets based on their requirement. For example, if the staff of S-REAs wish to know whether the status of their own objects has changed, they can query through the dropped worksheet; if they wish to know the information regarding other real estate objects and develop potential objects, they can use the added worksheet. Chain real estate agents can use cooperative methods. For example, if one discovers from the added worksheet that the other party possesses an object that the customer likes, then he/she can negotiate with the other party to sell the object based on a win–win situation. Tab. 2 shows the cross-comparison results for the objects of the S-REA, Y-REA, and F-REA.
Based on the comparison results pertaining to the object text information, the S-REA, Y-REA, and F-REA owned 666, 1,570, and 1,493 objects, respectively; both the S-REA and Y-REA jointly owned eight objects, the S-REA and F-REA jointly owned 33 objects, and the Y-REA and F-REA owned 48 objects simultaneously, whereas the three agents did not own any object simultaneously (see Fig. 7). It was clear the not many identical objects were owned by the different real estate agent networks. Therefore, to develop new objects or increase search efficiency, a comparison with other textual information is a more relevant reference for decision-making.
Figure 7: Cross-comparison results of S-REA, Y-REA, and F-REA for object content similarity
4.2.2 Price Fluctuations and Number of Days for Sale of S-REA, Y-REA, and F-REA Objects
Based on the text mining of daily crawled object information, the price of 135 among 3,907 objects was reduced during the data acquisition period, the price of 47 objects was reduced by more than twice, and the most number of days an object was advertised for sale on the real estate agency website was 97 days. This real-time and dynamic object information can be provided to real estate agents and consumers as a reference for decision-making, as shown in Tabs. 3 and 4.
The content of each image in the image database was read out and converted to the VGG input format, the feature vector was calculated using the VGG16 model, and the images were compared in pairs. Subsequently, the similarity was calculated to identify the two most similar images. The closer the similarity value to 1, the higher was the similarity. The number of similar images can be adjusted as shown in the red box in Fig. 8.
Figure 8: Image comparison program code
The object appearance image was captured as the search image for a comparison with 213,325 object images in the amplification image database. Subsequently, the similarity was calculated and then used to identify the top two objects that are the most similar to each other (see Fig. 9). This method allows consumers to identify similar objects through image comparison with the object of interest for purchase or bargaining, thereby reducing the cost of web search. In addition, the results of image comparison not only allows real estate agents to easily understand the object based on the text information of the object and then develop potential objects, but also allows them to identify similar objects in the object database through image comparison to serve as a reference to consumers when the latter use an image to inquire regarding real estate agents.
Figure 9: Image comparison result
In this study, a dynamic recommendation system for real estate objects was developed using web crawler technology and deep learning. Subsequently, web crawlers were employed to capture object information in real estate agents’ networks and real estate trading platforms. Next, the captured text and images of the objects were compared to identify different objects. Finally, a dynamic object recommendation system was established based on the comparison analysis results to allow real estate agents and consumers reduce search costs.
The dynamic recommendation system for real estate objects developed in this study can assist real estate agents in developing potential objects rapidly, provide consumers with real-time and dynamic information, and identify similar objects rapidly using object images. The results of this study are as follows:
(1) The efficiency of acquiring information on real estate objects can be improved using the Python web crawler. The survival of real estate agents in industry depends on the development of objects. However, the current digital application of the real estate agency industry focuses on the manner in which objects are presented to assist consumers in understanding objects, whereas advanced technologies such as image recognition or web crawlers are seldom employed to assist real estate agents in identifying objects rapidly. In this study, a Python web crawler was used to retrieve object information regarding real estate trading platforms within a data collection period of five months. Information pertaining to 3,907 objects was acquired via the daily crawling of objects located in Nanzih district (Kaohsiung City) on two B2C websites (S-REA and Y-REA) and one C2C website (F-REA).
(2) Text and image comparison were used to provide object information promptly to real estate agents and consumers. Many webpages are available for selling houses with diversified information and frequently changing contents. By automatically acquiring and comparing the information, real estate agents and consumers will be able to grasp and understand the sales status of the objects. Based on the comparison results of the text information of the objects, the S-REA, Y-REA, and F-REA each had 666, 1,570, and 1,493 objects, respectively. The S-REA and Y-REA jointly owned eight of the same objects, the S-REA and F-REA jointly owned 33 of the same objects, and the Y-REA and F-REA owned 48 of the same objects; meanwhile, none of the same objects were owned by all three agents simultaneously. Real estate agents can learn the object information via different worksheets based on their requirement for the rapid search and development of potential homeowners, as well as objects that they serve as sales agents. The text mining of the same real estate object enables consumers and real estate agents to understand the price reduction of the object and the number of days the object has been advertised for sale on the real estate agency website; consequently, they will be better informed when purchasing or bargaining for the object. When consumers are interested in an object, they can identify similar objects based on the image of the object, by adjusting the number of search results according to their requirements, and using comparison results to purchase or bargain, thereby reducing the cost of web search. The image comparison result allows real estate agents to easily understand the object based on the text information of the object, or identify similar objects in the image database through image comparison to provide a reference to consumers.
(3) Both text and image comparison were combined to construct a dynamic object recommendation system to provide innovative applications in the real estate industry. In this study, the information of 3,907 objects in Nanzih district (Kaohsiung City) was acquired. At least one week was required to compare the objects manually; even though the object information was stored as an Excel file using a crawler, and the filter or search function was used for auxiliary comparison, a significant amount of time was required to organize the information. The object dynamic recommendation system developed based on text mining and deep learning images can complete the object information comparison in only 2 min. Hence, it enables real estate agents and consumers to obtain the latest information regarding objects promptly; this significantly reduces the search time and cost, rendering real estate agency operations more efficient and providing more convenience to consumers.
Developing objects is vital to real estate agents. Previously, real estate agents distributed leaflets, visited strangers’ homes, and used other methods for object development. These methods are not only time consuming, but also costly. Using the dynamic object recommendation system proposed herein, real estate agents can obtain comparison results from crawled data within 2 min. Real estate agents can easily understand the status of the objects that they own through text mining; furthermore, they can select the most likely customers or objects for development from a significant amount of data at any time, thereby significantly reducing the search time and cost. In addition to allowing real estate agents to match the object text narrative to gain a better understanding of the object, the image comparison enables the agents to identify similar objects for recommendation and improve work efficiency by promptly comparing an object with the image database in real time; this is accomplished by inputting the object image when the consumer demands for similar objects, regardless of whether the customer is in a physical store or browsing for an object from a website.
Owing to the current digital age, consumers rely significantly on the search function provided by the Internet when purchasing products. However, the information provided by the Internet is diverse, and it is difficult to detect whether the commodity information on the website is changing. The dynamic object recommendation system can provide relevant object difference analysis and information difference analysis based on the various requirements of users, such as allowing consumers to obtain the dynamic price reduction information of the object more easily, as well as to understand the number of days the object has been listed on the website. When consumers identify an object of interest, they can search through images at any time to identify similar objects, and then purchase the object or bargain for it. Furthermore, image comparison allows the number of similarities to be set based on the consumers’ requirements, thereby reducing the time required to browse for the objects of interest; this can enhance the consumer experience and establish multiple interactive channels.
Information is increasing at a rapid pace owing to the advent of the big data era. The extraction of data required from this significant amount of data and the acquisition of important information from the data to facilitate decision making have become important topics. Previously, most studies associated with recommendation systems were performed in the fields of film, retail, and shopping, whereas those pertaining to real estate were primarily based on investment forecasting or housing price forecasting, where few studies used advanced technology to establish recommendation systems. Deep learning has achieved favorable results in the fields of image recognition, speech recognition, natural language processing, and others. In addition, the application of deep learning has introduced new changes to recommendation systems. The increasingly frequent e-commerce transactions occurring currently has caused more companies to advertise their products online, which results in a significant amount of commodity information. Results are difficult to achieve in a short duration using the recommendation system; therefore, the time frame must be extended accordingly. The daily or hourly continuous update of object information by web crawlers can optimize the recommendation system and then reveal information regarding the object of the user’s potential interest.
Based on the results of the dynamic recommendation system for real estate objects, the following are suggested for future studies: (1) The dynamic difference analysis information can be presented on a web page: The presentation of object dynamic recommendation system through a web page allows consumers to rapidly browse the dynamic changes of information through the web page and use images to identify similar objects. When the image of the object is linked to the real estate agent’s website wherein the object is located, if consumers are interested in the object, then they can directly proceed to the real estate agent’s webpage to view the object’s information through the link, thereby reducing search costs. (2) Establishing a difference analysis database: This study was conducted only in Nanzih district of Kaohsiung. To analyze the information of objects in Kaohsiung or in the entire country of Taiwan, the amount of data involved will be significant. By storing the crawled text and images in a cloud database and linking them to a query screen, the information can be quickly analyzed, and real-time difference analysis can be conducted in response to changes in the object status.
Funding Statement: The authors received no specific funding for this study.
Conflicts of Interest: The authors declare that they have no conflicts of interest to report regarding the present study.
 | This work is licensed under a Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. |