MSM: A Method of Multi-Neighborhood Sampling Matching for Entity Alignment

The heterogeneity of knowledge graphs brings great challenges to entity alignment. In particular, the attributes of network entities in the real world are complex and changeable. The key to solving this problem is to expand the neighborhoods in different ranges and extract the neighborhood information efficiently. Based on this idea, we propose Multi-neighborhood Sampling Matching Network (MSM), a new KG alignment network, aiming at the structural heterogeneity challenge. MSM constructs a multi-neighborhood network representation learning method to learn the KG structure embedding. It then adopts a unique sampling and cosine cross-matching method to solve different sizes of neighborhoods and distinct topological structures in two entities. To choose the right neighbors, we apply a down-sampling process to select the most informative entities towards the central target entity from its one-hop and two-hop neighbors. To verify the effectiveness of matching this neighborhood with any neighborhood in the corresponding node, we give a cosine cross-graph neighborhood matching method and conduct detailed research and analysis on three entity matching datasets, which proves the effectiveness of MSM.


Introduction
Entity alignment is designed to determine whether two or more entities with different knowledge graphs point to the same object in the real world. It not only contributes to the construction and expansion of knowledge base, but also plays an important role in solving cross-network crimes. For example, entity alignment is now widely used in graph networks and social networks [1,2]. The most advanced entity alignment solutions mainly rely on the structure information of knowledge map to judge the equivalence of entities, but in the real-world knowledge map, most entities only have low node degree and little structure information. In addition, the lack of annotated data greatly limits the effectiveness of the entity alignment model. Unfortunately, entity alignment is not trivial, because real-life knowledge graphs are often incomplete and different knowledge graphs typically have heterogeneous schemas and the equivalent entities in different graphs often have different neighborhood structures.
The aim of network entity alignment technique is to find the same entity in different networks. At present, increasing attention has been paid to the embedding-based methods [3][4][5][6][7]. Its core idea is to transform the nodes in the network into low-dimensional vectors by certain methods, whose vector dimensions are much lower than the number of nodes and contain the attributes of the nodes themselves and the semantic relationship information between the nodes. Unfortunately, these methods can only deal with homogeneous networks or multilingual knowledge graphs. The types of nodes and the relationships between nodes are same [8,9]. However, most of the network data in reality are heterogeneous networks. There are many kinds of node types or multiple node relationships. For example, in Fig. 1, United States of America is among the one-hop (direct) neighbors of Barack Obama in Wikidata. However, in Dbpedia, it is a two-hop neighbors.
At the same time, the information contributed by the same order neighbor of the central entity is nonequivalent [8,9]. To elaborate on this point, let's move on to Fig. 1. There are many city entities for USA which also have the entity United States of America. Consequently, the contribution of United States of America to the central entity is significantly less than that of Christian. Because existing embedding-based method are unable to choose the right neighbors, we need a better approach.
The challenge of resolving this issue lies in the difficulty of fully mitigating the non-isomorphism in the neighborhood structures of counterpart entities from different KGs. We present the Multi-neighborhood Sampling Matching Network (MSM), a novel Network embedded and Sampling-based framework. The goal of MSM is to obtain the neighborhood with the most valuable information and accurately estimate the similarity of neighborhood among entities in different knowledge graphs. To learn the embedding of knowledge graph structure, MSM utilizes multi-neighborhood network representation learning method to aggregate higher degree neighboring structural information for entities. We evaluate MSM by applying it to benchmark datasets DBP15K [10][11][12][13][14][15][16].
2 Multi-Neighborhood Sampling Matching

KG Structure Embedding
In addition to the one-hop neighborhood information, the non-directly related high-order neighborhood information is also very important for the representation of the central entity [16]. Therefore, MSM first constructed a multi-neighborhood network representation learning method (MNE), using this representation method learns the structure embedding of KG to gather the high-order neighborhood structure information of the entity. The overall architecture and processing pipeline of MNE is shown in Fig. 2. MSM uses pre-trained word vectors to initialize MNE.
We use G ¼ ðE; R; T Þ to represent a knowledge graph, in which E; R; T represents entities, relationships, and sets of triples. Without loss of generality, we consider the entity alignment task between two knowledge graphs G 1 and G 2 based on a set of pre-aligned equivalent entities. Our goal is to find an equivalent entity pair between G 1 and G 2 . We put G 1 and G 2 as a large input graph into MSM, each MNE layer consists of two layers of GCN aggregated entities of one-hop neighborhood information, a layer of attention-network (ATT-Net) aggregated entities two-hop neighborhood information and each GNN updates the node representation as: where h ðlÞ i;1 is the output node features of l-th GCN layer. E i is the normalization constant. N i is the set of neighbor indices of entity i. W ðlÞ 1 2 R d ðlÞ Âd ðlÀ1Þ is the attention weight of l layer neighborhood of node i obtained by GNN training. To aggregate the two-hop neighborhood information of entity i, we have adopted ATT-Net. The hidden representation of entity i, denoted as h l i;2 , is computed as follows: where h ðlÞ i;2 is the output node features of l-th ATT-Net layer. a ðlÞ ij is a learnable attention weight for entities i and its neighbor j. W ðlÞ 2 2 R d ðlÞ Âd ðlÀ1Þ is the wight matrix. Finally, for entity i, we use the gating mechanism to combine its one-hop and two-hop neighborhood information. The node representations are given as below: the weight matrix and bias vector, respectively. In order to control the accumulated noise, we also introduce the highway network [15] into the MNE layer, which can effectively control the propagation of noise in the MNE layer.

Neighborhood Sampling
The first-order and second-order neighborhoods of an entity are the key to determining whether the entity should be aligned with other entities. However, as we discussed before, not all first-order and second-order neighborhoods make a positive contribution to entity alignment. In order to select the correct neighbor, we use a down-sampling process to select the entity that provides the most information to the central target entity from its first-order and second-order neighborhoods. Previously, many random sampling methods were used, but the method have some randomness. The neighbor nodes sampled may not necessarily contribute the most. Therefore, we give a sampling method based on probability. Formally, given an entity e i , the calculation formula for sampling its first-hop domain e iÀj and second-hop domain e iÀj 0 are [16]: where N 1 i and N 2 i are the first-hop and second-hop neighborhood index set of the central entity e i , respectively. h i , h iÀj and h iÀj 0 are the entity representation vectors of the entities e i , e iÀj and e iÀj 0 respectively, and W 1 , W 2 are shared weight matrix.
By selectively sampling first-hop and second-hop neighborhoods, MSM gets the most valuable neighborhood information of each entity. In the end, MSM achieves the alignment of G1 and G2 through neighborhood matching and neighborhood aggregation.

Neighborhood Matching
In order to judge whether the two entities should be aligned, we need a similarity calculation to its neighbor nodes. MSM builds hence a neighborhood subgraph generated by the sampling process for each entity. Then MSM will only operate neighbors within the subgraph, achieving the neighborhood matching.
Inspired by the graph matching method [5], our specific matching method is as follows.
We need to compare the subgraphs of each candidate entity in its sampling neighborhood subgraph E 2 to select the best aligned entity of the entity e i in E 1 . But if each entity neighborhood in E 2 is compared and calculated with e i , this requires too much calculation. Therefore, MSM adopts an approximate alternative method. Inspired by the candidate selection method [16], MSM samples an entity alignment set f gof e i , and then calculates the subgraph similarity between e i and these candidate entities. Therefore, for the entity e j in E 2 , the calculation formula for it to be a candidate entity for e i is: where h i and h j are the node vector representations of e i and e j output by MNE.
We propose the following graph matching network, which changes the node update module in each propagation layer. It not only considers the aggregated messages on the edge of each graph as before, but also considers a cross-graph matching vector, which measures the degree of matching between a node in one graph and one or more nodes in another graph. Define ðe i ; e j Þ as the entity pair to be measured, where e i 2 E 1 , e j 2 E 2 . The cross-matching vector of e i is calculated as follows: Here a j!i is the cosine similarities of entity e i in E 1 with the entity e j in E 2 , m i is the matching vector of node e i , N i is the sampled neighbor set of e i , h i and h j are the node vector representations of e i and e j output by MNE, respectively. h À1 i is an attentive vector for the entire graph E 2 . Then, we combine h i and m i to get the sampled neighbor representation of e i : where È indicates vector concatenation.

Entity Alignment
We give the loss function of MNE: where dði; jÞ ¼ h i À h j L 1 , Á k k L 2 means L 2 À norm; c > 0 is a margin hyper-parameter; X is the alignment seeds and X 0 is the set of negative aligned entity pairs generated by nearest neighbor sampling [15].
The loss function of MSM after the pre-training phase, defined as: where f ði; jÞ ¼ĥ i À _ h j L 1 ; T is the positive alignments set and T 0 the negative alignments set.

Experiment
In this section, we evaluated the multi-neighborhood sampling matching model (MSM) on three data subsets of the open-source data set DBP15k by comparing with other methods. Tab. 1 gives detailed descriptions of the DBP15k datasets. We use the same split with previous work, 30% for training and 70% for testing.
The configuration we use in the DBP15K datasets is: b ¼ 0:1, c ¼ 1:0, and we sample 5 neighbors for each entity in the neighborhood sampling stage.
The comparison method we use is as follows: (1) JAPE [17]: A model for preserving attribute embedding for cross-language entity alignment. It embeds the structure of the two networks into a unified vector space, and further optimizes it by using the attribute correlation in the network. This model is a supervised learning model. In the experiment, 30% of the nodes in the data set are used as training data and 70% of the nodes are used as test data. The model parameters adopt the original default best parameter combination. In the experiment, 10 negative samples are drawn for each pre-aligned entity pair.
(2) AliNet [18]: In order to solve a new KG alignment network in which the replica entity has a nonisomorphic neighbor structure, it aims to alleviate the non-isomorphism problem of the neighbor structure in an end-to-end manner. Due to the heterogeneity of the schema, the direct neighbors of the replica entity are usually not similar. AliNet introduces distant neighbors to extend the overlapping part of the neighbor structure. The attention mechanism is also used to emphasize helpful distant neighbors and reduce noise. Then it uses the gate mechanism to aggregate the information of direct neighbors and distant neighbors.
(3) BoostEA [16]: The entity alignment based on the embedded usually depends on the alignment of the existing entities as training data. But the first alignment can be usually accounted for a small part. BoostEA proposes a bootstrapping method to solve the above challenges.
(4) GMNN [5]: Since the previous cross-language knowledge graph alignment research relies on the idea of entity embedding, it cannot be applied to the two knowledge graphs. GMNN combines the two kinds of map matching problems with entity context information into a map matching problem and proposes a matching model of a graph neural network, which includes map matching and pixel matching information.
(5) MSM: Firstly, the low-dimensional vector representation of nodes in the network is obtained by using the multi-neighborhood network representation learning method (MNE) proposed in Section 2 of this paper. Then, entity alignment is carried out by using the information entropy sampling and crossgraph neighborhood matching entity alignment method proposed in this paper.

Evaluation Index
To compares the various methods more objectively and comprehensively, in the experiment, the commonly used model evaluation indicators Hit@k in the field of entity alignment are used, and the calculation formula is as follows: where n rk represents the number of times the target node appears in the k closest candidate target nodes, and N is the number of tests. We select Hits@1 and Hits@10 indicators according to the commonly used experimental methods.

Analysis of Results
(1) The experimental results of each method on different data subsets of the data set DBP15k are shown in Tab. 2. From Tab. 2, we can see that under the evaluation index of Hits@1, MSM has an improvement of 3% to 5% relative to GMNN, and also has an improvement of about 3% under the evaluation index of Hits@10. The result meets 8% improvement on zh-en data subset. Compared with JAPE, AliNet and BoostEA, MSM has a greater improvement.
To evaluate the sampling method on the accuracy of the model, we implement a model variant (referred as MSM (s-two) ) that samples two-hop neighbors. From Tab. 2, we find that the sampling two-hop neighbors leads MSM a gain of 0.4%~0.8% by Hits@1 and 0.1%~0.3% by Hits@10 relative to the sampling one-hop neighbors (referred as MSM (s-one) ) on DBP15K. The result confirms the importance of the two-hop neighborhoods in KG structure embedding and neighborhood sampling.
(2) To explore the impact of using the pre-trained word embeddings to initialize the MSM. We remove the initialization part of the MNE, choosing the different sampling size for comparative analysis. From Fig. 3, for DBP15K ZH-EN , we observe that the initialization of MNE is very important of MSM. More exactly, removing the initialization from MSM, leads to around a 43% drop in Hits@1 and a 40% drop inHits@10 on average. On the other hand, MSM is sensitive to the size of sampling and the model works better when the size of sampling is 5.
(3) To verify the superiority of our cross-graph neighborhood matching method, we compare it with the common direct aggregate matching method. As shown in Fig. 4, when we choose different candidate set sizes, the cosine cross-neighborhood matching method performs better. It increases 0.5%~1% relative to the overall the cross-neighborhood matching method under the evaluation index of Hits@10 and about 0.2% under the evaluation index of Hits@1. At the same time, the different matching candidate size also has some influence on MSM performance. When the size of candidate selection is 18, the performance of the MSM is better on DBP15K ZH-EN .
(4) To show the effect of entity alignment more intuitively, this part visually shows the node distribution consistency before and after entity alignment. We reduce the 300-dimensional vector to two-dimensional space through t-sne. Fig. 5 show the visualization of 4000 pairs of nodes randomly selected from two KG on DBP15K JA-EN . These result show that our pre-trained vectors, sampling and matching modules are particularly important, when the neighborhood sizes of equivalent entities greatly differ and especially there may be few common neighbors in their neighborhoods.

Conclusion
This paper proposes a multi-neighborhood sampling matching entity alignment method, which aims to solve the problem of different neighborhood sizes and topological structures in heterogeneous networks. We build a multi-neighborhood network representation learning method to achieve effective aggregation of entity neighborhood information and use a new sampling-based method to select the most informative neighbor for each entity, using the method of cosine cross-graph neighborhood matching to achieve rapid alignment of different network entities. We conducted extensive experiments on real-world data sets and compared MSM with four embedded-based entity alignment methods. Experimental results show that MSM obtains the best and more robust performance, and consistently outperforms competing methods in data sets and evaluation metrics. For future work, we plan to incorporate the multi-neighbor information of entities in other modes into our model structure. At the same time, since some alterative sampling techniques based on ranks information may lead more efficient procedure [19,20], this will be our followup work.