2.1 Word Embedding

Word representation (Distributed Representation, Word Embedding) was first proposed by Hinton⁷ . Word representation is learned a vector from a large number of unlabeled free text corpus by language model. The idea of using neural network to train language model was first presented by Xu¹⁵. Word representations learned by neural network language models can be used for many NLP tasks such as POS tagging, chunking, named entity recognition, semantic and syntactic similarity. And subsequently many language models are proposed in⁵ . (Mikolov et al. (2013) proposed two state-of-the-art models (CBOW and Skip-gram)¹ to capture better semantic and syntactic word similarity⁸,⁹,¹⁰.

2.2 Embedding-based Model

Many recent energy-based embedding models ⁷,⁸,⁹ are proposed, focusing on increasing expressivity, but resulting in higher computational cost. Bordes et al. ¹⁰ proposed a simpler model (TransE), however, the drawback of it is that it only can model linear triplets. Wang et al. proposed an extended model of TransE, TransH, which faces the same issue. Several models have been recently proposed for that purpose¹⁶.

2.3 Deep Neural Network

The Deep Learning models is very hot recently, such as DBN¹³ , DBM¹⁴, Deep-autoencoder, etc. They have been used in many applications and perform excellent, such as speech recognition, image recognition and natural language processing, etc.

3.1 Word Representation learned by Language Model from Unstructured Data (Free Text) and Encoding Structured Data (Knowledge Base)

Word representation are word vector that can be used as features for other models. In this work, we choose two typical word representations. One is the approach of (Mikolov et al., 2013), known as Word2vec. It used two main language models: n-grams and skip grams for learning. Here resulting word vectors have some interesting character and capture many linguistic regularities. Take the famous two as example here, vector operations vector(’China’) - vector(’Beijing’) + vector(’Tokyo’) results in a vector that is very close to vector(’Japan’), and vector(’king’) - vector(’man’) + vector(’woman’) is close to vector(’queen’). In Word2vec, it has offered more than 1.4M pre-trained entity vectors with naming from Freebase. The other work is proposed in⁶ , as SINNA. We also directly utilize the resulting work vectors from it, pre-trained word embedding of Wordnet, and how to train them is not our focus in this paper.

In order to connect latent knowledge of Knowledge Base with word representations learned by language model from free text, we propose to encode the entities and relationships of the triplets into latent embedding space. This allow us to build a model, like deep neural network model, which can calculate the plausibility of additional new triplets for extending knowledge base. As introduced in Section 2, several models have been recently proposed for that purpose. In this work, we choose to follow the Pairwise-interaction Differentiated Embedding model (PIDE)¹⁶ , which is claimed to performed excellent.

In the next, we briefly review the Pairwise-interaction Differentiated Model. Given a training set H of triplets (subject s, predicate p, object o), s,o∈E (the set of entities) and p∈R (the set of predicate relationships), the PIDE model learns entities and predicate relationships latent vector representations. It considers that the reason why a triplet (subject s, predicate p, object o) should be hold is due to its inner pairwise interactions: (subject, object), (subject, predicate) and (predicate, object), each of which should contribute to the confidence of the triplet. The more closer between subject and object ((subject and predicate) or (predicate and object)), the more likely the knowledge fact triplet hold. In addition, it also consider that the entity in knowledge triplet should possess semantic and syntactic information, and the predicate relationship should have syntactic information. The semantic information of entity represents its content information, while the syntactic information represents their position and order information. So it can obtain the interaction function between subject entity and object entity fsem+syn(s,o) using their semantic and syntactic information. Also it can get the interaction function between subject entity and predicate relationship fsyn(s,p) and the interaction function between predicate relationship and object entity fsyn(p,o) using their syntactic information. The PIDE model scoring function is defined as follows:

where es=[es1,es2], eo=[eo1,eo2] and ep=[ep1,ep2], es1,es2,ep1,ep2,eo1 and eo2∈Rk. The model is trained with contrastive max-margin optimization criterion. The main idea is that each triplet in the training set should receive a higher score than a corrupt triplet in which one of the entities is replaced with a random entity.

3.2 Learning Multimodel DNN with Connecting Word Embedding and Entity Latent Representation

As we have introduced above, we could learning two type of latent representations for a same word (entity)². The word embedding of the entity is learning from free text by word2vec. We also learn the another latent representation of the word (entity) from structured data (knowledge base). Theoretically both the two latent representation of the same entity should indicate its semantic information. But its two semantic vector expressions would be different because they derive from two different latent semantic space, one is free corpus ant the other is structured knowledge base. So we are trying to find a pipeline that can somehow connect them or transform them from each other. In this paper, we propose to use Deep neural network model for connecting word embedding and entity latent representation. We use a great amount of pre-learned pairs (word embedding, entity latent representation) to train the deep neural network (DNN) model. By this, we could build a pipeline across the two latent semantic space. If there have a additional new entity which exists in free text but does not exist in knowledge base, namely the word embedding of the additional new entity in free text is known and not in knowledge base. Through the DNN model as pipeline, we can obtain latent representation of the additional new entity by put the its word embedding as input.

3.3 Implementation for Extending Knowledge Base

In the next, we present the workflow of the implementation and show how to extend the knowledge base from the beginning. As in the Fig.1, here comes a new entity at testing phase. We find out its word embedding pre-learned from free text. We put the word embedding as input into Multimodal DNN. Through the model, we obtain the entity representation that would be the latent semantic vector in knowledge base space. Finally we can calculate the scoring function of all the possible triplets with the new entity. The candidate triplet holds if its score is larger than the threshold. So we extend the knowledge base in this way.

4.1 Data Set

WordNet is a large lexical database of English. Nouns, verbs, adjectives and adverbs are grouped into sets of cognitive synonyms (synsets), each expressing a distinct concept. Synsets are interlinked by means of conceptual-semantic and lexical relations. Examples of triplets are ( _payment_NN_1, _hyponym, _recompense_NN_1 ) or ( _flint_NN_3, _part_of, _wolverine_state_NN_1).³ Here we do not yet distinguish the same entity with different part-of-speech tag because their word representations learn by language model from free text is unique. We create a data set with WordNet and Word Representations (which were trained for about 2 month over Wikipedia) provided by SENNA⁴. This data set include 47176 triplets with 10556 entities and 18 relationships which were randomly split into three parts (Train, Valid, Test). The training data set includes 35,016 triplets with 8,541 entities and 18 relationships, which would be further split into three parts for PIDE model learning in KB. The valid data set includes 4,283 triplets with 723 extra new entity which would be not emerge in training data set. LEFT-NEW, RIGHT-NEW, BOTH-NEW indicated that the left entity or right entity or both entities of triplets were new entities. The test data set includes 7,877 triplets with 1,292 extra new entity. This data set is denoted WN-WR10K in the rest of this section

Freebase is a large collaborative knowledge base of general facts, currently including around 1.2 billion triplets and more than 80 million entities. We created the data set with FB15K³ extracted from Freebase and Freebase entities vectors⁵ (word represenation) which trained on 100B words from various news articles. FB15K is subset of Freebase including 592,213 triplets with 14,951 entities and 1,345 relationships. We remove the entities which do not have the word representation from FB15K. This resulted in 471,648 triplets with 13,868 entities and 1271 relationships. Likewise we deal with it further as WordNet. This data set is denoted FB-WR14K in the rest of this section which is shown in Table 1

Table 1 Statistics of the data sets used for the experiment and extracted from the two knowledge bases, WordNet and Freebase ³. 

4.2 Evaluation Metrics

In the experiment, we use the ranking criteria (Bordes et al. 2011) for evaluation. Firstly for each test triple, we remove the subject entity and replace it by each of the entities of the dictionary in turn. The function values g(s, p, o) of the negative triples would be computed by the related models and then sorted by descending order. We can obtain the exact rank of the correct entity in the candidates. Similarly, we repeat the whole procedure while removing the object entity instead of the subject entity of the test triple. We have three kinds of test triplets: LEFT-NEW, RIGHT-NEW AND BOTH NEW. So we use two evaluation metrics for comparison in the three data sets: the mean of those predicted ranks and the proportion of correct entities ranked in the top 10 (Hits@10(%)). Finally they are LR: Left Rank⁶; LH10: Left Hits@10(%); RR: Right Rank; RH10: Right Hits@10(%); MR: Mean Rank; MH10: Mean Hits@10(%).

4.3 Baseline and Configuration

In this experiment, we choose two baseline models for comparison. The first one is the Un-transform model, in which we directly utilize the word embedding as the entity latent representation without any model as the pipeline to connect the two latent semantic space. The other one is linear transform model namely shallow neural network, using artificial neural network (ANN). We select 4 layers for the ANN model and use BP learning algorithm to train the model.

4.4 Learning the Entity Representation with PIDE Model

For learning the entities and predicate relationships latent representations in TRAIN-TN using PIDE model, we selected the best parameters: { λe (learning rate of entities)= λr (learning rate of predicate relationships) = 0.1(when epoches ≤ 300), λe=λr = 0.01(when epochs > 300), κ =50, γ = 1} using valid data set TRAIN-VD on WN-WR10K TRAIN EX.; { λe =0.01, λr = 0.001 , κ =50, γ = 1} on FB-WR14K TRAIN EX.. And in this experiment we also try use the TransE model as comparison, we selected paramters: { λe =0.1(when epoches ≤ 300), λe = 0.01(when epochs > 300), λr = 0.01, κ =20, γ =2}on WN-WR10K TRAIN EX.;{ λe =0.01, λr = 0.001 , κ =50, γ = 1} on FB-WR14K TRAIN EX.. The number of iteration is 1000. We test it on test data set TRAIN-TT of the two data sets and evaluation results are shown in Table 2. Finally, we choose the entities representations encoded by PIDE model via comparison with TransE model. Other smarter methods to model knowledge base could be used but this is not our focus.

Table 2 Knowledge base completion results of the different models. (The lower the better for Mean Rank, whereas Hits@10(%) is on the contrary.) fLR: Left Rank; LH10: Left Hits@10(%); RR: Right Rank; RH10: Right Hits@10(%); MR: Mean Rank; MH10: Mean Hits@10(%) g 

4.5 Extending Knowledge Bases

Using the two data set: WN-WR10K and FB-WR14K, we test different models with extending the knowledge bases. Firstly, we directly use the word embedding in knowledge base for extending knowledge base (denoted Un-transform). For FB-WR14k, we first use PCA⁷ to reduce the 1K dimension to 50 dimension for consistent with the dimension of PIDE model. As in ANN model, for WordNet configuration: 4 layers (50 500 200 100), the learning rate:0.0001. We choose the sigmod function as the active function. We train it with the epochs is 500, and we cascade the semantic and syntactic representation that pre-trained by PIDE model as the output of training data (dimension = 100). On FB-WR14K, we also choose 4 lyaers (1000, 500, 200,100), the other parameter is the same. For DNN model, we choose the deep autoencode model, the setting of layer is :( 50 500 1000 200 100) on WN-WR10K and (1000 2000 1000 500 100) on FB-WR14K. We use the typical learning algorithm contrastive divergence for optimization.

We can observe from Table 3 and 4 that the Un-transform method performs worse. I think it exactly indicate that the two latent semantic space: word embedding latent space and entity latent space are not the same. Even the same word or entity, the semantic expression of them are different in the two latent space. We can not directly exchange them from one to another through no additional model. And then we can find that the results is better after we use the ANN as the pipeline for connecting the two latent space. We also could observe from the tables that the DNN performs best. ANN is the typical neural network, and it is hard and may not be effective to optimization. DNN is a deep learning model, it could be trained by contrastive divergence effectively.

Table 3 Results of extending knowledge base in Wordnet with different models. (The lower the better for LR, RR and MR, whereas LH10, RH10 and MH10 are on the contrary.) {LR: Left Rank; LH10: Left Hits@10(%); RR: Right Rank; RH10: Right Hits@10(%); MR: Mean Rank; MH10: Mean Hits@10(%) } 

Table 4 Results of extending knowledge base in Freebase with different models. (The lower the better for LR, RR and MR, whereas LH10, RH10 and MH10 are on the contrary.) {LR: Left Rank; LH10: Left Hits@10(%); RR: Right Rank; RH10: Right Hits@10(%); MR: Mean Rank; MH10: Mean Hits@10(%) }