2.1 Overview

Here, we construct word networks using Doen et al.’s method [⁴], based on a large newspaper article dataset (Figure 1). The procedure is as follows:

Fig. 1 Network construction overview (Source: Figure 1 of the paper [⁴]) 

In this work, we also remove unrelated words from the network by adding additional procedures to Steps 4 and 5 by referring to the method proposed by Doen et al. The method is explained in detail in the following sections.

2.2 Extracting Candidate Nodes

Denote the initial keyword by k. First, we extract the articles that include k, and denote the resulting article set as S. Next, we use morphological analysis (specifically, ChaSen [⁷]) to extract the nouns from S. Here, we eliminate single-character words and those consisting only of hiragana characters (Japanese functional characters) or figures, because such words are unlikely to be important. Then, we use the remaining words as candidate nodes.

2.3 Selecting Nodes using TF-IDF-based Related Word Extraction

From the candidate nodes, we select the ones to actually add to the network via TF-IDF-based related word extraction. Specifically, we score the words using a TF-IDF based method and identify the five highest-scoring ones as being most closely related to the keyword. Then, we add the highest-scoring candidate words to the network as nodes, and use the TF-IDF scores as edge weights.

The TF-IDF based related word extraction method produces scores indicating the importance of particular words (candidate nodes) in the extracted articles, which are calculated as follows:

Here, tft is the frequency with which candidate node word t appears in the extracted articles, while dft is the number of articles where the candidate node word t appears in the entire dataset and N is the total number of articles in the dataset.

Equation 1 uses dft to give low weights to less-important words that appear in many articles, and conversely gives high weights to significant words that only appear in a small number of articles.

2.4 Expanding the Network

First, we extract five words based on the keyword k, as described in Section 2.3, and connect them to k. Next, we expand the network by extracting five additional words, based on using the first word a of the initial five as a new keyword, and connecting them to a (Figure 2). Then, we continue to expand the network by iteratively repeating this process.

Fig. 2 Example of building a network by extracting nodes (Source: Figure 3 of the paper [⁴]) 

2.5 Deleting Unrelated Words

Next, we use the topic-restricted extraction method proposed by Doen et al. to delete unrelated words from the network. This involves one change to the method given in Section 2.1. Specifically, while extracting articles by repeating Step 5, we only extract those that include both the initial and current keywords. Since this means we are focusing on articles that include the initial keyword, we are likely to obtain words related to it, and unlikely to extract unrelated words (Figure 3).

Fig. 3 Overview of topic-restricted extraction (Source: Figure 4 of the paper [⁴]) 

4.1 Methods

In these experiments, we constructed word networks for the theme keywords “Toyota,” “universe,” and “Greece.” These networks consisted of 191, 228, and 99 word pairs, respectively. To build the networks for “Toyota” and “universe,” we used 102,547 articles taken from the Mainichi Newspaper (all from 2014). To build the network for “Greece,” we used 92,807 articles taken from the Mainichi Shimbun (all from 2010). Figures 5-7 give the networks for “Toyota,” “universe,” and “Greece,” respectively. These show that all the networks used here consisted of four levels.

Fig. 7 Word network for “Greece” 

4.2 Evaluation based on Human Judgment

Next, we evaluated whether or not the character strings given to the network links were appropriate. For this, we used 20 randomly chosen word pairs from each network (“Toyota,” “universe,” and “Greece”), for a total of 60 word pairs. In addition, ten randomly chosen newspaper articles including each word pair were used as reference data.

The pairs were then evaluated by a human participant against the top five highest-priority strings (as determined in Section 3), as determined by each of the three priority equations. The participant evaluated them on a four-step scale after consulting the reference newspaper articles. Hereafter, we will describe the methods embodied by Equations 2, 3, and 4 as “high-frequency,” “short,” and “ratio,” respectively. Tables 2–5 show the evaluation criteria and examples representing each of the four possible grades.

The example in Table 2 was evaluated as “Good” because the character string essentially says “astronaut Wakata Koichi has become the ISS captain,” which was judged to appropriately indicate the relationship between “flying” and “captain.” Similarly, the example in Table 3 was evaluated as “OK1” because, although the character string appropriately indicates the relationship between the two words, it also includes the additional phrase “at 7:58 am on the 14th.”

Next, the example in Table 4 was evaluated as “OK2” because, while the character string does indicate the relationship between the two words, it is missing information such as the person’s name and thus does not make the relationship as easy to understand as possible. Finally, the example in Table 5 was evaluated as “Bad” because the character string differed significantly from the correct information, as determined from the reference data, so it was judged as not appropriately indicating the relationship.

4.3 Evaluation Using the Mean Reciprocal Rank (MRR)

After conducting the manual evaluation described in Section 4.2, we evaluated the results using the MRR, based on the highest-ranked correct answer among the top five highest-priority results for each target. We calculated the MRR as follows:

where N is the total number of targets to be evaluated and ri is the rank of the highest-ranked correct answer produced for target i. In this study, since we output the top five results, 1≤ri≤5. Here, we considered three evaluation criteria: only “Good” answers are correct; “Good” and “OK1” answers are correct; and “Good,” “OK1,” and “OK2” answers are correct.

4.4 Evaluation Using the Top-n Accuracy Rate

Next, we evaluated the results based on the Top-n accuracy rates by assessing which of the top five highest-priority outputs corresponds to the correct answer. In the Top-n accuracy rate, a value of 1 is given in each question when the top n candidate answers contain the correct answer.

The Top-n accuracy rate is the result dividing the sum of the given values by the number of questions. In this study, we evaluated the Top-1 and Top-5 accuracy rates.

As for the MRR evaluation, we considered three correctness criteria: only “Good” answers are correct; “Good” and “OK1” answers are correct; and “Good,” “OK1,” and “OK2” answers are correct.

4.5 Evaluation of the First String Extraction Method (Periods and Commas)

In this section, we evaluate the string extraction method based on using periods and commas as delimiters. First, we randomly selected 20 word pairs each from the “Toyota,” “universe,” and “Greece” networks, for a total of 60 word pairs. For each word pair, we used the proposed method to extract the five highest-priority strings, then evaluated them in terms of their MRR, Top-1 accuracy rate, and Top-5 accuracy rate.

The experiments were carried out as described in Section 4.1, while the evaluations were carried out as described in Sections 4.2, 4.3, and 4.4. Table 6 shows the MRR evaluation results, while Table 7 shows the Top-1 accuracy rates and Table 8 shows the Top-5 accuracy rates. In addition, Tables 9–11 show some example outputs. “High-frequency,” “Short,” and “Ratio” corresponds to the use of Equations 2 to 4.

4.6 Evaluation of the Second String Extraction Method (Periods Only)

In this section, we evaluate the string extraction method based on using just periods as delimiters. The experimental conditions were as before (Section 4.1), and the evaluations were carried out as described in Sections 4.2, 4.3, and 4.4. Here, we used the same 60 word pairs for evaluation as in Section 4.5. Table 12 shows the MRR evaluation results, while Table 13 shows the Top-1 accuracy rates and Table 14 shows the Top-5 accuracy rates. Tables 15–17 show some example outputs.

5.1 Assigning Strings to Links

Assigning character strings to the links made it possible to identify word relationships that would otherwise have been difficult to understand. However, it was not always possible to extract suitable character strings to attach to the links, because when punctuation marks were present in the character string between two words extracted by the proposed method, the character string was omitted and no character string could be extracted. This issue was particularly prevalent for word pairs that only appeared in a small number of articles. In future work, we plan to improve our approach in this respect.

5.2 Priority Equation

Next, we investigated the equations used to determine the priorities of character strings in the method using both commas and periods. We considered three priority equations: one emphasized the frequency with which the string appeared, another emphasized short character strings, and the third based the priority on the ratio between appearance frequency and length.

First, we found that the equation that focused on short character strings had the lowest performance according to all evaluation methods and criteria. We believe this was because its emphasis on short strings reduced the amount of information available to indicate the word relationships.

Next, we examine the equations emphasizing appearance frequency and the frequency/length ratio. These both produced character strings that appropriately represented the word relationships, and we found that they both yielded nearly equivalent performance when we considered output strings with extra or missing information to be correct. However, the frequency-based equation performed slightly better when only strings that properly indicated the word relationships were considered correct, and also when we allowed answers with additional information.

In addition, although the Top-1 accuracy rates for the two equations were equal, the Top-5 accuracy rate of for the frequency-based equation was higher than that of the ratio-based equation. This indicates that the frequency-based equation is likely to produce more correct answers among the top five outputs. Given that, we believe that emphasizing the occurrence frequency makes it easier to acquire character strings that appropriately indicate the word relationships.

5.3 Use of Commas and Periods

We used two methods of extracting character strings, one that only uses periods to divide the strings and another uses both commas and periods. These produced very similar evaluation results when we considered answers with additional or missing information to be correct. However, when we focused purely on answers that appropriately indicated the word relationships, the method using both commas and periods performed better. Conversely, when we allowed answers including additional information, the period-only method performed better. We believe this was because the character strings produced using periods as delimiters were longer and thus included many strings with extra information.

5.4 Discussion Comparing our Proposed Method and Other Methods

Murata et al. [⁹] proposed a method of using strings between two words in a sentence as the relationship of the two words in Japanese sentences. The definition sentence for an entry word “snowy moonlit night” is “a moonlit night with the presence of snow.” This shows that “snowy moonlit night” consists of two terms, “snow” and “moonlit night,” and the relationship between them is expressed using the phrase “with the presence of.” They extracted relationships such as “that is in,” “that has,” and “that was made from.”

On the other hand, in our study, a substring containing two words is extracted as representing the relationship between the two words. A substring containing two words has a wider range than a string between two words and can show the relationship between two words more clearly than a string between two words.

In the example sentence of Figure 4, “Koichi Wakata astronaut (universe flying pilot): Captain ISS, communicate with the prime minister,” the string between two words “flying” and “captain” is only “pilot):.” “pilot):” is inadequate as an expression between two words. In contrast, our method gets “Koichi Wakata astronaut (universe flying pilot): Captain ISS.” “Koichi Wakata astronaut (universe flying pilot): Captain ISS” has more information than “pilot):” and shows the relationship between two words in an easy-to-understand manner. Our method is superior to the method using a string between two words.

We conducted experiments using the method based on strings between two words in a sentence. We used frequency corresponding to the use of Equation 2 in the experiments. The results are shown in Table 18. The results were lower than those in our methods. We confirmed that our methods are more effective than the method of using strings between two words.