2.1 Information Retrieval (IR)

Information retrieval has received much attention of the research community given the importance of providing the users with organized and relevant information about a desired topic ⁵. The purpose of an Information Retrieval System (IRS) is to find those documents that meet the user information need, expressed in the form of a query ⁶, p. 221. The “ideal” response of an IRS consists only of documents relevant to the query ⁷. Hence the primary aims of IRS are ⁸, p. 17: to maximize the probability that the query recovers relevant elements, and to produce the best ranking possible according to the relevance probability estimates ³, p. 446.

Some of the important references that have made it possible to analyze strategies used for adapting each IR phase are as follows.

In 2013, Zhang and Zhan ⁹ presented the construction of a retrieval system based on Lucene. The architecture and components of Lucene used for the construction of Compass Framework are described therein.

In 2012, ARGOsearch ¹⁰ was presented, which is an IRS that improves the quality of search in a Library Management System (LMS), supported on a general and extensible architecture based on relevance criteria that orders the results of user queries by considering the similarity of texts, user profile, and statistical information collected by the LMS.

In 2011, Cui, Chen, and Li ¹¹ presented the construction of an IRS for Chinese based on the Lucene model and it is applied to the retrieval of research data of a forestry information system, finding the system to be fast and efficient.

In 2004, Hollink et al. ¹² presented research done on eight European languages (Dutch, English, Finnish, French, German, Italian, Spanish and Swedish), which describes the IR tasks and how they were applied to these languages, obtaining good results.

2.2 Tokenizer

Tokenization is based on the recognition of lexical units of a text. A document (text) is introduced as input and a list of lexical units, or tokens, is obtained as output ¹³. It is one of the first tasks carried out when performing the pre-processing of text ¹⁴. In most cases, tokens match the corresponding words. However, they comprise the so-called lexical units from the text and are known as tokens. Likewise, programs responsible for carrying out this lexical decomposition are called tokenizers ¹³.

A tokenizer has two important properties: (1) It is the only component that requires to be used in any application of natural language processing that involves analysis tasks; and (2) it is always used at the beginning of the analysis process ¹⁵.

Some important studies that reveal the different uses and strategies for text tokenization are as follows.

In 2015, Hammo, Yagi, et al. ¹⁶ described how an IR process can be used for Arabic, studying the semantic changes of the language in relation to the frequency of use of a word over time and how it changes its meaning over time. The tokenizer implemented for this processing provides two types of output: tokens that correspond to lexical units whose characters are recognizable, and tokens that need more morphological analysis, while tokens of one character in length were not taken into account.

In 2011, Jamil et al. ¹⁷ presented a system that initiates the retrieval process with query transformation using tokenization, stemming (Fatimah) and stop-word removal (Bahasa Melayu morphological rules). The IRS was tested with 30 documents and was able to operate effectively.

In 2008, Bijankhan et al. ¹⁸ discussed linguistic aspects taken into account in building a Corpus in Persian and reducing problems when making a suitable tokenization process specific to Persian.

In 2006, Jiang and Zhai ¹⁹ presented the generalization of various tokenization strategies in a set of heuristics organized to be applied to IR in biomedicine, concluding that the removal of non-functional characters is safe and different types of tokenization can be applied.

2.3 Nasa Yuwe Language

Nasa Yuwe is the most important ethnic language in Colombia. It is spoken in various Nasa reservations and settlements across a number of municipalities in the Departments of Caquetá, Putumayo, Meta, Tolima, Valle del Cauca, Cauca, and Huila ²⁰. The Nasa population is currently estimated to total around 200,000 people, of which 75% are active speakers of the Nasa language ². Nasa Yuwe is in danger of extinction. It has been losing ground due to a range of cultural, social, geographical, and historical factors ².

The alphabet, unified only as recently as 2000, consists of 32 vowels (oral, interrupted oral, aspirated oral, long nasal oral, interrupted nasal, and aspirated nasal) and 61 consonants (basic, aspirated, palatalized, nasal, fricative, approximant, among others). The different variants are associated with the geographical area in which the reserves of each community are located. An approximate guide identifies the different variants of Nasa as ²¹: Northern region (Munchique - Tigres, Huellas, Toribío, San Francisco, and Tacueyó); Central region (on one side, Jambaló and Pitayó, and on the other side, Caldono, Pioyá, and Pueblo Nuevo); southern region (Novirao and Paniquitá); and the interior region.

The main works to describe the language are as follows. “Paéz or Nasa Yuwe Grammar” ²² describes the phonology, morphology, and syntax of Paéz or Nasa Yuwe. “A Nasa Yuwe-Spanish Dictionary” ²³ contains vocabulary for Nasa editors and gives a description of Nasa Yuwe based on Spanish grammar. “A look at the Nasa Yuwe of Novirao” ²⁴ provides a phonological and morphological description of Nasa Yuwe in Novirao and draws a comparison with other variants. Finally, “The Paéz language-an overview of its grammar” ²¹ presents a more detailed description of the language.

2.4 Test Collection for the Nasa Yuwe Language

A test collection is the most widely used method for evaluating information systems. As such, in the process of building an IR system for Nasa Yuwe it was necessary to have a test collection for such a purpose. The first test collection for Nasa Yuwe ²⁵ was therefore built. Its description is shown in Table 1; it is published at the webpage with the URL http://www.ewa.edu.co/coleccion.

Table 1 Atributes of the Nasa Yuwe collection. Source: ²⁵  

The documents that form this collection were selected following fieldwork. They are written in the unified alphabet and treat topics on the Nasa culture and worldview. The queries were performed by prioritizing the issues most in need of information in reference to the selected texts. The work was carried out with expert speakers and teachers of the language, and finally the queries were obtained. To make the classification of the relevant documents for each query, the judgment of experts for each document-query pair was used, since there were no previous judgements nor pre-selections from other IR systems. The experts used a scale of four levels of relevance in order to return a judgment as follows: very relevant, relevant, not very relevant, and not relevant. In the collection, 63% of the documents were relevant for one or more of the eight queries ¹⁷. It was also established that Nasa Yuwe complies with Zipf’s law ¹⁷.

3.1Defining the Baseline

To begin, the Lucene standard tokenizer (ST) ²⁶ was selected (version 2.9.4). It separates the words by punctuation characters, removing punctuation marks, but full stops that are not followed by a blank space are considered part of the token; it separates words joined together by hyphens (-) unless there is a number contained in the token, in which case they are not separated. It also identifies website addresses, emails addresses, and acronyms ²⁷. Secondly, the baseline was established using the Lucene standard tokenizer, making use of the queries and documents in the test collection ¹⁷. Thirdly, the precision-recall curve was selected as a measure of evaluation, taking into account the result of each of the eight queries defined in the test collection.

Finally, with these elements it was possible to establish the baseline using different processing techniques on texts written in Nasa Yuwe, such as: (1) the Lucene standard tokenizer; (2) a Lucene standard filter used to remove, among other features, the 's word endings and the full stops in acronyms ²⁷; and (3) a converter of words to lowercase (LC) added to the Lucene standard tokenizer ²⁷.

3.2 Process of Adaptation to Nasa Tokenizer

Once the processing of the test collection texts has been carried out with: (1) ST; (2) ST + Filter; and (3) ST + LC, the difficulties in processing text written in Nasa Yuwe were identified. The aim was then to resolve these difficulties, looking to obtain a Nasa tokenizer that correctly processes texts written in that language.

In Table 2, a list of words from a snippet of a Nasa document from the collection is presented along with the tokens obtained using the standard tokenizer. A high number of errors were seen to occur even in such small snippets. It was therefore necessary to focus attention on identifying the causes of the errors and detecting a solution. These are explained as follows.

Table 2 Tokenized text snippet 

The Nasa tokenizer therefore unifies these pairs of characters into the character to which they correspond. It was thus necessary to make an initial unification both in the documents in the collection and in the terms of the query. Table 6 presents some data about the errors previously described, highlighting the impact of each one and showing the importance of correcting them.

Table 6 Data about the errors 

Figure 2 summarizes the additional steps that the Nasa tokenizer carries out compared to the standard tokenizer.

Fig. 2 The resulting Nasa tokenizer (NT) 

3.3 Performance Evaluation

With the Nasa tokenizer already implemented, we proceeded to carry out various comparisons for each case of the baseline contrasted with the Nasa tokenizer (NT), determining the quality of the test collection text tokenization and establishing the various conclusions.

4.1 Baseline

Figure 3 shows the performance of the standard tokenizer, standard tokenizer with filter, and standard tokenizer with the converter of text to lowercase (LC).

Fig. 3 Baseline for standard tokenizer for Nasa Yuwe 

The filter removes endings but preserves Internet addresses, e-mail addresses, some nouns that contain numbers, etc. Figure 3 shows that:

4.2 Nasa Tokenizer (NT)

First, in the process followed in order to obtain the Nasa tokenizer, several precision-recall curves were analyzed by correcting each of the errors, each curve giving better precision values until Figure 3 was obtained. Figure 4 shows the precision-recall curve that presents the Nasa tokenizer compared with the standard tokenizer over the eight queries on the test collection. In this figure, it can be seen clearly that the Nasa tokenizer obtains better precision values in the first recall values, which is lower than 40%.

Fig. 4 Comparison between NT and ST 

The filter was then applied to the Nasa tokenizer and no changes were obtained in the values for the precision-recall curve.

Next, the converter of text to lower case (LC) was applied to the Nasa tokenizer and compared with the standard tokenizer, as shown in Figure 5, where it can be seen that in general the precision-recall curve for NT + LC shows the best precision values, with the exception of the first value (a recall of 10%). The latter is understandable given that as the standard tokenizer separates terms incorrectly, the chances are increased of finding similarities in the words in the queries and the documents. For example, the words a’te, a'te', a'te's, a'tek, and a'tepa', on being incorrectly tokenized (a te, a te, a te s, a te pa) produce greater possibilities of coincidences than in the case of the Nasa tokenizer, where these words are different.

Fig. 5 Comparison between ST + LC and NT + LC 

Table 7 provides a summary of the best results that can be obtained with the proposed tokenizer.

Table 7 Text snippet correctly tokenized