2.1 Content-Based Recommender System

As explained in the previous section, it may be the case that a database with certain grades d (for instance, movie ratings) is incomplete due to the fact that not all users have watched every movie. To complete this rating dataset, a method proposed by ¹⁸ called content-based recommender system can be used. Following the example, assume that for each movie we possess a feature vector x¹,…,x^m containing α movie features (i.e. amount of romance, amount of action, etc.). Then, for each user we learn likewise a feature vector θ ¹, ...,θ^u that represents the user's appeal for the α movie features. With this information, we are able to predict the user's movie rating d using the following calculation:

(1)

where θⁱ represents the user's feature vector, x^j represents the movie's feature vector, and Τ denotes the transposed matrix.

In this methodology, two main drawbacks arise. First, to learn the user's appeal for a movie θ¹,...,θ^u , we would need to have some kind of information that explicitly or implicitly describes it. Based on the user's previous ratings, we could perform a linear regression minimization ¹⁸ to find the values that most appropriately describe the users. Second, we would need to know the features of each movie x¹,…,x^m by watching all movies one by one and identifying their a features. Even if these two problems are solved, all of these features are subjective and vary from case to case, since we cannot confirm nor deny that a certain movie has a discrete amount of features such as romance or action.

2.2 Justification of a New Algorithm

As it has been exposed, content-based recommendation is effective when we possess the information of either the ranker or the ranked object, but when this information is not explicit or logical to extract, we need to explore more possibilities. In order to increase the accuracy of a systematic review and meta-analysis where some data is missing, such data must be neither ignored nor completed randomly but via statistical methods. Moreover, this data must reflect a good approximation of what such study would have presented if such outcome had been evaluated.

3.1 Basic Definitions

Given a data matrix Y of size u x m, where u represents the number of articles that study some outcome or users that rank some phenomenon, and m represents the number of outcomes studied or features ranked, certain data e may be present and some other data t may be missing (ø) due to the reasons explained in Section 1. Once confirmed that the total number of data d = u· m = |e| + |t|, where |e| and |t| represent the cardinality of sets e and t, respectively, we first define a logical matrix R of size u x m, where

(2)

Before processing the current data, a normalization process is suggested, given that many ML methods, in particular the ones related to recommender systems, work better with prenormalized data to avoid large deviations in the calculated data ¹⁸. We propose a 0-1 normalization by first calculating a vector of minimum values min_1,j and a vector of maximum values max_1,j for every 1 ≤ j ≤ m.

(3)

Notice that the normalization must be only performed for data as long as R_i,j = 1 for such data position. Once the data in Υ is normalized and Yⁿ is obtained, we calculate a mean vector µ_1,j for every m as long as R_i,j = 1. This is done to have values on each feature vector with a zero mean.

(4)

Due to equation 4, the values of Y^s will not be in the range of 0 and 1. Nevertheless, this will not be a problem given the real purpose of normalization was, as commented before, to avoid large data variations. Other methods, such as standard score normalization (applying first equation 4 and then dividing by the variance), may be applied for this purpose as well.

Once d ∈ e have been normalized and Y^s has been obtained, we calculate the data's covariance matrix C_i,j , verifying that the dimensions of Y^s and C agree. Afterwards, we apply to the covariance matrix C _i,j any ML algorithm such as PCA ²⁰, k-nearest neighbors ²¹, or SVM ²² to obtain the eigenvalues vector Ω_1,j and the eigenvectors matrix ɛ_j,j. Other approaches such as the Singular Value Decomposition (SVD) have been discarded, given they work as dictionary approaches, whereas our goal is to complete the data without explicitly relying on it. Also, non-linear and non-parametric learning spaces have not been considered for this solution given that ranking systems and medical outcomes usually follow a linear pattern.

Using the eigenvectors matrix ɛ_j,j we apply the function

(5)

to find the weight's matrix W. By performing the inverse operation of equation 5, we calculate

(6)

thus obtaining the normalized and centered values which complete dataset Y. In Y', the estimated values for all d∈ e and d ∈ t are contained, therefore we must only select the data d ∈ t which completes the missing values of Y (where R_i,j = 0).

After applying first the inverse operations of equations 4 and 3 (in such order) on Y', we can compute a final completed dataset Y'' by using the following rule:

(7)

For the case that new users or updated data have to be inserted into the database, the whole process must be executed from the beginning. Therefore, this new information is inserted in the original dataset Y, and then the method is run from scratch to complete every d ∈ t based on the new information.

3.2 Tuning the System's Variance

Not every time we perform this process we need to use the whole eigenvector matrix ɛ _j,j As explained in ¹⁹, if we desire the system to have a certain retained percentage of variance n, we can use the eigenvalues in vector Ω _1,j to reduce ɛ_j,j into a submatrix ɛ'_j,k with only the first k columns of the original one. This tuning allows us to define the variance retained by the system.

If we want a retained variance of n percent, Ω _1,j must be normalized by repeating the steps made with equation 2. Afterwards, we execute the algorithm shown in Figure 1 to obtain k.

Fig. 1 Algorithm to calculate k for an accuracy of η percent using the eigenvalues vector 

Typically, a 95-99% retained variance is used when applying a learning algorithm.

4.1 Application in a Recommender System based on a Movie Rating Database Scenario

To evaluate the functionality of our proposal, the first tests involve the use of the Movie Rating Database Scenario ¹⁸. This database was specifically designed to work with the state of the art content-based recommender systems described in section 2.1. Even though this dataset is not related to medical research fields at all, it possesses every characteristic that appeals to our method. Consider m = 1682 movies existing in a certain movie server and u = 943 registered users that could watch those movies and assign to them a rating based on a scale y, where y = {1,2,3,4,5} represents the user's opinion ranging from "very bad" to "very good". Since it is very plausible that not all users have seen all movies, many ratings are missing in this rating dataset Y.

Thus, the database counts with 100,000 ratings distributed unevenly for every user and it represents barely 6.31% of the total possible ratings. For this dataset, the authors provide both the feature vector θ_i for every user u and the feature vector x_j for every movie m, where θ_i contains α = 10 types of "ground truth" movie features (i.e. romance content, action content, etc.) and x_j contains a = 10 types of "ground truth" movie appeals (i.e. romance appeal, action appeal, etc.). Notice that the α features are the same for each θ_i and x_j , respectively.

As explained in section 2.1, having this information is highly unlikely in a real scenario, not only since it would be a long and exhaustive work, but also because intending to map a feature such as "level of action in a movie" or "amount of user's attraction to a romantic movie" onto a numerical scale is very difficult and subjective.

For a first validation, we compared the state of the art content-based recommender system method (SOA) with our proposal (OUR) by implementing a 100-fold cross validation ²³ with the 100,000 preexisting ratings of the database. This type of validations is especially useful to detect if any of the two methods is incurring in data overfitting.

We split the preexisting ratings in 100 random partitions Ρ containing 1,000 ratings each and ran each method 100 times, each time leaving one partition out of the training step. Afterwards, we measured the total average errors φSOA- and φOURk- between the ratings obtained by the ML methods and the ratings in the left-out partition, using equation 8 and 9 respectively:

(8)

(9)

The results of these tests are shown in Figure 2. For the case of OUR, each cross-validation test is performed for every possible value of k in order to test different levels of retained variance. First, we observe that OUR reports error values deviation slightly higher than SOA. Moreover, notice that SOA reports a constant value since this method does not depend on the parameter k. Nevertheless, OUR method does not use any previously compiled feature vectors θ _i and x _j , thus it can be considered effective given the slight difference with the error computed by SOA. Finally, the low and constant values for the total average errors on both lines indicate that none of the methods was overfitting data.

Fig. 2 Final Average Errors φSOA- and φOUR- with respect to k (parameter in our method). For k>40, the value of φOUR- remains constant 

In the second evaluation, our goal is to obtain the missing t = 1,486,126 rankings (93.69%) to compute Y" _i,j and eventually calculate a final rating vector for each movie. For this purpose, we applied SOA and OUR to the original dataset Y in order to obtain the complete uxm dataset matrix Y _SOA " and Y _OUR ", respectively. Once again, OUR was computed for every possible k. In this case, we registered the average error φ between the results obtained with OUR and SOA assuming the ratings obtained by SOA were the ground truth. For this purpose, the following equation was used:

(10)

By multiplying the difference of both datasets by the term (1 = R _i,j ), the error is calculated only between the ratings that were completed by both methods and not on the preexisting ones.

In Figure 3 we present the results for φ_k, where we can appreciate that the method obtains the lowest error φ_k = 0.58 rating at k = 17 (96.7%) variance according to the eigenvector tuning. The average error is kept constant with an error of φ_k = 0.64 rating at k > 40.

Fig. 3 Average error φ _k with respect to k (parameter in our method). For k > 40, the value of φ_k remains constant 

We consider that having an error of φ_k = 0.58 rating in a dataset where 93.69% of the data was completed is a very good outcome, since predicting a value for each movie m with around half a rating of difference would not diverge considerably from the user's real opinion. In fact, using the worst k scenario (k = 1) results in an error of φ_k = 0.7 rating, which still is a very good reflection of the ground truth ratings.

The database and code used for these tests is available in ²⁴.

4.2 Application in a Medical Research Systematic Review and Meta-Analysis Scenario

As noted before, one of the most well-known forms to compare the effectiveness of several medical studies is by performing a systematic review and meta-analysis. Nevertheless, it is common that not all of the selected studies have used the same outcome to measure the effectiveness of their intervention. For this reason in the second scenario presented, we will show how our method could be applied to complete missing data in a dataset of outcomes that intend to measure medical effectiveness. This data was extracted from a systematic review performed in ²⁵.

The systematic review proposed in ²⁵ aimed at comparing the effectiveness of studies across Europe whose main purpose was to reduce obesity in children. After a rigorous inclusion and exclusion process where multiple health study sources were screened (i.e. PubMed), we selected u = 34 studies which satisfied certain criteria such as number of participants, age of participants, among others. The whole list of selected studies can be found in ²⁵, but for the reader to have a reference of the used data, u ¹ = [10] and u ⁷ = [11]. Later, we collected the measurements m that each study used to demonstrate whether they considered that their intervention prevented childhood obesity or not. We collected only the outputs that belonged to one of the six different m ⁿ shown in Table 1.

First, it is important to note that for the case of outcomes m¹, m², m⁴ and m⁶ , the ideal aim of a study is to decrease their values. Contrarily, for outcomes m ⁴ and m ⁶, the aim would be to increase them. Additionally, every measurement has a different unit associated. To solve both issues, we calculate for each measurement the Effect Size (ES) with the double difference method ²⁶. This way every measurement is replaced by a number on a scale where m ⁿ ≤ 0 represents ineffectiveness, 0 < m ⁿ ≤ 0.2 represents low effectiveness, 0.2 < m ⁿ ≤ 0.5 represents medium effectiveness, and m ⁿ > 0.5 is considered high effectiveness when intending to improve such outcome. The resulting dataset Y is shown in Table 2. Notice that if our ML method is not used and we only consider the existing effectiveness measures, an immediate observation would be that, for instance, u¹ was a less effective study that u ⁵.

After applying our ML method to generate 7" (shown in Table 3), several interesting observations can be drawn from the resulting dataset, even if no comparison with some kind of ground truth information is possible. For the previously stated example, both u ¹ and u ⁵ would now have a combined effectiveness score of 1.22 (adding the values of each row), indicating that both studies were equally effective although they used different outcomes and obtained different results. This situation is very plausible in medical research, given that certain studies are better at improving certain outcomes than others.

Moreover, notice that the results that have been completed present a low deviation from the original data, such as in the m ² outcome, where the completed results range from -0.02 to 0.48, thus ensuring that none of the completed data is below or above a calculated value. Also, the fact that a certain study did not present any positive ES does not necessarily imply that the rest of outcomes will be negative as well, but it will decrease such values. That is the case of u¹⁶ which only presented the outcome m¹ = -0.28. When the rest of data is completed, we notice that only positive values were added. Nevertheless, this study only scores a total effectiveness of 0.86.

This particular database presented |t| = 136 (66.6%) data to be completed using a 99.23% variance for the eigenvector tuning (k = 4).