1 Introduction

International standards for software documentation revision stand out that well-designed documentation not only assists users and helps to reduce training and support costs, but also enhances the reputation of the final product.

Through processes, such as verification, validation testing, and expert review of content during the development of a software product, developers and managers obtain valuable feedback about the accuracy and usability of their work [²⁰, ²⁷]. Throughout the State-of-the-Art, it has been stated that non-functional requirements are a key aspect to the success of a project [¹, ¹¹, ¹⁰, ²⁸].

For example, the non-functional requirements related to usability issues [¹⁰, ²⁸, ²³, ²², ²⁶], such as usability and system testing of documentation [²⁰].

Notwithstanding, it is also known that the requirements engineering community has not established a definition for non-functional requirements as well as a methodology or process that allows eliciting, documenting, and validating them [⁶, ⁵, ¹¹, ¹², ⁹, ²⁵] in an easy and transversal multidisciplinary manner.

Therefore, this article aims to address one of the gaps previously mentioned: a methodology that supports the assessment of a non-functional requirement through a set of metrics and a qualification criterion.

This approach directly benefits development enthusiasts, software development companies, the engineering community and researchers in general who seek to maintain adequate documentation based on internationally accepted guidelines and standards without having to prepare different documents depending on a particular evaluation agency.

Specifically, the proposed metrics and methodology allow any software development, including those services that comprehend any integration of Artificial Intelligence algorithms, to have a suitable, fast, and multidisciplinary way to accelerate the incorporation of improvements through the life cycle of a product and guarantee its quality.

The main contributions of this article are four-fold:

The rest of the article is organized as follows. Section 2 provides the State-of-the-Art related to software quality, emphasising concepts and definitions associated with functional and non-functional requirements, and evaluation standards focused on testing software as well as documentation.

Later, Section 3 presents the proposed evaluation metrics and guideline to support the testing of documentation according to the non-functional requirements: soundness and completeness. Finally, Section 4 provides the conclusions of this research.

2 State of the Art

Nowadays, software products offer increasingly robust functionality in both within applications and across multiple platforms. In order to offer this, software designs include underlying programming mechanics, rules, calculations, and algorithms that are only discernible through several testings. In consequence, user documentation endures as an essential component of software products [²⁷].

As previously stated, there is a need for a standardized evaluation metric and/or methodology that can be used transversally in the different application fields of the documentation of an API. Consequently, this section presents the foundations and standards that support this research work.

3 Proposed Evaluation Metrics

As pointed out in the State-of-the-Art, the process of documentation evaluation involves several qualities (aspects) that developers or specialists must take into account to ensure the overall quality of the documentation.

This article provides metrics focused on the evaluation of end-users’ documentation by supporting the testing of non-functional requirements of Application Program Interfaces (APIs). Specifically, this proposal allows evaluating the soundness and completeness qualities of the documentation available on the official websites of APIs.

The proposed metrics aim to help the development of an accurate, usability, and completeness users’ documentation of APIs within the life-cycle of product development.

It is noteworthy to mention that, as similar to the ISO/IEC/IEEE standards previously described, the proposed evaluation metrics can highlight problems with the evaluated API; however, resolving these problems is out of the scope of this research work.

The proposed metrics aim to maintain an accurate and useful content comprehended in the APIs’ documentation. Furthermore, to facilitate the documentation evaluation, a guideline that comprehends the metrics is proposed.

This guideline aims not only to help developers and/or specialists to ensure an accurate and complete evaluation of the documentation of APIs during the life-cycle of product development but to ease the tasks during the usability evaluation.

To achieve this goal, several previously presented aspects are concentrated in a guideline in a structured form that facilitates both the evaluation of documentation during the non-functional evaluation and the management and control of updating tasks. The general structure of the proposed guideline is shown in Figure 1.

Fig. 1 General structure of the proposed evaluation guideline 

To seek clearness and fluency, the proposed metrics are concentrated in four categories according to their evaluation purpose. The four established categories are: General overview, Installation, Troubleshooting, and Functional test sub-process.

The categories and their respective metrics are detailed in Table 1, Table 2, Table 3, and Table 4.

The structure of the evaluated aspects in each category is concentrated in four columns to ease the evaluation process.

Once the metrics that help in the evaluation of non-functional requirements are defined and a guideline is provided, metrics for evaluating the completeness and soundness qualities of documentation must be selected.

In consequence, the evaluation metrics taken into account need to be mapped regarding their purpose of evaluating these two qualities. The selection of metrics that integrates the completeness and soundness qualities measurements were based on the standards presented in Section 2.

Despite that all metrics concentrated in the guideline are relevant for the overall documentation evaluation, not all of them are required to evaluate the two previously introduced qualities. Therefore, 23 metrics were considered to evaluate the completeness quality and 27 metrics were contemplated to measure the soundness quality. The metrics to evaluate the completeness and soundness qualities are detailed in Table 5.

Afterward, a criterion must be contemplated to give a rating to the tested documentation. This rating will serve as an easily readable base-line for users and developers to notice the quality of the documentation that is reviewing. The main goal of this rating is to simplify the process of reviewing documentation of software, specifically APIs documentation.

Hence, a straightforward criterion that adds only the positives values of the metrics in each quality is applied. That is, the rating score of the qualities will be established by adding only the “Yes” values of each metric that embrace that quality.

Metrics values of “No”, “Partially”, and “NA” are not taken into account since these values represent that the corresponding information evaluated for that metric is incomplete or is missing. Consequently, the maximum value of the criterion for the completeness quality is 23 and for the soundness quality is 27.

The addition of these qualities by themselves does not help to get an insight into the documentation’ quality. Therefore, a seven rating scale is chosen to provide an abstraction and clearance about the quality of the documentation tested.

The rating scale scores are: 1) “Very Poor/Bad”, 2) “Poor”, 3) “Fair”, 4) “Average”, 5) “Good”, 6) “Very Good”, and 7) “Excellent”.

This representation is friendly-user and easy to understand by both developers and general users of the documentation. Moreover, the seven rating scale clearly defines the status of the documentation qualities.

Following the straightforward approach, the maximum value of each quality is divided into this seven rating scale to obtain the ranges that will determine each score on the scale. The ranges of the ratings of the completeness and soundness qualities are concentrated in Table 6.

As noticed from Table 6 the “Excellent” rating is the only one that has no range of scores, being achievable only if it has all metrics covered with the “Yes” value. By scoring the maximum rate in this form, the quality evaluation of the documentation ensures that there is no lack or deficiency in any aspect of the content.

On the opposite, the minimum rating can not be expressed through the zero value for both qualities. The zero value itself could represent that the documentation does not have any single metric with the “Yes” value or that there is no documentation at all.

Thus, to concentrate both representations with no distinctions for the purposes of evaluation, the zero-score is integrated into the range of the “Very Poor/Bad” rating.

4 Conclusion

From the State-of-the-Art, it can be noticed that there is a need for the engineering community, to reach a consensus and define a methodology that can evaluate the non-functional requirements of a project in an easy, multidisciplinary, and standardized manner.

This will allow a fast and reliable way to review and integrate improvements in the documentation during the development of software products. In this regard, the proposed approach aims to obtain an accurate and complete non-functional evaluation of completeness and soundness qualities of any API.

The proposed metrics, the methodology, and the guideline allow delivering high-quality documentation of APIs to developers within the minimum time required.

This is achieved by taking care of relevant metrics that are of interest to developers during the development of documentation in the software’s life-cycle process. Therefore, the proposed metrics concentrated in each guideline make it possible to ensure a comprehensive review of the soundness and completeness qualities of the documentation since these metrics are based on the standards introduced in the State-of-the-Art.

Moreover, another advantage of following the approach proposed in this work is that after evaluating the API’s documentation with the proposed metrics concentrated on the proposed guideline, several relevant qualities and aspects related to the usability of the documentation will also be covered before performing a process of usability evaluation by itself.

Consequently, the APIs’ documentation will require less effort in the validation and correction tasks during the usability evaluation. Thus, developers and end-users can rely on to have an accurate, useful, and friendly API’s documentation. Consequently, the proposed approach can be applied to other types of domains that need to evaluate their software documentation.

Finally, the approach of this proposal supports following one of the recommendations established in the standard ISO/IEC 26514:2008 [²⁷]; that is, to obtain high-quality software documentation, its development, verification, and evaluation should be an integral part of the software life-cycle process and should not be regarded after the software’s implementation.

Further work could focus on developing a form of pilot evaluation to provide a complete and reliable non-functional evaluation model. For this, the incorporation of more evaluation metrics related to other qualities of the documentation would have to be addressed, thus achieving a complete evaluation of properties related to the non-functional evaluation, such as usability.

Moreover, standards and frameworks related to risk management and governance of developments incorporating Artificial Intelligence, such as [¹⁸, ²⁴], will be included to strengthen the proposed metrics and evaluation methodology. Furthermore, to test the proposed approach, the cloud-based technological platform called FIWARE [⁷] will be considered.

The test case will focused on testing the installation manuals of the FIWARE Generic Enablers (GEs) [⁷], whose task is also relevant to the FIWARE initiative itself [⁷], by using the set of metrics, the qualification criterion, the assessment methodology, and the guideline proposed in this research.