Introduction
Tuberculosis (TB) has been historically linked with adverse socioeconomic conditions. However, in recent decades, other related factors such as Human Immunodeficiency Virus, the emerging of resistant strains, and exacerbated poverty have been reported, particularly in low-income countries. This disease is still a serious public health issue due its economic and clinical burden1.
Given its worldwide impact, the World Health Organization (WHO) poses it as a global health priority, calling on countries to develop and implement prevention and control strategies adjusted to each environment’s features and needs2.
The basic tuberculosis control strategy is interrupting contagion, cutting the transmission chain between host, reservoir, and infectious agent3. Therefore, WHO and CDC have developed guidelines on tuberculosis prevention and transmission in healthcare institutions. These guidelines suggest activities to improve practices on tuberculos is prevention and transmissioncontrol within such settings. Control measures established by those agencies are divided into administrative, environmental and respiratory protection measures4,5.
Tuberculosis events within healthcare facilities have gained importance, by putting patients and health staff at risk, being the most contagion vulnerable settings, as all kinds of patients with several different conditions and illnesses access these facilities. Thus, it is a must for healthcare institutions to adopt and incorporate protective measures for patients, visitors, and health staff6.
Countries such as Peru have incorporated measures enactedbytheseagenciesintotheirownoperationalplan7. In Colombia, there are few studies regarding implementationof tuberculosis control measures in healthcare facilities8, which is why it is necessary to carry out processes of research and validation for instruments that provide support on this matter. Therefore, this research article will show the results of the design, construction and validation of this instrument, taking into account the administrative, environmental and respiratory protection components that health establishments must take into account for compliance with infection control measures. Administrative controlmeasures are those measures that define the health establishment with the objective of preventing and controlling the transmission of tuberculosis between health personnel and patients; Environmental control measures are aimed at reducing the concentration of infectious droplet nuclei and finally respiratory protection measures seek to protect health personnel through personal protectionelements.
The existence of these measures will help to address the problems related to the risk the tuberculosis infection.
The purpose of this study was to validate an instrument that will assess infection control compliance of Healthcare Providers in Cali (Colombia).
Materials and methods
A cross-sectional study, previously approved by the Ethics Committee of Libre University.
Instrument Formulation (domains identification):
The development of the instrumet by the author, considered sections of guidelines for preventing the transmission of Mycobacterium tuberculosis in health care settings (CDC)4, guidelines for implementation of infection control of tuberculosis in the Americas5, and evaluation control tools for Hospital-Acquired Infections9, designing an easy to apply instrument with answer opportunities to respondents. The initial instrument covered 68 questions within the two domains: Administrative measures, and environmental and respiratory protection measures.
Construction/ Formation and evaluation of the instrument:
This research methodology development corresponded to a cross-sectional study (psychometric), where an instrument was designed to check perceived validity and content validity, which would allow a measuring phenomenon that should exist but cannot be directly assessed.
Pilot test:
Once the instrument was built, it was validated by a group of seven experts (two bacteriologists, two nurses, one psychologist, one sociologist, and one statistician) -who knewabouttuberculosisandhaveworkedinthisfield-using the Delphi method. This method is a structured and multi- stage process with an expert panel invited to participate in a series of rounds to identify, clarify and achieve consensus on a particular topic10-12. The selected health specialties experts hold a higher education training (Masters). The objective of this testwas to evaluate structure and general design of the survey, the utility, content, and validity of each questionand its reliability amongexperts.
Delphi Method- First round:
The direct delivery of the instrument to selected experts for reading was carried out previously andpersonally.
The objectives of the study, and work plan for the Delphi When evaluating the usefulness of the instrument and Method development were verbally explained. Their consent to participate in the study wasprovided.
Delphi Method-Second round (item generation):
In this round, evaluation of the instrument was excecutedby each expert. Thefollowingissues were to be considered:
Evaluation of items: Each expert qualified each item according to understanding, ambiguity, writing, range and frequency of responses.
Evaluation on the usefulness of the instrument: Eachexpert qualified eachitemaccordingtoapplication time, need for training, and ratingconvenience.
Evaluation on the format of the instrument: Each expert assessed that the instrument had clear instructions, response method and appropriate typeface.
Evaluation on the validity of the instrument (apparent and content validity): For the study of the apparent validity and content, the Aiken Content Validity Coefficient was used where the judges considered each item of the instrument relevant
Each expert evaluated all issues using the following scale: Adequate (3 points), Slightly Adequate (2 points), and Inadequate (1 point).
Statistical analysis:
Dataquality: Once informationwas obtained, a tabulation of experts’ answers was made, building a database on Microsoft Excel 2013. We assessed database quality by randomly take 10% of the 3332 data, finding no inconsistencies on audited data (333). Information analysis was performed with R version 3.2.0. statistical package. The analysis was focused on evaluating psychometric properties of the instrument, including evaluation of items, instrument’s usefulness and format, validity and reliability.
Evaluation of items: On the items’ evaluation, they were rated ‘excellent’ when70% of experts set it in the ‘Adequate’ category. Evaluatingcriteriaforeachitemwas:
Understanding: Items should be easy to understand regardless level of education.
Ambiguity: The use of words that may have different Interpretations should be avoided.
Writing: Contents should not be confusing or provide emotional burden. Wordingmust be clear.
Range of responses: Identifying items that need more answer opportunities or the possible removal of unnecessary options.
When evaluating the usefulness of the instrument and format, a frequency distribution analysis was carried out according to the answers provided by the experts.
Evaluation of apparent validity and content: Apparent validity and contents were assessed using the Aiken coefficient, which is one of the quantifying techniques on item relevance to a content’s domain, with the participation of N number of judges orexperts, and with ranges going from 0 to 1. 1 being the possible magnitude that indicates perfect agreement among judges regarding higher validity score on evaluated contents 13,14. A minimum Aiken coefficient standard value was considered as equal to or greater than 0.70, which allowed quantifying the relevance of each item regarding the domain established according to the numberof experts15,16.
Reliability regarding the subject was assessed by Bangdiwala’s original and weighted coefficient. The following classification of Bangdiwala’s coefficient was scored: 0.81 - 1.0 (Very Good), 0.61-0.80 (Good), 0.41-
0.60 (Moderate), 0.21-0.40 (Weak) and 0.0 - 0.20 (Poor). The value of these coefficients oscillates between “0” (Null Agreement) and “1” (Total Agreement)17.
Correlations among judges: To determine correlations among judges’ agreement for topics of understanding, ambiguity, writing, and range of responses on each domain, Spearman’s coefficient (rho) was used. This coefficient fluctuates between -1 to 1. A value of -1 to + 1 indicates a perfect negative or positive linear relationship. Correlational relationships can be demonstrated through correlograms using circles filled in a clockwise direction for positive values, and counterclockwise for negativevalues18.
Final instrument construction: We considered not only statistical analysis results, but also suggestions and recommendations provided by the experts.
Results
68 items were initially evaluated by the judges, to obtain65. The first instrument covered two domains (administrative, and environmental/respiratory), which was adjusted to three domains (administrative, environmental, andrespiratory).
Items Analysis. The results obtained from expert’s items evaluation showed that 19 of the 68 items needed adjustments regarding understanding, ambiguity, writing, and range of responses as they were not rated as ‘Adequate’by the experts. Questions with any evaluated criteria percentage below 70 were modified according to experts’ recommendations and reflected in table 1.
Administrative Control Measures | Items | Understanding | Ambiguity | Writing | Rangeof Responses |
---|---|---|---|---|---|
Infection Control Committee issues | Item 3 | 58% | 86% | 58% | 100% |
Item 4 | 58% | 72% | 58% | 72% | |
Item 8 | 29% | 29% | 58% | 86% | |
Item 9 | 58% | 72% | 72% | 86% | |
Item 11 | 100% | 100% | 100% | 29% | |
Item 13 | 100% | 100% | 86% | 58% | |
Item 14 | 58% | 72% | 58% | 15% | |
Health staff issues | Item 17 | 44% | 72% | 44% | 86% |
Diagnosis issues | Item 21 | 44% | 58% | 58% | 86% |
Sample collection, processing and reporting issues | Item 27 | 58% | 44% | 58% | 100% |
Item 28 | 72% | 72% | 86% | 44% | |
Hospitalizationissues | Item 34 | 58% | 58% | 58% | 72% |
Item 35 | 58% | 86% | 58% | 72% | |
Patient follow-up issues | Item 38 | 72% | 44% | 44% | 100% |
Item 44 | 100% | 100% | 100% | 58% | |
Item 51 | 58% | 86% | 72% | 100% |
Source: Personal Reference
Up next the findings for each domain:
Administrative Control Measures domain: On Infection Control Committee, item 6 was deleted (same contents of item 13), so this domain has only 13 questions out ofwhich, items 3, 4, 8, 9, 11, 13 and 14 weremodified.
On health staff, experts rated items 15, 16, 18 and 19 with an ‘Adequate’ percentage. Item 17 only presented low percentage in the experts’ evaluation; they recommended separating both questions contained in it.
On diagnosis, the experts rated items 20, 22, and 23 with an ‘Adequate’ percentage. Only item 21 showed low percentages in the experts’ evaluation.
On sample collection, processing and reporting, the experts rated items 24, 25, 26, 29 and 30 with an ‘Adequate’ percentage, while items 27 and 28 were rated with low percentages, so they were adjusted. Experts agreed on deleting item 30 as it had same contents of item 57.
On hospitalization, the experts rated items31,32,33,36and 37 with an ‘Adequate’ percentage. They recommended to keep the answer option of “DoesNotApply” in the event
of healthcare facilities that don’t provide hospitalization services. Items 34 and 35 showed low percentages in the experts’ evaluation.
On patients’ follow-up part, the experts assessed items 38 to 55; where items 38, 44 and 51 showed low percentages in the experts’ evaluation. Following their recommendation item 46 was re-located under Sample Collection Processing and Reporting. Items 47 and 48 merged into a single one. Items 52 and 53 were removed from the instrument as their contents are same of item 42 and 39respectively.
On Environmental Control Measures and Respiratory Protection, experts recommended separating thesemeasures with their respective questions in two different domains. They assessed items 56 to 68, where only items 61, 62 and 68 showed low percentages in their evaluation. Questions with any evaluated criteria percentage below 70 were modified according to experts’ recommendations table 2.
Evaluation of Instrument’s format and usefulness:
Regarding the usefulness of the instrument, it was found that 5 of the 7 expert judges rated instrument filling out time as good (71%), given that it has quickmeasuringscalesthat do not fatigue respondents and saves time. Five experts believe there is no need for training prior the application of the instrument, since items are general questions anyone in charge of a healthcare institution or an Epidemiological Surveillance Committee must know and can answer.
As for the format evaluation of the instrument, six experts stated that the guides in the instrument were clear, and there is no need for an example on them. Overall, the format, has an appropriate, easy to read font type, for respondents.
The following were the experts’ recommendations aboutthe instrument:
Do not useacronyms.
Change the word information byissue.
Remove any authorship reference from instrument.
Use the word Health Institution instead of Health Establishment.
Place ‘Yes’ in the answers and then‘No’.
Classify questions according to logicalorder.
Change the phrasing ‘level of complexity’ by ‘level of assistance’.
Validity Analysis:
It was submitted to thecriterion of the judges- seven altogether- the evaluation on understanding, ambiguity, writing and range of responses from each of theitems in the instrument. For result systematization, the Aiken coefficient was used, complemented with the use of confidence intervals using the program proposed by Merino and Livia14. A minimum standard of an Aiken’s V coefficient equal to or greater than 0.70 was used, accepting those items which Aiken’s coefficient was above 0.70 asvalid.
Items which confidence intervals were above 0.45 were accepted as valid as well.
Table 3 results show that 11 of the 68 items built in the instrument showed Aiken coefficients below the minimum standard value. We reviewed the suggestions made by the judges and recommendations thoroughly for the respective adjustment of those items according to criterion assessed.
Items | Understanding | IC 95% | Ambiguity | IC 95% | Writing | IC 95% | Rangeof responses | IC 95% |
Item 8 | 0.57 | 0.32-0.79 | 0.43 | 0.21-0.67 | 0.79 | 0.52-0.92 | 0.93 | 0.69-0.99 |
Item 9 | 0.64 | 0.39-0.83 | 0.79 | 0.52-0.92 | 0.79 | 0.52-0.92 | 0.93 | 0.69-0.99 |
Item 11 | 1.0 | 0.78-1.0 | 1.0 | 0.78-1.0 | 1.0 | 0.78-1.0 | 0.57 | 0.32-0.79 |
Item 14 | 0.64 | 0.39-0.83 | 0.79 | 0.52-0.92 | 0.71 | 0.45-0.88 | 0.14 | 0.04-0.40 |
Item 17 | 0.64 | 0.39-0.83 | 0.86 | 0.60-0.96 | 0.64 | 0.39-0.83 | 0.93 | 0.69-0.99 |
Item 21 | 0.64 | 0.39-0.83 | 0.71 | 0.45-0.88 | 0.71 | 0.45-0.88 | 0.86 | 0.60-0.96 |
Item 27 | 0.71 | 0.45-0.88 | 0.64 | 0.39-0.83 | 0.79 | 0.52-0.92 | 1.0 | 0.78-1.0 |
Item 28 | 0.79 | 0.52-0.92 | 0.79 | 0.52-0.92 | 0.93 | 0.69-0.99 | 0.64 | 0.39-0.83 |
Item 34 | 0.57 | 0.32-0.79 | 0.71 | 0.45-0.88 | 0.57 | 0.32-0.79 | 0.86 | 0.60-0.96 |
Item 38 | 0.71 | 0.45-0.88 | 0.57 | 0.32-0.79 | 0.57 | 0.32-0.79 | 1.0 | 0.78-1.0 |
Item 51 | 0.64 | 0.39-0.83 | 0.86 | 0.60-0.96 | 0.86 | 0.60-0.96 | 1.0 | 0.78-1.0 |
Source: Personal Reference
Subject-Related reliability.
Through this analysis, reliability was represented by measuring the experts’ scores to aspects of understanding, ambiguity, writing, and range of responses. It was found that values on original agreement coefficients oscillated between 0.66 and 0.81, while weighted agreement values oscillated between 0.83 and 0.91. It was found that original and weighted agreements were higher than the “poor” classification table 4. The aspect showing greater concordance was range of responses with a weighted coefficient of 0.91.
Aspects | Original concordance coefficients (bw) | Weighted concordance coefficients (bwn) |
Understanding | 0.71 | 0.83 |
Ambiguity | 0.75 | 0.87 |
Writing | 0.66 | 0.84 |
Rangeof responses | 0.81 | 0.91 |
Source: Personal Reference
Correlations between understanding, ambiguity, writing and range of responses in the Administrative measures domain. Results show that judges tended to agree on aspects of understanding, writing, and ambiguity. Through the Correlogram, it was possible to demonstrate high positive correlations between “understanding” and “writing” (rho = 0.79) and between “understanding” and “ambiguity” (rho = 0.73). Lowest correlation was present between “range of responses” and “ambiguity” (rho = 0.16). It was also evident that judges ‘agreeing on “range of responses” was not related to the other aspects figure 1. On the other hand, the environmental and respiratory measures, the results shown pose that judges tended to agree on aspects of understanding, writing, and ambiguity. Through Correlograms, it is possible to demonstrate high positive correlations between “understanding” and “writing” (rho = 0.71), and between “understanding” and “ambiguity” (rho = 0.81). The lowest correlation was between “understanding” and “range of responses” (rho = 0.11) figure 2.
Final version. Instrument’s final version comprised of 65 items evaluating administrative control measures (items 152), environmental control measures (items 53 - 60), and respiratory protection control measures (items 61-65) (Supplementary file 1). Analysis of these results allowed to estimate that items’ formulation was relevant, coherent, and sufficient, grouped under the right domain, obtaining the needed information from each. Likewise, answering options were improved to make each a more appropriate question.
Application of the Instrument. Below are the results obtained in three domains of tuberculosis infection control measures after instrument’s application in healthcare facilities. Five Health-Care Institutions were visited (two private, two public, and a mixed entity) according to authorizations from the Ethics Committee of eachentity.
Administrative control Measures: In connection with the existence of a committee to monitor healthcare-associated infections, four out of the five healthcare institutions stated the existence of such committee. Two of the healthcare institutions surveyed indicated having a tuberculosis control program implemented which helps them monitortheir patients according to their level of complexity. On the other hand, four of the five institutions had a professional or individual in charge of the tuberculosis program. Tuberculosis events reported to domestic’s Public Health System (SIVIGILA- by its acronym in Spanish) are reported by four of the five healthcare institutions. All health care facilities provide education to health staff in the management of the disease. Regarding health staff involved in tuberculosis management, all healthcare institutions refer having an interdisciplinary team of nurse’s assistants, psychologists, social workers, infection disease specialists, internists, respiratory therapists, and pharmaceutical chemists; all health staff know the tuberculosis care path. And in terms of continuous training there is an education and training record as well. Four of the five healthcare institutions claim having had tuberculosis events among healthcare staff. 80% of these health care facilities do not run a regular PPD testing to their personnel as part of a health staff care follow-up.
As for medical diagnosis, all healthcare institutions surveyed perform this procedure according to patient’s signs and symptoms. The identification of respiratory symptomatic or tuberculosis suspected patients is performed by all health care facilities at their emergency service. Only four of the five institutions sort patients diagnosed with confirmed tuberculosis away from other patients.
The Clinical lab in four of the healthcare institutions carry out a smear sample collection and had containers or inputs (laminar flow cabin) sufficient for such collection and processing.
Report delivery is done within a 24-hour period. Three of the healthcare facilities send molecular tests to labs in other healthcare institutions.
Finally, on patient follow-up, four of the five healthcare institutions had an updated record on tuberculosis diagnosed patients, medical consultation is the primary tool for this follow-up to begin. All healthcare facilities fill out an individual patient card and in all of them are keep sufficient drugs of good quality for these patients.
Environmental Control Measures:
On this regard, two of the five institutions had a site established for sample collection. All facilities had waiting rooms where TB diagnosed patients had natural or mechanical ventilation, but are not separated from other patients. Only one healthcare institution had central air conditioning. On tuberculosis patient’s confinement, all facilities had at least one exclusive bathroom or toilet; four of the five had the right visible confinement signaling. Regarding ventilation system maintenance, only four of the five entities surveyed had it done monthly or every three months.
Discussion
Measurement is a value appointing process for certain real life events; this process is part of the daily practice to all professionals since scientific knowledge is obtained from it. So much so, that all fields using measurements can take advantage of it to create theories and test hypothesis19. The application of surveys or instruments generates measurement and data which once analyzed, can determine important decisions most of the time.
Consequently, it is necessary to have valid and reliable instruments to use in clinical practice and research, able to respond to human needs and conditions, and promote treatments, early diagnoses and therapeutic decisions.
Summary of main findings:
As for methodological aspects, the expert consensus is highlighted. The level of expertise of the judges on this particular topic, managed to issue assessments on each evaluated aspect, changing instrument items’ form and semantics and then, finally building an instrument version 2 to evaluate institutional compliance with tuberculosis infection control measures of healthcare institutions in the city of Cali.
The proposed survey will make it possible to learn whether or not health-care institutions comply with tuberculosis infection control measures and are able to lead research on this matter. The instrument has the validity of appearance and content to be applied in the health sector.
On appearance and contents validity, through Aiken’s coefficient obtained relevance of items regarding contents of established domain was quantified, which helps to accurately evaluate what is intended to measure. Only 11 items showed Aiken coefficient below 0.70. Items 6, 30, 52 and 53 were eliminated. The others showed coefficients above 0.70, which according to Merino Soto and Livia Segovia, is a more conservative value. The above provides evidence of content validity of surveyed domains.
Regarding subject reliability, it shows a moderate to very good result, understood as though two or more experts evaluated each item, on understanding, ambiguity, writing and range of responses, with similar results. The aspect showing greater concordance was “range of responses” with a Bangdiwala weighted coefficient of 0.91, followed by “ambiguity” (BWN = 0.87), then “writing” (BWN = 0.84) and finally “understanding “ (BWN = 0.83).
Whereas in correlation, findings show that both, administrative measures and environmental and respiratory measures, have concordance among judges as equal in understanding, writing, and ambiguity. There is a positive linear trend between these aspects.
Overall, results of the instrument’s validation process confirm compliance with minimum requirements onvalidity and reliability.
Considering results regardingthe application of the instrument, the commitment and control exercised by each healthcare institution surveyed ontuberculosis is evident. It is worth highlighting the work performed by managers of these healthcare facilities, in the face of administrative, environmental and respiratory strategies, being immersed in their institutional operational plan.
Contrast with literature:
Literature shows several studies linking the Delphi method as a methodological strategy in the formulation and validation of an instrument. In the study of Longobardi-Vasquez and collaborators, the first one developed was submitted to six experts 20; likewise, in a research carried out by Noriega Bravo and collaborators, they used the Delphi method with the help of seven experts to evaluate the total of items 21. Hence confirmed, the number of experts used in this research was appropriate.
As exposed by Cruz-Avelar Agles 18, Mousavi and collaborators22 and other authors, validity measures the degree to which items seem to measure what is proposed, and should include the judgment or opinions of some experts in the matter. F o r t h e appearance and content validation stage, there are many methods used, among them are quoted: Method based on factorial analysis, content validity index, item-objective congruence index, congruence index, content validity coefficient, among others. Aiken’s coefficient was chosen to evaluate contents’ validity by judges’ criterion; since, according to the literature this coefficient is easy to calculate, it guarantees objectivity of the procedure, and allows to obtain values feasibly contrasted with statistics depending on the size of selected judges’ sample The instrument’s 68 items were evaluated by the seven experts in all four aspects: understanding, writing, ambiguity, and range of responses, which made it possible to obtain rates which helped modify or eliminate the instrument’s item.
On the other hand, different procedures have been used to determine the degree of agreement among judges. The Kappa coefficient became the most widely used agreement index23. Despite that, the Bangdiwala weighted coefficient of this research was chosen because it is more stable than Kappa’s as sample size decreases. In addition, Bangdiwala’s coefficient relies on the number of judges and is more powerful to determine concordance among them24.
Different studies evaluate the level of implementation of control measures. Brouwer and collaborators conducted a study in three provinces of Mozambique where they assessed management, administrative, and environmental measures in the 29 healthcare centers, concluding that guidelines on tuberculosis diagnosis and treatment were not present in those facilities25.
Another study conducted in Bogotá - Muñoz Sanchez et al.-, describes the implementation of control measures in 51 healthcare institutions; evidence showed 68% of them had an infection control plan in place, out of which 16% had mechanical or natural ventilation, and 84% had N95 respirators available for their staff26. Validation of instrument to comply with institutional measures on tuberculosis infection control
Strengths and Weaknesses:
Obtaining this valid and reliable instrument will identify the level of compliance measures for tuberculosis infection control in health-care institutions. Conversely, this instrument becomes a useful and practical tool to guide these entities’ directors/managers into incorporating changes or to reshape institutional procedures on tuberculosis prevention and control. Weaknesses for this instrument’s validation were the type of questions, and the nominal and dichotomous variables used, which reduced the incorporated statistical analyses associated with these characteristics.
Application of this instrument’s final version was only possible in five out of the seven healthcare institutions chosen. For future research on this topic, would be important to choose more facilities with a different inclusion criterion. Most surveys applied were answered by nurses who were members of COVE* (Epidemiological Surveillance Committee -*by its acronym in Spanish), the Hospital Committee, or the tuberculosis program, which makes us think that the assisting staff is trained and skilled on tuberculosis and knows all about the approach to this infection.
Public Health implications:
Finally, it can be stated that it is a ready-to-use instrument, useful to identify compliance of administrative, respiratory, and environmental measures at any level of health care, including facilities in charge of management, diagnosis, and treatment of tuberculosis patients.
On the other hand, the possibility to apply this instrument allows to evaluate the existence of such measures, thus carrying out other studies in that regard which may lead to identify and reduce the risk of tuberculosis infections.
As a conclusions: On the analysis of items’ evaluation, it is concluded that the number of items (68) initially considered in the instrument’s 1st version, needs to be modified according to the experts, in all four scoring aspects, thus leaving an instrument Version 2 with 65 questions.
In that sense, the instrument’s features, such as the number of replies, vocabulary used, the distribution of the questions, are factors regarded within the four aspects evaluated: understanding, ambiguity, writing and range of responses.
Contents’ validity and degree of reliability were adequate and acceptable. Only 11 items showed an Aiken coefficient below 0.70. Incidentally, reliability among experts for evaluated aspects showed a moderate to very good concordance between every pair of experts.
The above, allowed obtaining a reliable and valid tool to be used in health institutions, where administrative, environmental, and respiratory control measures are the main tool to prevent contagion of this infection.