Dear Editors of CoDAS Journal,
The purpose of this letter is to present a few considerations on test validation in Speech-language Pathology. The validation of a diagnostic clinical trial does not occur in a single moment, it is conversely a process of increasing complexity. This process can be divided into two phases: internal validation of the performance of this new assay and validation of its performance in a diagnostic clinical trial.
The internal validation of the performance of a new assay is a mechanism of control over the internal consistency of the developed protocol. The development of a protocol whose relevance is justified by epidemiological data and existence of specific tests occurs during this phase. The items of this protocol are based on the available, preferably international specific literature and the analysis of its items and application procedures should be conducted by unbiased referees.
At this phase, the protocol is first assessed with respect to its intra-observer (the same referee analyzes the test results repeatedly, observing the proportion of times that he/she will agree with his/her own previous interpretations) and inter-observer (two or more trained referees analyze the outcomes obtained in the assay, observing the proportion of times that they will agree with the interpretations of the first examiner of the responses) variability. There are specific statistical tests for determination of agreement. In general, concordance <50% indicates low reproducibility, that is, the trial must be modified or abandoned; concordance between 50 and 75% indicates average reproducibility; between 75 and 83%, good reproducibility; and above 83%, excellent reproducibility. If adequate reproducibility indices are observed, the trial can then be conducted.
The validation phase of test performance in diagnostic clinical trials is indicated to rigorously analyze an assay. Diagnostic clinical trials are performed in phases. Each of these phases presents different goals and defined objectives, and the following should be considered in each of them:
-
•
definition of the target population;
-
•
inclusion criteria;
-
•
exclusion criteria;
-
•
sample selection strategies (the spectrum of the sample surveyed should represent what is expected for the general population);
-
•
sample randomization (random selection of participants);
-
•
blinding and masking (prevent those who interpret the test outcomes from having access to information about the participants, avoiding bias and prejudice that may affect judgment).
PHASE I DIAGNOSTIC CLINICAL TRIAL
The purpose of this phase is to assess the safety and identify the possible undesirable effects of the diagnostic trial. Adequacy of procedures, safety, and possible undesirable effects of the trial are tested. The assay is conducted with a small group of participants (20-100) based on patients with a disease spectrum to be diagnosed.
PHASE II DIAGNOSTIC CLINICAL TRIAL
The objective of this phase is to expand the sample size of the first phase and, if possible, vary the research group (different city, risk group, etc.). The assay is conducted with a larger group of participants (100-300) to determine its magnitude of efficacy and further assess its safety.
PHASE III DIAGNOSTIC CLINICAL TRIAL
The aim of this phase is to compare the new assay with an existing one, or rather, compare the new assay with a gold standard one, to determine the accuracy of the new diagnostic trial. A new trial can only be considered validated when the results obtained in its application, compared with those of a reference assay, can clearly discriminate who actually has the disease and who does not have it. Equations to calculate the performance of the Phase III Diagnostic Clinical Trial are as follows:
-
1
Sensitivity – an assay is considered sensitive when it identifies a participant who actually has the disease as positive;
-
2
Specificity - an assay is considered specific when it identifies a participant who does not have the disease as negative;
-
3
Positive predictive value - it is the proportion of true positive participants compared with that of participants diagnosed in the assay as positive;
-
4
Negative predictive value - it is the proportion of true negative participants compared with that of participants diagnosed in the assay as negative;
-
5
ROC (receiver operator characteristics) Curve - it is a graphical plot that establishes cut-off points based on the values of specificity and sensitivity. These points are defined, forming an area under the curve (varying from 0.5 (worthless test) to 1.0(perfect test)). Trail quality is defined by the curve design, the closer to the upper left corner of the graph, the better the test performance.
PHASE IV CLINICAL TRIAL
The goal of this phase is to apply the new assay to large groups of participants (1,000-3,000), with different examiners at different institutions, to confirm its efficacy and monitor its undesirable effects. Once a test has undergone all of these phases, researchers are expected to begin studies on diagnostic performance, that is, the direct effect of the outcomes of a trail on clinical decisions.
There are few gold standard clinical trials in Speech-language Pathology. Sometimes, the only outcome indicator is the follow-up of the patient to confirm the presence or the absence of disease. At other times, the gold standard assay is imperfect (in the case of language disorders, the gold standard assay was validated in another language). In the absence of a gold standard test for the early detection of a communication disorder, it is necessary to trace the risk factors for early diagnosis and prognosis estimation of the case.
Speech-language Pathology is initiating its diagnostic clinical trials. Few speech-language therapists are familiar with these concepts and methodologies and have undergone specific training in the interpretation of diagnostic assays. Most therapists do not consider diagnostic accuracy relevant. This attitude perpetuates a precarious standard of clinical decision, which is flawed because it is based on self-judgment (the well-known “clinical practice” that may be mistaken), based on part of the literature (almost always domestic and often lagging in quality) and on a brief, informal approach that can, to a large extent, be misleading. The adoption of clinical trials - in the case of diagnostic clinical trials - will allow an increased accuracy index of clinical activity, with a more accurate approach, registering as positive what is really positive and as negative what it really is negative.
BIBLIOGRAFIA
- ASHA: American Speech-Language-Hearing Association [Internet]. Evidence-based practice in communication disorders. Rockville: ASHA; 2005 [citado em 2003 Nov 14]. Disponível em: www.asha.org/policy
- Clinical Trials [Internet] Statement Bethesda National Institutes of Health 2003 citado em 2003 Nov 14 Disponível em: www.clinicaltrials.gov
- Gehan EA, Lemak NA. Statistics in medical research: developments in clinical trials. New York: Plenum Medical Book Co.; 1994.
- Giffels JJ. Clinical trials: what you should know before volunteering to be a research subject. New York: Demos Vermande; 1996.
- Hegde MN. Clinical research in communicative disorders: principles and strategies. 2nd ed. Austin: Pro-Ed.; 1994.
- Hulley SB, Cummings SR, Browner WS, Grady DG, Newman TB. Delineando a pesquisa clínica. 3. ed. Porto Alegre: Artmed; 2008. 384 p.
- Luna B Fo. A ciência e a arte de ler artigos médicos. São Paulo: Atheneu; 2010. 176 p.
- Maxwell DL, Satake E. Research and statistical methods in communication sciences and disorders. Canada: Thompson; 2006. 561 p.
- Meinert CL. Clinical trials: design, conduct, and analysis. New York: Oxford University Press; 1986.
- NIH: National Institutes of Health [Internet]. An introduction to clinical trials. Bethesda: NIH; 2003 [citado em 2003 Nov 14]. Disponível em: http://clinicaltrials.gov/ct/info/whatis
» http://clinicaltrials.gov/ct/info/whatis - Robey RR. A five-phase model for clinical-outcome research. J Commun Disord. 2004;37(5):401-11. PMid:15231420. http://dx.doi.org/10.1016/j.jcomdis.2004.04.003
» http://dx.doi.org/10.1016/j.jcomdis.2004.04.003 - Schulz KF, Altman DG, Moher D. CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. BMJ. 2010;340:c332. PMid:20332509. http://dx.doi.org/10.1136/bmj.c332
» http://dx.doi.org/10.1136/bmj.c332
Data availability
Data citations
Clinical Trials [Internet] Statement Bethesda National Institutes of Health 2003 citado em 2003 Nov 14 Disponível em: www.clinicaltrials.gov
Publication Dates
-
Publication in this collection
2017
History
-
Received
11 Aug 2016 -
Accepted
11 Aug 2016
