Reports of uncontrolled clinical trials for directly placed restorations in vital teeth

Abstract

Uncontrolled trials are criticized as unreliable. This study aimed to establish how the number of published reports from uncontrolled clinical trials compares to that of controlled trials for directly placed restorations in vital teeth and whether their annual number is increasing, stable or decreasing. PubMed was searched and suitable citations of uncontrolled and controlled trial reports published between 1990–2016 were included. Reference check and hand searching were conducted. The median annual report number with 25 and 75% percentile was calculated for both types of trials. 695 reports were found. The median number of reports per year was 4 (3–7) and 22 (15–26) from uncontrolled and controlled trials, respectively. A statistically significant decreasing ratio of uncontrolled to controlled trial reports was observed (p = 0.01) by linear regression analysis. The number of reports of uncontrolled clinical trials listed in PubMed over the last 27 years appears at least five times smaller than that of controlled clinical trials and its number in relation to that of controlled trials seems to decrease over time.

Clinical Trial; Research Design; Dentistry

Introduction

Uncontrolled clinical trials are defined as trials with one single treatment arm during which all patients receive the same intervention and whose outcomes are followed up over a certain period of time.¹,² The conduct of uncontrolled clinical trials has been considered to be less expensive, more convenient and faster than that of randomised control trials (RCT).¹ Uncontrolled clinical trials are further recommended as pilot studies for the exploration of associations between variables and outcome measures, as well as for the estimation of effect sizes as basis for sample size calculation in subsequent RCTs.³

On the other hand, uncontrolled clinical trials have been criticized as being based on the logical post hoc ergo propter hoc (“After this, therefore because of this” = false cause) fallacy⁴ – which can be considered as a subset of the common ‘Affirming the consequent’ error⁵ - and its results are considered to be unreliable, due to regression to the mean, particularly with increasing follow-up period.⁶ Since regression to the mean is related to continuous measurements (e.g. that of body height, weight or blood pressure), this problem may be less prevalent in uncontrolled clinical trials with binary (success/failure) outcomes that investigate the clinical merits of directly placed tooth restorations.

However, the logical post hoc ergo propter hoc fallacy may be considered as the main reason not to rely on uncontrolled trial results for clinical guidance. When, for example, directly placed restorative materials (or any other property of a particular tooth restoration type) are investigated during uncontrolled clinical trials, these materials are used to place tooth restorations in patients and the success or failure of these restoration are evaluated after specified time period. Once failures or successes are established, these are then ascribed to the particular material properties and the usefulness of the material for daily dental practice is then inferred and even recommended.⁷,⁸

This process of inference is explained as follows: If the restorative material properties are sufficient then it is inferred that they will lead to a successful tooth restoration outcome and if it is insufficient then the restoration will be a failure. However, the success or failure of a tooth restoration may only partially depend on the properties of the placed material alone or not at all.⁹,¹⁰ Other factors will usually play a role as well. The characteristics of tooth cavities (e.g. size, depth);⁹,¹⁰ patient factors, such as bruxism, caries risk or age, including frequency of sugar intake through diet or their oral hygiene habits,⁹,¹⁰ the quality of dental service applied by the dentist¹⁰ or indeed any unknown factors that may exist in the studied patient cohort may affect the success or failure of a placed tooth restoration in time. Therefore, in reality the outcome of a placed tooth restoration may be a success even if material properties are found insufficient in in-vitro studies, due to the beneficial effect of other factors. This in turn creates a logical contradiction, which states that a placed tooth restoration may be a success when the properties of the placed material are insufficient and sufficient. Such statement disagrees with itself and thus cannot hold true. The consequence of such contradiction is that no causal inferences can be drawn.

Since the causal relationship of material properties to restoration outcome is uncertain in uncontrolled trials, the very utility of such trials for the exploration of associations between variables and outcome measures, as well as for the estimation of effect sizes as basis for sample size calculation is negated. In uncontrolled trials, the exploration of associations between variables and outcome measures is undertaken by sub-grouping subjects according to specific variables and then by establishing whether the outcome per subgroup lies above or below the total average of the study sample.³ However, the result of such exploration is again challenged by the uncertainty regarding which of the potentially influencing factors is cause for the outcome per subgroup as each subgroup would differ not only in the variable under investigation but also in the set of other influencing variables. The differences in the latter and not in the former may then be cause for the observed difference of the subgroup from the total average.

Thus, in uncontrolled trials estimated effect sizes may not be useful for sample size calculation in subsequent RCTs either, because the real set of factors that have affected the measured effect size may substantially differ in the RCT sample of subjects. As the uncontrolled trial design does not include any control group, it is simply not able to eradicate such possible confounder influence, which thus render uncontrolled trials not useful as pilot studies. Instead, prospective cohort studies (with test and control groups) and smaller, less expensive randomised control trials would appear more suitable as pilot studies, particularly for the exploration of associations between variables and outcome measures and for the estimation of effect sizes as basis for sample size calculation in subsequent larger RCTs, respectively.

The uncontrolled clinical trial design has also been utilized for the investigation of clinical merits of directly placed restorations in vital teeth.¹¹,¹² For this reason the objectives of this study were to establish:

1. How the number of published reports from uncontrolled clinical prospective trials compare to that of reports from controlled clinical prospective trials for direct restorations in vital teeth;

2. Whether the annual number of reports from uncontrolled trials is either increasing, stable or decreasing in comparison to that of controlled trials.

Methodology

Both authors searched PubMed independently until January 26, 2017. Reference check of all included trial reports and hand searching of additional journals were conducted. The details of the search strategy and the including search terms are presented in Table. Citations were included in line with the following criteria:

Table
Search strategy: electronic database search.

Electronic database	Search term number	Number of Citations found (without dublications)
PubMed search strategy: 26.01.2017 Online: http://www.ncbi.nlm.nih.gov/pubmed	[1] (((tooth restoration) OR tooth filling) OR dental filling) OR “Dental Restoration, Permanent”[Mesh] Sort by: PublicationDate Filters: Clinical Trial; Abstract; Humans	2563
	[2] (amalgam OR composite OR glass-ionomer OR compomer) AND restoration Sort by: PublicationDate Filters: Clinical Trial; Abstract; Humans	1470
	[3] atraumatic restorative treatment (no filters)	413
	[4] composite restorations (Filters activated: Clinical Trial, Abstract.)	1195
	[5] compomer restoration (Filters activated: Clinical Trial, Abstract.)	152
	[6] amalgam restoration (Filters activated: Clinical Trial, Abstract.)	267
	[7] glass ionomer restoration (Filters activated: Clinical Trial, Abstract.)	461
	Total included from database search	6521
	Hand-search included (PubMed listed)	36
	Total citations included	6557

1. Report of a clinical prospective trial;

2. Direct restoration in vital primary and permanent teeth (anterior and posterior);

3. Restoration longevity, survival or failure as outcome (Definition of failure also included all aspects of patient satisfaction, e.g.: including post operative sensitivity, unsatisfactory aesthetic);

4. Abstract of trial report listed in PubMed;

5. Uncontrolled, single treatment arm study design or controlled (randomised or non-randomised) trial with at least two treatment arms;

6. Report published between 1990–2016.

No limitation on publication languages of the trial reports was set. Both authors scanned titles and abstracts of identified citations in duplication. Potential citations of trial reports without a listed abstract were scanned in full copy. Trial reports were excluded, if: no restoration failure or success was assessed; when no direct restorations were placed; no vital teeth were restored; if the study was not a clinical and a prospective trial. Duplications and any citations without PubMed listed abstracts, which could not be traced in full, were also excluded. The authors resolved any disagreements with regard to citation inclusion/exclusion through discussion and consensus. Reports, which could not be traced in full copy but for which an abstract was listed in PubMed were included for sensitivity analysis.

Data collection from accepted trials and statistical analysis

In line with the stated inclusion criteria, the number of uncontrolled and controlled trial reports listed in PubMed between 1990 and 2016 was recorded per calendar year. All trials that included more than one intervention group were classified as ‘controlled trial’. As the number of controlled trial reports per year served only as comparator to that of reports from uncontrolled trials, no further investigation into the details of reports from controlled trials was made during this study.

From the number of annual reports per type of trial, the median trial report number with 25 and 75% percentile, as well as the annual ratio of uncontrolled versus controlled trial reports was calculated. In addition, the annual report numbers and ratios where explored through linear regression analysis (Biostat 2009 statistical software) with the calendar year being the independent variable and the annual trial report number or the ratio of uncontrolled to controlled trial reports the dependent variable. Alpha was set at 5%.

Sensitivity analysis was conducted with additional data extracted from abstracts of trial reports, identified in PubMed that could not be traced in full.

Results

A total of 6521 citations were identified through the initial PubMed search and an additional 36 citations through hand searching (Table). From the total citations found, 5804 citations considered irrelevant to the topic were excluded and 753 were included for review. From these, 63 reports could not be traced in full. A reference check of the 690 traced trial reports yielded an additional 83 relevant citations that could be traced in full. A total of 78 reports were excluded for the following reasons: no restoration failure or success assessed = 17; no direct restorations were placed = 20; no vital teeth were restored = 9; not a clinical trial = 13; duplication = 17; not a prospective trial = 2. A total of 695 trial reports, 118 from uncontrolled (17%) and 577 from controlled trials (83%) were accepted. Details of the included, excluded trials and trials not traced in full copy are presented in separate work sheets in the Additional file.

The median number (with 25–75% percentile) of reports from uncontrolled trials per year was 4 (3–7) and that of controlled trials 22 (15–26), thus the latter being between four and five times larger than that of uncontrolled trials. The annual report numbers of the two different trial types are shown in Figure 1 and a decreasing trend (Coefficient = -0.008, p = 0.01) of publications of uncontrolled trial reports in relation to published controlled trial reports per year is shown in Figure 2. Further exploration of the trend established that changes in the number of uncontrolled trial reports (Coefficient = -0.06, p = 0.31), and of controlled trial reports (Coefficient = 0.37, p = 0.06) per year were not statistically significant. Further details of the linear regression analysis, as well as all numbers of annual trial reports and ratios are presented in the Additional file.

Figure 1
Number of trials per year (1990–2016).

Figure 2
Ratio of number of reports from uncontrolled to controlled trials per year.

Sensitivity analysis

Abstracts from 63 trial reports that could not be traced in full copy were reviewed in order to establish whether their absence would have significantly altered the results from the data extracted from the traced full reports. Because the full reports could not be reviewed, in order to verify the correctness of the abstract data, it was decided not to include this data in the main analysis. Details of the abstract data are presented in the Additional file.

When data from the abstracts were added, the median number (with 25–75% percentile) of reports from uncontrolled trials per year was 5 (3–8) and that of controlled trials 23 (18–28), thus the latter would remain four to five times larger than that of uncontrolled trials. In addition, the changed proportions still show a decreasing trend (Coefficient = -0.008, p = 0.01) of publications of uncontrolled trial reports in relation to that of controlled trial reports per year (Figure 2). Further exploration of the number of uncontrolled trial reports (Coefficient = -0.09, p = 0.17) and the number of controlled trial reports (Coefficient = 0.24, p = 0.21) per year appeared to confirm the main results.

Discussion

Limitations of study method

A systematic literature search based on specific strings of search terms (Table) was conducted. Limitations of this search included the restriction to one single database (PubMed), only; restriction to the time period from 1990–2016 and possible limited inclusiveness of search terms. The latter appeared likely as further citations were found by use of reference check that was missed during the database search. However, these constituted 12% (n = 83) of the total number of included trial reports, only. In addition, the results of the sensitivity analysis showed that exclusion of 8% (n = 63) of trials that could not be traced in full did not affect the conclusions of this study. For this reason it can be assumed that the number of possibly still unidentified trial reports is small and that its absence would not have substantially altered the overall study results.

Limiting the search to PubMed only may have further reduced the number of identifiable trial reports by exclusion of reports that are listed in other databases but not in PubMed. However, this study intended to focus on PubMed listed reports, only. PubMed is a well-known global evidence source that is at the same time freely accessible on the Internet (http://www.ncbi.nlm.nih.gov/pubmed). While other databases such as LILACS (http://lilacs.bvsalud.org/en/) or DOAJ (https://doaj.org/) are also freely accessible, they are either only of regional nature, less well known or are largely redundant, due to overlap with the listed PubMed content. On the other hand, other well-known global evidence sources such as EMBASE are not freely accessible and require subscription. For these reasons, the results and conclusions of reports listed in PubMed are more salient to those who seek clinical evidence and thus reflect the most influential results to the topic.

A further limitation of this study may have arisen from the fact that it did not include any quality assessment of the identified controlled trials reports. Therefore, the established ratio of uncontrolled to controlled trial reports does only reflect the number of reports of uncontrolled trials in relations to that from trials where more than one intervention group was included and not in relation to that of e.g. high quality RCT reports to the topic. Furthermore, the focus of this study was on the number of published reports and not on that of trials. Some trials may have generated several published reports listed in PubMed over the years, e.g. for different follow-up periods and such repetition would need to be considered when evaluating the risk of obtaining definitive clinical guidance from uncontrolled clinical trials.

In addition, the search was also limited to the period from 1990 – 2016, thus excluding all trial reports listed before 1990. The reason for such limitations was that restorative materials studied before 1990 may have mostly lost their clinical relevance today. It was also noted that clinical trials were increasingly listed without abstracts in the years before 1990, thus potentially leading to an increasing number of trial exclusions on that basis.

Discussion of study results

The results of this study suggest that the number of listed reports from uncontrolled trials is small in relation to that of controlled trials and that this number in relation to the number of controlled trials is decreasing.

The low number of published reports of uncontrolled trials seems surprising, as the former study design is generally considered to be cheaper and faster.¹ The reason may be related to the relative early recognition of the advantages of prospective comparisons in clinical study design. Hickel et al.¹³ reconfirmed this recognition for studies of restorative dental materials by stating that “a study without a control group is of limited value”. A further reason for the relative small number of uncontrolled trial reports is an apparent lack of direct recommendations and guidelines in favour of the uncontrolled study design.

While the shortcomings of the uncontrolled trial design have been highlighted in the past,⁴,⁵,⁶ the dangers of subsequently using results from uncontrolled trials for naïve-indirect comparisons have been discussed further an the example of high-viscosity GIC versus dental amalgam restorations.¹² A ‘naïve-indirect comparison’ is made when results for competing clinical interventions are extracted from unrelated studies and compared.¹⁴ When results from naïve-indirect comparisons between high-viscosity GIC versus dental amalgam were compared to that from RCT to the same types of materials, the latter were found to be highly unreliable.¹² This unreliability was further illustrated by a simulation study.¹⁵ In this study, the same data was compared using either naïve-indirect comparison or direct comparison on RCT basis. The results showed a poor inter-comparison type agreement with kappa = 0.06 and a 64% overestimation of the true results through naïve-indirect comparison.¹⁵ Against this background, the observation that the use of the uncontrolled study design appears to be decreasing is encouraging.

Conclusions

Within the limits of the applied study methodology it can be concluded that the number of reports of uncontrolled clinical trials listed in PubMed over the last 27 years appears more than five times smaller than that of controlled clinical trials for directly placed restorations in vital primary and permanent teeth and its number in relation to that of controlled trials seems to decrease over time.

Acknowledgement

The authors thank Dr Petra Gaylard for her assistance during the statistical analysis of the data.

References

• ¹
  Wang D, Bakhai A. Clinical trials: a practical guide to design, analysis, and reporting. London: Remedica; 2006.
• ²
  Huitema BE. The analysis of covariance and alternatives: statistical methods for experiments, quasi-experiments, and single-case studies. 2nd ed. Willey; 2011. Chapter 30, Uncontrolled clinical trials; p. 609-17.
• ³
  White A, Ernst E. The case for uncontrolled clinical trials: a starting point for the evidence base for CAM. Complement Ther Med. 2001;9(2):111-6. https://doi.org/10.1054/ctim.2001.0441
  » https://doi.org/10.1054/ctim.2001.0441
• ⁴
  Türp JC, Schwarzer G. [The effectiveness of therapeutic measures: the post-hoc-ergo-propter-hoc fallacy]. Schweiz Monatsschr Zahnmed. 2003;113(1):36-46. German.
• ⁵
  Kaye SM. Critical thinking: a beginner’s guide. Oxford: Oneworld; 2012.
• ⁶
  James KE. Regression toward the mean in uncontrolled clinical studies. Biometrics. 1973;29(1):121-30. https://doi.org/10.2307/2529681
  » https://doi.org/10.2307/2529681
• ⁷
  Luengas-Quintero E, Frencken JE, Muñúzuri-Hernández JA, Mulder J. The atraumatic restorative treatment (ART) strategy in Mexico: two-years follow up of ART sealants and restorations. BMC Oral Health. 2013;13(1):42. https://doi.org/10.1186/1472-6831-13-42
  » https://doi.org/10.1186/1472-6831-13-42
• ⁸
  Burrow MF, Tyas MJ. Clinical investigation of G-Bond resin-based adhesive to non-carious cervical lesions over five years. Aust Dent J. 2012;57(4):458-63. https://doi.org/10.1111/j.1834-7819.2012.01729.x
  » https://doi.org/10.1111/j.1834-7819.2012.01729.x
• ⁹
  McCracken MS, Gordan VV, Litaker MS, Funkhouser E, Fellows JL, Shamp DG et al. A 24-month evaluation of amalgam and resin-based composite restorations: Findings from The National Dental Practice-Based Research Network. J Am Dent Assoc. 2013;144(6):583-93. https://doi.org/10.14219/jada.archive.2013.0169
  » https://doi.org/10.14219/jada.archive.2013.0169
• ¹⁰
  Demarco FF, Corrêa MB, Cenci MS, Moraes RR, Opdam NJ. Longevity of posterior composite restorations: not only a matter of materials. Dent Mater. 2012;28(1):87-101. https://doi.org/10.1016/j.dental.2011.09.003
  » https://doi.org/10.1016/j.dental.2011.09.003
• ¹¹
  Manhart J, Chen H, Hamm G, Hickel R. Buonocore memorial lecture: review of the clinical survival of direct and indirect restorations in posterior teeth of the permanent dentition. Oper Dent. 2004;29(5):481-508.
• ¹²
  Mickenautsch S, Yengopal V. Direct contra naïve-indirect comparison of clinical failure rates between high-viscosity GIC and conventional amalgam restorations: an empirical study. PLoS One. 2013;8(10):e78397. https://doi.org/10.1371/journal.pone.0078397
  » https://doi.org/10.1371/journal.pone.0078397
• ¹³
  Hickel R, Roulet JF, Bayne S, Heintze SD, Mjör IA, Peters M et al. Recommendations for conducting controlled clinical studies of dental restorative materials. Clin Oral Investig. 2007;11(1):5-33. https://doi.org/10.1007/s00784-006-0095-7
  » https://doi.org/10.1007/s00784-006-0095-7
• ¹⁴
  Glenny AM, Altman DG, Song F, Sakarovitch C, Deeks JJ, D’Amico R et al. Indirect comparisons of competing interventions. Health Technol Assess. 2005;9(26):1-134. https://doi.org/10.3310/hta9260
  » https://doi.org/10.3310/hta9260
• ¹⁵
  Mickenautsch S. How should competing clinical interventions be compared in dentistry? J Minim Interv Dent. 2013;6(6):73-80.

Publication Dates

• Publication in this collection
  2017

History

• Received
  07 Feb 2017
• Reviewed
  05 May 2017
• Accepted
  10 May 2017