Identifying Depression Early in Adolescence: assessing the performance of a risk score for future onset of depression in an independent Brazilian sample

Abstract

Objective:  The Identifying Depression Early in Adolescence Risk Score (IDEA-RS) was recently developed in Brazil using data from the Pelotas 1993 Birth Cohort to estimate the individualized probability of developing depression in adolescence. This model includes 11 sociodemographic variables and has been assessed in longitudinal studies from four other countries. We aimed to test the performance of IDEA-RS in an independent, community-based, school-attending sample within the same country: the Brazilian High-Risk Cohort.

Methods:  Standard external validation, refitted, and case mix-corrected models were used to predict depression among 1442 youth followed from a mean age of 13.5 years at baseline to 17.7 years at follow-up, using probabilities calculated with IDEA-RS coefficients.

Results:  The area under the curve was 0.65 for standard external validation, 0.70 for the case mix-corrected model, and 0.69 for the refitted model, with discrimination consistently above chance for predicting depression in the new dataset. There was some degree of miscalibration, corrected by model refitting (calibration-in-the-large reduced from 0.77 to 0).

Conclusion:  IDEA-RS was able to parse individuals with higher or lower probability of developing depression beyond chance in an independent Brazilian sample. Further steps should include model improvements and additional studies in populations with high levels of subclinical symptoms to improve clinical decision making.

Adolescent; clinical prediction; depression; prevention

Introduction

Depression is a leading cause of years lived with disability worldwide.¹ It often begins in adolescence and early adulthood,²,³ placing a significant burden on adolescents’ lives and increasing the likelihood of many negative outcomes.⁴ Depression also has considerable social and economic impacts⁵ and not infrequently presents as a chronic recurring disorder associated with lifelong disability, even when affected individuals receive appropriate treatment.⁶

Early identification of adolescents at high risk for the onset of major depressive disorder (MDD) may be an essential step for the development and implementation of preventive interventions with potential to improve the prognosis of at-risk individuals.⁷ In contrast to the search for single risk factors, the agenda of prognostic medicine now hinges on the combination of multiple risk factors into single composite scores and use of predictive modeling techniques to provide estimates of risk on an individual level rather than group-based averages.⁸ Several risk models for predicting mental health disorders have been developed in recent years,⁹ although most still lack independent external validation.¹⁰ There has been an increase in the development and validation of prediction models for child and adolescent mental health, but poor performance, methodological limitations, and lack of external validation are still obstacles to translation into clinical practice.¹¹

The Identifying Depression Early in Adolescence (IDEA) consortium⁷ recently developed a model to predict individualized risk of developing depression in late adolescence using data from the population-based 1993 Pelotas Birth Cohort, located in the south of Brazil.¹² The IDEA risk score (IDEA-RS) encompasses 11 easily obtainable predictors assessed at the age of 15 in that cohort (sex, skin color, drug use, having ever failed school, social isolation, involvement in fights, relationship with mother, relationship with father, relationship between parents, childhood maltreatment, and having ever run away from home) and presented good ability to discriminate between adolescents who did and did not develop MDD at the age of 18 years (area under the receiver operating characteristic curve [AUC] = 0.78).¹² Developing models for common mental health disorders including known risk factors that can be investigated in low-cost settings may represent a major advance toward implementing early recognition and intervention. Testing the performance of the IDEA-RS in predicting MDD in external settings is an essential step to facilitate its translation into clinical practice and inform targeting of preventive interventions. To date, the IDEA-RS has been externally validated in four independent cohorts: nationally representative birth cohorts in the United Kingdom (AUC = 0.59) and New Zealand (AUC = 0.63),¹² a sample of children exposed to war who were receiving services from humanitarian organizations in Nepal (AUC = 0.73),¹³ and a representative school-based sample in Lagos, Nigeria (AUC = 0.62).¹⁴ In the present study, we had the opportunity to investigate the ability of the IDEA-RS to predict the individualized risk of developing a depressive episode by late adolescence in a second independent cohort of adolescents from the same country as the original study (Brazil), in a sample enriched for parental psychopathology.

Methods

Sample and participants

We analyzed data from a sample of the Brazilian High-Risk Cohort for Psychiatric Disorders (BHRC), a school-based sample enriched for youth at high-risk of psychiatric disorder. The BHRC is a longitudinal cohort designed to assess the trajectory of mental health problems in young people.¹⁵ The BHRC sample recruited students from 57 public schools in two major Brazilian cities (35 in São Paulo and 22 in Porto Alegre). Parents of 9,937 children were interviewed with the Family History Screen.¹⁶ An index of family load was calculated considering the presence of symptoms in the biological mother, biological father, biological siblings, and half-siblings.¹⁵ A total of 2,511 participants were assessed at baseline (Wave 0, at age 6-14 years): 957 youths were randomly selected and 1,554 were selected as being at high-risk for psychiatric disorders, based on the presence of high family loading of psychopathology as determined by the Family History Screen. The BHRC study received approval from the local ethics committee, and written informed consent was obtained from a parent or guardian, in addition to the participant’s assent.

Data included in this study were extracted from Wave 1 (collected in 2014-2015, at age 9-17 years, retention rate = 80.0%) and Wave 2 assessments (2018-2019, at age 13-23 years, retention rate = 75.9%). Further details on data collection and retention for the BHRC are available as online-only supplementary material in Figure S1. We selected these two timepoints to closely resemble the characteristics of the original IDEA-RS development study,¹² particularly in regard to puberty status at baseline and to the interval between assessments.

Following the protocol used in the development of the IDEA-RS,¹² the exclusion criteria were having an intelligence quotient (IQ) lower than 70, having any depressive disorder (major depressive disorder, other depressive episode, or any undifferentiated anxiety/depression disorder) at previous assessments, and having no evidence of puberty (Tanner stage < 2). After applying the exclusion criteria, a total of 1,442 participants (71.7% from Wave 1) were included in the present study (Figure 1). Participants included vs. not included in this study were similar in age and number of years of education (p > 0.05) but differed significantly in sex and socioeconomic status (p < 0.05). There were proportionally more males in the included sample (58.8%) than in the not-included sample (50.2%) and fewer adolescents of lower socioeconomic status in the included vs. not-included sample (8.8% vs. 12.8%, respectively).

Figure 1
Flow diagram of participants from the Brazilian High-Risk Cohort included in the analysis. DAWBA = Development and Well-Being Behavior Assessment.

Predicting variables

Eight predictor variables were used in this model: sex, skin color, childhood maltreatment, school failure, social isolation, fights, ran away from home, and drug use. Predictor variables were evaluated at Wave 1 in the BHRC. Three variables from the original IDEA-RS model were not available in the BHRC dataset and were not included: relationship with mother, relationship with father, and relationship between parents. All included variables were selected and harmonized with the original IDEA-RS. Table S1, available as online-only supplementary material, presents further details on the definition and assessment of the variables, which closely matched the predictors used in the IDEA-RS model.

Outcome variable

The outcome of interest was a categorical diagnosis of major depressive episode (MDD), other depression, or mixed anxiety-depression disorder at Wave 2 of the BHRC, assessed using the Development and Well-Being Behavior Assessment (DAWBA).¹⁷,¹⁸ Depressive symptoms were assessed during the past month (current depression). The DAWBA is a structured interview that generates diagnoses according to the DSM-IV criteria.¹⁹ It was administered by lay interviewers to parents or primary caregivers in Wave 0 and both to parents or primary caregivers and to the child or adolescent in Waves 1 and 2, and further rated by trained supervised psychiatrists. All interviewers were extensively trained by the research team and their work was subject to constant supervision throughout the project. Diagnoses were assigned by one of nine trained psychiatrists via a computerized platform using the information acquired during the interviews. These psychiatrists were trained by attending several meetings led by a senior child psychiatrist with significant experience rating the DAWBA. A second child psychiatrist rated a total of 200 interviews, and the kappa values between raters for the main diagnosis was high (0.80 for any disorder and 0.85 for emotional disorders).¹⁵

Statistical analysis

The original IDEA-RS using data from the Pelotas 1993 Birth Cohort was a logistic-regression predictive model developed using penalized maximum likelihood estimation (to minimize overfitting). The original model comprising 11 predictors of adolescent depression¹² was recreated in the original sample, eliminating the three predictors that were not available in the BHRC. Standard external validation was used to predict depression in the BHRC sample. The regression coefficients of the model’s predictors and intercept were re-estimated in the BHRC sample (refitted model). A case mix-corrected model was also evaluated, as suggested by Steyerberg & Vickers.²⁰ Differences in participants’ characteristics between the development and validation samples can impact external validation. To account for that, the impact of differences in case-mix on the model’s validation performance was quantified. For this analysis, we assume that the regression coefficients for assessed predictors and the model intercept are fully correct for the validation setting. On simulating the outcome from the observed case-mix in the development sample, assuming the prediction model is correct for the new sample, differences in performance between the development and validation assessments suggest real differences in the weights of the regression coefficients.

The performance of the model was evaluated regarding discrimination and calibration properties. Discrimination – the model’s ability to separate those at higher risk of having an event from those at lower risk – was assessed by the AUC,²¹ a widely used metric of performance of prediction models.²² The AUC-evaluated discriminative ability of the test can be interpreted as the probability that a randomly selected subject with the condition has a higher risk score than that of a randomly chosen subject without the condition.²² The AUC value should be between 0.50 and 1; values > 0.50 indicate better-than-chance discrimination, and the closer the AUC value to 1, the better the model is able to discriminate between adolescents with and without depression at follow-up. Calibration refers to the agreement between model predictions and observed endpoints – in this case, actual observed rates of depression, This was assessed using calibration-in-the-large and calibration slope statistics.²³ The calibration intercept (calibration-in-the-large) compares the mean of all predicted risks with the mean observed risk and has a target value of 0; negative values indicate overestimation, whereas positive values suggest underestimation. The calibration slope evaluates the spread of the estimated risks and has a target value of 1. The Brier score is influenced by both discrimination and calibration simultaneously, and is used to give an indication of the overall performance of the model.²⁰ The Brier score estimates the mean squared distance between the observed (actual rates of depression) and expected (model predictions) outcomes. A lower Brier score (closer to zero) indicates more accurate predictions. Harrel’s Emax, describing the maximal absolute difference between observed and predicted probabilities of the outcome, was used to assess model calibration.

Results

A total of 1,442 students were included in the final analyses (a flowchart is shown in Figure 1), predominantly male (58.3%). Mean age at Wave 1 was 13.47 years (SD 1.88); the youngest individual was 9.21 and the oldest was 17.46. Mean age at Wave 2 was 17.72 years (SD 1.92), with the youngest individual aged 13 and the oldest aged 23.

Figure 2 presents descriptive variables for predictors included in the BHRC and Pelotas samples. There were some differences between the two samples regarding the frequencies of these variables (Figure 2). Female sex, school failure, drug use, fight involvement, and ran away from home were proportionally more frequent in the BHRC sample than in the original Pelotas sample. The prevalence of a depressive episode during follow-up was 12.4% (n=158), with an events per variable (EPV) ratio of 19.75.

Figure 2
Prevalence of risk variables in the Pelotas 1993 Birth Cohort (n=7,229) vs. Brazilian High-Risk Cohort (n=1,442), with 95% CIs. Proportions do not reach 100% because the figure only shows responses that indicate risk for the variables.

The model’s AUC (the measure assessing its discriminative capacity) was 0.65 (bootstrap-corrected 95%CI 0.60-0.68), lower in the BHRC than in the original sample (Table 1). The AUC for the case mix-corrected model was 0.70, suggesting differences in the weights of the regression coefficients. The AUC for the refitted model – i.e., corrected using coefficient estimates deemed optimal for the validation data – was 0.69. Overall performance was good (Brier score of 0.12), but worse than in the original dataset. There was some degree of miscalibration, improving significantly in the refitted model, where calibration-in-the large reduced from 0.77 to 0.

Table 1
Comparative results for each step of model validation in the two cohorts

		Pelotas cohort		Brazilian High-Risk Cohort
Performance parameter	Description	Apparent validation	Internal validation	External validation	Case-mix-corrected model†	Refitted model‡
C-statistic	Concordance statistic, equal to the area under the curve of the receiver operating characteristic (AUC-ROC) in binary endpoints.	0.78	0.71	0.65	0.70	0.69
Calibration-in-the-large	An overall measure of calibration, compares mean observed with mean predicted in the validation dataset.	0.00	0.02	0.77	-0.01	0.00
Calibration slope	Measure of agreement between observed and predicted risk of the event (outcome) across the whole range of predicted values.	1.26	1.00	0.71	1.00	1.00
R2	Measure of overall goodness-of-fit of the model.	0.12	0.06	-0.43	0.06	0.10
Brier score	Quadratic scoring rule that combines calibration and discrimination.	0.03	0.03	0.12	0.02	0.10
Emax	Maximum absolute error in predicted probabilities.	0.19	0.03	0.40	0.00	0.12
Available information for model validation		100%		93.10%

• Apparent validation: the model is evaluated directly in the derivation cohort; internal validation using bootstrapping evaluation with 1,000 iterations; external validation using the linear predictor derived from the selected Pelotas model.
• Higher results for C-statistic and R², lower results for Brier score and Emax, results closer to 0 for Calibration-in-the-large, and results closer to 1 for Calibration slope indicate better model performance.
• ^†
  Reference values indicating the model’s performance under the assumption that the Pelotas model coefficients are fully correct for the validation setting, simulating a similar case mix between samples.²⁰
• ^‡
  Reference values indicating the model’s performance after refitting predictor variable coefficients that would be optimal for the validation sample.²⁰

Discussion

External validation of prognostic models in medicine is a critical step to support utilization of these models in clinical and research settings. The IDEA-RS is a tool to predict adolescent depression that was developed in the Brazilian Pelotas 1993 Birth Cohort and validated in four samples in different countries. In the present study, we extend its replication to a fifth independent sample based in Brazil to assess how it performs within the country in which it was developed.

There was a 65% probability that a randomly selected adolescent who developed depression at ages 13 to 23 would have a higher risk score at a mean age of 13.5 than a randomly selected adolescent who did not develop depression at a mean age of 17.7 years. The discriminative ability in the BHRC cohort (AUC = 0.65) was slightly superior to its performance in the UK (AUC = 0.59), New Zealand (AUC = 0.63), and Nigerian (AUC = 0.62) cohorts, but lower than in Nepal (AUC = 0.73).¹²-¹⁴ The AUC increased to 0.70 in the case mix-corrected model, indicating acceptable discrimination.²²

The model underestimated the true observed rate of depression in the sample and achieved unsatisfactory calibration; the refitted model, however, demonstrated a significant improvement in calibration. The difference in outcome prevalence (lower in the Pelotas sample) may have impacted calibration. This may be at least in part explained by the difference between samples. The sample in this study differed from the original study, since the majority of children included were selected on the basis of a high index of family load for psychiatric disorders. The risk of developing a mental disorder, including depression, is significantly higher among the offspring of parents with a mental disorder.²⁴,²⁵ Having a parent diagnosed with a mental disorder also seems to impact prognosis when children receive a psychiatric diagnosis. Depression in adolescents is more severe and is associated with higher impairment when parents also have a depressive disorder.²⁶ Differences in the prevalence of predicting variables may also impact the results, since it expectedly changes pretest and post-test probabilities. This can ultimately lead to a loss of observed performance in external validation procedures.

External validation helps determine if the model’s predictive performance will remain stable in different populations with similar data acquisition, determining the extent to which information in the model can be applied generally across different samples. However, inherent differences in populations and data collection methods usually impacts model performance. This is the case for differences in disease prevalence, which may have a significant impact on the accuracy of prediction models.²⁷ As expected, correcting for such differences in the refitted model yielded better performance measures. Another reason for differences in model performance is that the effect of predictor variables may vary in different subgroups in a non-linear way, which may be missed when developing the model.²⁸ In this high-risk sample, family load of mental illness may add additional layers of complexity to how predictor variables interact, which is in turn difficult to predict.

This study is not without limitations. Data harmonization was not perfect (three variables were not available) and this may have impacted the external validation results. Risk factors in the Pelotas cohort were assessed at age 15. Since the BHRC is not a birth cohort, risk factors were assessed at different ages (between 9 and 17 years), so both samples were different when considering the timing of assessment of the at-risk component as well as the outcome of a depressive episode. Cumulative prevalence of the risk factors is expected to vary, increasing as the population gets older, which may account for differences in prevalence of the predictor variables. Differences in the prevalence of risk factors were also evident between the two samples, which contributes to case-mix variation and is a key cause of heterogeneity in model performance across different populations.²⁸ Additionally, differences in outcome prevalence may impact calibration. Finally, how the model performs in real world situations is still to be determined.

In conclusion, this model with eight easily obtainable sociodemographic predictors was able to predict depression beyond chance among adolescents in a diverse sample from the same country where it was developed. Further steps should include model improvements by adding other risk factors for depression, including biological metrics²⁹ such as polygenic risk scores, and pilot studies to assess its feasibility and acceptability in healthcare settings, as well as its utility in clinical decision making. A non-comparative research design is usually employed in early stages, such as a cross-sectional survey to determine how clinicians incorporate the investigation of risk factors to routine assessment and identify possible barriers to implementation. As a next step, comparative research designs may investigate potential effects of implementing the model, such as reduction in the incidence of a depressive episode. Risk thresholds may be needed to identify high-risk patients in clinical practice, and this decision must consider the possible benefits and harms of true and false positives, which may vary according to the clinical context. For this purpose, cost-effectiveness analysis for nonpharmacological interventions may be appropriate.

Acknowledgements

This study was supported by the Instituto Nacional de Psiquiatria do Desenvolvimento para Crianças e Adolescentes (INPD), funded by Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq; grant 465550/2014-2) and Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP; grant 2014/50917-0 and 2021/05332-8). This study was funded in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES; Finance Code 001). The IDEA project was funded by an MQ Brighter Futures grant (MQBF/1 IDEA). Additional support was provided by the UK Medical Research Council (MC_PC_MR/R019460/1) and the Academy of Medical Sciences (GCRFNG_100281) under the Global Challenges Research Fund. HLF was partly supported by the Economic and Social Research Council (ESRC) Centre for Society and Mental Health at King’s College London (ES/S012567/1). CZ received a Young Talent Research Scholarship from CAPES (grant 88887.575201/2020-00). BAK and CK are supported by the U.S. National Institute of Mental Health (R21MH124072). CK is a CNPq researcher and an Academy of Medical Sciences Newton Advanced Fellow. VM was supported by the National Institute for Health Research (NIHR) Biomedical Research Centre at South London and Maudsley NHS Foundation Trust and King’s College London. The views expressed are those of the authors and not necessarily those of the National Health Service, the NIHR, the Department of Health and Social Care, the ESRC, or King’s College London.

We are extremely grateful to the individuals who participated in this study and to all members of the IDEA team and the Brazilian High-Risk Cohort study for their dedication, hard work, and insights.

References

• ¹ GBD. Global, regional, and national incidence, prevalence, and years lived with disability for 354 diseases and injuries for 195 countries and territories, 1990-2017: a systematic analysis for the Global Burden of Disease Study 2017. Lancet (London, England). 2018;392:1789-1858.
• ² Olin B, Jayewardene AK, Bunker M, Moreno F. Mortality and suicide risk in treatment-resistant depression: an observational study of the long-term impact of intervention. PloS one. 2012;7:e48002.
• ³ Stevanovic D, Jancic J, Lakic A. The impact of depression and anxiety disorder symptoms on the health-related quality of life of children and adolescents with epilepsy. Epilepsia. 2011;52:e75-78.
• ⁴ Clayborne ZM, Varin M, Colman I. Systematic Review and Meta-Analysis: Adolescent Depression and Long-Term Psychosocial Outcomes. Journal of the American Academy of Child and Adolescent Psychiatry. 2019;58:72-79.
• ⁵ Liu Q, He H, Yang J, Feng X, Zhao F, Lyu J. Changes in the global burden of depression from 1990 to 2017: Findings from the Global Burden of Disease study. Journal of psychiatric research. 2020;126:134-140.
• ⁶ Curry J, Silva S, Rohde P, Ginsburg G, Kratochvil C, Simons A, et al. Recovery and recurrence following treatment for adolescent major depression. Archives of general psychiatry. 2011;68:263-269.
• ⁷ Kieling C, Adewuya A, Fisher HL, Karmacharya R, Kohrt BA, Swartz JR, et al. Identifying depression early in adolescence. The Lancet Child & adolescent health. 2019;3:211-213.
• ⁸ Mitchell GF, Hwang SJ, Vasan RS, Larson MG, Pencina MJ, Hamburg NM, et al. Arterial stiffness and cardiovascular events: the Framingham Heart Study. Circulation. 2010;121:505-511.
• ⁹ Bernardini F, Attademo L, Cleary SD, Luther C, Shim RS, Quartesan R, et al. Risk Prediction Models in Psychiatry: Toward a New Frontier for the Prevention of Mental Illnesses. The Journal of clinical psychiatry. 2017;78:572-583.
• ¹⁰ Salazar de Pablo G, Studerus E, Vaquerizo-Serrano J, Irving J, Catalan A, Oliver D, et al. Implementing Precision Psychiatry: A Systematic Review of Individualized Prediction Models for Clinical Practice. Schizophrenia bulletin. 2021;47:284-297.
• ¹¹ Meehan AJ, Lewis SJ, Fazel S, Fusar-Poli P, Steyerberg EW, Stahl D, et al. Clinical prediction models in psychiatry: a systematic review of two decades of progress and challenges. Molecular psychiatry. 2022.
• ¹² Rocha TB, Fisher HL, Caye A, Anselmi L, Arseneault L, Barros FC, et al. Identifying Adolescents at Risk for Depression: A Prediction Score Performance in Cohorts Based in 3 Different Continents. Journal of the American Academy of Child and Adolescent Psychiatry. 2021;60:262-273.
• ¹³ Brathwaite R, Rocha TB, Kieling C, Gautam K, Koirala S, Mondelli V, et al. Predicting the risk of depression among adolescents in Nepal using a model developed in Brazil: the IDEA Project. European child & adolescent psychiatry. 2021;30:213-223.
• ¹⁴ Brathwaite R, Rocha TB, Kieling C, Kohrt BA, Mondelli V, Adewuya AO, et al. Predicting the risk of future depression among school-attending adolescents in Nigeria using a model developed in Brazil. Psychiatry Res. 2020;294:113511.
• ¹⁵ Salum GA, Gadelha A, Pan PM, Moriyama TS, Graeff-Martins AS, Tamanaha AC, et al. High risk cohort study for psychiatric disorders in childhood: rationale, design, methods and preliminary results. International journal of methods in psychiatric research. 2015;24:58-73.
• ¹⁶ Weissman MM, Wickramaratne P, Adams P, Wolk S, Verdeli H, Olfson M. Brief screening for family psychiatric history: the family history screen. Archives of general psychiatry. 2000;57:675-682.
• ¹⁷ Fleitlich-Bilyk B, Goodman R. Prevalence of child and adolescent psychiatric disorders in southeast Brazil. Journal of the American Academy of Child and Adolescent Psychiatry. 2004;43:727-734.
• ¹⁸ Goodman R, Ford T, Richards H, Gatward R, Meltzer H. The Development and Well-Being Assessment: description and initial validation of an integrated assessment of child and adolescent psychopathology. Journal of child psychology and psychiatry, and allied disciplines. 2000;41:645-655.
• ¹⁹ American Psychiatric Association: Diagnostic and statistical manual of mental disorders: DSM-IV. 4 ed. Washington, APA; 1994.
• ²⁰ Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology (Cambridge, Mass). 2010;21:128-138.
• ²¹ Steyerberg EW. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. New York, NY, Springer; 2019.
• ²² Mandrekar JN. Receiver operating characteristic curve in diagnostic test assessment. Journal of Thoracic Oncology. 2010;5:1315-1316.
• ²³ Fenlon C, O’Grady L, Doherty ML, Dunnion J. A discussion of calibration techniques for evaluating binary and categorical predictive models. Preventive veterinary medicine. 2018;149:107-114.
• ²⁴ Rasic D, Hajek T, Alda M, Uher R. Risk of mental illness in offspring of parents with schizophrenia, bipolar disorder, and major depressive disorder: a meta-analysis of family high-risk studies. Schizophrenia bulletin. 2014;40:28-38.
• ²⁵ Rice F, Harold G, Thapar A. The genetic aetiology of childhood depression: a review. Journal of child psychology and psychiatry, and allied disciplines. 2002;43:65-79.
• ²⁶ Weissman MM, Warner V, Wickramaratne P, Moreau D, Olfson M. Offspring of depressed parents. 10 Years later. Archives of general psychiatry. 1997;54:932-940.
• ²⁷ Morise AP, Diamond GA, Detrano R, Bobbio M, Gunel E. The effect of disease-prevalence adjustments on the accuracy of a logistic prediction model. Medical decision making : an international journal of the Society for Medical Decision Making. 1996;16:133-142.
• ²⁸ Riley RD, Ensor J, Snell KI, Debray TP, Altman DG, Moons KG, et al. External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges. bmj. 2016;353.
• ²⁹ Kieling C, Buchweitz C, Caye A, Manfro P, Pereira R, Viduani A, et al. The Identifying Depression Early in Adolescence Risk Stratified Cohort (IDEA-RiSCo): Rationale, Methods, and Baseline Characteristics. Frontiers in psychiatry. 2021;12:697144.

Publication Dates

• Publication in this collection
  03 July 2023
• Date of issue
  May-Jun 2023

History

• Received
  11 July 2022
• Accepted
  23 Mar 2023