The use of e-ASPECTS in acute stroke care: validation of method performance compared to the performance of specialists

O uso do e-ASPECTS no AVC agudo: validação da performance do método comparada com a performance dos especialistas

ABSTRACT

Background:

The rapid evaluation of non-contrast-enhanced computed tomography (NCCT) brain scans in patients with anterior stroke symptoms saves time and favors optimal and prompt treatment. e-ASPECTS is a tool that automatically calculates the Alberta Stroke Program Early CT Score (ASPECTS) values, leading to a more accurate and timely image evaluation.

Objective:

To determine the ability of e-ASPECTS in differentiating images with and without injury.

Methods:

One-hundred sixteen patients admitted to a stroke unit in a Brazilian tertiary hospital underwent a CT scan at admission and at least one control brain imaging (NCCT or magnetic resonance imaging - MRI) 24 hours after admission. ASPECTS evaluation was performed by three neuroradiologists, three neurologists, and three neurology residents, all blinded to the symptoms and the injury side. The scores were compared to the ground truth, and an ASPECTS score was provided by two independent non blinded evaluators. Sensitivity and specificity were analyzed, and receiver operating characteristic curves, Bland-Altman plots with mean error score, and Matthews correlation coefficients (MCCs) were obtained for ASPECTS scores, assuming values equal to 10 for images without injury and values other than 10 for images with ischemic injury.

Results:

e-ASPECTS demonstrated similar performance to that of neuroradiologists and neurologists, with an area under the curve of 0.78 and an MCC value of 0.48 in the dichotomous analysis. The sensitivity and specificity of e-ASPECTS were 75% and 73%, respectively.

Conclusion:

e-ASPECTS is a validated and reliable tool for determining early signs of ischemia in NCCT.

Keywords:
Stroke; Infarction, Anterior Cerebral Artery; Tomography, X-Ray Computed; Diagnostic Imaging

RESUMO

Introdução:

A avaliação rápida da tomografia de crânio sem contraste (TCSC) em pacientes com AVC de circulação anterior economiza tempo e permite um tratamento rápido e otimizado. O e-ASPECTS é um software que calcula automaticamente os valores do ASPECTS e permite uma avaliação da imagem mais precisa e ágil.

Objetivo:

Determinar a habilidade do e-ASPECTS em diferenciar imagens com e sem lesão.

Métodos:

Cento e dezesseis pacientes admitidos em uma unidade de AVC de um hospital terciário brasileiro foram submetidos a uma TCSC na admissão e pelo menos uma imagem de controle (TC ou Ressonância de Crânio) 24 horas após a admissão. A avaliação do ASPECTS foi realizada por três neurorradiologistas, três neurologistas e três residentes em neurologia, todos cegados para os sintomas e para o lado da lesão. Os valores foram comparados ao ground truth (GT) e uma pontuação ASPECTS foi obtida por dois avaliadores independentes não cegos. Análise da sensibilidade e especificidade, características das curvas ROC, gráficos de Bland-Altman com média de escore de erro e coeficientes de correlação de Matthews (CCM) foram realizados para os valores de ASPECTS, assumindo valores iguais a 10 como imagens sem lesões e valores diferentes de 10 como imagens com alguma lesão isquêmica.

Resultados:

o e-ASPECTS demonstrou uma performance similar aos neurorradiologistas e neurologistas, com uma área sob a curva de 0,78 e um valor de CCM de 0,48 na análise dicotômica. Sensibilidade e especificidade do e-ASPECTS foram, respectivamente, 75 e 73%.

Conclusão:

O e-ASPECTS é uma ferramenta confiável e validada para determinar sinais precoces de isquemia nas TCSC.

Palavras-chave:
Acidente Vascular Cerebral; Infarto da Artéria Cerebral Anterior; Tomografia Computadorizada por Raios X; Diagnóstico por Imagem

INTRODUCTION

Acute ischemic stroke management has advanced in the last three decades, from the introduction of intravenous thrombolysis to the validation of mechanical thrombectomy¹. In all studies, brain image analysis has become a significant tool for verifying the presence of early ischemic changes^2,3,4,5 in order to exclude the presence of intracranial hemorrhage² and indicate the correct reperfusion therapy management^6,7.

In recent years, a software-based machine learning algorithm (e-ASPECTS, Brainomix®, Oxford, UK; www.brainomix.com) has been developed based on the Alberta Stroke Program Early Computed Tomography Score (ASPECTS), a simple and reliable method for assessing early ischemic stroke signs in non-contrast-enhanced computed tomography (NCCT) brain scan. e-ASPECTS is a 10-point quantitative score used in patients with symptoms suggestive of middle cerebral artery (MCA) ischemic stroke. The scores range from 10 (normal image) to 0 (all regions present early signs)⁸. e-ASPECTS has demonstrated a good correlation compared to assessments by neuroradiologists, stroke neurologists, and residents in neurology, but current analyses have been performed only in high-income countries. In low- and middle-income countries, the use of this software may improve time-to-treatment, which is still far less than ideal^6,9.

The current study aimed to evaluate the ability of e-ASPECTS in identifying injuries in NCCT in a tertiary care hospital and compare the performance of e-ASPECTS with that of three groups of trained professionals.

METHODS

All patients with suspected acute ischemic stroke admitted to the Hospital de Clínicas of the Universidade Federal do Paraná between March 2017 and February 2018 were evaluated and considered for the current study. The inclusion criteria were as follows:

1. hospital admission within the first 24 hours of symptom onset;

2. NCCT performed upon admission;

3. anterior circulation stroke symptoms;

4. NCCT or magnetic resonance imaging (MRI) performed to confirm anterior circulation ischemic stroke 24 hours after admission.

The exclusion criteria were a diagnosis other than ischemic stroke, such as hemorrhagic stroke, the presence of transient ischemic attack (TIA), and the absence of stroke (stroke mimics).

All patients underwent brain NCCT upon admission and NCCT or MRI to confirm the ischemic stroke 24 hours after admission. All patients were treated according to the current guidelines.

Patients were examined on a multislice CT system using a 0.5-mm thick slice with 64 simultaneous detector rows (Aquilion 64, Toshiba Medical Systems, Japan). Axial brain NCCT images were reconstructed in 2-mm and 5-mm thick slices using FC64 kernels.

NCCT images were retrospectively analyzed by three neuroradiologists (NR1, NR2, NR3) with at least 5 years of experience in stroke imaging analysis, three neurologists (N1, N2, N3) with at least 5 years of experience in acute stroke care, and three neurology residents (R1, R2, R3) with one year of experience in stroke medicine. All groups were trained on the ASPECTS method prior to the analysis of the images, and they were able to discern acute and chronic signs of ischemic stroke. All evaluators were blinded to the symptoms and the injury side.

e-ASPECTS is a fully automated software that quickly calculates the ASPECTS value based on machine learning algorithms. This software can detect acute and non-acute hypodensity in NCCT, but only early signs of acute ischemic stroke are scored in the ASPECTS final value¹⁰.

The ground truth corresponded to the e-ASPECTS values provided by an independent stroke neurologist and a neuroradiologist who were not blinded to any clinical information. The values were based on CT or MRI images after 24 hours, as well as all other imaging or clinical information in the patient medical records.

A receiver operating characteristic (ROC) curve was calculated for each evaluator to obtain a dichotomized ASPECTS score based on normal (ASPECTS=10) or abnormal (ASPECTS<10) findings. The objective was to evaluate whether the observer had identified anormalCTwhen the CT was in factnormal, and, similarly, whether they had identified an injury where there was an injury.

Sensitivity, specificity, positive predictive value, negative predictive value, and accuracy were calculated for e-ASPECTS and all three groups, using the standard method and the sandwich estimator, the generalized estimating equation (GEE). While the standard method presents the results obtained by the simple proportion of individuals without considering other effects of the sample, GEE reveals the average sensitivity and specificity across all patient-specific estimates of sensitivity. GEE, therefore, is interpreted as the weighted average across the study population¹¹ and is more robust than the traditional model¹². The Matthews correlation coefficient (MCC), which ranges between -1 and +1, was calculated for each data point. For an optimal demonstration of error distribution, Bland-Altman plots and histograms were constructed for the three neuroradiologists and e-ASPECTS. Data were tabulated in Microsoft Excel 2016, and statistical analyses were performed with IBM SPSS Statistics, version 25.0, and RS Studio, version 1.2.1335.

RESULTS

Among the 445 initially selected patients, 116 fulfilled the inclusion criteria of the study. We excluded 279 patients due to hemorrhagic stroke, non-anterior stroke, symptom improvement, or lack of NCCT or MRI data after 24 hours. Another 100 patients had different final diagnoses (TIA and stroke mimics).

Table 1 presents the baseline characteristics of the sample. e-ASPECTS showed an accuracy of 74.14%, which was only lower than that of one neuroradiologist (83.62%) and higher than the accuracies of all other experts and neurology residents (Table 2).

Table 1.
Baseline patient characteristics.

Characteristics	116
Age, in years (mean±SD)	65.83±14.20
Female n (%)	54 (49.1)
Medical history
Hypertension, n (%)	89 (76.72)
Previous stroke, n (%)	17 (14.66)
Previous TIA, n (%)	8 (6.9)
Diabetes mellitus, n (%)	32 (27.59)
Atrial fibrillation, n (%)	20 (17.24)
Hypercholesterolemia, n (%)	50 (43.1)
Current smoker, n (%)	27 (23.28)
Imaging features
MRI	36 (31.03)
CTA or MRA	67 (57.76)
CTA occlusion	28 (24.14)
Treatment
NIH stroke scale at admission (median) (IQR)	8 (4-16)
IV rtPA, n (%)	60 (51.72)
Mechanical thrombectomy, n (%)	2 (1.7)
Onset-to-door time, in min (mean±SD)	171.81±124.06
Door-to-needle time, in min (mean±SD)	42.81±27.45
Symptom-to-needle, in min (mean±SD)	179.60±51.17

• SD: standard deviation; TIA: transient ischemic attack; MRI: magnetic resonance imaging; CTA: computed tomography angiography; MRA: magnetic resonance angiography; NIH: National Institutes of Health; IV rtPA: intravenous recombinant tissue plasminogen activator.

Table 2.
Score-based analyses and Matthews correlation coefficient for all methods.

	Method	Measure	e-ASPECTS	NR1	NR2	NR3	N1	N2	N3	R1	R2	R3
Score-based	-	TP	46	60	25	49	43	37	27	23	26	22
	-	TN	40	10	51	48	40	35	47	43	43	33
	-	FP	15	45	4	7	15	20	8	12	12	22
	-	FN	15	1	36	12	18	24	34	38	35	39
	Standard	Sensitivity (CI)	75.4% (62.7; 85.5)	98.4% (91.2; 100)	41% (28.6; 54.3)	80.3% (68.2; 89.4)	70.5% (57.4; 81.5)	60.7% (47.3; 72.9)	44.3% (31.6; 57.6)	37.7% (25.6; 51)	42.6% (30; 55.9)	36.1% (24.2; 49.4)
	Standard	Specificity (CI)	72.7% (59.0; 83.9)	18.2% (9.1; 30.9)	92.7% (82.4; 98.0)	87.3% (75.5; 94.7)	72.7% (59; 83.9)	63.6% (49.6; 76.2)	85.5% (73.3; 93.5)	78.2% (65.0; 88.2)	78.2% (65.0; 88.2)	60% (45.9; 73.0)
	GEE	Sensitivity (CI)	60.8% (52.1; 68.9)	74.9% (65.8; 82.2)	27.5% (19.8; 36.8)	67.3% (61.9; 72.2)	36.3% (27.9; 45.5)	37.4% (30; 45.5)	19.9% (13.8; 27.8)	14% (8.8; 21.6)	16.4% (10.2; 25.3)	5.9% (3.6; 9.4)
	GEE	Specificity (CI)	96.2% (93.4; 97.8)	69.1% (58.2; 78.2)	97.5% (96.3; 98.3)	91.5% (88.6; 93.7)	92.1% (87.6; 95.1)	89.6% (86; 92.3)	96.2% (94.5; 97.3)	94.4% (92.9; 95.6)	95.2% (93.2; 96.6)	93.5% (91; 95.4)
	-	PPV	75.41%	57.14%	86.21%	87.50%	74.14%	64.91%	77.14%	65.71%	68.42%	50.00%
	-	NPV	72.73%	90.91%	58.62%	80.00%	68.97%	59.32%	58.02%	53.09%	55.13%	45.83%
	-	Accuracy	74.14%	60.34%	65.52%	83.62%	71.55%	62.07%	63.79%	56.0%	59.48%	47.41%
		MCC (Score-based)	0.48	0.28	0.39	0.68	0.43	0.24	0.32	0.17	0.22	-0.04

• TP: true positive; TN: true negative; FP: false positive; FN: false negative; NR: neuroradiologist; N: neurologist; R: neurology resident; GEE: generalized estimating equations; MCC: Matthews correlation coefficient; PPV: positive predictive value; NPV: negative predictive value; CI: confidence Interval (95%).

When analyzing the ROC curve, e-ASPECTS had area under the curve values similar to those of all experts and higher than those of two neurology residents (Figure 1). MCC demonstrated a positive correlation (0.48) for e-ASPECTS, which was only lower than that for one neuroradiologist (0.68). One neurology resident presented a negative MCC (-0.04).

Figure 1.
Receiveroperating characteristic (ROC) curves for e-ASPECTS compared to neurology residents (a); neurologists (b), and neuroradiologists (c).

The error distribution for e-ASPECTS was 0.250 (-2.307 - 2.807), which was lower than that observed for all neuroradiologists, -0.552 (-4.931 - 3.827), as presented in Figure 2.

Figure 2.
Bland-Altman and associated histograms of mean error score. Bland-Altman plots (left panel) with mean error score (gray dashed line) and histograms (right panel) of error score for e-ASPECTS (a) and neuroradiologists (NRDs) (b).

DISCUSSION

The present study showed that, considering all diagnostic criteria, e-ASPECTS has similar performance to that of neuroradiologists and neurologists in evaluating early signs of ischemia from NCCT; thus, e-ASPECTS is a safe method that can be used in stroke care.

e-ASPECTS is a fast and reliable tool for the initial evaluation of stroke images. Furthermore, recent investigations have shown a strong relationship between e-ASPECTS and clinical outcomes¹³.

The current study confirms previous noninferior results of e-ASPECTS compared to neuroradiologists in the analysis of early signs of acute ischemic stroke from NCCT¹⁴. In addition, this study corroborates the good performance of e-ASPECTS that had been previously reported by Herweh et al.¹⁵.

e-ASPECTS does not seem to have the same performance as neuroradiologists when the images demonstrate other lesions besides the early signs of ischemia, including leukoaraiosis, old infarcts, and atypical parenchymal defects such as calcification. Although e-ASPECTS is able to differentiate between old and recentinfarcts, previous studies have not found a significant correlation for patients with pre-existing changes¹⁰. Our research did not exclude patients with old infarcts or any other lesion. Nevertheless, these confounders did not affect the software performance¹⁰.

As a reliable tool comparable to neuroradiologist assessments, e-ASPECTS can assist neurology residents in their acute stroke image analysis training. The lower performance observed among neurology residents may be improved with the use of e-ASPECTS, which can check their mistakes in real time.

To the best of our knowledge, this study was the first to validate e-ASPECTS in a large developing country. Nonetheless, this study has limitations. This was a single-center investigation with a small sample size. The independent evaluators used different machines and screening tools to assess the images. Due to operational and availability issues, diffusion-weighted MRI (DW-MRI) was not the only method used for ground truth determination, as was the case in other studies¹⁵.Some follow-up images were obtained after thrombolysis, which may have altered the final area of infarction.

In conclusion, the current study found that e-ASPECTS demonstrated similar performance to that of neuroradiologists and neurologists, but it performed better than neurology residents in differentiating NCCT images with or without early signs of ischemia, thus making it a reliable method in Brazil.

References

• ¹
  Zerna C, Thomalla G, Campbell BCV,Rha JH, Hill MD et al. Current practice and future directions in the diagnosis and acute treatment of ischaemic stroke. Lancet 2018 Oct; 392(10154): 1247-56. https://doi.org/10.1016/S0140-6736(18)31874-9
  » https://doi.org/https://doi.org/10.1016/S0140-6736(18)31874-9
• ²
  The National Institute of Neurological Disorders and Stroke rt-PA Stroke Study Group. Tissue plasminogen activator for acute ischemic stroke. N Engl J Med. 1995 Dec; 333(24):1581-7. https://doi.org/10.1056/nejm199512143332401
  » https://doi.org/https://doi.org/10.1056/nejm199512143332401
• ³
  Hill MD, Demchuk AM, Goyal M, Jovin TG, Foster LD, Tomsick TA, et al. Alberta stroke program early computed tomography score to select patients for endovascular treatment: interventional management of stroke (IMS)-III trial. Stroke 2014 Feb; 45(2): 444-9. https://doi.org/10.1161/strokeaha.113.003580
  » https://doi.org/https://doi.org/10.1161/strokeaha.113.003580
• ⁴
  Dzialowski I, Hill MD, Coutts SB, Demchuk AM, Kent DM, Wunderlich O, et al. Extent of early ischemic changes on computed tomography (CT) before thrombolysis: prognostic value of the Alberta stroke program early CT score in ECASS II. Stroke 2006 Apr; 37(4): 973-8. https://doi.org/10.1161/01.str.0000206215.62441.56
  » https://doi.org/https://doi.org/10.1161/01.str.0000206215.62441.56
• ⁵
  Hacke W, Kaste M, Fieschi C, Toni D, Lesaffre E, von Kummer R, et al. Intravenous thrombolysis with recombinant tissue plasminogen activator for acute hemispheric stroke. The European Cooperative Acute Stroke Study (ECASS). JAMA 1995 Oct; 274(13):1017-25. PMID: 7563451.
• ⁶
  Saver JL, Goyal M, van der Lugt A, Menon BK, Majoice CB, Dippel DW, Campbell BC, et al. Time to treatment with endovascular thrombectomy and outcomes from ischemic stroke: a meta-analysis. JAMA 2016 Sep; 316(12):1279-88. https://doi.org/10.1001/jama.2016.13647
  » https://doi.org/https://doi.org/10.1001/jama.2016.13647
• ⁷
  Emberson J, Lees KR, Lyden P, Blackwell L, Albers G, Bluhmki E, et al. Effect of treatment delay, age, and stroke severity on the effects of intravenous thrombolysis with alteplase for acute ischemic stroke: a meta-analysis of individual patient data from randomized trials. Lancet 2014 Nov; 384(9958):1929-35. https://doi.org/10.1016/s0140-6736(14)60584-5
  » https://doi.org/https://doi.org/10.1016/s0140-6736(14)60584-5
• ⁸
  Barber PA, Demchuk AM, Zhang J, Buchan AM. Validity and reliability of a quantitative computed tomography score in predicting outcome of hyperacute stroke before thrombolytic therapy. ASPECTS Study Group. Alberta Stroke Programme Early CT Score. Lancet 2000 May; 355(9216):1670-4. https://doi.org/10.1016/s0140-6736(00)02237-6
  » https://doi.org/https://doi.org/10.1016/s0140-6736(00)02237-6
• ⁹
  Lange MC, Braga GP, Novak EM, Harger R, Felippe MJDB, Canever M, et al. Key performance indicators for stroke from the Ministry of Health of Brazil: benchmarking and indicator parameters. Arq. Neuro-psiquiatr 2017 Jun; 75 (6): 354-8. https://doi.org/10.1590/0004-282x20170051
  » https://doi.org/https://doi.org/10.1590/0004-282x20170051
• ¹⁰
  Guberina N, Dietrich U, Radbruch A, Goebel J, Deuschl C, Ringelstein A, et al. Detection of early infarction signs with machine learning-based diagnosis by means of the Alberta Stroke Program Early CT score (ASPECTS) in the clinical routine. Neuroradiology. 2018 Sep;60(9):889-901. https://doi.org/10.1007/s00234-018-2066-5
  » https://doi.org/https://doi.org/10.1007/s00234-018-2066-5
• ¹¹
  Genders TS, Spronk S, Stijnen T, Steyerberg EW, Lesaffre E, and Hunink MG. Methods for calculating sensitivity and specificity of clustered data: a tutorial. Radiology 2012 Dec; 265(3): 910-6. https://doi.org/10.1148/radiol.12120509
  » https://doi.org/https://doi.org/10.1148/radiol.12120509
• ¹²
  Kirkwood BR, Sterne JAC. Essential medical statistics. 2nd ed. Oxford, (England): Blackwell Publishers; 2003.
• ¹³
  Nagel S, Wang X, Carcel C, Robinson T, Lindley RI, Chalmers J, et al. Clinical utility of electronic Alberta Stroke Program early computed tomography score software in the ENCHANTED Trial Database. Stroke 2018 Jun;49(6):1407-11. https://doi.org/10.1161/strokeaha.117.019863
  » https://doi.org/https://doi.org/10.1161/strokeaha.117.019863
• ¹⁴
  Nagel S, Sinha D, Day D, Reith W, Chapot R, Papanagiotou P, et al. e-ASPECTS software is non-inferior to neuroradiologists in applying the ASPECT score to computed tomography scans of acute ischemic stroke patients. Int J Stroke 2017 Aug; 12(6):615-22. https://doi.org/10.1177/1747493016681020
  » https://doi.org/https://doi.org/10.1177/1747493016681020
• ¹⁵
  Herweh C, Ringleb PA, Rauch G, Gerry S, Behrens L, Möhlenbruch M, et al. Performance of e-ASPECTS software in comparison to that of stroke physicians on assessing CT scans of acute ischemic stroke patients. Int J Stroke 2016 Jun; 11(4): 438-45. https://doi.org/10.1177/1747493016632244
  » https://doi.org/https://doi.org/10.1177/1747493016632244

Publication Dates

• Publication in this collection
  14 Dec 2020
• Date of issue
  Dec 2020

History

• Reviewed
  03 Mar 2020
• Received
  23 Apr 2020
• Accepted
  16 May 2020