Performance evaluation of a Sars-CoV-2 rapid test and two automated immunoassays

ABSTRACT

Introduction: Due to urgency and demand of a response to the Covid-19 pandemic, numerous Sars-CoV-2 immunoassays have been rapidly developed. Objective: This study aimed at assessing the performance of rapid Sars-CoV-2 antibody test in comparison to high-throughput serological assays. Methods: A total of 86 serum samples were evaluated in the three assays: a lateral flow immunoassay - Wondfo Sars-CoV-2 Antibody Test (WRT) - and two chemiluminescence immunoassays: Elecsys Anti-Sars-CoV-2 (ECLIA), and Sars-CoV-2 IgG (CMIA-IgG). Results: The estimated diagnostic sensitivities of serological tests in the evaluation of serum samples from the epidemiological survey were: WRT 59% [95% confidence interval (CI) 43.4%-72.9%], ECLIA 66.7% (51%-79.4%), and CMIA-IgG 61.5% (47.1%-73%). Meanwhile, the estimated diagnostic specificity was for WRT 78.7% (95% CI 65.1%-88%), ECLIA 72.3% (58.2%-83.1%), and CMIA-IgG 76.6% (74%-95.5%). The sensitivity and specificity values were lower than manufacturers’ claimed. Although 16.2% (14/86) of serological results were discordant among the three Sars-CoV-2 serological assays, the degree of agreement by the kappa index was adequate: WRT/CMIA-IgG [0.757 (95% CI 0.615-0.899)], WRT/ECLIA [0.715 (0.565-0.864)], and ECLIA/CMIA-IgG [0.858 (0.748-0.968)]. Conclusion: The serological testing may be a useful diagnostic tool, which reinforces its careful evaluation, and, as well as the correct time to use it.

Key words: coronavirus infections; serology; antibodies; point of care testing; immunoassay; betacoronavirus

RESUMEN

Introducción: Debido a la urgencia y la demanda de una respuesta a la pandemia de Covid-19, se han desarrollado rápidamente numerosos inmunoensayos del Sars-CoV-2. Objetivo: Este estudio tuvo como objetivo evaluar el rendimiento de la prueba rápida de anticuerpos contra el Sars-CoV-2 en comparación con los ensayos serológicos de alto rendimiento. Métodos: Se evaluaron un total de 86 muestras de suero en los tres ensayos: un inmunoensayo de flujo lateral - Wondfo Sars-CoV-2 Antibody Test (TRW) - y dos inmunoensayos de quimioluminiscencia: Elecsys Anti-Sars-CoV-2 (ECLIA) y Sars-CoV-2 IgG (CMIA-IgG). Resultados: Las sensibilidades diagnósticas estimadas de las pruebas serológicas en la evaluación de muestras de suero de la encuesta epidemiológica fueron: WRT 59% [intervalo de confianza (IC) del 95%: 43,4%-72,9%], ECLIA 66,7% (51%-79,4%) y CMIA-IgG 61,5% (47,1%-73%). Mientras tanto, la especificidad diagnóstica estimada fue para WRT 78,7% (95% CI 65,1%-88%), ECLIA 72,3% (58,2%-83,1%) y CMIA-IgG 76,6% (74%-95,5%). Los valores de sensibilidad y especificidad fueron más bajos que los declarados por los fabricantes. Aunque el 16,2% (14/86) de los resultados fueron discordantes entre los tres ensayos serológicos del Sars-CoV-2, el grado de concordancia del índice kappa fue adecuado: WRT/CMIA-IgG [0,757 (IC del 95%: 0,615-0,899)], WRT/ECLIA [0,715 (0,565-0,864)] y ECLIA/CMIA-IgG [0,858 (0,748-0,968)]. Conclusión: La prueba serológica puede ser una herramienta diagnóstica útil, lo que refuerza su evaluación cuidadosa, así como el momento adecuado para usarla.

Palabras clave: infecciones por coronavirus; serología; anticuerpos; pruebas en el punto de atención; inmunoensayo; betacoronavirus

RESUMO

Introdução: Em função da urgência e demanda de uma resposta à pandemia do novo coronavírus (Covid-19), vários testes de detecção de anticorpos para a síndrome respiratória aguda grave do coronavírus 2 (Sars-CoV-2) têm sido desenvolvidos. Objetivo: Este estudo teve como objetivo avaliar o desempenho do teste rápido utilizado em um inquérito epidemiológico para Sars-CoV-2 em comparação com outros ensaios sorológicos. Métodos: Foram avaliadas 86 amostras de soro em três ensaios sorológicos: um imunoensaio de fluxo lateral - Wondfo Sars-CoV-2 Antibody Test (TRW) - e dois imunoensaios de quimioluminescência: Elecsys anti-Sars-CoV-2 (ECLIA) e Sars-CoV-2 IgG (CMIA-IgG). Resultados: As sensibilidades diagnósticas estimadas dos testes sorológicos na avaliação dessas amostras foram: TRW 59% [95% intervalo de confiança (IC) 43,4%-72,9%], ECLIA 66,7% (51%-79,4%) e CMIA-IgG 61,5% (47,1%-73%). Enquanto isso, a especificidade diagnóstica estimada para TRW foi 78,7% (95% CI 65,1%-88%), ECLIA 72,3% (58,2%-83,1%) e CMIA-IgG 76,6% (74%-95,5%). Os valores de sensibilidade e especificidade foram inferiores aos afirmados pelos fabricantes. Embora 16,2% (14/86) dos resultados tenham sido discordantes entre os três ensaios serológicos para Sars-CoV-2, o grau de concordância pelo índice Kappa foi adequado: TRW/CMIA-IgG [0,757 (95% IC 0,615-0,899)], TRW/ECLIA [0,715 (0,565-0,864)] e ECLIA/CMIA-IgG [0,858 (0,748-0,968)]. Conclusão: O teste sorológico pode ser uma ferramenta diagnóstica útil, o que reforça sua avaliação criteriosa, bem como o momento correto de sua utilização.

Unitermos: infecções por coronavírus; sorologia; anticorpos; testes imediatos; imunoensaio; betacoronavírus

INTRODUCTION

Laboratory diagnosis has played a fundamental role in rapid diagnostic of coronavirus disease 2019 (Covid-19) pandemic(¹). Some different methods are used routinely to confirm the diagnostic or in epidemiological survey and post-vaccination seroconversion.

Among nucleic acid tests, the reverse-transcriptase polymerase chain reaction (RT-PCR) method is considered as the ‘gold standard’ for the detection of severe acute respiratory syndrome coronavirus 2 (Sars-CoV-2) because of its benefits as specificity and for representing a simple qualitative assay(²). However, in some situations, the sensitivity of RT-PCR tests has been sub-optimal due to different issues as: low viral loads at the time of infection, improper sample collection, conservation and transport of sample, and characteristics of gene targets(^2-5).

In cases with high clinical suspicion of infection with negative PCR results, antibody detection can be a useful tool to establish exposure, infection, and immunity to Sars-CoV-2, as well as to perform epidemiological studies(^5,6). Compared to PCR, serological test is advantageous because of its faster turnaround time, high-throughput, less workload, and affordability of samples(^3,7).

Serology plays a key role in contact tracing, epidemiological survey, identification of convalescent plasma donors (which may serve as a possible treatment for those who are seriously ill from Covid-19), study of the efficacy and pattern of immune response to vaccines, and in the estimation of seroprevalence(^4,8,9). While further evaluation is needed, due to urgency and demand of pandemic response, numerous Sars-CoV-2 immunoassays were rapidly being developed and placed on the market with limited validation on clinical samples(^1,8). However, to properly use serology tests, it is important to understand their performance characteristics and limitations(¹⁰).

That way, validation of serological test is required for accurate diagnosis of Sars-CoV-2 infection(⁸). However, there is much concern about lateral flow immunochromatographic assays, which has widespread due to their easy and fast performance but many of these tests lack adequate sensitivity and specificity evaluation(⁴).

Different types of assays can be used to determine different aspects of immune response and functionality of antibodies. The tests can be broadly classified to detect either binding or neutralizing antibodies. The humoral immune response produces antibodies against different viral antigens including the nucleocapsid (N) protein and spike (S) protein; S protein is present on the viral surface and is essential for virus entry and is present on the viral surface, N protein is the most abundantly expressed immunodominant protein that interacts with ribonucleic acid (RNA)(¹¹).

Although current utility of serological testing is mostly for epidemiological studies, proper evaluation of the clinical sensitivity and specificity of these tests within the tested population is very important. The test specificity directly affects its positive predictive values (PPV) and thus the reliability of a positive result. Although high specificity is reported for many commercial Sars-CoV-2 serologic assays, not all of them consistently meet this specificity threshold(¹²). In this context, identification of individuals showing a serological response to Sars-CoV-2 provides important complementary information by giving an evaluation of the fraction of individuals who have previously been infected(¹³). In this study was aimed at assessing the performance of rapid test used in an epidemiological survey for Sars-CoV-2 compared to other serological assays, taking into account the specificity and sensitivity of the methods.

METHODS

Detection of antibodies anti-Sars-CoV-2

All serological tests were performed according to the manufacturers’ instructions:

• lateral flow immunochromatographic assay [Wondfo Sars-CoV-2 Antibody Test (Guangzhou Wondfo Biotech Co., Ltd., China)]. The Wondfo rapid test (WRT) detects specific antibodies [immunoglobulin class G (IgG) and class M (IgM)] to the Sars-CoV-2 binding domain of the spike protein (S);

• electrochemiluminescence immunoassay [Elecsys anti- Sars-CoV-2 (Roche Diagnostics, USA)]. Roche ECLIA intended for the detection of total antibodies driven to Sars-CoV-2 nucleocapsid (N) antigen;

• chemiluminescent microparticle immunoassay [Sars-CoV-IgG (Abbott Diagnostics, Ireland)]. Abbott assay chemiluminescent immunoassay (CMIA)-IgG is intended for the detection of IgG antibodies driven to Sars-CoV-2 nucleocapsid (N) antigen.

In addition to these assays, a CMIA [Sars-CoV-IgM (Abbott Diagnostics, Ireland)], test for specific IgM antibodies to the Sars-CoV-2 (CMIA-IgM) binding domain of the spike protein (S) was used only in the control group samples.

All samples were evaluated in the three Sars-CoV-2 serological tests.

Serum sample

The flowchart (Figure) shows the samples evaluated in this study.

FIGURE
Serum samples evaluated in this study Sars-CoV-2: severe acute respiratory syndrome coronavirus 2; RNA: ribonucleic acid.

Epidemiological survey

Eighty six respiratory and serum samples were collected from July 5^th to July 26^th, 2020, as part of the epidemiological survey samples carried out at the Instituto Adolfo Lutz (IAL) to assess the exposure of its employees to Sars-CoV-2. All respiratory samples were evaluated for Sars-CoV-2 RT-PCR, 39 (45.3%) with detectable viral RNA and 47 (54.7%) with undetectable Sars-CoV-2 RNA (12 with Covid-19-compatible symptoms, and 35 without symptoms), and all serum samples by the three serological tests WRT, Roche ECLIA and Abbott CMIA-IgG.

The eighty six participants were grouped according to the time interval between the onset of symptoms and the blood collection (stage of the disease) for analysis in serological tests - initial phase (seven to 13 days after the symptoms of the disease), intermediate (14 to 20 days after the symptoms of the disease), and late phase (≥ 21 days after the symptoms of the disease)(⁸).

In this study, the association of signs and symptoms to define the diagnosis of Covid-19 was according to the criteria established by the Ministry of Health of Brazil(¹⁴).

Control group

Eighty four control serum samples were included to assess the performance of the assays used in this study, which were distributed in two groups:

• positive control group - 38 (45.2%) serum samples were obtained from 22 hospitalized patients with Covid-19 to assess the sensitivity of assays. All enrolled cases were confirmed to be infected with Sars-CoV-2 by means of RT-PCR test in samples from the respiratory tract. Serum samples were obtained in different stages of Covid-19 - 15 (39.5%) samples in the initial phase, 12 (31.6%) in the intermediate and 11 (28.9%) in the late phase. The classification of the stage of the disease was the same used in the epidemiological survey samples(⁸). The interval between the respiratory sample collection (RT-PCR) and the serum sample collection varied from -1 to +45 days (mean = 13.6 ± 11.6 days; median = 10 days);

• negative control group - 46 (54.8%) remnant serum samples from diagnostic routine of infectious diseases stored at the IAL from December 2017 to March 2018 were used to assess the specificity of assays and were characterized as follows: forty (87%) sera with positive results for one or more markers in serological tests of human immunodeficiency virus (HIV), syphilis, hepatitis B virus (HBV), hepatitis C virus (HCV), Mycoplasma pneumoniae (MP), dengue virus (DENV), Chikungunya virus (CHIKV), yellow fever virus (YFV), and six (13%) samples with negative results for serological markers in blood banks. All negative samples used were collected prior to 2020, before the Sars-CoV-2 is known to have circulated in Brazil.

STATISTICAL ANALYSIS

The diagnostic sensitivity of antibody test was defined as the percentage of positive results for antibodies against Sars-CoV-2 in the serum samples from confirmed patients with Covid-19 (with detectable Sars-CoV-2 RNA by RT-PCR), whereas the diagnostic specificity was defined as the percentage of negative results for antibodies against Sars-CoV-2 in the serum samples from individuals without Covid-19 (without detectable Sars-CoV-2 RNA by RT-PCR or negative control group samples). For the calculation of diagnostic sensitivity, diagnostic specificity, positive predictive value, negative predictive value, accuracy, and seropositivity the Microsoft Office Excel Program was used.

The measurement agreements between different antibody tests were interpreted by means of the Kappa (k) index, as proposed by Altman (1999)(¹⁵) and adapted from Landis and Koch (1977). k value < 0.2 represents poor agreement; 0.21-0.4, fair; 0.41-0.6, moderate; 0.61-0.8, good; 0.81-1, very good agreement. Fisher Exact Test, two tailed, was used for categorical variables.

This study was approved by the Human Research Ethics Committee of Instituto Adolfo Lutz (CAAE: 31924420.8.0000.0059 and CAAE: 43250620.4.1001.0059).

RESULTS

Control group samples

Based on the results of 84 control serum samples, the estimated values of sensitivity and specificity of assays are described in the Table 1. The positive control samples were classified according to the stage of the disease.

TABLE 1
Estimated values of sensitivity and specificity of assays in control group samples

Serum samples	n	WRT (IgM/IgG)			ECLIA (Total Ig)			CMIA (IgG)			CMIA (IgM)
		P	N	Result	P	N	Result	P	N	Result	P	N	Result
Positive control group	38	Sensitivity value (95% CI)
Initial phase	15	15	0	100% (79.6%-100%)	12	3	80% (54.8%-93%)	13	2	86.7% (62.1%-96.3%)	15	0	100% (79.6%-100%)
Intermediate phase	12	12	0	100% (75.8%-100%)	12	0	100% (75.8%-100%)	12	0	100% (75.8%-100%)	12	0	100% (75.8%-100%)
Late phase	11	11	0	100% (74.1%-100%)	11	0	100% (74.1%-100%)	11	0	100% (74.1%-100%)	11	0	100% (74.1%-100%)
Total		38	0	100% (90.8%-100%)	35	3	92.1% (79.2%-97.3%)	36	2	94.7% (82.7%-98.5%)	38	0	100% (90.8%-100%)
Negative control group	46	Specificity value (95% CI)
Antibodies to other pathologies*	40	0	40	100% (92.3%-100%)	0	40	97.8% (88.7%-99.6%)	0	40	100% (92.3%-100%)	0	40	100% (92.3%-100%)
Negative**	6	0	6		1	5		0	6		0	6
Total		0	46		1	45		0	46		0	46

• WRT: Wondfo rapid test; ECLIA: electrochemiluminescence immunoassay (Roche); total Ig: total immunoglobulin; CMIA: chemiluminescent microparticle immunoassay (Abbott); IgM: immunoglobulin class M; IgG: immunoglobulin class G; n: total number of samples; P: positive; N: negative; CI: confidence interval;
• ^*
  samples collected before 2019 and reactive to other infectious diseases including: HIV (human immunodeficiency virus), syphilis, HBV (hepatitis B virus), HCV (hepatitis C virus), DENV (dengue virus), YF (yellow fever), MP (Mycoplasma pneumoniae), CHIKV (chikungunya virus);
• ^**
  for serological markers of blood banks.

Diagnostic sensitivity was assessed using 38 positive serum samples for Sars-CoV-2 infection, confirmed by RT-PCR, and grouped according to the time interval between the onset symptoms and the blood collection (stage of disease). Among 15 patients in the initial phase of the disease, 100% (15/15) were positive by the WRT and CMIA-IgM, 80% (12/15) by ECLIA, and 86.7% (13/15) by CMIA-IgG. The samples grouped in the intermediate and late phases were positive in all four assays.

In the evaluation of the control group samples, WRT and CMIA-IgM presented the maximum sensitivity and specificity values (100%), while ECLIA showed the sensitivity and specificity values of 92.1% and 97.8%, respectively. The global sensitivity value of CMIA-IgG was 94.7%, with two samples of positive group control yielding negative results. On the other hand, these two samples showed positive results in CMIA-IgM and negative results in ECLIA, even though the later one is able to detect both G and M specific immunoglobulins. For the negative control group samples, which have been collected and stored before the appearance of Sars-CoV-2, we observed one negative sample for serological markers of blood banks with positive result by ECLIA (4.39 index: reference value < 1.00).

In the assessment of the 84 control group sera, the degree of agreement between serological tests is described in the Table 2.

TABLE 2
Agreement between four serological tests in the evaluation of control group samples

Serologic test	Percent agreement	k index	Standard error	95% CI	Observed agreement	Expected agreement	Degree of agreement
WRT/CMIA IgM	100%	1.00	0.000	(1.000-1.000)	1.00	0.50	Very good
WRT/CMIA IgG	97.6%	0.952	0.034	(0.886-1.018)	0.98	0.51	Very good
WRT/ECLIA	95.2%	0.903	0.047	(0.811-0.996)	0.95	0.51	Very good
ECLIA/CMIA IgM	95.2%	0.903	0.047	(0.811-0.996)	0.95	0.51	Very good
ECLIA/CMIA IgG	97.6%	0.951	0.034	(0.885-1.018)	0.98	0.51	Very good

• WRT: Wondfo rapid test; CMIA: chemiluminescent microparticle immunoassay (Abbott); ECLIA: electrochemiluminescence immunoassay (Roche); IgM: immunoglobulin class M; IgG: immunoglobulin class G; k index: Kappa index; CI: confidence interval.

The agreement between the two automated immunoassays and one rapid test for Sars-CoV-2 specific antibodies detection varied from 95.2% (k = 0.903) to 100% (k = 1.00). Although percent agreement between ECLIA and CMIA-IgG has been very good 97.6% (k = 0.951), both showed two false-negative results in samples of positive control group, classified in the initial stage of the disease.

Epidemiological survey samples

The samples were separated into four distinct groups according to the results of the RT-PCR and reports of symptoms: (i) detectable Sars-CoV-2 RNA with symptoms; (ii) detectable Sars-CoV-2 RNA without symptoms; (iii) undetectable Sars-CoV-2 RNA with symptoms; and (iv) undetectable Sars-CoV-2 RNA without symptoms. These results and those obtained from serological tests are detailed in the Table 3.

TABLE 3
Performance of the three serological tests for anti-Sars-CoV-2 in serum samples from the epidemiological survey according to RT-PCR results

Assay/Result				Number of samples n = 86	Stage of disease (n)
RT- PCR	Wondfo (RT)	Roche (ECLIA)	Abbott (CMIA)
Presence of Sars-CoV-2 RNA with symptoms n = 30 (34.9%)	P	P	P	17 (56.7%)	Intermediate (4), and late (13)
	N	N	N	8 (26.7%)	Initial < 7 days (2), and late (6)
	N	P	P	3 (10%)	Late
	P	N	N	1 (3.3%)	Initial (< 7 days)
	N	P	N	1 (3.3%)	Late
Presence of Sars-CoV-2 RNA without symptoms n = 9 (10.4%)	P	P	P	3 (33.3%)	_________
	N	N	N	3 (33.3%)	_________
	N	P	N	1 (11.1%)	_________
	P	P	N	1 (11.1%)	_________
	P	N	P	1 (11.1%)	_________
Absence of Sars-CoV-2 RNA with symptoms n = 12 (14%)	P	P	P	4 (33.3%)	Intermediate (1), and late (3)
	N	N	N	3 (25%)	Late
	N	P	P	3 (25%)	Late
	P	N	N	1(8.3%)	Late
	P	P	N	1 (8.3%)	Late
Absence of Sars-CoV-2 RNA without symptoms n = 35 (40.7%)	P	P	P	4 (11.4%)	_________
	N	N	N	30 (85.7%)	_________
	N	P	N	1 (2.9%)	_________

• Sars-CoV-2: severe acute respiratory syndrome coronavirus 2; RT-PCR: reverse-transcriptase polymerase chain reaction; RT: rapid teste; ECLIA: electrochemiluminescence immunoassay; CMIA: chemiluminescent microparticle immunoassay; RNA: ribonucleic acid; N: negative; P: positive; n: sample size.

As shown in Table 3, 48.8% (42/86) of participants that reported symptoms of Covid-19 were classified according to the stage of the disease at the time of blood collection for serological tests - 7.1% (3/42) before the initial stage of disease (< 7 days), 11.9% (5/42) in intermediate phase, and 81% (34/42) in late stage. The other 51.2% (44/86) did not report any symptoms of Covid-19. No participant reported having moderate or severe symptoms.

The occurrence of an antibody positive result (any assay) among those with Covid-19 symptoms but with negative RNA test was significantly higher than among those RNA negative without symptoms (9/12 vs 5/35, p = 0.0002). Even if we consider as positive only those with reactivity in all three tests, the difference is still significant (p = 0.02), Table 3.

The evaluation of the 86 samples showed eight cases with undetectable Sars-CoV-2 RNA and positive result in the three serological tests.

Although 16.3% (14/86) of serological results of survey showed some discordant result, the agreement between serological tests was adequate according to interpretation of Kappa index; the data are summarized in Table 4.

TABLE 4
The degree of agreement between serological tests in epidemiological survey samples

Serologic test	Number of samples	Percent agreement	k index	Standard error	95% CI	Observed agreement	Expected agreement	Degree of agreement
WRT/CMIA-IgG	86	88.4% (76/86)	0.757	0.072	(0.615-0.899)	0.88	0.52	Good
WRT/ECLIA		86% (74/86)	0.715	0.076	(0.565-0.864)	0.86	0.51	Good
ECLIA/CMIA-IgG		93% (80/86)	0.858	0.056	(0.748-0,968)	0.93	0.51	Very good

• WRT: Wondfo rapid test; ECLIA: electrochemiluminescence immunoassay (Roche); CMIA: chemiluminescent microparticle immunoassay (Abbott); IgG: immunoglobulin class G; k index: Kappa index; CI: confidence interval.

The highest percent agreement (93%) was observed between the ECLIA and CMIA-IgG, both chemiluminescent tests, with Kappa index equal to 0.858 (very good). Regarding the WRT and CMIA-IgG the percent agreement was 88.4% (k = 0.757), and 86% (k = 0.715) WRT and ECLIA (total antibody).

The estimated sensitivity and specificity values of the serological tests in the serum samples from the survey participants were calculated in relation to the Sars-CoV-2 RT-PCR results (presence of RNA/absence of RNA). Table 5 shows the estimated values of sensitivity, specificity, accuracy, positive predictive and negative predictive values.

TABLE 5
Estimated values of sensitivity, specificity, PPV, VPN, and accuracy obtained for different Sars-CoV-2 serological assays in the participants samples

Assay	Parameter					Seropositivity	Samples
	Total sensitivity	Total specificity	PPV	NPV	Accuracy
WRT (IgM/IgG)	59% (95% CI 43.4%-72.9%)	78.7% (95% CI 65.1%-88%)	69.7%	69.8%	69.8%
ECLIA (Total Ig)	66.7% (95% CI 51%-79.4%)	72.3% (95% CI 58.2%-83.1%)	66.7%	72.3%	69.8%	45.3%	Epidemiological inquiry (n = 86)
CMIA-IgG	61.5% (95% CI 47.1%-73%)	76.6% (95% CI 74%-95.5%)	68.6%	70.6%	69.8%

• Sars-CoV-2: severe acute respiratory syndrome coronavirus 2; WRT: Wondfo rapid test; ECLIA: electrochemiluminescence immunoassay (Roche); CMIA: chemiluminescent microparticle immunoassay (Abbott); IgG: immunoglobulin class G; IgM: immunoglobulin class M; CI: confidence interval; PPV: positive predictive value; NPV: negative predictive value.

In the evaluation of the 86 serum samples of epidemiological survey in the WRT, the estimated seropositivity was 45.3% [100* (23 true positive samples + 16 false-negative samples)/86]. The WRT PPV demonstrated that 69.7% chance that the individual to be sick, against 30.3% (100 - 69.7) not having disease, despite the positive result. The WRT showed 18.6% (16/86) of false-negative results compared to 11.6% (10/86) of false-positive, confirmed by the estimated sensitivity (59%) and specificity (78.7%) values, respectively.

DISCUSSION

In this study we evaluated three serological assays with different designs to assess their performance in serum samples from documented Sars-CoV-2 RNA positive cases, hospitalized patients and participants of an epidemiological survey that included asymptomatic individuals and cases with symptoms compatible to Covid-19. RNA testing was performed at least once by all participants. As also samples collected before the pandemic, with or without other prevalent infections, were tested to estimate the specificity of antibodies to Sars-CoV-2. Although Sars-CoV-2 specific antibodies detection cannot be assumed as an immunity correlate to the virus, it is a useful tool for public health tasks especially in epidemiological surveys.

We compared the results of Sars-CoV-2 IgG/IgM WRT, performed on serum samples, and two high-throughput automated serological tests, all according to respective manufacturer’s instructions. The use of point of care (POC) serological rapid tests facilitates access to diagnosing and allows for their use outside traditional health system settings. However, use of an antibody-detection POC test with serum, instead of whole blood, might improve its sensitivity, in addition to allowing direct comparisons with other serological assays that use serum samples. The results in this work refer to laboratory test of serum and extrapolation to field POC testing must consider this.

Sensitivity and specificity estimates shown may not be indicative of the real world performance of the tests(¹⁰). The predictive value of a test may be different when it is applied in a general population rather than in the study sample in which it was first developed(¹⁶). There was an increase in the sensitivity and specificity values of WRT when the samples of the control groups were assessed (Table 1). However, some differences may exist due to the different clinical conditions, period of onset of symptoms (“time of serological turning point”), epidemiological data, among others(¹⁷). The positive control group consisted of samples from hospitalized patients who had moderate to severe symptoms of Covid-19, while survey participants reported only mild symptoms of the disease. The WRT sensitivity was increased with clinical severity and days of symptoms, corroborating data in the literature(¹⁸). The negative control group was composed of serum samples collected long before the first case of Sars-CoV-2 reported in Brazil, unlike the survey participants samples with negative RNA Sars-CoV-2 results by RT-PCR, who may have acquired the infection previously or even other infections that may cause cross-reactive antibody binding reactions. The agreement between ECLIA and CMIA-IgG was very good, but both showed disagreement in three samples from the control group - one from the negative control group and two from the positive control group. The two samples of positive control group, when evaluated in CMIA-IgM and WRT (IgM/IgG) were positive, however ECLIA, for total antibodies, did not detect the presence of the immunoglobulins in these samples in the initial stage of the disease. In patients with Covid-19, the production of IgM and IgG can be simultaneous and reach in some patients a plateau level after six days (initial stage of the disease)(¹⁹). And about the negative control group sample, there was the presence of cross-reactive antibodies in the ECLIA.

In eight cases of the epidemiological survey with positive results in the three serological tests but without detected Sars-CoV-2 RNA, four of which did not report symptoms, the infection may have occurred previously and was not documented by the RT-PCR. In other four symptomatic cases it is possible that the diagnostic sensitivity of RT-PCR has been influenced by the timing of specimen collection and period of the disease development(^2,5). Even though we were not able to rule out the hypothesis that this occurred due to the collection of the PCR sample after the oro/nasopharyngeal viremia period, for the purpose of the present study in the absence of Sars-CoV-2 RNA the serological reagent results were considered false-positive. The interpretation of our rates has to consider the fact that in most cases only one RT-PCR was obtained.

As observed in the study by Costa et al. (2020)(²⁰), the performance of the serological test may be superior to that of RT-PCR. Discordant results must be cautiously interpreted. In the case of RT-PCR negative result with clinical features suspicion for Covid-19, the sampling timing and the stage of the disease development play an important role in assay results(²). Although the nucleic acid test or genetic sequencing serves as the gold standard method for confirmation of infection, yet several recent studies have reported Sars-CoV-2 false-negative results by RT- PCR(²¹). One study indicates that 2/10 negative cases by the RT-PCR were confirmed as positive for Covid-19, resulting in a false-negative rate of approximately 20%(²²).

In the same way, the negative results of the serological tests in samples with presence of Sars-CoV-2 RNA by RT-PCR were considered false-negative. These samples included six participants in the late stage of the disease (29-50 days of onset symptoms), with sufficient time for seroconversion. False-negative results can make it difficult to prevent and control the Covid-19 pandemic, in the same way that false-positive results can lead to unnecessary stress for patient, contacts and the health system. To improve accuracy of diagnosis, it is necessary a rational approach based on the usefulness of each type of test, the how to collect the sample from the patient, and the right time to perform the sample collection(²²).

The serological surveys can aid investigation of an ongoing outbreak and retrospective assessment of the attack rate or extent of an outbreak. In cases where molecular assays are negative and there is a strong epidemiological link to Covid-19 infection, paired serum samples (in the acute and convalescent phase) could support diagnosis with available validated serology tests(²³). Testing algorithms with more than one test may be necessary to rule out false-positives by initial tests, such that has been the rule for HIV and hepatitis. The potential of some assays to have high specificities depending on their target isotypes, the antigens used, and establishment of cut-off values(²⁴).

In this context, the clinical data of individuals based on signal and symptoms of Covid-19 may be useful for the evaluation of serological test results. Serology is suited for supporting the diagnosis of Covid-19 infection in RNA-negative symptomatic patients, and establishing seroprevalence in population studies(²⁵). Twelve survey participants with Sars-CoV-2 RNA not detected had reported the presence of symptoms of Covid-19, however these clinical indicators were not used to confirm the presence of disease for the purposes of this evaluation, because inevitably this can introduce some incorporation bias, it is expected to find more positive results in the tests used, and this tends to inflate the measured sensitivity of these tests. In this study, the result of RT-PCR was used as a parameter to define the infection by Sars-CoV-2. It’s worth mentioning that, serological assays do not typically replace direct detection methods as the primary tool for diagnosing an active Sars-CoV-2 infection, but they do have several important applications in monitoring and responding to the Covid-19 pandemic(¹¹).

Eleven survey samples with detectable Sars-CoV-2 RNA and negative results in the three serological tests (eight cases with symptoms and three without symptoms), mainly in the six cases of late stage of the disease, possibly are false-negative serological results or factors related to the immune system of individuals. It is also important to note that some individuals do not develop detectable IgG or IgM antibodies following infection (absence of humoral immune response to Sars-CoV-2) or the antibodies may not be long-lasting. Thus, the absence of detectable antibodies does not necessarily rule out that they could have previously been infected(^{5, 11, 17}). In this context, although it’s in the early stages of critical evaluation, can be included the relationship between disease severity and antibody titer(¹). All participants in this epidemiological survey who reported symptoms were mild, that reinforces this theory. In three (3/11) asymptomatic cases they may show lower antibody responses(²⁶). In the other two cases (2/11), serology for Sars-CoV-2 was performed less than seven days after the onset of symptoms. In the early days of an infection when the body’s immune response is still building, antibodies may not be detected (low antibody titer). Antibodies most commonly become detectable 1-3 weeks after symptom onset, at which time evidence suggests that infectiousness likely is greatly decreased and that some degree of immunity from future infection has developed(¹¹). In the initial stage of illness, molecular detection of the virus is the primary tool for early and accurate diagnosis of disease, as antibody production is usually delayed or absent in this phase(²⁷).

Of the 39 participants with Covid-19 (30 cases with symptoms and nine without symptoms) confirmed by the presence of RNA Sars-CoV-2, for the three serological tests used, the lowest estimated sensitivity rate was 59% for the WRT (43.4%-72.9%), and the highest was 66.7% for the ECLIA (51%-79.4%) followed by 61.5% (47.1%-75%) for the CMIA-IgG. Even though different tests show comparable results in detecting anti-Sars-CoV-2 antibodies, factors such as test-specific methodology, the type of antigen used, cut-off value determination, the temporal dynamics of virus-specific IgM and IgG immune responses, and the presence of cross-reactive antibodies remain important and affect the overall performance of serological tests(^{5, 17}).

A better sensitivity for serum samples with WRT was reported by Cota et al. (2020)(¹⁸) in a Brazilian study accessing the performance of 12 serological tests for Covid-19 diagnosis, that described an overall sensitivity for Wondfo at 71.7% (64.3%-78.2%). In another evaluation of the Wondfo, Conte et al. (2021)(¹⁹) have shown a sensibility of 47.6% (37.2%-58.1%) in capillary whole blood. However, it is important to note that the WRT may have a lower sensitivity when used with capillary blood. The antibody detection is more effective in plasma or serum samples than in whole blood(^16,28).

In the pairwise comparison of results from epidemiological survey participants testing the highest percent agreement (93%) [k = 0.858 (very good)] was observed between the ECLIA (total antibodies) and CMIA-IgG, both assays employing the Sars-CoV-2 nucleocapsid protein as antigen. Followed by WRT (antibody assay targeting the Sars-CoV-2 spike protein) and CMIA-IgG with 88.4% [k = 0.757 (good)], and WRT and ECLIA with 86% [k = 0.715 (good)]. Previous study showed that antibody to the nucleocapsid protein of Sars-CoV-2 is more sensitive than spike protein antibody for detecting early infection(²⁶). It is important to highlight that the results presented here cannot directly support a correlation between detectable Sars-CoV-2 antibodies and immunity to Covid-19.

Regarding the WRT used in this epidemiological survey, the sensitivity, specificity, and accuracy values reported in the Table 5 are lower than the manufacturers’ claimed [sensitivity 86.43% [95% confidence interval (CI) (82.51%-89.58%); specificity 99.57% (95% CI 97.63%-99.92%); accuracy 91.61% (95% CI 89.10%-93.58%)]. Specificities of at least 99.5% are required to achieve a high positive predictive value in low-prevalence populations(¹¹). About the results of serological tests, in most of the country, including areas that have been heavily impacted by Covid-19, the prevalence of Sars-CoV-2 antibody is expected to be low, ranging from < 5% to 25%, so that testing at this point might result in relatively more false-positive results and fewer false-negative results(¹¹). In this study, the seropositivity (45.3%) is higher due to the fact that many participants present a positive result on RT-PCR, and the WRT showed slightly more false-negative (16/86) than false-positive (10/86) results, with similar positive (69.7%) and negative (69.8%) predictive results.

It should be noted that there are some limitations of this study as the number of serum samples evaluated, which was based on the availability of chemiluminescence assays, at that time. It’s essential for refinement of the sensitivity values of assays a larger number of sera be tested, especially in samples from patients at different stages of the disease, because the sensitivity values of antibody tests are highly relevant for calculating the frequency of Sars-CoV-2 exposure in seroprevalence studies(²⁹). Samples from individuals at an initial stage of the disease are very important in assessing the performance of a serological test to check their sensitivity, in addition monitoring the seroconversion process. The Sars-CoV-2-specific antibodies can be detected in the serum of approximately 40% of Covid-19 patients as early as seven days after the onset of symptoms, with seroconversion rates rapidly increasing to > 90% by day 14(^8,30). As well as the evaluation of Sars-CoV-2 negative samples containing antibodies to other different diseases that have not been evaluated, such as cytomegalovirus (CMV), Epstein-Barr (EBV), other coronavirus, and autoantibodies to rule out cross-reactivity (false-positive result)(⁵). Another limitation of the study was a small number of samples in the initial and intermediate stages, which made it impossible to assess sensitivity and specificity in groups by stages of the disease, and be able to compare with other studies. Another caveat is that the results refer only to the batches of serological tests used, so it is important further studies for evaluating the clinical performance in other different batches to ensure accuracy.

CONCLUSION

In addition to the RT-PCR for Sars-CoV-2, the serological testing may be a useful diagnostic tool as long as a correct timing regarding antibody production is considered. Serologic tests can be used in epidemiological survey, but should be assessed before implementation to ensure analytical performance. On the other hand, a proportion of persons who are infected with Sars-CoV-2 may not develop measurable antibodies, therefore limiting the sensitivity of any antibody test.

ACKNOWLEDGEMENTS

The authors thank Andre R. Campos from by the technical laboratory support in the conduction of this work. They also thank the participants and members of the study project “Inquérito epidemiológico de funcionários e colaboradores do Instituto Adolfo Lutz frente à pandemia da Covid-19”. Special thanks to the survey participants for making this study possible.

REFERENCES

• ¹ Harley K, Gunsolus I. Comparison of the clinical performance of the Abbott alinity IgG, Abbott architect IgM, and Roche elecsys total Sars-CoV-2 antibody assays. J Clin Microbiol. 2020; 58: e01243-20.
• ² Tahamtan A, Ardebili A. Real-time RT-PCR in Covid-19 detection: issues affecting the results. Expert Rev Mol Diagn. 2020; 20(5): 453-54.
• ³ Zhao J, Yuan Q, Wang H, et al. Antibody responses to Sars-CoV-2 in patients with novel coronavirus disease 2019. Clin Infect Dis. 2020; 71(16): 2027-34.
• ⁴ Serrano MM, Rodríguez DN, Palop NT, et al. Comparison of commercial lateral flow immunoassays and ELISA for Sars-CoV-2 antibody detection. J Clin Virol. 2020. 129: 104529.
• ⁵ Chen S-Y, Lee Y-L, Lin Y-C, et al. Multicenter evaluation of two chemiluminescence and three lateral flow immunoassays for the diagnosis of Covid-19 and assessment of antibody dynamic responses to Sars-CoV-2 in Taiwan. Emerg Microbes Infect. 2020; 9(1): 2157-68.
• ⁶ Gutiérrez-Cobos A, Frutos SG, García DD, et al. Evaluation of diagnostic accuracy of 10 serological assays for detection of Sars-CoV-2 antibodies. Eur J Clin Microbiol Infect Dis. 2020; 1-7.
• ⁷ Wu J-L, Tseng W-P, Lin C-H, et al. Four point-of-care lateral flow immunoassays for diagnosis of Covid-19 and for assessing dynamics of antibody responses to Sars-CoV-2. J Infect. 2020; 81(3): 435-42.
• ⁸ Lassaunière R, Frische A, Harboe ZB, et al. Evaluation of nine commercial Sars-CoV-2 immunoassays. MedRxiv (Preprint).
• ⁹ Sasisekharan V, Pentakota N, Jayaraman A, Tharakaraman K, Wogan GN, Narayanasami U. Orthogonal immunoassays for IgG antibodies to Sars-CoV-2 antigens reveal that immune response lasts beyond 4 mo post illness onset. PNAS. 2021; 118(5): e2021615118.
• ¹⁰ Food and Drug Administration. Independent evaluations of Covid-19 serological tests. USA 2020. Available at: https://open.fda.gov/apis/device/covid19serology/
  » https://open.fda.gov/apis/device/covid19serology/
• ¹¹ Centers for Disease Control and Prevention. Interim guidelines for Covid-19 antibody testing. Atlanta 2020. Available at: https://www.cdc.gov/coronavirus/2019-ncov/lab/resources/antibody-tests-guidelines.html
  » https://www.cdc.gov/coronavirus/2019-ncov/lab/resources/antibody-tests-guidelines.html
• ¹² Xu G, Emanuel AJ, Nadig S, et al. Evaluation of orthogonal testing algorithm for detection of Sars-CoV-2 IgG antibodies. Clin Chem. 2020; 66(12): 1531-37.
• ¹³ Vauloup-Fellous C, Maylin S, Périllaud-Dubois C, et al. Performance of 30 commercial Sars-CoV-2 serology assays in testing symptomatic Covid-19 patients. Eur J Clin Microbiol Infect Dis. 2021; 1-7.
• ¹⁴ Brasil. Ministério da Saúde. Secretaria de Vigilância em Saúde. Guia de vigilância epidemiológica. Emergência de saúde pública de importância nacional pela doença pelo coronavírus 2019. Brasília; 2020. Available at: https://coronavirus.saude.gov.br/guia-de-vigilancia-epidemiologica-covid-19
  » https://coronavirus.saude.gov.br/guia-de-vigilancia-epidemiologica-covid-19
• ¹⁵ Altman DG. Practical statistics for medical research. Inter-rather agreement. 8 ed. London: Chapman & Hall; 1999. p. 403-9.
• ¹⁶ Usher-Smith JA, Sharp SJ, Griffin SJ. The spectrum effect in tests for risk prediction, screening, and diagnosis. BMJ. 2016; 353: i3139.
• ¹⁷ Dias VMCH, Carneiro M, Michelin L, et al. Testes sorológicos para Covid-19: interpretação e aplicações práticas. J Infect Control. 2020; 9(2).
• ¹⁸ Cota G, Freire ML, Souza CS, et al. Diagnostic performance of commercially available Covid-19 serology tests in Brazil. Int J Infect Dis. 2020; 101: 382-90.
• ¹⁹ Conte DC, Carvalho JMA, Luna LKS, Faíco-Filho KS, Perosa AH, Nancy Bellei N. Comparative analysis of three point-of-care lateral flow immunoassays for detection of anti-Sars-CoV-2 antibodies: data from 100 healthcare workers in Brazil. MedRxiv preprint. April 2021.
• ²⁰ Costa SF, Buss L, Espinoza EPS, et al. Performance of a qualitative rapid chromatographic immunoassay to diagnose Covid-19 in patients in a middle-income country. J Clin Virol. 2020; 131: 104592.
• ²¹ Li D, Wang D, Dong J, et al. False-negative results of real-time reverse-transcriptase polymerase chain reaction for severe acute respiratory syndrome coronavirus 2: role of deep-learning-based CT diagnosis and insights from two cases. Korean J Radiol. 2020; 21(4): 505-8.
• ²² Verotti MP, Ramos MC, Henriques CMP, Elias FT, Camargo EB. Testes diagnósticos para Covid-19 registrados na Agência Nacional de Vigilância Sanitária: sensibilidade e especificidade reportadas pelos fabricantes. Com Ciências Saúde. 2020; 31(1): 217-29.
• ²³ World Health Organization. Laboratory testing for coronavirus disease (Covid-19) in suspected human cases: interim guidance. Geneva; 2020. Available at: https://apps.who.int/iris/handle/10665/331329
  » https://apps.who.int/iris/handle/10665/331329
• ²⁴ Tehrani ZR, Saadat S, Saleh E, et al. Performance of nucleocapsid and spike based Sars-CoV-2 serologic assays. PLoS One; 15(11): e0237828.
• ²⁵ Turbett SE, Anahtar M, Dighe AS, et al. Evaluation of three commercial Sars-CoV-2 serologic assays and their performance in two-test algorithms. J Clin Microbiol. 2021; 59(1): e01892-20.
• ²⁶ Burbelo PD, Riedo FX, Morishima C, et al. Sensitivity in detection of antibodies to nucleocapsid and spike proteins of severe acute respiratory syndrome coronavirus 2 in patients with coronavirus disease 2019. J Infect Dis. 2020; 222: 206-13.
• ²⁷ Charlton CL, Kanji JN, Johal K, et al. Evaluation of six commercial mid- to high-volume antibody and six point-of-care lateral flow assays for detection of Sars-CoV-2 antibodies. J Clin Microbiol. 2020; 58(10): e01361-20.
• ²⁸ Santos VA, Rafael MM, Sabino EC, Duarte AJS. Comments. Sensitivity of the Wondfo one step Covid-19 test using serum samples. Clinics. 2020; 75: e2013.
• ²⁹ Schnurra C, Reiners N, Biemann R, Kaiser T, Trawinski H, Jassoy C. Comparison of the diagnostic sensitivity of Sars-CoV-2 nucleoprotein and glycoprotein-based antibody tests. J Clin Virol. 2020; 129: 104544.
• ³⁰ Krüttgen A, Cornelissen CG, Dreher M, Hornef M, Imöhl M, Kleines M. Comparison of four new commercial serologic assays for determination of Sars-CoV-2 IgG. J Clin Virol. 2020; 128: 104394.

Publication Dates

• Publication in this collection
  29 Nov 2021
• Date of issue
  2021

History

• Received
  12 May 2021
• Reviewed
  16 May 2021
• Accepted
  20 May 2021
• Published
  20 July 2021