PID-5-SRF online administration: psychometric indicators and measurement invariance between different formats of data collection

Abstract

Objective:  The Personality Inventory for DSM-5 (PID-5) is a tool used to assess maladaptive personality traits according to the Diagnostic and Statistical Manual of Mental Disorders, 5th edition (DSM-5) alternative model. The objective is to seek evidence of the validity and reliability of the PID-5 - Self Reported Form (PID-5-SRF) administered online and assess its measurement invariance compared to the paper-and-pencil administration.

Method:  A sample of 274 individuals from the general population (73.4% of women; 34.76 years old ± 11.6) completed the instrument online after the study was disseminated on social media and among the authors’ contacts.

Results:  Internal consistency (facets α ≥ 0.70; domains α ≥ 0.89) and test-retest reliability (15 to 30 days: facets intraclass correlation coefficient [ICC] ≥ 0.63; domains ICC ≥ 0.82) were satisfactory, but a floor effect was found in almost all the items. A large number of facets (n = 9) showed better fit to a bifactorial structure, and the exploratory factor analysis (EFA) suggested that a six-factor model better fits the data. Measurement invariance between the online and paper-and-pencil administrations was not attested at a configural level.

Conclusion:  The results revealed satisfactory psychometric indicators when the instrument was applied online, confirming its feasibility in collecting data. However, the instrument's structure is not invariant, and caution must be adopted when comparing and interpreting data collected through different formats.

Keywords:
PID-5-SRF; online; administration; psychometric; indicators; measurements

Introduction

The internet is a tool increasingly used in scientific research.¹ Some researchers note the benefits of collecting data online, suggesting that this format will become even more disseminated and eventually replace the traditional paper-and-pencil format.² Collecting data online is less expensive, faster, and more accurate, and larger populations can be accessed while the confidentiality of the participants’ identities can be ensured.³

The psychometric properties of instruments used to collect data online must be tested, regardless of the results obtained in paper-and-pencil administrations.³ Many researchers argue that the measurement of an instrument does not vary when administered in different formats⁴; however, there is no consensus around this notion.⁵ Studies show that web-based surveys present some specificities. These specificities concern low bias associated with social desirability or,⁶ on the contrary, high sampling bias due to barriers to accessibility, especially in less developed countries with more restricted digital access or with older, or less educated populations.⁷,⁸ Such biases may change how an instrument is completed impacting its configurations and parameters.⁹,¹⁰

With the publication of the Diagnostic and Statistical Manual of Mental Disorders, 5th edition (DSM-5), a new self-report instrument, the Personality Inventory for DSM-5 - Self Reported Form (PID-5-SRF), was proposed to support a dimensional assessment of maladaptive traits. It was written in English and is composed of 220 items rated on a 4-point Likert scale.¹¹ It has been the subject of several studies since its publication, involving the analysis of its psychometric qualities and cross-cultural adaptation to other languages.¹²,¹³ Furthermore, some of these studies applied it online, such as Bo et al.,¹⁴ Suzuki et al.,¹⁵ and Zimmerman et al.,¹⁶ and reported appropriate psychometric indicators comparable to the paper-pencil administration.¹⁷–¹⁹ However, thus far, measurement invariance according to different delivery formats has not yet been verified.

We recently conducted the cultural adaptation of the PID-5-SRF to the Brazilian context, and the psychometric study of the paper-pencil version presented satisfactory psychometric indicators.²⁰ Hence, given the current context marked by the advent of online technologies, the objective is to assess the validity and reliability of the Brazilian version of the PID-5-SRF when applied online and investigate whether there is measurement invariance between the paper-pencil and web-based administrations.

Method

The local institution's institutional review board approved the study (process no. 4058/2018), and the participants provided their consent through a free and informed consent form accessed in the data collection platform.

Participants

The study was disseminated in social media, to the researchers’ contacts, and through institutional e-mails. Data were collected online between July 2019 and January 2020 via Google Forms. After accessing the link, the participants were asked to give their consent through a free and informed consent form to complete the instrument. Eligible individuals were 18 years old or older, both sexes, literate, and with good comprehension skills.

The initial sample comprised 327 individuals, 53 of whom were excluded due to missing data, as they did not submit their responses. Hence, the final sample comprised 274 participants. Fifteen days later, the participants received a link via e-mail for the retest, and 73 participants completed the instrument in this stage.

The sample of a previous study in which PID-5-SRF was applied in the paper-pencil format was used to test the measurement invariance.²⁰ Of the 2,000 eligible individuals, 832 did not return the questionnaires, 58 did not answer the form correctly, 380 missed data, and 730 were included in the final sample. The inclusion and exclusion criteria of both studies are identical.

Instruments

The data collection protocol comprised the following instruments: PID-5-SRF developed by Krueger et al.¹¹ and culturally adapted to Brazil by Barchi-Ferreira et al.¹³; the Response Inconsistency Scale developed by Keeley et al.,²¹ whose validity and clinical usefulness was verified by Sellbom et al.,²² to detect potentially invalidating response style; and sociodemographic and clinical questionnaire, 19-item form specifically developed for this study.

Data analysis

PID-5-SRF data were coded according to its technical guidelines. The analyses were performed using IBM SPSS and Mplus, with the significant level established at p ≤ 0.05. Descriptive statistics (mean, standard deviation [SD], frequency, and percentage) and group comparison tests (t student and chi-square [χ²]) were used to characterize and analyze the sample. Cronbach's alpha was used to verify internal consistency, which is adequate when above 0.70.²³ The intraclass correlation coefficient (ICC) was used for the test-retest reliability with a 95% confidence interval (95%CI).

The polychoric correlation matrix and unweighted least squares extraction (ULS) (a method that does not require normal distributions) were used to verify the facets’ unidimensionality.²⁴,²⁵ Parallel analysis,²⁶ Velicer's minimum average partial (MAP),²⁷ and the Hull method were used to assess the most appropriate number of factors.²⁸ The adequacy of the one-factor solution was verified through the following indexes: chi-square, Tucker-Lewis index (TLI), root mean square error of approximation (RMSEA), and root mean square residual (RMSR), adopting the following parameters: χ²/degrees of freedom (df) equal to or below 3,²⁹,³⁰ TLI values close to 1.00 or higher than 0.90, and RMSR close to or below 0.08,³¹ RMSEA close to or below 0.08.²⁹

The study of exploratory factor analysis (EFA) was conducted at the facet level considering a Pearson correlation matrix. The ULS method was used for extraction²⁴,²⁵ with Promax rotation.

The invalidating response style was assessed using the Response Inconsistency Scale, following the criteria proposed by Keeley et al.,²¹ adopting a cutoff score ≥ 17, whose sensitivity is 97%, specificity 95%, and accuracy 96%.²¹

The measurement invariance analysis that considered the two delivery formats was performed using the multi-group confirmatory factor analysis (MGCFA) (estimated via the maximum likelihood method [MLE]) at four levels (configural, metric, scalar, and residual). Because the order of the parameters follows a hierarchy, a more complex model is only assessed if the previous one presented invariance.²⁹,³⁰ Therefore, the configural invariance will be tested (and confirmed if an unrestricted baseline model, which verifies whether the same latent variables explain the same item, presents a good fit without specifying any measurement parameter). If confirmed, the other analyses will follow successively. Significant worsening in the model fit would indicate non-invariance between groups in all the comparisons. Since the literature indicates that chi-square difference tests detect minor discrepancies without practical or theoretical implications among samples above 200, a decrease in CFI by 0.01 and an increase in RMSEA by 0.015 were considered the best comparison indicators.³²

Results

Socio-demographic characteristics

The final sample (online administration) was composed of 274 individuals, most of whom were women (73.4%), aged 34.76 years old (SD = 11.6), with 12 or more years of schooling (78.5%). Approximately 37.60% of the participants lived with a partner, and 80.3% had a paid job. Regarding health conditions, 31.8% of the sample reported current health problems, predominantly hypertension (25.3%) and respiratory problems (11.5%), while 32.1% reported a psychiatric diagnosis. Of these, 55.7% reported depression and 42% anxiety. The paper-pencil sample²⁰ comprised 730 individuals from the general population: 67.8% were women, aged 33.84 on average (SD = 15.2 years), 69.5% reported 12 or more years of schooling, and 13.7% reported a psychiatric disorder. More detailed information regarding both samples is provided in Supplementary Table S1.

Reliability indicators

Analysis of items, facets, domains, and reliability

The analysis of the items’ scores (mean raw scores) indicates that the item with the highest score was P96R (reverse score): "I rarely worry about things" (mean = 2.53; SD = 0.71) while the item with the lowest score is P198: "I sometimes hit people to remind them of who is in charge" (mean = 0.06; SD = 0.27). Asymmetry and kurtosis indexes showed that none of the items had a normal distribution. A floor effect (more than 15% percentage of the responses were in the category "Very false or often false"³³) was found for almost all the items (n = 214). In turn, a ceiling effect was found in 42 items (more than 15% of the responses were rated as "Very true or often true"). Data are presented in detail in Supplementary Tables S2 and S3.

The scores concerning facets and domains are presented in Table 1. All domains and most facets are correlated with a score above 0.50. The scale's internal consistency (Cronbach's alpha) was 0.98. All the facets individually presented appropriate alpha values (> 0.70); the domains obtained alpha values above 0.89. The test-retest reliability was performed for each item individually, and most items obtained indicators above 0.51 (Supplementary Table S2). The facets and domains obtained strong/very strong indexes (> 0.50).

Table 1
Raw and weighted scores, distribution measures, correlations, and reliability indicators of the different facets and domains of the PID-5-SRS – Online administration (n = 274)

Domains	Facets	No. items	Distribution shape				Weighted scored		Item-total correlation	Facet-total correlation	α	T/R ICC (95%)
			Asym	SE	Kurt	SE	Mean	SD
NA	Emotional Lability	7	0.15	0.15	-0.78	0.29	1.43	0.74	0.49-0.65	0.57	0.83	0.86 (0.75-0.92)
NA	Anxiousness	9	-0.33	0.15	-0.71	0.29	1.75	0.77	0.44-0.77	0.66	0.90	0.83 (0.73-0.89)
NA	Separation Insecurity	7	1.00	0.15	0.51	0.29	0.74	0.67	0.53-0.74	0.41	0.85	0.86 (0.78-0.91)
NA	Submissiveness	4	0.49	0.15	-0.39	0.29	1.01	0.72	0.87-0.93	0.42	0.82	0.83 (0.71-0.90)
NA	Hostility	10	0.47	0.15	-0.16	0.29	1.04	0.65	0.34-0.73	0.69	0.88	0.71 (0.56-0.81)
NA	Perseveration	9	0.58	0.15	0.03	0.29	0.97	0.62	0.40-0.69	0.76	0.84	0.74 (0.60-0.83)
DET	Withdrawal	10	0.43	0.15	-0.65	0.29	1.09	0.76	0.55-0.81	0.59	0.92	0.81 (0.69-0.88)
DET	Intimacy Avoidance	6	1.34	0.15	1.52	0.29	0.63	0.64	0.36-0.74	0.46	0.80	0.63 (0.45-0.76)
DET	Anhedonia	8	0.38	0.15	-0.72	0.29	1.15	0.75	0.44-0.75	0.69	0.89	0.77 (0.64-0.85)
DET	Depressivity	14	0.81	0.15	-0.41	0.29	0.89	0.78	0.50-0.82	0.74	0.95	0.87 (0.80-0.92)
DET	Restrict Affect	7	0.63	0.15	-0.39	0.29	0.90	0.71	0.58-0.66	0.51	0.85	0.82 (0.71-0.88)
DET	Suspiciousness	7	0.21	0.15	-0.52	0.29	1.26	0.58	0.18-0.64	0.63	0.70	0.69 (0.53-0.80)
ANT	Manipulation	5	1.14	0.15	0.77	0.29	0.54	0.57	0.56-0.58	0.47	0.77	0.79 (0.67-0.87)
ANT	Deceitfulness	10	1.52	0.15	2.00	0.29	0.46	0.50	0.30-0.71	0.61	0.87	0.85 (0.76-0.91)
ANT	Grandiosity	6	0.87	0.15	0.60	0.29	0.72	0.55	0.23-0.55	0.37	0.70	0.79 (0.67-0.87)
ANT	Attention Seeking	8	0.83	0.15	-0.10	0.29	0.74	0.65	0.44-0.77	0.47	0.87	0.67 (0.51-0.78)
ANT	Callousness	14	1.68	0.15	2.61	0.29	0.34	0.39	0.15-0.75	0.53	0.82	0.78 (0.67-0.86)
DIS	Irresponsibility	7	1.18	0.15	1.01	0.29	0.51	0.51	0.41-0.52	0.63	0.74	0.75 (0.62-0.84)
DIS	Impulsivity	6	0.64	0.15	-0.25	0.29	0.91	0.71	0.60-0.78	0.58	0.89	0.79 (0.67-0.87)
DIS	Distractibility	9	0.46	0.15	-0.53	0.29	1.14	0.75	0.44-0.78	0.65	0.90	0.85 (0.76-0.91)
DIS	Risk Taking	14	0.62	0.15	0.24	0.29	0.92	0.54	0.39-0.70	0.16	0.87	0.82 (0.72-0.89)
DIS	Rigid Perfectionism	10	0.27	0.15	-1.01	0.29	1.23	0.76	0.58-0.79	0.39	0.90	0.73 (0.59-0.83)
PSY	Unusual Beliefs	8	1.01	0.15	0.48	0.29	0.67	0.61	0.39-0.59	0.54	0.80	0.79 (0.68-0.87)
PSY	Eccentricity	13	0.91	0.15	-0.25	0.29	0.74	0.78	0.73-0.84	0.75	0.96	0.88 (0.81-0.93)
PSY	Cognitive and Perceptual Dysregulation	12	1.19	0.15	1.67	0.29	0.57	0.51	0.35-0.67	0.78	0.85	0.82 (0.72-0.89)
NA		46	0.02	0.15	-0.68	0.29	1.18	0.51	0.19-0.69	0.80	0.94	0.86 (0.75-0.92)
DET		52	0.47	0.15	-0.46	0.29	0.99	0.56	0.15-0.78	0.61	0.96	0.82 (0.72-0.89)
ANT		43	1.06	0.15	1.06	0.29	0.62	0.43	0.17-0.65	0.81	0.92	0.84 (0.75-0.91)
DIS		46	0.59	0.15	0.62	0.29	0.97	0.39	0.01-0.61	0.77	0.89	0.84 (0.75-0.91)
PSY		33	0.75	0.15	1.01	0.28	1.58	0.69	0.31-0.83	0.77	0.95	0.89 (0.82-0.93)

• ANT = Antagonism; Asym = asymmetry; DET = Detachment; DIS = Disinhibition; ICC = intraclass correlation index; Kurt = kurtosis; NA = Negative Affect; PID-5-SRS = Personality Inventory for DSM-5 - Self Reported Form; PSY = Psychoticism; SD = standard deviation; SE = standard error; T/R = test-retest reliability; α = Cronbach's alpha.

Validity indicators based on the internal structure

Facets unidimensionality. Three different methods were used to estimate the number of factors associated with the facets. As presented in Table 2, the parallel analysis suggests that most facets present a multi-dimension structure. Even though the other methods predominantly suggested a one-dimension structure, the goodness of fit indexes (GFI) associated with this condition was satisfactory only for the Intimacy Avoidance, Restricted Affect, Irresponsibility, Impulsivity, Separation Insecurity, Submissiveness, Withdrawal, Anhedonia, and Distractibility facets (n = 9). The two-factor structure presented a better fit for the Emotional Lability, Anxiousness, Hostility, Perseveration, Suspiciousness, Grandiosity, Attention Seeking, Rigid Perfectionism, and Eccentricity facets (n = 9) (Supplementary Table S4). However, the two-factor model was not satisfactory for the Depressivity, Manipulativeness, Deceitfulness, Callousness, Risk Taking, Unusual Beliefs, and Perceptual and Cognitive Dysregulation facets (n = 7).

Table 2
Analysis of the facets’ unidimensionality according to different delivery formats – Online administration (n = 274) – One-factor model – measures

Domains	Unidimensionality Facets	No. of factors suggested factors			Fit measure for the one-factor model
		Parallel analysis	Velicer's MAP	Hull test	χ2 (df)	TLI	RMSEA	RMSR
NA	Emotional Lability	2	2	1	640 (14)	0.299	0.404	0.210
NA	Anxiousness	3	1	1	250 (27)	0.826	0.174	0.060
NA	Separation Insecurity	3	1	1	83 (14)	0.908	0.130	0.050
NA	Submissiveness	1	1	-	7.8 (2)	0.964	0.103	0.030
NA	Hostility	2	1	1	240 (35)	0.829	0.147	0.070
NA	Perseveration	2	1	1	170 (27)	0.827	0.139	0.080
DET	Withdrawal	2	1	1	160 (35)	0.928	0.113	0.040
DET	Intimacy Avoidance	1	1	1	28 (9)	0.963	0.088	0.040
DET	Anhedonia	2	1	1	96 (20)	0.927	0.117	0.050
DET	Depressivity	2	1	4	745 (77)	0.830	0.178	0.050
DET	Restrict Affect	1	1	1	46 (14)	0.949	0.091	0.040
DET	Suspiciousness	2	1	1	88 (14)	0.788	0.139	0.080
ANT	Manipulation	3	1	-	95 (5)	0.765	0.256	0.070
ANT	Deceitfulness	-	1	1	5824 (35)	0	0.777	0.070
ANT	Grandiosity	2	1	1	59 (9)	0.854	0.142	0.060
ANT	Attention Seeking	2	1	1	180 (20)	0.855	0.172	0.070
ANT	Callousness	-	1	1	710.2(77)	0.740	0.173	0.070
DIS	Irresponsibility	2	1	1	59 (14)	0.889	0.109	0.060
DIS	Impulsivity	1	1	1	24 (9)	0.979	0.078	0.020
DIS	Distractibility	2	1	1	160 (27)	0.900	0.132	0.060
DIS	Risk Taking	3	2	3	494 (77)	0.766	0.141	0.080
DIS	Rigid Perfectionism	3	1	1	260 (35)	0.842	0.153	0.060
PSY	Unusual Beliefs	4	1	1	160 (20)	0.790	0.160	0.070
PSY	Eccentricity	2	2	1	609.3 (65)	0.861	0.180	0.040
PSY	Cognitive and Perceptual Dysregulation	-	1	2	351.9 (54)	0.799	0.142	0.070

• ANT = Antagonism; DET = Detachment; df = degrees of freedom; DIS = Disinhibition; NA = Negative Affect; PSY = Psychoticism; RMSEA = root mean square error of approximation; RMSR = root mean square residual; TLI = Tucker-Lewis index; χ² = chi-square.

Exploratory factor analysis. The factorability of the matrix was verified via Kaiser-Meyer-Olkin (KMO) (0.923) and Bartlett's test of sphericity (p < 0.001). The techniques used to retain factors suggest the presence of four (Hull test and Velicers MAP) or six factors (parallel analysis). The GFI for each factor solution suggested and the five-factor model proposed by Krueger et al.¹¹ are presented in Table 3. The distribution of the items’ factor loadings in the five factors is presented in Table 4.

Table 3
PID-5 adjustment indexes associated with different factor models analyzed through EFA – Online administration (n = 274)

Indexes	Models
	4 factors	5 factors	6 factors
χ2 (df)	747.290 (206) p < 0.0001	537.800 (185) p < 0.0001	400.250 (165) p < 0.0001
TLI	0.804	0.858	0.893
RMSEA	0.098	0.083	0.072
RMSR	0.04	0.03	0.03

• df = degrees of freedom; EFA = exploratory factor analysis; PID-5-SRS = Personality Inventory for DSM-5 - Self Reported Form; RMSEA= root mean square error of approximation; RMSR = root mean square of residuals; TLI = Tucker-Lewis index; χ² = chi-square.

Table 4
Factor loadings of the facets in the different domains (n = 6) based on an exploratory factor analysis (EFA) of the PID-5-SRF – Online administration (n = 274)

Facets	Six-factor model
	Factor 1	Factor 2	Factor 3	Factor 4	Factor 5	Factor 6
Emotional Lability	0.69	-0.17	-0.23	0.09	0.20	0.16
Anxiousness	0.83	-0.10	-0.08	-0.17	0.04	0.28
Separation Anxiety	0.51	0.16	-0.15	-0.01	-0.05	0.11
Submissiveness	0.57	0.26	0.01	-0.14	-0.02	-0.21
Hostility	0.17	0.01	0.19	0.26	-0.13	0.76
Perseveration	0.48	0.05	0.27	-0.08	0.29	-0.05
Withdrawal	0.25	0.00	0.69	-0.24	-0.12	0.19
Intimacy Avoidance	0.02	-0.02	0.58	0.06	0.06	-0.04
Anhedonia	0.59	-0.08	0.56	-0.15	-0.20	0.12
Depressivity	0.74	-0.08	0.32	-0.04	-0.09	0.07
Restrict Affect	-0.26	0.08	0.82	-0.02	0.15	0.03
Suspiciousness	0.31	0.08	0.07	-0.08	0.09	0.46
Manipulation	0.03	0.78	0.05	0.09	-0.03	0.01
Deceitfulness	0.28	0.79	0.12	0.11	-0.19	-0.01
Grandiosity	-0.19	0.48	0.08	-0.03	0.25	0.12
Attention Seeking	0.44	0.46	-0.43	0.26	0.15	0.00
Callousness	-0.27	0.26	0.49	0.28	-0.03	0.33
Irresponsibility	0.41	0.10	0.24	0.42	-0.11	-0.02
Impulsivity	0.35	-0.13	-0.07	0.58	0.05	0.26
Distractibility	0.53	-0.10	0.27	0.27	0.07	-0.12
Risk Taking	-0.20	0.18	-0.02	0.53	0.04	0.04
Rigid Perfectionism	0.14	0.09	0.07	-0.48	0.39	0.22
Unusual Beliefs	0.01	-0.02	0.00	0.01	0.80	-0.03
Eccentricity	0.14	-0.01	0.35	0.19	0.48	-0.05
Cognitive and Perceptual Dysregulation	0.31	-0.02	0.16	0.16	0.62	-0.13

• EFA = exploratory factor analysis; PID-5-SRS = Personality Inventory for DSM-5 - Self Reported Form.
• Bold type indicates highest factor loading.

The analysis of all GFI indicates that the five- and six-factor models present better adequacy. Even though the five-factor model suggests a greater theoretical association with Krueger's original model, which is also composed of five factors, the distribution of the facets’ factor loadings in the domains suggests that the six-factor model is more appropriate. Hence, Factor 1 (Negative Affect) is composed of the model's original facets (Emotional Lability, Anxiousness, Separation Insecurity, Submissiveness, Perseveration, except Hostility) and the Distractibility facet (which originally belonged to the Disinhibition domain). Factor 2 corresponds to the Antagonism facet (Manipulativeness, Deceitfulness, Attention Seeking, and Grandiosity, except for Callousness). Factor 3 comprises the facets of the Detachment domain (Withdrawal, Intimacy Avoidance, Anhedonia, Depressivity, Restricted Affect, except for Suspicioness). Factor 4 corresponds to the Disinhibition domain, except for the Distractability facet, which, as previously described, presented a higher factor loading in Factor 1. Factor 5 corresponds to Psychoticism's original facets (Unusual Beliefs, Eccentricity, and Perceptual and Cognitive Dysregulation), and Factor 6 is composed of the Hostility, Suspiciousness, and Callousness facets.

Response inconsistency analysis. The response inconsistency analysis showed that in the sample in which PID-5-SRF was applied in online format, 4.7% of the subjects (n = 13) presented indicators at this level. In the sample whose application of the PID-5-SRF was via paper-and-pencil, the percentage of subjects with inconsistency indicators was 6.3% (n = 46). These indices are not statistically different (p = 0.35).

Measurement invariance analysis

A MGCFA was performed considering the different delivery formats. First, the test started at the configural level, and the results indicated that the instrument's structure was unstable (χ² = 4362.268, df = 530; RMSEA = 0.085; CFI = 0.736), so we did not advance to the remaining analyses.

Discussion

This study's objective was to analyze the psychometric properties of the PID-5-SRF Brazilian version applied online and verify the measurement invariance between the web-based and paper-and-pencil formats. The internal consistency and temporal stability were appropriate (≥ 0.69). It is similar to the original version in English (α ≥ 0.72),¹¹ which was also applied online, and above the Brazilian version applied in the paper-and-pencil format (α ≥ 0.51). Regardless of the administration format, the PID-5-SRF reliability indicators were adequate for all domains and most facets, even among cross-cultural studies.¹⁷,³⁴–³⁶

Almost all items (n = 217) had a floor effect (the answers were concentrated on the measure's lowest levels), while a much lower number of items (n = 37) presented a ceiling effect. This finding is similar to the study in which the instrument was applied in the paper-and-pencil format. These effects may negatively impact an instrument's sensitivity and specificity, which should be further analyzed. These findings may be linked to the fact that the sample studied, in both studies, was population-based. As these effects can negatively impact the sensitivity and specificity of an instrument, they must be analyzed in more detail, especially in clinical samples, in order to demonstrate whether the applicability of the instrument for this context may or may not be affected.

Testing the facets’ unidimensionality showed that many facets did not fit this model, which had already been observed in the Brazilian study in which the instrument was applied in the paper-and-pencil administration.²⁰ The best fit to the two-factor model of the Emotional Lability, Hostility, Perseveration, Anxiety, Attention Seeking, and Distrust facets was previously reported.²⁰,³⁶–³⁸

Apart from that, for the first time in this study, the Grandiosity, Rigid Perfectionism, and Eccentricity facets showed a better fit to the two-factor structure. The Rigid Perfectionism facet was composed of items representing the pursuit of perfection itself and another factor concerning rigidity and other people's perceptions of this behavior. The Eccentricity facet was composed of a factor that grouped items focused on eccentric behaviors and the perception of others (heteroperception) and another factor with items related to eccentric thoughts and perception itself (self-perception). On the other hand, the Grandiosity facet was composed of a factor related to the grandiose quality and importance compared to others and another factor linked to personal achievements and devaluation of others. Unlike the paper-and-pencil administration, the Depressivity, Manipulation, Risk Exposure, Unusual Beliefs, and Cognitive and Perceptual Dysregulation facets did not present an adequate fit in the online administration, not even to the two-dimensional model. The Deceitfulness and Callousness facets did not fit the one-dimension or two-dimension models also in the paper-and-pencil administration.²⁰

As for the PID-5-SRF factor structure, the previous literature indicates that the five-factor structure is the most commonly found,¹⁷,³⁶,³⁸,³⁹ illustrating the theoretical model that underpins the instrument.¹¹ However, in this study, the six-factor structure proved more adequate. This model somehow reflects the original five-factor structure.¹¹ The most differentiating point is the emergence of a new factor composed of the Hostility, Suspiciousness, and Callousness facets, which portrays a different dimension that brings together traits associated with social maladjustment. This factor can be seen as composed of the pathological variants of the Social Concordance domain of the Severity Indices of Personality Problems (SIPP-118),⁴⁰ composed of the Aggression Regulation, Frustration Tolerance, Cooperation, and Respect facets. A better fit to the six-factor structure also observed in the study by Zhang et al.⁴¹ These authors investigated the psychometric properties of PID-5-SRF in the paper-and-pencil administration in a sample of Chinese adolescents. However, the composition of each factor differs significantly from the one found in this study. It also presents little correspondence to the original model, which the authors attributed to the participants’ age in which personality is still in formation.

Finally, the PID-5-SRF invariance in the most initial level (configural) according to the format in which the instrument was administered was not verified, showing that only some items/facets are better explained by the same latent variables. As previously noted, the instrument administered online showed a better fit to the six-factor model, while the paper-and-pencil format fit the five-factor model better.²⁰ Invariance between the different formats in which psychological instruments are administered is controversial. For example, a previous study involving the Big Five Personality Test (BFQ-2) reported invariance,⁴ while another study using instruments to assess emotional functioning (Negative Mood Regulation Scale [NMRS]; Trait Meta-Mood Scale [TMMS]) and attachment (Inventory of Parent and Peer Attachment [IPPA]) did not.⁴² The presence of measurement invariance considering a given variable is necessary to compare scores between groups with different characteristics, so that differences in the latent construct of interest can be measured.⁴³,⁴⁴

Different variables may impact the answers provided to an instrument when the format in which it is administered differs. Among these variables, potential bias linked to social desirability stands out. However, there are also biases related to the use of technology, such as the respondents’ skill level and non-standardization of an instrument's presentation (e.g., different screens may be used when the instrument is applied online, such as a desktop, notebook, or smartphone with different resolutions). There is also sampling bias, considering that participants in online environments are subject to numerous physical and psychological variables and may become more distracted than when taking tests under supervised conditions.⁴⁵,⁴⁶ It is noteworthy that the rate of subjects whose responses to the PID-5-SRF were considered inconsistent did not differ significantly between the samples.

In this study, although statistically significant differences were observed in some variables of the samples recruited for the two application formats, in general, they are not very significant, maintaining the general profile of the samples homogeneous. However, a slightly higher percentage of people with psychopathology indicators may have an influence, even though a previous study showed measurement invariance among clinical and community samples¹⁷

This is the first study investigating whether the format in which the PID-5 is administered influences the data variance. In addition to the samples’ clinical and non-clinical conditions, previous studies have already analyzed the impact of culture³⁴,⁴⁷ and sex,⁴⁸ reporting invariance at various levels. However, the study by Sorrel et al.⁴⁷ is an exception. It analyzed a larger number of cultures and did not report invariance at the scalar level.

The conclusion is that the PID-5-SRF online administered presents good psychometric indicators, compatible with the paper-and-pencil administration, reinforcing previous results reported in the literature and its feasibility for assessing pathological personality traits. However, the instrument structure seems to differ, whether at the facets or domains level, depending on the type of application. This fact has no implications for the applicability of the instrument in any of the analyzed formats. Hence, those interested in using PID-5 at a clinical or research level should consider this aspect to avoid measurement bias. Based on these results, comparing and interpreting data collected through different formats is not recommended, given a lack of invariance, as it may influence diagnostic reasoning and clinical decisions.

Acknowledgements

This study received financial support from Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP; process no. 2019/27022-0).

Data availability statement

The data that support this study are available from the authors upon request.

SUPPLEMENTARY MATERIAL

SUPPLEMENTARY MATERIAL

SUPPLEMENTARY MATERIAL

References

• ¹ Al-Salom P, Miller CJ. The problem with online data collection: predicting invalid responding in undergraduate samples. Curr Psychol. 2019; 38:1258-64.
• ² Al-Dajani N, Gralnick TM, Bagby RM. A psychometric review of the Personality Inventory for DSM–5 (PID–5): current status and future directions. J Pers Assess. 2016; 98:62-81.
• ³ Aluja A, Rossier J, Zuckerman M. Equivalence of paper and pencil vs Internet forms of the ZKPQ-50-CC in Spanish and French samples. Pers Individ Dif. 2007;43:2022-32.
• ⁴ Vecchione M, Alessandro G, Barbanelli C. Paper-and-pencil and web-based testing: the measurement invariance of the Big Five personality tests in applied settings. Assessment. 2012;19:243-6.
• ⁵ Ward P, Clark T, Zabriskie R, Morris T. Paper/pencil versus online data collection. J Leis Res. 2014;46:84-105.
• ⁶ Aust F, Diedenhofen B, Ullrich S, Musch J. Seriousness checks are useful to improve data validity in online research. Behav Res Methods. 2013;45:527-35.
• ⁷ Granello DH, Wheaton JE. Online data collection: strategies for research. J Couns Dev. 2004;82:387-93.
• ⁸ Lefever SC, Dal M, Matthíasdóttir R. Online data collection in academic research: advantages and limitations. Br J Educ Technol. 2006;38:574-82.
• ⁹ Vandenberg RJ, Lance CE. A review and synthesis of the measurement invariance literature: suggestions, practices, and recommendations for organizational research. Organ Res Methods. 2000;3:4-69.
• ¹⁰ Borsa JC, DeSousa DA. Invariância de medida e evidências de validade externa da Peer Aggressive Behavior Scale (PAB-S). Psico. 2018;49:178-86.
• ¹¹ Krueger RF, Derringer J, Markon KE, Watson D, Skodol AE. Initial construction of a maladaptive personality trait model and inventory for DSM-5. Psychol Med. 2012;42:1879-90.
• ¹² Al-Dajani N, Gralnick TM, Bagby RM. A psychometric review of the Personality Inventory for DSM–5 (PID–5): current status and future directions. J Pers Assess. 2016; 98:62-81.
• ¹³ Barchi-Ferreira AM, Osório FL. Personality Inventory for DSM-5 (PID-5): Cross-cultural adaptation and content validity in the Brazilian context. Trends Psychiatry Psychother. 2019;41;297-300.
• ¹⁴ Bo S, Bach B, Mortensen EL, Simonsen E. Reliability and hierarchical structure of DSM-5 pathological traits in a Danish mixed sample. J Pers Disord. 2016;30:112-29.
• ¹⁵ Suzuki T, South SC, Samuel DB, Wright AGC, Yalch MM, Hopwood CJ, et al. Measurement invariance of the DSM–5 Section III pathological personality trait model across sex. Personal Disord Theory Res Treat. 2019;10:114-22.
• ¹⁶ Zimmermann J, Altenstein D, Krieger T, Holtforth MG, Pretsch J, Alexopoulos J, et al. The structure and correlates of self-reported DSM-5 maladaptive personality traits: findings from two German-speaking samples. J Pers Disord. 2014;28;518-40.
• ¹⁷ Roskam I, Galdiolo S, Hansenne M, Massoudi K, Rossier J, Gicquel L, Rolland JP. The psychometric properties of the French version of the personality inventory for DSM-5. PLoS One. 2015;10;e0133413.
• ¹⁸ Bach B, Sellbom M, Simonsen E. Personality inventory for DSM-5 (PID-5) in clinical versus nonclinical individuals: generalizability of psychometric features. Assessment. 2018;25:815-25.
• ¹⁹ Debast I, Rossi G, van Alphen SP. Construct validity of the DSM-5 Section III maladaptive trait domains in older adults. J Pers Disord. 2017;31(5):671-88.
• ²⁰ Barchi-Ferreira AM, Osório FL. Psychometric study of the Brazilian version of the personality inventory for DSM-5-paper-and-pencil version. Front Psychiatry. 2022;13.
• ²¹ Keeley JW, Webb C, Peterson D, Roussin L, Flanagan EH. Development of a response inconsistency scale for the personality inventory for DSM–5. J Pers Assess. 2016;98:351-59.
• ²² Sellbom M, Dhillon S, Bagby RM. Development and validation of an Overreporting Scale for the Personality Inventory for DSM-5 (PID-5). Psychol Assess. 2018;30:582-93.
• ²³ Hair Jr JF, Black WC, Babin BJ, Anderson RE, Tathan RL. Análise multivariada de dados. 6ª ed. Porto Alegre: Bookman; 2009.
• ²⁴ Fabrigar LR, Wegener DT, MacCallum RC, Strahan EJ. Evaluating the use of exploratory factor analysis in psychological research. Psychol Methods. 1999;4:272-99.
• ²⁵ Henson RK, Roberts JK. Use of exploratory factor analysis in published research: common errors and some comment on improved practice. Educ Pschol Meas. 2006;66:393-416.
• ²⁶ Hayton JC, Allen DG, Scarpello V. Factor retention decisions in exploratory factor analysis: a tutorial on parallel analysis. Organ Res Methods. 2004;7:191-205.
• ²⁷ Velicer WF. Determining the number of components from the matrix of partial correlations. Psychometrika. 1976;41:321-27.
• ²⁸ Lorenzo-Seva U, Timmerman ME, Kiers HAL. The hull method for selecting the number of common factors. Multivariate Behav Res. 2011;46:340-64.
• ²⁹ Brown TA. Confirmatory factor analysis for applied research. New York: Guilford Press; 2006.
• ³⁰ Byrne BM. Structural equation modeling with AMOS: Basic concepts, applications, and programming. 2nd ed. London: Routledge Taylor & Francis Group; 2010.
• ³¹ Hu L, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Struct Equ Modeling. 1999;6:1-55.
• ³² CHEN FF. Sensitivity of goodness of fit indexes to lack of measurement invariance. Struct Equ Modeling. 2007;4;464-504. https://doi.org/10.1080/10705510701301834
  » https://doi.org/10.1080/10705510701301834
• ³³ Terwee CB, Bot SD, de Boer MR, van der Windt DA, Knol DL, Dekker J, et al. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol. 2007;60:34-42.
• ³⁴ Thimm JC, Jordan S, Bach B. Hierarchical structure and crosscultural measurement invariance of the Norwegian version of the personality inventory for DSM-5. J Pers Assess. 2017;99:204-10.
• ³⁵ Al-Attiyah AA, Megreya AM, Alrashidi M, Dominguez-Lara SA, Al-Sheerawi A. The psychometric properties of an Arabic version of the Personality Inventory for DSM-5 (PID-5) across three Arabic-speaking Middle Eastern countries. Int J Cult Ment Health. 2017;10:197-205.
• ³⁶ Gutiérrez F, Aluja A, Peri JM, Calvo N, Ferrer M, Baillés E, et al. Psychometric properties of the Spanish PID-5 in a clinical and a community sample. Assessment. 2017;24:326-36.
• ³⁷ Zimmermann J, Altenstein D, Krieger T, Holtforth MG, Pretsch J, Alexopoulos J, et al. The structure and correlates of self-reported DSM-5 maladaptive personality traits: Findings from two German-speaking samples. J Pers Disord. 2014;28:518-40.
• ³⁸ Riegel KD, Ksinan AJ, Samankova D, Preiss M, Harsa P, RF Krueger RP, et al. Unidimensionality of the personality inventory for DSM-5 facets: Evidence from two Czech-speaking samples. Pers Ment Health. 2018;12:281-97.
• ³⁹ Bo S, Bach B, Mortensen EL, Simonsen E. Reliability and hierarchical structure of DSM-5 pathological traits in a Danish mixed sample. J Pers Disord. 2016;30:112-29.
• ⁴⁰ Andrea H, Verheul R, Berghout C, Dolan C, Vanderkroft P, Busschbach JJ, et al. Measuring the core components of maladaptive personality: Severity indices of personality problems (SIPP-118). Report of the Viersprong Institute for Studies on Personality Disorders (VISPD) in cooperation with the Department of Medical Psychology & Psychotherapy, Erasmus University Rotterdam, The Netherlands. [Internet]. 2007. www.researchgate.net/publication/237140125_Measuring_the_Core_Components_of_Maladaptive_Personality_Severity_Indices_of_Personality_Problems_SIPP-118
  » www.researchgate.net/publication/237140125_Measuring_the_Core_Components_of_Maladaptive_Personality_Severity_Indices_of_Personality_Problems_SIPP-118
• ⁴¹ Zhang W, Wang M, Meng Yu M, Wang J. The hierarchical structure and predictive validity of the Personality Inventory for DSM-5 in Chinese nonclinical adolescents. Assessment. 2022;29:1559-75.
• ⁴² Fouladi RT, McCarthy CJ, Moller N. Paper-and-pencil or online? Evaluating mode effects on measures of emotional functioning and attachment. Assessment. 2002;9:204-15.
• ⁴³ Chen FF. Sensitivity of goodness of fit indexes to lack of measurement invariance. Struct Equ Modeling. 2007;14:464-504.
• ⁴⁴ Contractor AA, Caldas SV, Dolan M, Lagdon S, Armour C. PTSD's factor structure and measurement invariance across subgroups with differing count of trauma types. Psychiatry Res. 2018;264:76-84.
• ⁴⁵ Tippins NT, Beaty J, Drasgow F, Gibson WM, Pearlman K, Segall DO, Shepherd W. Unproctored Internet testing. Pers Psychol. 2006;59:189-225.
• ⁴⁶ Trippe DM. Equivalence of online and traditional forms of a Five Factor Model measure. Paper presented at the 20th Annual SIOP Conference, Los Angeles, California; 2005.
• ⁴⁷ Sorrel MA, García LF, Aluja A, Rolland JP, Rossier J, Roskam I, Abad FJ. Cross-cultural measurement invariance in the Personality Inventory for DSM-5. Psychiatry Res. 2021;304:114-34.
• ⁴⁸ Suzuki T, South SC, Samuel DB, Wright AG, Yalch MM, Hopwood CJ, Thomas KM. Measurement invariance of the DSM–5 Section III pathological personality trait model across sex. Personal Disord Theory Res Treat. 2019;10:114-22.

Edited by

• Handling Editor:
  Adriane Rosa

Publication Dates

• Publication in this collection
  23 Mar 2026
• Date of issue
  2026

History

• Received
  03 Dec 2023
• Accepted
  14 May 2024