Reliability and validity of a questionnaire on vulnerability to sexually transmitted infections among adults with chronic mental illness: PESSOAS Project

Confiabilidade e validade de um questionário sobre vulnerabilidade para infecções sexualmente transmissíveis entre adultos com transtorno mental crônico: Projeto PESSOAS

BRIEF REPORT

Reliability and validity of a questionnaire on vulnerability to sexually transmitted infections among adults with chronic mental illness - PESSOAS Project

Confiabilidade e validade de um questionário sobre vulnerabilidade para infecções sexualmente transmissíveis entre adultos com transtorno mental crônico – Projeto PESSOAS

Mark Drew Crosland Guimarães^I,II; Helian Nunes de Oliveira^II,III; Lorenza Nogueira Campos^II; Carolina Ali Santos^II; Carlos Eduardo Resende Gomes^II; Suely Broxado de Oliveira^IV; Maria Imaculada de Fátima Freitas^II,V; Francisco de Assis Acúrcio^II,VI; Carla Jorge Machado^II,VII

^IDepartment of Preventive and Social Medicine, Medical School, Universidade Federal de Minas Gerais (UFMG), Belo Horizonte (MG), Brazil

^IIGrupo de Pesquisas em Epidemiologia e Avaliação em Saúde (GPEAS/CNPq), Department of Preventive and Social Medicine, Medical School, Universidade Federal de Minas Gerais (UFMG), Belo Horizonte (MG), Brazil

^IIIHospital de Ensino Instituto Raul Soares, Fundação Hospitalar do Estado de Minas Gerais, Belo Horizonte (MG), Brazil

^IVInstitute of Psychiatry, Universidade Federal do Rio de Janeiro (UFRJ), Rio de Janeiro (RJ), Brazil

^VSchool of Nursing, Universidade Federal de Minas Gerais (UFMG), Belo Horizonte (MG), Brazil

^VIDepartment of Social Pharmacy, School of Pharmacy, Universidade Federal de Minas Gerais (UFMG), Belo Horizonte (MG), Brazil

^VIICenter of Regional Development and Planning, School of Economic Sciences, Universidade Federal de Minas Gerais (UFMG),

Belo Horizonte (MG), Brazil

^{Correspondence}

ABSTRACT

OBJECTIVE: To describe reliability/validity of a semi-structured questionnaire designed to assess risk behavior to sexually transmitted diseases among adults with chronic mental illness.

METHOD: A cross-sectional pilot study was conducted in one psychiatric hospital and one mental health outpatient clinic. Clinical, behavioral and demographic data were collected from semi-structured interviews and medical charts. One-hundred and twenty patients were randomly selected from pre-defined lists in both centers while 89 (74%) were interviewed, indicating 26% nonparticipation rate. Protocol, participation rates, consent form and feasibility issues were assessed. The semi-structured interview was evaluated with regard to reliability (intra- and inter-rater) and construct validity by randomly repeating the interviews in a 1:1 ratio up to one-week interval. Reliability was estimated by percent agreement and Kappa statistics (95% confidence interval). Construct validity was assessed by Grade of Membership model.

RESULTS: Kappa statistics ranged from 0.40 to 1.00 for most variables. Grade of Membership analysis generated three profiles. Profile one was represented by mostly women with no condom use in stable relationships; profile two revealed mostly men in stable relationship but with multiple risk behaviors; while profile three indicated a higher proportion of licit or illicit substance use.

CONCLUSIONS: Reliability and construct validity assessment using Grade of Membership analysis indicated that the semi-structured interview was suitable for capturing risk behavior among patients with chronic mental illness.

Descriptors: Sexually transmitted diseases; Risk-taking; Reproduction of results; Validity of tests; Mental health

RESUMO

OBJETIVO: Descrever a confiabilidade/validade de um questionário semi-estruturado para avaliar comportamento de risco para as infecções sexualmente transmissíveis entre adultos com doença mental crônica.

MÉTODO: Estudo-piloto transversal em um hospital psiquiátrico e um centro de atenção psicossocial. Dados clínicos, comportamentais e sociodemográficos foram coletados por entrevistas semi-estruturadas/prontuários médicos. Cento e vinte pacientes foram selecionados aleatoriamente a partir de listas pré-definidas de ambos os centros, enquanto 89 (74%) foram entrevistados, indicando uma taxa de não-participação de 26%. Protocolo, participação, consentimento e outros fatores foram avaliados. A entrevista semi-estruturada foi avaliada em sua confiabilidade (intra e inter-observador) e validade de construto, por repetição aleatória (razão 1:1), com até uma semana de intervalo. A confiabilidade foi estimada pelo percentual de concordância/coeficiente Kappa (intervalo de confiança 95%). A validade de construto foi avaliada pelo modelo Grade of Membership.

RESULTADOS: Os coeficientes Kappa variaram de 0,40-1,00 para a maioria das variáveis. A análise Grade of Membership gerou três perfis de vulnerabilidade. O perfil um representou principalmente mulheres em relacionamentos estáveis que não usavam preservativo; o perfil dois revelou principalmente homens em relacionamentos estáveis, mas múltiplos comportamentos de risco; o terceiro perfil indicou uma alta proporção de uso de substâncias lícitas e/ou ilícitas.

CONCLUSÕES: A confiabilidade e validade de construto indicaram que a entrevista foi adequada para captar os comportamentos de risco nesta população.

Descritores: Infecções sexualmente transmissíveis; Assunção de riscos; Reprodutibilidade dos testes; Validade dos testes; Saúde mental

Introduction

Chronic mentally ill patients (CMIP) are known to have an increased vulnerability, i.e., susceptibility or factors related to individuals or groups that could increase the risk of an adverse event, to several health conditions including sexually transmitted infections (STI) such as HIV and syphilis.¹ Social and behavioral characteristics comprise the main factors that pose CMIP at increased vulnerability to such conditions.^2,3 Early assessment of potential increased risks must be assured in order to properly develop public health interventions and prevention efforts at both individual and mental health institution levels. However, tested instruments specifically designed to describe risk behavior among CMIP in Brazil are unknown, while the existing instruments have mostly been developed for severe mentally ill patients in developed countries.^4-7 In addition, there is scarce data on risk behavior among CMIP in Brazil, usually with highly selected populations. PESSOAS Project (Pesquisa em Soroprevalência de Aids na Saúde Mental) is the result of a joint effort between the Brazilian AIDS/STD and Mental Health Programs and the Universidade Federal de Minas Gerais, and it was designed to assess risk behavior and STI/HIV prevalence in a national representative sample of CMIP under public hospital or outpatient care in Brazil. In order to assess and obtain reliable and valid sociodemographic, behavioral and clinical characteristics of the CMIP participating in PESSOAS project, a semi-structured questionnaire was specifically designed and tested in a short pilot study. The objective of this paper is to describe the reliability and validity of this instrument.

Method

1. Subjects and protocol

The pilot study was conducted in one public psychiatric hospital and one public mental health outpatient clinic (CAPS) after ethical approval (UFMG/ETIC 195/04). Participants should be 18 years old or over and should provide written informed consent. In order to achieve a priori sample size of 90 patients, 45 in each service, we randomly recruited 120 patients from predefined lists, considering a 40% nonparticipation rate. A seven-day period was allowed for clinical stabilization of hospitalized patients. Before actual recruitment, a preliminary assessment was carried out by trained mental health professionals in order to evaluate the clinical and mental conditions of the patients. The focus of this preliminary assessment was to evaluate proper spatial and timing orientation, ability to understand the aims of the project, ability to answer the questionnaire and legal capacity for signing the informed consent. In addition, mental health professionals were asked to register any clinical occurrence during interviews that could potentially alter reliability.

2. Risk assessment questionnaire

After signing the consent form, a semi-structured interview specifically developed for PESSOAS Project was administered. It consisted of five parts according to the following groups: 1) sociodemographic data (e.g.: age, gender, race, schooling, income); 2) clinical characteristics (e.g.: psychiatric diagnosis and treatment, STI history, nonpsychiatric diagnosis); 3) sexual behavior and substance use characteristics and risk assessment for STI (e.g.: sexual practices, alcohol, tobacco and illicit substance use, history of violence); and, 4) assessment of HIV/AIDS knowledge and perception of risk. The fifth part included diagnosis and treatment data obtained from medical chart. A detailed list of all variables can be found in http://saudepublica.medicina.ufmg.br/dissert_teses.htm. ⁸

The semi-structured questionnaire was designed and pretested before the pilot study as follows: 1) extensive literature review to select questions pertinent to the risk behavior assessment; 2) expert discussion panel with the research staff for selection of questions; 3) pretest assessment with a few patients; 4) re-discussion with panel of experts for final version to be used in the pilot study. During the pretest assessment, trained interviewers were asked to phrase questions in a sequential manner until a final version had been obtained in order to guarantee understanding of each question by the patient. A manual was designed containing a detailed description of the study, procedures for recruitment and specific instructions for each question. The final version of the questionnaire is available at http://saudepublica.medicina.ufmg.br/instrumentos.htm.

Interviews were administered by experienced and trained mental health care professionals. They should have higher education training and be staff members of each center. After the main interview, participants were randomly invited to have a second interview with the same interviewer (intra-rater or test-retest comparison) or with a different interviewer (inter-rater or observer comparison) for reliability purposes. In both situations, a maximum of seven-day interval between the two applications was allowed.

3. Statistical analysis

Descriptive analysis was carried out regarding protocol procedures and overall population characteristics. Intra-rater and inter-rater reliabilities were estimated using percent agreement and Kappa statistic with 95% confidence interval for categorical variables.⁹

The ability of the questionnaire to capture vulnerability characteristics (construct validity) was indirectly assessed by Grade of Membership analysis (GoM). GoM is a multivariate statistical technique suited for analyzing data consisting of several different measurements for each individual, where each of these measurements defines a dimension. The GoM model estimates vulnerability profiles and a grade of membership of each patient within each profile. Profiles can be described by the probability of the occurrence of specific characteristics of individuals with full membership.⁹ However, in GoM analysis, individuals in the sample may be a partial member of two or more profile groups. Therefore, the method estimates the degree to which each patient in the data set belongs to each of these profiles and are represented by a set of "grades of membership" or "scores" that ranges between zero and one, and adds up to one for each individual.⁹ As a rule, individuals with scores = 1 are defined as complete membership or pure/extreme profile. We assumed that individuals with scores > 0.75 were considered as admissible or pure profile, those with scores d 0.75 and f 0.50 were considered to have a membership of a mixed profile, while scores < 0.50 were considered as amorphous or residual. The vulnerability profiles were then described dividing the estimated probability (E) that an individual from a pure profile belongs to a defined level (e.g.: male) of a specific vulnerability characteristic (e.g.: gender) by the corresponding marginal observed frequencies (O) in order to assess the relative importance of each level.¹⁰ Categories of each variable with quotient E/O f 1.2 indicated higher discrimination power, i.e. increased vulnerability, within each profile, as suggested by Sawyer.¹¹

For the present analysis, selection of variables used to characterize the vulnerability profiles to STI was based on the literature review and the reliability assessment of the current data as follows: gender, age, marital condition, history of homelessness, history of STI, lifetime alcohol use, condom use (past six months, last intercourse and refusal of use by partner), sexual violence, self-perceived risk, number of sexual partners, illicit drug use, psychoactive drug use during sex and exchange of sex for money or drugs. GoM analysis was performed assuming three profiles and the significance of an additional profile was tested as an independent increment in the fit of the model adjusting for the larger number of degrees of freedom in the larger model. Akaike information criterion (AIC) for small sample sizes was used to assess whether the three generated profiles were adequate to represent the vulnerability characteristics in this population.¹² Analyses were carried out by SAS^®, SPSS^® and GOM3^® software, while data was entered using Paradox^®.

Results

1. Recruitment

Among the 120 recruited patients, 89 (74%) were interviewed while 14 (12%) were not capable of participating according to the preliminary assessment, and 17 (14%) refused, missed the appointment or were discharged before the interview. Participation rate was similar with regard to type of care (p = 0.53); schooling (p = 0.62) or type of diagnosis (p = 0.31), while older patients (> 35 years old) and those living outside Belo Horizonte tended to be nonparticipants (p < 0.05). There was no difference when comparing type of diagnosis (psychosis and others) among hospital (p = 0.41) or CAPS (p = 0.12) patients. Most patients were cooperative (98%) and relaxed during interviews (88%), while few were uncomfortable (14%), uninterested (10%), sleepy (8%), disorganized (7%) or hostile (3%). Mean duration of the interview was 42 minutes (median = 39).

Most participants were < 35 years old (54%), male (59%), nonwhite (72%), single (63%), had < 5 years of schooling (51%), and 88% were able to read and write. A large proportion had a history of homelessness (27%), and had multiple hospitalizations (62%). Schizophrenia was the most common diagnosis (80%) followed by bipolar disorder (10%) and others (10%). Recent (past month) alcohol, cannabis, crack or injection drug use were reported by 37%, 32%, 14% and 4% of participants, respectively. History of STI and exchange sex for money were reported by 29% and 40%, respectively, while only 10% had always used condoms ever and 33% in the last sexual intercourse. Moreover, 26% and 36% of the participants declared at least one episode of sexual violence or imprisonment, respectively.

2. Reliability

Most variables tested for reliability showed Kappa coefficients varying from 0.40 to 1.00 (81%) while only 19% had poor agreement (Kappa < 0.40). Among variables fitted in GoM analysis (Table 1), age, gender, marital condition, sex for money, history of homelessness and sexual violence, last intercourse condom use, substance use during sex, number of lifetime partners and alcohol use had Kappa coefficients above 0.60, while condom use in the last six months, partner's refusal to use condom, HIV risk perception and illicit drug use had lower coefficients (< 0.60). There was no difference in Kappa estimates when comparing patients with delusional symptoms during interview, except for risk perception of acquiring HIV – those with delusional symptoms had lower agreement as compared to those without these symptoms [Kappa = 0.17 (95% CI = -0.05; 0.39) and 0.55 (95% CI = 0.37; 0.72), respectively]. Detailed analysis of all variables can be found in http://saudepublica.medicina.ufmg.br/dissert–teses.htm. ⁸

3. Vulnerability profiles

Three profiles, P1, P2 and P3, were generated among the 89 subjects available for GoM analysis – 41.5%, 22.4% and 28.1%, respectively. Among these, 33 individuals (37.1%) were classified as pure or admissible profiles; 49 individuals (55.1%) as mixed profile and 7 individuals (7.9%) were classified as amorphous or residual profile. Goodness-of-fit evaluation indicated a lower AIC for the three-profile model, which is evidence that it was adequate to describe vulnerability among this population (AIC = -984.0).

Table 1 shows the observed values and E/O for each level of indicator variables. As indicated (E/O > 1.2), pure or extreme profile one (P1) consisted predominantly of women, 44 years old or older, married or living with partner, with no prior history of STI, who reported no alcohol use and no condom use in the last six months and in the last sexual intercourse. Individuals in this extreme profile also reported sexual violence, believed that they were not at risk for HIV infection, had 10 or fewer sexual partners and had no prior history of exchanging sex for money or drugs.

Pure or extreme profile two (P2) consisted mainly of men, 35 to 43 years old, married or living with partner, with prior history of homelessness, past history of STI, lifetime alcohol use, irregular condom use (not always or never), partner's refusal to use condom. They also had more than 10 sexual partners during lifetime, exchanged sex for money, drugs or other assets, reported no condom use in the last sexual intercourse, use of illicit drug in lifetime and psychoactive substance use during sexual activity. Finally, extreme profile three (P3) can be described by a higher proportion of men, 18 to 34 years old, single or divorced, with no history of being homeless or having any STI, with a past history of alcohol use, who reported regular condom use (always) in the last six months or had no sexual partner in the last six months. They also had no prior history of partner's refusal to use condoms. Additionally, individuals in P3 believed they were not at risk for HIV infection, reported condom use in the last sexual intercourse, had 10 or fewer sexual partners in lifetime, used illicit drugs in lifetime and denied using psychoactive substance during sexual activity.

Discussion

In the present pilot study, we were able to assess feasibility and reliability issues for a larger national representative risk behavior study among CMIP under care (PESSOAS Project). As shown, most patients were able to understand and sign a written informed consent, while the preliminary assessment enabled interviewers to obtain reliable answers. However, one potential limitation of the study is the exclusion of patients unable to participate due to cognitive deficit or mental deficiency. This may potentially underestimate risk behavior characteristics among CMIP.

Studies of sexual behavior and vulnerability conditions rely mostly on self-reported answers, which can be affected by memory errors, social stigma, embarrassment or clinical conditions, especially for individuals with chronic mental illness. In general, our Kappa statistics indicated reasonable to excellent agreement for most questions assessed. However, questions more difficult to be understood or requiring previous knowledge tended to have lower agreement. Also, more embarrassing or stigma/legal related questions, such as illicit drug use and partner's refusal to use condom had lower agreement. In addition, a more careful approach should be conducted if patients present any delusional symptoms during interview. Of public health importance is the lower agreement on HIV perception of risk among those with delusional symptoms, although not statistically different. This may be due to poor understanding and/or indicate impaired judgment of actual risks, which may potentially put patients with delusional states at an increased risk for STI. It is clear, though, that our findings indicated that reliable information for most variables including sociodemographic, clinical, sexual behavior, and substance data can be obtained among chronic mentally ill patients.

Few studies in the medical literature have indicated consistent validity for sexual behavior assessment of psychiatric patients, mostly in developed countries.^4-7 In the absence of direct measures of validity for risk behavior characteristics, indirect assessment by means of GoM analysis was a reasonable approach herein adopted. The vulnerability profiles generated revealed a number of risk behavior characteristics clustered together which are corroborated by the literature.^13-15 Profile one (42%) was represented by mostly women with no condom use in stable relationships (married or in union). On the other hand, profile two (23%) revealed mostly men in union relationship with a number of additional risky behaviors. It seems reasonable to assume that individuals who share predominant characteristics of this profile have a greater probability of transmitting HIV/STI to their partners. Finally, most individuals in profile three (28%) reported regular condom use and no history of sexual activity in the last six months. However, a higher proportion of licit (alcohol) or illicit psychoactive substance abuse could potentially place them at higher risk of infection. In conclusion, our results indicated the ability of our questionnaire to adequately capture key risk behavior characteristics among CMIP with reasonable reliability. In addition, it can be a valuable tool for screening high risk populations in similar health care settings by trained professionals, so proper counseling and early intervention be appropriately taken.

Financing: Programa Nacional de DST/Aids – MS (UNESCO/DST-Aids 914/BRA/3014 – Project 169/03).

Conflict of interests: None

Submitted: May 25, 2007

Accepted: November 27, 2007

This study is part of the MSc thesis in Public Health/UFMG of one of the authors (Helian Nunes de Oliveira) concluded in 2006.

• 1. Mann J, Tarantola D. AIDS in the World II: global dimensions, social roots, and responses New York (NY): University Press, Oxford; 1996.
• 2. Cournos F, Mckinnon K. HIV seroprevalence among people with severe mental illness in the United States: a critical review. Clin Psychol Rev. 1997;17(3):259-69.
• 3. Rosenberg SD, Goodman LA, Osher FC, Swartz MS, Essock SM, Butterfield MI, Constantine NT, Wolford GL, Salyers MP. Prevalence of HIV, hepatitis B, and hepatitis C in people with severe mental illness. Am J Public Health. 2001;91(1):31-7.
• 4. McKinnon K, Carey MP, Cournos F. Research on HIV, AIDS, and severe mental illness: recommendations from the Ninth National Conference. Clin Psychol Rev. 1997;17(3):327-31.
• 5. McKinnon K, Cournos F, Meyer-Bahlburg HF, Guido JR, Caraballo LR, Margoshes ES, Herman R, Gruen RS, Exner TM. Reliability of sexual risk behavior interviews with psychiatric patients. Am J Psychiatry. 1993;150(6):972-4.
• 6. Sohler N, Colson PW, Meyer-Bahlburg HF, Susser E. Reliability of self-reports about sexual risk behavior for HIV among homeless men with severe mental illness. Psychiatr Serv 2000;51(6):814-6.
• 7. Auerbach JD, Coates TJ. HIV prevention research: accomplishments and challenges for the third decade of AIDS. Am J Public Health. 2000;90(7):1029-32.
• 8. Oliveira HN. Usuários de serviços de saúde mental e vulnerabilidade para DST/HIV: estudo piloto [dissertação]. Belo Horizonte (MG): UFMG; 2005. Available from: http://saudepublica.medicina.ufmg.br/dissert_teses.htm
• 9. Szklo M, Nieto JF. Epidemiology: Beyond the Basics 2nd ed. Baltimore, MD: Aspen Publishers; 2007.
• 10. Manton KG, Woodbury MA, Tolley HD. Statistical Applications Using Fuzzy Sets New York, NY: John Wiley & Sons; 1994.
• 11. Cassidy F, Pieper CF, Carroll BJ. Subtypes of mania determined by Grade of Membership analysis. Neuropsychopharmacology 2001;25(3):373-83.
• 12. Sawyer DO, Leite IC, Alexandrino R. Profiles of health services utilization in Brazil. Cien Saude Colet 2002;7(4):757-76.
• 13. Akaike H. Information theory and extension of the maximum likelihood principle. Proceedings of the 2nd International Symposium on Information Theory; 1973; Budapest, Hungary. p. 267-81.
• 14. Rosenberg SD, Swanson JW, Wolford GI, Osher FC, Swartz MS, Essock SM, Butterfield MI, Marsh BJ. The five-site health and risk study of blood-borne infections among persons with severe mental illness. Psychiatr Serv. 2003;54(6):827-35.
• 15. Meade CS, Sikkema KJ. HIV risk behavior among adults with severe mental illness: a systematic review. Clin Psychol Rev. 2005;25(4):433-57.

Publication Dates

• Publication in this collection
  31 Jan 2008
• Date of issue
  Mar 2008

History

• Accepted
  27 Nov 2007
• Received
  25 May 2007