Influence of speech stimuli in the auditory perceptual identification of hypernasality in individuals with cleft lip and palate

ABSTRACT

Purpose

To investigate the influence of speech stimuli in the auditory perceptual identification of hypernasality in individuals with Cleft Lip and Palate (CLP).

Methods

Speech samples from 80 individuals with operated unilateral CLP, ages ranged from nine to 17 years (the mean age of: 12y7m), both genders, were edited for this study. Samples were recorded over the production of nine different speech stimuli, including counting and short sentences characterized by oral sounds, one loaded with low pressure consonants and seven loaded with high pressure consonants. Three speech-language pathologists rated the presence or absence of hypernasality while analyzing 864 recordings (80 individuals X 9 stimuli + 144 repeated recordings, for measuring the intra-rater agreement). Intra-rater and inter-rater indexes of agreement were established for all nine stimulus conditions. The indexes of inter-rater agreement were compared using the Z test (p<0.005), with samples comprising significant indexes of agreement interpreted as better stimuli for identifying the hypernasality in these individuals.

Results

Intra-rater agreement for high pressure stimuli with voiced consonants were significantly lower than indexes for other stimuli. Inter-rater agreement between each pair of SLPs ranged from 0.11 (plosive voicing stimuli) to 0.57 (12 short sentences, one of each high pressure consonant). The values of mean inter-rater agreement between all SLPs was 0.47 indicating moderate agreement for identifying hypernasal speech.

Conclusion

Speech recordings obtained over the production of longer speech samples including 12 short sentences, for instance one for each high pressure consonant, may favor inter-rater agreement for identifying hypernasality.

Keywords
Cleft Palate; Velopharyngeal Insufficiency; Speech; Speech Disorders; Speech Perception; Resonance

RESUMO

Objetivo

Investigar a influência de estímulos de fala distintos na identificação perceptivo-auditiva da hipernasalidade em indivíduos com fissura labiopalatina operada (FLP).

Método

Foram editadas amostras de fala gravadas em áudio de 80 indivíduos com FLP unilateral operada, de ambos os sexos, com idades entre 9 e 17 anos (média=12 anos e 7 meses). As amostras foram gravadas durante a produção de 9 estímulos de fala distintos: contagem de números e conjuntos de frase orais, sendo 1 constituído por consoantes de baixa pressão e 7 constituídos por consoantes de alta pressão. Três fonoaudiólogas identificaram a presença ou ausência da hipernasalidade ao analisarem 864 gravações (80 indivíduos X 9 estímulos + 144 gravações repetidas para análise de concordância intra-avaliador). Os índices de concordância intra e interavaliadores foram estabelecidos para todos os 9 estímulos de fala e comparados entre si por meio do Teste Z, com nível de significância de 5%, com maiores índices de concordância interpretados como melhores estímulos para identificação da hipernasalidade.

Resultados

Índices de concordância intra-avaliadores de estímulos de fala vozeados foram significativamente menores do que outros estímulos. Índices de concordância entre os pares de fonoaudiólogas variaram de 0,11 (concordância estímulos plosivos vozeados) a 0,57 (12 frases, uma com cada consoante de alta pressão), com média de 0,47 entre as três avaliadoras, indicando concordância moderada para identificação da hipernasalidade.

Conclusão

Gravações de fala obtidas durante a produção de estímulos mais longos, incluindo 12 frases, uma com cada consoante de pressão, podem favorecer a concordância interavaliador na identificação da hipernasalidade.

Descritores
Fissura Palatina; Insuficiência Velofaríngea; Fala; Percepção de Fala; Distúrbios de Fala; Ressonância

INTRODUCTION

Individuals with cleft lip and palate (CLP) surgically repaired are considered at risk for speech disorders⁽¹⁾. The non-anatomical and functional conditions to establish adequate velopharyngeal closure in speech after primary palatoplasty results in characteristic speech symptoms, with hypernasality being the most common and most representative symptom of velopharyngeal dysfunction (VPD)⁽²⁾. Hypernasality occurs when there is an excess of nasal resonance during the production of oral sounds due to the abnormal coupling of the resonance cavities (oral, nasal and pharyngeal). Since resonance is an acoustic-perceptual phenomenon, the clinical evaluation of this speech characteristic is performed through perceptual-auditory analysis⁽²⁾. According to some authors, no acoustic measure can completely replace the information that a well-trained ear can reveal about speech nasality⁽³⁾, this variable being essential in the diagnosis of VPD.

In the VPD diagnostic process, instrumental measurements (nasoendoscopy, videofluoroscopy, nasometry and flow-pressure technique) are commonly used, as they offer valuable information that corroborates perceptual findings⁽⁴⁾. However, the auditory-perceptual assessment is the initial resource used by the speech-language pathologist to identify the speech disorders of the VPD⁽⁵⁾, being the same considered “gold standard” for the identification of these changes⁽¹⁾. The auditory-perceptual evaluation must be conducted by experienced professionals^(6,7) and the findings obtained through such evaluation favor clinical decision-making, taking into account the principle that treatment should only be indicated when speech disorders are perceived by the patient or people around him^(3-5). Despite being indispensable in the assessment and definition of conduct of the VPD, the subjective nature of the auditory-perceptual assessment makes it a challenging process for the clinical speech-language pathologist, being able to suffer errors and variations⁽⁸⁾, even when conducted by experienced professionals⁽⁷⁾. To improve it, in addition to training the listener^(7,9,10), it is recommended to record audio and/or video speech with quality equipment for analysis by multiple evaluators, to increase the reliability of the auditory-perceptual assessment of speech⁽⁹⁾.

Clinically, the identification of the presence of hypernasality is obtained through auditory-perceptual assessment using a binary scale (normal or altered) or through numerical scales of equal intervals (for example, 4-point scale⁽¹¹⁾), in which the evaluator attributes an index to the assessed speech aspect, indicating its level of severity, with the lowest value referring to the absence of the alteration and the highest value to the maximum degree of the alteration⁽¹⁾. Other types of scales, including direct magnitude estimation, paired comparisons, with or without reference samples⁽¹⁾ and visual analog scale^(3,12,13) have been proposed to identify hypernasality. More recently, different methods (2-step method, VISOR method) and Borg scale⁽¹⁴⁾ were introduced to assess the reliability of the auditory-perceptual analysis of hypernasality. Discussions about procedures that can better favor the auditory-perceptual analysis of speech nasality remain among researchers^(5,13). According to the literature, descriptive categories and the use of scales of equal intervals are the procedures commonly used to document hypernasality⁽¹⁾. However, the CLP management process must include procedures for analyzing speech hypernasality that initially emphasize the identification of its presence and absence⁽¹¹⁾.

The identification of hypernasality is not an easy task, even for experienced listeners, and several factors can influence this task⁽⁸⁾, for example, the extension of the speech stimulus^(14,15) and the phonetic context that constitutes the sample of analyzed speech^(2,14). In a previous study, the listener's reliability in analyzing speech nasality was greater for longer than shorter stimuli (sentences larger than isolated words, and isolated words greater than isolated vowels⁽¹⁵⁾). A recent study⁽¹⁴⁾ suggested that the use of repetition of nine non–nasal single word strings (similar to a short sentence), with a controlled phonetic context (high-pressure consonants followed by high vowels), favored the analysis of hypernasality by the perceptual assessment methods used. As for the phonetic context, the literature suggests that the high vowels facilitate the identification of hypernasality^(2,14,16) and that a certain individual may be perceived as more nasal when the phonetic context includes nasal consonants due to the assimilation effect of these consonants in voiced vowels or consonants that precede or follow them⁽²⁾. Speech stimuli consisting of vowels, semivowels and liquid consonants can be useful to isolate hypernasality from other speech symptoms of VPD⁽¹⁷⁾. However, a study showed greater reliability among examiners in the perceptual analysis of hypernasality for high-pressure speech samples (plosives and fricatives), when compared with low-pressure samples (liquid)⁽¹⁸⁾.

The selection of speech material is therefore considered an important factor when analyzing hypernasality^(10,14,16). Among the stimuli used to capture speech samples for later identification of hypernasality, the isolated production of words, sentences, spontaneous conversation^(2,16,17) and numerical counting⁽²⁾ are included. Although spontaneous conversation offers important information about the presence and degree of hypernasality⁽⁴⁾, this type of speech material can make the task of the evaluator more difficult due to the influence of several factors (phonetic context of the speech, rate of speech and pitch⁽²⁾ and the presence of compensatory articulations⁽¹⁶⁾). The repetition of sentences or word sequences allows the control of the phonetic context of the speech stimulus^(14,19), which may favor the auditory-perceptual analysis of hypernasality. However, the possible effect of the co-occurring articulation disorder in the auditory-perceptual analysis of hypernasality⁽²⁰⁾ must be considered. Also, better intra-rater reliability rates were reported for a set of 11 exclusively high-pressure oral sentences, compared to samples of spontaneous conversation during the assessment of hypernasality⁽²¹⁾, suggesting that differences in this assessment may occur for the same individual according to the speech stimulus.

Although speech stimuli involving sentences consisting of oral vowels and high-pressure consonants have shown good intra-rater reliability⁽²¹⁾ and, also, greater reliability among examiners when compared to low-pressure (liquid) sentences⁽¹⁸⁾, so far, it is not clear whether the extension of this type of sentence and/or its phonetic constitution can influence the identification in hypernasality. Considering that the clinical evaluation of hypernasality is a challenging task for speech-language pathologists and that this variable is the most important indicator of the results of surgery and the primary symptom of VPD⁽¹¹⁾ it is essential to seek strategies that make the identification of hypernasality more reliable, concerning the intra- and inter-rater reliability coefficients. In this sense, the objective of the study was to investigate the influence of different speech stimuli in the auditory-perceptual identification of hypernasality in individuals with operated CLP.

METHODS

Observational, cross-sectional study approved by the Human Research Ethics Committee at the Hospital for Rehabilitation of Craniofacial Anomalies (HRCA-USP), Universidade de São Paulo, Bauru, Brazil (No. 1.938,881/2017). The Free and Informed Consent Term was waived in the study since it involved speech samples already recorded and pre-existing in the database of the institution.

In the study, speech recordings belonging to 80 individuals with operated unilateral CLP, with or without VPD, of both sexes, aged between 9 and 17 years (mean of 12 years and 7 months) were selected. The recordings were selected from those with good audio quality stored in the database of the Laboratory of Experimental Phonetics (HRCA-USP). In this laboratory, the recordings are performed routinely, in an acoustically treated room, with the speech material recorded directly on the computer, equipped with a Sound Blaster Audigy 2 sound card and the Sony® Sound Forge program, version 7.0, with a sampling rate of 44100 Hz, in single-channel, 16 bits. The speech samples of the study were captured using a head microphone (model AKG C420®), positioned approximately 5 cm from the side of the patient's labial commissure.

Speech stimuli

Each of the 80 recordings selected for the study consisted of 9 types of speech stimuli, categorized in: 1) set with 12 short senteces containing predominantly fricative and plosive consonants - 12FRIPLO; 2) set of 3 short senteces containing predominantly voiced fricative consonants - 3FVOZ; 3) set of 3 short senteces containing predominantly unvoiced fricative consonants - 3FNVOZ; 4) set of 3 short senteces containing predominantly voiced plosive consonants - 3PVOZ; 5) set of 3 short senteces containing predominantly unvoiced plosive consonants - 3PNVOZ; 6) set of 6 short senteces containing predominantly voiced consonants, 3 with voiced fricative consonants and 3 voiced plosives consonants - 6FPVOZ; 7) set of 6 short senteces containing predominantly unvoiced consonants, 3 with unvoiced fricative consonants and 3 with unvoiced plosive consonants - 6FPNVOZ; 8) set of 4 short senteces containing liquid consonants - LIQ and 9) counting numbers from 1 to 10 - CONT (Chart 1). Of the nine speech stimuli, seven consisted of oral vowels and high-pressure consonants that differed according to the phonetic context (plosives and/or fricatives, voiced or unvoiced) and their length, that is, the number of sentences (in that the most extensive stimulus was made up of 12 short senteces and the less extensive stimuli were made up of 6 or 3 short senteces). One of the stimuli consisted of oral vowels and oral consonants predominantly of low intra-oral pressure (liquids) and another speech stimulus consisted of oral and nasal sounds (count from 1-10).

Chart 1
Phonetic context, extension (number of short sentences) of speech stimuli and set of corresponding sentences

Phonetic Context of Speech Stimulus	Stimulus extension	Set of Short Sentences (in Brazilian Portuguese)
Fricatives/Plosives (12FRIPLO)
High intraoral pressure	12 consecutive fricative/plosive short sentences	“O piupiu piou, o tatu é da Talita, A Cuca correu e caiu; A rosa azul é da Zezé; Júlia ralou o joelho; A vovó viu a uva; Fafá foi a feira, Cecília laçou o saci, a Xuxa achou o xale”
Fricative and plosive (manner of articulation)
Voiced and unvoiced
Voiced Fricatives (3FVOZ)
High intraoral pressure	Only 3 short sentences	“A rosa azul é da Zezé; Júlia ralou o joelho; A vovó viu a uva”
Fricative (manner of articulation)	(voiced fricatives)
Voiced
Unvoiced Fricatives (3FNVOZ)
High intraoral pressure	Only 3 shot sentences	“Fafá foi a feira, Cecília laçou o saci, a Xuxa achou o xale”
Fricative (manner of articulation)	(unvoiced fricatives)
Unvoiced
Voiced Plosives (3PVOZ)	Only 3 short sentences
High intraoral pressure	(voiced plosives)	“O bebê babou; O dedo da Duda doeu; O Gugu é gago”
Plosive (manner of articulation)
Voiced
Unvoiced Plosives (3PNVOZ)	Only 3 short sentences
High intraoral pressure	(unvoiced plosives)	“O piupiu piou, o tatu é da Talita, a Cuca correu e caiu”
Plosive (manner of articulation)
Unvoiced
Voiced Fricatives/Plosives (6FPVOZ)	6 short sentences
High intraoral pressure	(voiced fricatives/plosives)	“A rosa azul é da Zezé; Júlia ralou o joelho; A vovó viu a uva; O bebê babou; O dedo da Duda doeu; O Gugu é gago”
Fricative/Plosive (manner of articulation)
Voiced
Unvoiced Fricatives/Plosives (6FPNVOZ)	6 short sentences
High intraoral pressure	(unvoiced fricatives/plosives)	“Fafá foi a feira, Cecília laçou o saci, A Xuxa achou o xale; O piupiu piou, O tatu é da Talita, A Cuca correu e caiu”
Fricative/Plosive (manner of articulation)
Unvoiced
Liquids (LIQ)	4 short sentences
Low intraoral pressure	(2 sentences in repetition)	“Lalá olhou a lua; Rui é o rei”; Lalá olhou a lua; Rui é o rei”
Sonorant (manner of articulation)
Count (CONT)	.........	1 to 10
Oral/Nasal

Procedures

The recordings included in the study were retrieved from the database of the laboratory, saved and later edited. After excluding the audio speech record that corresponded to the interlocutor's participation in all recordings, the sets of sentences corresponding to each of the nine speech stimuli of interest were edited separately. When editing short sentences with predominantly low pressure (liquid) consonants, it was decided to consecutively repeat the two liquid sentences available in the database, totaling four short sentences. In all editions, an interval of 1 second was standardized between the sentences in each set. After the edits, the speech samples corresponding to each speech stimulus were numbered and copied at random on a flash drive, in 9 separate folders. Besides, 20% of the total samples were inserted in each of the 9 folders, for further analysis of the intra-rater reliability index.

Auditory-perceptual analysis of speech hypernasality

A prospective analysis of the selected speech samples was carried out by three speech-language pathologists with at least five-year experience in evaluating the speech of individuals with CLP and/or VPD. The evaluators analyzed, according to their criteria, the speech samples recorded, individually, using headphones of the type K414P. They were instructed to listen to the samples as many times as they deemed necessary for their analysis and, also, to perform a 5-minute rest every 20 minutes of analysis. The speech-language pathologists received material for evaluation composed of 9 audio files (wave) corresponding to the speech samples constituted by the 9 speech stimuli of interest. Each evaluator was instructed to identify the presence or absence of hypernasality in the 96 speech samples (80 for analysis and 16 for intra-evaluator reliability), of each of the 9 stimuli, being able to insert information about the presence or absence of hypernasality or other concomitant speech aspects, if desired. Thus, each evaluator analyzed 720 speech samples (80 recordings x 9 stimuli = 720), in addition to 20% of this total (144 samples) for intra-evaluator reliability analysis.

Analysis of results

The results of identifying the occurrence of speech hypernasality were analyzed considering the binary scale in which 1 represents the absence of hypernasality and 2 the presence of hypernasality. The inter and intra-rater reliability index was established for the 9 types of speech stimuli, using the Kappa coefficient. The results were interpreted, according to Landis and Koch⁽²²⁾: below 0 = without reliability; 0 to 0.19 = poor reliability; from 0.20 to 0.39 = regular reliability; from 0.40 to 0.59 = moderate reliability; from 0.60 to 0.79 = substantial reliability; from 0.80 to 1.00 = almost perfect/perfect reliability. The comparison between the inter and intra-rater reliability indexes for the speech samples was made using the Z test. P <0.05 were accepted as significant.

RESULTS

Intra-rater reliability

The intra-rater reliability coefficients regarding the presence and absence of hypernasality, obtained in the analysis of each of the nine types of speech stimuli, are shown in Table 1.

Table 1
Intra-rater reliability in the perceptual analysis of hypernasality for the 9 types of speech stimuli: agreement percentage (%), Kappa coefficients (K) and its interpretation

	EV1			EV2			EV3
	%	K	Interpretation	%	K	Interpretation	%	K	Interpretation
12FRIPLO	93.75	0.86	almost perfect	100	1.00	perfect	93.75	0.88	almost perfect
3FVOZ	100	1.00	perfect	93.75	0.85	almost perfect	75	0.47	0, 47 moderate
3FNVOZ	87.50	0.71	substantial	81.25	0.54	moderate	87.50	0.75	substantial
3PVOZ	87.50	0.75	substantial	75	0.20	regular	87.50	0.75	substantial
3PNVOZ	100	1.00	perfect	75	0.47	moderate	93.75	0.87	almost perfect
6FPVOZ	100	1.00	perfect	100	1.00	perfect	50	0.04	poor
6FPNVOZ	100	1.00	perfect	75	0.46	moderate	100	1.00	perfect
LIQ	87.50	0.75	substantial	100	1.00	perfect	87.50	0.75	substantial
CONT	93.75	0.86	almost perfect	93.75	0.64	substantial	81.25	0.59	moderate

• Caption: EV1=Evaluator 1; EV2= Evaluator 2; EV3= Evaluator 3; 12FRIPLO=12 short sentences fricative/plosives; 3FVOZ=3 short sentences voiced fricatives; 3FNVOZ=3 short sentences unvoiced fricatives; 3PVOZ=3 short sentences voiced plosives; 3PNVOZ=3 short sentences unvoiced plosives; 6FPVOZ=6 short sentences voiced fricative/plosives; 6FPNVOZ=6 short sentences unvoiced fricative/plosives; LIQ=liquids; CONT=counting; % = percentage value; K = Kappa coefficient

For evaluator 1, there was no significant difference among the Kappa values ​​of the nine studied stimuli. For evaluator 2, there was a significant difference between the Kappa index of 0.20 (interpreted as regular) obtained for 3PVOZ and the Kappa index of 1.0 (interpreted as perfect), obtained for the following stimuli: 12FRIPLO (p = 0.018), 6FPVOZ (p = 0.018) and LIQ (p = 0.018) (Table 2). These data suggest that the stimulus of three stop short sentences (3PVOZ) disfavored the analysis by this evaluator, when compared to the findings of the stimuli of greater extension (12FRIPLO and 6FPVOZ) and, also, of low-pressure (LIQ). For evaluator 3, there was a significant difference between the Kappa index of 0.04 (interpreted as poor) obtained for 6FPVOZ and the Kappa index for the other stimuli (interpreted as almost perfect or perfect), with two exceptions: 3FVOZ and counting. These data suggest that the voiced stimulus (6FPVOZ) disfavored the analysis of hypernasality by this evaluator.

Table 2
Statistical comparison between the intra-rater reliability indexes in the perceptual analysis of hypernasality for the 9 types of speech stimuli: evaluator 2 and evaluator 3

		12FRIPLO	3FVOZ	3FNVOZ	3PVOZ	3PNVOZ	6FPVOZ	6FPNVOZ	LIQ	CONT
12FRIPLO	EV2	-	-	-	-	-	-	-	-	-
	EV3	-	-	-	-	-	-	-	-	-
3FVOZ	EV2	p=0.670	-	-	-	-	-	-	-	-
	EV3	p=0.405	-	-	-	-	-	-	-	-
3FNVOZ	EV2	p=0.426	p=0.375	-	-	-	-	-	-	-
	EV3	p=0.244	p=0.421	-	-	-	-	-	-	-
3PVOZ	EV2	p=0.018*	p=0.054	p=0.313	-	-	-	-	-	-
	EV3	p=0.708	p=0.421	p=1.0	-	-	-	-	-	-
3PNVOZ	EV2	p=0.134	p=0.280	p=0.842	p=0.426	-	-	-	-	-
	EV3	p=0.977	p=0.277	p=0.729	p=0.729	-	-	-	-	-
6FPVOZ	EV2	p=1.0	p=0.670	p=0.191	p=0.018*	p=0.134	-	-	-	-
	EV3	p=0.011*	p=0.197	p=0.030 *	p=0.030*	p=0.012*	-	-	-	-
6FPNVOZ	EV2	p=0.098	p=0.229	p=0.805	p=0.403	p=0.976	p=0.098	-	-	-
	EV3	p=0.733	p=0.134	p=1.0	p=0.473	p=0.712	p=0.004*	-	-	-
LIQ	EV2	p=1.000	p=0.670	p=0.191	p=0.018*	p=0.134	p=1.000	p=0.098	-	-
	EV3	p=0.712	p=0.428	p=1.0	p=1.0	p=0.733	p=0.032*	p=0.480	-	-
CONT	EV2	p=0.292	p=0.536	p=0.768	p=0.178	p=0.619	p=1.0	p=0.566	p=0.292	-
	EV3	p=0.408	p=0.733	p=0.643	p=0.644	p=0.424	p=0.097	p=0.244	p=0.649	-

• Caption: EV2= Evaluator 2; EV3= Evaluator 3; 12FRIPLO=12 short sentences fricative/plosives; 3FVOZ=3 short sentences voiced fricatives; 3FNVOZ=3 short sentences unvoiced fricatives; 3PVOZ=3 short sentences voiced plosives; 3PNVOZ=3 short sentences unvoiced plosives; 6FPVOZ=6 short sentences voiced fricative/plosives; 6FPNVOZ=6 short sentences unvoiced fricative/plosives; LIQ=liquids; CONT=counting; % = percentage value.
• ^*
  p value (Z test)

Interrater reliability

The interrater reliability coefficients for the 9 types of speech stimuli analyzed are shown in Table 3. There were lower rates of Kappa coefficient among the three evaluators, together, for the following speech stimuli: 6FPVOZ (Kappa = 0.24), 3FVOZ (Kappa = 0.37) and LIQ (Kappa = 0.31) (regular reliability), suggesting that the voiced component of high-pressure and low-pressure stimuli (liquid short sentences) disfavored the identification of hypernasality among the three evaluators. The highest Kappa coefficient index for the three evaluators was 0.47 (12FRIPLO), interpreted as moderate. In general, the lowest Kappa coefficient index was 0.11 (6FPVOZ), obtained for evaluators 2 and 3 and the highest Kappa coefficient index was 0.57 (12FRIPLO).

Table 3
Inter-rater reliability in the perceptual analysis of hypernasality for the 9 types of speech stimuli: agreement percentage (%), Kappa coefficients (K) and its interpretation

	EV1 and EV2			EV1 and EV3			EV2 and EV3			EV1, EV2, EV3
	%	K	Interpretation	%	K	Interpretation	%	K	Interpretation	K	Interpretation
12FRIPLO	77.50	0.40	moderate	81.25	0.57	moderate	73.75	0.37	regular	0.47	moderate
3FVOZ	73.75	0.27	regular	76.25	0.46	moderate	75.00	0.38	regular	0.37	regular
3FNVOZ	73.75	0.37	regular	81.25	0.55	moderate	75.00	0.44	moderate	0.45	moderate
3PVOZ	77.50	0.41	moderate	73.75	0.45	moderate	76.25	0.49	moderate	0.45	moderate
3PNVOZ	78.75	0.48	moderate	75.00	0.42	moderate	73.75	0.43	moderate	0.44	moderate
6FPVOZ	77.50	0.44	moderate	61.25	0.17	poor	58.75	0.11	poor	0.24	regular
6FPNVOZ	72.50	0.34	regular	77.50	0.52	moderate	70.00	0.33	regular	0.40	moderate
LIQ	63.75	0.21	regular	77.50	0.56	moderate	51.25	0.16	poor	0.31	regular
CONT	73.75	0.33	regular	81.25	0.56	moderate	75.00	0.43	moderate	0.44	moderate

• Caption: EV1= Evaluator 1; EV2= Evaluator 2; EV3= Evaluator 3; 12FRIPLO=12 short sentences fricative/plosives; 3FVOZ=3 short sentences voiced fricatives; 3FNVOZ=3 short sentences unvoiced fricatives; 3PVOZ=3 short sentences voiced plosives; 3PNVOZ=3 short sentences unvoiced plosives; 6FPVOZ=6 short sentences voiced fricative/plosives; 6FPNVOZ=6 short sentences unvoiced fricative/plosives; LIQ=liquids; CONT=counting; % = percentage value; K = Kappa coefficient

Analyzing the reliability coefficient between the pairs of evaluators separately, it was found, as shown in Table 3, that between the evaluators 1 and 2, the Kappa indexes ranged from 0.21 (interpreted as regular) (LIQ) to 0.48 (interpreted as moderate) obtained for 3PNVOZ, with a statistically significant difference (p = 0.04) only between the Kappa coefficients of these two speech stimuli.

Between evaluators 1 and 3, the Kappa indexes varied from 0.17 (interpreted as poor) obtained for 6FPVOZ to 0.57 (interpreted as moderate) obtained for 12FRIPLO (Table 3). There was a significant difference in the Kappa index of 0.17 (6FPVOZ) and the following Kappa indexes: 0.57 (12FRIPLO; p = 0.009); 0.55 (3FNVOZ; p = 0.013), 0.52 (6FPNVOZ; p = 0.024) and 0.56 (LIQ; p = 0.001), all interpreted as moderate. There was also a significant difference in the Kappa index of 0.17 (6FPVOZ) and Kappa index of 0.56 (interpreted as moderate) obtained for CONT (p = 0.005) (Table 4). The data suggest that the stimulus consisting of 6 exclusively voiced short sentences disfavored the identification of hypernasality among these evaluators when compared to the findings of the most extensive stimuli (12 short sentences, one with each pressure consonant) and unvoiced (3 FNVOZ and 6FPNVOZ) and also with the findings of counting numbers and liquid stimuli (low-pressure).

Table 4
Statistical comparison between the inter-rater reliability indexes in the perceptual analysis of hypernasality for the 9 types of speech stimuli: evaluator 1 and evaluator 3

EV1 X EV3	12FRIPLO	3FVOZ	3FNVOZ	3PVOZ	3PNVOZ	6FPVOZ	6FPNVOZ	LIQ	CONT
12FRIPLO	-	-	-	-	-	-	-	-	-
3FVOZ	p=0.482	-	-	-	-	-	-	-	-
3FNVOZ	p=0.898	p=0.563	-	-	-	-	-	-	-
3PVOZ	p=0.436	p=0.948	p=0.513	-	-	-	-	-	-
3PNVOZ	p=0.325	p=0.793	p=0.391	p=0.841	-	-	-	-	-
6FPVOZ	p=0.009 *	p=0.060	p=0.013*	p=0.064	p=0.095	-	-	-	-
6FPNVOZ	p=0.750	p=0.703	p=0.847	p=0.650	p=0.513	p=0.024*	-	-	-
LIQ	p=0.948	p=0.516	p=0.948	p=0.467	p=0.350	p=0.001*	p=0.796	-	-
CONT	p=0.848	p=0.371	p=0.748	p=0.330	p=0.238	p=0.005*	p=0.612	p=0.795	-

• Caption: EV1= Evaluator 1; EV3= Evaluator 3; 12FRIPLO=12 short sentences fricative/plosives; 3FVOZ=3 short sentences voiced fricatives; 3FNVOZ=3 short sentences unvoiced fricatives; 3PVOZ=3 short sentences voiced plosives; 3PNVOZ=3 short sentences unvoiced plosives; 6FPVOZ=6 short sentences voiced fricative/plosives; 6FPNVOZ=6 short sentences unvoiced fricative/plosives; LIQ=liquids; CONT=counting; % = percentage value.
• ^*
  p value (Z test)

For evaluators 2 and 3, the Kappa indexes varied from 0.11 (interpreted as poor) obtained for 6FPVOZ to 0.49 (interpreted as moderate) obtained for 3PVOZ (Table 5). There was a significant difference between the Kappa coefficient of 0.11 (6FPVOZ) and the following Kappa indexes: 0.043 (3PNVOZ; p = 0.039), 0.44 (3FNVOZ; p = 0.036), 0.49 (3PVOZ; p = 0.007) and 0.43 (CONT; p = 0.024), all interpreted as moderate. The data suggest that the stimulus constituted by the set of 6 exclusively voiced short sentenes (6FPVOZ) favored the identification of hypernasality among these evaluators, particularly when compared with the findings obtained for unvoiced stimuli and number counting. For these evaluators, there was also a significant difference in the Kappa index of 0.16 (interpreted as poor) obtained for LIQ and the following Kappa indexes: 0.49 (3PVOZ; p = 0.004), 0.43 (3PNVOZ; p = 0.036), 0.44 (3FNVOZ; p = 0.034) and 0.43 (CONT; p = 0.017), all interpreted as moderate (Table 5). These findings suggest that the low-pressure stimulus also disfavored the identification of hypernasality for these evaluators.

Table 5
Statistical comparison between the inter-rater reliability indexes in the perceptual analysis of hypernasality for the 9 types of speech stimuli: evaluator 2 and evaluator 3

EV2 X EV3	12FRIPLO	3FVOZ	3FNVOZ	3PVOZ	3PNVOZ	6FPVOZ	6FPNVOZ	LIQ	CONT
12FRIPLO	-	-	-	-	-	-	-	-	-
3FVOZ	p=0.945	-	-	-	-	-	-	-	-
3FNVOZ	p=0.658	p=0.699	-	-	-	-	-	-	-
3PVOZ	p=0.400	p=0.429	p=0.742	-	-	-	-	-	-
3PNVOZ	p=0.699	p=0.739	p=0.796	p=0.687	-	-	-	-	-
6FPVOZ	p=0.080	p=0.063	p=0.036 *	p=0.007*	p=0.039*	-	-	-	-
6FPNVOZ	p=0.786	p=0.728	p=0.482	p=0.255	p=0.515	p=0.134	-	-	-
LIQ	p=0.083	p=0.060	p=0.034*	p=0.004*	p=0.036*	p=0.678	p=0.153	-	-
CONT	p=0.673	p=0.719	p=0.948	p=0.658	p=1.0	p=0.024*	p=0.476	p=0.017*	-

• Caption: EV2= Evaluator 2; EV3= Evaluator 3; 12FRIPLO=12 short sentences fricative/plosives; 3FVOZ=3 short sentences voiced fricatives; 3FNVOZ=3 short sentences unvoiced fricatives; 3PVOZ=3 short sentences voiced plosives; 3PNVOZ=3 short sentences unvoiced plosives; 6FPVOZ=6 short sentences voiced fricative/plosives; 6FPNVOZ=6 short sentences unvoiced fricative/plosives; LIQ=liquids; CONT=counting; % = percentage value.
• ^*
  p value (Z test)

DISCUSSION

The study verified the influence of speech stimuli in the auditory-perceptual assessment of the occurrence of hypernasality in patients with CLP operated without and with VPD. More specifically, this study sought to investigate which speech stimuli, concerning the phonetic context and/or its extension (number of short sentences), could be used to favor the documentation of the clinical findings of the speech aspects of this population, to make the identification of the most reliable hypernasality, regarding the intra- and inter-rater reliability coefficients. In general, the results showed that the consistency of the hypernasality assessment for the same examiner may vary depending on the speech stimulus, with a higher reliability index obtained, for each examiner, for the most extensive high-pressure stimulus (12 short sentences, one with each pressure consonant).

It is speculated that short sentences consisting of 12 high-pressure oral consonants may favor the perceptual assessment of speech hypernasality by the same evaluator, since more extensive stimuli (greater number of short sentnences) may provide information to the listener for a longer time and, therefore, enable him to be more consistent in his responses, even though there is a risk that he is exposed concomitantly to aspects unrelated to nasality (for example, distortion caused by the escape of audible nasal air, nasal turbulence, use of compensatory articulations, dento-occlusal distortions or dysphonia). In one study⁽¹⁵⁾ the listener's reliability in assessing speech nasality was higher for more extensive than short stimuli (sentences greater than isolated words, and isolated words greater than isolated vowels). The authors suggested that the acoustic cues of the consonants that precede or precede the vowels may have favored the assessment of hypernasality in sentences and words by the listeners when compared to that performed for the vowels alone. However, they do not report whether the phonetic constitution of these stimuli influenced the listeners' reliability in the analysis of hypernasality.

In this study, lower indexes of intra-rater reliability were obtained for stimuli with voiced high-pressure consonants and short in extension (three voiced plosive short sentences, evaluator 2 and six voiced fricative/stop short sentences, evaluator 3). Particularly, for evaluator 3, there was a significant difference between the reliability indexes obtained for the six voiced short sentences and almost all other exclusively oral stimuli, except for the three voiced fricative short sentences, pointing at a possible influence of the voiced component, in more extensive stimuli exclusively voiced, in the analyzes of this evaluator.

Speech resonance is a complex acoustic phenomenon in which the sound energy generated by the vibration of the vowel folds is directed upwards in the vocal tract and will vibrate through the resonance cavities (pharyngeal, oral and/or nasal)⁽²⁾. The relative balance of sound energy vibration in the resonance cavities will determine whether the speech quality will be perceived as normal or altered. When there is an excess of nasal resonance during the production of oral sounds due to the abnormal coupling of the oral and nasal resonance cavities, the listener perceives an excessive nasality (or hypernasality)^(2,4) particularly in the vowels (as they have a longer duration)⁽⁴⁾ and in the voiced consonants^(2,4). The findings of the study, therefore, suggest that the composition of the stimuli (voiced) may impair the consistency of the auditory-perceptual assessment of hypernasality by an evaluator.

External variables can influence the auditory-perceptual assessment of speech hypernasality and, among them, stand out the atypical articulation patterns, the escape of audible air/nasal turbulence that is commonly observed in high-pressure oral consonants^(4,16). In the study, the presence of adverse speech conditions was not a controlled variable. However, the intra-evaluator reliability of evaluators 2 and 3 was favored for the liquid stimulus (low-pressure), when compared to findings of the high-pressure stimulus consisting of voiced consonants (three voiced plosive short sentences, evaluator 2 and six fricative/voiced plosives short sentences, evaluator 3), suggesting that the presence of possible adverse speech conditions (compensatory articulation, for example) may have impaired the identification of hypernasality by these evaluators, particularly when the speech stimulus was voiced.

The identification of speech hypernasality is a challenging task, even when performed by experienced listeners⁽⁷⁾. The internal criteria that one evaluator uses in his analyzes differ from the other⁽⁶⁾ and may be unstable for the same evaluator, regardless of the level of experience⁽²³⁾. These criteria can be influenced by internal factors (lapses in memory, attention) and external variables, including the speech stimulus used to capture the samples to be evaluated^(5,18). The use of specific criteria may explain differences in the intra-rater reliability indexes obtained for the three evaluators in the study. For evaluator 1, there was no significant difference in the reliability rates for the nine types of speech stimuli and her reliability rate ranged from substantial to perfect. Except for this evaluator, for the two others, the indexes varied from regular to perfect and from poor to perfect for the 9 speech stimuli, with the variability of the findings attributed, at least in part, to the constitution of the speech stimuli.

When considering the findings of the three evaluators, moderate to perfect intra-evaluator reliability indexes were found for most of the investigated stimuli, with two exceptions: regular reliability index for the 3 voiced plosive short sentences (evaluator 2) and poor reliability index for the six voiced fricative/plosive short sentences (evaluator 3). These expressive indexes of intra-evaluator reliability obtained for most of the stimuli analyzed corroborate previous investigations that also indicated expressive indexes of intra-evaluator reliability, when classifying speech hypernasality, using repetition of sentences consisting of high-pressure oral sounds^(5,21,24) or by the combination of high and low-pressure sounds⁽²⁵⁾. Other studies have also indicated discrete to almost perfect indexes of intra-evaluator, when classifying speech hypernasality, in speech samples constituted by the repetition of sentences combining high and low-pressure oral consonants⁽¹⁸⁾ or, even, from regular to good for more extensive samples (counting from 1 to 10 followed by the repetition of high-pressure oral sentences) analyzed before training⁽⁷⁾.

The repetition of sentences is, therefore, commonly reported in studies that verified indexes of intra-evaluator reliability of speech hypernasality^{(11,18,21,25)}. This speech stimulus favors the perceptual analysis of the speech by the evaluator as it constitutes a precise type of speech sample⁽²¹⁾. In the literature, some studies report that the repetition of sentences favored the reliability of the auditory-perceptual assessment of hypernasality of the same evaluator when compared with intra-evaluator reliability indexes obtained for spontaneous conversation⁽²¹⁾. In another study, good intra-rater reliability indexes were reported for the repetition of sentences, with regular reliability for spontaneous conversation only for one of four evaluators⁽⁵⁾. The use of repetition of standardized stimuli has been recommended for the documentation of speech results (including hypernasality) of individuals with CLP⁽¹⁶⁾. However, no previous study has attempted to verify whether the use of a sentence with specific contexts could influence such documentation.

The auditory-perceptual assessment of speech is considered a “gold standard” procedure for assessing the speech of individuals with CLP⁽¹⁾. However, several factors can influence the reliability among evaluators in the identification of hypernasality, including the selected speech stimulus^(2,5,18). In the present study, when considering the findings of the three evaluators, lower (regular) indexes of reliability were found for the voiced stimuli (three voiced fricative short sentences and six voiced fricative/plosive short sentences), pointing out that the voiced component of these stimuli may have influenced the evaluators' analyzes.

The statistical analysis between the pairs of evaluators showed a significant difference between the reliability index of the stimulus 6 voiced short sentences and the indexes of other stimuli (unvoiced short sentences, set of 12 high-pressure short sentences, liquid short sentences and count) for the evaluators 1 and 3, suggesting that the voicing present in the 6 plosive/fricative short sentences disfavored the reliability of the auditory-perceptual analysis of these evaluators. When analyzing the findings obtained between evaluators 2 and 3, a lower Kappa index was also observed for the 6 voiced fricative/plosive short sentences concerning unvoiced stimuli (3 unvoiced plosive short sentences and 3 unvoiced fricative short sentences) and also to short plosive voiced stimulus (3 voiced plosive short sentences). Based on these results, it is speculated that more extensive voiced oral pressure consonants may impair auditory-perceptual analysis of hyponasality among evaluators. The influence of voicing on oral consonants to determine speech hypernasality has not been explored and, in general, it is recommended to use the set of high-pressure oral speech samples (voiced/unvoiced) to assess speech characteristics of individuals with CLP and/or VPD^(16,26).

In the study, lower (regular) indexes of reliability between the three evaluators were also found for the liquid stimulus (low intraoral pressure). The statistical analysis between the pairs of evaluators showed that the low pressure (liquid) speech stimulus disfavored the identification of hypernasality for pairs of evaluators 1 and 2 and, also, of evaluators 2 and 3. It is speculated that speech stimulus with low intra-oral pressure consonants may impair auditory-perceptual analysis of hypernasality among evaluators. As they do not involve plosion and friction, these productions can minimize the possible impact of adverse conditions (use of compensatory articulations such as glottal stop, for example) and, consequently, impair the perception of hypernasality in sounds that move acoustic energy with less intraoral pressure (and also lower intranasal pressure in cases of oronasal coupling), as is the case with liquid sounds. In a previous study⁽¹⁸⁾, a regular inter-rater reliability index was obtained for speech samples consisting of low-pressure consonants.

The literature reports regular (pre-training) and moderate (post-training) reliability levels in the classification of hypernasality among evaluators for high-pressure oral stimuli of greater extent (sections containing counts from 1 to 10 and repetition of sentences with plosive and fricatives)⁽⁷⁾. The authors justify the difficulty in obtaining a high level of reliability in the perceptual assessment of hypernasality between different evaluators due to the subjective nature of this evaluation and, particularly, the procedure used (4-point scale) to graduate this aspect of speech⁽⁷⁾. In another study, however, a moderate reliability index was also obtained among evaluators in the assessment of hypernasality for short stimulus, using a binary scale (presence and absence)⁽²⁷⁾. Based on the findings of the present study, it is suggested to choose a more extensive set of short sentences, involving all fricative and plosive consonants in the assessment of hypernasality, to avoid the selection of speech stimuli that may impair the reliability of responses among evaluators.

The focus of the present study was to verify if the evaluators' reliability would be, in some way, affected by the selected speech stimuli. The findings, in general, suggest that the type of speech stimulus may influence the reliability of the analyzes between different evaluators. A limitation of the study concerns the vowel context lack of control of the stimuli selected for study comparisons. The literature reports that listeners tend to perceive high vowels as more nasalized than low vowels in the speech of individuals with hypernasality⁽²⁸⁾. In future studies, it is suggested to control the vowel context by using different sentences to verify the influence of these sentences in the identification of speech hypernasality.

CONCLUSION

The speech stimulus influenced the reliability of the auditory-perceptual evaluation for the identification of hypernasality since the intra-evaluator reliability in the analyzes was, in general, lower for exclusively voiced stimuli and the reliability among different evaluators was lower for high-pressure voiced short sentences and low-pressure short sentences. A higher index of reliability among the three speech-language pathologists was obtained for the set of 12 short sentences, one with each pressure consonant, suggesting that longer oral stimuli may favor inter-rater reliability in the identification of hypernasality.

• Study conducted at Hospital de Reabilitação de Anomalias Craniofaciais, Universidade de São Paulo – USP, Bauru (SP), Brasil.
• Financial support: nothing to declare.

REFERÊNCIAS

• ¹
  Kuehn DP, Moller KT. Speech and language issues in the cleft palate population: the state of the art. Cleft Palate Craniofac J. 2000;37(4):1-35. http://dx.doi.org/10.1597/1545-1569_2000_037_0348_saliit_2.3.co_2
  » http://dx.doi.org/10.1597/1545-1569_2000_037_0348_saliit_2.3.co_2
• ²
  Zajac DJ, Vallino LD. Evaluation and management of cleft lip and palate: a developmental perspective. San Diego: Plural Publishing; 2017. Speech and resonance characteristics; p. 193-226.
• ³
  Baylis A, Chapman K, Whitehill TL. Validity and reliability of visual analog scaling for assessment of hipernasality and audible nasal emission in children with repaired cleft palate. Cleft Palate Craniofac J. 2015;52(6):660-70. http://dx.doi.org/10.1597/14-040 PMid:25322442.
  » http://dx.doi.org/10.1597/14-040
• ⁴
  Kummer AW. Speech evaluation for patients with cleft palate. Clin Plast Surg. 2014;41(2):241-51. http://dx.doi.org/10.1016/j.cps.2013.12.004 PMid:24607192.
  » http://dx.doi.org/10.1016/j.cps.2013.12.004
• ⁵
  Bettens K, De Bodt M, Maryn Y, Luyten A, Wuyts FL, Van Lierde KM. The relationship between the Nasality Severity Index 2.0 and perceptual judgments of hypernasality. J Commun Disord. 2016;62:67-81. http://dx.doi.org/10.1016/j.jcomdis.2016.05.011 PMid:27310727.
  » http://dx.doi.org/10.1016/j.jcomdis.2016.05.011
• ⁶
  Kreiman J, Gerratt BR, Kempster GB, Erman A, Berke GS. Perceptual evaluation of voice quality: review, tutorial, and a framework for future research. J Speech Hear Res. 1993;36(1):21-40. http://dx.doi.org/10.1044/jshr.3601.21 PMid:8450660.
  » http://dx.doi.org/10.1044/jshr.3601.21
• ⁷
  Oliveira ACASF, Scarmagnani RH, Fukushiro AP, Yamashita RP. The influence of listener training on the perceptual assessment of hypernasality. CoDAS. 2016;28(2):141-8. http://dx.doi.org/10.1590/2317-1782/20162015163 PMid:27191877.
  » http://dx.doi.org/10.1590/2317-1782/20162015163
• ⁸
  Kent RD. Hearing and believing some limits to the auditory-perceptual assessment of speech and voice disorders. Am J Speech Lang Pathol. 1996;5(3):7-23. http://dx.doi.org/10.1044/1058-0360.0503.07
  » http://dx.doi.org/10.1044/1058-0360.0503.07
• ⁹
  Lee A, Whitehill TL, Ciocca V. Effect of listener training on perceptual judgement of hypernasality. Clin Linguist Phon. 2009;23(5):319-34. http://dx.doi.org/10.1080/02699200802688596 PMid:19399664.
  » http://dx.doi.org/10.1080/02699200802688596
• ¹⁰
  Chapman KL, Baylis A, Trost-Cardamone J, Cordero KN, Dixon A, Dobbelsteyn C, et al. The Americleft Speech Project: a training and reliability study. Cleft Palate Craniofac J. 2016;53(1):93-108. http://dx.doi.org/10.1597/14-027 PMid:25531738.
  » http://dx.doi.org/10.1597/14-027
• ¹¹
  Padilha EZ, Dutka JCR, Marino VCC, Lauris JRP, Silva MJF, Pegoraro-Krook MI. Assessment of speech nasality in individuals with cleft palate. Audiol Commun Res. 2015;20(1):48-55. http://dx.doi.org/10.1590/S2317-64312015000100001444
  » http://dx.doi.org/10.1590/S2317-64312015000100001444
• ¹²
  Castick S, Knight RA, Sell D. Perceptual judgments of resonance, nasal airflow, understandability, and acceptability in speakers with cleft palate: ordinal versus visual analogue scaling. Cleft Palate Craniofac J. 2017;54(1):19-31. http://dx.doi.org/10.1597/15-164 PMid:28067575.
  » http://dx.doi.org/10.1597/15-164
• ¹³
  Bettens K, Bruneel L, Maryn Y, De Bodt M, Luyten A, Van Lierde KM. Perceptual evaluation of hypernasality, audible nasal airflow and speech understandability using ordinal and visual analogue scaling and their relation with nasalance scores. J Commun Disord. 2018;76:11-20. http://dx.doi.org/10.1016/j.jcomdis.2018.07.002 PMid:30071470.
  » http://dx.doi.org/10.1016/j.jcomdis.2018.07.002
• ¹⁴
  Yamashita RP, Borg E, Granqvist S, Lohmander A. Reliability of hypernasality rating: comparison of 3 different methods for perceptual assessment. Cleft Palate Craniofac J. 2018;55(8):1060-71. http://dx.doi.org/10.1177/1055665618767116 PMid:29634363.
  » http://dx.doi.org/10.1177/1055665618767116
• ¹⁵
  Counihan DT, Cullinan WL. Reliability and dispersion of nasality ratings. Cleft Palate J. 1970;7(1):261-70. PMid:5266336.
• ¹⁶
  Henningsson G, Kuehn DP, Sell D, Sweeney T, Trost-Cardamone JE, Whitehill TL. Universal parameters for reporting speech outcomes in individuals with cleft palate. Cleft Palate Craniofac J. 2008;45(1):1-17. http://dx.doi.org/10.1597/06-086.1 PMid:18215095.
  » http://dx.doi.org/10.1597/06-086.1
• ¹⁷
  Glade RS, Deal R. Diagnosis and management of velopharyngeal dysfunction. Oral Maxillofac Surg Clin North Am. 2016;28(2):181-8. http://dx.doi.org/10.1016/j.coms.2015.12.004 PMid:27150305.
  » http://dx.doi.org/10.1016/j.coms.2015.12.004
• ¹⁸
  Ferlin F, Yamashita RP, Fukushiro AP. Influence of high and low intraoral pressure consonants on the speech nasality and nasalance in patients with repaired cleft palate. Audiol Commun Res. 2017;22:e1851.
• ¹⁹
  Lohmander A, Persson C, Willadsen E, Lundeborg I, Alaluusua S, Aukner R, et al. Scandcleft randomised trials of primary surgery for unilateral cleft lip and palate: 4. Speech outcomes in 5-year-olds - velopharyngeal competency and hypernasality. J Plast Surg Hand Surg. 2017;51(1):27-37. http://dx.doi.org/10.1080/2000656X.2016.1254645 PMid:28218551.
  » http://dx.doi.org/10.1080/2000656X.2016.1254645
• ²⁰
  Lee A, Potts S, Bressmann T. Speech-language therapy students’ auditory-perceptual judgements of simulated concurrent hypernasality and articulation disorders. Clin Linguist Phon. 2020;34(5):479-92. http://dx.doi.org/10.1080/02699206.2019.1655666 PMid:31429313.
  » http://dx.doi.org/10.1080/02699206.2019.1655666
• ²¹
  Medeiros MNL, Fukushiro AP, Yamashita RP. Influence of speech sample on perceptual rating of hypernasality. CoDAS. 2016;28(3):289-94. http://dx.doi.org/10.1590/2317-1782/20162015202 PMid:27409419.
  » http://dx.doi.org/10.1590/2317-1782/20162015202
• ²²
  Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159-74. http://dx.doi.org/10.2307/2529310 PMid:843571.
  » http://dx.doi.org/10.2307/2529310
• ²³
  Chan KM, Yiu EM. A comparison of two perceptual voice evaluation training programs for naive listeners. J Voice. 2006;20(2):229-41. http://dx.doi.org/10.1016/j.jvoice.2005.03.007 PMid:16139475.
  » http://dx.doi.org/10.1016/j.jvoice.2005.03.007
• ²⁴
  Scarmagnani RH, Barbosa DA, Fukushiro AP, Salgado MH, Trindade IET, Yamashita RP. Relationship between velopharyngeal closure, hypernasality, nasal air emission and nasal rustle in subjects with repaires cleft palate. CoDAS. 2015;27(3):267-72. http://dx.doi.org/10.1590/2317-1782/20152014145 PMid:26222944.
  » http://dx.doi.org/10.1590/2317-1782/20152014145
• ²⁵
  Waknis AP, Tak HRS, Kulkarni SP. Perceptual and instrumental analysis of hypernasality in children with repaired cleft palate. J Cleft Lip Palate Craniofacial Anomalies. 2016;3(2):67-72. http://dx.doi.org/10.4103/2348-2125.187508
  » http://dx.doi.org/10.4103/2348-2125.187508
• ²⁶
  Pegoraro-Krook MI, Marino VCC, Dutka JCR. Tratado de motricidade orofacial. 1. ed. São José dos Campos: Pulso; 2019. Avaliação das alterações de fala na fissura labiopalatina e disfunção velofaríngea; p. 695-706.
• ²⁷
  Prado-Oliveira R, Marques IL, Souza L, Souza-Brosco TV, Dutka JC. Assessment of speech nasality in children with Robin Sequence. CoDAS. 2015;27(1):51-7. http://dx.doi.org/10.1590/2317-1782/20152014055 PMid:25885197.
  » http://dx.doi.org/10.1590/2317-1782/20152014055
• ²⁸
  Lewis KE, Watterson T, Quint T. The effect of vowels on nasalance scores. Cleft Palate Craniofac J. 2000;37(6):584-9. http://dx.doi.org/10.1597/1545-1569_2000_037_0584_teovon_2.0.co_2 PMid:11108528.
  » http://dx.doi.org/10.1597/1545-1569_2000_037_0584_teovon_2.0.co_2

Publication Dates

• Publication in this collection
  11 Dec 2020
• Date of issue
  2020

History

• Received
  11 Dec 2019
• Accepted
  11 Feb 2020