EFFECT OF STANDARDIZING OBSERVATION AND RATING CRITERIA FOR THE FRONT CRAWL STROKE TECHNIQUE ON INTER- AND INTRA-RATER RELIABILITYABSTRACT
This study aimed to assess the effect of standardizing observation and rating criteria for the front crawl stroke technique on inter- and intra-rater reliability in order to propose an observation list. Two teachers and two competitive swimming coaches with different academic backgrounds were responsible for evaluating the front crawl stroke technique of 44 swimmers using video. Each swimmer completed a 25m course at a comfortable speed, having their displacement recorded in the lateral and frontal planes, both submerged and out of water, in the final 10 m of the pool. Throughout the recording, the swimmers did not breath laterally. Only the movement performed by the right arm was the raters’ focus of observation and was based on a checklist. After the shooting, the raters participated in three stages: in the first stage, there was an intervention to standardize the observation and rating criteria; inter-rater agreement was tested in the second stage, while intra-rater agreement was obtained in the third. In the second stage of the study, the agreement ranged from small to low (k=0.08 and k=0.38). In the third stage, some raters showed poor and low agreement (k=-0.13 and k=0.35), while others showed agreement between reasonable and virtually perfect (k=0.64 and k=0.87). From the results presented, it was concluded that the intervention proposal did not have the desired effect of standardizing the observation and rating criteria among the raters.
Keywords:
Swimming; Sport; Rating and Pedagogy
RESUMO
Esse estudo teve por objetivo avaliar o efeito da padronização dos critérios de observação e de avaliação da técnica da braçada do nado crawl na confiabilidade inter e intra avaliadores para uma proposta de lista de observação. Dois professores e dois treinadores de natação competitiva com formação acadêmica diferentes foram responsáveis por avaliar a técnica da braçada do nado crawl de 44 nadadores por meio de vídeo. Cada nadador realizou um percurso de 25m em velocidade confortável tendo seu deslocamento gravado nos planos lateral e frontal, ambos submersos e fora d’água, nos 10 m finais da piscina. No percurso de filmagem os nadadores não realizaram a respiração lateral. Somente o movimento realizado pelo braço direito foi foco de observação dos avaliadores e com base em uma lista de verificação. Após as filmagens os avaliadores participaram de três etapas: na primeira etapa houve uma intervenção para padronização dos critérios de observação e de avaliação. Na segunda etapa testou-se a concordância inter-avaliadores e, na terceira, a concordância intra-avaliador. Na segunda etapa do estudo a concordância variou de pequeno a baixa (k=0,08 e k=0,38). Na terceira etapa alguns avaliadores apresentaram concordância pobre e baixa (k=-0,13 e k=0,35), enquanto outros apresentaram concordância entre razoável e praticamente perfeita (k=0,64 e k=0,87). A partir dos resultados apresentados concluiu-se que a proposta de intervenção não surtiu o efeito desejado de padronização dos critérios de observação e avaliação entre os avaliadores.
Palavras-chave:
Natação; Esporte; Avaliação e Pedagogia
Introduction
Swimming techniques are influenced by the interaction of several components1), (2),(3. In this sense, evaluation procedures are important in controlling learning4),(5, besides being a frequent task for teachers. This evaluation occurs by means of one observing the movement performed by the learner, who, subsequently, is given suggestions for correction4),(6),(7. With this purpose, the literature suggests using checklists as a rating instrument8)-(11) for a better analysis of the main components of the movement and adequate judgment5. However, the checklist must be objective, valid, easy to use, and meet reliability requirements9),(11),(12.
Reliability refers to the condition for a rating or a measurement to present similar results on different occasions, as long as there is no interference with the objective of changing performance4),(11),(13. This is an important procedure that will ensure that the raters’ results are consistent11),(14. Despite its importance, previous studies evaluating swimming techniques did not achieve the desired success in all its aspects9),(11. This can happen because teachers’ observation and rating criteria are different and, though particular, seem to lack internal consistency9. In other words, a teacher does not use the same observation and rating criteria when analyzing a student twice in a row in a short period of time9.
Considering the abovementioned aspects, the present study is justified by the need to devise a standardization proposal intended to evaluate the swimming learning process. Such standardization can bring about significant levels of intra- and inter-rater agreement. This agreement is important in practice because, with similar rating criteria, the swimming learning process would be made easier. Swimming studies indicate that there is a tendency to find greater intra-rater reliability among teachers with more academic training and experience9),(15. However, these results are not completely satisfactory. In addition, throughout swimming learning, a student has classes with different teachers. Thus, it becomes imperative for procedures to be developed aiming at greater reliability regarding the observation and rating process among different teachers. After investigations on the subject, only one study9 was concerned with providing opportunities for interaction between evaluating teachers, discussing the adopted observation and rating criteria. However, this interaction was not systematized, and the hypothesis that it could improve inter- and intra-rater reliability was not fully confirmed.
In light of the foregoing, the objective of the present study was to assess the effect of standardizing observation and rating criteria for the front crawl stroke technique on inter- and intra-rater reliability. As a hypothesis, it was estimated that a pre-rating intervention, by enabling analysis and an organized and systematized discussion, would foster the standardization of technique observation and rating criteria by teachers.
Methods
Sample
Two swimming teachers and two competitive swimming coaches - referred to as raters, with different academic backgrounds and with a minimum of three years of experience teaching swimming -, were responsible for evaluating the front crawl stroke technique of 44 swimmers. Swimmers (9 to 11 years old) of both sexes were randomly selected; they were in the stage of learning/enhancing swimming techniques and on different levels of swimming skills. All individuals involved in the research (raters and swimmers) received oral and written instructions for voluntary participation in the present study, which was also approved by the Research Ethics Committee under number 4.578.750.
Procedures
The technical behavior of the swimmers’ stroke when performing the front crawl was observed by means of video recordings and following the qualitative movement analysis method16. The choice was to evaluate the stroke because it is considered the main propulsive agent of swimming3),(15. Each swimmer completed a 25-meter crawl swim at a comfortable speed; their displacements were recorded in the lateral plane (submerged and out of the water) and frontal plane (submerged and out of the water) in the final 10 m of the pool, in order to prevent the exit effect. The swimmers were instructed not to breath laterally during the recording for a standardization in swimming and prevention of possible technical changes resulting from this body action. For this purpose, markings on the bottom of the pool indicated the beginning and end of this distance.
The swimmers were filmed by sports cameras (GoPro Hero 8; 1080p at 240 frames per second), and the one used in the water was inside a diving box specifically designed for this purpose (GoPro) (0.5 m depth from the waterline). The four images were later digitized, synchronized and transformed into a single file for video editing using the Final Cut Pro software, version 10.4.9.
Only the movement performed by the swimmers’ right arm was used as an object of observation by the raters and was based on the same checklist, as already described in the literature9. The checklist used, considered as the one that presents the most important movements to be observed for the stroke action, is composed of 5 items describing the stroke phases and the main variations of how the movement can be performed9. The rater was free to watch the footage as many times as necessary. From the observation of the movement, it was indicated in which descriptor (A, B, C or D) the movement fit. Chart 1 presents the checklist used.
Checklist used by the raters to observe the front crawl stroke. The table presents the Items and their movement descriptors (MDs)
Intervention stages:
The study was divided into three stages, as described below.
1st Stage: intervention to standardize the criteria for observation and rating of the front crawl stroke by the teachers
One of the project’s researchers met remotely with the raters to discuss the criteria for observation and rating of the front crawl stroke. Three online meetings lasting about 60 minutes were held. In the first meeting, the crawl swimming technique was discussed, with an emphasis on the essential movements of the stroke. In this sense, there was: (1) a theoretical presentation with the purpose of providing a technical description of the swimming movement and discussing the procedures for the qualitative analysis of human movement, using textual elements and photographs, and (2) a presentation of the videos (in different observation planes) of swimmers on different levels of performance. At all times, the raters were free to intervene and ask questions. In the second meeting, (1) the technical checklist was presented (composed of five items), and the essential movements of the front crawl stroke were discussed through the presentation of the videos (in different observation planes) and (2) there was a discussion on how the computer resources would be used for a better observation of the movement, such as slow-speed viewing. The expectation for that meeting is that the raters would become familiar with the tool and develop similar observation criteria. In the third meeting, the raters had the opportunity to simultaneously watch three swimmers of different levels. Each observation and rating criterion was openly discussed among them. All meetings were held through video lessons.
2nd Stage: inter-rater reliability
Seven days after the end of the first stage, the raters received a link by email containing the videos of the swimmers identified by letters and had 30 days to fill out the observation form. The videos were transferred to each rater’s computer before observation and rating. Each rater used their own resources, criteria and methods to observe the movement. With the intention of preventing flaws in the observation process due to fatigue, a maximum of five swimmers were observed per day. The swimmers’ videos could be viewed as many times as the rater desired, but without contact with the other raters.
3rd Stage: intra-rater reliability
This stage consisted of assessing intra-rater reliability and was carried out 20 days after the end of the second stage. The raters received a link by email containing the videos of the swimmers identified by letters (different from those presented in the second stage) and had 30 days to re-evaluate them. The videos were transferred to their computers before observation and rating. Each rater used their own resources, criteria and methods to observe and rate the movement. With the intention of preventing flaws in the observation process due to fatigue, a maximum of five swimmers were observed per day. The swimmers’ videos could be viewed as many times as the rater desired, but without contact with the other raters.
Statistical analysis
The kinematic descriptors used in the present study can be classified as nominal. The level of inter-rater reliability (2nd stage) was tested using Fleiss’s Kappa Reliability Coefficient. On the other hand, the intra-rater reliability level (3rd stage) was tested using Cohen’s Kappa Reliability Coefficient. Results lower than zero represent poor reliability; between 0 and 0.20, small reliability; between 0.21 and 0.40, low reliability; between 0.41 and 0.60, moderate reliability; between 0.61 and 0.80, reasonable reliability, and above 0.81, a virtually perfect reliability((17)). Using Fleiss’ Kappa, it was also possible to obtain the level of inter-rater reliability for each of the kinematic descriptors. In all cases, a significance level of 95% was considered, and reliabilities above 0.41 (moderate) were considered relevant. The data were processed with the aid of IBM SPSS Statistics, version 25.
Results
Table 1 presents the results for inter-rater reliability. Its values are between small and low, ranging from 0.13 to 0.38 (p<0.05).
Regarding the intra-rater reliability results, it can be highlighted that rater A showed moderate reliability in questions 1 and 5 (k=0.56 and p<0.001). In their turn, rater B showed reasonable reliability in questions 1, 3 and 5 (k=0.65 and p<0.001; k=0.68 and p<0.001; k=0.65 and p<0.001). Rater C presented the best reliability indexes among the 4 raters: virtually perfect reliability in questions 1, 3, 4 and 5 (k=0.82 and p<0.001; k=0.87 and p<0.001; k=1 and p< 0.001; k= 0.92 and p<0.001. Finally, rater D showed moderate reliability in question 5 (k=0.49 and p<0.001) and reasonable reliability (k=0.64 and p<0.001) in question 2. The intra-rater reliability classification for the front crawl stroke rating items is presented in Chart 2.
Discussion
The present study aimed to assess the effect of standardizing observation and rating criteria for the front crawl stroke technique on inter and intra-rater reliability. Values classified as small and low (0.13 to 0.38; p<0.05) were obtained for inter-rater reliability. This result fell short of the expectation and is an indication that the proposed intervention did not have the desired effect on this aspect of reliability. In its turn, as for intra-rater reliability, there was a large variation among the raters. Some of them obtained a “virtually perfect” classification for the proposed items, while others, low reliability. What was possible to notice, in this case, was that the swimming coaches presented, in general, better intra-rater reliability than the teachers did. However, even so, intra-rater reliability needs to be improved.
The reliability findings obtained with the present study did not confirm the hypothesis that the intervention offered with the aim of standardizing the observation and rating criteria would result in high inter- and intra-rater reliability values. Therefore, it seems that the way in which the intervention was carried out - remotely and with video classes -, did not have the desired effect on the standardization of these criteria. One of the limitations that may have occurred in the intervention may be precisely the use of video lessons. Adopting them as a pedagogical resource maintains a unidirectional relationship of knowledge, that is, the researcher who shot them was responsible for passing on the information. As much as studies show the benefits of video lessons aimed at training professionals18),(19, the present one did not achieve the expected success using this tool. This does not mean that the proposal is invalid, but that it needs to be improved.
The results found suggest the hypothesis that the opportunity for a discussion among the raters, with the latter reflecting on and presenting their own observation and rating criteria - which are oftentimes not clear to themselves even (as per the intra-rater reliability results), followed by guidance as to the standardization of these criteria, would be positive. Thus, interaction and communication among raters is important for the training and teaching process to be implemented20. Moreover, the use of different learning tools during the video lesson (such as animations, charts, and links to articles or texts based on the chosen theme) can contribute positively19. In this way, for other studies addressing this topic, the use of these strategies is suggested. It is thus believed that the intervention process would be better structured and could result in significant inter and intra-rater reliability values. In any case, the offer of four possibilities for performing the movement makes it more difficult for random agreement to occur, and this factor must be considered when interpreting reliability results.
In a study conducted by Wizer et al.11, which can be used for comparison with the second stage of the present study, inter-rater reliability ranged from poor to moderate. Said study evaluated the water skills of children during the phase of adaptation to the aquatic environment. Intra-rater reliability, in its turn, ranged from substantial to perfect. This reliability was lower for tasks that were rarely performed at this level of learning, mainly because they required greater confidence to be performed. In this way, the children may have presented different movement patterns, making the evaluation difficult due to inconsistency in the execution of motor tasks. Despite the reliability results presented, it is believed that, for this level of learning, the teacher has even greater difficulty establishing criteria and the internal standards of the movement.
On the other hand, Vidal et al.10 found reasonable reliability, in the inter-rater stage, when evaluating several technical components of crawl swimming. As for intra-rater reliability, it ranged from reasonable to substantial. Some limitations were presented by the authors, during both the intra and inter-rater stages.
Corazza et al.21, in their turn, found virtually perfect agreement in a study that designed and validated a motor performance test for crawl swimming. These findings are still similar to those of other studies that reported reliability ranging from moderate to virtually perfect9),(22. As much as comparisons can be made between these studies, it is important to stress that swimmers’ levels, evaluation forms, observation mode (live or video recording), as well as the experience and level of training of the rating professionals are different. In the end, all these differences make it difficult to compare results between studies.
Another study evaluating only the front crawl stroke obtained reliability results ranging from mild to substantial, passing through moderate9. It is worth mentioning that the intra-rater reliability testing stage was carried out after a meeting between the raters and one of the researchers responsible for the research. Even with the meeting and a debate to choose the most relevant points of the evaluation form, and the greater academic training of one of the teachers, the intra-rater agreement was not high for all items, especially for teachers with less academic training. As in our study, the authors point out that the intervention did not generate the expected effect of standardization among the raters. This was evident because they continued to use their previously constructed personal knowledge to rate the swimmers9.
Two other studies evaluating crawl swimming obtained similar intra-rater reliability, which was virtually perfect21),(22. These findings are close to some of the results of the present study, in which considerable intra-rater reliability was obtained in questions 1, 3 and 5 for rater B, and virtually perfect for rater C in questions 1, 3 and 4. However, in the study by Corazza et al.21, the observation of the swimmers was repeated six days after the first observation and by means of direct observation of the swimmers in the pool. This short time interval between observations may allow some evaluation criteria to be kept in one’s memory and influence the second evaluation. Moreover, the direct observation of the swimmer may allow an adjustment in the position of the observer from their observation focus needs. The same does not happen when one watches the video, which has a fixed observation plane throughout the entire movement. Even in the current study, which had four fixed observation points in two different planes (lateral and frontal), the observation capacity is predetermined. Furthermore, initially, the swimmer is behind the observation plane allowed by the camera, then aligned in parallel and, finally, positioned ahead.
The study by Madureira et al.22 took into account only the errors that the executor can make in each crawl swimming stage. Such a strategy may have facilitated agreement at this stage, as the rater’s only function was to evaluate possible execution errors by the swimmers, which is easier to do than evaluating the movement as a whole and fitting it into one of the options highlighted in the evaluation form, which was the case in our study. As much as this facilitates the evaluation by the observers, we have to emphasize that there is no rigid, immutable technique for crawl swimming, but variations of what may be the ideal technique. Therefore, one should bear in mind the characteristics and limitations of the practitioner and allow something within the spectrum of that which is desirable for the crawl swimming technique10.
The training and experience of the professionals participating in the different studies presented should be discussed as well. This research hypothesizes that coaches are more concerned with observing the movement from the perspective of its contribution and propulsive efficiency. On the other hand, swimming teachers keep their focus on observing the movement from the aesthetics of the technique23. For instance, the results found showed that the teachers had greater intra-rater reliability in questions 1 and 5. The coaches, in questions 1, 3, 4 and 5. Whereas questions 1 and 5 are descriptors of movements executed out of the water and which are not propulsive, questions 3 and 4 refer to movements executed in the water and with a real influence on propulsion. Therefore, coaches seem to be concerned with the movements performed in and out of the water, since the technique is built with a propulsive purpose. In this way, they develop their own and spontaneous criteria that allow this observation. On the other hand, teachers seem to be more concerned with the movement performed outside the water, developing observation criteria which are more related in this sense.
In any case, movements executed in the water are difficult to be perceived by the raters, even in underwater images10, as this is not a common practice for most of them. All this influences the ability of these professionals to observe, perceive and evaluate the movement and, consequently, the inter and intra-evaluator agreement in each study. Also noteworthy is the difficulty presented by both groups in the current study in identifying the catch moment, described in question 2. Although this moment is not propulsive, the position of the upper limbs is important, because it is from them that the propulsive action of the stroke begins24. Given the above, a suggestion for studies on this theme is for the intervention to maintain the proposal of presenting the swimming technique, but to allow for a discussion among the raters, who will expose their own observation and rating criteria. This initiative may allow their criteria to be known and go through personal reflection and systematization. As a result, these criteria are expected to be standardized and contribute to raising inter- and intra-rater reliability in studies on swimming technique evaluation. An additional recommendation is the conduction of studies on other components of the crawl swimming technique, such as the movement of both arms (not just the right limb), the kicking movement, trunk position, and breathing phases.
Conclusion
The intervention carried out with the raters in an attempt to standardize the observation and rating criteria did not have the desired effect on inter- and intra-rater agreement. The results found are in line with those of other studies in the literature, but there was a belief in the possibility of improving reliability. Actually, despite the discussions raised in the intervention, what seems to have prevailed during the evaluations were the raters’ prior experiences. Thus, it is important to propose an intervention with quality and sufficient criteria to enable greater standardization as to the criteria for swimming teachers with different training and professional experiences to observe and rate swimmers.
References
-
1 da Silva LM, Giuliano AF, de Souza Castro FA. Ensino, aprendizagem e avaliação da técnica do eggbeater em aulas de natação. R. Bras. Ci. e Mov. 2016;24(2):138-45.. DOI: https://doi.org/10.18511/rbcm.v24i2.6187.
» https://doi.org/10.18511/rbcm.v24i2.6187 -
2 de Oliveira TA, Apolinário MR, Freudenheim AM, Corrêa UC. Análise sistêmica do nado crawl. Bra. Journ. of Motor Behav. 2009;4(1):15-21. Acesso em: 05 de março de 2022. DOI: https://doi.org/10.20338/bjmb.v4i1.20
» https://doi.org/10.20338/bjmb.v4i1.20 -
3 Silveira RP, de Souza Castro FA, Figueiredo P, Vilas-Boas JP, Zamparo P. The effects of leg kick on swimming speed and arm-stroke efficiency in the front crawl. Int J Sports Physiol Perform. 2017Jul;12(6):728-35. DOI: https://doi.org/10.1123/ijspp.2016-0232
» https://doi.org/10.1123/ijspp.2016-0232 -
4 Aleixo I, Vieira M. Análise do Feedback na instrução do treinador no ensino da Ginástica Artística. Motric. 2012[acesso em 06 mar 2021];8(2):849-59. Disponível em: https://www.proquest.com/openview/dc051d5d2e9f856ba7df3a756f88e560/1?pq-origsite=gscholar&cbl=616555
» https://www.proquest.com/openview/dc051d5d2e9f856ba7df3a756f88e560/1?pq-origsite=gscholar&cbl=616555 -
5 Federolf P, Reid R, Gilgien M, Haugen P, Smith G. The application of principal component analysis to quantify technique in sports. Scand J Med Sci Sports. 2014Jun;24(3):491-9. DOI: https://doi.org/10.1111/j.1600-0838.2012.01455.x
» https://doi.org/10.1111/j.1600-0838.2012.01455.x -
6 Lees A. Technique analysis in sports: a critical review. J Sports Sci. 2002Oct;20(10):813-28. DOI: https://doi.org/10.1080/026404102320675657
» https://doi.org/10.1080/026404102320675657 -
7 Rosado A, Virtuoso L, Mesquita I. Relação entre as competências de diagnóstico de erros das habilidades técnicas e a prescrição pedagógica no voleibol. Rev. Br. de Ed. Fis. Esp. 2004;18(2):151. DOI:https://doi.org/10.1590/S1807-55092004000200003
» https://doi.org/10.1590/S1807-55092004000200003 -
8 Canossa S, Fernandes RJ, Carmo C, Andrade A, Soares SM. Ensino multidisciplinar em natação: reflexão metodológica e proposta de lista de verificação. MotriC.. 2007;3(4):82-99. DOI: https://doi.org/10.6063/motricidade.656
» https://doi.org/10.6063/motricidade.656 -
9 Tucher G, Quintão GF, Garrido ND, Castro FAdS. Inter-and intra-rater reliability of swimming teachers with different skill levels, in different conditions, evaluating front crawl arm movement in non-expert swimmers. Motriz: Rev. Educ. Fis.. 2020;26. DOI: https://doi.org/10.1590/s1980-6574202000010025
» https://doi.org/10.1590/s1980-6574202000010025 -
10 Vidal JdM, Tucher G, Nogueira L, Novaes RC, Vale RGdS, Castro MORd, et al. Crawl technique observation sheet for beginning swimmers: an evaluation proposal for swimming teachers. Motriz: Rev. Educ. Fis. 2021;27. DOI: https://doi.org/10.1590/S1980-65742021016920
» https://doi.org/10.1590/S1980-65742021016920 -
11 Wizer R, Franken M, Castro F. Concordância intra e inter-observador de protocolo de avaliação de habilidades aquáticas de crianças. Rev Br. Ci. e Mov. 2016;24(1):101-7. DOI: https://doi.org/10.18511/rbcm.v24i1.5835
» https://doi.org/10.18511/rbcm.v24i1.5835 -
12 Leandro C, Ávila-Carvalho L, Sierra-Palmeiro E, Bobo-Arce M. The evaluation rules in the view of the rhythmic gymnastics judges. Journ. of Sports. Sci. 2016;4:232-40. DOI: 10.17265/2332-7839/2016.04.007
» https://doi.org/10.17265/2332-7839/2016.04.007 -
13 Currell K, Jeukendrup AE. Validity, reliability and sensitivity of measures of sporting performance. Sports Med. 2008;38(4):297-316. DOI: https://doi.org/10.2165/00007256-200838040-00003
» https://doi.org/10.2165/00007256-200838040-00003 -
14 Peixoto JG, Dias AG, Miranda LM, Defilipo ÉC, Feitosa MB, Chagas PSdC. Análise de confiabilidade de medidas das pressões plantares estática e dinâmica de crianças e adolescentes com desenvolvimento normal. Fisioter e Pesqui. 2017;24:46-53. DOI: https://doi.org/10.1590/1809-2950/16222224012017
» https://doi.org/10.1590/1809-2950/16222224012017 -
15 Zamparo P, Carrara S, Cesari P. Movement evaluation of front crawl swimming: Technical skill versus aesthetic quality. PLoS One. 2017;12(9):e0184171. DOI: https://doi.org/10.1371/journal.pone.0184171
» https://doi.org/10.1371/journal.pone.0184171 - 16 Knudson DV, Knudson CS. Análise qualitativa do movimento humano. Barueri-SP: Editora Manole Ltda; 2001.
-
17 Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977Mar;33(1):159-74. DOI: https://doi.org/10.2307/2529310
» https://doi.org/10.2307/2529310 -
18 Lima VS, Azevedo NAdA, Guimarães JMX, Pereira MM, Neto JA, Souza LM, et al. Produção de vídeo-educacional: estratégia de formação docente para o ensino na saúde. Rev. Elet. de Com. 2019. acesso em 06 de março de 2022. DOI: 10.29397/reciis.v13i2.1594
» https://doi.org/10.29397/reciis.v13i2.1594 -
19 Mogetti RS, Brod FAT, Lopes JLB. VIDEOAULA INTERATIVA COMO RECURSO DE ENSINO PARA A EDUCAÇÃO PROFISSIONAL A DISTÂNCIA. Redin- Rev. Edu. Interd.. 2019;8(1). acesso em 07 de março de 2022. Disponível em:Disponível em:http://seer.faccat.br/index.php/redin/article/view/1531
» http://seer.faccat.br/index.php/redin/article/view/1531 -
20 Lauriti NC. A comunicação na avaliação de desempenho docente no ensino superior. EccoS-Rev. Cient. 2002;4(1):111-30. DOI: https://doi.org/10.5585/eccos.v4i1.296
» https://doi.org/10.5585/eccos.v4i1.296 -
21 Corazza ST, Pereira ÉF, Villis JMC, Katzer JI. Criação e validação de um teste para medir o desempenho motor do nado crawl. Rev Bras Cineantropom Desempenho Hum. 2006[acesso em 08 mar 2022];8(3):73-8. Disponível em: Disponível em: https://pesquisa.bvsalud.org/portal/resource/pt/lil-469812
» https://pesquisa.bvsalud.org/portal/resource/pt/lil-469812 -
22 Gollegã DG, de Oliveira TAC, Freudenheim AM, Madureira F, Rodrigues HF, Dubas JP. Validação de um instrumento para avaliação qualitativa do nado “Crawl”. Rev. Bras. de Ed. Fis. e Esp. 2008;22(4):273-84. DOI: https://doi.org/10.1590/S1807-55092008000400004
» https://doi.org/10.1590/S1807-55092008000400004 -
23 Soares S, Fernandes R, Carmo C, Santos Silva J, Vilas-Boas J. Avaliação qualitativa da técnica em Natação. Apreciação da consistência de resultados produzidos por avaliadores com experiência e formação similares. Rev. Port. de Ci. do Desp. 2001[acesso em 08 mar 2022];1(3):22-32. Disponível em: Disponível em: https://www.researchgate.net/profile/J-Paulo-Vilas-Boas/publication/307781939_Avaliacao_qualitativa_da_tecnica_em_Natacao_Apreciacao_da_consistencia_de_resultados_produzidos_por_avaliadores_com_experiencia_e_formacao_similares/links/5836d0db08aec3fe331e0d56/Avaliacao-qualitativa-da-tecnica-em-Natacao-Apreciacao-da-consistencia-de-resultados-produzidos-por-avaliadores-com-experiencia-e-formacao-similares.pdf
» https://www.researchgate.net/profile/J-Paulo-Vilas-Boas/publication/307781939_Avaliacao_qualitativa_da_tecnica_em_Natacao_Apreciacao_da_consistencia_de_resultados_produzidos_por_avaliadores_com_experiencia_e_formacao_similares/links/5836d0db08aec3fe331e0d56/Avaliacao-qualitativa-da-tecnica-em-Natacao-Apreciacao-da-consistencia-de-resultados-produzidos-por-avaliadores-com-experiencia-e-formacao-similares.pdf -
24 Toussaint HM, Hollander AP, Van den Berg C, Vorontsov A. Biomechanics of swimming.In: Garrett WE, Kirkendall DT, editors. Exercise and Sport Science Philadelphia: Lippincott, Williams & Wilkins; 2000. [acesso em 08 mar 2022]:639-60. Disponível em:Disponível em:https://www.researchgate.net/profile/Peter-Hollander-3/publication/313617617_Biomechanics_of_swimming/links/60718dc14585150fe998b00d/Biomechanics-of-swimming.pdf
» https://www.researchgate.net/profile/Peter-Hollander-3/publication/313617617_Biomechanics_of_swimming/links/60718dc14585150fe998b00d/Biomechanics-of-swimming.pdf
Source: authors
