Translation and cultural adaptation to the portuguese language of the American Shoulder and Elbow Surgeons Standardized Shoulder assessment form (ASES) for evaluation of shoulder function

Abstracts

Dores na região do ombro estão presentes em grande parte da população. Um instrumento de medida utilizado para avaliar a função do ombro, independente da enfermidade ou do distúrbio, é o American Shoulder and Elbow Surgeons Standardized Shoulder Assessment Form (ASES). No entanto, até o presente estudo, o ASES não contava com uma versão em português. O objetivo deste trabalho é realizar a tradução e a adaptação cultural do ASES à língua portuguesa.A versão original do ASES passou pelo processo específico de tradução e adaptação cultural, constituído pelas etapas de tradução inicial, retrotradução, comitê, pré-teste e aprovação do autor da versão original. O pré-teste foi aplicado em vinte pacientes com distúrbios variados na articulação do ombro (9 mulheres; 41,1 ± 13,0 anos de idade; 11,2 ± 8,9 meses de acometimento; 12,5 ± 3,1 anos de escolaridade). A versão definitiva do ASES em português foi estabelecida após os pacientes julgarem todos os seus itens compreensíveis e claros, e o autor do questionário original considerar a versão adequada. O resultado deste trabalho auxiliará os profissionais de reabilitação e os pesquisadores brasileiros, que passam a contar com mais um instrumento de medida a ser aplicado em pacientes com alterações funcionais no ombro.

ombro; reabilitação; avaliação funcional

ORIGINAL ARTICLE

^IMaster's Degree in Rehabilitation (Université de Montréal - Canada). Research Nucleus, Physical Therapy Department, Pontifícia universidade Católica do Paraná, Curitiba, Paraná, Brazil

^IIPhD in Production Engineering (UFSC). Professor, Physical Therapy Department, Pontifícia Universidade Católica do Paraná, Curitiba, Paraná, Brazil

^IIIMaster's Degree in Rehabilitation (Université de Montréal). Rehabilitation School, Université de Montréal, Montreal, Canada

^IVPhysician, Professor of the Surgery Department, Medical School, University of Toronto. Chief of the Surgery Department of Sunnybrook Health Sciences Centre, toronto, Canada

Correspondence to

ABSTRACT

Shoulder pain affects a significant percentage of the population. The American Shoulder and Elbow Surgeons Standardized Shoulder assessment form (ASES) is an outcome tool used to assess shoulder function, regardless of the disorder. However, at the moment the current study was undertaken, a Portuguese version of the ASES was not available. The objective of this work was to translate and make a cultural validation of the ASES to the Portuguese language. The original version of the ASES underwent the specific process of translation and cultural adaptation, comprising of the initial translation, back translation, committee, pre-test and the approval by the original author. The pre-test was applied in 20 patients with shoulder disorders (9 women, 41.1 ± 13.0 years of age, 11.2 ± 8.9 months with the disorder, and 12.5 ± 3.1 schooling years). The final Portuguese version of the ASES was established after patients considered all items of this tool comprehensible and clear, and the author of the original questionnaire considered it adequate. The results obtained with this study will help Brazilian rehabilitation professionals and researchers, since they have one more outcome measure to be applied in patients with functional disabilities of the shoulder.

Keywords: shoulder, rehabilitation, functional assessment.

INTRODUCTION

Disorders in the shoulders are commonly seen in the Occident¹. Studies have demonstrated that 14-21% of the people experience pain in the area of the shoulder^2,3, and it has been estimated that two out of every three individuals will have at least one episode of neck or shoulder pain some time in their lives.⁴

Although the prevalence and incidence of shoulder disorders are elevated in the general population, they tend to increase in workers and athletes exposed to specific risk factors, such as repetitive movements and excessive strain.^5-9 Thirty-seven to 45% of industry and service sector workers experience shoulder pain, and eight new cases in 100 workers are seen each year.^5-9

Shoulder injuries cause pain and reduce articular mobility and, consequently, have a negative effect on functional abilities, work activities, and quality of life of the individuals.¹⁰ Besides, disorders of this joint represent an important socio-economical burden¹ because they affect workers assiduity and productivity^11,12, leading to a high investment in the treatment of those patients,¹³ and, in European countries, they are responsible for 18% of health care costs with disabled workers.² Although a large percentage of the information in the literature comes from European and North American studies, their data can be applied to the Brazilian reality, since most individuals work in less than ideal ergonomic conditions.

To minimize the impact of those injuries in the lives of individuals and in society, the quality and duration of rehabilitation and prevention programs should be maximized. Evidence indicates the efficacy of different physical therapy methods in shoulder rehabilitation.¹⁴ Initially, during and after the treatment period, measuring tools are used by physical therapists to plan the rehabilitation program and monitor its efficacy.

Functional assessments represent one of the ways of measuring the impact of a musculoskeletal injury in the life of an individual.¹⁵ Usually, assessment of muscular strength and amplitude of movement of the affected joint is used for this end. However, those measurements are poorly associated with the functional status for activities of daily living (ADLs) and work activities.¹⁰

Several questionnaires for the functional evaluation of the shoulder have been developed since the 1980s.¹⁶ In a systematic review of the literature, Bot et al. found 16 questionnaires with this objective in the English literature.¹⁷ Currently, only three of those questionnaires have a Brazilian version: The Disabilities of the Arm, Shoulder, and Hand Questionnaire (DASH),^18,19 Western Ontario Rotator Cuff Index (WORC),^20,21 and the modified version of The University of California at Los Angeles Shoulder Rating scale (UCLA).^{22, 23}

The Disabilities of the Arm, Shoulder, and Hand Questionnaire (DASH) has been developed by the American Academy of Orthopaedics Surgeons (AAOS) in cooperation with the Institute for Work & Health (Toronto, Canada). It is composed of 30 items that evaluate symptoms and physical, social, and psychological function, and it was developed to be used in patients with any disorder in any joint of the upper limbs. This generalization favors the use of the DASH in different medical areas, but makes it less responsive and, therefore, less effective as a research tool.^20,24

The Western Ontario Rotator Cuff Index (WORC) is a questionnaire composed of 21 items graded by visual analogue scales (VAS), and it encompasses physical symptoms, sports, recreation, work, life style, and emotional status. Unlike the DASH, the WORC is a specific tool for patients with changes in the rotator cuff.^21,25

The modified version of the UCLAwas introduced by Ellman et al. for the pre-and postoperative evaluation of degenerative lesions of the rotator cuff.^22,23 The modified UCLA has a maximal score of 35 points, 10 for pain, 10 for function, 5 for amplitude of movement for flexion of the shoulder, 5 for the strength of the group of flexors of the shoulder, and 5 for patient satisfaction.²⁴ Although widely used in Brazilian literature, the UCLA does not evaluate the functional status of the shoulder in specific movements and physical activities. Besides, in the pain-evaluation section, the options of answers combine the frequency and severity of pain, as well as the type and amount of medication used for pain relief. This increases patient difficulty to choose the answers that best apply to him/her, since the existing answers do not always correspond to the characteristics of the pain experienced by a patient.^23,24

Among the tools for the shoulder that are not restricted to specific disorders and that do not have a Portuguese version, the American Shoulder and Elbow Surgeons Standardized Shoulder assessment form (ASES)²⁶ should be mentioned due the number of times it is mentioned in the literature. The ASES was elaborated by the Research Committee of the American Shoulder and Elbow Surgeons,²⁶ and it is divided in two sections, one administered by a physician, and the other self-applied by the patient. This last section is composed by one item related to pain and ten items related to function, and the validity, reliability, and responsivity of this section have been demonstrated.^27,28 The item related to pain is evaluated by a VAS (10 cm) that ranges from "no pain at all" to "pain as bad as it can be". The items related to function are evaluated by a four-point Likert scale. The scores of the pain and function subsections are transformed in percentages and each one represents 50% of the final score, which can range from 0 (absence of function) to 100 (normal function).²⁴ This questionnaire, which evaluates shoulder function through questions related to common daily activities, is easy to apply. Unlike other tools that require recollection of the pain behavior, the ASES assesses pain at the time of the evaluation, without focusing on the past, allowing it to be applied in shorter intervals. Besides, studies have demonstrated that the ASES has better responsiveness than similar tools.²⁸

Besides providing a new tool for Brazilian rehabilitation professionals for clinical-and research-oriented uses, the Portuguese translation and cultural adaptation of the ASES will help standardize functional evaluation methods of the shoulder, similar to that of the methods used in other countries, allowing comparison of studies undertaken in different populations. Thus, the objective of the present study was to establish the Portuguese translation and cultural adaptation of the ASES to be used in patients with different shoulder disorders.

PATIENTS AND METHODS

Patients

Twenty patients (9 females and 11 males; mean age 41.1 ± 13.0 years) whose clinical diagnosis, schooling, socioeconomical status, and cultural status are summarized in Table 1, participated in the Portuguese translation and cultural adaptation processes of the ASES.

Inclusion criteria were as follows: 1) 18 years of age or older; 2) clinical diagnosis of a shoulder disorder; and 3) native speaking Portuguese. Exclusion criteria were: 1) the presence of neurological or musculoskeletal disorder other than the shoulder; 2) cognitive changes; and/or 3) illiterate, incapable of understanding and filling out the translated questionnaire.

All participants were recruited at a teaching outpatient clinic in Curitiba. Initially, medical records were reviewed to identify potential participants. The project was presented to patients who fulfilled the inclusion criteria and they had all the time they required to make a decision. Besides, any questions participants could have, after the presentation of the project, were answered. Patients under treatment, as well as new patients who started their treatment during the study period, were invited to participate in this study. Patients who agreed to participate signed an informed consent. The study was approved by the Research on Humans Ethics Committee under protocol #5257.

Translation and cultural adaptation

Only the section of the ASES administered by the patient underwent Portuguese translation and cultural adaptation, according to the steps proposed by Guillemin.²⁹

Two Portuguese native speaking English teachers were responsible for the literary and conceptual translation of the ASES. Only one of the translators was aware of the objective of the study, and translations were independently done. The two versions were compared and analyzed, by both translators and two investigators, to reach a consensus for the first Portuguese translation.

After the first Portuguese version of the questionnaire was defined, back translation, in which the Portuguese version was translated into English, independently, by two native English speaking translators who were fluent in Portuguese, who were unaware of the objective of the study and did not have access to the original questionnaire, was undertaken. Those newer versions of the questionnaire were then compared to the initial translation.

After the back translation,a committee formed by the four translators, three investigators, and three physical therapists, compared the original version, the first Portuguese translation, and the version of the back translation. The objectives of the committee included: 1) guarantee the translation was fully understandable; and 2) verify the semantic (i.e., vocabulary, grammar), idiomatic (i.e., colloquial expressions), and conceptual (i.e., concept validity) equivalence between the original version and the translation. At the end of this step, a second Portuguese version of the ASES was approved.

The clarity, comprehension, and acceptance of the second Portuguese version were tested in 20 individuals with shoulder disorders who were treated at the teaching outpatient clinic. On this step, called Pre-test, the investigator gave the questionnaires to the patients, who answered whether they understood or not each item and made comments on what they understood. If any item was not understood, the patient was stimulated to suggest changes. It was established that items with 15% or more of "lack of understanding" would be reformulated in a new meeting of the committee and, afterwards, tested in the patients.^15,30 Thus, the Portuguese version of the ASES would only be accepted when all items were considered understandable.

Finally, the documentation demonstrating all steps of the translation and cultural adaptation was sent to the author of the original questionnaire (i.e., the English version) to guarantee the process used was adequate and that the translation was appropriate.²⁸

RESULTS

After the initial translation, both translators presented similar versions. Small differences were observed in the description of the VAS, in the enunciate, and in items 4, 7, 8, and 10 of the section that evaluates function, which can be seen in Table 2. After discussion among the translators and investigators, the word "ability" was substituted by "capacity". Finally, on this step, item 7 was established as "lift 5 kg above the shoulder".

After the back translation, four items were modified. In this step, the definition of the score "3" was changed from "easy to perform" to "performs without difficulties". In item 9 of the functional assessment, the word "yours" was included, while item 10 was changed from "perform daily sports activity" to "practice the usual sport" (^{Annex 1}).

In the committee composed of specialists, three changes were made after the back translation. Item 3 was described as "reaching the upper portion of the back", item 4 was substituted by "wipe oneself after using the bathroom", and item 8 was changed to "to throw a ball above the head" (Table 3).

Table 4 shows the level of "non-comprehension" by patients of the items and words of the ASES in Portuguese. None of the items were considered "non-comprehensible", since all of them had "comprehension" levels higher than 85% and, therefore, it was not necessary to adjust the questionnaire to apply them again to the patients. The time necessary for patients to answer the ASES ranged from 2 to 7 minutes, corresponding to the mean time necessary to answer the original version of the ASES, which ranges from 3 to 5 minutes.¹⁷

The back translation of the pre-test version was sent to the original author, who considered the process to be adequate and the translation to be appropriate. Thus, the final Portuguese version of the ASES, adapted to the Brazilian culture, was established (^{Annex I}).

DISCUSSION

Several tools to evaluate shoulder function can be found in the literature;¹⁷ however, very few have been translated to Portuguese.^15,19,23 Although the ASES has been mentioned in the Portuguese literature,³¹ it is one of the questionnaires that, before the present study, did not have a Portuguese version elaborated after a specific methodological process.²⁹

Only the section of the ASES self-applied by the patient was translated and culturally adapted in our study. This can be justified because the section applied by the physician is not adopted as a measuring tool in studies found in the literature, probably because this section does not contribute for the final score.²⁴

The Portuguese version of the ASES was determined after the following steps: initial translation, back translation, committee, pre-test, and approval by the author of the original version. The main cultural and linguistic adaptation during the process was done in the first three phases. In the initial translation, the measuring unit "pounds" (lb) was substituted by "kilogram" (kg) in item 7. This is justified because, unlike the United States (i.e., British system), Brazil uses the International Units System. Although 10 lbs do not correspond exactly to 5 kg (but to 4.54 kg), the approximation of this value was done to help patients imagine the effort on the shoulder during the movement, since many products in the daily lives of those patients comes in 5-kg packages (i.e., rice and sugar).

In this step, the word "ability", seen in the second part of the original questionnaire, was translated as "habilidade" by both translators. However, during the meeting of those translators and the investigators, it was decided that the word "capacidade" would be more appropriate, since the objective of the questionnaire is to determine the level of difficulty to perform a task regardless of the technical condition of the patient.

In the back translation phase, item 10 was modified from "perform every day sports" to "practice the usual sport", mainly because the expression "every day" is related to "daily", while "usual" suggests a "habit", regardless of being performed every day or not.

Adaptations were also done during the meeting with specialists. In item 3, it was stated that patients should reach the "upper portion" of the back, according to the original questionnaire. Without this specification, the movement to be performed would be subjected to several interpretations, affecting both the level of difficulty and the group of muscles used. For example, instead of reaching the upper portion of the back through flexion, adduction, and external rotation of the shoulder, the patient could reach for the lower portion of the back, combining extension, adduction, and internal rotation.

The pre-test was applied in patients with different diagnosis and occupations, 41.1 (± 13.0) years of age, shoulder disorders for 11.2 (± 8.9) months, and 12.5 (± 3.1) years of schooling. The heterogeneity of the participants is important to reproduce the level of understanding of the general population. In this phase, two out of 20 patients had difficulties with item 8 of the functional assessment, in which the task mentioned is similar to the movement of throwing a handball, i.e., throwing a ball with one of the hands. Both patients were in doubt of whether the ball had to be thrown with 1 or both hands and the direction of the movement. However, the item was considered understandable because it had a level of comprehension of 90%. This indicates a small possibility of different interpretations, which could affect the reliability and responsivity of the questionnaire. Establishing the level of pain by the VAS, in which only the absence of pain and maximal pain are represented numerically (i.e., 0 and 10, respectively), was another difficulty met by patients. However, this only happened with one patient and, therefore, the level of comprehension of this item was also considered to be good. However, we chose to modify the original scale, as former studies have shown that Brazilian population understands better a numerical scale.³²

Only one pre-test was necessary to define the final Portuguese version of the ASES. This was due to the simple language, objectivity of the items of the questionnaire, and the fact that it uses common everyday tasks, making it easy to be understood by patients. Since this is a self-applied tool, the clarity observed in ASES items is fundamental for its validation. However, self-administration restricts the use of this tool to people who can read, which represents a limitation in the Brazilian reality.

Although the psychometric properties of the original ASES have been investigated,^27,28 determination of the validity, reliability, and responsivity of the Portuguese version is ongoing, since this information is fundamental to support its use.

The conclusion of all steps of the elaboration of the Portuguese version of the ASES will contribute with the daily practice of Brazilian physical therapists since, with this tool, they will be able to improve treatment planning and followup, allowing objective comparisons of shoulder-specific function in individuals with different clinical diagnosis. Besides, the Portuguese version of the ASES is important for the Brazilian scientific community because it represents one more type of evaluation that can be used as a measuring tool in clinical assays.

ACKNOWLEDGEMENTS

The authors are grateful to the patients who participated in this study, as well as to Marc Breyer, Leonardo Paixão, Denilso de Lima, and Brunnel Lufti for their technical support. Portuguese version of the aSES for assessment of shoulder function

REFERENCES

References

• ¹ Grooten WJ, Mulder M, Josephson M, Alfredsson L, Wiktorin C. The influence of work-related exposures on the prognosis of neck/shoulder pain. Eur Spine J 2007; 16(12):2083-91.
• ² Bongers PM. The cost of shoulder pain at work. BMJ 2001;322(7278):64-5.
• ³ Urwin M, Symmons D, Allison T, Brammah T, Busby H, Roxby M et al Estimating the burden of musculoskeletal disorders in the community: the comparative prevalence of symptoms at different anatomical sites, and the relation to social deprivation. Ann Rheum Dis 1998; 57(11):649-55.
• ⁴ Makela M, Heliovaara M, Sievers K, Impivaara O, Knekt P, Aromaa A. Prevalence, determinants, and consequences of chronic neck pain in Finland. Am J Epidemiol 1991; 134(11):1356-67.
• ⁵ Andersen JH, Haahr JP, Frost P. Risk factors for more severe regional musculoskeletal symptoms: a two-year prospective study of a general working population. Arthritis Rheum 2007; 56(4):1355-64.
• ⁶ Bongers PM, Kremer AM, ter Laak J. Are psychosocial factors, risk factors for symptoms and signs of the shoulder, elbow, or hand/ wrist?: A review of the epidemiological literature. Am J Ind Med 2002; 41(5):315-42.
• ⁷ Cassas KJ, Cassettari-Wayhs A. Childhood and adolescent sportsrelated overuse injuries. Am Fam Physician 2006; 73(6):1014-22.
• ⁸ Eltayeb S, Staal JB, Kennes J, Lamberts PH, de Bie RA. Prevalence of complaints of arm, neck and shoulder among computer office workers and psychometric evaluation of a risk factor questionnaire. BMC Musculoskelet Disord 2007; 8:68.
• ⁹ Leclerc A, Chastang JF, Niedhammer I, Landre MF, Roquelaure Y. Incidence of shoulder pain in repetitive work. Occup Environ Med 2004; 61(1):39-44.
• ¹⁰ Mousavi SJ, Parnianpour M, Abedi M, Askary-Ashtiani A, Karimi A, Khorsandi A et al Cultural adaptation and validation of the Persian version of the Disabilities of the Arm, Shoulder and Hand (DASH) outcome measure. Clin Rehabil 2008; 22(8):749-57.
• ¹¹ Hagberg M, Tornqvist EW, Toomingas A. Self-reported reduced productivity due to musculoskeletal symptoms: associations with workplace and individual factors among white-collar computer users. J Occup Rehabil 2002; 12(3):151-62.
• ¹² Hooftman WE, Westerman MJ, van der Beek AJ, Bongers PM, van Mechelen W. What makes men and women with musculoskeletal complaints decide they are too sick to work? Scand J Work Environ Health 2008; 34(2):107-12.
• ¹³ Meislin RJ, Sperling JW, Stitik TP. Persistent shoulder pain: epidemiology, pathophysiology, and diagnosis. Am J Orthop 2005; 34(12 Suppl):5-9.
• ¹⁴ Faber E, Kuiper JI, Burdorf A, Miedema HS, Verhaar JA. Treatment of impingement syndrome: a systematic review of the effects on functional limitations and return to work. J Occup Rehabil 2006; 16(1):7-25.
• ¹⁵ Lopes AD, Stadiniky SP, Masiero D, Carrera EF, Cicconelli RM, Griffin S. Tradução e adaptação cultural do WORC: um questionário de qualidade de vida para alterações do manguito rotador. Rev. Bras. Fisioter. 2006; 10(3):309-15.
• ¹⁶ Paul A, Lewis M, Shadforth MF, Croft PR, Van Der Windt DA, Hay EM. A comparison of four shoulder-specific questionnaires in primary care. Ann Rheum Dis 2004; 63(10):1293-9.
• ¹⁷ Bot SD, Terwee CB, van der Windt DA, Bouter LM, Dekker J, de Vet HC. Clinimetric evaluation of shoulder disability questionnaires: a systematic review of the literature. Ann Rheum Dis 2004; 63(4):335-41.
• ¹⁸ Hudak PL, Amadio PC, Bombardier C. Development of an upper extremity outcome measure: the DASH (disabilities of the arm, shoulder and hand) [corrected]. The Upper Extremity Collaborative Group (UECG). Am J Ind Med 1996; 29(6):602-8.
• ¹⁹ Orfale AG, Araujo PM, Ferraz MB, Natour J. Translation into Brazilian Portuguese, cultural adaptation and evaluation of the reliability of the disabilities of the arm, shoulder and hand questionnaire. Braz J Med Biol Res 2005; 38(2):293-302.
• ²⁰ Kirkley A, Griffin S, McLintock H, Ng L. The development and evaluation of a disease-specific quality of life measurement tool for shoulder instability. The Western Ontario Shoulder Instability Index (WOSI). Am J Sports Med 1998; 26(6):764-72.
• ²¹ Lopes AD, Ciconelli RM, Carrera EF, Griffin S, Faloppa F, Dos Reis FB. Validity and reliability of the Western Ontario Rotator Cuff Index (WORC) for use in Brazil. Clin J Sport Med 2008;18(3): 266-72.
• ²² Ellman H, Hanker G, Bayer M. Repair of the rotator cuff. End-result study of factors influencing reconstruction. J Bone Joint Surg Am 1986; 68(8):1136-44.
• ²³ Oku EC, Andrade AP, Stadiniky SP, Carrera EF, Tellinni GG. Tradução e adaptação cultural do Modified-University of California at Los Angeles Shoulder Rating Scale para a língua portuguesa. Rev Bras Reumatol 2006; 46(4):246-52.
• ²⁴ Kirkley A, Griffin S, Dainty K. Scoring systems for the functional assessment of the shoulder. Arthroscopy 2003; 19(10):1109-20.
• ²⁵ Kirkley A, Alvarez C, Griffin S. The development and evaluation of a disease-specific quality-of-life questionnaire for disorders of the rotator cuff: The Western Ontario Rotator Cuff Index. Clin J Sport Med 2003; 13(2):84-92.
• ²⁶ Richards RR, An K-N, Bigliani LU, Friedman RJ, Gartsman GM, Gristina AG et al A standardized method for the assessment of shoulder function. J Shoulder Elbow Surg 1994; 3:347-52.
• ²⁷ Michener LA, McClure PW, Sennett BJ. American Shoulder and Elbow Surgeons Standardized Shoulder Assessment Form, patient self-report section: reliability, validity, and responsiveness. J Shoulder Elbow Surg 2002; 11(6):587-94.
• ²⁸ Angst F, Goldhahn J, Drerup S, Aeschlimann A, Schwyzer HK, Simmen BR. Responsiveness of six outcome assessment instruments in total shoulder arthroplasty. Arthritis Rheum 2008; 59(3):391-8.
• ²⁹ Guillemin F. Cross-cultural adaptation and validation of health measures. Scand J Rheumatol 1995; 24(2):61-3.
• ³⁰ Marques AP, Santos AMB, Assumpção A, Matsutani LA, Lage LV, Pereira CAB. Validação da versão brasileira do Fibromyalgia Impact Questionnaire (FIQ). Rev Bras Reumatol 2006; 46(1):24-31.
• ³¹ Lino Júnior W, Belangero WD. Resultado funcional do tratamento artroscópico da instabilidade do ombro sem a lesão do lábio da glenóide. Rev Bras Ortop. 2007; 42(7):195-200.
• ³² Gallasch CH, Alexandre NM. The measurement of musculoskeletal pain intensity: a comparison of four methods. Rev Gaucha Enferm 2007; 28(2):260-5.

Annex 1

Translation and cultural adaptation to the portuguese language of the American Shoulder and Elbow Surgeons Standardized Shoulder assessment form (ASES) for evaluation of shoulder function

Luiz A. Knaut^I; Auristela D. L. Moser^II; Sibele De Andrade Melo^III; Robin R. Richards^IV

Publication Dates

• Publication in this collection
  20 May 2010
• Date of issue
  Apr 2010

History

• Accepted
  14 Feb 2010
• Received
  01 Sept 2009