A Critique on Gomez-Parra and Company’s Design and Validation of a Questionnaire for the Measurement of Students’ Perceptions of Intercultural Practices within Bilingual Secondary Schools in the European Context

Summary

The article was written by highly profiled researchers aimed at producing an appropriate research tool with regard to intercultural practices in the context of bilingual education. I  chose this article because as a researcher, the thing that I would really seek to master is the proper conduct of the methodology. Through the years,  I have  learned that writing a good research paper lies in the proper methods to be utilized. No matter how effective  the title is and no matter how data-driven the presentation of the theoretical foundations in the Review of Related Literature, if the methods are not properly executed, the strength of the paper becomes  weak. Consequently, reading  its  title about designing a good research instrument  interests me to  delve in deeper into the core of the text.  The title alone can give us already the overview of what we should expect to be reading and learning in the article. It will give us a comprehensive description of how the researchers were able to design and validate the questionnaires that could successfully measure the  students’ perception of intercultural practices within bilingual secondary schools in the European context. 

The very reputable authors are Dr. María-Elena Gómez-Parra, Dr. Irina Golubeva, and Dr. Roberto Espejo Mohedano. Dr. Mara-Elena Gómez-Parra earned her English philology degree from the University of Granada and her PhD from the University of Córdoba. She has a master's degree in distance education. She is currently an interdisciplinary researcher focused on Intercultural Education, Bilingual Education, and English as a Foreign Language. She is an Associate Professor at the University of Córdoba's Faculty of Education. Interculture, multilingual education, and early second language acquisition are some of her main research interests. Dr. Gómez Parra is the director of the Bilingual and Intercultural Education Research Group, the Ibero-American Bilingual and Intercultural Education Joint Master's Degree, the Erasmus Mundus Joint Master's Degree, and the prestigious UIMP (Universidad Internacional Menéndez Pelayo) training courses for Spanish Primary School Teachers. She is also a reviewer for a number of international journals on education, second languages and culture. Dr. Irina Golubeva, the second author, is an Associate Professor and the Director of the Graduate Program in Intercultural Communication at the University of Maryland, Baltimore County (Baltimore, MD, USA), as well as the Co-Director of the Intercultural Leadership Certificate Program. English, Hungarian, Russian, and Spanish are the four languages in which she teaches and publishes. Professor Golubeva's primary research interests include the development of multilingual awareness and intercultural competence, higher education globalization, and the idea of intercultural citizenship. She is passionate about non-profit work and served for seven years as vice-president of the European Association of Teachers. In 2020, she was granted the Pedagogy and Teaching Award as well as the title of "UMBC Innovation Fellow," which was particularly encouraging amid the COVID-19 pandemic's unexpected changeover to online classrooms. Dr. Roberto Espejo Mohedano, like the first author, is also a professor at the University of Córdoba. He has worked as a university professor for 29 years, teaching several engineering degrees and participating in multiple doctoral and master's degree programs. He has a PhD in Computer Science and a degree in Mathematical Sciences with a focus in Statistics (computational statistics). For numerous years, he served as the academic coordinator for university degree programs in computer engineering and was involved in several curriculum development procedures at the University of Córdoba's Polytechnic School. He has written in high-impact journals in the field of applied statistics and has managed a number of educational innovation projects. He also serves as a peer reviewer for a number of science education journals. These researchers successfully completed the paper in 2021.

Now let us see the context of where they are coming from to understand the rationale behind this article. This study is conducted in the European continent which includes countries such as Netherlands, Hungary, Germany, France, England, Spain, and Poland. The aforementioned countries have language education principles that are similar to those recognized by the Common European Framework of Reference for Languages, hence their language policies promote plurilingual and pluricultural skills in their people' educational curriculum. This research is based on the commonly held belief that language and culture are inextricably linked, which is shared by educators who teach in multilingual and intercultural settings (Byram, 2012). "Language and culture should indeed constitute part of all teachers' activities, regardless of the disciplines they teach," writes Porto (2013). Furthermore, these countries, along with many others around the world, think that bilingual education should be at the forefront of international educational policies due to the wide range of benefits it can provide, including intellectual, sociocultural, and neurolinguistic benefits, as well as increased employment opportunities for bilingual workers. "Learners need not only knowledge and proficiency in the grammar of a language, as well as the ability to use the language in socially and culturally relevant ways," according to this concept (Byram, Gribkova, & Starkey, 2002). The interplay of language and culture is one of the aspects that contribute to the effectiveness of a bilingual program. They recognize the significance of measuring the school's scope in order to properly complete one of its most crucial tasks: the efficient application of multicultural practices in secondary education. As a result, the development of an assessment instrument seeks to close the gap between the level of L2 vocabulary knowledge which is usually assessed and the level of cultural intelligence which is rarely assessed. This goal is supported by a literature review, which highlights the importance of bilingual education in the twenty-first century (Senge, 2010). There are a few studies that look at bilingual education from the perspective of students (Ramrez-Verdugo, & Gerena, 2020), but none of them, according to the authors, provide an instrument to measure students' perceptions.

The study's aims were to design a questionnaire to measure students' perceptions of intercultural practices in bilingual schools, then establish the content validity and comprehension validity of the questionnaire using appropriate methods and statistical tools, and finally analyze the questionnaire's validity and reliability. design, pilotage, and validation are the three key procedures covered in this study.

The first stage entails the development of preliminary questionnaires, the content of which is based on theoretical premises taken from the literature studied. Knowledge, attitudes, skills, awareness, and intercultural school practices are the five categories that it is divided into at first. These categories, according to literature, are the areas where intercultural competency can be tested and analyzed. Depending on the material or location they would fit, each part has a varying number of questions. A total of 51 questions were spread throughout the five parts at this level. The researcher chose to utilize a 4-point Likert Scale, with 4 indicating Strongly Agree, 3 indicating Agree, 2 indicating Disagree, and 1 indicating Strongly Disagree. The study group agreed on the format of the initial questionnaire with the goal of making it easier for the pilot testing sample to read and respond to the questionnaire. Aside from the questions, the instrument contains some basic information about the respondents, such as their age, gender, school course, and country of origin.

The questionnaires were subsequently put through two rounds of validity testing: content validity and comprehension validity. The Delphi Method was used to test the content validity. As stated in the article, the Delphi methodology is one of the most extensively used and approved methods for collecting replies from experts in a given field of study in order to achieve a convergence of opinion on a certain real-world issue, according to the literature they read. This strategy was chosen because it is best suited to achieving consensus. Delphi, in comparison to other research methods, has several unique features. For example, the selected participants have the option to review and revise their initial judgements regarding the topic matter based on input from other panelists during the series of rounds, which was critical to the current study. The researchers in the study went into great detail about how they chose the experts to participate in their investigation. They listed the characteristics of the experts who took part in the Delphi process. Experts are asked to respond to multiple rounds of written questions as part of the Delphi approach. After each set of surveys is finished, the facilitator compiles all of the responses and provides a summary report of the responses to each expert. The experts next review the summary report and express agreement or disagreement with the other experts' responses. After completing a second questionnaire, the experts are able to provide updated comments based on their understanding of the summary report. The Delphi process is finished when all forecasts are in agreement. Throughout the process, the experts determined whether each piece was relevant enough to be included in the original section. The researchers were in charge of keeping track of the experts' reports on their evaluations of the study instrument's content. The results were then analyzed. The validation procedure included criteria such as clarity, pertinence, adequacy, and feedback. The questionnaires were then adjusted and finalized based on the Delphi panelists' feedback at the end of the two rounds of the Delphi procedure. The surveys were reduced down to 21 items from the original 51 questions at the end of the Delphi Method. After that, the completed questionnaires were put through a second test, the comprehension validity test, which was done through pilot testing. The comprehension validity test will be used to determine whether the respondents understand all of the questions in the instrument in the pilot testing sample. The researchers agreed to evaluate the comprehension of 40 student-respondents aged 14-16 in the specified bilingual schools as part of this pilot study. Following that, the data was evaluated using statistical methods. Cronbach's alpha, which expresses the degree to which the items measure one common and unique variable, which is intercultural activities in bilingual secondary schools, was used to verify the scale's reliability. Items that reduce the internal consistency coefficient have been eliminated. Once the scale has reached a good level of internal consistency, the process is finished. The five sections of the questionnaires were reduced to four using various tests in pilot testing, with Cronbach alpha ranging from 0.60 to 0.7, which was regarded sufficient by the author and backed by the literature they mentioned. The low value of the variation coefficient led the researchers to increase the Likert scale range from 1-4 to 1-6 in order to increase the variability of the respondents' answers. With all of these modifications, the researchers believe the tool has become acceptable for measure the extent of intercultural practice implementation in bilingual secondary schools as perceived by students, with an 88.1 percent reliability rating. The modified research instrument was subsequently validated and assessed by administering it to 213 bilingual secondary school students from numerous European nations with similar demographics in the pilot testing.

New data was collected and new analyses were performed at the final step. Using a snowball strategy, the researchers selected the sample for the last step using a non-probabilistic convenience sampling method. The final sample of 213 students was evenly distributed based on the gender of the participants. The sample size was established using the traditional guideline of a minimum of 10 sample elements for each of the instrument's items to be confirmed subsequently using factorial analysis. The data acquired in this step was likewise verified for internal consistency using Cronbach alpha, as it was in the pilot testing stage. Total item correlation was also employed to refine the number of final items at this step. The item total correlation is a relationship between the question score and the overall assessment score. It is assumed that participants who correctly answer a question will have higher overall evaluation scores than those who incorrectly answer a question. Although the values acquired on the scales are acceptable in the final analysis of the data, the researchers perform a confirmatory factor analysis as a final step in the validation of the construct. It's a method for condensing a big number of variables into a smaller number of parameters. This method takes the largest common variance from all variables and converts it to a single score. This confirmatory factorial analysis allowed the researchers to identify three factors that corresponded to the groups of questions for the items included in the study's table. Two more items were dropped since their factor saturations didn't match any of the multidimensional factors. According to the results of the Kaiser-Meyer-Olkin (KMO) and Barlett's sphericity tests, the result is relevant.

The instrument earned an exceptional reliability score at the culmination of the last stage of testing. Following the indicated approaches for increasing the psychometric quality of the instrument, the 21 questions were further refined to 19 items. Finally, the new instrument was forwarded to a group of experts who finally came to an agreement during the validation procedure. The questionnaire's final form was determined to have valid metric values derived from several validity and confirmatory tests that can measure the items for which they were originally developed. Overall, the authors asserted that the proposed questionnaire has good psychometric quality, based on the results of multiple statistical tests.

The authors also discussed some of the study's limitations, which may limit the extent to which it can be applied, as well as potential solutions to those constraints. The first is on the justification for utilizing a single quantitative instrument, which was found to be important in order to collect as objective data as possible from the target demographic (14 to 16-year-old secondary school students who have likely never been dealt with these types of questions before). Another constraint is the questionnaire administration, as the target population's age and cognitive level can lead to statistical biases in the final results. The authors highlighted a study by Soubelet and Salthouse (2011) that found children of various ages having varying judgments of what is socially acceptable, which can lead to statistical biases connected to each individual's psychological development stage. As a result, this fact may have an impact on the validity and consistency of their responses to a trait-sensitive questionnaire. On the other hand, the effect of what is considered as socially desirable peaks between the ages of 10 and 14, whereas higher social resistance is displayed between the ages of 14 and 16. Because the test was created primarily to assess the perceived intercultural practices of bilingual secondary school students aged 14 to 16, extra attention should be paid to this fact. Furthermore, the presence of an interviewer or an experienced guide can increase the quality of the responses, according to another limitation identified. Other investigations have confirmed that their presence can cause a 5% to 10% difference in the replies. As a result, self-administration of the instrument is recommended, avoiding the interviewer effect, based on the recommendations of the panel of experts on the design of the research instrument (regarding the clarity, pertinence, context, and adequacy of the questions for secondary education students) and the fact that it is a "short questionnaire" (as was carried out in the piloting of this study). Furthermore, the current survey does not gather information from key stakeholders such as school administrators, classroom instructors, or parents (which will be the goal of a complementary questionnaire that is under construction at the moment of writing this paper by the same research team). This fact restricts the conclusions that can be drawn from these findings. As a result, the researchers intend to be scientifically rigorous in declaring what may legitimately be asserted from this analysis: students' impressions of intercultural behaviors in bilingual secondary schools in Europe.

Critique

The study is very interesting as it gives readers the opportunity to learn designing and validating research instruments. As to the introduction part of the paper, I am impressed with the good sequence of information. All information is data-driven and packed with supporting literature. Salient concepts from relevant literatures are systematically presented. The authors successfully highlighted the research gap and successfully established the basis for conducting the study. 

In the methodology part of the paper, I find the presentation very detailed with some observations that can improve the paper to better suit the readers who are very good with numbers. The authors were able to justify the appropriateness of the methods used (form the use of Delphi method, to the use Cronbach Alpha in testing the reliability of the instrument in both the pilot testing and final stage, the use of Total Item Correlation and Factorial Analysis in the last stage). They also presented the detailed process of the Delphi Method in the study. However, the sampling procedure employed for the final validation of the questionnaire is quite questionable (Snowball Technique- a non-probabilistic convenience sampling). Non-probability sampling technique is most useful for exploratory studies like pilot survey where sample size is smaller compared to the pr-determined sample. Researchers usually use this method in studies where it is impossible to draw random probability sampling due to time or cost considerations. And snowball sampling technique specifically helps researchers find a sample when they are difficult to locate, the sample size is small and not easily available. The authors failed to justify the use of the sampling method considering that they use this in the final stage of validation where 213 student respondents were involved. Also, the distribution of respondents according to sex as mentioned above is also questionable since snowball technique was employed. On the lighter note, one of the strengths of the paper is the fact that the research tool was tested for internal consistency for both Pilot and Final Sample. There is also a good presentation of how data will be analyzed for both for qualitative and quantitative. Though the presentation of the stages of the validation is very detailed, the arrangement and the order of presentation must be improved because there were some parts that are confusing. Confusing most specially for readers who are not so familiar with various statistical test. All items in the final version of the questionnaire was expressed in positive statements. The addition of negatively worded items in a questionnaire/scale is a popular strategy for reducing acquiescent answer bias. Questionnaires that include both positive and negative comments encourage attentive responders to disagree with some of the statements. The resulting composite score should have minimized acquiescence bias under the assumption that negative and positive items are substantially comparable, and by reverse rating the negative items. However, more recent studies suggest that providing a mix of positively and negatively worded items causes more difficulties than it solves. Internal reliability is lowered, the factor structure/construction is distorted, and there are issues with criterion related validity (Lewis and Sauro, 2009).

The results were presented in logical manner and is very understandable. Overall, the article is a very good paper with very helpful tool to measure students’ perception of their intercultural practices. 

Comparison and Contrast

I searched for other articles that have something to do with designing and validating research instruments in measuring identified variables in the study. Three articles were compared to this study: Fernández-Rio et al.’s (2017) Design and validation of a questionnaire to assess cooperative learning in educational contexts; Saillour-Glénisson et al.’s  (2016) Design and validation of a questionnaire to assess organizational culture in French hospital wards; and Alvarado et al. (2016) Design and validation of a questionnaire to measure research skills: experience with engineering students. I compared their objectives and the methods they used in the study. All articles are with common goals, that is to design and validate research instruments. In terms of their methodology, I looked into the series of validity and reliability tests employed in the study. The first article  had its first version of the questionnaire assessed by a group of experts.  Then conducted a pilot study with 60 students. And the final version underwent several statistical tests. The second paper  conducted pre-test and face validation with 15 health care professionals. It has implemented series of phases:  the exploratory phase, the confirmatory phase, and the reproducibility phase.  The third study had its questionnaire validation through literature review, semantic and content validation by experts. Then,  they made use of factorial and reliability validation. All articles underwent a series of validity and reliability tests. This made their study credible, generalizable, and replicable like that of the study of  Gomez-Parra and company. When a study utilizes a wide range of validity and reliability testing, it aims to reduce errors and biases on  the process. Their articles have a solid methodology base, which is similar to the work of  Gomez-Parra. 

Conclusion

A good research paper requires an instrument with a  good psychometric quality.  It   needs to have high reliability and validity scores to capture the main goal of the study. To be able to design and validate,  one requires careful study and analysis. The article presented a good description  how the questionnaire in measuring students’ perception of Intercultural Practices within Bilingual Secondary Schools in the European Context was designed and analyzed. This paper is very helpful as this can contribute to the fields of both bilingual and intercultural education, where the measurement of the perceptions of bilingual students can guide stakeholders to improve pedagogical and in-classroom practices. Overall, the study is very relevant as it is aimed in improving the way to measure perception towards intercultural practices in bilingual schools. This is a good outset to new research in the future. A study that would address its indicated limitations and expand the scope of the study, maybe extending the participants to stakeholders like teachers and parents. The focus would be still on designing and validating questionnaires but this time it is multi-perspective questionnaires that can measure intercultural practices. Specifically, dealing on research questions like, what psychometric qualities can be incorporated in the design of the multi-perspective questionnaires, how can the validity be established, and how can reliability be ensured on  the process? With this, the study would not only expand the views about intercultural practices in the context of education but also provide an avenue for promoting strategies in the context of cultural inclusivity.

References

Alvarado, F. C., León, M. P., & Colon, A. M. O. (2016). Design and validation of a questionnaire to measure research skills: experience with engineering students. Journal of Technology and Science Education, 6(3), 219-233.

Byram, M. (2012). Language awareness and (critical) cultural awareness – relationships, comparisons and contrasts. Language Awareness, 21(1-2), 5-13. https://doi.org/10.1080/09658416.2011.639887

Byram, M., Gribkova, B., & Starkey, H. (2002). Developing the intercultural dimension in language teaching: A practical introduction for teachers. 

Fernández-Rio, J., Cecchini, J. A., Méndez-Giménez, A., Méndez-Alonso, D., & Prieto, J. A. (2017). Design and validation of a questionnaire to assess cooperative learning in educational contexts. Anales de psicología, 33(3), 680-688.

Lewis, J. R., & Sauro, J. (2009). The factor structure of the system usability scale. SpringerLink. https://link.springer.com/chapter/10.1007/978-3-642-02806-9_12

Porto, M. (2013). Language and intercultural education: An interview with Michael Byram. Pedagogies: An International Journal, 8(2), 143-162. https://doi.org/10.1080/1554480x.2013.769196

Rio, J. F., Cecchini, J. A., Giménez, A. M., Alonso, D. M., & Prieto, J. A. (2017). Design and validation of a questionnaire to assess cooperative learning in educational contexts.  https://pdfs.semanticscholar.org/deb8/144e327ef46af5b7e0a128e66f6b751d76b2.pdf

Saillour-Glénisson, F., Domecq, S., Kret, M., Sibe, M., Dumond, J. P., Michel, P., & TheOReM Group. (2016). Design and validation of a questionnaire to assess organizational culture in French hospital wards. BMC health services research, 16, 1-14.

Senge, P. M. (2010). Education for an interdependent world: Developing systems citizens. Second International Handbook of Educational Change, 131-151. https://doi.org/10.1007/978-90-481-2660-6_8

Soubelet, A., & Salthouse, T. A. (2011). Influence of social desirability on age differences in self-reports of mood and personality. Journal of Personality, 79(4), 741-762. https://doi.org/10.1111/j.1467-6494.2011.00700.x

AUTHOR: 

Aubrey May F. Balignot, Ed.D. 

Department of Education, Philippines 

aubreymay.balignot@deped.gov.ph