Critiquing a qualitative study using Tracy ’ s big-tent criteria

This paper presents a critique of an article that explores the internationalisation of the curriculum. First, to facilitate critical reflection, the main Western paradigms of positivism, interpretivism, critical theory and postmodernism are described and the differences between them emphasised. The article focused on in this paper can be situated largely within an interpretivist paradigm. Big-tent markers, which are employed to evaluate qualitative research, are then critically implemented to highlight the strengths and weaknesses of this article and offer relevant suggestions regarding ways in which it can be improved.


Introduction and rationale
Lecturing in a private college in Oman has provided me with numerous opportunities to interact with international students from various backgrounds. The presence of non-Omani students alongside Omani students, together with the internationalisation of the curriculum (IoC), has been of increased academic interest to me. Within the scope of this paper, it is important to clarify the distinction between IoC and IoHE (Internationalisation of Higher Education). IoC refers to "the incorporation of an international and intercultural dimension into the curriculum as well as the teaching and learning arrangements and support services of a program of study" (Leask, 2009, p. 209). This indicates that both a formal and informal curriculum, including optional services, are internationalised in line with global dimensions to fulfil students' needs. Conversely, IoHE in Oman is "a way of producing graduates capable of explaining their country to the wider world" (Trahar, 2011, p. 5) and is similar to the system applied in Japan (Trahar et al., 2015).
1 Mr. Saud Albusaidi,University of Bristol,s_s_8mile@hotmail.com Research in Social Sciences and Technology (RESSAT) 2019: 4 (1), [105][106][107][108][109][110][111][112][113][114][115][116][117][118][119][120][121][122] Several researchers have investigated themes relating to internationalisation in Oman (e.g. Al Abduwani, 2017;Al'Abri, 2016a;2016b;Brandenburg, 2013). Sawir's article, however, provides a holistic view of how teachers and domestic students perceive internationalisation and explores the contributions international students make in Australia. IoC is a major area of interest within higher education and Sawir's article has motivated me to explore whether similar observations can be made in Oman. Indeed, my selection of this article was driven by my pedagogical questioning of IoC in Oman and the research I am conducting for my thesis. In practical terms, this paper will also help me to acquire fundamental skills and knowledge needed to understand, dissect, and critique any future research papers.
First, I present a summary of Sawir's research article, along with its aims, methodology, and the main findings. The paradigmatic perspective from which Sawir conducted her research will then be classified and justified. Tracy's (2010) criteria for evaluating qualitative research will then be utilised to evaluate the strengths and weaknesses of her study. This will be followed by an overall evaluation of issues pertinent to conducting research in social sciences, along with suggestions on how Sawir's study could be improved. Finally, I will explain how the stated aims of this paper have been achieved.

Summary of Sawir's article
In her research, Sawir investigates whether and how international students have simplified IoC in Australia. A more thorough examination of internationalisation is an ongoing requirement in higher education (Leask, 2015;Green & Whitsed, 2015) and also in relation to Oman (e.g., Al Abduwani, 2017;Al'Abri, 2016a;2016b). Sawir (2013) argues that the presence of international students and their facilitation in learning and teaching has not been extensively investigated in Australia. Her article therefore aims to fill this gap in the research literature.
To conduct her study, she recruited potential participants by sending an email to four academic faculties. In response, 80 academic staff members agreed to participate and signed a consent form prior to taking part in a semi-structured interview. Once the interviews were conducted, a thematic approach was utilised to analyse the data. The main findings were then presented and discussed.
Sawir's study highlights the following issues: Albusaidi 1. International students are tied to their own diverse cultures.
2. Academic staff were inspired in their pedagogical preparation for an internationalised classroom.
3. Staff appreciated and acknowledged the value of such classrooms. 4. Australian students were unaware of how internationalisation impacted the cultural environment.
5. Given the lack of interest among Australian students, the cultural resources that international students bring were not fully exploited.

Review of theories and classifying the article
The different paradigms of 'Western' social science research can be classified as positivism, interpretivism, critical theory, and post-modernism (e.g. see Bassey, 1996;Crotty, 1998;Usher, 1996). These paradigms "systematically" guide the researcher in addressing and exploring an issue (Usher, 1996, p. 10) and distinguish research from everyday questioning and problem-solving.
Each paradigm is underpinned by epistemological and ontological assumptions, with the latter addressing "what is reality" (Usher, 1996, p. 11) and the former addressing the nature of knowledge (Usher, 1996;Morrison, 2012).
Positivists base their knowledge on the assumption that research should be objective, unbiased, value-free, and involve the implementation of quantitative methods (Morrison, 2012;Usher, 1996). For positivists, there is one single reality that exists, "waiting to be discovered" (Feilzer, 2010, p. 6). Usher, however, claims that "human actions are interpretable… hence any knowledge of them is indeterminate" (1996, p. 20). Acknowledging that positivism and post-positivism paradigms are both employed in research, Usher suggests that a positivist epistemology in social sciences research is inappropriate because values can affect findings and "we cannot be positive about our claims of knowledge when studying the behavior and actions of humans" (Creswell, 2014, p. 7).

Research in Social
Unlike positivists, the context within which meanings are constructed and embedded, for them, is vitally important when interpreting data and findings.
Critical theory, however, has challenged the epistemology of both positivism and interpretivism.
According to the Marxist tradition, critical theory challenges the established traditions of society and opposes the stance taken by both positivists and interpretivists. The position critical theorists adopt is that each context provides its own unique perspective. Usher explores the suggestion that critical theory constitutes an "ideology critique" (1996, p. 23). Critical theory primarily aims to engender social change based on consensus and robust arguments: thus, for such theorists, reality is not permanent and enduring but crystallised through dialogic, dialectic, and transformative methodologies (Guba & Lincoln, 1994).
Post-modernism, by contrast, "questions… the foundations of knowledge and understanding" (Usher, 1996, p. 25). Within this framework there is no certainty in truth and therefore social science is full of ambiguity. The binary subjective-objective framework is disrupted by postmodernism, which contends that deeper scrutiny is needed to challenge and question preeminent knowledge (Kelly, 1995;Usher, 1996). According to this argument, reliable and rational science is disclaimed knowledge. Overall, however, researchers do not always specify which paradigm they are working within (Guba & Lincoln, 1994).
Ostensibly, Sawir's article could be situated within the interpretivism paradigm based on its methodology, epistemology, and ontology. The methodology, according to Creswell (2003), represents the research paradigm. Although Guba and Lincoln (1994) suggest qualitative and quantitative research methods can be implemented within any paradigm, interpretivists mainly construct their knowledge through the use of qualitative research methods, such as the interviews employed in Sawir's article. These were therefore not interpreted quantitatively and numerically; Albusaidi instead, the findings were based on an interpretation and understanding of the data by Sawir (Creswell, 2014). This is confirmed by Thanh and Thanh (2015, p. 24) who suggest that "researchers who are using an interpretivist paradigm and qualitative methods often seek experiences, understandings, and perceptions of individuals for their data to uncover reality rather than rely on numbers of statistics".
Sawir interviewed academic staff "to discuss their perspectives" and "comment on the academic and social issues experienced by international students" (2013, p. 364). Creswell (2003, p. 9) argues that interpretivists "generate or inductively develop a theory or pattern of meanings". This was evident in the way Sawir examined the experiences of academic staff and interpreted them qualitatively. Her aim was not to invoke social change, which is a feature of the critical theory paradigm (Usher, 1996;Creswell, 2014), but to 'understand' and 'explore' features of the data.
Hence, the epistemological assumptions of Sawir as an interpretivist underpin her research aims. Creswell (2003), furthermore, highlights the way researchers within the interpretivism paradigm construct their science based on the experiences and perceptions of participants. Creswell's claim can be linked to Sawir's epistemology, which is evidently interpretivist, in that her aim is to 'understand' and 'explore' staff experiences in relation to IoC. Furthermore, she inserted herself into the research (Scott & Morrison, 2006) by personally asking staff about the experiences of international students. Importantly, her previous knowledge did not influence the findings as the participants' own words provided a route to their reality. This shows that Sawir accepts the existence of multiple realities. Scotland (2012) claims that the ontological assumption of the interpretivist is embedded within the topic that the researcher is exploring. This can be seen in Sawir's mission to understand the contribution international students make to IoC. Her ontology is relativist as her exploration leads to different answers and experiences and is thus different "… from positivists who often accept only one correct answer" (Thanh & Thanh, 2015, p. 25). Thus, a concrete reality does not exist; instead, there is a socially constructed reality based upon the experiences of staff.

Discussion and critiquing the selected article
For a novice researcher, critiquing a published research article is an intricate and challenging task.
Moreover, selecting the appropriate criteria introduces yet more complications. Although the big-

Research in Social Sciences and Technology (RESSAT)
2019: 4 (1), 105-122 tent criteria by Tracy (2010) are holistic in nature, I believe they can be implemented to evaluate the quality of all theoretical research and/or a combination of criteria can be created that encourages "imagination, growth, and improvisation" (Tracy, 2010, p. 837). Indeed, the field of qualitative research includes an ongoing dialogue on values and the quality of research, a clear contrast with the consistent aims of quantitative research. Consequently, Tracy and Hinrichs (2017, p. 1) believe it is essential to develop the big-tent model because "there is…great value and need for more standardized criteria". As I am new to the process of critiquing, my position is that the big-tent model provides a proper framework within which to evaluate Sawir's article. This framework compromises eight criteria: worthy topic, rich rigour, sincerity, credibility, resonance, significant contribution, ethics, and meaningful coherence (Tracy & Hinrichs, 2017;Tracy, 2010), all of which are addressed in this paper.

Worthy topic
Tracy (2010) believes a worthy topic is one that is relevant, timely, significant, and interesting.
Over the past 25 years, research papers on internationalisation have increased because of globalisation (Rezaei et al., 2018), "the economic benefits" (Sawir, 2013, p. 359;Trahar, 2007) that accrue, and the number of students travelling to other countries for study and employment (Elliot et al., 2015). This has also been witnessed within higher education institutions (HEIs) in Oman, which compete to provide an education that equips learners to work in a global environment. In regard to HEIs, Australia (Sawir, 2013) and the UK (Trahar, 2007) set different fee structures for international students: conversely, in Oman, local and international students pay the same fees. However, the process of IoC has accelerated in Australia and the UK because of the growing number of international students joining HEIs. Both countries argue that these fees ensure the provision of an adequate and internationalised curriculum. Sawir is not alone in arguing that "many international students led institutions to redesign their curricula" in Australia (2013, p. 361). Haigh (2002), Leask (2005;2015), and Schuerholz-Lehr (2007) detail how Australian and international HEI graduates are being prepared for global positions through IoHE and IoC. Thus, an exploration of IoC is relevant both to Sawir's concerns and the wider international context.
From a narrower perspective, the annual report issued by Central Queensland University (CQU), the institution where Sawir works, states that "international, intercultural and indigenous Albusaidi perspectives will be evidenced in our academic offering" (2012, p. 12). Similarly, the report emphasises the importance of "international prestige ratings" to CQU, which praises research in this area (2012, p. 12). Sawir's findings therefore provide significant insights that will be of value to CQU decision-makers and practitioners. The CQU report shows that in 2010 there were 8,054 international students. This was followed by a sharp decrease in 2012 to 5,779 students. This underlines the timeliness of Sawir's research as it directly relates the experiences of staff in Australia to those of CQU staff. For the same reason, Sawir's topic has educational authenticity as it raises awareness among both CQU staff and decision-makers (Guba & Lincoln, 1989, 2005.
Acknowledging that several factors have led to the decrease in international students, Sawir admits: …too much focus on research on international students has created a gap in the literature of the internationalisation of higher education. To have comprehensive understanding of processes of internationalisation, more research needs to be conducted, and particularly with domestic students as participants. (2013, p. 374) This suggests that interviewing international students will provide more insight into this topic. This is a clear strength of Sawir's article as she had admitted there is a gap in her research and has highlighted this as an area for future research. Finally, although the topic of IoC may seem 'obvious' to some researchers, I believe it is personally of great value in the context of Oman and, moreover, has remained relevant beyond 2013.

Rich rigour
Rich rigour refers to the depth the researcher goes into when discussing the methodology and findings of the topic they are investigating (Tracy & Hinrichs, 2017). Although the discussion of the methodology in Sawir's article covers most of the main concepts, it would have been strengthened further if she had justified the choices made. For instance, one anonymous university was chosen with only prosaic justifications provided for doing so. Sawir (2013, p. 363) based this choice on the fact that it was "the second oldest university in Australia" and its position as "a research-intensive university". However, no clear reasons were provided as to whether and how Sawir was related to that university.

Research in Social Sciences and Technology (RESSAT)
2019: 4 (1), 105-122 Tracy (2010, p. 841) argues that "demonstrations of rigor include the number and length of interviews, the appropriateness and breadth of the interview sample". Sawir, however, only states that semi-structured interviews were implemented. I believe that explaining her reasons for choosing semi-structured interviews over structured or unstructured interviews would have strengthened her methodology. Qualitative or open-ended questionnaires could, arguably, be a viable alternative method of data collection (Page et al., 2014;Schuh et al., 2009). It is therefore not clear why Sawir chose interviews over questionnaires. Moreover, in addition to interviews, she could have used open-ended questionnaires to enrich her findings. This would have been useful as some participants may require time to think about their answers, which is more feasible with questionnaires than it is with interviews. Furthermore, their use would also have enabled Sawir to triangulate her findings, a point that will be returned to later. Nevertheless, due to staff interest, the interviews took approximately 30-60 minutes which would indicate the collection of sufficient data. Although she recorded the interviews, the use of appropriate procedures and practices (Tracy, 2010;Tracy & Hinrichs, 2017), such as an acknowledgment of the gap between recording and transcribing, the length of time taken for transcription, example interview questions and transcripts, and a sample consent form were not explained or provided. The only practice mentioned by Sawir was "reading the text [transcribed interviews] several times" to identify the main themes (2013, p. 364).
In terms of sampling, four out of the ten faculties were selected purposefully, yet the reasons for choosing these venues were not given. Moreover, there were 7,000 staff working at that university, yet only 80 participated in the study. This in itself is less of an issue as qualitative researchers generally tend to work with small samples because they aim for deep and rich data. However, Sawir did not state how long she engaged with the staff in order to build trust, which may affect the credibility of her research. Nevertheless, the thematic approach she used to analyse the data was effectively implemented. Overall, however, explaining 'how' the approach works does not justify 'why' it was used.
I chose Sawir's article to broaden my understanding of the IoC situation in Australia; yet, surprisingly, Sawir explicitly states that "[o]ther findings from the larger study such as the views of academic staff towards the presence of international students, … are written and published separately" (2013, p. 364). As a researcher, this surprised me because a reader will pay to review Albusaidi this version of the article, but many of the results are presented in other versions. This is despite the fact that she does acknowledge that this research is part of a larger study.
Nevertheless, the results and discussion sections are profound in terms of both analysis and evaluation. This includes a full consideration of the features of a thematic approach, such as finding and categorising similar and different patterns (Braun & Clarke, 2006), all of which are deployed and explained by Sawir. Tracy argues that "rigorous analysis is marked by transparency" (2010, p. 841). However, one could ask whether the data are "coded in an epistemological vacuum" (Braun & Clarke, 2006, p. 84). Such a query implies that the data may sometimes be purposely selected to present only the positive aspects of a topic. Braun and Clarke (2006, p. 84) term this "theoretical thematic analysis" whereby data selection is influenced by the researcher's own values and beliefs, reducing the quality of the discussion. In her research, Sawir explicitly shows how the presence of international students was perceived negatively by domestic students, which presents a challenge for academic staff who must strive to encourage the latter to appreciate the life experiences of the former. For example, she notes that "it was a challenge for them [staff] to get domestic students to appreciate and exploit these cultural resources [the life experiences international students bring with them]" (2013, p. 367). Thus, in this case, transparency in the selection of the data is evident.

Sincerity
A sincere researcher would be self-reflexive and transparent (Tracy & Hinrichs, 2017). It could be argued that Sawir has not entirely fulfilled this goal. For example, in the methodology section, she fails to state any aims or limitations beyond choosing a qualitative approach or the use of semistructured interviews. When clarifying the choice of university, she mentions she works for another university. However, she does not state whether this was for "a prolonged period" (Creswell & Miller, 2000, p. 127). This casts doubt over her engagement within the context of the university. her values and experience were not clearly explained. An interpretivist researcher jointly constructs the meaning of the research with the participants (Creswell, 2003). Although Sawir conducted the interviews, the extent to which she was critically engaged with the context and the participants was not made clear. She simply presents themes and supports them with direct quotes from the interviewees. One could therefore question how Sawir validated the stories, experiences, and situations related to her by staff. She did not reflect upon these situations and thus address the essentially subjective nature of qualitative research.
A transparent researcher would also describe how the study was funded, the challenges it presented, and the "twists and turns" (Tracy & Hinrichs, 2017, p. 5) of the research. These remain unstated in Sawir's article. Overall, the inclusion of such information could have justified the methodological practices she implemented. This is crucial as it helps avoid any misunderstanding and removes all doubts about the authenticity of the findings.

Credibility
Researchers 'show' rather than 'tell' readers how detailed their description is (Tracy, 2010). For instance, Sawir demonstrates the richness of her data by supporting each theme with ideas, indepth descriptions, and selected quotes from the participants. In so doing, she improves the credibility of the research. However, Sawir's voice was less clear when contextualising her research in the literature review as she mentions numerous noteworthy studies yet does not question the methodology or context of those studies. For example, a claim such as "international students in tertiary institutions have lowered academic standards" (Sawir, 2013, p. 360) arguably requires deeper scrutiny, yet Sawir fails to mention the context surrounding the studies making such claims or the data collection instruments they employ. Taking the findings of other studies for granted rather than questioning them is a weakness of her article.
In terms of triangulation, Sawir only employed interviews. She did not state why she did so as the use of more than one method is believed to increase the credibility and trustworthiness of research (Flick, 2006;Patton, 2002). One might claim that, because Sawir is an interpretivist, using different methods will yield different realities, none of which are necessarily "true". However, Sawir did not state which paradigm she is working within and why she only used interviews, which means her methodology can be questioned. One point mentioned previously relating to the interviews Albusaidi concerns the extent to which Sawir engaged with the participants. Tracy argues that "[m]ultivocality can also be achieved through intense collaboration with participants" (2010, p. 844). It is therefore possible to argue that the study would have been better suited for CQU as Sawir would have been more familiar with the context and had a better understanding of the participants. Although being both a researcher and a staff member would have presented a challenge, participants may have felt safer revealing their experiences, thus yielding richer and more authentic data.

Resonance
Sawir's study is qualitative and therefore her aim was not to generalise the findings. It could, however, provide significant insights that could be useful in other contexts. From the perspective of an international student, and as a teacher of international students, I believe several findings are particularly noteworthy. For example, Sawir found that the "presence [of international students] has led them [the teachers] to think about their teaching methods" (2013, p. 365). International students come from different backgrounds and have experienced a diverse range of teaching and learning experiences. They will therefore expect to be taught in a similar or perhaps more effective way. Having experienced a similar situation when I moved from being an international student in the UK to teaching international students at a college in Oman, this compelled me to think critically about the teaching methods used in my classroom. Prior knowledge and expectations are not only built on the methods of teaching but also on how an international student is viewed by their domestic counterparts. This is because the lecturer plays a vital role in enabling students to understand and learn from each other. For instance, Sawir states: Interviewees asserted that those domestic students who interacted more with the international students probably gained a much better (or more interesting) learning experience. Developing relationships with domestic students, however, is more complicated and difficult, resulting in divisions between international and domestic students. (2013, p. 368) Sawir here touches on important and profound issues, such as how teachers and local students in Oman should interact and learn from international students. Her findings therefore increased my desire to investigate the Omani context, providing further resonance to her work. Sciences and Technology (RESSAT) 2019: 4 (1), 105-122

Significant contribution
Sawir's study has still made significant contributions to my work as a novice researcher. Her research has provided a deeper understanding of the contribution international students can make.
For example, her research addresses the fact that "there is little research examining the extent to which the presence of international students on campus contributes to internationalisation of Australian education and training" (2013, p. 363). This confirms Tracy's claim that qualitative research "provides new conceptual understandings" (2010, p. 846). Thus, Sawir's study profoundly explains IoC in an Australian university and provides the basis for further research, not just in Australia, but globally. As mentioned previously, her findings have increased my desire to examine IoC in Oman. Overall, her study can be assessed as both theoretically and heuristically significant (see Tracy, 2010). Tracy also argues that "[a] research project that yields unsurprising theoretical findings may nonetheless provide a significant contribution by introducing and explicating a new methodological approach" (2010, p. 846). This may justify Sawir's use of a qualitative approach. In the literature review, she extensively highlighted previous studies in relation to IoC and how international students are perceived and the contribute they make. Beyond her stated aims, her objective was to investigate the topic from within a qualitative paradigm. This may represent the methodological contribution of her research in a local context, which consequently could contribute to other qualitative studies conducted globally (e.g. see Leask, 2013;Leask & Bridge, 2013;Trahar et al., 2015;Trahar, 2014).

Ethics
Sawir has followed ethical practices to at least some degree. She ensured that the name of the university and participants' names and ages remained anonymous, emphasising "[t]o preserve confidentiality interviewees, if quoted, were referred to by a code placed in brackets" (2013, p. 364). Issues such as voluntary participation, arranging convenient interviews, and signing the consent form were all addressed. However, Tracy contends that "we constantly reflect on our methods and the data worth exposing" (2010, p. 847), and Sawir does not explain how she provided a safe environment within which the interviewees could freely express their ideas (Creswell & Miller, 2000). Moreover, she does not state how the recordings and transcribed interviews were Albusaidi saved, and there was no evidence of member-checking, all of which seem to suggest a lack of transparency in regard to ethical research practices.

Meaningful coherence
A meaningful and coherent study connects its methodology to its paradigm and aims (Tracy, 2010). Sawir's interpretivist position thus provides a clear link between employing a qualitative approach and qualitative data collection. She achieves this by inserting herself into the research in order to elicit and understand participants' views and construct meaning from the data using their voices, all of which are commensurate with the goals of her study. Although the research questions are not explicitly stated, the gap the research aimed to fill is clearly addressed, and the findings have largely achieved this. Thus, the methods used to collect and analyse data were appropriately implemented. Furthermore, the section 'A way forward' considers several implications that provide an insight into the concerns and issues surrounding IoC. This is one of the strengths of her research as she situates her study within a wider context. In her conclusion, she states how the aims were achieved. Her study could have been made more coherent by elaborating and reflecting upon the methodology and justifying the particular methods employed. This would help the reader to understand and interpret the methodological rationale for the study, thereby averting any misinterpretations.

Overall evaluation
The weaknesses and strengths of Sawir's study constitute a basis upon which her study can be improved. Some researchers suggest that acting on the findings and implications of qualitative research is not a secure undertaking as it is built on a subjective foundation. Unlike quantitative research, the findings are not generalisable. Nevertheless, Tracy emphasises that "[a] parsimonious framework for qualitative quality can help us communicate value for our work to a variety of audiences" (2010, p. 838). This has empowered me to add a novel flavour to my journey as a novice interested in qualitative research. It has also guided me in critiquing Sawir's article and encouraged me not to take any research for granted. For instance, Sawir could have stated her research questions more clearly as she leaves the reader confused about certain aspects of her study. A clearly stated research paradigm could also have rendered both the methodological aspects (i.e. the approach, the method, the strategy) and the findings more coherent and explicit. Sciences and Technology (RESSAT) 2019: 4 (1), [105][106][107][108][109][110][111][112][113][114][115][116][117][118][119][120][121][122] Additionally, the results and discussion sections could have been combined and Sawir's voice heard more loudly throughout the study. Although the discussion is rich, issues such as member checking, triangulation, peer reviewing, and external audits (Tracy, 2010;Creswell & Miller, 2000) are not addressed in Sawir's article. Specifically, "who is not talking and what is not said" (Tracy, 2010, p. 843) are not clear in her literature review, results, and discussion sections. An improvement in these areas will render her research more credible and trustworthy.

Conclusion
In conclusion, this paper has made me think deeply about writing my own thesis. I feel that using Tracy's model will provide me with an effective way to critique research papers in the future. Each criterion addresses how research weaknesses can be a turning point on which to produce a better study in the future. It is also important to emphasise that, regardless of the limitations of Sawir's study, her research adds to my understanding of IoC and the contribution international students make. Her findings have further increased my desire to investigate IoC in Oman. Although my critique is not exhaustive, it has been invaluable in deepening my understanding of the research process at an early stage of my career.