A Pilot Investigation of Critical Thinking in Undergraduate Students of Communication Sciences and Disorders

Recommended Citation Rehfeld, David M.; Sulak, Tracey N.; Rainer, Deborah; Stewart, Janice; and Welch, Nadine (2021) "A Pilot Investigation of Critical Thinking in Undergraduate Students of Communication Sciences and Disorders," Teaching and Learning in Communication Sciences & Disorders: Vol. 5 : Iss. 2 , Article 2. DOI: https://doi.org/10.30707/TLCSD5.2.1624983591.707528 Available at: https://ir.library.illinoisstate.edu/tlcsd/vol5/iss2/2

Critical thinking is essential for successful practice in the field of speech-language pathology (American Speech-Language-Hearing Association [ASHA], 2018). Although ASHA has acknowledged its importance in pre-service professional development, there remains a dearth of information regarding the critical thinking abilities of undergraduate students pursuing future careers in the field of communication sciences and disorders (CSD; Mok et al., 2008). Many studies investigating critical thinking in the field of CSD have relied on contextualized or researcher-developed outcome measures that make comparisons across the literature difficult. A better understanding of undergraduate students' critical thinking is necessary to ensure that the future of the profession is prepared to find, interpret, and apply evidence in the assessment and treatment of communication disorders-especially as they are bombarded with information from myriad sources of varying quality.
Literature Review. Critical thinking has been defined as "the ability and willingness to assess claims and make objective judgments on the basis of well-supported reasons and evidence rather than emotion or anecdote" (Wade & Tavris, 2008, p. 7). It has also been discussed as a heterogeneous set of skills used to analyze facts, judge opinions, and facilitate goal-directed behavior (Almeida & Franco, 2011). Finn (2011) succinctly stated that critical thinking is "applied rationality" (p. 69), arguing that this skill is necessary for the implementation of evidence-based practice in the field of speech-language pathology. Although critical thinking appears to be a heterogeneous set of skills, it is possible to group them into approximately three broad categories: interpretation, evaluation, and metacognition (Almeida & Franco, 2011;Finn, 2011). In this conceptualization, critical thinking is used to interpret data, evaluate the sources from which they were obtained, and monitor one's own thoughts, motivations, and biases in the process. This framework is directly relevant to the solicitation, evaluation, and implementation of evidencebased practice in speech-language pathology.
More broadly speaking, the development of critical thinking has been previously identified as an integral focus of postsecondary education in the modern era (Almeida & Franco, 2011;Fink, 2013;Huber & Kuncel, 2016;Roth, 2010). Graduate school performance has also been linked to metrics of critical thinking (Behar-Horenstein & Niu, 2011). Behar-Horenstein & Niu's (2011) review of the literature found 42 studies investigating outcomes of interventions reported to develop critical thinking skills in postsecondary students published between 1994 and 2009. In a similar vein, Chan (2016) conducted a systematic review of critical thinking in medical education and found 41 articles published between 1981 and 2012. Chan's review indicated that the majority of studies found were focused on the development of novel teaching methods hypothesized to promote the development of critical thinking. This is consistent with Behar-Horenstein & Niu's finding that much of the existing literature has been focused on teaching critical thinking within specific contexts. These reviews also found wide variation in how critical thinking development has been measured across studies, with some using existing instruments and a large number using studyspecific outcome metrics. Such heterogeneity impedes the meaningful comparison of critical thinking development across studies, especially when considering that many represent specific intervention procedures.
Huber & Kuncel (2016) conducted a meta-analytic review of critical thinking development during the college years, along with the effects of cross-sectional and longitudinal research designs on critical thinking outcomes. The results of their work suggest that college students develop critical thinking skills over the course of their studies-even without formal intervention. This is good news, considering that faculty value the development of critical thinking (DeAngelo et al., 2009), and students believe they are better critical thinkers at the end of their studies (Tsui, 1998). The magnitude of this development, however, may well vary across majors even when measured as a domain-general skill rather than one that is discipline specific. Unfortunately, some previous work has also suggested that current undergraduate students are developing critical thinking at a slower rate than their historical peers (Arum & Roska, 2011). While Huber & Kuncel reported substantial gains in domain-general critical thinking over the college years, much of the existing literature in CSD has focused on domain-specific (i.e., contextualized) measures associated with formal interventions. The general critical thinking abilities of undergraduate students in CSD, however, remains important in considering how these individuals will represent the field to the public and engage in collaboration with members of other professions. As aspiring speech-language pathologists, students must be prepared to think critically about not just the evidence behind their assessment and treatment decisions but also those practices that affect referrals, enrollment, and equitable access to their services. Mok et al. (2008) conducted a review of the literature on developing critical thinking in students of CSD, again stressing its importance in evidence-based clinical decision making. Mok and colleagues focused specifically on the effects of problem-based learning on critical thinking. While their results indicated that problem-based learning is effective in developing critical thinking skills, they did not report on general levels of critical thinking across the student population in CSD. In concluding their review, they cited Wang's (2005, p. 22) assertion that "universities need to deliver not simply specific skills and specific knowledge but also the attitudes, aptitudes and problem solving skills for lifelong learning". Almost a decade later, Procaccini et al. (2016) still reported that many students coming into the field of CSD lack sufficient critical thinking skills and must be taught them during their program. They also argued that skills such as critical thinking that are taught in the classroom do not always transfer to clinical practice and recommended several different models of instruction for use in developing critical thinking in students. These authors, however, did not report any specific quantitative evidence regarding the critical thinking abilities of students unrelated to the investigation of specific intervention approaches.
In order to facilitate students' development of critical thinking skills, Procaccini et al. (2016) described a number of different instructional methods suitable for use in higher education. While they discussed the strengths and weaknesses of several of these approaches, it remains difficult for clinical educators to make decisions regarding which method to use without quantitative data regarding the critical thinking skills-and needs-of their students. Without a clear understanding of if and how critical thinking differs between students at different points in their undergraduate careers, it is unclear whether specific instructional strategies designed to promote it are even necessary-especially given mixed results regarding their efficacy in the current literature (Battaglia, 2020;Behar-Horenstein & Niu, 2011) and evidence that these skills develop on their own to some degree over the course of the college experience (Huber & Kuncel, 2016).
CSD faculty need more data on the critical thinking abilities of their students before investing in novel teaching approaches. The Cornell Critical Thinking Tests (CCTT; Ennis et al., 2005) are one such way to collect these data; they are a pair of domain-general critical thinking ability tests designed to cover the developmental spectrum, with Level Z recommended for use with most adults. The CCTT have historically been some of the most frequently used standardized tests of critical thinking used to investigate pre-and post-test outcomes of critical thinking interventions conducted with postsecondary students (Behar-Horenstein & Niu, 2011). As mentioned above, however, critical thinking is not a unitary construct and there is interdependence between the abilities thought to comprise it. The Cornell Critical Thinking Test -Level Z (CCTT-Z; Ennis et al., 2005) thus allows for the calculation of several scores for each participant, reflecting their performance overall as well as on items measuring induction, deduction, observation and credibility judgments, assumption identification, and meaning interpretation. A description of these constructs based on the work of Ennis (1996) and Ennis et al. (2005) is provided in Table 1.

Interpretation of Meaning
Identifying the influence of the way in which words are used An SLP receives a referral for a child with a fluency disorder. During the evaluation, no stuttering is observed. The teacher is unsurprised because they are concerned about the child's oral reading rate. The SLP then realizes there was a miscommunication during the referral process.
The CCTT has been used in a large number of previous studies outside the field of speech-language pathology, which suggests some measure of face validity in that other researchers have found it to measure what they expected (Behar-Horenstein & Niu, 2011). Ennis et al. (2005) reported correlations of approximately .50 between the CCTT-Z and other frequently used assessments of critical thinking and noted that this was reasonable given "the differences in approach of different test makers" (p. 32). No differential effects of gender or academic achievement have been reported, although the CCTT-Z has been reported to serve as a relatively reliable indicator of early graduate school success and developmental effects have been observed in other studies (Garett & Wulf, 1978;Mines et al., 1990). These specific features are of particular relevance to faculty in CSD, where future employment in the field is dependent on acceptance into and completion of formal graduate training. A shortage of such programs and their associated clinical placements means that acceptance into accredited graduate training programs is highly competitive for undergraduate students. Because of its domain-general construction, ease of administration, and previous use in the literature, the CCTT-Z was used to assess participants' critical thinking abilities in the present study.
Hypothesis, Aims, and Objectives. The aim of the present study is to serve as a pilot in describing and comparing the critical thinking abilities of lower-and upper-level undergraduate students majoring in CSD. The descriptive nature of the present research design precludes causal determinations regarding the development of critical thinking during the undergraduate period. The present study can, however, provide clinical educators and researchers with some insight into the critical thinking abilities of lower-and upper-level undergraduate students majoring in CSD.

Methods
This research was approved by the university's Institutional Review Board prior to the occurrence of any study activities. Because we were interested in examining the performance of CSD students at opposite ends of their undergraduate training, students were considered for participation if they were enrolled in one of two courses during the Fall 2019 semester: a freshman-level introduction to CSD (LL) or senior-level clinical methods course (UL). These two courses were selected because they are both required for all students majoring in CSD at the university, students cannot enroll in them simultaneously, and a review of their historical enrollment records indicated that the majority of students take them during Fall terms. There were no exclusionary criteria used to identify the target sample. By recruiting students from these two courses during the same semester, the present study utilized a cross-sectional research design. Based on the inclusion criterion, the total possible sample for this study was 148. An a priori power analysis was conducted using G*Power (Version 3.1.9.4) to determine the viability of the available sample in powering an independent samples t-test. This analysis indicated that the available sample would be sufficient to detect a medium-sized difference in critical thinking between the two groups, while a small difference would require upwards of 700 participants (Cohen, 1988).

Participant Characteristics.
Major demographic characteristics are reported in Table 2. Based on the results of an independent samples t-test, the groups did not differ significant by gender (p = .750). The two groups differed by age (p < .001); this was expected since the two groups were at opposite ends of their undergraduate training. Both samples consisted of predominately White females, and although specific race and ethnicity data are unavailable, such a majority is consistent with finding that only 8.3% of ASHA members identify as racial or ethnic minorities (ASHA, 2019). Data Collection. In collaboration with authors three through five, the first author met with potential participants during a regularly-scheduled course meeting approximately one to two months into the semester. Data were collected from lower-level students in September while data were collected from upper-level students in October in order to accommodate the course instructors' existing schedules. Therefore, the data collection points were approximately one month apart. Given the existing curricular gap of several years between the two groups, however, this small lag in data collection was unlikely to meaningfully affect the results of the study. The first author provided students with information about the study and time for them to ask questions before obtaining informed consent for their participation. One hundred and forty-two students elected to participate, representing 96% of the total possible sample. After obtaining informed consent, the first author administered the CCTT-Z during the remainder of the single class meeting for each of the courses sampled for this study. Participants were given approximately fifty minutes to complete it in accordance with standardized test administration procedures. All participants completed the CCTT-Z within the allotted time period.
Scores on the CCTT-Z can be calculated in two ways, one of which is a traditional count of the number of correct responses (informally termed "rights only"), while the other imposes a halfpoint penalty for incorrect items in order to discourage guessing. For the purposes of this study, the "rights only" approach was used and both total and subscale scores for each participant were calculated based on the number of relevant items to which they correctly responded. CCTT-Z scales and the number of items comprising them are presented in Table 3. Using the "rights only" approach results in a minimum possible score of zero and maximum possible score of 52. Meaning 15 Observation & Credibility 04 Note. Scale item counts do not sum to the total item count because several items are used to calculate multiple scaled scores. Data Diagnostics. Prior to formal statistical analysis, common data diagnostic procedures were planned; complete data on all CCTT-Z variables were available for all participants. Because complete data were available for all participants, no formal investigation or treatment of missingness was necessary. Additionally, we investigated the tenability of the assumptions underlying the general linear model: score distribution normality, homogeneity of variance, and independence of the error terms. The results of Shapiro-Wilks tests of normality are reported in Table 4, indicating that this assumption was only satisfied for both groups regarding participants' total CCTT-Z scores. The results of Levene's tests investigating the equality of variances between groups was tenable for all measures and are also presented in Table 4. Because participant scores were collected from two independent sources (i.e., two distinct undergraduate courses) with mutually exclusive samples, the assumption of independence is satisfied as well.
While t-tests are generally robust to violations of normality, the significance of the departures in the present results coupled with the size discrepancy between the samples suggests that their use would be less than optimal and likely affected by an inflated Type I error rate (Field, 2018). Accordingly, a series of Mann-Whitney tests were planned to investigate the differences in outcomes between the two samples. As a non-parametric test, the Mann-Whitney test does not require participants' scores to be normally distributed and is a better match for the present data. Descriptive statistics for all CCTT-Z variables are reported in Table 5 with no significant differences observed between the two groups on any variable. While a review of these data suggests no difference between either group means or medians, the results of formal statistical analyses are also reported in Table 6. Although the upper-level students, on average, consistently appear to have achieved scores higher than those of their lower-level counterparts, the magnitude of this difference was statistically negligible for all measures. In fact, when reviewing the median scores for both groups across variables, both groups were equivalent on four of the five areas assessed by the CCTT-Z. Regarding participants' overall performance on the CCTT-Z, a Mann-Whitney U-test indicated that the difference between lower-and upper-level students was not significant (U = 2626, p = .087, r = .144). This non-significance was also observed in the results of Mann-Whitney tests for participants' induction (U = 2241, p = .970, r = .003), deduction (U = 2554, p = .160, r = .118), observation (U = 2424, p = .384, r = .073), assumptions (U = 2448, p = .342, r = .080), and meaning (U = 2553, p = .155, r = .119) scores. All of these group differences represent negligible to small effect sizes, with r = .100 commonly reported as the benchmark for identifying a small effect (Cohen, 1988;Field, 2018).

Discussion
We investigated the difference in critical thinking skills between lower-and upper-level undergraduate students using the CCTT-Z, a standardized measure of global critical thinking abilities, in a sample of 142 participants recruited from a single university. While we anticipated that upper-level students would demonstrate higher critical thinking abilities in comparison to their lower-level counterparts, we found no statistically significant differences between the groups on either global critical thinking ability or any of the specific skills measured by the CCTT-Z. Although the mean scores of upper-level participants were slightly greater than those of the lowerlevel students on all six measures calculated, their median scores were greater on only two: overall performance and deduction. For both groups and on all measures, participants' scores fell in the middle of the range possible on the CCTT-Z. The interquartile range (i.e., the difference between the 25th and 75th percentiles) for both groups consistently overlaps across the CCTT-Z as well, suggesting an overall similar dispersion of CCTT-Z scores for both LL and UL students. Even a review of the maximum scores achieved by each group across subscales finds roughly equivalent performance between the two groups. Based on the data currently available, the reason for the considerable overlap between the two groups is unclear. Although the a priori power analysis indicated that the present sample was sufficiently powered to detect a medium-level effect, the negligible-to-small effect sizes observed in the present data (Table 6) indicate that the difference in critical thinking performance between the two groups-if any-was much smaller than anticipated. It is possible that a larger sample, collected through replication or extension of the present study, would provide better insights into the critical thinking abilities of LL and UL students in CSD. Although no statistically significant results were obtained, it must be acknowledged that small effects were obtained for overall performance on the CCTT-Z (r = .144) as well as the deduction (r = .118) and meaning (r = .119) subscales. Conceptually, this might suggest that UL students have made some gains-albeit small ones-in their overall critical thinking, ability to confirm or disprove generalizations using data, and interpret obscure meaning during their undergraduate training program. While future study is needed, the present small effect sizes in these areas might be explained by natural development of overall critical thinking ability during college and CSD training at the undergraduate level, which involves exposing students to new, sometimes surprising, concepts (e.g., the SLP scope of practice, the idea that sounds and letters do not always have a one-to-one correspondence).
Because there are minimal existing data regarding the critical thinking skills of undergraduate students in CSD, it is difficult to compare the present study to other findings from different universities. The present study only assessed students from a single university. Additionally, the cross-sectional nature of the present study precludes causal determinations. Because the present pilot study utilized a cross-sectional as opposed to a longitudinal design, it is possible that unexplored confounding factors might have affected the results. As such, the present study contributes tentative findings to be confirmed through further replication.
As it stands, data from the current study suggest that the critical thinking skills of lower-and upperlevel undergraduate students in CSD can overlap considerably. Even though specific developmental trends for undergraduate students are unavailable, the results of the present study are consistent with previously reported data on the development of critical thinking dispositions (Giancarlo & Facione, 2001). Although the results of the present study are surprising in that statistically significant differences between the two groups were expected, it is possible that-for whatever reason-critical thinking as it is measured by the CCTT-Z does not develop substantially over the three-year gap between the groups studied at present. Apart from issues of measurement sensitivity, it is also possible that some cohort-related factor not accounted for in the present crosssectional study explains the lack of differential performance (differences in K-12 education, university admissions policies at time of application, etc.).
For those studies in the future also considering the use of a cross-sectional research design, more robust demographic information should be collected from participants in order to account for differences in group composition that might have influenced the present results. Future studies should also utilize a larger sample that would be sufficiently powered to detect even a small effect between the two groups and/or consider the use of a longitudinal research design. It is also possible that the magnitude of critical thinking change between the beginning of freshman and senior years is not significant enough to demonstrate the effect anticipated in the present study. Future studies might consider comparing entering freshmen to exiting seniors or even graduate students to better understand the development of critical thinking across the entire pre-service training period for CSD professionals.
In considering the weak differences observed in the present study, it is possible-and likely-that the findings would be larger if a domain-specific measure were used to assess critical thinking skills as they specifically apply to CSD. Such measures, however, are likely to be biased in favor of upper-level students because of their dependence on content knowledge. These contextualized measures are often used to report the effects of critical thinking interventions both in CSD (e.g., Khamis-Dakwar & DiLollo, 2018;LeJeune & Gunter, 2003) as well as in other fields (e.g., Simpson & Courtney, 2002). It may be that contextualized assessments of critical thinking are an efficient method of measuring students' development of these skills across the scope of our practice. For example, case-based learning might be incorporated into students' academic coursework to gauge their critical thinking both within and, ideally, across classes. Instructors might present students with specific scenarios requiring them to make clinically relevant observations, identify assumptions pertaining to a specific diagnosis, interpret simulated data, and make both inductive and deductive inferences to demonstrate their understanding of and ability to apply course content in novel contexts. The benefit of measuring critical thinking in ways specific to the SLPs' scope of practice is that faculty would have information on students' critical thinking as demonstrated on discipline-specific tasks. The downside, however, is that these contextualized assessments would simultaneously be testing students' content knowledge as well. For students who perform poorly on such assessments, it may be difficult to identify the reason(s) underlying their poor performance-do they lack sufficient content knowledge, struggle with critically thinking and application, or both? The benefit of decontextualized measures such as the CCTT-Z used in the present study is that, to the extent possible, they reduce the number of confounding variables (e.g., content knowledge) faculty must consider in interpreting student performance. Decontextualized measures might also give instructors insight into how critical thinking being taught in the classroom specific to CSD applications is transferring to scenarios outside of students' pre-service professional work. Critical thinking is integral to success across SLPs' wide clinical scope (Finn, 2011) and situational measures of critical thinking may not be sensitive to students' ability to identify, evaluate, and integrate information relevant to clinical practice until the end of their studies-far too late for interventions to address any deficiencies. Researchers, academic faculty, and clinical educators should continue to collaborate in developing effective and efficient methods of developing critical thinking skills in undergraduate students pursuing careers in the field of CSD.

Implications.
Instructors in the field of CSD should continue to integrate critical thinking into their course syllabi at the undergraduate level. Meaningful learning occurs at a variety of levels and promoting students' critical thinking in relationship to specific coursework (e.g., anatomy and physiology, language development) should facilitate their development of domain-specific critical thinking that will benefit them as they prepare for graduate education and training. As students progress through any undergraduate training program, coursework should build upon the knowledge and skills they have learned previously. To achieve this goal, faculty should collaborate to ensure that their courses fully capitalize on what students have been expected to learn in previous coursework while extending that knowledge as they move forward toward graduation. As reviewed in Procaccini et al. (2016), there are a variety of ways to promote domain-specific clinical thing in students' coursework; all of these methods require instructors to integrate one or more elements of metacognition into instruction.
In attempting to stimulate students' critical thinking in line with the suggestions of Procaccini et al. (2016), faculty might choose to incorporate any number of "inquiry-based" instructional methods into their existing coursework. Faculty might, for instance, choose to incorporate problem-based learning into their undergraduate curricula as part of students' end-of-course assignments. For students enrolled in a freshmen-level introductory course such as the one used in the present study, faculty might assign students a problem such as differentiating which conditions would be more likely to be addressed by an audiologist as opposed to a speech-language pathologist. Students would then work together to sort through a list of provided conditions to use their recently acquired course knowledge to solve the problem presented to them; such a task would require students to think critically about scopes of practice for both professions in order to complete the assignment. In a sophomore-level anatomy and physiology course, faculty might assign students a problem requiring them to identify which bodily functions would be likely to be affected by damage to specific anatomy or vice versa. By scaling the difficulty of these problems, faculty have considerable latitude in assessing students' ability to think critically about course content. For more advanced students, such as those enrolled in a senior-level clinical methods course such as the one featured in the present study, faculty might require students to work together in identifying appropriate treatment approaches for different communication disorders. In situations such as the last example, faculty might even utilize the same assignment at multiple times throughout the course for different content units (e.g., fluency, language, voice) to assess students' ability to think critically about in-depth section content rather than assigning one much larger, comprehensive problem or problem set at the end of a course. Each of these suggestions, however, is limited in their ability to measure students' growth over time because they are dependent on students' acquisition of course content knowledge. As mentioned above, it is unclear what faculty's next steps would be if students perform poorly on an end-of-course critical thinking assessment-at this point, it would be difficult to completely tease out students' lack of content knowledge from their ability to critically think through and apply that knowledge.
As instructors focus on developing domain-specific critical thinking in their courses, they can also likely leverage these instructional strategies to promote the development of domain-general critical thinking as well. For example, in coursework on language disorders and interventions, instructors could emphasize the underpinnings of these interventions and the logic supporting or refuting their use prior with the existing evidence base. Discussions such as this can help students to focus on underlying constructs rather than dichotomizing interventions as evidence-based or not. By reviewing several interventions both within the scope of speech-language pathology (e.g., contextualized instruction, discrete trial training) as well as those that are likely to be marketed to caregivers (e.g., essential oils, packaged interventions), instructors can keep activities relevant to the course while also helping students to generalize otherwise domain-specific critical thinking skills. Undergraduate faculty should strive to assess and develop both the domain-general and domain-specific critical thinking of students in CSD in order to develop future professionals capable of interpreting, evaluating, and reflecting on emerging research to best meet client needs.
Future research should investigate the instructional utility of domain-general critical thinking assessments (such as the CCTT-Z used here) as well as domain-specific measures in better understanding the critical thinking abilities and needs of undergraduate students in CSD. Next steps would likely include longitudinal studies of students' critical thinking over the span of their undergraduate-and potentially graduate-careers. Faculty interested in incorporating specific critical thinking interventions into their coursework should also consider systematic methods of data collection and analysis to assess the effects of such instruction on student learning outcomes. For those faculty working in larger institutions with multiple sections of the same course, the field could benefit from data collected via random assignment of these sections to traditional instruction or instruction incorporating specific critical thinking strategies such as problem-based learning, case-based learning, or concept mapping activities.

Disclosures
The first author has no relevant financial or non-financial disclosures related to this research. The second, third, fourth, and fifth authors all receive a salary from the university at which this research was conducted but have no relevant non-financial disclosures.