THE EFFECTS OF INDIVIDUAL INTEREST AND GOAL-ORIENTATION ON ORDINARY AND WORTHY PERFORMANCE

Individual Interest and Achievement-Goal Orientation Scales

The Motivational Strategies for Learning Questionnaire (MSLQ) was used to measure the individual interest levels and achievement-goal orientations of the participants. This questionnaire is a Likert-scale with seven levels where 1 indicates the lowest and 7 indicates the highest level of the attribute. The MSLQ was developed by Pintrich et al. (1991) and the Turkish version of the questionnaire was developed by Büyüköztürk et al. (2004). Looking at significant overlap between task value and individual interest (see O’Keefe & Linnenbrink-Garcia, 2014; Wigfield & Cambria, 2010), the task value dimension of MSLQ was used to measure individual interest because task value is similar to individual interest (McWhaw & Abrami, 2001). It consists of six items scored on a seven-point Likert scale from 1 (not at all true for me) to 7 (very true for me).

The intrinsic- and extrinsic-goal orientation dimensions of MSLQ were used to measure participants’ intrinsic- and extrinsic-goal orientations, which will be referred to as mastery- and performance-goal orientations, respectively. Each of these goal orientation dimensions of the MSLQ instrument include four items. The items belong to both intrinsic- (mastery) goal and extrinsic-goal orientations and are scored on a seven-point Likert scale from 1 (not at all true for me) to 7 (very true for me). All the items intended to measure individual interest, mastery-goal orientation, and performance-goal orientation can be seen in the appendix. Even though MSLQ is a preexisting valid and reliable instrument (see Pintrich et al., 1991) and the validity and reliability of its adapted version is reported by Büyüköztürk et al. (2004), we also piloted the instrument with 108 undergraduates to test its reliability. The data yielded Cronbach alpha internal consistency statistics for task value (individual interest), intrinsic-goal (mastery-goal), and extrinsic-goal (performance-goal) dimensions as .93, .88, and .84, respectively.

Achievement Test to Measure Rote Learning

The researchers developed an achievement test to measure rote-learning outcomes on the Critical Information Seeking and Reporting (CISR) course which is developed by the researchers to conduct this study. Details about the course is provided in procedure section of the paper. In parallel with the objectives and the content of the course which are outlined using table of specification, the achievement test covered the following topics: basic web concepts, Boolean operators and search engine filters, evaluation criteria for online information source, information literacy, basics of text structure, ethical use of information source, APA style citation, and APA style referencing. All questions in the test intended to measure lower cognitive skills (i.e., remembering, and understanding) because the test is developed for measuring rote learning only.

The achievement test was piloted with 76 undergraduate students. Based on item-total correlations, six items demonstrating weak or negative correlations were removed. After removing six items, we proceeded with 28 remaining multiple-choice questions. KR-20 and split-half reliability were computed for internal consistency. KR-20 statistic and Gutman split-half coefficients were .78 and .82, respectively. Based on the results of these analyses, test reliability was acceptable. Item difficulties were ranged between .20 and .80 with a mean of 0.54. Most of the items fell into the moderate difficulty range. Discrimination indices of items varied between .21 and .89 with a mean of .48. Content validity was assessed as suggested by Davis (1992). The final version of the achievement test was reviewed by nine subject-matter experts for adequacy, representativeness, comprehension, ambiguity, and clarity. Based on the reviewers’ ratings on adequacy, a content validity index for each item was computed as described in Yurdugül (2005). These indices were then compared to the critical value of .80 as suggested by Davis (1992). Because all item validity indices were above the critical value of .80 and considering the content validity of these items, validity of the achievement tests was adequately high.

Performance Task to Measure Meaningful Learning

To measure the meaningful learning outcomes of the participants on the CISR course, the researchers designed and developed a performance task by carefully considering the objectives of the course and situational assessment. The performance task had four primary sections: (1) finding and evaluating sources of information, (2) writing concise paragraphs, (3) providing an in-text citation, and (4) creating a bibliography according to the APA Seventh edition writing guidelines. Each participant was given a performance score based on a rubric containing carefully selected criteria that match the skills and knowledge taught during the CISR course. The opinions of field experts and instructional designers were taken while designing and developing the performance task as well as the rubric. While designing the performance task, the following steps were followed:

• Identify content and skills in accordance with course objectives,

• generate initial task ideas,

• choose from alternative performance task ideas through peer discussion,

• design and develop the task,

• develop the scoring rubric,

• develop performance criteria for success,

• review and revise the performance task.

The performance task intended to measure higher cognitive skills (i.e., applying, analyzing, evaluating, and creating) since the main purpose of the task is to assess meaningful learning.

Measure of Worthy Performance

Human competence is a function of worthy performance (WP), which is defined as the ratio of valuable accomplishments (A) to costly behavior (B), which can be formulated as WP = A/B (Gilbert, 2007). To calculate the worthy performance in our case, we divide task performance (i.e., meaningful learning) by the cognitive effort the participant uses during task completion. Above we have presented how the task performance was measured. Now we turn to explaining the assessment of cognitive effort in our study.

The cognitive effort that participants use during task performance is estimated via three different variables: (1) the used attentional working memory capacity, (2) depleted self-control resources, and (3) task duration. The literature suggests that the two predominant measurement techniques for cognitive effort are the dual-task and muscle strength techniques. The dual task relies on the fact that attentional working memory has a limited capacity for which two tasks compete when they are simultaneously performed; the muscle strength relies on the assumption that depletion of cognitive resources resembles muscle fatigue. Further, the dual-task technique is grounded in attention theories while the muscle strength technique is grounded in the strength model of self-control. Moreover, Christensen-Szalanski, (1980) and Cooper-Martin (1994) suggest taking task duration into account while measuring cognitive effort. Therefore, we measure cognitive effort by a single composite score that is the weighted sum of the measures of the following: (1) allocated attentional working memory capacity, (2) depletion in the limited self-control resource, and (3) task duration.

To measure attentional working memory capacity, the computer-assisted tool ScriptKell (see Piolat et al., 1999, for more detail) is used throughout the performance task. This technique and the tool have been successfully applied in many studies to measure cognitive effort. (e.g., Alves et al., 2008; Olive et al., 2009; Piolat et al., 2008). Based on the muscle strength technique, Muraven et al. (1998) suggest using a handgrip to measure cognitive resource depletion based on the assumption that depletion of cognitive resources resembles muscle fatigue. In this measurement technique, participants squeeze a handgrip before and after the performance of a cognitive task. The difference between pre- and posthandgrip squeeze time is used as an indicator of depleted cognitive resource during the performance task. This technique has also been used to measure cognitive effort exertion in the cognitive psychology literature (e.g., Baumeister et al., 2007; Muraven & Baumeister, 2000; O’Keefe & Linnenbrink-Garcia, 2014). Lastly, because longer persistence on the task leads to more cognitive effort exertion, each participant’s task duration is also measured using a stopwatch.

After measuring all three aspects of cognitive effort, we conduct a principal component analysis (PCA) to take a proper linear combination of the three measures and obtain a single composite cognitive effort score. The worthy performance score is then obtained by dividing performance score (i.e., meaningful learning score) by composite cognitive effort score.

The CISR Course Is Designed and Developed

The course was designed following the steps described in the Layers of Necessity Instructional Design Model. A situational assessment was conducted first in a public university at which the participants are enrolled to gather information regarding the participants’ academic writing abilities and information literacy skills. The situational analysis was conducted by carefully reviewing students’ existing work (i.e., project proposal materials, written homework, etc.) as well as the content of the project management course offered by their department. Additional information was gathered from academic staff at the department for preschool teacher training regarding their students’ knowledge and skills in information literacy. Information gathered through the situational assessment was used to conduct an extensive goal analysis where the goals and objectives of the course are written. Instructional strategies corresponding to the achievement goals of each group were then determined. The course covers the following topics: information literacy, Boolean operators, search engines and their filters, evaluation of source of information, ethical use of information, basics of writing structure, and citing and referencing information properly.

Next, the achievement test and performance task, discussed in the measures section were developed. To determine the extent of each assessment tool (i.e., achievement test to measure rote learning, and performance task to measure meaningful learning), Bloom’s revised taxonomy and table of specification created by the researchers under supervision from two curriculum and instruction academics were used. Based on their recommendations, only two levels in the cognitive domain of Bloom’s Revised Taxonomy (remembering and understanding) were taken into consideration while forming the achievement test items. To evaluate instruction, the researchers consulted with subject matter experts (i.e., academic staff of the Computer Education and Instructional Technology Department) and checked adequacy of the instructional strategy through one-to-one formative evaluation trials with one senior undergraduate student. The necessary revisions were then carried out.

The Course Is Presented to the Participants

The content coverage, instructional objectives, teaching strategies and materials, and assessment strategies and tools of the CISR course were explained to the participants.

The Participants Are Grouped Based on Their Interest Levels and Goal Orientations

To determine the individual interest levels and the achievement-goal orientations of the students, the respective instruments, discussed in the measures section, were administered. First, the participants were divided into two groups based on their disposition on achievement goal types. To identify their disposition, mastery- and performance-goal orientation scores were used. To be able to make a meaningful comparison, the natural logarithm of the ratio of mastery- and performance-goal orientation scores (i.e., ln(Xmastery/Xperformance) were used. The scale for these scores varied between −1.95 and 1.95 (i.e., ln[4/28] to ln[28/4]), where the minimum and maximum available scores for these achievement-goal orientations were 4 and 28, respectively. In this scale, a score of 0.00 indicates that the ratio is 1.0, or the participant’s mastery- and performance-goal orientations are equal. To make sure that the groups were distinct, the participants with ln[mastery-goal/performance] scores around 0 (i.e., between −0.1 and +0.1) were excluded from the study.

Next, based on the participants’ responses to the individual interest questionnaire, the participants were further divided into three groups within the two achievement goal groups (i.e., mastery goal group, and performance goal group). These three groups were representative of high-, moderate-, and low-level individual interest. Participants were divided using the median split technique. Because we are more interested in the effects of high and low individual interest in the dependent variables, moderate-interest participants were omitted.

After extracting (1) the outliers, (2) the participants without any predominant goal orientations, and (3) the participants with a moderate level of individual interest, our final four groups consisted of 125 participants out of an initial 187. The final groups and the number of participants in each group are given Table 1.

TABLE 1 Number of Participants Fall into Each Group Defined by Independent Variables

View Fullscreen

Achievement Pretest Administered

After setting the experiment the participants’ preknowledge of CISR course content was assessed with an achievement test. The achievement test was administered to all groups simultaneously.

CISR Course Is Implemented

During the four weeks of CISR instruction, mastery-goal oriented participants received mastery-goal support. For this, in-class strategies suggested by Svinicki (2010) for fostering mastery-goal orientation were followed. The instructor acted as role model, enabled participants to learn from their own mistakes through constructive feedback, provided participants with options during instructional activities, emphasized the importance of the learning outcomes of the course, and consistently reminded participants of the effects of the course outcomes on their personal development.

In a similar vein, performance-goal oriented participants received performance-goal support during the instruction. To foster performance-goal orientation, the instructor suggested participants strive for success by being in the top twenty percent in the class. The instructor also consistently informed participants that their achievement scores would affect their final grades as the CISR course was part of the project management course in which they were enrolled. Moreover, during the in-class activities, the instructor awarded high performers with energy drink.

Achievement Posttest Is Administered

After the CISR course was implemented, the postknowledge of the participants was assessed with an achievement test. The achievement test was administered to all groups simultaneously. The gain score between post- and preachievement test was used as an indicator of rote learning. This rote learning data was gathered and analyzed to answer the first research question.

Task Duration Measurement Procedure

The task duration (i.e., elapsed time in which each participant completed the task) was measured using a stopwatch.

Self-control Resource Depletion Measurement Procedure

To measure self-control resource depletion, each participant squeezed a handgrip before and after performing the task. The duration of the participants’ squeeze was measured in seconds using a stopwatch. The gap between pre- and posthandgrip squeeze time scores counted toward the amount of depleted self-control resource during the performance task.

Used Working Memory Capacity Measurement Procedure

To measure allocated working memory capacity to perform the task, the software program ScriptKell (Piolat et al., 1999) was used. This program requires participants to respond to an auditory probe by right clicking a mouse connected to another computer running ScriptKell. This can be considered as giving a secondary task to participants with the aim of assessing how much attention was paid to the primary task (i.e., the internet-based performance task to measure the participants’ meaningful learning). In the secondary task, we first took the baseline of participants’ auditory probe response time before starting the primary task.

During the measurement of baseline response times, participants were presented with 14 auditory probes (i.e., “beep” sound), and were asked to give an immediate response. These 14 baseline auditory signals were presented within 5 to 15 seconds as suggested by Kellogg (1988). The primary and secondary tasks were then performed simultaneously. The auditory signals throughout the primary task were presented every 90–120 seconds. The program provided the mean response time for each participant. The attentional working memory capacity allocated to the primary task was then estimated by the delay in responding to the auditory probe in the secondary task. This delay is determined by the difference between the mean response times and the baseline response time (i.e., mean response time to auditory probes in the absence of the primary task) and the response time during the primary task (i.e., mean response time to auditory probes in the presence of the primary task).

Generating a Worthy Performance Score

These three measures on task duration, self-control resource depletion, and allocated attentional working memory capacity in task performance were combined into a single composite cognitive effort score. We conducted a Principal Component Analysis to determine the proper weights for totaling the three measures. Because the largest variation in the matrix of the measures is explained by the first principal component obtained from PCA, the Eigen-vector loadings for the first principal component were used as the weight-vector in totaling the three measures. Finally, the worthy performance scores were calculated for each participant by dividing their performance task scores by their composite cognitive effort score. The data for worthy performance was gathered and analyzed to answer the third research question.