國立虎尾科技大學 |

The impact of background characteristics on rater severity in a cross-discipline composition assessment.

Record Type:	Language materials, manuscript : Monograph/item
Title/Author:	The impact of background characteristics on rater severity in a cross-discipline composition assessment./
Author:	Riemer, Deborah K.
Description:	1 online resource (171 pages)
Notes:	Source: Dissertation Abstracts International, Volume: 76-10(E), Section: A.
Contained By:	Dissertation Abstracts International76-10A(E).
Subject:	Educational evaluation. -
Online resource:	click for full text (PQDT)
ISBN:	9781321742381

The impact of background characteristics on rater severity in a cross-discipline composition assessment.
Riemer, Deborah K.

The impact of background characteristics on rater severity in a cross-discipline composition assessment. - 1 online resource (171 pages)

Source: Dissertation Abstracts International, Volume: 76-10(E), Section: A.

Thesis (Ph.D.)

Includes bibliographical references

This study investigated the extent to which rater background variables influenced rater inclinations towards severity or leniency when evaluating students' performance on six composition criteria. The raters in this study were faculty members within the context of a university online classroom environment. Participants included 144 faculty members, representing multiple and diverse programs from four university colleges, who evaluated and scored the performance of 3,149 students on select integrated classroom assessments. The raters utilized an electronic rubric to score and record student performance. The subjective nature of performance evaluations, however, may undermine the validity of the resulting assessment data. A systematic source of measurement error can be attributed to rater bias, and more specifically the level of severity or leniency by which raters score student performance. Not so clear, however, is the source of such rater bias. This study sought to determine whether certain rater characteristics impacted a tendency towards severity or leniency when scoring performance assessments. The rater background variables studied were gender, nationality, educational attainment, teaching experience, academic discipline, and linguistic background. This study was guided by validity theory and situated within the context of the online classroom environment. The quantitative analysis utilized FACETS software and the multifaceted Rasch methodology to provide measurements of rater severity and leniency. The results indicated that approximately 33% of the total faculty-student variance could be explained by rater bias. Further analysis indicated a significant degree of bias in approximately 4.2% of all faculty-student interactions, with 79.5% of those on the more severe end of the spectrum. The gender, educational attainment, and teaching experience of participants indicated no significant influence on severity and leniency. Nationality and linguistic background could not be measured due to the homogeneity of the demographic information collected for these variables. Academic discipline, on the other hand, indicated a significant (p<0.05) influence on bias when considering faculty members from the business and management discipline, suggesting a tendency towards leniency. While further study is warranted, implications for discipline-specific bias could necessitate modifications to curriculum and assessment development, methods for student evaluations, and development opportunities for faculty.

Electronic reproduction.
Ann Arbor, Mich. :
ProQuest,
2018

Mode of access: World Wide Web

ISBN: 9781321742381Subjects--Topical Terms:

555501
Educational evaluation.
Index Terms--Genre/Form:

554714
Electronic books.

The impact of background characteristics on rater severity in a cross-discipline composition assessment.
LDR:03858ntm a2200361Ki 4500 001 912091
005 20180605073454.5
006 m o u
007 cr mn||||a|a||
008 190606s2015 xx obm 000 0 eng d
020 $a 9781321742381
035 $a (MiAaPQ)AAI3702767
035 $a (MiAaPQ)capella:17125
035 $a AAI3702767
040 $a MiAaPQ $b eng $c MiAaPQ
099 $a TUL $f hyy $c available through World Wide Web
100 1 $a Riemer, Deborah K. $3 1184289
245 1 4 $a The impact of background characteristics on rater severity in a cross-discipline composition assessment.
264 0 $c 2015
300 $a 1 online resource (171 pages)
336 $a text $b txt $2 rdacontent
337 $a computer $b c $2 rdamedia
338 $a online resource $b cr $2 rdacarrier
500 $a Source: Dissertation Abstracts International, Volume: 76-10(E), Section: A.
500 $a Adviser: Rod Sims.
502 $a Thesis (Ph.D.) $c Capella University $d 2015.
504 $a Includes bibliographical references
520 $a This study investigated the extent to which rater background variables influenced rater inclinations towards severity or leniency when evaluating students' performance on six composition criteria. The raters in this study were faculty members within the context of a university online classroom environment. Participants included 144 faculty members, representing multiple and diverse programs from four university colleges, who evaluated and scored the performance of 3,149 students on select integrated classroom assessments. The raters utilized an electronic rubric to score and record student performance. The subjective nature of performance evaluations, however, may undermine the validity of the resulting assessment data. A systematic source of measurement error can be attributed to rater bias, and more specifically the level of severity or leniency by which raters score student performance. Not so clear, however, is the source of such rater bias. This study sought to determine whether certain rater characteristics impacted a tendency towards severity or leniency when scoring performance assessments. The rater background variables studied were gender, nationality, educational attainment, teaching experience, academic discipline, and linguistic background. This study was guided by validity theory and situated within the context of the online classroom environment. The quantitative analysis utilized FACETS software and the multifaceted Rasch methodology to provide measurements of rater severity and leniency. The results indicated that approximately 33% of the total faculty-student variance could be explained by rater bias. Further analysis indicated a significant degree of bias in approximately 4.2% of all faculty-student interactions, with 79.5% of those on the more severe end of the spectrum. The gender, educational attainment, and teaching experience of participants indicated no significant influence on severity and leniency. Nationality and linguistic background could not be measured due to the homogeneity of the demographic information collected for these variables. Academic discipline, on the other hand, indicated a significant (p<0.05) influence on bias when considering faculty members from the business and management discipline, suggesting a tendency towards leniency. While further study is warranted, implications for discipline-specific bias could necessitate modifications to curriculum and assessment development, methods for student evaluations, and development opportunities for faculty.
533 $a Electronic reproduction. $b Ann Arbor, Mich. : $c ProQuest, $d 2018
538 $a Mode of access: World Wide Web
650 4 $a Educational evaluation. $3 555501
650 4 $a Educational tests & measurements. $3 1180442
650 4 $a Curriculum development. $3 1148494
655 7 $a Electronic books. $2 local $3 554714
690 $a 0443
690 $a 0288
690 $a 0727
710 2 $a ProQuest Information and Learning Co. $3 1178819
710 2 $a Capella University. $b School of Education. $3 1179598
773 0 $t Dissertation Abstracts International $g 76-10A(E).
856 4 0 $u http://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=3702767 $z click for full text (PQDT)