At Michigan Technological University, we offer approximately 23 sections of first-year engineering courses (ENG1001, ENG1101, and ENG1102) every fall semester. For course assessment and accreditation reporting, it is important to have a reliable metric of student performance. Perhaps even more important is for this metric to produce comparable results when used by different instructors. We reviewed the reliability of a rubric developed by Washington State University. For our courses, this rubric was not applicable for all assignments and not reliable between instructors teaching different sections of the same course. Therefore, the rubric was modified to reduce inconsistencies in grading between different instructors and standardize it so that the same rubric could be used for first-year engineering technical communication assignments. Our current Technical Communication Rubric looks at the following criteria:
- Identifies, summarizes the problem, question, or issue. (r = 0.670)
- Identifies the assumptions and/or constraints and how they apply to the problem. (r = 0.576)
- Addresses specific technical requirements of assignment. (r=0.668)
- Data Presentation and Evaluation: analyzes supporting data/evidence (r = 0.752) AND discusses implications and findings (r=0.891)
- Conclusions: identifies conclusions based on previously discussed material (r=0.458) AND clearly discuss implications and recommendations (r=0.588)
- Communicates effectively. (r>0.7)
We used a Pearson-r Correlation was use to assess the inter-rater reliability of the rubric between a single instructor an the TAs in three different sections (26 student teams). Correlation coefficients above 0.70 are considered adequate; those above 0.80 are the norm in establishing reliability. All results were significant at the p=0.01 level except conclusions (identifies conclusions based on previously discussed material), which was significant at the p=0.5 level. The highest correlations appeared for data presentation and analysis and communicates effectively. These results indicate that the rubric is very reliable for these two criteria.