A Collaborative Data Driven Teaching Experiment

The below case study details how two teachers collaboratively implemented a series of common assessments and analyzed data to alter instruction with the stated goal of improving student achievement in a public middle school.

Assessment Methodologies

Teacher A and Teacher B decided to collaborate and analyze student achievement data based upon pre-test, formative and summative common assessments for one unit of study.  The teachers agreed to focus these common assessments towards two student performance standards:

1. Student understanding of US History content
2. Reading comprehension of primary and secondary source documents

Before any instruction was provided in the US History classes, both teachers gave students the same pre-test. The use of Einstruction's CPS clicker response pads expedited the grading time for these assessments.  This pre-test clicker data was disaggregated by two standards, those questions that were content-specific or questions assessing reading comprehension skill.

The ability to quickly use student achievement data was also improved by exporting CPS clicker data directly into an excel spreadsheet (CSV) format.  This data was then uploaded onto a google doc to enable real time collaboration and analysis between the teachers.

Pre-test results

Comparing the pre-test performance of students between teacher A and teacher B yields a few differences statistically however,  analyzing the combined pre-test performance demonstrates a larger sample size for later post-test analysis with the goal of determining effect size.  Not surprisingly, the teachers learned that students knew very little about the upcoming content of the unit with the combined average score on all content specific questions for all students was 44%.

The data also showed all students assessed using the pre-test had an average total score of 47%.  On average, students did marginally better on the reading skills questions, 57%.  One interesting statistic to note was the large combined standard deviation of 15 points from the average score for all students on the reading skill questions.

This is an interesting statistic to note as it indicates that we have a wide range of student reading ability in our classes.  See the pre-test student achievement data summary below from google docs.

Formative Assessment of Reading Standard Proficiency

Next both teachers provided a formative common assessment to all students to gauge change upon only the reading standard.  It is important to note that the formative achievement data is based upon a very small sample size of questions and this potentially skewed the results below.  

Teacher A: 87%

Teacher B: 88%

Summative Data Results

Following the formative assessment results teachers A and B decided to use different teaching  methodologies to address student reading comprehension.  Teacher A partnered students based upon proficiency levels proven from pre-test results and formative results.  Teacher B focused upon active reading and thematic reading skills and introduced and reviewed text reading strategies.

After altering instruction following the pre and formative assessment results a summative assessment was provided and an effect size was calculated based upon the formula below:

Teacher A:

Sample: 53

Mean: 88%

Std Dev: 11.77

Mean reading standard: 81%

Mean content standard: 91%

Effect Size = 1.681

Teacher B:

Sample: 38 students

Mean: 90.7%

Std Dev: 8.118

Mean reading standard: 83%

Mean content standard: 93%

Effect size = 1.513 

Analysis

Comparing the student achievement data throughout the assessment cycle of pre/formative/summative common assessments it is clear the student achievement increased.  Students gains were greatest on content standards largely due to a lack of previous knowledge.  Student achievement results on reading standard also increased but not at a similar rate.  Comparing the standard deviation on the assessment cycle both teachers significantly narrowed the variability of scores while increasing performance. This data indicates that both teachers were able to raise the achievement for all students and narrow the gap between proficiency and non-proficient students.  In addition, the effect size data clearly shows the teaching methods used, while divergent, were both highly effective instructional strategies.  It is difficult to isolate the causality in this case study and it is important to stress the extremely small sample size.

From MCAS to Teacher Evaluation

A recent task force report from the Massachusetts Department of Elementary and Secondary Education proposes to overhaul the teacher evaluation system across the Commonwealth.

Described in a NPR article, the task force recommendations call for teachers to be evaluated using results from two types of student assessment, one of which must be the growth data from the Massachusetts Comprehensive Assessment Systems exam where it applies. This task force analyzed the current teacher evaluation systems and provided guidelines for improvement. This entire NPR article available online here.

This task force published the report "Building a Breakthrough Framework for Educator Evaluation in the Commonwealth". The entire report is available online and it outlined the below reasons for changing the existing system on page five:

The Task Force concludes that current educator evaluation practice in Massachusetts:

• Rarely includes student outcomes as a factor in evaluation
• Often fails to differentiate meaningfully between levels of educator effectiveness
• Fails to identify variation in effectiveness within schools and districts
• Rarely singles out excellence among educators
• Does not address issues of capacity, or “do-ability”
• Fails to calibrate ratings, allowing inconsistent practices across the state
• Fails to ensure educator input or continuous improvement
• Is often under-resourced or not taken seriously"

The task force recommends that evaluators use a wide variety of other local, district, state or commercially-available standardized exams. In addition the recommendations include student work samples can also be used and that teachers should also be judged during classroom observations on elements such as instruction, student assessment and curriculum measures.

These changes were recently described in a Boston Globe article titled, Rating Teachers on MCAS Results: Sweeping changes pushed by state education leader on April 17, 2011.

The Boston Globe article states that the new teacher evaluation system "also gives teachers who do not make the grade a year to show improvement or face termination. A fiery debate subsequently emerged over how much weight testing data should have in determining the overall effectiveness of a teacher or administrator." The entire Boston Globe article can be found online here.