The Effects of a Site-based Teacher Professional Development Program on Student Learning

A mixed methods study is used to investigate the effectiveness of a professional development program intended to enhance teacher knowledge and student learning so as to systematically improve student achievement in elementary literacy. In this study, a large urban school district partnered with a local university to provide intervention in a Title 1, low-performing elementary school. Measures included teacher knowledge and practices based on surveys, classroom observation, and student achievement data. Teachers self-reported their perspectives on school-based teacher training in terms of its significance, requirements, challenges, and possible solutions to teacher training. Schools were selected based on their Adequate Yearly Progress (AYP) in reading/ language arts’ status. Adequate Yearly Progress (AYP) is a measurement defined by the federal No Child Left Behind Act that allows the U.S. Department of Education to assess the academic performance of every public school and school district in the country by using the results on standardized tests. The collaboration effort involved supporting the school’s goal of enhancing reading, language arts, and math achievement of students by providing interventions targeted toward grades four and five.


Introduction
The paper features a school district's collaborative effort with an institute of higher education to provide professional development programs to build teacher capacity and thereby improve student learning. Increasingly, research confirms that teacher and teaching quality are the most powerful predictors of student success. As Linda Darling Hammond says, "we can have all kinds of educational reforms underway in the US -curriculum reform, governance reforms and so on; but at the end of the day, if you don't have a strong, qualified teacher in the classroom, nothing else in education can work" (Darling-Hammond and Oakes, 2019). By investing in teacher development, districts assure higher student achievement. School divisions are constantly making significant efforts to retain teachers by providing teacher development opportunities. Customized, sustained professional development programs that align with the needs of both the schools and the staff is an important means of retaining high-quality teachers. Professional learning culture must be fostered in the school to create and sustain the dynamics of ongoing learning. Although professional development is a viable and effective way to improve student achievement in reading, old models consisting of single workshops presented by outsiders that lack an in-depth understanding of the school, community, and the curriculum are not effective enough for today's teachers and students. In this study, an American university's College of Education faculty members helped provide various professional development and other activities for staff and students at a local elementary school. This collaborative effort was aimed at supporting the school's goal of enhancing reading, language arts, and math achievement of its students by providing interventions that were targeted at grades four and five.

Research on Continuing Sustainable Professional Development
The need for robust and systematic teacher preparation and development has gained increasing momentum. Various research studies have highlighted the benefits of training that targets the professional development of in-service teachers with the goal of improving teacher quality (Day, 2002;Niemi, 2015). Research clearly indicates that the quality of teaching has a significant impact on the learning quality of students (Anderson, Hiebert, Scott, & Wilkinson, 1985). Having highly qualified teachers in the classroom "does more to assist students who are academically at-risk than any other policy-controllable issue" (Denson, 2001, p. 34), including smaller student-teacher ratios (Darling-Hammond, 1999;Fuller, 1999). High-quality professional development training positively influences student achievement in general, including student achievement in reading (Wharton-McDonald, Pressley, & Hampston, 1998). As a result, school districts frequently utilize professional development training to improve reading instruction as well as student learning (Kinnucan-Welsch, Rosemary, & Grogan, 2006). In providing professional development training for reading, it is imperative that school districts have a clear framework for designing and implementing such programs.
Student achievement is clearly influenced by the capacity of the individual classroom teacher (Youngs & King, 2002). Students entering school from economically disadvantaged homes are more likely to have difficulty developing early literacy skills, leading to poor academic performance. As teachers realize that their instruction can have a direct impact on student learning outcomes, they often become motivated to strive for excellence in their instruction. This motivation improves the quality of teaching, resulting in greater student achievement. Results of various studies support this push for improved teacher professional development to, in turn, improve student academic performance (Kinnucan-Welsch, Rosemary, & Grogan, 2006).
The impact of a teacher's strong knowledge base and effective teaching skills on student test performance is evidenced by statistically significant correlations across multiple states (Darling-Hammond, 2000). In addition, research supports the need for improved teacher preparation due to low efficiency or low-quality teacher education programs. Teachers are constantly given additional tasks, and their workloads are increasing due to increasing state standards and pressure to promote effective change in student achievement. However, these expectations are arguably difficult for a teacher to achieve without proper training or professional development. This study was an attempt to support a school's goal of improving its students' reading, language arts, and math achievement. This was done by training both pre-service and in-service teachers in language arts, reading, and math competencies, with a focus on the interventions targeted toward students in grades four and five.

Research Questions
The present study investigates the effects of school-based teacher training on teachers' base knowledge, using a cohort of twelve teachers who were given university-led courses and workshops. The following three questions were used to guide the research study: • How well did teachers master the content and skills covered during the professional development activities?
• To what extent were the content and skills covered during the training implemented in the fourth and fifth grade classrooms?
• How did the fourth and fifth grade students perform on measures related to reading achievement after their teachers went through the professional development training?

Research Design
The mixed method evaluation design used both qualitative and quantitative data to examine participant perceptions and experiences, methods of instruction, and student achievement. To that end, we relied on various data sources to address the research questions outlined above. These sources included teacher questionnaires, teacher grades in the course, classroom observations, and student achievement scores related to reading. The study analyzed data from the teacher participants, classroom observations, and analysis of student achievement data.

School Demographics and Participants
The school that participated in the study was a Title 1 school with 780 students enrolled. As many as 98% of the students were African American and 95% received free or reduced-price lunch (a determining factor toward Title 1 eligibility). Based on the state's Standards of Learning (SOL) test scores, the school had been accredited with warning for the last two years. Schools with this rating receive extra help to improve their standing by developing a two-year improvement plan with the assistance of local agencies. In order to be fully accredited, an average of 70% of the students enrolled in the school must pass the SOL tests in each subject area at each grade level. Table 1 shows the SOL scores from the previous two years by grade level and subject area. The scores show that the criteria for full accreditation was not met across multiple subject areas and grade levels.
Target grade levels for the study were fourth and fifth grade students with 12 full-time teachers from the school also participating in the study. Three university instructors provided teacher training workshops and courses on Reading and Math. Table 1 shows the consistent low performance of the school's third and fifth graders, which are the two grade levels in which students are tested for state Standards of Learning (SOLs). It is evident from the table that the average scores in subject areas such as English, Math, History, Science, Writing, and Technology are significantly below state averages.

Intervention
During the fall semester, university faculty taught a reading course titled Survey of Reading Instruction for three hours once a week during evenings. The teachers in the participating school were given priority enrollment, tuition waivers, and the opportunity to earn three graduate level course credits. This was a foundational level course in literacy education. Of the 30 candidates enrolled in the course, 12 teachers were from the target school where the intervention took place. A list of course competencies appears in table 2. In addition, two professional development workshops were offered. Two workshops, one each semester, were conducted on early release days so that the teachers were available. Throughout the academic year, there were more than twenty other workshops and follow-up sessions provided by university faculty at the school site during the universal planning time for the fourth grade and fifth grade teachers. Nearly all of the fourth and fifth grade teachers attended each of the respective sessions. In addition, the school principal, reading specialist, and special education teachers were frequently in attendance.
Most of the topics covered during these professional development workshops led by faculty members at the university focused on classroom reading education strategies. The university faculty members informally surveyed the teachers and administrators on their curricular needs and requests. Based on these responses, two series of workshops were conducted on classroom math and technology strategies, which were identified as the teachers' curricular needs. One example of a topic covered in a technology workshop was the Inspiration software package. A list of specific topics covered in the three domains of Reading, Mathematics and Technology during the workshops appears in Table 3.

Teacher Course Questionnaire
Two questionnaires were developed to gain feedback from participating teachers. The first questionnaire was administered to teachers who were enrolled in the reading course. The second was administered to all fourth and fifth grade teachers from the participating school who were also enrolled in the course. The questionnaires provided both quantitative and qualitative responses. Limitations of this self-reported data were noted. Descriptive qualitative classroom observations were conducted in all fourth and fifth grade classrooms to corroborate this measure. In addition to the input from the teacher surveys, a measure of course grades provided additional information on how well the teachers mastered the course competencies.
The questionnaire on the reading course (Survey of Reading Instruction) was administered to teachers on the last day of class at the end of fall semester. The teachers were assured that their responses would remain confidential and the instructor would not have access to the questionnaire. All 12 of the participating teachers responded to the questionnaire.
For each of the twelve course competencies, teachers were asked to rate how well they mastered the competency and applicability of the competency to their classroom instructional practices. The five-point, Likert type, rating scale for mastery ranged from ''Not at all" (1) to "Very well" (5). The rating scale for applicability to the classroom ranged from ''Not at all" (1) to "Very or highly" (5). The second page of the questionnaire contained three open-ended questions and one checklist question. The two open-ended questions asked teachers to describe how the course changed the way they think about reading instruction and the way they teach reading. The final open-ended question solicited any other comments that would help us evaluate the course. On the checklist question, teachers were instructed to "check all that apply" to indicate whether they would be interested in (1) other reading courses, (2) workshops or other professional development opportunities related to reading, and/ or (3) reading more professional articles or books related to reading. A copy of the questionnaire appears in Appendix A.

Mastery Ratings
The results of teacher ratings related to how well they thought they had mastered the course competencies are provided in Table 4. The average (mean) ratings were high across all twelve of the competencies, ranging from 4.17 to 5.00. The percentages by response category reveal that all teachers indicated at least some mastery of the competencies with most teachers indicating that the competencies had been very well mastered. The mean rating of 5.00 for three of the competencies (reading taught as a process; phonemic, morphemic, syntactic, and pragmatic systems; literacy as a learning tool across the curriculum) indicate that 100% of teachers thought these competencies were "very well" mastered. One competency that received relatively lower ratings (M= 4.17) was for "Strategies for including parents as partners in literacy development." Twenty-five percent of teachers said this competency was only "somewhat" mastered.

Applicability Ratings
Similar to the pattern of results obtained for mastery, the mean rating results observed for the applicability of course competencies to classroom instructional practices were also consistently high as shown in Table 4. For two of the competencies (Vocabulary strategies for comprehension and development; Questioning strategies and techniques), the mean rating of 5.00 was even higher than that obtained for mastery. In addition, the mean rating of 5.00 was again obtained for the same two competencies as observed for mastery (Reading taught as a process; Literacy as a learning tool across the curriculum). Again, a relatively lower mean rating of 4.17 was obtained for the competency, "Strategies for including parents as partners in literacy development."

Open-ended items
Ten of the twelve teachers responded to the first open-ended item which asked how the course changed the way they think about reading. The responses were very favorable in tone, and all responses indicated that their thinking about reading had changed as a result of the course. Some stated that they were more knowledgeable about how to apply different types of reading strategies and techniques in their classrooms. One stated, "The course has made me excited about the different strategies to teach reading.'' Another teacher said that she must "decide if the teaching techniques are benefiting the students" and if not, "find other strategies that work." Others appreciated the new knowledge and insights gained from the course. "The balanced literacy framework has given me different insights about teaching reading." A more general comment was that "it has empowered me to be a well-prepared literacy instructor. " The second open-ended question asked teachers to describe how the course changed the way they teach reading. All ten teacher responses were positive and suggested changes in their teaching as a result of the course. The predominant theme of using various effective strategies to teach reading emerged in responses ("I have learned the application of many strategies and skills"). More specific strategies included the "method framework" aligned with district frameworks and reader response journals that "allowed students to use this opportunity to connect their reading with their writing." Only four teachers responded when asked for any other comments that would help us to evaluate the course. Two of the responses indicated their appreciation of the instructor who was described as "sensitive," "cooperative," and "great." The other two praised the course more generally. "This has been a great learning experience to help me with reading. I would like to take other reading courses." The checklist questions asked teachers whether they would be interested in (1) other reading courses; (2) workshops or other professional development opportunities, and (3) reading more professional articles or books related to reading. A high percentage of teachers (73%) said they would be interested in both taking other reading courses and other workshops, and/or other professional opportunities related to reading. A smaller percentage (55%) expressed an interest in reading more professional articles or books related to reading.

Teachers' Performance in the Reading Course
All twelve teachers in the reading course (Survey of Reading Instruction) completed the class, and nearly all performed well in the course. Table 5 shows the final grade distributions for these teachers.
The table 5 shows that the largest percentage (58.33%) of participating teachers obtained between 90% and 100% of the available points and earned an A grade in the course. Another 33% scored between 80% and 89% and earned a B. Only one teacher candidate earned lower than a B (a C+). These percentages were very similar to the class averages. The average percentage for the class was 89.86% and the average percentage for participating teachers was 89.33%. The distribution of grades was very similar across exams and other assignments (e.g. the literature review).

Fourth and Fifth Grade Teacher Questionnaires
The second questionnaire was administered to all fourth and fifth grade teachers at the end of the school year. The teachers were asked to mail back the questionnaires to the lead evaluator in the return envelope provided. All nine teachers in these grade levels completed and returned the questionnaire.
The questionnaire contained both quantitative rating scale questions and open-ended questions. The rating scale questions corresponded to the topics or strategies covered in the professional development workshops. On the first scale,  teachers were asked to rate the extent to which they mastered the workshop strategies. The scale ranged from ''Not at all" (1) to "Very well" (5). The second scale asked teachers to rate the frequency with which they used these course competencies in the classroom. This scale ranged from "Never" (1) to "Frequently"(5). Teachers were instructed to leave the question blank if they had not attended the workshop covering particular strategies or concepts. The first open-ended question addressed how the intervention activities influenced classroom teaching. The second solicited suggestions for other kinds of activities that would be beneficial for improving instruction and student achievement. A copy of the instrument appears in Appendix B.

Mastery Ratings
The descriptive statistics for rating scale items on how well the strategies were mastered appear in Table 6. The items are organized by the general topic of the workshop (i.e. Content Reading, Math, and Technology).
For the workshops that covered content reading, the mean ratings on the thirteen strategies ranged from 3.00 to 4.75. These ratings suggest that, on average, teachers perceived that the strategies were at least "somewhat" mastered. The highest mean ratings were obtained for the following five strategies: PAR (Prepare, Assist and Reflect) lesson framework, Two column note taking, GIST (Generating Interactions between Schema and Text) procedure, Graphic representations (4.75, respectively) and Anticipation guide (4.71). The lowest mean rating of 3.00 was observed for the strategy, PQR2ST+ (Preview, Question, Read, Remember, Scan, Touch-up). Nearly 13% of respondents rated this item as "not at all" mastered.
Another 13% rated the INSERT technique, which is a text coding strategy that uses symbols to self-monitor reading comprehension without taking elaborate notes, as "not at all" mastered (M= 3.75). In reference to the items covered in the Math workshops, the mean ratings were a bit lower and ranged from 2.56 to 4.33 across the four strategies. The highest ratings were obtained for the strategy of using pattern blocks to determine functional relationships (M= 4.33). Nearly 67% indicated that this strategy was "very well" mastered. The two strategies related to using data for decision-making (SOL and Flanagan scores) showed more variation in ratings. Whereas 38% of teachers indicated that these strategies were "very well" mastered, another 38% rated them as "not at all" mastered. The lowest mean rating (2.65) was obtained for the strategy of using the 100 board for patterns.
The final series of workshops addressed strategies related to application of technology to instructional practices. The mean ratings on these seven questions ranged from 3.00 to 4.67. Internet searches and using existing WebQuests for content received the highest mean ratings of 4.67, respectively. 83% of teachers judged that they had mastered these strategies "very well." Using Microsoft Word to develop "How-to sheets" was rated lower by teachers (M= 3.00) with 20% indicating that the strategy was not mastered. Teachers perceived that they had at least "somewhat" mastered the remaining strategies related to technology.

Frequency of Strategy Use Ratings
The second scale required teachers to rate how frequently they used the strategies covered in the workshops. The descriptive statistics for these mean ratings across the three types of workshops are presented in Table 7. The ordinal ranking of means comparing mastery to frequency of use was nearly identical; however, the mean ratings for frequency of use were, in almost all cases, lower. The only exceptions (higher mean ratings) were in cases where the number of respondents differed on the two scales. This pattern of results is not surprising because though teachers felt they had mastered most of the strategies, they may not frequently use them in their classrooms. For strategies covered in the Content Reading workshops the mean ratings ranged from 3.22 to 4.33. Mean ratings that ranged from 3.33 to 4.20 were observed for the strategies covered in the Math workshops. On average, the mean ratings obtained on the strategies related to technology were substantially lower and ranged from 2.00 to 4.00. It should be noted, however, that teachers did not receive the Inspiration software, provided by grant funds, until the end of the academic year.
Open-ended items. Six of the nine teachers responded to the open-ended question that asked teachers to describe how the professional workshops and reading course activities influenced their classroom teaching. Some appreciated the array of strategies that they could apply in their classrooms and their impact on student learning and motivation. For instance "The grant activities have enabled me to incorporate a variety of reading strategies and activities, which has increased classroom participation and comprehension. The students show interest, enthusiasm, and often state the strategy they want to use." Other strategies that the teachers found to be beneficial for their instructional practices included cooperative learning, student generated questions and vocabulary, Inspiration software and other technology, PAR framework in reading, and two-column notes.
Six teachers also responded when asked what other kinds of activities would be beneficial for improving instruction and student achievement. Two of the teachers noted that an instructor modeling the implementation of these strategies in the classroom would be "a great follow-up and reinforcement of the concepts." Two others suggested classroom manage-ment or "behavior techniques" that would promote on-task behaviors and minimize disruptions. The final two respondents suggested additional "technology techniques" and strategies for involving parents.

Classroom Observations of Fourth and Fifth Grade Classrooms
The evaluations were conducted during the month of May before SOL testing began. This timing to conduct our observations at the end of the school year was intentionally chosen to increase the likelihood that teachers would have been previously exposed to the course and workshop strategies and would have had a chance to try them out in their classrooms. We observed the fourth and fifth grade classes during the Reading/ Language Arts block in the mornings to further increase the likelihood that targeted strategies would be observed. Although teachers were aware of the timeframe for observations, they were not informed about who would be observed on which dates or times. We conducted a total of 19 observations (10 at the fourth grade level and nine at the fifth grade level). All nine of the fourth and fifth grade teachers were observed between one to three times. The observations were approximately 30 minutes in duration.
The observation form used appears in Appendix C. At the top of the form, observers recorded the teacher's name, grade level, subject area(s), and start and end times. In the next section, observers provided a general description of classroom instruction and activities. The third part of the questionnaire presented a checklist of all strategies covered in the courses and workshops with space to write open-ended comments about strategy implementation, student reactions, and other contextual information. The final section prompted observers  We observed three targeted reading strategies frequently used by the teachers. The first, observed most often in the fifth grade classrooms, was GIST. For this procedure, students first generated summaries of paragraphs individually or in groups and then wrote summaries on the blackboard. Previewing WIKA (What I Know Already) was another frequently observed strategy, especially in the fourth grade. This strategy involves providing information in columns about what they already know, what they would like to know (questions), answers to these questions, and any remaining questions about the reading. It was obvious from observation that these strategies were commonly implemented in the classroom because the teachers had handouts prepared, and the students were clearly familiar with and largely enthusiastic about the techniques. In some cases, the students retrieved a GIST or WIKA lesson in progress from their desks and automatically returned to these tasks in their learning groups. The third most frequently observed strategy was reading across the curriculum. For example, one reading lesson focused on Fine Arts and History, while another focused on astronomy. The reading text integrated the different subject areas, and the teachers were focusing on particular chapters.
Although we frequently observed the teachers' use of questioning techniques, they did not consistently illustrate higher-order or cognitively challenging types of questioning. For example, we observed a good deal of instructional time devoted to preparing students for the SOL test. In this context the questions tended to address multiple choice items and test-taking strategies. The questions were directed at identifying correct alternatives and ruling out incorrect alternatives. In contrast, more sophisticated kinds of questioning were often associated with the reading strategies already described. Students might be asked to make predictions about what might happen in a story and then to provide an explanation for their predictions. They might also be asked to project an artist's intent and audience reaction from a historical perspective.
Another instructional strategy that had some overlap with the reading strategies already discussed was the use of group work in the classrooms. We observed group work primarily in the context of these other strategies (i.e., Previewing WIKA or GIST). Still, there were a couple of exceptions to this trend that occurred in SOL preparation and grammar lessons. Other strategies observed on only single occasions were the INSERT technique, two-column note taking, DRTA (Directed Reading Thinking Activity), incorporating cultural/ diversity issues, and using patterns in Mathematics. Again, we sometimes observed these strategies implemented in conjunction with other targeted strategies. The number of strategies we did not observe is equally noteworthy with the caveat that some of these strategies would be difficult to observe directly. For example, the use of technology software or data to tailor instruction might be used in lesson preparation rather than in the lesson itself. Furthermore, we intentionally conducted most of our observations during the Reading block and observed little Math and no Technology instruction.

Fourth and Fifth Grade Student Reading Achievement
To assess the impact of the activities in the training workshops on student achievement, we focused on any available test scores related to reading at the fourth and fifth grade levels. We analyzed fourth and fifth grade STAR data, fourth grade Benchmark scores in reading, fifth grade Benchmark scores in reading and writing, and the percentage of students who passed the fifth grade SOL English exam. The SOLs are not administered in the fourth grade. STAR scores. Star reading is an online assessment program developed by Renaissance Learning for students in grades K-12. The program uses a combination of the Cloze method and reading comprehension passages to assess various reading skills. Table 8 shows the average scores of STAR assessment obtained from a computerized diagnostic reading assessment program developed by the Renaissance Learning company. Students respond to "cloze" type or incomplete sentences for which they are presented 3 or 4 options (depending on reading level) and then asked to select the most appropriate word to complete the sentence. The difficulty level of the items is calibrated to the students' responses and becomes more or less difficult depending on the students' reading level.
The assessment program yields grade level equivalent (GLE) and percentile scores. The raw score is converted to a grade level equivalent, which provides a normative referent indicating a grade level and month. A GLE equivalent of 5.3 would indicate that students in the norm group who obtained this score were in the third month of the fifth grade. Percentile ranks also are interpreted with reference to the norm group at the same grade levels. For instance, a percentile rank of 65 indicates that this student outscored 65% of the students in the norm group. The percentile rank for a particular GLE will change as months in the school year increase. On average, a beginning third grader would be expected to score about a 3.0 and would be ranked at about the 50th percentile. If the score does not increase after a number of months, the percentile rank would drop accordingly.  Table 9 shows the results of the STAR assessments for fourth and fifth grade students. To examine the amount of improvement in reading that occurred over the entire school year, we compared scores from the first administration of the test (the pretest given in late September or early October) with scores from the last administration of the test (the post-test given in late May or early June). We only included scores from students who were tested on both of these particular dates (matched pairs). Fourth grade students did not take a writing test. The Wilcoxan signedrank test was used to determine the statistical significance of change in rankings. Benchmark scores. Benchmark scores were used to gauge fourth grade students' progress in Reading and fifth grade students' progress in Reading and Writing. The Benchmark tests are aligned with and modeled after the SOL tests. They are developed by the state of Virginia to provide early diagnostic information and practice for the SOLs. We compared the scores obtained after the first nine weeks of the school year (late November) with those obtained in the third nine weeks (early April). Although this was a relatively short time span, it was another source of evidence pertaining to student progress in the areas of Reading and Writing. With respect to the performance of fifth graders on the Benchmark tests, we found virtually no change in reading scores. The mean percentages hovered around 55% on both the pre-and post-tests. Writing scores showed a modest, but nonsignificant increase. The average pre-test scores were nearly 46%, and the average post-test scores were close to 49%.

Standards of Learning Scores
The SOL data related to reading achievement available at the time this report was for the number of fifth-grade students at each proficiency level on the English test. The fifth-grade writing tests of students were being re-scored and therefore were not yet available. The SOLs are not administered to fourth grade students. In terms of comparative data, we contrasted how fifth grade students have scored on the English SOL across three years as shown in Table 11. An examination of the percentages reveals a steady increase in the number of students who passed at the proficient level or above. Although the 2003 passing percentage had not reached the state benchmark of 7%, it is certainly approaching this goal. Slowly but surely, the gap between the state average and the school's percentage is decreasing.

Discussion and Conclusions
Teachers were generally positive about the content and relevance of the academic curriculum and its effectiveness in preparing them to teach reading. However, we learned that teachers came to professional development workshops with various levels of content understanding. In addition, there were significant differences in their background experiences and expectations from the training program, making it critical to assess teacher understanding of subject matter before deciding on the topics for the training workshops so that the workshop curriculum could be tailored to their exact needs. The first question used to guide the evaluation of the professional development program was, "How well did teachers master the content and the skills covered during the course and in the workshops?" According to the teachers themselves, the content and skills were well mastered as evidenced by the consistently high mean ratings on the questionnaire items related to the mastery of the reading course competencies as well as the strategies and the skills covered during the workshops. In addition, the teachers performed well in the course as indicated by the fact that all but one of the participating teachers earned a grade of either A or B in the course.
The second survey question was, "To what extent were the content and skills covered during the workshops implemented in the fourth and fifth grade level classrooms?" Teachers' self-ratings for the applicability of course competencies and the frequency with which they used the strategies covered in the workshops provide indirect evidence of classroom implementation. Again, consistently high mean ratings were obtained on both scales targeting classroom applicability and the use of competencies and strategies, respectively. More direct evidence about the extent of implementation came from our observations of fourth and fifth grade classrooms. Specifically, we noted that some of the targeted skills and strategies had been implemented in the classroom, and a familiarity with these strategies was suggested by the behavior of teachers and students. While a number of strategies appeared to have been consistently implemented in the classroom, many other strategies covered in the workshop and the course were not observed.
The third and final question was, "How did the fourth and fifth grade students perform on measures related to reading achievement?" The results were mixed, as the students did not perform well on the STAR assessments when compared to the norm group. At both grade levels, we saw a small gain in the grade level equivalents, and although these gains did not keep pace with the progress of the norm group over the course of the school year, most students did show progress on the benchmark scores. We also saw significant gains in the fourth grade Reading scores and a modest gain in fifth grade Writing scores. The SOL English scores revealed that the largest percentage of fifth grade students passed this subtest in comparison with any other tests. A comparison of the SOL pass rates across multiple years suggests an increasing trend of fifth grade students scoring as proficient in English.
The limitations of this evaluation are worth noting. First, research in the real world of schools often precludes the control of variables to isolate cause-and-effect inferences.
In particular, because of the simultaneous implementation of several other programs and interventions within the school, it is impossible to establish a causal link between the activities addressed by the intervention and the academic achievement by the students. Instead, we can only rely on the descriptive, comparative data. Another limitation of this study was that all achievement measures were not administered pre and post at all grade levels. In addition, we may have missed opportunities to observe the teachers' application of the skills and strategies targeted by the interventions. Additional observations throughout the academic year would have improved the likelihood of these strategies being observed. Conducting observations only at the end of the school year, i.e., just prior to the SOL testing may not yield the data that is fully representative of the classroom practices throughout the school year. Finally, our study relied heavily on self-reporting by the teachers. While the teachers themselves are largely responsible for carrying out any educational reforms, and their feedback is crucial, self-reported data may not always be completely honest.