Quality of Research
Documents Reviewed
The documents below were reviewed for Quality of Research. The research point of
contact can provide information regarding the studies reviewed and the availability
of additional materials, including those from more recent studies that may have been conducted.
Study 1Brown, E. C., Low, S., Smith, B. H., & Haggerty, K. P. Outcomes from a school-randomized controlled trial of Steps to Respect: A Bullying Prevention Program. School Psychology Review, 40(3), 423-443.
Outcomes
| Outcome 1: Student climate |
|
Description of Measures
|
Student climate was assessed using the following:
-
The 4-item Student Climate scale of the Student Survey, which is a revised version of the Colorado Trust's Bullying Prevention Initiative Student Survey. The Student Survey is a paper-and-pencil survey that contains 65 items for students and captures information on the school environment. Using a scale ranging from 1 (strongly disagree) to 4 (strongly agree), students rate the 4 items in the Student Climate scale regarding their perceptions, since the beginning of the school year, of their school's climate regarding trust, willingness to help, and cooperation among students.
- The 4-item Student Climate scale of the School Environment Survey (SES), which was adapted for the study from the Colorado Trust's Bullying Prevention Initiative Student Survey. The SES is a paper-and-pencil survey that contains 36 items for school staff and captures information on the school environment. Using a scale ranging from 1 (strongly disagree) to 4 (strongly agree), school staff rate the 4 items in the Student Climate scale regarding their perceptions, since the beginning of the school year, of their school's climate regarding trust, willingness to help, and cooperation among students.
In both the Student Survey and the SES, the Student Climate scale score is calculated from the average rating of the 4 scale items. Higher scores indicate a more positive student climate.
|
|
Key Findings
|
A study was conducted with students and staff (i.e., paid and volunteer staff, including administrators, teachers, paraprofessionals, support staff, custodial and cafeteria personnel, and bus drivers) from elementary schools that were matched on school demographic characteristics and randomly assigned to the intervention condition or the wait-list control condition. Results of the study indicated the following:
-
From pre- to posttest, the Student Climate scale score from the Student Survey increased in the intervention schools and decreased in the wait-list control schools (p < .05). This finding is associated with a very small effect size (Cohen's d = 0.19).
- From pre- to posttest, the Student Climate scale score from the SES increased in the intervention schools and decreased in the wait-list control schools (p < .01). This finding is associated with a small effect size (Cohen's d = 0.21).
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
2.9
(0.0-4.0 scale)
|
| Outcome 2: Student social competency |
|
Description of Measures
|
Student social competency was assessed using the 5-item Social Competency scale of the Teacher Assessment of Student Behavior (TASB). The TASB is a 20-item online survey that captures information on students' classroom behavior, scholastic aptitudes, and demographics, and teachers complete a separate TASB for each student in their class. Using a scale ranging from 1 (never) to 5 (always), teachers rate the 5 items in the Social Competency scale regarding their perceptions, since the beginning of the school year, of each student's competency in interpersonal social skills (e.g., getting along with classmates, working cooperatively with other students). The Social Competency scale score is calculated from the average rating of the 5 scale items. Higher scores indicate greater social competency.
|
|
Key Findings
|
A study was conducted with students and staff (i.e., paid and volunteer staff, including administrators, teachers, paraprofessionals, support staff, custodial and cafeteria personnel, and bus drivers) from elementary schools that were matched on school demographic characteristics and randomly assigned to the intervention condition or the wait-list control condition. From pre- to posttest, the Social Competency scale score increased in the intervention schools and decreased in the wait-list control schools (p < .05). This finding is associated with a very small effect size (Cohen's d = 0.13).
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
2.9
(0.0-4.0 scale)
|
| Outcome 3: Bullying behaviors |
|
Description of Measures
|
Bullying behaviors of students were assessed using the 4-item Physical Bullying Perpetration scale and the 4-item Nonphysical Bullying scale of the Teacher Assessment of Student Behavior (TASB). The TASB is a 20-item online survey that captures information on students' classroom behavior, scholastic aptitudes, and demographics, and teachers complete a separate TASB for each student in their class. Using a scale ranging from 1 (never) to 5 (always), teachers rate the items in the Physical Bullying Perpetration scale and the Nonphysical Bullying scale regarding their perceptions, since the beginning of the school year, of each student's physical and nonphysical bullying behaviors (e.g., "pushed, shoved, or tripped a weaker student," "spread rumors about another student"). The Physical Bullying Perpetration and Nonphysical Bullying scale scores are calculated from the average ratings of the 4 items in each scale. Higher scores indicate more bullying behaviors.
|
|
Key Findings
|
A study was conducted with students and staff (i.e., paid and volunteer staff, including administrators, teachers, paraprofessionals, support staff, custodial and cafeteria personnel, and bus drivers) from elementary schools that were matched on school demographic characteristics and randomly assigned to the intervention condition or the wait-list control condition. Although the Physical Bullying Perpetration scale score increased from pre- to posttest in both the intervention and wait-list control schools, the increase was smaller in the intervention schools (p < .01). There was no significant difference between the intervention and wait-list control schools in the overall Nonphysical Bullying scale score.
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
2.9
(0.0-4.0 scale)
|
| Outcome 4: School bullying-related problems |
|
Description of Measures
|
School bullying-related problems were assessed using the 7-item School Bullying-Related Problems scale of the School Environment Survey (SES), which was adapted for the study from the Colorado Trust's Bullying Prevention Initiative Student Survey. The SES is a paper-and-pencil survey that contains 36 items for school staff and captures information on the school environment. Using a scale ranging from 1 (not a problem) to 4 (a huge problem), school staff rate the 7 items in the School Bullying-Related Problems scale regarding their perceptions, since the beginning of the school year, of bullying-related problems among students in their school (e.g., "How big of a problem in your school is students spreading rumors or lies about students they are mad at or don't like?"). The School Bullying-Related Problems scale score is calculated from the average rating of the 7 scale items. Lower scores indicate fewer school bullying-related problems.
|
|
Key Findings
|
A study was conducted with students and staff (i.e., paid and volunteer staff, including administrators, teachers, paraprofessionals, support staff, custodial and cafeteria personnel, and bus drivers) from elementary schools that were matched on school demographic characteristics and randomly assigned to the intervention condition or the wait-list control condition. Although the School Bullying-Related Problems scale score from the SES decreased from pre- to posttest in both the intervention and wait-list control schools, the decrease was greater in the intervention schools (p < .01). This finding is associated with a small effect size (Cohen's d = 0.35).
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.0
(0.0-4.0 scale)
|
| Outcome 5: Bystander behavior |
|
Description of Measures
|
Bystander behavior was assessed using the 5-item Positive Bystander Behavior scale of the Student Survey, which is a revised version of the Colorado Trust's Bullying Prevention Initiative Student Survey. The Student Survey is a paper-and-pencil survey that contains 65 items for students and captures information on the school environment. Using a scale ranging from 1 (never) to 5 (a lot), students rate appropriate bullying bystander behaviors (e.g., "I tried to defend students who always get pushed or shoved around"). The Positive Bystander Behavior scale score is calculated from the average rating of the 5 scale items. Higher scores indicate more positive bystander behavior.
|
|
Key Findings
|
A study was conducted with students and staff (i.e., paid and volunteer staff, including administrators, teachers, paraprofessionals, support staff, custodial and cafeteria personnel, and bus drivers) from elementary schools that were matched on school demographic characteristics and randomly assigned to the intervention condition or the wait-list control condition. From pre- to posttest, the intervention schools had a greater increase in the Positive Bystander Behavior scale score from the Student Survey compared with the wait-list control schools (p < .05). This finding is associated with a very small effect size (Cohen's d = 0.14).
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
2.9
(0.0-4.0 scale)
|
Study Populations
The following populations were identified in the studies reviewed for Quality of
Research.
|
Study
|
Age
|
Gender
|
Race/Ethnicity
|
|
Study 1
|
6-12 (Childhood)
|
51% Male
49% Female
|
52% White
42% Hispanic or Latino
35% Race/ethnicity unspecified
7% Black or African American
6% Asian
|
Quality of Research Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the Quality of Research for an intervention's
reported results using six criteria:
For more information about these criteria and the meaning of the ratings, see Quality of Research.
|
Outcome
|
Reliability
of Measures
|
Validity
of Measures
|
Fidelity
|
Missing
Data/Attrition
|
Confounding
Variables
|
Data
Analysis
|
Overall
Rating
|
|
1: Student climate
|
2.0
|
2.6
|
2.5
|
3.8
|
3.0
|
3.8
|
2.9
|
|
2: Student social competency
|
2.3
|
2.0
|
2.5
|
3.8
|
3.0
|
3.8
|
2.9
|
|
3: Bullying behaviors
|
2.3
|
2.0
|
2.5
|
3.8
|
3.0
|
3.8
|
2.9
|
|
4: School bullying-related problems
|
2.0
|
2.8
|
2.5
|
3.8
|
3.0
|
3.8
|
3.0
|
|
5: Bystander behavior
|
2.0
|
2.5
|
2.5
|
3.8
|
3.0
|
3.8
|
2.9
|
Study Strengths Fidelity of implementation was measured by self-reported data that were systematically collected through the teachers' use of the online program implementation log; these data indicated that 92% of the teachers covered all program objectives and that 75% of the students were exposed to 95% of the lessons. The study had low attrition. Imputation analyses accounted for missing data. Participating schools were selected and matched on the basis of socioeconomic and racial/ethnic diversity, then randomly assigned to the intervention or wait-list control condition. The study used appropriate analyses, and sample size and power were adequate.
Study Weaknesses Psychometrics were not provided for the modified versions of known outcome instruments. Fidelity data were not collected through a method other than teacher self-report (e.g., by independent observers), and a validated fidelity instrument was not used.
|
|
Readiness for Dissemination
|
Materials Reviewed
The materials below were reviewed for Readiness for Dissemination. The implementation
point of contact can provide information regarding implementation of the intervention
and the availability of additional, updated, or new materials.
Committee for Children Web site
Schoolwide implementation support kit:
-
Committee for Children. Steps to Respect: A Bullying Prevention Program. Program guide: Review of research, program overview, program implementation, and resources. Seattle, WA: Author.
- Committee for Children. Steps to Respect: A Bullying Prevention Program. Training manual: All-staff training, coaching training, curriculum orientation training, booster trainings, family overview sessions, staff training video, and staff poster. Seattle, WA: Author.
Readiness for Dissemination Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the intervention's Readiness for Dissemination
using three criteria:
- Availability of implementation materials
- Availability of training and support resources
- Availability of quality assurance procedures
For more information about these criteria and the meaning of the ratings, see Readiness for Dissemination.
Implementation
Materials
|
Training and Support
Resources
|
Quality Assurance
Procedures
|
Overall
Rating
|
|
4.0
|
4.0
|
4.0
|
4.0
|
Dissemination Strengths Program materials, including an implementation support kit detailing an overall approach for integration of the program into a school community and providing clear standards for maintaining fidelity, support a schoolwide approach for staff, families, and students to target bullying behaviors. The curriculum and supporting resources for students are age appropriate and of high quality, and resources are provided for families. The developer's Web site contains an abundance of free tools and resources for implementers, including newsletters, research reports, case studies, sample protocols and policies, and tools and tips on securing funding. On-site training, on-site consultation, and Webinars are available to build the competency of implementers. Checklists, surveys, and worksheets assist implementers in monitoring program fidelity. A student experience survey is available to measure the impact of the program, and guidance is provided on using resulting data to support quality assurance.
Dissemination Weaknesses No weaknesses were identified by reviewers.
|
|