Quality of Research
Documents Reviewed
The documents below were reviewed for Quality of Research. The research point of
contact can provide information regarding the studies reviewed and the availability
of additional materials, including those from more recent studies that may have been conducted.
Study 1Bradshaw, C. P., Zmuda, J. H., Kellam, S. G., & Ialongo, N. S. Longitudinal impact of two universal preventive interventions in first grade on educational outcomes in high school. Journal of Educational Psychology, 101(4), 926-937.
Furr-Holden, C. D. M., Ialongo, N. S., Anthony, J. C., Petras, H., & Kellam, S. G.. Developmentally inspired drug prevention: Middle school outcomes in a school-based randomized prevention trial. Drug and Alcohol Dependence, 73(2), 149-158.
Ialongo, N., Poduska, J., Werthamer, L., & Kellam, S. The distal impact of two first-grade preventive interventions on conduct problems and disorder in early adolescence. Journal of Emotional and Behavioral Disorders, 9(3), 146-160.
Ialongo, N. S., Werthamer, L., Kellam, S. G., Brown, C. H., Wang, S., & Lin, Y. Proximal impact of two first-grade preventive interventions on the early risk behaviors for later substance abuse, depression, and antisocial behavior. American Journal of Community Psychology, 27(5), 599-641.
Petras, H., Masyn, K., & Ialongo, N. The developmental impact of two first grade preventive interventions on aggressive/disruptive behavior in childhood and adolescence: An application of latent transition growth mixture modeling. Prevention Science, 12(3), 300-313.
Supplementary Materials Becker, K. D., Bradshaw, C. P., Domitrovich, C., & Ialongo, N. S. Coaching teachers to improve implementation of the Good Behavior Game. Administration and Policy in Mental Health, 40(6), 482-493.
Outcomes
| Outcome 1: Conduct and problem behaviors |
|
Description of Measures
|
Conduct and problem behaviors were assessed using the following measures:
- Teacher Observation of Classroom Adaptation--Revised (TOCA-R). A structured interview administered to teachers by trained assessment staff, the TOCA-R is designed to measure child behaviors that affect school and social adaptation. Using a scale from 1 (not at all) to 6 (always), teachers rate the frequency of child behaviors over the past 3 weeks on the following three subscales: authority acceptance/aggressive behavior (e.g., breaks rules, harms property, fights), social participation/shy behavior (e.g., plays with classmates, initiates interactions), and concentration (e.g., pays attention, stays on task, is easily distracted). Some analyses used a total TOCA-R score (a mean of all the subscale items), while others used only the score on the authority acceptance/aggressive behavior subscale. This instrument was administered in the fall of first grade and the spring of first, second, and third grades.
- Peer Assessment Inventory (PAI). The PAI, designed to assess a child's adaptation to the demands of the classroom peer group, consists of 10 items taken from the Pupil Evaluation Inventory on the basis of their relevance to accepting authority/aggressive behavior, social participation/shy behavior, and likeability/rejection. For each item, a question is read aloud to the class (e.g., "Which children play alone a lot?"), and children circle the pictures of their classmates that they think are described in the question. Raw scores are converted to standard scores. Two composite scores from the PAI were used: bullying/victimization and social participation/shy behavior. This instrument was administered in the fall and spring of first grade.
- Teacher Report of Classroom Behavior--Checklist Form (TRCB-CF). Based on the TOCA-R, the TRCB-CF is designed to obtain teacher reports of child conduct problems in the school setting. Each of 7 items presents a behavior consistent with DSM-IV criteria for conduct disorder that can be observed in school (i.e., starts physical fights with classmates, lies, hurts others physically, steals, damages other people's property on purpose, skips school, bullies classmates into getting his or her own way). The teacher rates the frequency of each behavior on a scale from 1 (not at all) to 6 (always). The teacher also reports whether the child has been suspended from school in the past year and whether the child is in need of mental health services. When more than one teacher assessed the child (e.g., English/language arts, reading, and math teachers in sixth grade), the teachers' ratings were averaged to create one TRCB-CF score, and responses to questions about suspension and mental health service needs were coded "yes" if any of the teachers responded "yes." This instrument was administered in the spring of 6th through 12th grades.
- The Conduct Disorder Module of the Diagnostic Interview Schedule for Children (DISC-IV). The DISC-IV is a structured interview used to determine DSM-IV diagnoses. Child and parent reports on the Conduct Disorder Module were used to assess lifetime diagnosis of conduct disorder. A conduct disorder symptom was counted as present if either the parent or child reported it. This instrument was administered in the spring of sixth grade.
|
|
Key Findings
|
Three first-grade classrooms in each of nine schools in Baltimore City were randomly assigned to one of three conditions:
- PAX GBG. This condition included PAX GBG, provided over the duration of first grade, in combination with weekly classroom meetings to promote group problem solving, curriculum enhancements (new and supplementary curriculum materials in language arts and mathematics), and additional support for children who did not respond adequately to the intervention (e.g., tutoring, modifications in the curriculum to address individual learning styles).
- Family-School Partnership (FSP). FSP, provided over the duration of first grade, included training for teachers, workshops for parents, and weekly home-school activities aimed to enhance parent-teacher communication and provide parents with effective teaching and child behavior management strategies. As FSP was not compared with PAX GBG in the analyses, no findings on FSP are presented.
- Control, in which children received instruction as usual.
Findings from this study included the following:
- In intent-to-treat analyses that controlled for pretest scores on the TOCA-R, PAX GBG students had fewer teacher-reported problem behaviors on the TOCA-R than control group students in the spring of first grade (p = .03 for boys; p = .01 for girls) and spring of second grade (p = .001 for boys; p = .0001 for girls).
- In intent-to-treat analyses that controlled for pretest scores on the PAI, boys in the PAX GBG group had fewer peer nominations for aggression on the PAI bullying/victimization composite than boys in the control group (p = .02) in the spring of first grade. No significant effects were found for boys on the social participation/shy behavior composite, and no effects were found for girls on the bullying/victimization or social participation/shy behavior composite.
- In the spring of sixth grade, relative to students in the control group, PAX GBG students had lower levels of teacher-reported conduct problems on the TRCB-CF (p < .001). According to teacher reports, PAX GBG students were also less likely to be in need of mental health services (odds ratio = 0.37, p < .05) and to have been suspended from school (odds ratio = 0.73, p < .05). PAX GBG students were less likely than control group students to have a lifetime conduct disorder diagnosis based on parent or child report on the DISC-IV Conduct Disorder Module (odds ratio = 0.42, p < .05).
- In first through third grades, PAX GBG students were three times more likely than control group students to be in the low-aggressive/disruptive behavior trajectory (i.e., displaying low, stable levels of aggression) based on teacher reports on the TOCA-R authority acceptance/aggressive behavior subscale (odds ratio = 3.117, p < .01 for boys; odds ratio = 3.059, p < .05 for girls). A significant difference between groups was sustained into middle and high school for boys (odds ratio = 0.497, p < .01) but not for girls.
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.5
(0.0-4.0 scale)
|
| Outcome 2: Academic success |
|
Description of Measures
|
Academic success was assessed using the following measures:
- Comprehensive Test of Basic Skills (CTBS). This standardized achievement battery assesses verbal skills (word analysis, visual recognition, vocabulary, comprehension, spelling, and language mechanics and expression) and quantitative skills (computation, concepts, and applications). This instrument was administered in the fall of first grade and the spring of first and second grades.
- Kaufman Test of Educational Achievement (KTEA). The KTEA is a measure of educational achievement assessing skills in reading, math, and spelling. Only the reading and math domains were used in the study. This instrument was administered in 12th grade.
- Special education service use. Official records were provided by the school district for students who had an Individualized Education Program (IEP) at any time during 1st through 12th grades.
- High school graduation. Data were obtained from the school district to determine whether the student had graduated from high school or had passed the General Educational Development (GED) test.
- College attendance. When they were 19, participants were asked during a phone interview whether they had attended college (e.g., 4-year college, junior college).
|
|
Key Findings
|
Three first-grade classrooms in each of nine schools in Baltimore City were randomly assigned to one of three conditions:
- PAX GBG. This condition included PAX GBG, provided over the duration of first grade, in combination with weekly classroom meetings to promote group problem solving, curriculum enhancements (new and supplementary curriculum materials in language arts and mathematics), and additional support for children who did not respond adequately to the intervention (e.g., tutoring, modifications in the curriculum to address individual learning styles).
- Family-School Partnership (FSP). FSP, provided over the duration of first grade, included training for teachers, workshops for parents, and weekly home-school activities aimed to enhance parent-teacher communication and provide parents with effective teaching and child behavior management strategies. As FSP was not compared with PAX GBG in the analyses, no findings on FSP are presented.
- Control, in which children received instruction as usual.
Findings in early reading achievement, as measured by the CTBS, included the following:
- In intent-to-treat analyses that controlled for pretest levels of reading achievement, boys in the PAX GBG group with pretest levels of reading achievement at or above the 40th normal curve equivalent (NCE) had higher levels of reading achievement than their counterparts in the control group in the spring of first grade (p = .01).
- In the spring of second grade, boys in high-implementation PAX GBG classrooms (those with more than 50% of the intervention implemented with high fidelity) had higher levels of reading achievement than boys in the control group (p = .001).
- No significant effects in reading achievement were found for girls.
Findings in early math achievement, as measured by the CTBS, included the following:
- In the spring of first grade, boys in high-implementation PAX GBG classrooms with pretest levels of math achievement at or above the 40th NCE had higher levels of math achievement than their counterparts in the control group (p < .001).
- In intent-to-treat analyses that controlled for pretest levels of math achievement, boys in the PAX GBG group with pretest levels of math achievement at or above the 60th NCE had higher levels of math achievement than their counterparts in the control group in the spring of second grade (p = .02).
- In the spring of first grade, girls in high-implementation PAX GBG classrooms with pretest levels of math achievement at or above the 40th NCE had higher levels of math achievement than their counterparts in the control group (p = .01).
- In the spring of second grade, girls in high-implementation PAX GBG classrooms with pretest levels of math achievement at or above the 60th NCE had higher levels of math achievement than their counterparts in the control group (p = .04).
Subsequent analyses to assess the longitudinal impact of PAX GBG controlled for academic readiness as measured by teachers on the TOCA-R in the fall of first grade. Findings included the following:
- In 12th grade, PAX GBG students had higher reading and math performance than control group students as measured by the KTEA (p < .01 for reading; p < .01 for math).
- PAX GBG students were about half as likely as control group students to have received any special education during 1st through 12th grades (odds ratio = 0.494, p < .05).
- PAX GBG students were more likely than control group students to graduate from high school (odds ratio = 1.702, p < .05) and attend college (odds ratio = 2.222, p < .05).
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.3
(0.0-4.0 scale)
|
| Outcome 3: Mental health service utilization |
|
Description of Measures
|
Mental health service utilization was assessed using the following measures:
- Service Assessment for Children and Adolescents Parent Report (SACA-P). This structured interview, designed to accompany the DISC-IV, is used to obtain information from parents on lifetime child and adolescent mental health service utilization, including use of inpatient and outpatient services provided by mental health professionals. This instrument was administered in the spring of sixth grade.
- School Mental Health Professional Report (SMHPR). This checklist, developed by researchers for use in the study, was completed by school mental health professionals who indicated whether they provided mental health services to a student in the past year and what type of services were provided (i.e., individual or group counseling, psychological assessment, consultation). This instrument was administered in the spring of sixth grade.
|
|
Key Findings
|
Three first-grade classrooms in each of nine schools in Baltimore City were randomly assigned to one of three conditions:
- PAX GBG. This condition included PAX GBG, provided over the duration of first grade, in combination with weekly classroom meetings to promote group problem solving, curriculum enhancements (new and supplementary curriculum materials in language arts and mathematics), and additional support for children who did not respond adequately to the intervention (e.g., tutoring, modifications in the curriculum to address individual learning styles).
- Family-School Partnership (FSP). FSP, provided over the duration of first grade, included training for teachers, workshops for parents, and weekly home-school activities aimed to enhance parent-teacher communication and provide parents with effective teaching and child behavior management strategies. As FSP was not compared with PAX GBG in the analyses, no findings on FSP are presented.
- Control, in which children received instruction as usual.
By the sixth grade, PAX GBG students were less likely to have received mental health services than control group students according to parent reports on the SACA-P (odds ratio = 0.53, p < .05) and school reports on the SMHPR (odds ratio = 0.56, p < .05).
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.4
(0.0-4.0 scale)
|
| Outcome 4: Initiation of substance use |
|
Description of Measures
|
Initiation of substance use was assessed using standardized questions regarding first use of tobacco, alcoholic beverages, inhalants, marijuana, and other illegal drugs (i.e., powder cocaine, crack, heroin). Students self-reported their initiation of substance use through audio computer-assisted self-interview (ACASI) methods during the spring of sixth through eighth grades.
|
|
Key Findings
|
Three first-grade classrooms in each of nine schools in Baltimore City were randomly assigned to one of three conditions:
- PAX GBG. This condition included PAX GBG, provided over the duration of first grade, in combination with weekly classroom meetings to promote group problem solving, curriculum enhancements (new and supplementary curriculum materials in language arts and mathematics), and additional support for children who did not respond adequately to the intervention (e.g., tutoring, modifications in the curriculum to address individual learning styles).
- Family-School Partnership (FSP). FSP, provided over the duration of first grade, included training for teachers, workshops for parents, and weekly home-school activities aimed to enhance parent-teacher communication and provide parents with effective teaching and child behavior management strategies. As FSP was not compared with PAX GBG in the analyses, no findings on FSP are presented.
- Control, in which children received instruction as usual.
In an analysis that controlled for teacher-rated problem behaviors in the fall of first grade, PAX GBG students were less likely than control group students to have initiated use of tobacco (relative risk = 0.55, p = .013) and illegal drugs other than marijuana (relative risk = 0.29, p = .028) by eighth grade.
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.4
(0.0-4.0 scale)
|
Study Populations
The following populations were identified in the studies reviewed for Quality of
Research.
|
Study
|
Age
|
Gender
|
Race/Ethnicity
|
|
Study 1
|
6-12 (Childhood)
|
53.2% Male
46.8% Female
|
86.8% Black or African American
13.2% White
|
Quality of Research Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the Quality of Research for an intervention's
reported results using six criteria:
For more information about these criteria and the meaning of the ratings, see Quality of Research.
|
Outcome
|
Reliability
of Measures
|
Validity
of Measures
|
Fidelity
|
Missing
Data/Attrition
|
Confounding
Variables
|
Data
Analysis
|
Overall
Rating
|
|
1: Conduct and problem behaviors
|
3.3
|
3.4
|
3.8
|
3.3
|
3.4
|
3.8
|
3.5
|
|
2: Academic success
|
3.2
|
2.9
|
3.8
|
3.1
|
3.3
|
3.6
|
3.3
|
|
3: Mental health service utilization
|
3.3
|
2.9
|
3.8
|
3.3
|
3.3
|
3.8
|
3.4
|
|
4: Initiation of substance use
|
3.3
|
3.3
|
3.8
|
3.0
|
3.3
|
3.8
|
3.4
|
Study Strengths An impressive array of instruments was used to provide a comprehensive picture of the variables examined. Most of the measures used were established and tested by independent researchers and shown to have good reliability and validity or were adaptations of these tested measures. Implementation fidelity was maximized through training and certification for teachers, ongoing expert support, monthly meetings, standardized manuals, and support materials (e.g., outlines, checklists). Fidelity was assessed through the systematic collection of data using psychometrically tested measures of classroom setup, classroom observations, and reviews of classroom records. Five of the nine intervention classrooms were identified as high-implementation classrooms, and all but two of the nine intervention teachers implemented more than 50 percent of the protocol. There was minimal attrition, which is particularly noteworthy considering the period of follow-up and breadth of data collected. There were no significant differences between the intervention and control groups in rates of attrition or sociodemographic characteristics of children with missing data. Sophisticated methods were used to account for missing data. Classrooms were randomly assigned to condition, and there were no significant differences between intervention and control groups in age, gender, ethnicity, or free lunch status. Teacher differences at pretest were controlled for in subsequent analyses. Data analyses were sophisticated, and sample size and power were adequate.
Study Weaknesses Several intervention classrooms did not implement the intervention with fidelity.
|
|