Quality of Research
Documents Reviewed
The documents below were reviewed for Quality of Research. The research point of
contact can provide information regarding the studies reviewed and the availability
of additional materials, including those from more recent studies that may have been conducted.
Study 1
Shaw, D. S., Dishion, T. J., Supplee, L., Gardner, F., & Arnds, K. Randomized trial of a family-centered approach to prevention of early conduct problems: 2-year effects of the Family Check-Up in early childhood. Journal of Consulting & Clinical Psychology, 74(1),1-9.
Study 2
Study 2a
Dishion, T. J., Shaw, D. S., Connell, A.M., Gardner, F., Weaver, C. M., & Wilson, M. N. The Family Check-Up with high-risk indigent families: Preventing problem behavior by increasing parents' positive behavior support in early childhood. Child Development, 79(5), 1395-1414.
Study 2b
Dishion, T. J., Brennan, L. M., Shaw, D. S., McEachern, A. D., Wilson, M. N., & Jo. B. Prevention of problem behavior through annual Family Check-Ups in early childhood: Intervention effects from the home to early elementary school. Journal of Abnormal Child Psychology, 42(3), 343-354.
Outcomes
| Outcome 1: Maternal Involvement |
|
Description of Measures
|
Maternal involvement was measured three items drawn from the HOME Involvement Scale, which assesses parenting practices and aspects of the home environment. The three items were: (1) parent keeps child in visual range, (2) parent talks to child while doing housework, and (3) parent structures child's play. Parent-child interactions were observed and rated by a trained examiner during a home visit. The measure had moderate reliability (Cronbach's a = 0.53 at age 2, Cronbach's a = 0.56 at age 3 , Cronbach's a = 0.68 at age 4).
|
|
Key Findings
|
In Study 1, from the age 3 to age 4 assessment period, maternal involvement scores increased for families in the treatment group but decreased for families in the control group. The difference in growth rates was .82 (p<.05).
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.1
(0.0-4.0 scale)
|
| Outcome 2: Destructive Behavior |
|
Description of Measures
|
Destructive behavior was measured by a narrow-band scale of the Child Behavior Checklist, or CBCL, a parent/caregiver-rated questionnaire that assesses behavioral problems in young children. Items to assess 2- and 3-year olds were derived directly from the version for ages 2 to 3 and items to assess 4-year-olds were based on items from the version for ages 4 to 18 that were also included in the version for 2- to -3 year olds. Internal consistency coefficients for the Destructive Scale were reported as: Cronbach's a = 0.60 at age 2, Cronbach's a = 0.71 at age 3, Cronbach's a = 0.73 at age 4.
|
|
Key Findings
|
The effect of the program was conditional on baseline risk status. The treatment condition was associated with lower rates of destructive behavior for children who had mothers with high and average levels of depressive symptoms (high slope =-1.67, p< .001; average slope=-0.47, p<.001) but was associated with higher rates of destructive behavior for children who had mothers whose depressive symptoms were initially low (low slope=0.73, p <.05). In addition, the treatment condition was associated with lower rates of destructive behavior for children with low and average levels of inhibition at baseline (mean slope=-0.39, p<.05; low slope=-1.61, p <.001). Interestingly, there was no effect on the Aggression scale of the CBCL.
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.1
(0.0-4.0 scale)
|
| Outcome 3: Positive Behavior Support from Caregivers |
|
Description of Measures
|
Positive Behavior Support from Caregivers was measured during a 2.5 hour home visit in which child and caregiver were videotaped engaging in a series of tasks that included free play, cleanup, delay of gratification, teaching, presentation of inhibition-inducing toys, and meal preparation. Parents were rated on his or her tendency to anticipate potential problems and to provide prompts or other structural changes to avoid young children becoming upset and/or involved in problem behavior. The following items were entered into positive behavior support scores: Parent Involvement; Positive Behavior Support; Engaged Parent–Child Interaction Time; and Proactive Parenting). This measure was found to have adequate reliability (interrater agreement, kappa, was .86 and percent agreement between raters was .87).
|
|
Key Findings
|
Study 2a found a statistically significant positive effect for positive behavior support, with the program significantly predicting improvements in positive behavior support from the age 2 to the age 3 assessment (b=.18, p<.05; Cohen's d=.33).
|
|
Studies Measuring Outcome
|
Study 2
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.0
(0.0-4.0 scale)
|
| Outcome 4: Problem Behavior |
|
Description of Measures
|
Problem behavior was measured by the Externalizing Scale of The Child Behavior Checklist (CBCL) for ages 1.5 – 5, or CBCL, a parent/caregiver-rated questionnaire that assesses behavioral problems in young children. This scale assesses attention problems, rule-breaking behavior, and aggressive behavior. The reliability for this scale was found to be adequate for all assessment points
(Cronbach's alphas=.86, .89, and .86 at ages 2, 3, and 4, respectively).
Problem behavior was also measured by the Problem factor of the Eyberg Child Behavior Inventory (ECBI), a widely used measure of early childhood problem behavior. Items used to create a Problem factor score ask caregivers to report the extent to which the behavior is a problem for the parent. The reliability for this factor was found to be adequate for all assessment points (Cronbach's alphas=.84, .90, and .94 at ages 2, 3, and 4, respectively).
|
|
Key Findings
|
Program effects on ECBI problem behavior was varied by risk levels, with differing effects between children in the elevated-problem class and the lower-problem class. The treatment condition was associated with reduced growth in problem behavior for the high-risk, elevated-problem class (b=-1.03, p<.06) but was associated with no significant change for the lower-problem class. Program effects on CBCL problem behavior were also favorable, with less growth in problem behavior among children in the treatment group compared to children in the control group (b=-.82, p<.05).
|
|
Studies Measuring Outcome
|
Study 2
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.1
(0.0-4.0 scale)
|
| Outcome 5: Oppositional Defiant Behavior |
|
Description of Measures
|
Oppositional defiant behavior was measured using a parent-rated measure at ages 2 to 5 and a teacher-rated measure of oppositional defiant behavior at school at age 7.5. The parent-rated measure of child oppositional defiant behavior was created from the Child Behavior Checklist (CBCL) for ages 1.5–5 and ages 6–18, an empirically validated measure of child behavior problems. The measure was constructed using eight items from the CBCL that mapped onto DSM-IV criteria for oppositional defiant disorder (ODD) and aggressive items from conduct disorder and were continuously present on the CBCL across ages 2 through 5. These items were cruelty to animals, destroys own things, destroys others' things, gets in many fights, physically attacks people, is defiant, is disobedient, and has temper tantrums. A composite ODD variable was then computed by averaging the values for these items at each assessment age. The reliability of this measure was acceptable at all four ages (Cronbach's alpha=.71 at age 2, .75 at age 3, .78 at age 4, and .80 at age 5). The teacher-rated measure to assess behavior at school was the Oppositional Defiant Problems scale from the Teacher Report Form (TRF), a well-validated and reliable measure of child problem behavior (Cronbach's alpha=.90).
|
|
Key Findings
|
Findings from an intent-to-treat (ITT) structural equation model found that children assigned to the intervention group had slower growth in problem behavior than children assigned to the control group, from ages 2 to 5 (B = -.150, p < .05), and lower levels of teacher-reported ODD behavior at age 7.5 (B = -.129, p < .05; small Cohen's D effect size=.26).
|
|
Studies Measuring Outcome
|
Study 2
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.1
(0.0-4.0 scale)
|
Study Populations
The following populations were identified in the studies reviewed for Quality of
Research.
|
Study
|
Age
|
Gender
|
Race/Ethnicity
|
|
Study 1
|
0-5 (Early childhood)
|
100% Male
|
48.3% Black or African American
40% White
11.7% Race/ethnicity unspecified
|
|
Study 2
|
0-5 (Early childhood)
|
51% Female
49% Male
|
49.9% White
28% Black or African American
22% Race/ethnicity unspecified
|
Quality of Research Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the Quality of Research for an intervention's
reported results using six criteria:
For more information about these criteria and the meaning of the ratings, see Quality of Research.
|
Outcome
|
Reliability
of Measures
|
Validity
of Measures
|
Fidelity
|
Missing
Data/Attrition
|
Confounding
Variables
|
Data
Analysis
|
Overall
Rating
|
|
1: Maternal Involvement
|
3.0
|
2.9
|
2.9
|
3.3
|
3.3
|
3.3
|
3.1
|
|
2: Destructive Behavior
|
3.0
|
3.0
|
2.9
|
3.3
|
3.3
|
3.3
|
3.1
|
|
3: Positive Behavior Support from Caregivers
|
2.8
|
2.5
|
2.9
|
3.3
|
3.3
|
3.3
|
3.0
|
|
4: Problem Behavior
|
3.0
|
3.0
|
2.9
|
3.3
|
3.3
|
3.3
|
3.1
|
|
5: Oppositional Defiant Behavior
|
3.1
|
2.9
|
3.3
|
3.0
|
3.3
|
3.3
|
3.1
|
Study Strengths In both studies, outcomes were clearly stated and measured using measures that were reliable and valid based on established literature in the field; and analyses were appropriate given the design of the study. Also, the sample size was relatively large in Study 2, which means that the study had adequate power to detect small effects.
Study Weaknesses In Study 1, the psychometric properties of the assessment instruments could have been stronger. Also, although the intervention was monitored through ongoing, videotaped supervision and problematic cases were discussed during weekly videoconferencing sessions, stronger methods of ensuring fidelity could have been used. Finally, the sample size was relatively small, which means that the study may not have had adequate power to detect small effects. In Study 2, the authors reported a considerable loss of teacher report data at two of the three sites; however an analysis of attrition bias suggests no significant differences between missing and non-missing cases.
|
|
Readiness for Dissemination
|
Materials Reviewed
The materials below were reviewed for Readiness for Dissemination. The implementation
point of contact can provide information regarding implementation of the intervention
and the availability of additional, updated, or new materials.
· Implementation Model Figure
· Family Check-Up (FCU) Program Fact Sheet
· FCU 1-page summary
· Implementation Cost Sheet
· Organizational Readiness Checklist
·
Family-Centered Services Provider Perspectives Survey (or provider readiness survey)
·
Family Check-Up Provider Training Manual
· Standardized Training PPT Presentation
·
Training Evaluation
· Family Check-Up Intervention Manual (Phase 1 of FCU)
·
Everyday Parenting Manual (EDP; Phase 2 of FCU)
·
Instructional videos for providers and supervisors
·
Videos demonstrating parent enactment of positive intervention-targeted parenting behaviors
·
Intervention Session Materials (you can access all materials on the website after log in; hard copies of the materials are also in the Training Manual)
· FCU and Everyday Parenting COACH Rating Forms
· COACH Manual
·
Model video of FCU feedback (session 3)
·
Group Consultation Report Form
·
Individual Consultation Report Form
· FCU Certification Criteria
· FCU Certification Benchmarks
Readiness for Dissemination Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the intervention's Readiness for Dissemination
using three criteria:
- Availability of implementation materials
- Availability of training and support resources
- Availability of quality assurance procedures
For more information about these criteria and the meaning of the ratings, see Readiness for Dissemination.
Implementation
Materials
|
Training and Support
Resources
|
Quality Assurance
Procedures
|
Overall
Rating
|
|
4.0
|
4.0
|
4.0
|
4.0
|
Dissemination Strengths
The intervention can be further adapted to create Positive Family Support (PFS) for school-system implementation with existing tiered schoolwide Positive Behavioral Intervention and Supports (PBIS) programs currently embedded in schools. All material is easy to find and in a format that is easy to search/copy when needed/appropriate. The package of training and support is robust. The required and recommended implementation supports are customized to the implementing organization(s) and vary depending on the scope of implementation. Training workshops incorporate model examples and case studies via video and significant experiential practice, involving role play, and small- and large-group activities designed to enhance understanding of the FCU and Everyday Parenting models. An e-learning curriculum is available for FCU and PFS trainees unable to attend in-person trainings, and additional online training will be available. The manual format is easy to follow; clear examples and the use of multiple communication strategies are incorporated and electronic enhancements/video is useful. Tools to determine agency readiness are helpful. Ample and specific tools are available for staff and are supported by real-life examples noted in the material. Making these tools available via a password-protected website is beneficial. Timelines and costs are all available and the recertification process is clearly described. The fidelity measures can be used for self-assessment as well as for use by supervisory/consultation assessment purposes. The developer has the capacity for virtual supervision/consultation sessions. FCU assessments can be administered and scored online using mobile devices, thereby facilitating clinical outcome monitoring. Fidelity checks are done in diverse ways and use sophisticated measures. The "COACH" process is well documented and supported.
The new, user-friendly website is easily navigable and includes multiple formats for information including audio/video. It provides additional information, including FCU program overview, intervention process, impact, and related resources and training information, and also includes separate tabs containing specific information for providers, schools, and parents. Visitors to the website can contact the program by email, phone, or through an electronic service inquiry form. Information on training opportunities is readily available and registration for various types of training can be done by email or phone.
Dissemination Weaknesses Some of the forms seem very complex; however, with adequate training that is afforded by this program, the complex forms will likely be explained for use in ensuring the quality and fidelity of the implementation.
|
|
Costs
The cost information below was provided by the developer. Although this cost information
may have been updated by the developer since the time of review, it may not reflect
the current costs or availability of items (including newly developed or discontinued
items). The implementation point of contact can provide current information and
discuss implementation requirements.
|
Item Description
|
Cost
|
Required by Developer
|
|
Materials mailed to sites in introductory packet after initial inquiry about implementing the Family Check-Up (FCU) at their site.
|
Free
|
No
|
|
FCU Manual
|
Included in Training Manual cost
|
Yes
|
|
Everyday Parenting Manual
|
$21 per manual
|
Yes
|
|
Annual license for password-secured confidential video portal and information technology (IT) support
|
Ranging from $1,084 to $2,558 per site depending on organization size
|
Yes, if site wants providers to become certified
|
|
Video explaining how to upload session video on password-secured confidential video portal
|
Included in annual license for password- secured confidential video portal and IT support
|
No
|
|
Model video of FCU feedback (session 3) on password-secured confidential video portal
|
Included in annual license for password- secured confidential video portal and IT support
|
No
|
|
Videos demonstrating parent enactment of positive intervention-targeted parenting behaviors
|
$153 per provider for annual license that includes access to all web-based or online support; includes user license for online data collection system currently in production
|
No
|
|
Training Manual
|
$104.25 per provider trained
|
Yes
|
|
Initial Onsite 2-day FCU Training
|
$4,194 + trainer travel costs for 1 trainer and up to approximately 8–10 trainees
|
Yes
|
|
Initial Onsite 2-day Everyday Parenting Training
|
$4,194 + trainer travel costs for 1 trainer and up to approximately 8–10 trainees
|
Yes
|
|
Readiness Assessment and Report: In-person (includes organizational and provider readiness assessment and benchmark planning)
|
Ranging from $3,652 to $6,154 depending on size of organization (plus consultant travel if done onsite)
|
No, but highly recommended
|
|
Group Consultation
|
$221 per hour at 2 hours per month
|
Yes, if site wants providers to become certified
|
|
Individual Consultation Support For FCU Certification
|
Average cost: $1,063 to $1,584 per provider certified (5.5 to 8 hours of consultation and support)
|
Yes, if site wants providers to become certified
|
|
Individual Consultation Support for Everyday Parenting Certification
|
Average cost: $1,584 to $2,627 per provider certified (8 to 13 hours of consultation and support)
|
Yes, if site wants providers to become certified
|
|
Certify Provider as Site Supervisor and Trainer (e.g., Train-the-Trainer Model)
|
Average cost: $3,376– $5,583 per provider certified (28 to 45 hours of consultation and support)
|
Yes, if site wants certification and for model to be sustainable
|
|
Consultation Support to Resolve Implementation Barriers During Implementation Process
|
$185–$250 per hour of support dependent on consultant
|
No, but highly recommended
|
|
Annual license that includes access to all web-based or online support (includes user license for online data collection system currently in production)
|
$153 per provider
|
Yes, if site wants providers to become certified
|
|
COACH measure and COACH manual
|
Included in Training Manual Cost
|
Yes, if site wants providers to become certified
|
|
Instructional videos for providers and supervisors
|
Included in $153 annual license fee that includes access to all web-based or online support (includes user license for online data collection system currently in production)
|
No
|
|
Data Tracking and Support System
|
$1,670 one-time cost
|
Yes, if site wants certification and for model to be sustainable
|
|