Quality of Research
Documents Reviewed
The documents below were reviewed for Quality of Research. The research point of
contact can provide information regarding the studies reviewed and the availability
of additional materials, including those from more recent studies that may have been conducted.
Study 1Borduin, C. M., Mann, B. J., Cone, L. T., Henggeler, S. W., Fucci, B. R., Blaske, D. M., et al. Multisystemic treatment of serious juvenile offenders: Long-term prevention of criminality and violence. Journal of Consulting and Clinical Psychology, 63(4), 569-578.
Klietz, S. J., Borduin, C. M., & Schaeffer, C. M. Cost-benefit analysis of Multisystemic Therapy with serious and violent juvenile offenders. Journal of Family Psychology, 24(5), 657-666.
Schaeffer, C. M., & Borduin, C. M. Long-term follow-up to a randomized clinical trial of Multisystemic Therapy with serious and violent juvenile offenders. Journal of Consulting and Clinical Psychology, 73(3), 445-453.
Outcomes
| Outcome 1: Monetary benefit-to-cost advantage |
|
Description of Measures
|
Monetary benefit-to-cost advantage was derived from the Washington State Institute for Public Policy (WSIPP) benefit-cost model, an integrated set of estimates and computational routines that include (1) benefit-to-cost ratios based on monetary estimates of costs associated with crime, (2) formulas for contrasting costs associated with crime and operational costs of delivering treatment programs, and (3) formulas for contrasting relative costs and benefits of treatment programs. Three sets of study measures were used in the WSIPP model:
- Six categories of county juvenile justice office records and adult criminal arrest data from State police records: murder/manslaughter, sexual, robbery, aggravated assault, property, and drug arrests. These data were retrieved yearly from the end of treatment through an average posttreatment follow-up period of 13.7 years (11.8-15.2 years after treatment; standard deviation = 1.2 years). Estimates of the costs associated with each type of arrest were adjusted values using the Consumer Price Index from the U.S. Bureau of Labor Statistics.
- The estimated costs of providing MST and individual therapy (the comparison intervention). The cost of MST was estimated with data from the budget of a private service organization delivering the intervention. The cost of individual therapy was calculated by multiplying the reimbursement rate per treatment session by the average number of treatment sessions delivered by a family counseling center.
- Estimates of monetary benefits to taxpayers and tangible and intangible monetary benefits to crime victims. Monetary benefits to taxpayers were estimated from expenditures by public agencies in the State of Washington. Tangible monetary benefits were defined as expenses related to property damage or loss, medical care, mental health care, police and fire services, victim services, and lost productivity. Intangible monetary benefits were defined as out-of-pocket expenses associated with an injury added to the amount of compensatory damages awarded by a jury for lost quality of life associated with nonfatal injuries.
|
|
Key Findings
|
A randomized clinical trial was conducted with adolescent offenders (aged 12-17) and their families who were referred by juvenile court. Participants received either MST or individual therapy (i.e., usual community treatment, which consisted of psychodynamic, client-centered, and behavioral therapeutic approaches that focused on personal, family, and academic issues). Data from the average posttreatment follow-up period of 13.7 years were assessed with the WSIPP benefit-cost model to compare the monetary benefit-to-cost advantage of MST and individual therapy. Results indicated that over the follow-up period, the use of MST relative to individual therapy saved taxpayers and crime victims $75,110 per MST participant with one offense per arrest and $199,374 per MST participant with more than one offense per arrest. Also over the follow-up period, the savings to taxpayers and crime victims per dollar spent (benefit-cost ratio) on MST relative to individual therapy was $9.51 per MST participant with one offense per arrest and $23.59 per MST participant with more than one offense per arrest.
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.3
(0.0-4.0 scale)
|
Study Populations
The following populations were identified in the studies reviewed for Quality of
Research.
|
Study
|
Age
|
Gender
|
Race/Ethnicity
|
|
Study 1
|
13-17 (Adolescent)
|
69.3% Male
30.7% Female
|
76.1% White
22.2% Black or African American
1.7% Race/ethnicity unspecified
|
Quality of Research Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the Quality of Research for an intervention's
reported results using six criteria:
For more information about these criteria and the meaning of the ratings, see Quality of Research.
|
Outcome
|
Reliability
of Measures
|
Validity
of Measures
|
Fidelity
|
Missing
Data/Attrition
|
Confounding
Variables
|
Data
Analysis
|
Overall
Rating
|
|
1: Monetary benefit-to-cost advantage
|
3.3
|
3.3
|
2.5
|
3.5
|
3.7
|
3.7
|
3.3
|
Study Strengths Criminal arrest data (on which the cost estimates were based) were obtained from county juvenile office and State police records, which are considered gold standard measures for this type of data and have high reliability and validity. Investigators justified the cost calculation assumptions by using standard discounting procedures that have been employed in other cost studies and referencing cost estimates to actual legal costs associated with criminal acts and jury award data. The investigators used methods to enhance fidelity in the trial, and it appears that treatment was delivered as intended. The original study from which the cost data are derived used random assignment (which controlled for many potential confounds), had a high treatment completion rate (79.5%), and had a very high long-term follow-up rate (93.8% of participants in the original randomized sample were successfully located). The cost study used sophisticated data modeling, which was based on an intent-to-treat approach and included all randomized participant data in the analyses; the study also had sufficient statistical power to detect between-group differences.
Study Weaknesses Although arrest records from all randomized participants were used, some of these participants may have committed crimes in other States, which introduces a degree of uncertainty into the cost estimates. Additional costs associated with cost shifting could not be ruled out because other types of service utilization (e.g., social welfare, mental health, primary care) were not tracked during the treatment period or across the long-term follow-up period. The estimated cost of providing MST was not determined by an independent third party without a financial stake in MST, which raises concerns about potential bias.
|
Documents Reviewed
The documents below were reviewed for Quality of Research. The research point of
contact can provide information regarding the studies reviewed and the availability
of additional materials, including those from more recent studies that may have been conducted.
Study 1Henggeler, S. W., Melton, G. B., & Smith, L. A. Family preservation using multisystemic therapy: An effective alternative to incarcerating serious juvenile offenders. Journal of Consulting and Clinical Psychology, 60, 953-961.
Henggeler, S. W., Melton, G. B., Smith, L. A., Schoenwald, S. K., & Hanley, J. H. Family preservation using multisystemic treatment: Long-term follow-up to a clinical trial with serious juvenile offenders. Journal of Child and Family Studies, 2(4), 283-293. Study 2Borduin, C. M., Mann, B. J., Cone, L. T., Henggeler, S. W., Fucci, B. R., Blaske, D. M., et al. Multisystemic treatment of serious juvenile offenders: Long-term prevention of criminality and violence. Journal of Consulting and Clinical Psychology, 63, 569-578.
Schaeffer, C. M., & Borduin, C. M. Long-term follow-up to a randomized clinical trial of multisystemic therapy with serious and violent juvenile offenders. Journal of Consulting and Clinical Psychology, 73, 445-453. Study 3Henggeler, S. W., Melton, G. B., Brondino, M. J., Scherer, D. G., & Hanley, J. H. Multisystemic therapy with violent and chronic juvenile offenders and their families: The role of treatment fidelity in successful dissemination. Journal of Consulting and Clinical Psychology, 65, 821-833. Study 4Henggeler, S. W., Pickrel, S. G., & Brondino, M. J. Multisystemic treatment of substance-abusing and dependent delinquents: Outcomes, treatment fidelity, and transportability. Mental Health Services Research, 1, 171-184.
Schoenwald, S. K., Ward, D. M., Henggeler, S. W., Pickrel, S. G., & Patel, H. Multisystemic therapy treatment of substance abusing or dependent adolescent offenders: Costs of reducing incarceration, inpatient and residential placement. Journal of Child and Family Studies, 5, 431-444.
Brown, T. L., Henggeler, S. W., Schoenwald, S. K., Brondino, M. J., & Pickrel, S. G. Multisystemic treatment of substance abusing and dependent juvenile delinquents: Effects on school attendance at posttreatment and 6-month follow-up. Children's Services: Social Policy, Research, and Practice, 2, 81-93.
Henggeler, S. W., Clingempeel, W. G., Brondino, M. J., & Pickrel, S. G. Four-year follow-up of multisystemic therapy with substance-abusing and substance-dependent juvenile offenders. Journal of the American Academy of Child and Adolescent Psychiatry, 41(7), 868-874. Study 5Henggeler, S. W., Halliday-Boykins, C. A., Cunningham, P. B., Randall, J., Shapiro, S. B., & Chapman, J. E. Juvenile drug court: Enhancing outcomes by integrating evidence-based treatments. Journal of Consulting and Clinical Psychology, 74(1), 42-54.
Supplementary Materials Henggeler, S. W., & Borduin, C. M. Multisystemic therapy adherence scales. Unpublished instrument.
Outcomes
| Outcome 1: Posttreatment arrest rates |
|
Description of Measures
|
Archival arrest records were collected from the South Carolina Department of Youth Services approximately 60 weeks after referral to treatment, which was on average, about 46 weeks following completion of MST treatment. In another study, postprobation arrests for the 4 years following MST treatment were obtained from State police records.
|
|
Key Findings
|
Compared with youth receiving treatment as usual, youth receiving MST were arrested about half as often in the posttreatment period. Recidivism rates were 42% for the MST-treated youth compared with 62% for youth receiving usual services (p < .05). In a second study, MST was more effective than individual therapy in preventing rearrests for violent offenses during the follow-up period (p < .001).
|
|
Studies Measuring Outcome
|
Study 1, Study 2
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
2.9
(0.0-4.0 scale)
|
| Outcome 2: Long-term arrest rates |
|
Description of Measures
|
Juvenile and adult criminal arrest records were collected at 4.0 and 13.7 years following MST treatment versus individual therapy (IT) or usual services only.
|
|
Key Findings
|
At the end of 4 years of follow-up, the rate of criminal recidivism (rearrest) for the MST completers (22%) was less than one third the overall rate for IT completers (71%). At 13.7 years after treatment, MST participants (then aged 29 years) showed significantly lower rates of criminal recidivism (50%) than comparable youth (81%) (p < .0001).
|
|
Studies Measuring Outcome
|
Study 1, Study 2, Study 3
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.0
(0.0-4.0 scale)
|
| Outcome 3: Long-term incarceration rates |
|
Description of Measures
|
Incarceration histories were collected following treatment from archival databases (e.g., juvenile justice records) and from sentencing information (e.g., days sentenced to confinement).
|
|
Key Findings
|
MST participants had on average 73 fewer days of incarceration than youth receiving usual services (p < .006). More than two thirds (68%) of youth in the usual-services group were incarcerated after treatment, compared with only 20% of the MST group. Almost 14 years after treatment, MST youth were sentenced to fewer than half as many days of incarceration as the comparison youth (p < .01).
|
|
Studies Measuring Outcome
|
Study 1, Study 2
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.1
(0.0-4.0 scale)
|
| Outcome 4: Self-reported criminal activity |
|
Description of Measures
|
This outcome was measured using full-scale scores from the Self-Report Delinquency (SRD) scale.
|
|
Key Findings
|
In two studies, MST participants had significantly lower posttreatment scores on the SRD than youth in comparison conditions (p < .05 and p < .01, respectively).
|
|
Studies Measuring Outcome
|
Study 1, Study 5
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.2
(0.0-4.0 scale)
|
| Outcome 5: Alcohol and drug use |
|
Description of Measures
|
This outcome was measured using multimethod assessment of substance use combining self-reports of use during the past 90 days and urinalysis. Mean number of days of use was tabulated for alcohol use, heavy alcohol use (more than four drinks), marijuana use, and polydrug use. Urine screens for cannabis, cocaine, and amphetamines were collected at each measurement point.
|
|
Key Findings
|
Posttreatment reports of alcohol and marijuana use and other drug use were less frequent among MST participants compared with youth in the comparison condition, p < .05.
Four years after treatment, higher rates of marijuana abstinence were found among MST participants (55%) compared with young adults in the comparison condition (28%), p < .05.
|
|
Studies Measuring Outcome
|
Study 4, Study 5
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.0
(0.0-4.0 scale)
|
| Outcome 6: Perceived family functioning-cohesion |
|
Description of Measures
|
This outcome was measured using a self-report measure of parent and youth perceptions of family functioning using the 20-item Family Adaptability and Cohesion Evaluation Scales (FACES-III). Family cohesion assesses the degree of separation or connection of family members to the family.
|
|
Key Findings
|
Posttreatment assessments showed that family cohesion increased among families receiving MST and decreased among families in the usual-services condition (p < .05).
|
|
Studies Measuring Outcome
|
Study 1, Study 2
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.0
(0.0-4.0 scale)
|
| Outcome 7: Peer aggression |
|
Description of Measures
|
Peer relations were assessed through adult and youth reports concerning the adolescent's friendships using the 13-item Missouri Peer Relations Inventory (MPRI). Three dimensions were assessed: emotional bonding, peer aggression, and social maturity.
|
|
Key Findings
|
Reports of aggression with peers decreased significantly for MST participants, but remained the same for youth in the usual-services condition (p < .05).
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.1
(0.0-4.0 scale)
|
Study Populations
The following populations were identified in the studies reviewed for Quality of
Research.
|
Study
|
Age
|
Gender
|
Race/Ethnicity
|
|
Study 1
|
13-17 (Adolescent)
|
77% Male
23% Female
|
56% Black or African American
42% White
2% Hispanic or Latino
|
|
Study 2
|
13-17 (Adolescent)
|
67.5% Male
32.5% Female
|
70% White
30% Black or African American
|
|
Study 3
|
6-12 (Childhood)
13-17 (Adolescent)
|
82% Male
18% Female
|
81% Black or African American
19% White
|
|
Study 4
|
13-17 (Adolescent)
|
79% Male
21% Female
|
50% Black or African American
47% White
1% American Indian or Alaska Native
1% Asian
1% Hispanic or Latino
|
|
Study 5
|
13-17 (Adolescent)
|
83% Male
17% Female
|
67% Black or African American
31% White
2% Race/ethnicity unspecified
|
Quality of Research Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the Quality of Research for an intervention's
reported results using six criteria:
For more information about these criteria and the meaning of the ratings, see Quality of Research.
|
Outcome
|
Reliability
of Measures
|
Validity
of Measures
|
Fidelity
|
Missing
Data/Attrition
|
Confounding
Variables
|
Data
Analysis
|
Overall
Rating
|
|
1: Posttreatment arrest rates
|
3.0
|
2.5
|
2.5
|
3.5
|
3.0
|
3.0
|
2.9
|
|
2: Long-term arrest rates
|
3.5
|
2.5
|
2.5
|
3.5
|
3.0
|
3.0
|
3.0
|
|
3: Long-term incarceration rates
|
3.5
|
3.0
|
2.5
|
3.5
|
3.0
|
3.0
|
3.1
|
|
4: Self-reported criminal activity
|
3.5
|
4.0
|
3.0
|
3.0
|
2.5
|
3.0
|
3.2
|
|
5: Alcohol and drug use
|
3.0
|
3.0
|
3.5
|
3.0
|
2.5
|
3.0
|
3.0
|
|
6: Perceived family functioning-cohesion
|
3.5
|
3.5
|
2.5
|
3.0
|
3.0
|
2.5
|
3.0
|
|
7: Peer aggression
|
3.5
|
3.5
|
2.0
|
3.0
|
3.5
|
3.0
|
3.1
|
Study Strengths The authors generally employed widely used measures with fairly well known validity and strong psychometrics in all respects. The investigators used methods to enhance fidelity in the trial, and it appears that treatment was delivered as intended. At the time these studies were conducted, the approaches to handling missing data were state of the art. Similarly, the approaches to data modeling and analysis were well executed and were the best available at the time. Excellent retention of clients in these studies strengthens the findings.
Study Weaknesses As the authors seem to recognize, there are sometimes problems with computerized legal records (e.g., government agency arrest and incarceration data), but they also are considered by many to be a gold standard. Some studies had inadequate statistical power for assessing psychosocial measures/outcomes, mainly due to participant dropout. In some cases, there were no controls for variation in treatment contact between the interventions.
|
|