Quality of Research
Documents Reviewed
The documents below were reviewed for Quality of Research. The research point of
contact can provide information regarding the studies reviewed and the availability
of additional materials, including those from more recent studies that may have been conducted.
Study 1Swanson, J. W., Swartz, M. S., Borum, R., Hiday, V. A., Wagner, H. R., & Burns, B. J. Involuntary out-patient commitment and reduction of violent behaviour in persons with severe mental illness. British Journal of Psychiatry, 176, 324–331.
Swartz, M. S., Swanson, J. W., Hiday, V. A., Wagner, H. R., Burns, B. J., & Borum, R. A randomized controlled trial of outpatient commitment in North Carolina. Psychiatric Services, 52, 325–329.
Swanson, J. W., Swartz, M. S., Elbogen, E. B., Wagner, H. R., & Burns, B. J. Effects of involuntary outpatient commitment on subjective quality of life in persons with severe mental illness. Behavioral Sciences and the Law, 21, 473–491. Study 2Swartz, M. S., Wilder, C. M., Swanson, J. W., Van Dorn, R. A., Robbins, P. C., Steadman, H. J., Moser, L. L., Gilbert, A. R., & Monahan, J. Assessing outcomes for consumers in New York's Assisted Outpatient Treatment Program. Psychiatric Services, 61, 976–981. Study 3Phelan, J. C., Sinkewicz, M., Castille, D. M., Huz, S., & Link, B. G. Effectiveness and outcomes of assisted outpatient treatment in New York State. Psychiatric Services, 61, 137–143.
Outcomes
| Outcome 1: Assault or threat of violent behavior |
|
Description of Measures
|
In Study 1, subjects, family members, and case managers were asked whether the subject had been picked up by police or arrested for physical assault on another person, had been in fights involving physical contact, or had threatened someone with a weapon. A composite index was created, which measured whether at least one violent act was reported by any of the three sources.
In Study 3, assault or threat of violent behavior was assessed by the MacArthur Community Violence Interview. A score of 1 was given if the participant reported having done any of the following in the previous 3 months: (1) kicked, beaten, or choked another person; (2) hit with a fist or beaten up another person; (3) tried to physically force another person to have sex against his or her will; (4) threatened another person with a knife, gun, or other weapon; or (5) fired a gun at another person or used a knife or a weapon on him or her. A score of 0 was assigned if the participant reported that they had not done any of these.
|
|
Key Findings
|
In Study 1, participants randomly assigned to the experimental group received an initial period of involuntary outpatient commitment not longer than 90 days. Thereafter, the commitment order could be renewed for up to 180 days based on determinations by a psychiatrist and the court. Participants randomized to the control group received immunity from involuntary outpatient commitment for the year of the study, with one exception: subjects with a history of serious assault involving weapon use or physical injury to another person within the preceding year were required to undergo at least the initial period of involuntary outpatient commitment.
An analysis comparing only the randomized participants (i.e., excluding violent offenders) resulted in no significant difference in the rate of assault or threat of violent behavior at 12-month follow-up. However, when including violent offenders, a key risk group to which involuntary outpatient commitment policy may be targeted, those who received extended involuntary outpatient commitment (6 months or more) had a significantly lower incidence of assault or threat of violent behavior compared with those receiving regular outpatient services (p = .025).
In Study 3, individuals recently court-ordered to outpatient commitment were compared with individuals recently discharged from the same psychiatric hospitals and attending the same outpatient facilities. At the 12-month follow-up, the assisted outpatient treatment group was less likely to perpetrate serious violent behavior compared with the control group (p < .05).
|
|
Studies Measuring Outcome
|
Study 1, Study 3
|
|
Study Designs
|
Experimental, Quasi-experimental
|
|
Quality of Research Rating
|
2.0
(0.0-4.0 scale)
|
| Outcome 2: Hospitalization |
|
Description of Measures
|
In Study 1, hospital readmission data included any psychiatric or substance abuse readmission during the 12-month follow-up period.
In Study 2, case managers reported whether individuals had experienced psychiatric hospitalization in the prior 6 months.
|
|
Key Findings
|
In Study 1, participants randomly assigned to the experimental group received an initial period of involuntary outpatient commitment not longer than 90 days. Thereafter, the commitment order could be renewed for up to 180 days based on determinations by a psychiatrist and the court. Participants randomized to the control group received immunity from involuntary outpatient commitment for the year of the study, with one exception: subjects with a history of serious assault involving weapon use or physical injury to another person within the preceding year were required to undergo at least the initial period of involuntary outpatient commitment.
Patients who underwent sustained periods of outpatient commitment beyond the initial court order had about 57 percent fewer hospital admissions on average than those in the control group (p=.04). The sustained-order patients also were hospitalized for 20 fewer days on average than those in the control group (p=.01).
Study 2 employed a quasi-experimental design, comparing patients who received (1) assertive community treatment (ACT) alone (i.e., without a court order for assisted outpatient treatment [AOT]), (2) ACT with an AOT order, and (3) intensive case management with an AOT order. Compared with receipt of ACT alone, the addition of a court order to receive ACT significantly reduced the likelihood of hospitalization (OR = .43) during any 6-month period after the first 6 months of AOT. The combination of AOT with other forms of intensive case management was associated with a 43 percent reduction in the likelihood of hospitalization (OR=.57) compared with receipt of ACT alone.
|
|
Studies Measuring Outcome
|
Study 1, Study 2
|
|
Study Designs
|
Experimental, Quasi-experimental
|
|
Quality of Research Rating
|
2.7
(0.0-4.0 scale)
|
| Outcome 3: Quality of life |
|
Description of Measures
|
Quality of life was assessed with the abbreviated form of the Lehman Quality of Life Index (QOLI), which asks respondents to rate their feelings about their current life experience on a 7-point scale of satisfaction, covering eight domains: social relationships, daily activities, finances, residential living situation, and global life satisfaction. The summary score for subjective dimensions of quality of life was used in the study.
|
|
Key Findings
|
In Study 1, participants randomly assigned to the experimental group received an initial period of involuntary outpatient commitment not longer than 90 days. Thereafter, the commitment order could be renewed for up to 180 days based on determinations by a psychiatrist and the court. Participants randomized to the control group received immunity from involuntary outpatient commitment for the year of the study, with one exception: subjects with a history of serious assault involving weapon use or physical injury to another person within the preceding year were required to undergo at least the initial period of involuntary outpatient commitment.
An analysis comparing only the randomized participants (i.e., excluding violent offenders) resulted in no significant difference in quality of life at 12-month follow-up. However, when including violent offenders, a key risk group to which involuntary outpatient commitment policy may be targeted, total number of days on involuntary outpatient commitment during the year was associated with higher quality of life at 12 months, controlling for baseline risk factors (psychiatric symptoms and history of recent homelessness) and baseline quality of life (p < .05).
Study 3 compared individuals recently mandated to outpatient commitment with individuals recently discharged from the same psychiatric hospitals and attending the same outpatient facilities. In this study, quality of life was higher in the assisted outpatient treatment group at 12-month follow-up, but not significantly so.
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
2.8
(0.0-4.0 scale)
|
| Outcome 4: Suicide risk |
|
Description of Measures
|
Suicide risk was assessed by items from the Youth Risk Behavior Surveillance System (YRBSS). Participants were asked about recent self-harm, depression, serious consideration of suicide, suicide plans, number of suicide attempts, and injuries resulting from suicide attempts that required medical treatment.
|
|
Key Findings
|
In Study 3, individuals recently court-ordered to outpatient commitment were compared with individuals recently discharged from the same psychiatric hospitals and attending the same outpatient facilities. At 12-month follow-up, the assisted outpatient treatment group had a lower risk of suicide compared with the control group (p < .05).
|
|
Studies Measuring Outcome
|
Study 3
|
|
Study Designs
|
Quasi-experimental
|
|
Quality of Research Rating
|
2.5
(0.0-4.0 scale)
|
Study Populations
The following populations were identified in the studies reviewed for Quality of
Research.
|
Study
|
Age
|
Gender
|
Race/Ethnicity
|
|
Study 1
|
18-25 (Young adult)
26-55 (Adult)
55+ (Older adult)
|
53.4% Male
46.6% Female
|
66% Black or African American
33% White
1% Race/ethnicity unspecified
|
|
Study 2
|
18-25 (Young adult)
26-55 (Adult)
55+ (Older adult)
|
Data not reported/available
|
Data not reported/available
|
|
Study 3
|
18-25 (Young adult)
26-55 (Adult)
55+ (Older adult)
|
59% Male
41% Female
|
49% Black or African American
38% Hispanic or Latino
8% White
5% Race/ethnicity unspecified
|
Quality of Research Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the Quality of Research for an intervention's
reported results using six criteria:
For more information about these criteria and the meaning of the ratings, see Quality of Research.
|
Outcome
|
Reliability
of Measures
|
Validity
of Measures
|
Fidelity
|
Missing
Data/Attrition
|
Confounding
Variables
|
Data
Analysis
|
Overall
Rating
|
|
1: Assault or threat of violent behavior
|
1.3
|
1.8
|
0.9
|
1.8
|
2.8
|
3.8
|
2.0
|
|
2: Hospitalization
|
3.3
|
2.9
|
0.9
|
3.4
|
2.8
|
2.7
|
2.7
|
|
3: Quality of life
|
3.8
|
3.8
|
1.0
|
2.0
|
2.5
|
3.8
|
2.8
|
|
4: Suicide risk
|
1.6
|
2.6
|
0.9
|
3.4
|
3.0
|
3.8
|
2.5
|
Study Strengths Overall, the research evaluating this program has a number of strengths. First, the outcome measures in general demonstrated both reliability and validity. For example, assault or threat of violent behavior was assessed in Study 1 by report of an event (e.g., arrested for physical assault on another person, had been in fights involving physical contact) from any of three sources: the subject, family members, and case manager. This measure is face valid and has been used in other research. Study 3 used the MacArthur Community Violence Interview, which has been used extensively in the literature. The use of official medical records to assess hospitalizations in Study 1 produces data that are at least as valid and reliable as any other source. Lehman's Quality of Life Interview, which is used in Study 1, is a well-known measure with established internal consistency, test–retest reliability, and discriminant validity. Suicide risk was assessed using items derived from the Youth Risk Behavior Survey, with good internal consistency and content validity. In Study 2, the amount of missing data was low, and multiple imputation methods were used to account for missing data. In Study 3, although attrition was 30 percent by the 12-month data collection period, baseline differences between study completers and dropouts were controlled for in the analyses. Analyses for Study 2 controlled for time, relevant covariates, and multiple observations. Propensity scoring was used for Study 3, which helps control for the lack of random assignment to the AOT and comparison group conditions. For most outcomes, analyses were appropriate and controlled for important variables.
Study Weaknesses No evidence supporting the reliability of the measure of assault or threat of violent behavior in Study 1 was provided. Study 2 relied on case managers' reports of their clients' inpatient psychiatric hospitalizations, which were not confirmed by administrative data. The researchers reported that this is not an intervention that is amenable to fidelity assessment other than court oversight of the program. In Study 1, missing data and attrition were high, and although the amount of attrition did not vary by study group, the researchers did not establish that completers and dropouts were equivalent at baseline. Study 1 was essentially a quasi-experimental study, since the sample included a subgroup of subjects with a recent history of serious violent behavior who could not be assigned randomly. Renewal of outpatient commitment was a confounding factor, as renewals were left to the discretion of clinicians and courts, and, except for the Duke study, there was no attempt to control for the reasons some individuals received longer or shorter periods of AOT. Another confounding factor was the lack of information or control regarding the amount, type, and quality of outpatient services received. In Study 2, the use of repeated measures for logistical regression was not appropriate because the logistic regression model assumes that the observations are independent; however, since observations from the same subject are likely to be correlated, this is not a reasonable assumption. When the assumption of independent observations is violated, the estimated standard effects from logistic regression can lead to incorrect inference.
|
|
Readiness for Dissemination
|
Materials Reviewed
The materials below were reviewed for Readiness for Dissemination. The implementation
point of contact can provide information regarding implementation of the intervention
and the availability of additional, updated, or new materials.
"A Guide for Implementing Assisted Outpatient Treatment"
"Model Law for Assisted Treatment"
Readiness for Dissemination Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the intervention's Readiness for Dissemination
using three criteria:
- Availability of implementation materials
- Availability of training and support resources
- Availability of quality assurance procedures
For more information about these criteria and the meaning of the ratings, see Readiness for Dissemination.
Implementation
Materials
|
Training and Support
Resources
|
Quality Assurance
Procedures
|
Overall
Rating
|
|
3.5
|
1.8
|
2.3
|
2.5
|
Dissemination Strengths The implementation guide is well written, detailed, and comprehensive. It is formatted in a manner that is easy to navigate, and includes a large assortment of resources to assist with implementation including summaries of key research (with sources cited); in-depth case studies from communities across the nation; and an appendix with sample policies, letters, forms, templates, and PowerPoint presentations to provide foundational information to persons wishing to implement an AOT program. Examples listed in the guide are specific and audience-appropriate. The guide, which is available for download on the developer's website, includes embedded links (to web-based resources, sample policies, etc.). A variety of sample forms, information, pamphlets, and guides are available as exemplars: these can be used to shape implementation to suit local needs, practices, and realities.
The model law booklet includes an impressive assortment of laws related to AOT issues from several states. The comments section in the proposed articles offers helpful details of legal precedence, naming specific court cases and providing explanations and suggestions that are related to each proposed article. Materials are available to guide training and support, principally in the form of video interviews and testimonials, which are of high quality, readily available, and clear and concise. Backgrounders summarize prevalence data to support stakeholders' increased awareness and cite appropriate sources.
Some of the items in the appendix would help support implementation of the program, especially the sample PowerPoint presentations and letters from judges and lawyers sharing successes of their programs.
An implementation checklist provides direction and Web links/resources to assist with successful outcomes. Through visits, interviews, and interactions with five sites that use AOT, the identification of common practices and qualities in successful AOT programs were developed and are provided in the implementation manual's case studies. Examples of assessment, baseline, follow-up, and database outcome- tracking used by other AOT implementation sites are helpful. The manual's five appendices provide several items that assist in quality assurance tasks, including sample forms for procedures and evaluations, sample patient assessment and outcome measurement forms, sample database, and guidelines. The materials provide sufficient detail to support data collection for quality assurance procedures.
Dissemination Weaknesses The model law book would benefit from reformatting (i.e., create headings in larger, bold, or colored font; put quotations in italics; and use consistent spacing on all the numbered/indented sections). The introduction section of the booklet contains many statistics and studies that are not adequately cited, and the studies/statistics noted are all fairly dated. A brief narrative or overview of the information in the book would be helpful. Guidance on how to integrate this intervention with existing criminal justice and mental health systems would be helpful.
Although the AOT implementation manual urges early and frequent training and education, there were limited print materials available to directly and specifically guide training and support. Quality assurance standards, or descriptions of how data collection would support quality assurance, were not provided in sufficient detail to support development of adequate procedures.
|
|