Studies of Outpatient Commitment are Misused

Revised July 3, 2001.

• Controlled studies in New York and North Carolina
• Studies on hospital admissions
• Studies on treatment compliance
• RAND review of 8 states' experience
• The potential effect of IOC in California

The Effects of Outpatient Commitment on Use of Mental Health Services Are Greatly Exaggerated

Before making the sweeping changes that proponents of involuntary outpatient commitment suggest, policymakers and reporters covering this issue should review the research literature on involuntary outpatient commitment. The studies, relatively few in number, clearly show that it confers no benefit beyond access to effective community services—access that is too often nonexistent on a voluntary basis.

Involuntary "outpatient commitment" (IOC)—a statute authorizing courts to require an individual to accept outpatient mental health treatment or hospital release conditioned on treatment compliance—is being offered as a solution to the problem of people with mental illnesses in jails, homeless on the streets or acting out disruptively or violently in society. Proponents argue that only with such laws can certain individuals be persuaded to utilize mental health services. Yet most of the studies on which they rely are seriously flawed, and some are presented in misleading ways. A recent review of these studies by RAND Health and RAND Institute for Civil Justice offers a balanced look; we have added findings from RAND's analysis to ours.

As the Bazelon Center's and others' reviews demonstrate, arguments that involuntary outpatient commitment is a panacea in the treatment of individuals with mental illness are specious. The more scientific the study, the less evidence it offers that outpatient commitment orders have any effect beyond providing increased access to effective services.

A recent literature review identified 29 studies of mandated community treatment but found only two that met reviewers' criteria for randomization and control: the New York and North Carolina studies discussed below. The authors conclusion:

Based on current evidence, community treatment orders may not be an effective alternative to standard care. It appears that compulsory community treatment results in no significant difference in service use, social functioning or quality of life compared with standard care. There is currently no evidence of cost effectiveness. People receiving compulsory community treatment were, however, less likely to be victim of violent or non-violent crime. It is, nevertheless, difficult to conceive of another group in society that would be subject to measures that curtail the freedom of 85 people to avoid one admission to hospital or of 238 to avoid one arrest.

Kisely, S., Campbell LA, Preston N. Compulsory community and involuntary outpatient treatment for people with severe mental disorders. The Cochrane Database of Systematic Reviews 2005, Issue 3. Art No: CD004408.pub2. DOI:10.1002/146518.CD004408.pub2. Abstract available at http://www.update-software.com/abstracts/AB004408.htm.

In reality, where they exist, outpatient commitment laws are seldom used. It may be that those who understand the mental health system and its failings are not willing to penalize the individual with a mental illness for the lack of appropriate community services, or to subject innocent people to arrest and incarceration for the failures of the treatment system. The RAND team reviewed the experience of eight states with laws allowing IOC and found "significant problems" in all.

The Bazelon Center considers outpatient commitment a misguided approach to a systems problem (see our position on IOC). The pervasive lack of appropriate, accessible and acceptable services is the issue. Involuntary outpatient commitment appears to increase the use of services because it forces the system to make those services available to people for whom a court has ordered treatment. Expanding service options would accomplish the same ends without coercion, without the trauma of a court appearance and without violating the individual's right to make decisions about his or her own health care.

The Comparative Effectiveness of Involuntary Treatment and Its Alternatives

RAND separated involuntary outpatient commitment studies into two generations. The first generation was marked by studies indicating "limited positive results." However, RAND notes, these studies were "plagued by significant methodological limitations" and "did not specify for whom, how, or under what circumstances court-ordered outpatient treatment may work."

In the second generation of studies, only two randomized clinical trials have been completed: the Bellevue Hospital Center Study in New York City and the Duke Study in North Carolina. The studies, RAND found, reached "conflicting conclusions." The New York study found that outpatient commitment had no statistically significant effect on rehospitalization rates or days spent in hospital. The study also found that IOC did not improve compliance with medication and continuation of treatment, or reduce the number of arrests or violent acts committed. However, RAND considered the findings weakened by several limitations: 1) The IOC orders were inconsistently enforced throughout the study; 2) the IOC group included more individuals with co-occurring disorders than the control group; and 3) the sample size was small.

The overall findings of the North Carolina study, which RAND considers "the better of the two," generally support the New York finding that outpatient commitment has no effect on hospital use. The North Carolina study also found mixed results for subgroups, depending on the length of outpatient commitment, that require further investigation. Hospital use actually increased for those with a short duration of outpatient commitment (six months or less). The only group for whom hospital use decreased was the group who received more intensive services and outpatient commitment of six months or longer. RAND concluded that the North Carolina study "did not achieve outcomes that were superior to outcomes achieved in studies of assertive community treatment alone."

New York Study

Final Report: Research Study of the New York City Involuntary Outpatient Commitment Pilot Program, (at Bellevue Hospital). Policy Research Associates, December 4, 1998 (www.prainc.com/IOPT/opt_toc.htm)

The question this study attempted to answer was whether an outpatient commitment order by a court contributed to any additional beneficial results when compared with provision of intensive services only. All participants received the intensive services; only those subject to the court order were compelled to undergo treatment.

The findings are conclusive. Comparing those subjected to outpatient commitment with those who were offered access to the same intensive services, the study found:

• no additional improvement in patient compliance with treatment;
• no additional increase in continuation of treatment;
• no differences in rates of hospitalization;
• no differences in lengths of hospital stay; and
• no difference in arrests or violent acts committed.

Since people were randomly assigned to the two groups, the "difficult" cases were evenly distributed between the two approaches.

The results of this study help to explain why other studies of outpatient commitment have been misread to support its effect. Individuals subjected to a court order for outpatient treatment are provided services—often intensive services never before available to them. Not surprisingly, many of them do better. This is the very reason science is based on controlled-trial studies wherever possible. In a controlled trial, an attempt is made to isolate the variables and make it easier to identify the true effect of any one factor. While this is not always possible or easy to do, results from a controlled trial, like the Bellevue study, are more accurate than studies using other approaches.

Specifically, this study found:

• No statistically significant differences in the percentage of clients who discontinued treatment (27% court order, 26% intensive services only). Clients in assertive community treatment had the lowest dropout rate. This made it clear that assertive community treatment, not the court order, increases the likelihood that individuals will accept continued treatment.
• The assertiveness of the coordinating team ensured a level of care previously not experienced by providers or patients. Enhanced community services for all participants reduced rehospitalization rates (87.5% to 51.4% for those who did not have court orders, 80.1% to 41.6% for those with court orders).
• No statistically significant differences existed in compliance with case management services (71% for court-ordered clients and 61% for intensive services only).
• No statistically significant differences in the level of violence committed by either group. Few arrests were found (16% intensive services only, 18% court-ordered). There were no differences in any arrest, the number of arrests, or more serious charges.
• No statistically significant differences in medication compliance rates between the two groups.
• No statistically significant differences in quality of life or symptomatology between the two groups.

The study provides strong evidence that outpatient commitment has no intrinsic value. Where it does appear to have had an effect, this is because it has forced the mental health system to commit itself to helping consumers find acceptable and effective treatment for their illnesses.

The North Carolina Study

Swartz, M.S. et al., Can Involuntary Commitment Reduce Hospital Recidivism? Findings From a Randomized Trial with Severely Mentally Ill Individuals. American Journal of Psychiatry, 12: 1968-1974 (1999).

The findings of this study conducted at Duke University in North Carolina agree in part with the New York study discussed above. Overall, hospital admissions and days did not differ significantly for participants randomly assigned to outpatient commitment (of any length) and those in the comparison control group, who were not under commitment.

• Short term outpatient commitment increases hospital use and decreases participant cooperation.
• Outpatient commitment of less than 180 days actually increased hospital use. Participants on short outpatient commitment spent 35% longer, 38 days on average, in the hospital, compared to an average of 28 days for those not on outpatient commitment. The authors attribute this to an increased sense of coercion and decreased autonomy among participants under outpatient commitment.
• Long-term outpatient commitment and intensive services decreased hospital outcomes.

Unlike the New York study discussed above, this study found reduced hospital stays only for participants who remained under outpatient commitment for more than six months and who also received intensive services (a median of 7.5 services per month). Neither extended outpatient commitment nor higher level of service alone reduced the chance of hospital admission. These findings suggested to the RAND reviewers "that outpatient commitment may exert most of its effect on providers." In other words, the court order appears to increase the delivery of services to participants under outpatient commitment. "This use of outpatient commitment is not a substitute for intensive treatment; it requires a substantial commitment of treatment resources to be effective."

The RAND authors suggest two explanations for the findings on long-term outpatient commitment: 1) The larger North Carolina study was better able to detect differences between groups of outpatient commitment patients, and 2) the North Carolina outpatient commitment program has been up and running longer, whereas the program studied in New York was a pilot.

Both the Bazelon Center's analysis and the RAND review find weaknesses in the North Carolina study. RAND identified four issues that limit the applicability of the Duke findings to community-based settings beyond an academic research study.

1. RAND interviewed stakeholders in North Carolina who emphasized that "people in the study may have received more outpatient services, or services delivered more routinely, than individuals in other areas of North Carolina." The article does not describe service use among the non-outpatient commitment comparison group. It is therefore difficult to assess the impact of outpatient commitment on the service delivery system
2. Enforcement provisions are often a problem beyond academic research studies and may not "be as systematically implemented in usual community practice."
3. The Duke study recruited participants who were discharged from hospitals and therefore "the findings may not be generalizable to people initially placed under involuntary commitment in the community."
4. The length of involuntary commitment was not random, but depended on the situation of each individual. In other words, individuals under outpatient commitment for shorter periods differed from those under outpatient commitment for longer periods, and any differences between these groups are not reported. RAND researchers cautioned that "any bias of this type would probably operate to diminish the likelihood of finding an effect for outpatient commitment."

"Whether court orders without intensive treatment have any effect is an unanswered question," RAND concluded. "In sum, the Duke study does not prove that treatment works better in the presence of coercion or that treatment will not work in the absence of coercion and other evidence-based reviews prove that alternative interventions such as assertive community treatment have similar positive effects."

Does Outpatient Commitment Decrease Hospital Admissions?

Statements that outpatient commitment reduces hospital admissions or hospital stays are often based on data from four published studies, all flawed. The two reputable studies, described above, found no such correlation. Unpublished studies have also been cited in support of this claim.

1. Fernandez, G.A., and Nygard, S. Impact of Involuntary Outpatient Commitment on Revolving-Door Syndrome in North Carolina (1990). Hospital and Community Psychiatry 41:1001-1004 (1990)

Claims that this study shows a decrease in hospital admissions of from 3.7 to 0.7 per 1,000 days for those subjected to outpatient commitment are meaningless.

• This study has no comparison group, which means that changes in hospital admissions cannot be attributed to outpatient commitment. Other factors, such as improved access to services, changes in the state service system to make more services available, etc. could have caused this effect.
• The study examined only two measures: inpatient admissions and the number of inpatient days. No other data were evaluated, such as patient satisfaction or improvement in symptoms or functioning.
• The study examined the average rate of admission, instead of comparing the before and after rate for each individual. Further, it is not clear whether the time periods for the "before" and "after" measurements were even comparable.

2. Zanni, G., and deVeau, L. Inpatient Stays Before and After Outpatient Commitment (in Washington, D.C.) Hospital and Community Psychiatry 37:941-942 (1986).

Claims that this study shows a decrease in hospital admissions from 1.81 per year before to 0.95 per year after outpatient commitment are meaningless.

• The absence of a non-outpatient commitment comparison group means that any changes cannot be attributed only to outpatient commitment.
• The study included only 42 patients, too few to make any such generalizations.
• The study examined the average rate of hospital admission, instead of comparing the before and after rates for each individual.
• The study examined only two measures, inpatient admissions and average length of stay. No other data were evaluated, such as patient satisfaction or improvement in symptoms or functioning.

3. Munetz, M.R., Grande, T., Kleist, J., & Peterson, G.A. The Effectiveness of Outpatient Civil Commitment. Psychiatric Services 47:1251-1253 (1996).

Claims that hospital admissions decreased from 1.81 per year to 0.95 per year, as a result of outpatient commitment in Ohio, are flawed.

• The absence of a non-outpatient commitment comparison group means that any changes cannot be attributed only to outpatient commitment.
• The study included only 20 patients, too few to make any such generalizations.
• The study cannot separate the effects of the outpatient commitment order itself and the expanded services, including intensive case management, that the individuals had available to them.

4. Rohland, B. The Role of Outpatient Commitment in the Management of Persons with Schizophrenia. Iowa Consortium for Mental Health Services, Training, and Research. May 1998.

• Claims that hospital admissions per year decreased from 1.3 to 0.3 are based on a sample of only 39 patients under outpatient commitment—too few to make any such generalization. Further, the comparison group differs in important ways from the outpatient commitment group.
• Members of the comparison group were much less likely to use two or more antipsychotics and to have co-occurring substance abuse—factors that increase the likelihood of hospital admission. They were also much more likely to be compliant with medication. As a result, the comparison is meaningless.
• The study did not evaluate other data, such as patient satisfaction or improvement in symptoms or functioning.

Does Outpatient Commitment Increase Patients' Compliance with Psychiatric Treatment?

Statements that increased compliance with psychiatric treatment can be attributed solely to the effect of outpatient commitment are normally based on data from two studies—both flawed. The New York study at Bellevue finds just the opposite.

1. Hiday, V.A., and Scheid-Cook, TL. The North Carolina Experience with Outpatient Commitment: A Critical Appraisal. International Journal of Law and Psychiatry 10:215-232 (1987).

The study claims that only 33% of patients under outpatient commitment refused medication during a six-month period, compared to 66% of patients not on outpatient commitment. The claim is flawed because the 33% medication rate refusal was for everyone under outpatient commitment, including a rather large number of people who were inappropriately committed.

A correct comparison rate would have been for adults with serious mental illnesses who have a history of mental hospitalization, medication refusal or dangerous behavior the target group for outpatient commitment. Of this group, 53% refused medication while under outpatient commitment, compared to the 66% of voluntary patients—a far cry from the 50% differential claim by supporters of outpatient commitment. In addition, the study failed to report whether the 13-point difference was considered statically significant.

In addition, the absence of a voluntary outpatient control group receiving the same services as the involuntary commitment group means that any changes in medication compliance cannot be attributed exclusively to outpatient commitment. Furthermore, mental health services and additional assistance were not available equally across the state, to members of either group. The impact of these factors cannot be separated from the impact of the commitment order.

The study did not evaluate other data, such as patient satisfaction or improvement in symptoms or functioning.

2. Munetz, et al. (item #3, above).

Claims that this study shows that outpatient commitment increased patients' compliance with outpatient psychiatric appointments from 5.7 to 13.0 per year and with attendance at day treatment sessions from 23 to 60 per year are flawed.

• The absence of a non-outpatient commitment comparison group means that any changes cannot be attributed only to outpatient commitment.
• The study included only 20 patients, too few to make any such generalizations.
• Ten percent of the sample received Clozapine, which introduces another explanation for reduced inpatient stays. Clozapine has significantly fewer side effects than older psychotropics and generally results in individuals' being more willing to take their medication.
• The study cannot separate the effects of the outpatient commitment order itself and the expanded services, including intensive case management, which the individuals had available to them.

3. Van Putten, D.A.., Santiago, J.P., & Bergen, M.R. Involuntary Commitment in Arizona: ARetrospective Study. Hospital and Community Psychiatry 39:20005-5002 (1988).

Claims that this study shows improved compliance with treatment as a result of outpatient commitment are flawed (claims are made that 71% of those subjected to commitment maintained treatment contacts six months after expiration of the order, compared with 6% of patients who had not been subjected to outpatient commitment).

• The absence of a non-outpatient commitment comparison group means that any changes intreatment contacts cannot be attributed to outpatient commitment, but may be the result of other factors, including increased effort by treating professionals to work with these patients on prior problems with services offered to them.
• The sample size for this study, 66 individuals, is too small to make such generalizations.
• The comparison was between only 34 people (before outpatient commitment) and a different group of 32 people (after enactment of outpatient commitment). Not only are these small sample sizes, but this methodology is weak because it fails to track the same individuals over time.

Other Findings

In addition to reported research studies, proponents of involuntary outpatient commitment often refer to self-published monographs, presentations at conferences, anecdotal information and other such sources. These materials do not have the scientific rigor of peer-reviewed, published research studies. Furthermore, missing information on sample size, study design and the statistical analyses performed makes it impossible to evaluate the claims made in these various papers. Accordingly, these materials do not meet the traditional criteria to back up policy decisions.

While enhanced access to effective services is advantageous, involuntary outpatient commitment could also lead to unexpected and serious adverse outcomes for millions of individuals. Public policy should be based on serious scientific analysis, not on strident calls for a punitive response that is too simplistic to address the underlying problem. See our position paper on involuntary commitment at www.bazelon.org/involcom.html.

States' Experience with IOC

RAND conducted interviews on the experience of eight states (Michigan, New York, North Carolina, Ohio, Oregon, Texas, Washington, and Wisconsin) that have statutory provisions allowing IOC. Among the prosecuting and defense attorneys, behavioral health officials, and psychiatrists interviewed, they found both "widespread support" and "some skepticism and uncertainty about the practical application of these laws." RAND noted that in all states "significant problems were identified on the implementation of these laws." The researchers concluded that "perhaps that most important lesson drawn from this series of interviews is that making assumptions about the implementation of outpatient commitment based on statutory analysis alone is risky. A reading of what is permissible under statute may not accurately reflect the experience in a state."

The reviewers identified three critical requirements for successful implementation of IOC: 1) the infrastructure to monitor individuals on IOC; 2) adequate funding for the increased demand for mental health services; and 3) lack of consistent enforcement and service availability across jurisdictions. The key informants in the states "emphasized that outpatient commitment is not a 'silver bullet' and that it simply cannot work in the absence of intensive clinical services and mechanisms for enforcement of court orders."

RAND's review found that states are using outpatient commitment for individuals discharged from the hospital, instead of a community-initiated alternative to hospitalization. "...These states are using involuntary outpatient commitment at the time of discharge to extend close supervision and monitoring into the community."

In New York, RAND examined implementation of Kendra's Law and noted the following:

• More people were committed under IOC in New York City than in the rest of the state.
• Those under IOC have priority for limited case management services.
• The statute is used primarily for individuals discharged from inpatient care.
• Interview respondents noted that the Rikers Island jail in New York City plans to apply the statute to mentally ill individuals released from the jail.

The Bazelon Center considers the RAND examination of states' experience limited because it did not include consumers, consumer groups and advocacy organizations among the contacted groups. One of the major concerns regarding IOC is that mental health providers would have an enforcement role, therefore undermining the consumer trust necessary for treatment.

RAND Assessment of the Effect of Changes in California Civil Commitment Practices (Lanterman-Petris Act)

RAND was commissioned to analyze the potential effect of enactment of IOC in California. While an estimate of the number of people affected was precluded by the limits of existing data sources, RAND did examine the length of involuntary commitment of the 58,439 individuals involuntarily treated in California in 1997-1998. Few (0.41%) were under involuntary commitment longer than one month (165 for an additional 30-day commitment and 79 for an additional 180-days).

RAND cited the number of individuals under IOC in the eight other states as suggesting that IOC will be used primarily as a discharge planning vehicle for a small number of individuals. New York officials initially estimated that 7,000 individuals would be placed on outpatient commitment orders under Kendra's Law, yet as of September 2000 only 235 involuntary outpatient petitions had been filed.

RAND concluded that the data failed to answer the question of whether developing an IOC system in California is worth the added costs to mental health treatment systems, the courts and law enforcement. The researchers find "no direct evidence to suggest that simply amending the statutory language is likely to produce the desired results. Investments would need to be made in developing and sustaining an infrastructure for implementation."

February 2000, updated July 2001