|
|
Office of the Assistant Secretary for Planning and Evaluation |
Policy Information Center |
MISSION: To improve the quality and availability of prevention, treatment, and rehabilitation services for substance abuse and mental illness.
Evaluation Program
The Substance Abuse and Mental Health Services Administration (SAMHSA) is committed to evaluating its overall programs and individual grant projects to assess the effectiveness of prevention, treatment, and rehabilitation approaches and systems of care; the accountability of Federal funds; and the achievement of SAMHSA's programmatic and policy objectives.
To the greatest extent appropriate and feasible, SAMHSA encourages the use of comparable data elements and instruments across its evaluations in order to work toward a comprehensive evaluation system and to minimize respondent burden.
SAMHSA conducts grant programs under a variety of legislative authorities. These authorities can generally be grouped into two types: (1) services and (2) knowledge development and application (KDA). The evaluation required for a particular grant program is dependent on the type and purpose of the program. SAMHSA evaluates each of its service programs so as to provide information to program managers about the accountability of Federal funds. In addition, the evaluations of KDA programs will generate new knowledge to lead the field in the development of policies that improve services.
The two types of grant programs (service delivery and KDA) represent the two facets of SAMHSA's mission. SAMHSA's leadership in the field depends on the successful interaction of these two facets. Through KDAs, SAMHSA must identify effective approaches to prevention, treatment, and rehabilitation. Through service delivery funds, SAMHSA must provide incentives to the field to implement effective approaches. Major emphases of SAMHSA's mission are to develop, identify, and disseminate effective strategies and systems for prevention, treatment, and rehabilitation.
SAMHSA is implementing an integrated model of evaluation and planning. Strategic planning identifies priorities, such as managed care, that drive the development of grant programs and evaluations. In compliance with the Government Performance and Results Act (GPRA), SAMHSA is improving performance management by identifying annual performance objectives and measures. The formulation of programmatic and evaluation priorities includes consultation with SAMHSA Center Advisory Councils, and with other experts in the fields of evaluation and service delivery. Early and continuous coordination of program planning and evaluation design will result in the articulation of program objectives that may be evaluated. Evaluations will demonstrate the extent to which the grant programs have achieved their overall objectives, and SAMHSA will translate these results into information that can be used for program and policy development. The strategic planning and policy development processes will then use these results to refine SAMHSA's priorities and performance objectives.
This evaluation policy will help SAMHSA achieve its goal of continually informing policy and program development with knowledge culled from past performance. In this way, SAMHSA can best serve its customers by enhancing the quality of publicly-funded substance abuse and mental health services.
In compliance with the Public Health Service (PHS) guidelines for the technical review of evaluations, SAMHSA has established a standing committee of PHS staff who are evaluation specialists. Representatives of the Office of the Assistant Secretary for Planning and Evaluation serve as ex officio members of the committee. The SAMHSA evaluation officer is the committee chair. The committee does not generally review the evaluation proposals of individual grantees; rather, it reviews proposals for broader, more comprehensive evaluations, such as the cross-project evaluations of grant programs.
Evaluation project proposals are generally prepared by SAMHSA program staff in the various Centers. The standing committee reviews each proposal on the following criteria: clarity of evaluation objectives and research questions, appropriateness and feasibility of the specifications for evaluation design and methods, appropriateness of the plans for dissemination of results, and use of previous relevant evaluations and existing program data systems. Each proposal must clearly state the relationship of the evaluation to SAMHSA's overall policies, priorities, and evaluation program.
The National Evaluation Data Services (NEDS) of the Center for Substance Abuse Treatment (CSAT) conducted a secondary analysis of data collected in the National Treatment Improvement Evaluation Study (NTIES) to address the problems experienced by those who entered treatment for alcohol use. Results indicated that those using only alcohol tended to be older, more often white, somewhat more educated, and more likely to be employed before entering treatment. Those who used alcohol in addition to other drugs significantly reduced their consumption of drugs after treatment, but clients who had used only alcohol (who were by definition drug-free in the 12 months before treatment) showed minor but statistically significant post-treatment increases in various forms of drug use. With the exception of driving under the influence (DUI) and driving while intoxicated (DWI) offenses, clients who had used only alcohol exhibited fewer instances of criminal behavior and arrests before treatment. On the other hand, they were more likely to have been referred for treatment by the criminal justice system (presumably for these driving offenses) and less frequently sought treatment on their own. Nevertheless, all groups in the study demonstrated substantial reductions in criminal behaviors as well as improvement in general and mental health and employment circumstances during follow-up periods. No significant post-treatment reductions in the self-reported use of alcohol were observed among all concerned, although this finding may be of clinical significance only for those who entered treatment for alcohol use alone. Among the recommendations included a need to define "best practices" in the treatment of alcoholism, bearing in mind that the population of those whose problem is exclusively due to alcohol use may be "aging out".
AGENCY SPONSOR: Center for Substance Abuse Treatment
FEDERAL CONTACT: Smith, Ron
301-443-7730
PIC ID: 5994
PERFORMER: Caliber Associates
Fairfax, VA
Society continues to suffer the immense costs and consequences associated with drug use and crime. Drug use prevalence among arrestees has been 50-80 percent in some locations. Policy makers need better information about drug use among criminal populations to more effectively respond to these problems. This research project applied three types of synthetic estimation techniques to estimate the number of drug users in a larger population where the prevalence rates are not known. The report from this project was produced by the Department of Health and Human Services Center for substance Abuse Treatment, through the National Evaluation Data and Technical Assistance Center. To improve prevalence estimation methods, the UCLA Drug Abuse Research Center conducted an initial study of regression modeling to produce their estimates, which resulted in improved conceptualization and preliminary applications of a regression model. This report applies the same regression model to larger scale data and applies it to all jurisdiction levels. The major finding indicates multiple capture rates are significantly higher than those reported in the literature (typically 13%) and their rates dramatically differ by gender, race/ethnicity, and crime type. In conclusion, the report found regression models of synthetic estimation to be sensitive and capable of providing high-confidence estimates of drug use prevalence at local, state, and national levels.
AGENCY SPONSOR: Center for Substance Abuse Treatment
FEDERAL CONTACT: Smith, Ron
301-443-7730
PIC ID: 5994.6
PERFORMER: Caliber Associates
Fairfax, VA
This analytic report examines the outcomes of methadone-treated clients in the National Treatment Improvement Study (NTIES). It addresses two fundamental questions: "How much methadone treatment is required in order to achieve successful outcomes?" and "To what extent will favorable outcomes of methadone treatment persist following the client's termination from treatment?" The results were consistent with findings from other national studies which suggest that better treatment outcomes for outpatient methadone treatment clients are associated with both longer stays in treatment and still being in the program during follow-up. The current study found consistently favorable outcomes including reductions in drug use, HIV/AIDS risk, and reduced criminal behaviors for clients who were maintained on methadone for 12 or more months and discharged clients who were treated for 3 to 12 months. At the policy level, the findings provide support for the expansion of methadone treatment, the expansion of aftercare services for methadone treated clients and the expansion of ancillary services, such as transportation and day care. In terms of implications for practice, information and referral services should be provided to assist discharged clients locate alternative treatment resources.
AGENCY SPONSOR: Center for Substance Abuse Treatment
FEDERAL CONTACT: Smith, Ron
301-443-7730
PIC ID: 5994.5
PERFORMER: Caliber Associates
Fairfax, VA
The National Evaluation Data Services (NEDS) of the Center for Substance Abuse Treatment (CSAT) explored two methodological issues of importance to substance abuse treatment researchers and policy analysts alike: a lack of response by a group of those under treatment whose behavior is being studied over time and selection bias in recruiting such study participants. The purpose of the study conducted was to determine the extent and effects of these two issues in a recent series of four large-scale follow-up studies: the California Drug Alcohol Treatment Assessment (CALDATA), the Services Research Outcomes Study (SROS), and the National Treatment Improvement Evaluation Study (NTIES), and the Drug Abuse Treatment Outcome Study (DATOS). Differences in follow-up response rates among these four studies appear to be due partly to differences in research design and partly to a difference between survey organizations in follow-up effectiveness. These and other findings as well as recommendations were issued. (See also PIC ID #s 5994.1-5994.3.)
AGENCY SPONSOR: Center for Substance Abuse Treatment
FEDERAL CONTACT: Smith, Ron
301-443-7730
PIC ID: 5994.3
PERFORMER: National Opinion Research Center (NORC)
Chicago, IL
An important service of the Center for Substance Abuse Treatment (CSAT) is to create more effective substance abuse treatment and recovery services by developing and analyzing good data on which to base such programs. One of the specific activities of the National Evaluation Data Services (NEDS), created by CSAT, is to pull together databases for broad analytic purposes or to house databases produced under a wide array of activities. This study conducted by NEDS compares research methods, provider and patient characteristics, and outcome results from four large-scale follow-up studies of drug treatment during the 1990s: the California Drug and Alcohol Treatment Assessment (CALDATA), Services Research Outcome Studies (SROS), and the National Treatment Improvement Evaluation Study (NTIES), and Drug Abuse Treatment Outcome Study (DATOS) (See also PIC ID Numbers 5994, 5994.1, 5994.3 and 5994.4). From before to after treatment, each study revealed across-the-board positive changes in drug use, crime, and health status. Clients in both short-term and long-term residential treatment experienced positive changes in drug use, crime, employment and health more often than clients in outpatient treatment. Clients discharged from (rather than maintained on) methadone displayed positive changes less often than any other group in treatment. Clients in the SROS showed consistently fewer changes than others on drug use, crime, employment, and health. This finding may be due to the longer period encompassed by the study and lower precision in measuring long-term patterns of change through the study's measurement of drug use, crime, employment, and health. Further studies are needed on the potential of these data to capture differences in client populations, organization of services, and clinical practices that may explain differential outcomes and costs.
AGENCY SPONSOR: Center for Substance Abuse Treatment
FEDERAL CONTACT: Smith, Ron
301-443-7730
PIC ID: 5994.2
PERFORMER: National Opinion Research Center (NORC)
Chicago, IL
One important mission of the Center for Substance Abuse Treatment (CSAT) is to expand the knowledge about and the availability of effective treatment and recovery services. To aid in accomplishing this mission, CSAT has made a significant commitment to developing and acquiring high-quality data about treatment of clients, the services provided for them, and outcomes. By establishing National Evaluation Data Services (NEDS), CSAT has attempted to provide a wide array of data management and scientific support services to gain evidence on which to base answers to questions about what kinds of treatment are most effective for curbing addiction-related behavior. This study uses data from the National Treatment Improvement Study (NTIES), a national evaluation of the effectiveness of substance abuse treatment services delivered in comprehensive treatment demonstration programs supported by CSAT, to analyze what service-delivery unit factors such as providing vocational training, increase the odds of program completion. The study was based on NTIES intake and treatment episode completion status data for 3,117 clients across 61 service delivery units. Results indicated that: In general, shorter programs have higher treatment episode completion rates but longer programs retain clients for longer periods; long-term residential programs exhibit higher treatment episode completion rates than long-term outpatient (non-methadone) or correctional programs; and the median length of stay for clients in long-term outpatient (non-methadone) or correctional programs with intended lengths of treatment of 120 days or more were lower than in the 90 to 119-day programs, suggesting that setting the intended length of treatment at 120 days or more may be counterproductive to retaining clients. Among the avenues suggested for further analysis was investigating why clients leave programs. (See also PIC ID#s 5994, and 5994.2 through 5994.4.)
AGENCY SPONSOR: Center for Substance Abuse Treatment
FEDERAL CONTACT: Smith, Ron
301-443-7730
PIC ID: 5994.1
PERFORMER: Battelle Corporation
Arlington, VA
The Center for Mental Health Services (CMHS) is completing an evaluation of 22 grant sites of the Comprehensive Community Mental Health Services for Children and their Families Program funded in fiscal years 1993 and 1994. Local evaluators hired at each site have gathered basic descriptive information on about 40,000 children enrolled across all of the sites. They also have obtained information on functional outcomes at four points in time including intake, six months, 12 months, and 24 months for almost 1,000 children who remain continuously in services. External evaluators visited the sites annually to assess the degree to which each site implemented services in accordance with system of care goals and philosophy. A reliable system-of-care implementation tool was developed. Preliminary results indicate that a high percentage of children improve or maintain their functional outcomes, and that collaboration with the child welfare, juvenile justice, and education sectors has increased as indicated by a higher number of referrals from these sectors over the years in the program. A study comparing three systems of care with three service delivery systems will provide empirical evidence on the effectiveness of care systems.
AGENCY SPONSOR: Center for Mental Health Services
FEDERAL CONTACT: Santiago, Rolando L., Ph.D.
301-443-3808
PIC ID: 7269
PERFORMER: Macro International, Inc.
Calverton, MD
EXPECTED DATE OF COMPLETION: 12/31/2000
The Center for Mental Health Services (CMHS) is conducting an evaluation of 23 sites of the Comprehensive Community Mental Health Services for Children and their Families Program funded in fiscal years 1997 and 1998. Evaluators hired at each site currently gather descriptive data on such measures as demographics, functional status, diagnosis, and risk factors of enrolled children and their families as well as outcome data in such areas as social functioning, substance abuse, school performance, and juvenile justice involvement in a representative sample of 225 children. The children are assessed every 6 months for up to 36 months. A service-level substudy includes a provider survey and measures of multisector service contacts, client satisfaction, and adherence to a service plan. External evaluators visit the sites annually to assess implementation of services according to system care goals and philosophy. Three systems of care are being compared with three usual service systems to study how systems of care develop over time. Preliminary analyses indicate that 21 percent of referrals come from juvenile justice agencies, 21 percent from mental health providers, 20 percent from parents, 19 percent from schools, 9 percent from child welfare programs, and the remaining 10 percent from self or other sources.
AGENCY SPONSOR: Center for Mental Health Services
FEDERAL CONTACT: Santiago, Rolando L., Ph.D.
301-443-3808
PIC ID: 7269.1
PERFORMER: Macro International, Inc.
Calverton, MD
EXPECTED DATE OF COMPLETION: 9/29/2001
The Center for Mental Health Services (CMHS) will begin an evaluation of 20 new sites of the Comprehensive Community Mental Health Services for Children and their Families Program funded on September 30, 1999. Evaluators hired at each site will gather descriptive data on such measures as demographics, functional status, and participation in the Child Health Insurance Program of enrolled children and their families. These evaluators also will collect outcome data in such areas as social functioning, substance abuse, school performance, and juvenile justice involvement in a representative sample of children. External evaluators will visit the sites every 18 months to assess service implementation according to system of care program goals and philosophy. Local sites will receive technical assistance on how to create logic models to describe and self-monitor the development of their systems of care. The evaluation contractor will develop an Internet-based system to enter and monitor data collected at local sites to disseminate findings through Web-based technology. The evaluation contractor will also conduct a study on the effectiveness of specific services, treatments, or supports delivered within a selected system of care site.
AGENCY SPONSOR: Center for Mental Health Services
FEDERAL CONTACT: Santiago, Rolando L., Ph.D.
301-443-3808
PIC ID: 7269.2
PERFORMER: Macro International, Inc.
Calverton, MD
EXPECTED DATE OF COMPLETION: 9/29/2003
The ACCESS Program is a five year demonstration program that provides grants to nine States to develop integrated systems of treatment and supportive services and housing for homeless persons with serious mental illnesses. The goal of the program is to identify strategies for developing integrated service systems and to evaluate their effectiveness in providing services to homeless persons with serious mental illnesses. The evaluation of the ACCESS Program has two major components: The system-level evaluation identifies the different systems integration approaches, documents how they are implemented, and measures their impact on system capacity and system barriers. The client-level evaluation determines whether systems integration efforts result in improved service delivery, improvements in mental health and health status, rehabilitation, quality of life and permanent exit from homelessness.
AGENCY SPONSOR: Center for Mental Health Services
FEDERAL CONTACT: Frances, Randolph, Ph.D.
301-443-3706
PIC ID: 4980.1
PERFORMER: R.O.W. Sciences, Inc.
Rockville, MD
EXPECTED DATE OF COMPLETION: 7/31/2000
This project will conduct a cross-site evaluation of the Community Prevention Coalition (CPC) grant program over a five-year period. The goals of the evaluation are to assess: (1) the extent to which coalitions of service organizations created by the program are able to implement comprehensive and coordinated alcohol, tobacco, and other drug (ATOD) prevention programs through an expanded and non-duplicative service delivery base; (2) the extent to which coalitions are able to achieve measurable reductions in the incidence and prevalence of ATOD use, and improvements in ATOD-related social, health and safety consequences among all age, ethnic and vulnerable groups within grantee communities; (3) the contributions of various prevention strategies to site-specific outcomes and to general prevention outcomes in those sites where measurable changes are made; and (4) those characteristics common to coalitions where measurable changes are achieved, as well as those that differentiate successful coalitions from unsuccessful ones. The contractor will be required to collect process and outcome data for analysis from coalitions. In addition, data collected locally by grantees and archived community-wide indicators will be used.
AGENCY SPONSOR: Center for Substance Abuse Prevention
FEDERAL CONTACT: Kaftarian, Shakeh, Ph.D.
301-443-9136
PIC ID: 6042
PERFORMER: Cosmos Corporation
7474 Bethesda, MD
EXPECTED DATE OF COMPLETION: 5/31/2000
This project is designed to conduct an evaluation of the 48 High Risk Youth Substance Abuse Prevention Initiatives funded by the Center for Substance Abuse Prevention (CSAP) in 1994 and 1995, especially in regard to the following areas: (1) female adolescents, (2) high risk youth (HRY), and (3) replication of model programs for the prevention of alcohol, tobacco and other drug (ATOD) use among high risk youth. The purpose of the evaluation is to: (1) assess the impact of all programs in preventing or reducing the use of ATOD among youth at high risk for such behaviors, and (2) assess the effectiveness of specified intervention strategies in reducing the risk factors or enhancing the protective factors on which the specific programs are focused.
AGENCY SPONSOR: Center for Substance Abuse Prevention
FEDERAL CONTACT: Sambrano, Soledad, Ph.D.
301-443-9136
PIC ID: 6044
PERFORMER: EMT Associates
St. Louis, MO
EXPECTED DATE OF COMPLETION: 4/30/2000
The Center for Substance Abuse Treatment (CSAT) and other federal agencies are developing a regulatory framework for opioid substitution therapy that will change the current regulatory system for monitoring opioid treatment programs (OTPs) to an accreditation/regulatory approach. Over a five year period, CSAT will conduct and evaluate a phase-in program to obtain information that will aid in the full implementation of the new system. The goal of the evaluation is to obtain information that can guide the full national implementation of the new system for OTPs through systematic study of the processes, barriers, costs associated with a change from regulatory to an accreditation/regulatory process, and various other impacts.
AGENCY SPONSOR: Center for Substance Abuse Treatment
FEDERAL CONTACT: Diesenhaus, Herman, Ph.D.
301-443-6575
PIC ID: 6739
PERFORMER: Research Triangle Park
Research Triangle Park, NC
EXPECTED DATE OF COMPLETION: 9/29/2002
The National Evaluation Data Services (NEDS) was established to increase the number of scientifically based analyses to answer vital questions in the substance abuse treatment field. NEDS represents part of an overall CSAT evaluation strategy that builds upon prior findings and seeks to mine existing data whose potential has not been fully explored. One objective of the NEDS project is to provide a wide array of data management and scientific support services across various program and evaluation activities, including an information system of available data related to the evaluation of substance abuse treatment. Another objective is to provide CSAT with a flexible analytical capability to use existing data to address policy-relevant questions about substance abuse treatment, and to clearly articulate the resulting analyses as they relate to policy through fact sheets, evaluation summaries, technical reports, methods development documents and technical guidance documents.
AGENCY SPONSOR: Center for Substance Abuse Treatment
FEDERAL CONTACT: Smith, Ron
301-443-7730
PIC ID: 5994.4
PERFORMER: Caliber Associates
Fairfax, VA
EXPECTED DATE OF COMPLETION: 9/30/2001
The purpose of PETS is to evaluate the long-term effectiveness (up to thirty-six months) of substance abuse treatment services provided through a series of CSAT grants and cooperative agreements, and to conduct a number of special studies and policy analyses that address specific drugs of abuse, methods of treatment, populations, or policy issues. The primary follow-up studies will involve obtaining information at selected follow-up periods for persons who have completed the index treatment episode and for whom historical, intake and treatment exit data are available.
AGENCY SPONSOR: Center for Substance Abuse Treatment
FEDERAL CONTACT: Smith, Ron
301-443-7730
PIC ID: 6738
PERFORMER: Westat, Inc.
Rockville, MD
EXPECTED DATE OF COMPLETION: 9/30/2002
The State Substance Abuse Managed Care Evaluation Project will evaluate the effects of managed care on mental health and substance abuse services in five States. Each of these five State studies will inform future behavioral health care policy.
AGENCY SPONSOR: Center for Substance Abuse Treatment
FEDERAL CONTACT: Dilonardo, Joan
301-443-8555
PIC ID: 6454
PERFORMER: Brandeis University
Waltham, MA
EXPECTED DATE OF COMPLETION: 7/31/2001
More than 25 Treatment Improvement Protocols (TIPs) have been distributed to the public. The purpose of TIPs is to transfer protocols and guidelines for the treatment of alcohol and other drug (AOD) abuse from acknowledged clinical, evaluation and administrative experts to the nation's AOD treatment field. Institution of a field evaluation is needed to: (1) assess the extent to which target audiences realize, read and implement TIPs; and (2) learn ways to strengthen the development, formatting, marketing, dissemination, use and evaluation of TIPs to ensure effective use of future Federal resources devoted to TIPs. The overall objective of the proposed evaluation is to learn the impact of TIPs on the substance abuse treatment field. The study will examine methods of dissemination used by CSAT, the success of those methods in reaching the target audiences, users' perceptions of the value of TIPs, decisions to implement TIPs, and the successes, correlates and barriers associated with implementation.
AGENCY SPONSOR: Center for Substance Abuse Treatment
FEDERAL CONTACT: Mulvey, Kevin, Ph.D.
301-443-9472
PIC ID: 6737
PERFORMER: Johnson, Bassin and Shaw, Inc.
Silver Spring, MD
EXPECTED DATE OF COMPLETION: 9/20/2001