Performance Improvement 1997
Executive Summary
Performance Improvement 1997 is the second annual report of the U.S.
Department of Health and Human Services (HHS) on its evaluation activities. As a
report to Congress, it summarizes the findings of HHS evaluations completed in
fiscal year (FY) 1996. In that year, HHS agencies produced 87 evaluation reports and
supported more than 250 evaluation projects in progress.
In general, the report is intended for three audiences: decisionmakers, who need
information on program results across the broad spectrum of health and human
services; program managers, who need information on how they can improve program
operations and outcomes; and the health and human services community, which can
benefit by applying the knowledge and lessons learned from HHS evaluations.
In the Foreword, Secretary Donna Shalala and David Garrison, Principal Deputy
Assistant Secretary for Planning and Evaluation, stress that evaluation is an
important tool for producing the knowledge necessary to develop and improve the
performance of HHS's activities to meet the needs of the 21st century. HHS program
managers are continuously engaged in efforts to determine whether programs and
services reach the intended populations or communities, perform efficiently, and
achieve desired results.
Chapter I
Chapter I provides an overview of evaluation at HHS. It describes a variety of
evaluation activities; sources of funding; and evaluation management policies,
including planning and coordination, project management, quality assurance,
dissemination, and effective use of results. The last section in this chapter
discusses future directions for HHS evaluation. In the coming years, the Department
will examine the transformations now taking place in health and human services. The
transformations underscore the need of the Department to play a leadership role in
developing performance measures for implementation of the Government Performance
and Results Act of 1993 (GPRA). HHS agencies remain committed to seeking information
to continuously improve the operation of their programs.
Chapter II
Chapter II highlights 11 evaluations completed during FY 1996 and identified by the
panel of Senior Editorial Advisors as particularly useful to the public health and
human services community. For the selection criteria, see appendix C.
Evaluation of Child Access Demonstration Projects: Report to Congress
This Administration for Children and Families (ACF) Office of Child Support
Enforcement report evaluates different forms of interventions to bring
noncustodial parents closer to their children after divorce and separation. The
interventions included mediation, parenting training, counseling, enforcement of
visitation, and monitoring of visitation. The report found that mediation where
both parties attend resulted in parenting plans in 65 to 70 percent of the cases.
These parenting plans stimulated more visitation by noncustodial parents and better
compliance with child support. A majority of both parents were satisfied with
mediation. Other forms of intervention for more longer term and problematic cases
did not register impact.
Evaluation of Child Support Guidelines
This ACF report evaluated the implementation of State child support guidelines
mandated by the Family Support Act of 1988. The investigation concluded that States
need to increase the consistency of guideline application, including income
verification procedures, extended custody and visitation arrangements, and
consideration of multiple family situations, health insurance, and day care
expenses in the calculation of child support awards. Adopting a standardized
support order would improve documentation of departures from guideline award
calculations.
Serving Elders at Risk: The Older Americans Act Nutrition Programs. National
Evaluation of the Elderly Nutrition Program, 1993-1995
The Elderly Nutrition Program sponsored by the Administration on Aging provides
grants to State units on aging to subsidize the provision of daily meals to people
60 years of age or older. This evaluation report describes participant
characteristics, compares the characteristics of recipients of home-delivered
meals with those receiving meals in congregate settings, assesses how well the
program reaches disabled and poor elderly, estimates the impact on nutritional
intake and social contacts of participants, and examines program costs and other
sources of funding. Findings suggest that participants have higher daily intakes
of nutrients and more social contacts per month than a comparable group of
nonparticipants. The Elderly Nutrition Program succeeds in targeting older people
who are poor, live alone, are nutritionally "at risk" because they are
overweight or underweight, or are more functionally disabled than their age-group
peers. Federal expenditures were found to be highly leveraged with State, local, and
private funds.
Oregon Consumer Scorecard Project (OCS)
The final report on the Agency for Health Care Policy and Research (AHCPR)-supported
Oregon Consumer Scorecard Project describes the Oregon Consumer Scorecard
Consortium's development, pilot testing, and revision of a user-friendly
guidebook, "A Consumer Guide to Selecting a Health Plan." The prototype
scorecard was designed to help consumers choose a health plan consistent with their
individual needs and their health care service delivery preferences. The model may
also help purchasers and State policymakers develop their own comparative reports
about health plan performance.
Georgia State Clinic Assessments of Immunization Coverage: Impact of
Management and Clinic Immunization Practices
Between 1986 and 1994, the State of Georgia implemented a program of audits of the
immunization practices of all State public health immunization clinics in an effort
to improve child immunization rates. During this time, the immunization rate for
preschoolers increased from 31 percent to 90 percent. The Centers for Disease
Control and Prevention's (CDC's) evaluation of the program found that the
performance of these audits was significantly associated with higher immunization
rates for preschool children in the clinic catchment areas. This and other report
findings formed the basis of CDC recommendations to all States.
Ventilator-Dependent Unit Demonstration: Outcome Evaluation and Assessment
of Post-Acute Care
This Health Care Financing Administration evaluation assessed whether specialized
units serving chronic ventilator-dependent patients deliver better clinical
outcomes at a reasonable cost to Medicare than conventional hospital settings.
Findings included improved patient outcomes for patients treated in
ventilator-dependent units and lower hospital daily costs, yet higher overall costs
to Medicare because of greater patient longevity. The analysis of costs and outcomes
provides important information to policy makers and health care providers about a
seriously ill population with intensive resource needs.
Evaluation of the Bureau of Health Professions Strategic Directions:
Development of a Cross-Cutting Performance Monitoring System
This Health Resources and Services Administration (HRSA) evaluation reports on
Phase II of a three-phase effort to develop a fully integrated computerized data
system to facilitate planning and evaluation of the programs administered by the
Bureau of Health Professions. A performance monitoring system is proposed to
facilitate ongoing program management. Such a performance system is central to the
ability of the Bureau of Health Professions to measure whether program support is
meeting its national health workforce objectives, and to signal where program
course correction is necessary.
Analysis of the Effect of Regulation on the Quality of Care in Board and Care
Homes
This evaluation report, sponsored by the Office of the Assistant Secretary for
Planning and Evaluation (ASPE), assesses the effects of State board and care home
regulations on the quality of care provided by those institutions. The
investigation also attempts to identify the characteristics of the board and care
environments in the States selected for the study and to determine whether licensed
and unlicensed homes differed in terms of those characteristics, as well as in the
quality of care provided. The evaluation concludes that extensive State regulation
and licensure can improve the quality of care provided by board and care homes and
can increase the number of these facilities that hold licenses.
Health Care in Transition: Technology Assessment in the Private Sector
This report sponsored by ASPE briefly describes a number of private-sector health
organizations that are engaged in technology assessment. Technology assessment is
one of a family of evaluative activities circulating within these health care
organizations. Among private-sector organizations conducting technology
assessment, rigorous evaluation of clinical effectiveness based on a systematic
review of scientific and clinical evidence has become the norm, and technology
assessment has increasingly become "full service," encompassing drugs,
medical devices, and clinical procedures.
Subacute Care: Policy Synthesis and Market Area Analysis
The report of this ASPE study of subacute care presents an overview and historical
perspective of subacute care, a description of the current state of the art and how
it varies across selected market areas, and an examination of evidence about its
cost and effectiveness. Findings suggest that the emerging concept of subacute care
holds promise but also poses many challenges for the public and private sectors.
Cost-Effectiveness in Health and Medicine
Cost-effectiveness analysis (CEA) is one tool available to decisionmakers to help
determine the relative value of different approaches to improving health or life
expectancy. For a variety of reasons, however, CEA has not been widely used in
health policy settings. This Office of Public Health and Science report represents
a review of the methodological and theoretical aspects of CEA, its effectiveness,
and its expected outcomes and uses. Recommendations are made regarding framing and
designing such studies, identifying and valuing outcomes, assessing effectiveness,
estimating costs, discounting, reflecting analytical uncertainty, and reporting
analyses.
Chapter III
Chapter III provides an overview of the evaluation activities of the HHS agencies,
including information on the evaluation program, a summary of evaluations completed
during FY 1996 and evaluations in progress, and a discussion of future directions
for agency evaluations.
Administration for Children and Families
ACF administers a broad range of programs to support children and families and
targeted populations, such as the developmentally disabled and Native Americans.
The objectives of ACF evaluations are to provide information on the program design
and operations, to test delivery approaches of various services, to conduct policy
analyses, and to disseminate and apply evaluation results. In FY 1996, evaluations
were completed on child support guidelines, child access demonstration projects,
Head Start family self-sufficiency initiatives, home-based services for runaway
youth, gang families in public housing projects, foster youth mentors, child
protective and preventive services, family-based placement prevention programs,
and a case management enhancement project. Major evaluations in progress include
the Arkansas prenatal and postnatal paternity project, responsible fatherhood
initiatives, home visitor services for teen parents, community development
demonstrations, the Head Start program, and foster care placement. Two new National
Child Welfare Research Centers have also been established.
Administration on Aging
The Administration on Aging (AoA) supports studies that provide information on the
implementation of the Older Americans Act of 1992. This year, AoA completed a major
evaluation of the Elderly Nutrition Program, which provides an average of one
million meals per day to older Americans. Results showed that the program has been
successful in improving the nutritional intakes of elderly people and in decreasing
their social isolation. The report also provided guidance to help the program change
to meet future needs of the elderly population and adapt to changes under way in the
health care system. AoA is now helping many State and area agencies conduct
evaluations of programs supported by the Older Americans Act.
Agency for Health Care Policy and Research
The goals of the AHCPR evaluation program are to assess agency effectiveness in
meeting major and long-term priorities; to obtain information to respond to
critical agency and departmental concerns; and to conduct internal evaluations to
improve the efficacy of key program areas. Evaluation projects completed in FY 1996
included an evaluation of practice guidelines and science-based measures of
clinical quality; an evaluation of CONQUEST 1.0, a data base of clinical performance
measures useful for matching measures appropriate for different clinical
conditions; the Oregon Consumer Scorecard Project; and a synthesis of findings from
AHCPR's Medical Treatment Effectiveness Program (MEDTEP). In progress during FY
1996 were evaluation activities to assess the health care environment; evaluations
of AHCPR publications for research dissemination and training for minority health
services researchers; and several evaluation design studies, such as an assessment
of data release strategies for the AHCPR's Medical Expenditure Panel Survey.
Centers for Disease Control and Prevention
CDC places high priority on evaluations seeking to answer policy, program, and
strategic planning questions related to the Agency's mission. This year's major
evaluations include data policy and surveillance studies, such as developing a
comprehensive monitoring system to track the impact of health care reform. Program
evaluations completed include CDC's National Nosocomial Infections Surveillance
Program and the Dengue Hemorrhagic Fever Prevention and Control Program in San Juan,
Puerto Rico. CDC also assisted the State of Georgia in a study of immunization
auditing procedures and found positive results for increasing child immunization
rates. CDC's evaluations in progress include evaluations of data reporting systems,
such as one for public health laboratories and CDC's National Electronic
Telecommunications System for Surveillance. Program evaluations are under way in
the areas of occupational health hazards, epidemiology training, and prevention
centers. Other specialized projects are focusing on tuberculosis outreach workers,
minors' access to tobacco, and violence prevention.
Food and Drug Administration
Evaluation at the Food and Drug Administration (FDA) is largely a line-management
responsibility. Projects focus on performance management, customer participation,
and rigorous rulemaking. In FY 1996, FDA continued to identify performance measures
jointly with industry customers as part of the drug and biologic review processes
necessary for implementing the Prescription Drug User Fee Act. In addition, FDA
completed two evaluation projects on food labels. The first reports on the impact
of the new food label regulations on consumers, and the second assesses the status
of processed, packaged food labels subsequent to publication of the regulations.
During FY 1996 FDA program managers were engaged in evaluating their performance
measures in light of GPRA standards. Other evaluation projects at FDA include
assessing FDA's penalty-reduction and inquiries assistance program for small
businesses, influenza virus vaccines, and the national evaluation of x-ray trends.
Health Care Financing Administration
The Office of Research and Demonstrations performs and supports Health Care
Financing Administration (HCFA) evaluations to develop new health care financing
policies and to provide information on the impact of HCFA's programs. In FY 1996,
HCFA completed evaluations of ventilator-dependent hospital unit care, Medicaid
drug rebate policies, the Arizona health care cost-containment system,
telemedicine, home health prospective payments, physician payment reform, and
pharmacy benefit managers. HCFA has over 34 projects in progress, with 24 scheduled
to be completed in FY 1997. These projects are monitoring the Medicare and Medicaid
programs through demonstrations. Some of the demonstrations involve the Medicaid
managed care programs or waivers, Medicare case management demonstrations,
ambulatory patient groups, Medicaid uninsured demonstrations, rural health
clinics, community-supported living arrangements, Medicare Alzheimer's disease
demonstration, long-term care for persons with developmental disabilities, and
health plan report cards.
Health Resources and Services Administration
The objectives of the Health Resources and Services Administration (HRSA)
evaluation program are to improve program management and policy development and to
provide information for strategic planning, budget decisions, and legislation.
During FY 1996, HRSA continued to enhance its performance measurement capacity for
all line programs, with a notable study completed at the Bureau of Health
Professions. Special evaluation projects were performed on quality assurance
procedures for Ryan White Comprehensive AIDS Recovery Emergency (CARE) Act
grantees; managed care and the Federally Qualified Health Centers; the impact of
case management in community health centers; emergency medical services for
children; international medical graduates; advanced practice nursing education;
and primary care residency programs. Ongoing evaluations in FY 1996 include managed
care and community and migrant health centers; health maintenance organization, or
HMO, staffing strategies in underserved areas; casemix differences in health
centers; assessment of multilingual services offered at community and migrant
health centers; the Healthy Start Program to reduce infant mortality; Ryan White
CARE Act programs; organ transplantation; telemedicine; and the supply of nurses.
Indian Health Service
The Indian Health Service (IHS) evaluation program serves the agency's program and
policy objectives, developed in consultation with the tribal communities. IHS's
major evaluations completed in FY 1996 concentrated on developing a methodology for
adjusting IHS mortality data for inconsistent classifications; the IHS adolescent
regional treatment centers; and the second phase of an evaluation of alcohol and
substance abuse services for American Indian and Alaska Native women. Projects in
progress are focusing on the impact of a promotional initiative on elderly wellness
and use of comprehensive services; a study of prior trauma care of intoxicated
patients as a predictor of subsequently fatal injury; an assessment of Resource
Requirements Methodology as an IHS management tool; the development of an IHS health
services research agenda; and the continuing evaluation of alcohol and substance
abuse treatment services for women.
National Institutes of Health
Evaluation is an integral part of the role of the National Institutes of Health
(NIH) in supporting biomedical research, training, and public education. In FY
1996, NIH's evaluations focused on the NIH shared instrumentation grant programs,
clinical research at the National Institute of Diabetes and Digestive and Kidney
Diseases, an evaluation of the Navajo alcohol rehabilitation demonstration, and the
National Institute on Drug Abuse's study on the development of medications for the
treatment of opiate and cocaine addictions. The major evaluations in progress
during FY 1996 were an evaluation of the career status and satisfaction of NIH grant
applicants with grant application processes and procedures; comprehensive school
health programs in grades K-12; an evaluation of laboratory animal use; and the
development of a data base for NIH-supported research in rare diseases.
Office of the Assistant Secretary for Planning and Evaluation
The Assistant Secretary for Planning and Evaluation (ASPE) functions as principal
advisor to the secretary on policy development, conducting evaluations and policy
research studies. It is also responsible for HHS-wide coordination of legislative,
planning, and evaluation activities. During FY 1996, ASPE's evaluations looked at
health care technology assessment, information to guide physician practice,
assumptions underlying insurance participation modeling, board and care homes,
subacute care, integrating acute and long-term care for children with disabilities,
interdisciplinary professional training on disabilities, substance abuse
treatment for parents and welfare recipients, community response to domestic
violence, minority male violence prevention, and American Indian demography. The
major evaluations in progress include monitoring changes in health and human
services, assessing data needs, managed care and people with disabilities, assisted
living, family preservation services, moving welfare recipients to work, promoting
involvement of fathers in family life, and trend data on the well-being of children.
Office of Public Health and Science
The Office of Public Health and Science (OPHS) provides advice, policy and program
coordination, and leadership in the implementation, management, and development
of HHS public health and science activities. Several major OPHS-supported
evaluations were completed in FY 1996 on the cost-effectiveness methodology;
quantification of State public health expenditures; health needs of recently
arriving refugees; and scientific misconduct and the consequences of being accused.
Evaluations and policy studies in progress include dietary supplement labels;
framework for Healthy People 2010, school health programs, public health
infrastructure, and public health performance measures.
Substance Abuse and Mental Health Services Administration
The Substance Abuse and Mental Health Services Administration (SAMHSA) is committed
to evaluating its programs and grant projects to assess the effectiveness of
prevention, treatment, and rehabilitation approaches and system of care. In FY
1996, SAMHSA completed two evaluation projects. The first was an overview of
addiction treatment effectiveness; the second was a study of the employment
outcomes of indigent clients receiving alcohol and drug treatment in Washington
State. Ongoing evaluations include evaluations of SAMHSA's demonstration programs,
such as the Job Corps program of enriched substance abuse treatment for adolescents;
the services integration experiment for chronically mentally ill homeless persons;
and the National Treatment Improvement Evaluation Study. Evaluations are also under
way to examine the impact of managed care on the access, cost, and quality of
substance abuse treatment.
Appendices
A complete inventory of the 81 HHS evaluations completed in FY 1996 is provided in
appendix A, and HHS agency projects currently in progress are listed in appendix B.
Appendix C describes the Senior Editorial Advisor's review criteria for assessing
program evaluations. A glossary of acronyms used throughout this report is
presented in appendix D.
Return to Table of Contents.