Performance Improvement 1996
Executive Summary
Performance Improvement 1996 is the first in a series of
annual reports on evaluation activities of the U.S. Department of
Health and Human Services (HHS). As a report to Congress, it
summarizes the findings of HHS evaluations completed in fiscal
1995. In that year, HHS agencies produced 148 evaluation reports
and supported more than 200 evaluation projects in progress.
In general, the report is intended for three audiences:
decisionmakers who need information on program results across the
broad spectrum of health and human services; program managers who
need information on how they can make improvements in program
operations and outcomes; and the health and human services
community, which can benefit by applying the knowledge and
lessons learned from HHS evaluations.
In the Foreword, Secretary Donna Shalala and Principal Deputy
Assistant Secretary for Planning and Evaluation David Garrison
stress that evaluation is the primary source of knowledge for
improving the performance of HHS programs, which is the theme of
this report. Every year, program managers are engaged in efforts
to determine whether programs and services reach intended
populations or communities and achieve desired results. The
information obtained from HHS evaluations is useful for planning
new directions and adapting programs to meet new demands.
Chapter I
Chapter I provides an overview of evaluation at HHS. Described
are the variety of evaluation activities; sources of funding; and
evaluation management--including planning procedures, project
management, quality assurance, dissemination, and effective use
of results. The last section of the chapter discusses future
directions for HHS evaluation. First, in coming years, HHS will
examine the transformations now taking place in health and human
services. Second, the transformations underscore the need for HHS
to play a leadership role in developing and applying performance
measures. Third, HHS agencies remain committed to seeking
information to continuously improve the operation of current
programs.
Chapter II
Chapter II highlights 11 HHS evaluations completed during fiscal
1995 and selected by the panel of Senior Editorial Advisors as
evaluations considered as particularly useful to the public
health and human services community.
Strengthening Families and Neighborhoods: A
Community-Centered Approach
This Administration for Children and Families (ACF) report
describes the design, implementation, and evaluation of the Iowa
Patch project, which was undertaken to improve the quality,
accessibility, and accountability of social services to children,
youth, and families in a small "patch," or geographic
area, of Cedar Rapids, Iowa.
Youth With Runaway, Throwaway, and Homeless Experiences:
Prevalence, Drug Use, and Other At-Risk Behaviors
This report, also produced by ACF, describes the most detailed
and generalizable study undertaken to date on runaway, throwaway,
and homeless youth in the United States, and their associations
with high-risk behaviors such as substance abuse, suicide
attempts, unsafe sexual behavior, and criminal activities, both
inside and outside the home.
Real People, Real Problems: An Evaluation of the Long-Term
Care Ombudsman Programs of the Older Americans Act
This Administration on Aging report describes a 1-year project to
evaluate long-term care ombudsman programs, which investigate
complaints made by or on behalf of residents of nursing
facilities and board and care homes.
Understanding and Choosing Clinical Performance Measures for
Quality Improvement: Development of a Typology
Increasingly, private purchasers of health care, consumer groups,
Federal and State agencies, and health care plans are searching
for methods to compare clinical performance among providers and
health care systems. This Agency for Health Care Policy and
Research project develops a framework for evaluating the
structure and content of measures currently used to assess
clinical quality so that users can select measures that will help
guide choices, ensure accountability, provide data for quality
improvement, and track changes in the health care utilization and
quality.
The Effect of Mandated Managed Care for Medicaid Populations
on the Practice of Public Health: The Example of Childhood Lead
Poisoning Prevention
This Centers for Disease Control and Prevention study examined
the likely effects of managed care reforms on the delivery of
childhood lead poisoning prevention programs. The results
suggested that as managed care increases, appropriate childhood
lead screening will decrease, use of public health laboratories
for testing will decrease, and followup programs will be
performed outside the traditional health department.
Medicare Participating Heart Bypass Center Demonstration:
Evaluation Design
This negotiated bundled payment demonstration was implemented in
May 1991 by the Health Care Financing Administration (HCFA) as
one of many cost-containment demonstrations launched in response
to increases in Federal spending under Medicare. The study found
that both Medicare and hospitals could benefit from this type of
payment arrangement through reduced costs and better coordination
of care without sacrificing quality of care.
Monitoring the Impact of Medicare Physician Payment Reform on
Utilization and Access
This report to Congress prepared by HCFA examined the impact of
legislation that reformed Medicare payment methodology to
physicians. As Congress intended, the legislation shifted
utilization away from surgical procedures and toward medical
evaluation and management. In addition, the study showed that the
legislation did not create new barriers to care for vulnerable
patient groups. The report documents the strong racial and income
disparities in utilization.
Performance Indicators for Government Performance and Results
Act: Initial Assessment of Health Resources and Services
Administration Programs
There are increasing expectations across the Federal Government
that agency programs, with a given set of resources and through a
series of actions and decisions, will produce the desired
outcomes and products for the intended audiences or
beneficiaries. This project of the Health Resources and Services
Administration helped the Agency begin planning for performance
and outcomes measurement.
Evaluation of Diabetes Services Provided by IHS Model Diabetes Program:
Final Report
This study evaluates the utility of a model diabetes
demonstration program created by the Indian Health Service in
1979. The critical element of the model program was a team of
health professionals who augmented primary care by providing a
combination of education, outreach, service delivery, and
referrals.
Fetal Alcohol Syndrome: Diagnosis, Epidemiology, Prevention,
and Treatment
A committee of experts was convened by the National Academy of
Sciences, Institute of Medicine, to evaluate published studies on
fetal alcohol syndrome and related disorders. The evaluation
synthesis was requested by the National Institute on Alcohol
Abuse and Alcoholism at the National Institutes of Health.
Consequences of Whistleblowing for the Whistleblower in
Misconduct of Science
The Office of Research Integrity (ORI) in the Office of Public
Health and Science conducted a mail questionnaire survey of
individuals who had made allegations of scientific misconduct to
ORI to better understand the personal consequences of such
actions.
Chapter III
Chapter III provides an overview of the evaluation activities of
the HHS agencies, including information on the evaluation
program, a summary of evaluations completed during fiscal 1995
and evaluations in progress, and a discussion of future
directions for agency evaluations.
Administration for Children and Families (ACF)
ACF administers a broad range of entitlement and discretionary
programs. The objectives of its evaluations are to provide
information on the program design and operations; to test service
delivery approaches; to conduct policy analyses; and to
disseminate and apply evaluation results. In fiscal 1995,
evaluations were completed on welfare reform demonstrations;
welfare dependency reduction; foster care placement programs;
runaway/homeless youth; family strengthening; child maltreatment;
and low-income energy assistance. Major evaluations in progress
include welfare-to-work programs; responsible fatherhood
initiatives; teen pregnancy; intergenerational dependency; the
Head Start Program; foster care placement; and women and infant
nurturing services.
Administration on Aging (AoA)
AoA supports studies that provide information on the
implementation of the Older Americans Act of 1992 (OAA). The
major evaluation completed in fiscal 1995 examined the
effectiveness of State long-term care ombudsman programs. Results
documented the public purpose served by these programs and
endorsed the continuation of their mandate. A second AoA study
developed the framework and design for two studies authorized by
OAA to examine the quality of care in board and care facilities
and the quality of home care services for older and disabled
individuals. AoA is currently evaluating the Elderly Nutrition
Program.
Agency for Health Care Policy and Research (AHCPR)
The goals of the AHCPR evaluation program are to assess agency
effectiveness in meeting major and long-term priorities; to
obtain information to respond to critical agency and departmental
concerns; and to conduct internal evaluations to improve the
efficacy of key program areas. Evaluation projects completed in
fiscal 1995 focus on clinical practice guidelines; clinical
performance measures; and consumer assessments of health care
services. In progress are evaluation activities to generate
information to improve the health care system, such as surveys to
collect data on consumers' satisfaction with access to and use of
health services. AHCPR also is developing methods to disseminate
this information for improved consumer choice.
Centers for Disease Control and Prevention (CDC)
CDC places high priority on evaluations seeking to answer policy,
program, and strategic planning questions related to its mission.
The major evaluations in fiscal 1995 include data
policy/surveillance studies such as the injury risk factor
surveillance system and the influenza vaccination coverage.
Program evaluations include the Medical Examiner/Coroner
Information Sharing Program, the Fatality and Assessment and
Control Program, and the Lead Poisoning Prevention Program.
Evaluations of educational interventions were completed in the
areas of diabetes and adolescent school health. Projects in
progress are evaluating data management and reporting systems;
surveillance systems for drug-resistant Streptococcus pneumoniae;
occupational health hazards; epidemiology training; and dengue
hemorrhagic fever prevention and control in San Juan, Puerto
Rico.
Food and Drug Administration (FDA)
Evaluation at FDA is largely a line management responsibility.
Projects focus on performance management, customer participation,
and rigorous rulemaking. In fiscal 1995, FDA identified
performance measures jointly with industry customers as part of
the drug and biologic review processes necessary for
implementation of the Prescription Drug User Fee Act. In
addition, FDA completed an assessment of the Mammography Quality
Standards Act of 1992. This program is FDA's second major user
fee, performance-oriented, Government Performance and Results
Act-style initiative. During fiscal 1995, FDA program managers
were evaluating their performance measures in light of GPRA
standards.
Health Care Financing Administration (HCFA)
The Office of Research and Demonstrations performs and supports
HCFA evaluations to develop new health care financing policies
and provide information on the impact of HCFA's programs. In
fiscal 1995, HCFA completed evaluations in four areas: monitoring
the impact of Medicare physician payment reform on utilization
and access; the Medicare participating Heart Bypass Center
demonstration evaluation; the Essential Access Community
Hospital/Rural Primary Care Hospital Program; and the Rural
Health Care Transition Grant Program. The major evaluations in
progress include examinations of Medicaid 1915(b) waivers;
Community Support Living Arrangements; and evaluations of four
demonstration projects: Medicare Cataract Surgery Alternate
Payment, Medicare Case Management, Medicaid Uninsured, and
Medicare Alzheimer's Disease.
Health Resources and Services Administration (HRSA)
The objectives of the HRSA evaluation program are to enhance
strategic planning, budget decisions, and legislative planning,
and to improve program management. In fiscal 1995, HRSA reviewed
its performance measurement capacity for all line programs and
bureaus. Completed evaluations included studies in the following
areas: services for prevention and treatment of HIV/AIDS; the
cost-effectiveness of the Community Health Centers Program; and
grants to foster a permanent infrastructure for child and
adolescent injury prevention. Ongoing evaluations include
projects to enhance performance measurement capacity; the J-1
visa program; the National Health Service Corps; the future of
primary care; the Healthy Start Program; and the role of
telemedicine in improving the availability and quality of care to
underserved populations.
Indian Health Service (IHS)
The IHS evaluation program serves the Agency's program and policy
objectives, developed in consultation with the tribal
communities. IHS's major evaluations for fiscal 1995 concentrated
on family violence in Native American communities; the IHS Model
Diabetes Program; the IHS Mid-Level Health Providers; and child
abuse and neglect in American Indian and Alaska Native
communities. Projects in progress include the development and
implementation of the IHS Strategic Planning Model; an analysis
of underreporting of American Indian and Alaska Native deceased
persons on State death certificates; mental health service
delivery models for urban Native Americans; the IHS Adolescent
Regional Treatment Centers; and the IHS substance abuse treatment
programs for American Indian/Alaska Native women.
National Institutes of Health (NIH)
Evaluation is an integral part of the role of NIH in supporting
biomedical research, training, and public education. In fiscal
1995, NIH's evaluations focused on fetal alcohol syndrome;
nursing research training; an assessment of the National Cancer
Program; measuring social inequalities in health; evaluating
outreach activities of the National Library of Medicine; and
public and private funding for basic bioengineering research. The
major evaluations in progress are the National Research Service
Award Research Training Program; the Minority Access to Research
Careers Research Training Program; an assessment of NIH
implementation of Section 491 of the Public Health Service Act
mandating a program of protection of research subjects; the
utilization and regulation of genetic tests; a methodology study
to assess the impact of National Heart, Lung, and Blood Institute
research; the National Cancer Institute's Physician Data Query
Program; an assessment of the facilities and resources for
laboratory animals; and a survey of academic research equipment.
Office of the Assistant Secretary for Planning and Evaluation
(ASPE)
ASPE functions as principal advisor to the Secretary on policy
development, conducting evaluations and policy research studies.
It is also responsible for HHS-wide coordination of legislative,
planning, and evaluation activities. During fiscal 1995, ASPE's
evaluations looked at adolescent sexuality and parenthood;
privacy of private-sector health records; substance abuse
problems of individuals served by HHS programs; and transition to
work for welfare recipients. The major evaluations in progress
include a national study of assisted living; an analysis of the
effect of regulations on the quality of care provided in board
and care homes; the costs of domestic violence to the health care
system; family preservation programs; health care technology
assessment; physician information use; future long-term care
needs; managed care and people with disabilities; the JOBS
Program; performance measurement for the State service delivery
systems; managed care and the pharmaceutical marketplace;
programs to promote responsible fathering; and the cost and
quality of subacute care.
Office of Public Health and Science (OPHS)
OPHS provides advice, policy and program coordination, and
leadership in the implementation, management, and development of
HHS public health and science activities. Several major
OPHS-supported evaluations were completed in fiscal 1995 on the
following: blood supply safety; HHS standards for methadone
programs; clinical trials for the experimental drug fialuridine
and its parent drug fiacitabine; Healthy People 2000 baseline
data for disease prevention activities; measuring State and local
public health expenditures; HIV/AIDS prevention within minority
communities; breast and reproductive cancer prevention efforts at
community and migrant health centers; hepatitis B virus education
and outreach efforts; competitive health plans and academic
medical centers; and the supply, training, and distribution of
primary care providers. Evaluations and policy studies in
progress include dietary supplement labels; framework for Healthy
People 2010; school health programs; public health
infrastructure; cost-effectiveness methodology; and public health
performance measures.
Substance Abuse and Mental Health Services Administration
(SAMHSA)
SAMHSA is committed to evaluating its programs and grant projects
to assess the effectiveness of prevention, treatment, and
rehabilitation approaches and systems of care. In fiscal 1995,
SAMHSA completed evaluations on mental health services for
children and adolescents with serious emotional disabilities,
including the effectiveness of case management services and the
impact of multisystemic family preservation therapy on family
functioning and delinquent adolescent behavior. Ongoing
evaluations include evaluations of SAMHSA's demonstration
programs: the Job Corps program of enriched substance abuse
treatment for adolescents; the services integration experiment
for chronically mentally ill homeless persons; and the National
Treatment Improvement Evaluation Study. Evaluations are also
under way to examine the impact of managed care on access, cost,
and quality of substance abuse treatment.
Appendixes
A complete inventory of the 148 HHS evaluations completed in
fiscal 1995 is provided in Appendix A, and HHS agency projects
currently in progress are listed in Appendix B.
Return to Table of Contents.