Performance Improvement 1997

Chapter I

Evaluation in the Department of Health and Human Services

The mission of the U.S. Department of Health and Human Services (HHS) is to enhance the well-being and health of Americans by providing for effective health and human services and by fostering strong, sustained advances in the sciences underlying medicine, public health, and social services. The following HHS agencies and offices accomplish this mission through their program activities:

The Assistant Secretary for Planning and Evaluation (ASPE), located in the Office of the Secretary, coordinates evaluation activities throughout HHS.

Evaluation plays an integral role in carrying out the HHS mission: assessing various aspects of program performance of the HHS agencies enables staff to identify ways to improve that performance. The HHS evaluation function has three goals:

  1. To provide information on HHS programs that helps government officials and members of Congress make decisions related to programs, policies, budgets, and strategic planning;
  2. To help HHS managers improve program operations and performance; and
  3. To disseminate evaluation results and methodological tools useful to the larger health and human services community of State and local health and human service officials, researchers, advocates, and practitioners for improving the performance of their programs.
This last goal is very important to HHS. Its various agencies have an important obligation to foster the development of new knowledge about the effectiveness of health and human services programs, interventions, and evaluation tools for use by the larger health and human services community. Although the findings and recommendations of HHS evaluations are usually first used by the Administration and the Congress, they can also be applied by others in the research and practice communities to improve the performance of programs at the State and community levels. The purpose of this report is to disseminate information about recent HHS evaluations and to make sure that the potential for wider application is realized.

This chapter describes the organization and operation of evaluation at HHS. It first provides an overview of the kind of evaluation activities supported by HHS agencies and then describes the resources and funding mechanisms used to support them. It details HHS evaluation management, including planning procedures, project management, quality assurance, dissemination of results, and effective uses of evaluation. The chapter concludes with a discussion of future directions for evaluation at HHS.

HHS Evaluation Activities

HHS defines evaluation as the assessment of program performance (efficiency, effectiveness, and responsiveness) through the analysis of data or information collected systematically and ethically, and the effective use of resulting information in program or policy decision making and program management. This definition encompasses a range of evaluation activities that include the full spectrum of evaluation methodologies developed over the last quarter century. The classification of HHS evaluation activities presented in figure I-1 summarizes that diversity.

Figure I-1. Range of HHS Evaluation Activities

Evaluation projects

  1. Outcome evaluations: assessing the immediate or intermediate effects of a program with respect to the stated goals or objectives
  2. Impact evaluations: assessing the broader results, intended or unintended, of a program on populations or institutions involved
  3. Implementation or process evaluations: assessing the nature of program inputs and outputs and their relationship to stated goals and obectives
  4. Policy assessments: examining health policies with respect to their development, implementation, or impact on public health or program activities
  5. Cost-benefit or cost-effectiveness analyses: developing methodology and its application to assess the relationship of program results to program costs (direct and indirect), often in comparison with alternative programs
  6. Survey data analyses: evaluating the results of HHS programs or policies by analyzing data obtained from surveys
  7. Performance measurement and data systems: identifying and testing the validity and reliability of process, output, and outcome indicators to measure the performance of programs and develop data systems supporting implementation of the Government Performance and Results Act
  8. Simulations and models: using computer simulations and modeling techniques to analyze the impact of policy changes on service delivery systems and beneficiaries
  9. Management studies: examining the effectiveness or efficiency of the administration or operation of HHS programs and offices
  10. Evaluation syntheses: integrating the results from multiple independent evaluation studies within a defined program or policy area in a fashion that improves the accessibility and application of those results

Methodology projects

  1. Evaluation feasibility studies: assessing the clarity and importance of program goals and objectives, the consensus of program stakeholders on the potential utility of evaluation information, and the availability of relevant performance data before committing to a full-scale program evaluation
  2. Evaluation design projects: procuring assistance in the development of an evaluation design, measurement tool, or analytic model in preparation for fully implementing an evaluation
  3. Instrument development projects: developing evaluation instruments (design, measurement, or analytic) for a specific HHS program or for general use by the health and human services community

Evaluation support activities

  1. Evaluation technical assistance: providing assistance to HHS program managers or office directors on any aspect of evaluation planning, project design-implementation-analysis, or use of results
  2. Evaluation dissemination: identifying target audiences and mechanisms to inform program constituencies and evaluation stakeholders about evaluation results
  3. Evaluation training/conferences: maintaining the professional skills and expertise of evaluation staff through training opportunities, as well as promoting the dissemination of HHS evaluations through symposia

Evaluation Resources

Evaluation activities of the various HHS agencies are largely supported through two funding mechanisms: direct use of programs funds and use of special legislative set-aside authorities for evaluation. The first is a common mechanism giving program managers discretionary authority to use appropriated program funds to support contracts that will design, implement, and analyze evaluation data. In some cases, a program's legislative authority calls for a special mandated evaluation, and program funds are used directly to support the evaluation.

The second mechanism for evaluation funding is legislative set-aside authorities permitting the Secretary of HHS to use a proportion of overall program funds for evaluation purposes. The largest of such set-aside authorities is one established for evaluations conducted by several agencies of the U.S. Public Health Service (AHCPR, CDC, HRSA, NIH, and SAMHSA), ASPE, and the Office of Public Health and Science (OPHS) in the Office of the Secretary. The mechanism, called the 1-percent evaluation set-aside legislative authority, is provided for in Section 241 of the Public Health Service (PHS) Act. This authority was established in 1970 when Congress amended the Act to permit the HHS Secretary to use up to 1 percent of appropriated funds to evaluate authorized programs. Section 241 limits the base from which 1percent of appropriated funds can be reserved for evaluations of programs authorized by the PHS Act. Excluded are funds appropriated for FDA, IHS,1 and certain other programs that are managed by PHS agencies but not authorized by the Act (e.g., HRSA's Maternal and Child Health Block Grant and CDC's National Institute for Occupational Safety and Health).

In FY 1996, HHS invested more than $33 million in set-aside evaluation funds to carry out evaluation activities related to relevant public health programs. In FY 1997, HHS estimates that it will use approximately $35 million in PHS evaluation set-aside funds to continue current evaluation activities and to initiate new evaluation projects. Table I-1 provides a breakdown of the usage for FY 1996 by PHS agencies, ASPE, and OPHS, as well as the estimates for FY 1997.

Table I-1. Agency Use of Evaluation Set-Aside Funds,
in Thousands of Dollars
FY 1996FY 1997
Agency evaluation use:
    AHCPR$0$250
    CDC2,0002,000
    HRSA6,6806,868
    NIH4,3004,500
    SAMHSA9951,423
    ASPE15,50015,500
    OPHS3,8524,552
Total use$33,327$35,093

Evaluation Management

The management of HHS evaluations, carried out on a regular basis by the HHS agencies and offices and coordinated by ASPE, involves five basic functions:
  1. Evaluation planning and coordination
  2. Project management
  3. Quality assurance
  4. Dissemination of evaluation reports
  5. Ensuring effective use of evaluation results

A description of each function in general terms follows. Additional information on the individual HHS agencies, ASPE, and OPHS evaluation functions can be found in chapter III.

Evaluation Planning and Coordination

The HHS agencies, ASPE, the Office of the Inspector General (OIG), and OPHS develop evaluation plans annually, in concert with HHS's program planning, legislative development, and budgeting cycles. Plan development is coordinated by ASPE. Each agency or office plan generally states the evaluation priorities or projects under consideration for implementation. Typically, HHS evaluation priorities include congressionally mandated program evaluations, evaluations of Secretarial program or policy initiatives, and assessments of new programs, programs that are candidates for reauthorization, or programs for which key budget decisions are anticipated.

HHS evaluation planning activities are required more specifically in two ways. First, those agencies and offices that use the PHS 1-percent evaluation set-aside authority--AHCPR, CDC, HRSA, NIH, ASPE, OPHS, SAMHSA--submit a formal plan to ASPE, which coordinates and develops the individual plans into the HHS report to Congress on the use of the PHS 1-percent authority. This report must be submitted to Congress before HHS can implement the plan.

Second, Congress has asked HHS to coordinate all of its research, demonstration, and evaluation programs to ensure that the results of these projects address HHS's program goals and objectives. ASPE and the Assistant Secretary for Management and Budget now work with HHS agencies to provide Congress with a special annual research, demonstration, and evaluation budget plan that coincides with the preparation of the President's fiscal year budget. The plan outlines each HHS agency's research, demonstration, and evaluation priorities as related to overall HHS program goals and objectives.

Project Management

The design and management of evaluation projects at HHS is primarily decentralized--the various HHS agencies, OIG, and ASPE are all responsible for executing annual evaluation plans, developing evaluation contracts, and disseminating and applying evaluation results. Even within agencies, while there is some oversight responsibility and execution capability in the Office of the Director or Administrator, the various subunits (centers, institutes, bureaus) conduct much of the day-to-day evaluation activity.

OIG performs independent evaluations through its Office of Evaluations and Inspections (OEI). OEI's mission is to improve HHS programs by conducting inspections that provide timely, useful, and reliable information and advice to decisionmakers. This information (findings of deficiencies or vulnerabilities and recommendations for corrective action) is usually disseminated through inspection reports issued by the Inspector General. A summary of individual inspection reports and other OIG reports can be viewed on the Internet at http://www.sbaonline.sba.gov/ignet. OEI also provides technical assistance to HHS agencies in conducting their evaluations.

Quality Assurance

Most evaluation projects are developed at the program level. The initial quality review is generally conducted by a committee of agency- or office-level policy and planning staff members. Before a project is approved, however, it is also reviewed for technical quality by a second committee of staff who are skilled in evaluation methodology. Technical review committees follow a set of criteria for quality evaluation practice established by each agency. Some HHS agencies also have external evaluation review committees composed of evaluation researchers and policy experts from universities and research centers. More details on the quality assurance procedures for the various HHS agencies, ASPE, and OPHS are presented in chapter III.

Dissemination of Evaluation Reports

Maintaining report libraries and distributing information on evaluation results is an important component of HHS evaluation management. Project information and reports are continuously submitted to the HHS Policy Information Center (PIC)--the departmental evaluation data base and library maintained by ASPE. As an information data base and library resource, the PIC contains nearly 7,000 completed and in-progress evaluation and policy research studies conducted by the Department, as well as key studies completed outside HHS by the U.S. General Accounting Office and private foundations.

Typically, the results of HHS evaluations are disseminated through targeted distribution of final reports, articles in refereed journals, and presentations at professional meetings and conferences. Although the individual HHS agencies have primary responsibility for disseminating results, there is a departmentwide effort under way to expand dissemination to the larger research and practice communities through centralized computer communications and publications. Abstracts of all studies maintained in the PIC evaluation data base are on the Internet through the HHS World Wide Web server, at the following URL: http://aspe.hhs.gov/PIC/gate2pic.htm. Information is available on completed and in-process projects; it includes the name and telephone number of the HHS official responsible for the project.

In addition, HHS is widely distributing copies of its first departmentwide annual report on evaluation (Performance Improvement 1996: Evaluation Activities of the U.S. Department of Health and Human Services). The report's theme of performance improvement represents the numerous changes and initiatives throughout HHS to increase the effectiveness and efficiency of health and human services programs. This report to Congress summarizes the findings of HHS evaluations completed during FY 1995. In that year, HHS agencies produced 148 evaluation reports and supported more than 200 evaluation projects in progress. The report provides summaries or abstracts of the completed project reports and contacts for further information.

In addition to providing the report to members of Congress, copies were sent to State and local public health and social services officials, various associations for schools of public health, social welfare, and social work, and other national health and social policy research and practice associations. A similar dissemination strategy is being used for this report, the second annual HHS report on evaluations. Both reports, Performance Improvement 1996 and Performance Improvement 1997, are also available on the previously described HHS home page in three computer formats for downloading information.

Ensuring Effective Use of Evaluation Results

HHS is committed to ensuring that evaluations yield a high return on the investment of available program funds. In the last decade, HHS evaluations were used primarily by program managers for internal purposes of improving program operations and efficiency. The major outcome or impact evaluations conducted by HHS were generally congressionally mandated and involved a substantial commitment of evaluation resources. In the 1990's, however, the demand for more program outcome and impact evaluations has increased, largely in response to fulfilling HHS's responsibility for implementing reform legislation. To understand the effects of these transformations on HHS programs at the system and individual levels will require a substantial investment of evaluation resources in outcome or impact studies. In addition, HHS is giving priority to evaluation efforts related to implementation of the Government Performance and Results Act (GPRA), which concerns all HHS programs. The expectations for GPRA are that by FY 1999 HHS will have in place the capacity to measure the performance of its programs with data that are complete, accurate, and consistent enough to support decision making at various management levels. The measures should be tied to program performance goals and should have the capability of measuring program results in achieving those goals.

Future Directions in HHS Evaluation

Last year's annual report, Performance Improvement 1996, stated that HHS agencies, ASPE, and OPHS would focus future evaluation efforts in three directions: (1) the impact of transformations in health and human services; (2) the development of performance measures; and (3) overall program performance improvement. In FY 1997, those directions continue to be the principal evaluation priorities for HHS, and its evaluation resources will be used to generate new knowledge to improve the performance of its programs.

Impact of Transformations

The Secretary remains committed to an evaluation strategy to examine the transformations now taking place in health and human services and the impact of those changes on the well-being of Americans--especially the vulnerable populations that are a high priority for HHS programs, such as disadvantaged or low-income children and families, the elderly, racial and ethnic minorities, and individuals with disabilities.

The financing and delivery of health care in the United States is being transformed by the growth and evolution of managed care and by other innovations in the financing, management, and delivery of health services. HHS agencies need to better understand how these transformations directly affect their programs. For example, HCFA is now developing a comprehensive monitoring and evaluation plan to examine the Medicare and Medicaid programs on such dimensions as access to care, quality, efficiency, cost, and beneficiary satisfaction. They will implement new demonstrations that test alternative payment and health care delivery models and evaluate the results.

Public health programs supported by HHS are an essential part of the health care safety net for the uninsured and a vital element in protecting the health of all Americans. For HHS, an increasing challenge is to understand how managed care and other financing and service delivery innovations affect public health systems, public health programs, and the clients those programs serve. For example, HRSA and SAMHSA have already made significant investments to study the effects of new financing and service delivery mechanisms, especially managed care, on their programs, services, and clients. In upcoming years, ASPE will complement their efforts by undertaking additional evaluations focused on the public health system and programs. Particular emphasis will be given to evaluation issues that address the public health infrastructure; how changes affect vulnerable populations served by the public health system; and how program transformations taking place outside the health arena, such as welfare reform, interact with and affect the public health care system. In addition, HHS will continue to support evaluations to determine the effect of managed care on individuals with disabilities who have both health and long-term care needs.

In parallel, human services programs are undergoing transformations in their purpose, organization, and financing. Enactment of the Personal Responsibility and Work Opportunity Reconciliation Act of 1996 eliminated individual entitlement to cash assistance and replaced it with a fixed block grant to States. The law also placed a 5-year time limit on benefits, imposed strict work requirements on recipients, reduced benefits and services available to legal immigrants, and greatly expanded States' authority over welfare programs. The Nation has a great deal at stake in the success of this program, and timely and careful evaluation will be important to help ensure that success.

Studies are needed to evaluate the combined effects of changes taking place in health care delivery systems and in income supports for low-income children and families who often have special needs. For example, one issue is how to help substance abusers or victims of domestic violence whose entitlements and access to services will change under the terms of new welfare reform legislation. Under the new law, welfare recipients generally will be required to work to receive benefits and will face overall time limits on welfare benefits. In the past, many clients identified as substance abusers or victims of domestic violence were not required to participate in work activities. While some of these clients will be exempt under the new law, others must enter the workforce. Case managers need better tools to assess clients' barriers to sustaining employment and to assist these clients in addressing those barriers.

Children's issues will also be a priority. HHS will identify and evaluate effective tools--including methods of linking children and families under the new welfare system with health and family support services--and on understanding the extent and manifestations of health problems among the welfare population.

The interaction of health and welfare changes also raises questions about the impact of welfare changes on the health status and well-being of selected populations of disabled children, low-income mothers with disabilities, and elderly people with health and long-term care needs. For many in these groups, access to health care and supportive services was previously dependent on entitlements that may no longer exist. Studies are needed to examine how these populations obtain necessary health and long-term care services once their entitlements are terminated, as well as the extent to which their health status and quality of life changes. For example, the AoA is interested in examining how State aging agencies use funds under the Older Americans Act, along with other sources of funding, to develop comprehensive and coordinated systems of community-based services for the elderly. These issues become particularly important in light of the demographic changes facing the Nation as the population ages.

Development of Performance Measurements

Implementation of GPRA requires HHS to focus a substantial portion of its evaluation activities on the development of performance measures for its programs and on information systems necessary to produce the data expected to support GPRA's performance measurement requirements. By the end of FY 1997, HHS will have in place a departmentwide strategic plan. The plan will contain the HHS mission statement, the long-term strategic goals, and descriptions of how HHS intends to achieve those goals with its program resources. Next, HHS will develop its annual performance plan for FY1999, which will link the HHS strategic goals to specific program activities. Beginning with FY1999, the third major GPRA element, the Program Performance Report, will present performance data related to the strategic goals based on the previously identified performance indicators.

The performance plan for FY 1999 will identify the performance measures that HHS agencies and offices will use to assess program performance related to achieving the goals. GPRA has several expectations for these measures:

  1. They should to be tied to program goals and demonstrate the degree to which the desired results are being achieved.
  2. The number of measures for each goal should be limited to avoid confusion in assessing accomplishments and making adjustments.
  3. Measures should be responsive to multiple priorities, such as quality, cost, customer satisfaction, and stakeholder concerns.
  4. Performance measures should be linked directly to the offices responsible for making programs work in order to reinforce accountability.

Since GPRA was passed in 1993, HHS agencies have utilized evaluation projects to develop performance measures related to its health and human services programs. For example, AHCPR has initiated projects to develop and test the validity and reliability of agencywide performance measures of its health services research grants process and dissemination mechanisms. CDC now has projects to develop indicators to assess performance of its HIV-prevention community-planning programs and its State-based diabetes control cooperative agreements program. HRSA has invested its evaluation resources in training and technical assistance to program managers to develop measures for GPRA implementation. Featured in chapter II is an HRSA report about the efforts of the Bureau of Health Professions to develop a set of outcome-based performance measures and a plan for a monitoring system to support program management.

In addition to performance measures for HHS programs, the Federal grant relationships to States in public health are evolving into health outcomes-based performance management. For the past two years, ASPE and OPHS have collaborated on several cross-cutting efforts to involve States, communities, and service recipients in identifying such a data system. A major project with the National Research Council is identifying measures that can be used jointly by HHS agencies and States to measure their shared performance in grant management. The project will also recommend specific steps that can be taken to improve State and local community data capabilities. A related project is developing approaches to obtain comprehensive baseline and trend data on public health infrastructure. A third study is building on a Robert Wood Johnson Foundation project to assess the quality of data that States collect and determine whether there is a foundation on which to build a multi-State health data system that can be shared among States to improve health policy decision making.

HHS will continue to invest its evaluation resources in performance indicators to ensure a successful implementation of GPRA. The evaluation strategies of HHS agencies, mentioned in chapter III, include the priorities of projects that examine program performance objectives and develop useful measures of program outputs and outcomes. GPRA offers HHS agencies an opportunity to develop performance measurement systems that will eventually link program evaluation activities to budgeting. HHS's evaluation set-aside authority, such as the 1-percent authority for some PHS agencies, is an important resource to help program managers identify performance objectives and test the validity and reliability of progress indicators.

Performance Improvement

Program managers must initiate evaluations useful for improving the performance of HHS programs and ensuring that program operations are efficient and effective. Evaluations are an essential resource for HHS's Continuous Improvement Program, aimed at supporting development and operation of information systems and special studies to enable program managers to measure customer satisfaction with HHS services.

Several in-progress or planned projects illustrate HHS's evaluation priority of continuous improvement of services. AHCPR is initiating a study to examine the Medical Expenditure Panel Survey regarding the efficiency of its new design and its effectiveness in answering key health policy questions. They will look at the survey process, including data support contracts and the capacity of the survey data for developing health services analytical models.

FDA is examining its program to assist small businesses in complying with FDA rules, which includes determining FDA's success in coordinating different sources of agency advice and the appropriateness and timeliness of the advice. The evaluation will consider the training and materials available to FDA staff to help them answer questions from small businesses, identify barriers to timely and useful responses, and develop procedures for tracking inquiries.

One of HRSA's priorities is to improve access to the continuum of care for persons with HIV/AIDS through the Ryan White Comprehensive AIDS Recovery Emergency (CARE) Act programs. One project concerns the management of persons infected with both HIV and tuberculosis (TB) in metropolitan areas to identify models of successful collaboration between HIV/AIDS and TB control programs. The purpose is to determine the extent to which Ryan White CARE Act grantees are offering guidance to providers on TB screening and prophylactic and therapeutic regimes.

NIH is currently conducting a survey of individuals who applied to NIH for a research grant in FY 1992. The study will obtain information on the career progress and research productivity of a sample of grant applicants. The purpose is to assess the respondent's satisfaction with the grants application and award process and to get their opinions on specific modifications NIH is considering for improving policies and procedures.

SAMHSA's priority of evaluating its Knowledge, Development, and Application (KDA) grants is also a good example of HHS's commitment to continuous improvement of programs. Evaluating the current KDA grants will generate new data on the effectiveness of alternative models of managed care for treatment and prevention of substance abuse and mental disorders, including specific programs for homelessness among adults with serious mental illnesses and brief interventions for marijuana dependence.

Notes

1. FDA programs are principally authorized by legislation other than the PHS Act, specifically the authority of the Agriculture, Rural Development, Food and Drug Administration and Related Agencies Appropriations Act. IHS programs are authorized under the Indian Health Care Improvement Act and the Indian Self-Determination Act; they are appropriated under the Department of the Interior and Related Agencies Appropriations.

Return to Table of Contents.