| |
Office of the Assistant Secretary for Planning and Evaluation |
Policy Information Center |
The mission of the U.S. Public Health Service (PHS) is to protect and improve the mental and physical health of the American people and to close the gaps in the health status of disadvantaged populations through a variety of activities aimed at health policy development, research, public health practice, personal health services, regulation, infrastructure development, information dissemination, education, and training. Accomplishing this mission is the task of the eight Agencies that make up PHS: the Agency for Health Care Policy and Research (AHCPR), the Agency for Toxic Substances and Disease Registry (ATSDR), the Centers for Disease Control and Prevention (CDC), the Food and Drug Administration (FDA), the Health Resources and Services Administration (HRSA), the Indian Health Service (IHS), the National Institutes of Health (NIH), and the Substance Abuse and Mental Health Services Administration (SAMHSA). The Office of the Assistant Secretary for Health (OASH) is charged with program, policy, budget, and legislative coordination for the activities of these Agencies.
The PHS Evaluation Program plays an integral role in carrying out the PHS mission by reviewing various aspects of program performance of the PHS Agencies and OASH and by identifying means of improving that performance. The three major goals of the Evaluation Program are--
These goals parallel the major audiences for PHS evaluations: (1) government decisionmakers, (2) PHS program managers, and (3) the larger public health community. For the first two groups, PHS uses ongoing program evaluation as both a philosophy and a tool to support informed decisionmaking that affects the health of the Nation. The PHS Agencies and OASH assess and provide technically reliable information on the extent to which these efforts achieve their desired results. This information is the primary tool that PHS uses to improve program performance on an ongoing basis.
For nongovernmental stakeholders, PHS has an important obligation to foster the development of (1) new knowledge about the effectiveness of public health programs and interventions and (2) evaluation tools for use by the larger public health community. The 1988 Institute of Medicine (IOM) report, "The Future of Public Health," stressed the Federal Government's important responsibility to support knowledge development and dissemination through data gathering, research, and information exchange. Evaluations conducted by PHS play a significant role in fulfilling this responsibility.
This chapter describes the PHS Evaluation Program. It provides an overview of the kinds of evaluation activities supported by the program and then describes one of the funding sources that PHS Agencies and OASH use to support these evaluations. It details PHS evaluation policies and management, including planning procedures, quality assurance, dissemination mechanisms, and use of results. The chapter concludes with a discussion of future directions for the program.
PHS Evaluation Activities The evaluation activities sponsored by PHS and described in this report assess program performance (efficiency, effectiveness, responsiveness); analyze results based on those assessments; and use the resulting information in policymaking and program management. These activities are diverse and include the full spectrum of evaluation methodologies developed over the last quarter century. The classification of PHS evaluation activities presented in figure 1 summarizes that diversity.
PHS evaluation projects typically fall into a combination of these categories. For example, comprehensive PHS evaluations generally examine both process and outcome or impact. Knowing only whether goals and objectives are achieved is insufficient without also knowing how well the program was implemented and whether its goals and objectives were appropriate in the first place. Similarly, evaluation feasibility and design activities generally represent the crucial first phase of major PHS process and outcome/impact evaluations.
Evaluation Funding Funding for the PHS Evaluation Program is largely provided by a 1 percent evaluation set-aside legislative authority in Section 241 of the Public Health Service Act (42 U.S.C. \xa4 238j). This authority was established in 1970 when Congress amended the Act to permit the Secretary of Health and Human Services to use up to 1 percent of appropriated funds to evaluate authorized programs.[1] Amounts available from the 1 percent set-aside authority are determined annually and are subject to certain exclusions, such as program management and block grant funds, and deductions for health surveys and special evaluation projects. The remaining set-aside funds are distributed among the PHS Agencies and OASH for use under guidelines developed annually by the Assistant Secretary for Health.
Table 1 shows the set-aside evaluation funds used over the past 4 years by the PHS Agencies and OASH.[2] In addition to the set-aside, PHS Agencies support evaluation activities with appropriated program funds. Since other funds are used for demonstrations, health services research, or program developments that meet the general description of program evaluation, the total PHS funds used to evaluate its programs exceed the amount indicated in table 1.
Evaluation Policy and Management The policies and management of the PHS Evaluation Program, carried out on a regular basis by the PHS Agencies and coordinated by OASH, involve five basic functions:
These functions are described below in general terms. Additional information on individual Agency and OASH evaluation programs is included in chapter III.
1. Evaluation Planning Evaluation planning is conducted annually in concert with program planning, legislative development, and budgeting cycles. Before the start of each fiscal year, evaluation guidance is issued by the Assistant Secretary for Health (ASH) to signal the priority PHS program and policy areas for evaluation. Typically, they include new programs, those undergoing major change, those that are candidates for reauthorization, and those for which key budget decisions are anticipated. In addition, emphasis has been given to evaluations that support strategic planning goals and objectives; evaluation syntheses that address program areas currently under review by both Congress and the Department; and evaluations that cut across PHS Agencies and have broad implications for program or policy change. In FY 1995, a new element was introduced into the PHS evaluation planning process: a panel of experts from universities and private research centers offered consultation before the development of the PHS guidance for FY 1996.
In response to the ASH guidance, each PHS Agency submits a plan of its evaluation strategies and proposed projects for the immediate and subsequent fiscal years. Sharing information allows the Agencies to learn about evaluation proposals being considered throughout PHS, promotes PHS-wide coordination, and avoids duplicative efforts. The evaluation planning process also facilitates coordination with Agencies outside PHS, such as the Departments of Agriculture, Education, and Justice.
2. Quality Assurance Most evaluation projects are developed at the program level, and the initial review is conducted by a committee of Agency-level policy and planning staff members. Before a project is approved, it is reviewed for technical quality, generally by a second committee of staff who are skilled in evaluation methodology. Technical review committees follow a set of criteria for quality evaluation practice established by each Agency. Some PHS Agencies also have external evaluation review committees composed of evaluation researchers and policy experts from universities and research centers. More details on the quality assurance procedures for the various Agencies and OASH are presented in chapter III.
3. Project Management and Tracking A computer database, the PHS Evaluation Management System (PHS-EMS), has been operating since 1992 for managing information on evaluations and policy research studies. Information is continuously entered into the database, based on reports from PHS Agencies, and it is used to produce reports that track the progress of evaluations under way or to locate information about recently completed evaluations. In FY 1996, a component will be added to allow Agency evaluation staff remote computer access to the database. This database system is also coordinated with the HHS Policy Information Center (PIC)--the departmental evaluation database and library maintained by the Office of the Assistant Secretary for Planning and Evaluation. Updated records on PHS evaluations and the final reports on completed evaluations are regularly transferred to the PIC. As an information database and library resource, the PIC contains nearly 6,000 completed, ongoing, and planned evaluation and policy research studies conducted by the Department, including key studies completed outside the Department by the General Accounting Office (GAO) and private foundations.
4. Dissemination Typically, the results of PHS evaluations are disseminated through targeted distribution of final reports, articles in refereed journals, and presentations at professional meetings and conferences. Although the major responsibility for disseminating results lies with the PHS Agencies, departmentwide efforts are under way to expand dissemination to the public health community. Abstracts of all studies maintained in the PIC database are now accessible to the public health community through the Department's World Wide Web server (http://www.hhs.gov) on the Internet. The preparation of this report is another effort to expand dissemination of results; the report will be distributed widely to State and local public health agencies, schools of public health and other university health programs, and private foundations. Finally, a number of evaluation syntheses are being developed to enhance the accessibility of results from multiple evaluations that address similar issues.
5. Application PHS evaluations are generally used directly by program managers to improve program operations and efficiency. In addition, in accordance with a 1993 GAO recommendation (see Publication No. GAO/PEMD-93-13), more evaluation focus will be directed to outcome/impact evaluations that can be applied by Congress and others for program planning, budgeting, and legislative action. This shift is also consistent with implementation of the Government Performance and Results Act (GPRA) and the National Performance Review (NPR), which seek to improve the efficiency and effectiveness of Federal Government programs and the services provided to customers of those programs and directly to citizens.
Future Directions in Evaluation In addition to NPR and GPRA, PHS must be responsive to public health issues of the future. Public health has an opportunity to change the American health care system, shifting its focus toward protecting and improving health. This opportunity to reinvent public health is the theme of the PHS Strategic Plan for FY 1995. The plan centers on the following three goals, which PHS, working with the public health community, will pursue:
1. Shift from personal care to population-based services The PHS FY 1995 Strategic Plan focuses on strengthening the public health infrastructure of the Federal Government, tribal governments, States, and communities to identify and address high-priority health problems for their populations and to do so in a way that focuses on the outcomes of their activities. PHS intends to provide the States with more flexible funding by consolidating several categorical programs into Performance Partnership Grants (PPGs). Partnerships are proposed in six program areas--mental health, substance abuse, immunizations, infectious disease, chronic disease, and prevention. These grants would be based on the premise that giving the States greater flexibility to develop and manage programs in exchange for increased accountability for results will encourage more effective use of Federal funds to address national public health objectives, such as those developed for Healthy People 2000. The success of PPGs with the States depends on knowing what the grant funds can achieve (goals), whether the results over time can be known (measures), and how performance can be rewarded (incentives). Program goals, to be specified in legislation, would lay out national objectives for each grant. Performance measures would quantify national and State progress toward the goals. The measures must be results oriented, measuring success based on program outputs and health outcomes developed mutually by PHS Agencies and the States. Performance incentives would be an integral part of the individual contracts between the States and the relevant PHS Agency, including specific performance targets and timeframes for achieving them. The incentives might, for example, be to extend greater flexibility to States that meet their performance targets.
2. Public health partnership with managed care The managed care revolution has the potential to establish the foundation for a full partnership between public health and the personal care system on behalf of the health of defined populations. With payments based on annual premiums, managed care creates financial incentives that reward health plans for having healthy clients. Potentially, this incentive could encourage health plans to make sure that their enrollees take advantage of cost-effective preventive services. It also encourages plans to work with and support the efforts of public health agencies and community organizations, which can prevent unnecessary disease and health care costs through communitywide interventions. Realizing the benefits of organized systems of care, however, requires a strong system of performance monitoring to ensure that health plans are living up to their commitments and taking responsibility for improving the health of enrolled populations.
3. Leadership in the National Information Infrastructure The National Information Infrastructure (NII) refers to the web of electronically accessible information resources available to Americans. The NII holds potential for rapid and vastly enhanced interactive information flow. Multi-user list server technologies will allow infor-mation that is sent to a single location to go to all entities on the list. Electronic bulletin boards and multiple-recipient electronic mail (e-mail) are revolutionizing the way information is dis-seminated and the ability to have followup communication and dialogue. More and more communities or their critical components are becoming part of local and wide-area health information networks, often called community health information networks. These networks can ease transmission of and access to consumer health information.