| |
Office of the Assistant Secretary |
Policy Information Center |
The 10 agencies of the U.S. Department of Health and Human Services (HHS) and the Office of the Assistant Secretary for Planning and Evaluation (ASPE) and Office of Public Health and Science (OPHS) in the Office of the Secretary each maintain their own evaluation program, including the functions of evaluation planning and policy review, quality assurance through technical review, project coordination and management, dissemination of reports, and utilization of results. This chapter provides an agency-by-agency summary of the evaluation activities during fiscal 1995. Each summary begins with a description of the relevant evaluation program--its philosophy, policies, and procedures. Next is an overview of the major evaluations completed in fiscal 1995 and the major evaluations in progress during that time. Finally, each agency offers a discussion of new directions for its evaluations, including priorities for future program and policy evaluation projects. As stated in the foreword, abstracts and contact persons for all evaluation reports completed in fiscal 1995, organized by agency, are presented in appendix A. Listings, by agency, of all evaluation projects in progress are located in appendix B.
MISSION: To promote the economic and social well-being of families, children, individuals, and communities.
ACF administers a broad range of entitlement and discretionary programs including the welfare programs (Aid for Dependent Children [AFDC], Job Opportunities and Basic Skills Training [JOBS], Child Support); children and family services (Head Start, Child Welfare, Family Preservation and Support, youth programs, child care); four block grants; and special programs for targeted populations, such as the developmentally disabled and Native Americans.
The objectives of ACF's evaluations are to provide information on the design and operation of effective programs; test new service delivery approaches that build on the success of completed demonstrations; apply evaluation data to policy development, legislative planning, budget decisions, program management, and strategic planning and performance measures development; and disseminate findings of completed studies and promote application of results by State and local governments.
ACF actively engages with other Federal agencies, State and local policy and program officials, national organizations, foundations, professional groups and practitioners, and consumers to stay current on the emerging issues affecting its programs and to identify questions for evaluation studies. Study designs are carefully negotiated with the States and other interest groups. Studies are frequency funded as joint ventures with ASPE and other Federal agencies and foundations. Such collaborations enable efforts that are better informed, more representative of varying perspectives, and larger in scale. Proposals are reviewed by multidisciplinary experts. Work groups of various types are used throughout the projects to monitor progress and to advise on refinements in design and presentation of the findings.
ACF's evaluations are closely linked to its two strategic goals:
During fiscal 1995, ACF produced 23 evaluation reports on its various programs. Several major evaluations were related to the first goal of "Economic Independence and Productivity of Families." One study focused on increasing participation in work and work-related activities and the lessons learned from welfare reform demonstrations in five States: Colorado, Iowa, Michigan, Utah, and Vermont. The study identified a number of successful approaches that could be used by other States. The second study, "Something Old, Something New: A Case Study of the Post-Employment Services Demonstration in Oregon," evaluated a demonstration intended to help recently employed persons keep their jobs, help those losing their jobs return to work quickly, and reduce the amount of time spent receiving AFDC.
ACF's Office of Refugee Resettlement looked at its Key States Initiative Program, begun in fiscal 1987, to increase employment and reduce welfare dependency in refugee communities in Minnesota, New York, Pennsylvania, Washington, and Wisconsin. The study documented program design features, program participant characteristics, program outcomes, and lessons learned from each State.
Ten evaluations important to the goal of "Healthy Development of Children and Families" were completed in fiscal 1995. Three evaluations focused on children in foster care. The first, "Foster Youth Mentors," examined factors characterizing successful relationships between foster youth and older citizen mentors. The findings have been disseminated to independent living programs throughout the United States to facilitate use of mentors in older youths' transitions from foster care. The second study, "Outcomes of Permanency Planning for 1,165 Foster Children," examined foster care programs in San Diego County, California, and Pierce County, Washington. The study found an overrepresentation of minority children and a sizable number of children with mental health, physical health, and behavioral problems, calling for a more culturally sensitive practice in child welfare systems and appropriate services to this population with special health needs. The third study, "Update From the Multi-State Foster Care Data Archive: Foster Care Dynamics 1983-1993," contains foster care career histories for all children in State-supervised substitute care living arrangements. The study found that infants and young children are the fastest growing age groups in the foster care population.
The other seven studies focused on specific ACF program activities. "Strengthening Families and Neighborhoods: A Community-Centered Approach," highlighted in chapter II, is an innovative method of testing services improvement in a distressed urban neighborhood in Linn County, Iowa, modeled on the British patch system of community-centered service delivery. The 3-year demonstration project aimed to overcome categorical barriers that prevent the pooling and use of informal and formal resources needed for flexible services--services building on the strengths of individuals, families, and neighborhoods.
"Youth With Runaway, Throwaway, and Homeless Experiences: Prevalence, Drug Use, and Other At-Risk Behaviors" is a national study that examines substance use, suicide attempts, and other at-risk behaviors, which is also highlighted in chapter II. The findings were drawn from youth living in shelters, on the street, in family households, and from youth shelter directors. The study recommends that services address a continuum of need from primary prevention to intervention and treatment at the community level.
"Child Maltreatment 1993: Reports From the States to the National Center on Child Abuse" found that almost 2 million reports of child abuse and neglect were received by child protective services agencies and referred for investigation in 1993; neglect is the most common type of maltreatment, followed by physical, sexual, medical, and emotional abuse; and 1993 was the first year since 1976 that the rate of reported child abuse and neglect cases did not increase.
"Children on Hold: Improving the Response to Children Whose Parents Are Arrested and Incarcerated" surveyed patrol officers, narcotics officers, child protective services workers, foster care parents, and corrections staff in 100 counties and conducted site visits to four communities selected for their exemplary responses to children whose parents are arrested. The report assesses existing policies, procedures, and practices of child welfare, law enforcement, and local correctional agencies regarding children whose primary caretaker is arrested or incarcerated. It also assesses how well these agencies coordinate with each other in dealing with these families, identifies promising strategies to improve coordination, and assesses statutory enactments and case law regarding the termination of parental rights for incarcerated parents. The report recommends ways in which relevant agencies can improve services to these children. It highlights areas of concern within law enforcement and child protective services agencies and in the interaction of these agencies.
Next, ACF supported two literature review studies. First, the "Study of the Impact on Service Delivery of Family Substance Abuse" reviews research literature from 1986 through 1994, including intensive case studies of the effects of family alcohol and other drug (AOD) abuse programs funded by the Administration for Children, Youth and Families (ACYF). Child Protective Services, of all ACYF programs, is the most negatively affected by family AOD abuse. AOD abuse affects the mandates of all ACYF programs including accelerated termination of parental rights policies, postadoption and foster placement support programs, and Head Start's services to nonparental caretakers. The second study, "Selected Annotated Bibliography on Youth and Gang Violence Prevention, Community Team Organizing and Training, and Cultural Awareness Curriculums," developed a resource for youth-serving organizations and individuals, researchers, and policymakers concerned with youth issues.
Finally, ACF assessed its Low-Income Home Energy Assistance Program (LIHEAP) in a report to Congress for fiscal 1993, looking at State and Indian tribal use of funds to provide heating and cooling assistance, energy crisis intervention or assistance, low-cost home weatherization, or other energy-related home repairs. An estimated 5.6 million households received help with heating costs through heating or winter crisis assistance in fiscal 1993. This figure represents 20 percent of the 28.4 million households estimated to have incomes under the maximum income eligibility standard established by the LIHEAP statute. The mean home energy group burden for all eligible households (i.e., the ratio of home energy expenditures to income) was 4.0 percent of income compared with 1.1 percent for all U.S. households. Thirty-eight percent of all LIHEAP recipients received public assistance, and 65 percent received food stamps.
During fiscal 1995, ACF had 18 evaluation projects in progress. These projects are also linked to ACF's two strategic goals.
Three studies pertinent to the goal of "Economic Independence and Productivity of Families" are examining employment of welfare recipients. The JOBS evaluation, a major long-term study of the processes, impacts, and cost-effectiveness of the JOBS program, is designed to evaluate the effectiveness of alternative strategies for moving welfare recipients to work. Within the past year, the evaluation produced preliminary impacts on employment and welfare receipt at three sites (Atlanta, Georgia; Grand Rapids, Michigan; and Riverside, California), providing separate results for program models that are education focused (human capital development) and employment focused (labor force attachment).
The second study, which looks at the Parents' Fair Share (PFS) Demonstration, is testing the effects of requiring unemployed noncustodial fathers of children on AFDC to participate in employment and other services designed to increase their earnings so they can adequately support their children. PFS programs in seven States have developed effective procedures to identify eligible fathers, enroll them into employment services, and enforce regular participation. Preliminary data also show that PFS work and training requirements provide States with a promising mechanism to discover previously unreported income of nonpaying, noncustodial parents; approximately 25 percent of the men in the program had previously unreported income.
The third study, an evaluation of the Oregon Post-Employment Services Demonstration, addresses job loss among newly employed welfare recipients in four sites: Riverside, California; Chicago, Illinois; Portland, Oregon; and San Antonio, Texas. Each site furnishes job retention and reemployment services to recently employed JOBS program participants randomly assigned to receive the additional services, regardless of continued AFDC receipt.
Two studies under way address parenthood in welfare families. "Responsible Fatherhood: Theoretical and Empirical Foundations for Policy and Program Development," a joint project with the Office of the Assistant Secretary for Planning and Evaluation, is aimed at systematically developing credible information for States and localities about how to encourage and increase responsible conduct among fathers of disadvantaged children. Five community-based organizations are currently operating programs to teach fathers how to understand their children's development and positively affect their children's behavior.
A second study is assessing the effectiveness of the Home Visiting Services Demonstration in Chicago, Illinois; Dayton, Ohio; and Portland, Oregon. This demonstration requires first- time teenage parents on AFDC to participate in the JOBS program. It will test whether adding weekly home visitor services to mandatory JOBS programs will substantially strengthen the effectiveness of JOBS programs in helping young mothers better support themselves and their children while promoting positive parenting and reductions in repeat childbearing.
Two other studies are looking at child support enforcement issues. The first, "Evaluation of Child Support Guidelines," is funded by the Office of Child Support Enforcement (OSCE) and evaluates presumptive child support guidelines. The study will assess the impact of shifting from voluntary to presumptive guidelines by using the Current Population Survey--Child Support and Alimony Supplement of 1992 and will examine changes and activities of State Guideline Commissions.
A second OSCE study, the "Evaluation of Child Access Demonstration Projects," assesses demonstration projects in Idaho, Indiana, and Florida set up to test mediation services as a means to assist divorced, separated, and nonmarried parents reach parenting plans, as well as encourage greater involvement by noncustodial parents (usually fathers) with their children after divorce or separation from the custodial parent.
The final report in this group of related studies is titled "Identification and Prevention of Intergenerational AFDC Dependency: Promoting Long-Term Child Welfare." It will examine the causes of intergenerational welfare dependency, looking specifically at the critical age at which girls are most likely to be caught up in the dependency cycle. Factors distinguishing those who break free from dependency compared with those who cannot will be identified.
ACF has nine evaluations related to its second major strategic goal of "Healthy Development of Children and Families." Three of them address various facets of the Head Start program. First, the "Evaluation of the Head Start/Public School Early Childhood Transition Demonstration" is intended to assess the effectiveness of providing comprehensive, continuous, and coordinated services to Head Start families and children from the time of Head Start enrollment through the third grade in public school. The study will provide data on the effectiveness of the transition project models in maintaining the gains that children and families achieve while in Head Start.
A second study, "Evaluation of Head Start Family Child Care (FCC) Homes," will assess the quality of Head Start services provided in FCC homes and determine whether these services meet quality standards, including Head Start program performance standards. It will also compare services delivered in FCC homes with those delivered in Head Start centers.
The third study, a "Descriptive Study of Head Start Bilingual/Multicultural Program Services," will determine the number, geographic distribution, and sociodemographic characteristics of the Head Start eligible population for different cultural and linguistic groups, by region and nationally; will determine the number and nature of bilingual and multicultural children served by Head Start; and will identify the range of bilingual and multicultural services provided by Head Start nationally. The study includes an indepth assessment of service models, staff training approaches, community partnerships, and administrative plans and processes of a sample of 30 Head Start programs that use innovative methods to address the unique program needs of one or more of the diverse cultural and linguistic groups.
Several ACF projects in progress are looking at family protective services. The "National Study of Outcomes for Children Placed in Foster Care With Relatives" is examining the outcomes, including costs, for children and families in various configurations of relative foster care compared with similar configurations of nonrelative foster care. The "Family Preservation and Family Support Services (FP/FS) Implementation Study" will examine how FP/FS program funds for services to strengthen families have been used across States and communities and among different stakeholders. The main component of the study analyzes and synthesizes first-year State applications.
The "National Study of Protective, Preventive, and Reunification Services Delivered to Children and Their Families" will determine the number and percentages of children and families in the child welfare system that receive protective, preventive reunification, out-of-home care, and/or aftercare services. Case record abstracts will be completed on a nationally representative sample of 3,000 children and their families served by public child welfare agencies.
The "Evaluation of Nine Model Comprehensive Community-Based Child Abuse and Neglect Prevention Programs" is taking place in three phases over 3 years. Process and impact data are being collected across programs and through a series of studies conducted at each of the nine sites. Because each project has up to 10 service components, many of which vary across programs, a series of experimental designs has been developed for each service component.
ACF also is supporting two studies looking at family services from an international perspective. The "Transfer of International Innovations--Development of a Clinical Monitoring System To Support Foster Care in Michigan" features a computerized system based on a model combining structured and systematic monitoring of each individual child, with the aggregation of this information across the whole agency. Developed and now mandated in Israel, and modified for the U.S. foster care system, the system provides an integrated response to the needs of all partners in the agency, including administrators, managers, and policymakers. A second project is a knowledge transfer project assessing the application of a preteen Bedouin Arab dropout prevention program in a tribal community in the Negev Desert of Southern Israel to the Pascua Yaqui Indian Tribe in the Sonoran Desert of southern Arizona. The assessment will examine the effectiveness of a cooperative community/university model for empowering economically disadvantaged minority communities to respond to the unique needs of their at-risk children.
One ACF evaluation project addresses an important crosscutting issue of children and family services with substance abuse treatment. The "Woman and Infant Nurturing Services (WINGS)" program, designed to counteract the upward spiral of female incarceration and substance abuse, is being assessed. This demonstration project at the Rose M. Singer Correctional Facility, Riker's Island, New York City, targets pregnant, substance-abusing inmates and uses incarceration as a point of treatment intervention. The evaluation will test the effectiveness of a comprehensive service program, including substance abuse treatment, prenatal health and nutritional care, human immunodeficiency virus (HIV) education, parenting classes, mental health services, and assistance with entitlement preparation.
The passage of the Personal Responsibility and Work Opportunity Reconciliation Act of 1996 intensifies ACF's need to work even more closely with States and others to help improve the delivery, organization, and financing of human services to ensure positive impacts for children and families.
In recent years, State-initiated welfare reforms approved through waivers have been accompanied by carefully designed evaluation studies. For example, many States are testing the following: (1) requiring teen parents to attend school or training, (2) requiring minor parents to live at home or in other structured living settings, (3) increasing participation in JOBS, (4) increasing regular applications of larger sanctions for noncooperation with child support and JOBS requirements, and (5) encouraging savings by establishing special purpose accounts. One project described earlier has already provided States with very useful operational lessons on how they have increased participation in employment and training activities.
The law provides for continuation of these studies as well as for the initiation of new studies. For example, an area of current interest concerns the impact that welfare reforms will have on children. Several States have already expressed their interest to work in collaboration with ACF, ASPE, and other agencies to address this and related issues. Efforts are under way to develop measures of child well-being, to explore administrative databases as informational sources for evaluation, to identify methods for measuring outcomes and performance, and to move the field forward to benchmark, measure performance, and track results.
MISSION: To foster the development of services to help older persons maintain their independence.
AoA is the Federal focal point and advocate agency for older persons and their concerns. Under the Older Americans Act (OAA) of 1992, AoA administers programs to help vulnerable older persons remain in the own homes by providing supportive services, and offers opportunities for older Americans to enhance their health and to be active contributors to their families, communities, and the Nation through employment and volunteer programs. AoA works closely with a nationwide network of Federal regional offices, State and area agencies on aging, tribal organizations, national organizations, and representatives of business to plan, coordinate, and develop community-based systems of services that meet the unique needs of individual older persons and their caregivers.
As the responsibilities of this nationwide network of governmental and private organizations continue to grow, it is essential that they have the necessary information to meet these responsibilities. The overall evaluation priorities of AoA are to support studies that provide information on the success of existing program implementation in meeting the goals of the OAA; the design and operation of effective programs; and policy development, legislative planning, and program management.
During fiscal 1995, AoA completed two studies related to the assessment of the quality and effectiveness of services for older persons. The first project examined the effectiveness of the long-term care ombudsman programs. These programs were created more than two decades ago as a response to serious concerns about the quality of nursing home care. Currently operating in all 50 States, the District of Columbia, and Puerto Rico, these programs are funded from a combination of Federal, State, and local sources.
The role of the ombudsman has evolved to include both individual and systemic advocacy. Individual activities are designed to ensure that ombudsman services are accessible to all residents, systematically receive and respond to resident concerns about their long-term care services, and work cooperatively with a variety of agencies to resolve problems.
Systemic advocacy activities include evaluation of laws and regulations related to long-term care services for older adults, education of the public and facility staff, dissemination of data about the ombudsman program, and promotion of the development of citizen organizations and resident and family councils. Both roles are important, but the primary activity of ombudsman programs is identifying, investigating, and resolving individual resident complaints.
The evaluation was conducted by the Institute of Medicine (IoM) of the National Academy of Sciences/National Research Council. An IoM committee developed a formative evaluation strategy to identify program issues, strengths, and weaknesses. The evaluation results documented the public purpose served by long- term care ombudsman programs and endorsed the continuation of their mandate, offering several examples of ways in which they have provided valuable assistance to individuals and improved the service system. In some areas, the committee found that meaningful access to ombudsman services does not exist for all eligible nursing facility residents because of infrequent site visits and insufficient information outreach efforts. In general, complaints receive a timely response. Program implementation in board and care (B&C) homes is much more limited than in nursing facilities. Because the committee could not make a definitive assessment of program effectiveness, they recommend development of a process to facilitate uniform process and outcome evaluation in the future.
The IoM committee also recommended a strengthened leadership and oversight role for the Federal Government, including implementing an objective compliance review method for States. An improved data system is needed that includes information about program awareness, as well as the extent of complaint resolution, the extent of ombudsman input into systemic improvement activities, and the extent of ombudsman impact on the overall system of long- term care and services.
The second project completed by AoA developed the conceptual framework and provisional design for two studies that the 1992 OAA amendments authorize the Secretary of HHS to conduct on the quality of care provided by B&C facilities and on the quality of home care services for older and disabled individuals. The evaluation was conducted by IoM with the understanding that funding for the larger studies would need to come from other public and private resources. These recommendations were based on the deliberations of an outside expert study committee that directed the study, a literature review, and presentations and comments made by consumers, leading researchers, and State officials at an invitational workshop.
The final report recommends that one study be conducted that looks at quality assurance in both home care and residential care settings. It further recommends the focus be on all users of home- and community-based long-term care, reaching beyond just the elderly or disabled populations. The report lays out a study design to examine the key features that define home care services and the consumers receiving them, the frequency and severity of quality problems in home care and in residential care, the factors that enhance or impede quality care, the roles of consumers and their informal caregivers, and the need for national minimum standards or model standards to ensure the quality of home and residential care.
AoA is currently supporting an evaluation of the Elderly Nutrition Program (ENP). With the aging of the U.S. population, increased attention has been given to designing efficient service delivery systems for older people. Of particular concern is the development of service networks that can provide elders with a continuum of home- and community-based long-term care, thus avoiding premature or unnecessary institutionalization. An important component of any long-term care system is the provision of adequate nutrition services to ensure that optimal nutritional status in the older population is achieved and maintained.
The 1992 OAA amendments directed AoA and HHS's Assistant Secretary for Planning and Evaluation to evaluate the nutrition programs funded under OAA Title III and VI, ENP. ENP has not been evaluated since 1983, and the nutrition program funded under Title VI had never been evaluated. The project is being conducted by Mathematica Policy Research, Inc. Congress mandated that the evaluation address four objectives: (1) evaluate who is using the program and how effectively the program serves targeted groups, (2) evaluate the program's effects on participants' nutritional status and socialization, (3) assess how efficiently and effectively the program is administered and delivers services, and (4) clarify program funding sources and allocation of funds among program components.
The project data collection and preliminary analysis were completed during fiscal 1995; the final analysis and report are expected to be completed in fall 1996. The final report will be disseminated to Congress, to the aging network, and to the public, with public-use tapes also available for further research.
Given the evolving roles of AoA and State and area agencies on aging together with the projected growth of the Nation's elderly population, AoA's evaluation efforts will continue to focus on program planning and service delivery as well as continued monitoring of the effectiveness of programs in addressing the goals of OAA.
The changes occurring in the aging service delivery network present an opportunity to learn critical lessons in program planning and system development. These changes include the more systematic focus on home- and community-based long-term care, increased sophistication in addressing a variety of needs of the older population, and changes in the health care system such as the growth of managed care and the implementation of enhanced program information systems. Future evaluations will need to consider the impact of these and other similar developments on the delivery of OAA services to the Nation's older persons.
MISSION: To generate and disseminate information that improves the health care system.
AHCPR has designed a portfolio that responds to the evaluation needs for the following:
Evaluation efforts are built into virtually all AHCPR program activities. The evaluation mechanisms used by the agency include targeted evaluation studies undertaken through contracts or grants; peer review of grant applications and technical review of contract proposals for scientific integrity; obtaining information on the usefulness of AHCPR research efforts and findings through AHCPR's User Liaison Program, which provides information to State policymakers, health departments, and officials; and other targeted efforts such as focus groups and surveys to provide baseline information and assist the design of future AHCPR work.
AHCPR established a two-tier system for assessing proposed evaluation projects. The process begins with an executive-level review to assess policy relevance and relative priority of proposed projects. The review is conducted by the Administrator and senior staff. The second, or technical merit, review assesses policy-relevant project proposals for feasibility, soundness of design, costs, potential importance of the findings, and relation to ongoing evaluation activities. The second review is conducted by the Task Force on AHCPR Evaluation Projects, a group consisting of one individual with evaluation expertise from each office and center in AHCPR.
The 12 AHCPR evaluation projects completed in fiscal 1995 reflect AHCPR's mission of increasing scientific knowledge for improved clinical decisionmaking and for organizing public and private systems of health care delivery. The projects are part of AHCPR's efforts to develop information that will improve health care quality, reduce health care costs, and enhance access to health services; respond to information needs of consumers, clinical practitioners, payers, and policymakers; and promote dissemination and use of new information about health care systems, methods, and technologies.
The first group of projects consists of four studies to advance methodologies for evaluating and improving the quality of clinical care. The first project, a contract with the Center for Clinical Quality Evaluation in Washington, D.C., translated three AHCPR-supported clinical practice guidelines (on urinary incontinence, acute postoperative pain, and benign prostatic hyperplasia, or BPH) into clinical performance measurement systems. The project found it is possible to develop valid and reliable guideline-based performance measures from AHCPR- supported guidelines, and the structure of guideline recommendations can have a strong influence on developing such measures. The project also found that targeted educational efforts conducted by Medicare peer review organizations and based on the BPH guideline resulted in improved performance on specific aspects of clinical care.
The second project, a contract with Rand of Santa Monica, California, developed a quality review system based on the AHCPR- supported cataract guideline. This review system was designed for use in a range of settings including ophthalmologists in fee-for- service private practice, salaried ophthalmologists on staff at model health maintenance organizations (HMOs) or other managed care organizations, and ophthalmologists providing eye care services under capitated contracts or subcontracts with capitated primary care groups or other managed care organizations. The project found that it is feasible to develop performance measures based on the cataract guideline and that a medical record-based review system can be very useful for estimating guideline adherence and can highlight variations in practice and opportunities for quality improvement.
The third project, also conducted by Rand, developed a review system based on the guideline for prediction and prevention of pressure ulcers. The pressure ulcer guideline review system was developed and tested in collaboration with the Department of Veterans Affairs (VA) and was tested among VA and non-VA hospitals. The study revealed that because pressure ulcer care is multidisciplinary and recommends daily performance of many behaviors, the review systems required collection of data elements from a variety of data sources over a statistical sampling of hospital days.
In addition to developing clinical performance measures from science-based guidelines, these first three studies pointed to the need for organized and comprehensive information on the types of clinical performance measures in existence as well as the need for methods for evaluating the structure and quality of these measures. The fourth project therefore developed a typology or framework for collecting and evaluating clinical performance measures. AHCPR's fiscal 1995 measurement typology projects, described in chapter II, established a common language that can be used to compare measures. It also identified areas where additional clinical quality measures are needed.
Another category of projects completed in fiscal 1995 responds to the information needs of consumers, clinical practitioners, payers, and policymakers. The first of these projects, the Survey Design Project (SDP), developed prototype survey modules to obtain consumer assessments of their access to care, use of services, health outcomes, and patient satisfaction. The SDP forms the basis of AHCPR's Consumer Assessments of Health Plans Study (CAHPS), a five-year project to develop and test questionnaires that assess health plans and services, produce easily understandable reports for communicating survey data to consumers, and evaluate the usefulness of these reports for consumers in selecting health care plans and services. The second project in this category, using a qualitative case study approach, identified factors related to the selection, implementation, and use of information management systems. Project findings guided the development of an AHCPR research solicitation focused on understanding computerized decision support systems for clinical quality improvement. The third project in this category focused on improved clinical information from managed care settings. Conducted by the Institute for Health Policy Studies at the University of San Francisco, this study assessed the feasibility of using managed care data to assess the differential use of selected tests and procedures. The final project in this category developed a directory of minority health and human services data resources that can be used by researchers and others interested in examining data that contain race and ethnic identifiers.
The final group of projects completed in fiscal 1995 concentrated on evaluating the usefulness of AHCPR-supported clinical practice guidelines. A project conducted by George Washington University developed analytical designs and methods that could be used to evaluate the process of developing clinical practice guidelines. The project identified key variables related to the process of guideline development and assessed the feasibility of measuring the variables across guidelines. The next study, conducted by the American Institutes for Research, collected detailed and structured information from guideline users on issues associated with implementation of guidelines and their perceived impact. Approximately 100 physicians in office-based settings, physicians in academic settings, and nurses participated in 10 focus groups across the United States and provided information on how better to construct and disseminate information on improving clinical practice. A third project, conducted by the Institute of Medicine, described and evaluated methods for setting priorities for guideline topics. The final report in this category, AHCPR Clinical Practice Guidelines Program: Report to Congress, summarizes the Agency's evaluation work to date and highlights future directions for the Agency's work.
These findings, as well as those from other AHCPR-supported evaluation projects, suggest that the methods for introducing and implementing guidelines into clinical settings are extremely important in fostering conformance with science-based clinical recommendations. As a result of findings from this and other projects, AHCPR will no longer be developing clinical practice guidelines but instead will focus on science-based partnerships to improve health care delivery. This new three-faceted strategy will continue AHCPR's commitment to the synthesis of scientific and medical effectiveness information, establish public-private partnerships to make guidelines and other quality-related information accessible through a national clearinghouse, and continue to support research and evaluation focusing on methods to integrate scientific and medical effectiveness information into everyday clinical practice.
AHCPR's ongoing evaluation activities also concentrate on generating information to improve the health care system. In the area of information for clinical quality improvement, AHCPR has supported two followup efforts to the fiscal 1995 measurement typology project, described in chapter II. The first is the development of CONQUEST 1.0, the COmputerized Needs-Oriented QUality Measurement Evaluation SysTem. The CONQUEST project builds on its predecessor project in three ways. First, the project evaluates and strengthens the framework developed in the typology project by verifying the data with measure developers. Second, it creates an interlocking database of clinical conditions with coded information from AHCPR-supported and other clinical practice guidelines, so users can identify practice recommendations and link them to clinical quality measures. Third, the project creates a computerized system with a user- friendly interface to link measures to clinical information and guide the selection of measures. A related project currently under way involves evaluating this product by convening users to pilot test CONQUEST and to participate in focus groups about its usefulness.
Another project under way in fiscal 1996 builds on AHCPR's consumer choice work. The survey design project conducted by Research Triangle Institute and CAHPS develop surveys to collect data on consumers' satisfaction with access to and use of health services and methods to disseminate this information for improved consumer choice. Through the Survey Users Network, a related contract, AHCPR will conduct needs assessments with users to obtain information on how they use consumer surveys as well as their technical assistance needs in issues such as sampling, survey administration, data analysis, and reporting. The needs assessments conducted under the survey will help inform AHCPR of the CAHPS' work.
In the coming years AHCPR will focus its evaluation activities on three general areas. The first area is the need to develop AHCPR- wide performance measures that can be used to evaluate the quality of the Agency's work as well as to inform policymaking, budget planning, and program management. Efforts are under way to develop evaluation designs that will identify key customers or users of AHCPR information, identify their information needs, and obtain feedback on the usefulness of the Agency's products and reports.
The second general area centers on building AHCPR's research portfolio to translate research findings into forms of information that actively help consumers, practitioners, payers, and others make effective health care decisions. This area builds on research related to decision-support systems and hospital information systems, as well as psychosocial and organizational research examining factors that influence provider practice and clinical quality improvement.
The third area will build on AHCPR's work to examine and evaluate changes in the health care system, and how those changes affect access to care, use of services, quality of care, and patient outcomes. Through the use of targeted evaluation projects, analyses of data from the Medical Expenditure Panel Survey, and other media, this area of research will help HHS evaluate the effect of policy changes. For example, one area might focus on comprehensive evaluations and evaluation syntheses of the effect of managed care on vulnerable populations--and specifically, the impact of system changes such as the Section 1115 demonstration waivers on Medicaid beneficiaries, on health professionals, and on public health organizations and services within and across States.
MISSION: To promote health and quality of life by preventing and controlling disease, injury, and disability.
MISSION: To prevent exposure and adverse human health effects and diminished quality of life associated with exposure to hazardous substances from waste sites, unplanned releases, and other sources of pollution in the environment.
CDC places high priority on evaluations seeking to answer policy, program, and strategic planning questions. Performance improvement studies are of particular interest and importance to CDC. Evaluation studies are developed and selected based on CDC's eight strategies to achieve its mission. These strategies are as follows:
CDC's evaluation philosophy is consistent with its overall focus on the practice and application of sound public health strategies. This orientation is exemplified by CDC's "bottom up" nature of project development. Projects are conceptualized, developed, and monitored by CDC's public health professionals who are most closely aligned with the everyday practice of public health--the programmatic staff.
Annually, the Director of CDC provides guidance to the various center, institute, and office (CIO) directors on 1 percent set- aside evaluation activities. This memorandum generally includes information about the types of studies to be carried out with 1 percent evaluation funds. Each proposal undergoes multiple levels of review. Initial review is conducted by the Office of Program Planning and Evaluation (OPPE). Subsequent reviews are completed by the Office of the Assistant Secretary for Planning and Evaluation staff. Study authors are provided with comments, questions, and recommendations made by reviewers. In addition to providing their responses, authors may revise their proposals at this time.
A panel of CDC evaluators, scientists, and program managers are convened to review and rank proposals. Review criteria include (1) relevance to prevention effectiveness, (2) relative importance of the public health problem being addressed, (3) probability that the proposed project will accomplish its objectives, and (4) extent to which other CDC programs will derive benefit from the project. Results from the panel review are converted into a comprehensive ranking that is provided to the Director of CDC. Final funding decisions are made at that time.
Finally, staff within OPPE work closely with program staff to ensure development of a clear statement of work for selected projects. Prior to initiation of procurements, a final ad hoc review of the project statement of work is completed.
ATSDR receives its funds from the U.S. Environmental Protection Agency/Superfund appropriations rather than Public Health Service appropriations; therefore ATSDR does not receive a 1 percent evaluation set-aside. Nevertheless, ASTDR is responding to the changes mandated in its program planning and evaluation efforts by the National Performance Review and the Government Performance and Results Act (GPRA). To meet those requirements, ATSDR staff members modified ATSDR's planning process by incorporating implementation strategies and outcome/performance measures.
Prominent issues addressed in the new planning system emphasize ATSDR's priority to improve the health of people affected by hazardous substances polluting the environment. Using exposure assessments and demographic data to identify people at risk and, more directly, assessing/addressing the concerns of the Agency's customers represent some of its improvements. The new planning system provides the basis for measuring ATSDR performance and making systematic improvements a part of its internal evaluation activities.
Fourteen evaluations were completed in fiscal 1995. Of this number, 10 were funded through the 1 percent evaluation set- aside, and 4 were funded with program dollars. These studies are of four types: data policy/surveillance studies, program evaluations, evaluations of educational interventions, and organizational assessments.
Five studies sought to assess the implications of policy decisions on various existing or proposed data/surveillance systems. One of these studies addressed automated analysis of the four major data systems maintained by the National Center for Health Statistics: vital statistics, health examination, health interview, and health care. The primary domain of this project was health care and health interview data. Another study assessed the feasibility of developing an injury risk factor surveillance system; a third project entailed the conduct of two distinct surveys to explore issues related to influenza vaccination coverage.
Three program evaluation studies were completed in fiscal 1995. These studies included evaluations of the Medical Examiner/Coroner Information-Sharing Program, the Fatality and Assessment and Control Program, and the Lead Poisoning Prevention Program (LPPP). These program evaluations were context-specific. For example, the evaluation of LPPP examined an emerging trend in the program--the effects of managed care reforms on childhood lead poisoning prevention (CLPP) programs, focusing on CLPP programs funded by CDC.
Of the three evaluations of educational interventions completed in fiscal 1995, two of them focused on guides to diabetes education. One guide, written in Spanish, was evaluated for the appropriateness of its use for diabetic patients in Hispanic populations. The second diabetes educational intervention focused on health care providers. This project provided information to the Division of Diabetes Translation on the impact of three different provider interventions related to the dissemination of a provider guide. The final study evaluated the progress and activities conducted by the 71 HIV education programs funded by CDC's Division of Adolescent and School Health through cooperative agreements with State and local education agencies.
Three organizational assessments were carried out in fiscal 1995. Two of these studies addressed management issues of discrete programmatic areas: immunization and women's health. The third study sought to develop a comprehensive evaluation strategy that can be incorporated into planning budget and legislative processes in the National Center for Chronic Disease Prevention and Health Promotion.
During fiscal 1995, 18 evaluation studies were in progress. These studies consist of three types: evaluations of data policy/surveillance systems, program evaluations, and assessments of specialized aspects of proposed programs.
Six studies are under way that relate to either data management or reporting systems, or both. These studies include evaluations of racial and ethnic identification data, surveillance data for drug-resistant pneumococcal infections, and an assessment of the National Nosocomial Infections Surveillance (NNIS) Program.
In the study of racial and ethnic identification data, three major activities will be performed to improve the data quality of natality data systems. These activities include (1) a systematic evaluation of how respondents for birth records interpret current questions about their ethnic and racial identity; (2) an assessment of how respondents classify themselves on birth certificates regarding the current racial and ethnic categories used in the National Center for Health Statistics' natality statistics data systems; and (3) a comparison of respondents' interpretation and classification in response to the current categories of how they would interpret questions and classify themselves by using other racial and ethnic categories under consideration by the Office of Management and Budget.
The study evaluating CDC's current surveillance system for drug- resistant streptococcus pneumoniae will evaluate the system's hospital-based sentinel surveillance system for sensitivity and representativeness. Active population-based surveillance for invasive pneumococcal infections in two geographically distinct areas will be conducted. Surveillance areas for this evaluation are either adjacent to a sentinel surveillance hospital or in a community with one or more hospitals that are sufficiently large enough to serve as a comparison for the rest of the community.
Finally, the major purpose of the assessment of the NNIS Program is to develop a methodology for assessing NNIS hospitals' accuracy in identifying and reporting nosocomial infections in NNIS hospitals and to assess the utility of criteria used to diagnose these infections. Specifically, the study is designed to (1) confirm the presence of nosocomial infections, by site, in patients previously reported to NNIS as infected; (2) confirm the absence of infections, by site, in a hospital-selected population of patients who were at risk for acquiring nosocomial infections but for whom no infections were reported; (3) identify certain risk factors and nosocomial infection case-finding criteria to the identified infection site; and (4) determine whether consistent findings can be achieved by two similarly trained observers who apply specific nosocomial infection case-finding criteria to a retrospective review of a given medical record.
Seven major program evaluations are currently in progress. Included in this number are evaluations of the National Institute for Occupational Safety and Health's Health Hazard Evaluation (HHE) Program, the Field Epidemiology Training Program (FETP), the National Laboratory and Training Network (NLTN), and the Dengue Hemorrhagic Fever Prevention and Control Program in San Juan, Puerto Rico.
The HHE study is a process evaluation. As such, the overall purpose of the study is to develop a tool and procedure by which NIOSH can conduct an ongoing evaluation of the effectiveness of the HHE Program, which responds to 400 to 500 requests for onsite health hazard evaluations each year from employers; employees; employee representatives; or Federal, State, or local agencies.
By contrast, two of the studies mentioned are outcome evaluations. The evaluation of FETP will ascertain whether the program has achieved its objectives to train public health professionals in applied epidemiologic skills, to promote the sustainability of autonomous FETPs, and to develop a global network of national programs.
The evaluation of NLTN will assess the degree to which NLTN achieves its goals and mission. Study questions fall into five general categories: (1) offerings related to needs of laboratories and their staff, (2) quality of the training provided, (3) impact of the training provided, (4) quality of outreach and marketing, and (5) barriers to training.
Finally, the evaluation of the Dengue Hemorrhagic Fever Prevention and Control Program will assess three major components of the program: (1) proactive disease surveillance, (2) education of the medical community, and (3) community-based mosquito control.
Five specialized studies relating to discrete aspects of current or proposed programs are in progress. These studies include an evaluation of tuberculosis (TB) outreach worker activities as well as a community-context study seeking to understand minors' access to tobacco products. Two studies related to violence prevention are also included in this category. One of these studies relates to suicide in Native American communities, and the other considers various implementation aspects of youth violence prevention programs.
Each of the studies mentioned has as its core a desire to understand particular human behaviors and motivations. A clear understanding of these issues must be present before wholehearted program and fiscal commitments are made. For example, the Advisory Committee for the Elimination of TB recommended increased funding for TB outreach and workers conducting the outreach. However, prior to committing resources to this strategy, CDC must have a clear understanding of how outreach work is currently conducted, and what outreach workers do in their everyday work. It is through the evaluation of TB outreach workers that the National Center for HIV, STD, and TB Prevention will be provided with baseline data about how outreach workers function.
Similarly, a community context study is also being conducted that will help refine CDC's understanding of the relationships between public policies prohibiting minors' access to tobacco, the implementation and enforcement of such policies, tobacco-vendor perceptions and actions that may influence the sale of tobacco products to minors, and the use of tobacco products by minors.
Finally, two evaluations addressing violence prevention issues are also under way. The first study is being conducted jointly by CDC and the Indian Health Service. The study will evaluate surveillance systems, process indicators, and outcomes of four multifaceted suicide prevention programs. The second violence prevention study will identify key factors in the successful development and implementation of CDC-funded youth violence prevention programs. As part of the study, a document will be developed that will assist individuals, groups, or communities as they implement their own youth violence prevention programs.
CDC's evaluation program will continue to support studies aimed at improving public health programs. This focus on improvement is consistent with the congressional intent of GPRA. Efforts to develop and implement performance indicators are currently under way at CDC. Similarly, as programs have been developing and implementing performance indicators, projects that assess the effectiveness and efficacy of such indicators have been initiated. CDC's evaluation priorities in upcoming years will be based on five crosscutting strategies that address the programmatic needs of its 11 CIOs.
A number of key activities at CDC center around its commitment to strengthen the Nation's core public health functions. Tangible results of this commitment are reflected by CDC's work with State and local health departments and other partners throughout the country to obtain information necessary for monitoring and evaluating health, conducting epidemiologic and laboratory studies, developing new technologies, and providing training and technical assistance to help ensure that States and communities can protect the health of their citizens. Ongoing and proposed evaluation studies seek to answer how well CDC is fulfilling this commitment.
Epidemiologic investigations and laboratory work done by CDC and State and local health departments enable CDC to address urgent threats to public health in a timely and effective manner. Similarly, in a world increasingly threatened by emerging infections, CDC's leadership role in this area is critical. In fiscal 1996, a number of evaluation studies address important programmatic issues related to infectious disease.
One of CDC's key roles is to translate knowledge about effective methods of preventing disease and injuries into nationwide strategies that reach people in communities throughout the country. Several studies under way seek to assess the extent to which guidelines developed by CDC impact disease prevention activities throughout the Nation.
Recognizing that women's health issues have not always received the attention they have warranted, CDC has identified promotion of women's health as one of its top priorities. Although a number of projects are funded by CDC's Office on Women's Health, there are several ongoing evaluation studies that address important women's health issues. An example of one such study is an evaluation of guidelines for preventing perinatal HIV infection by measuring how recommendations are translated into routine clinical practice. Successful implementation of recommendations depends on the ability to maximize the opportunities for HIV- infected women to learn their HIV infection status, to be offered and receive preventive therapy, and to gain access to health and social services for themselves and their infants.
CDC's commitment to investing in the Nation's youth is exemplified by its school health education programs. However, CDC's investment is not limited to school-based intervention programs. For example, one recently initiated study is designed to assess the effectiveness of various teen pregnancy prevention interventions. This study will result in a systematic methodology to identify and evaluate community intervention components that show promise based on the behavioral science theory, available evaluation information, and consensus of leaders in the field.
MISSION: To protect and promote public health through food, drug, medical device, and cosmetic regulation.
Systematic changes in the government management environment are strongly influencing the setting, conduct, and use of evaluation activities in FDA. Three forces--all related to the goals of the Government Performance and Results Act--are reshaping the evaluation function within FDA.
The reorientation of all government managers toward performance management has shifted the responsibility for program evaluation from specialized staff offices and contractor studies to the day- to-day line managers. Picking appropriate program goals, establishing a valid measure for those goals, and collecting management information to record the measured progress toward the goals are now integral parts of the new government manager's responsibility. Relearning the role of management regarding these shifted responsibilities is a key priority.
Most of FDA's management performance measures are the same as those for regulated industries. Thus, although the beneficiary of FDA's performance is ultimately the general public, FDA operates in a manner that supplies industry with an essential component of commercial success. FDA's approval of a new drug, for example, not only satisfies a legal requirement but also assures the public of the safety and efficacy of the drug. Collaboration between FDA and its regulated customers regarding the design and coordination of the joint responsibilities to ensure effective high-quality products was a revolutionary concept, but it is becoming the new norm under the customer-conscious GPRA directives.
FDA establishes standards of safety and efficacy through rules published in the Federal Register. Many safety and efficacy standards are also performance standards that industry is obliged to meet. Examples are the Good Manufacturing Practices regulations. Today, virtually every FDA final rule of significant magnitude includes elements found in classic program evaluations- -a critique of the existing system, alternatives for better performance, performance and cost tradeoffs, reactions and suggestions of customers, and conclusions with an action timetable for implementation. The new directives of the Administration as well as the possible legislative action by Congress will bring the rulemaking process of the future even closer to the classic evaluation process.
In summary, FDA's evaluation efforts are driven by the mandates of GPRA and its corollaries; are carried out by line managers rather than specialized evaluation staffs; and are focused on performance management, customer participation, and rigorous rulemaking. Because it receives its funds from the U.S. Department of Agriculture's appropriations rather than Public Health appropriations, FDA does not conduct evaluations under the 1 percent evaluation set-aside authority.
The past year was the second year under the new evaluation paradigm. The following examples illustrate the manner in which managers have integrated evaluation into their line responsibilities and have conducted evaluation in cooperation with the affected customer.
"Implementation of the Prescription Drug User Fee Act." In fiscal 1995, FDA's drug and biologic review processes involving more than 1,000 Field Test Evaluations (FTEs) completed their third full year of successful performance management toward goals and performance measures jointly established by FDA managers and industry customers. The goals were ambitious. Line managers agreed to clear an overdue backlog equivalent to half a year of submissions, review a pending workload equal to nearly 2 years of submissions, and build a review capacity that could meet a phased schedule of substantially accelerated review goals. FDA managers met or exceeded all of the performance goals for the year.
"Assessment of the Mammography Quality Standards Act of 1992 (MQSA)." This program is FDA's second major user-fee, performance-oriented, GPRA-style initiative. MQSA requires facilities that perform mammography services to be certified as meeting standards developed by FDA. FDA managers want to ensure that as MQSA is implemented, patients can maintain access to quality services, especially patients in areas that are short of health professionals. To achieve this goal, FDA managers contracted for a cost-benefit analysis of the program. The contractor reviewed records from more than 10,000 facilities that had applied for accreditation and from more than 500 that had ceased operations. The contractor presented the results of the cost-benefit analysis to FDA managers who then used them to develop a performance-oriented regulatory policy coordinated with participating customers.
The evaluation agenda of line managers in fiscal 1996 will be influenced by the objectives discussed below, which merely reflect the forces shaping their evaluation role.
Managers of all FDA programs are evaluating their performance measures in light of GPRA standards. Broad-based training of managers to enable them to implement the performance design of their responsibilities is under way. This multiyear process toward a new standard of performance measure will end its first phase with the formulation of FDA's 1997 budget.
FDA managers are identifying further opportunities to involve their customers in the design and testing of alternative ways of doing business. The design and implementation of the Prescription Drug User Fee Act (PDUFA) is the most notable example of the successful pursuit of this objective. Initiatives with import brokers to facilitate the entry of safe products into the country is another example. Managers plan to involve customers in the redesign of several establishment inspection functions.
FDA managers are adding a new customer-sensitive dimension to their increasingly rigorous rulemaking function by implementing the President's directive to promote negotiated, consensual rulemaking. This is a marriage of customer participation and rigorous rulemaking that is attracting serious attention by FDA managers.
One example of the evaluation projects managers will undertake in fiscal 1996 to implement the stated objectives is the Fourth Annual Evaluation of PDUFA. Fiscal 1996 is a critical year for implementing PDUFA. Negotiations on its renewal will begin shortly thereafter. FDA managers will use performance data from this GPRA-style program to convince customers and Congress of the merits of continuing the program.
Changes in government management have created a systemic change in FDA's evaluation function. The new paradigm driven by line managers' performance responsibilities, by the imperatives for involving customers, and by directives for analytically rigorous rulemaking has replaced the traditional practice of evaluation guided and managed by centralized, specialized evaluation staffs.
MISSION: To promote the timely delivery of appropriate, quality health care to the Nation's aged, disabled, and poor through administration of the Medicare and Medicaid programs.
The research arm of HCFA, the Office of Research and Demonstrations (ORD), performs and supports research and demonstration projects (through intramural studies, contracts, and grants) to develop and implement new health care financing policies and to provide information on the impact of HCFA's programs. The scope of ORD's activities embraces all areas of health care: costs, access, quality, service delivery models, and financing approaches. ORD's research responsibilities include evaluations of the ongoing Medicare and Medicaid programs and of demonstration projects testing new health care financing and delivery approaches. These projects address four major themes:
During fiscal 1995, HCFA completed three major evaluations.
"Monitoring the Impact of Medicare Physician Payment Reform on Utilization and Access" represents a combination of a wide series of intramural and extramural projects and results in an annual report to Congress. This annual report was initiated when the new Medicare fee schedule for physician services was implemented to track the changes that might be caused by such a major shift in the Medicare payment approach. The fiscal 1995 report indicates that the new payment system has produced the kinds of shifts in payments that were anticipated. There are continuing indications that many vulnerable population groups face barriers to care; additional understanding of these barriers is needed to improve access to care.
The "Medical Participating Heart Bypass Center Demonstration Evaluation" covers the first 3 years of a demonstration carried out at seven hospitals across the United States. Its findings support the feasibility of an all-inclusive negotiated bundled payment arrangement for heart bypass surgery at high- volume hospitals. This evaluation focused on the assessment of surgical appropriateness, quality of care, hospital and Medicare program savings, and patient satisfaction.
Finally, an evaluation of the Rural Health Care Transition Grant Program produced information for an annual report to Congress on this HCFA grant program for rural hospitals. The report describes the new awards made in fiscal 1994, the characteristics of grantees compared with earlier cohorts, and the impacts of the grants on the hospitals and their communities.
ORD currently supports 39 evaluation projects in progress. These projects provide information for continued monitoring of the Medicare and Medicaid programs and assess the impacts of HCFA's Medicare and Medicaid demonstration projects. In-progress evaluations that are scheduled to be completed in fiscal 1996 include examinations of Medicaid program initiatives such as 1915(b) waivers and Community-Supported Living Arrangements and evaluations of the following several demonstration projects:
HCFA's evaluations in progress that are scheduled for completion in later years include evaluations of State health reform demonstrations, including Oregon and Tennessee, and of the following ongoing demonstrations:
As the U.S. health care system continuously changes, there is a clear need for the development, design, and testing of new ways to monitor and evaluate the performance of the system. It is important that monitoring and evaluation efforts for the Medicare and Medicaid programs include a number of critical dimensions to provide an understanding, on an ongoing basis, of how well these programs are performing in terms of access to care, quality, efficiency, costs, and beneficiary satisfaction. ORD is working to develop a comprehensive monitoring and evaluation plan for systematically examining the Medicare and Medicaid programs. It also will continue to work to develop a wider array of evaluation and measurement tools. Evaluation activities will continue to examine specific policy issues within the HCFA programs. For example, as Medicare and Medicaid continue to pursue managed care options, ongoing work will examine the cost-effectiveness and quality of and beneficiary satisfaction with managed care. HCFA also plans to carry out projects to monitor and compare the health status and/or health risk behaviors of beneficiaries in various delivery systems and how these change over time.
Finally, as HCFA develops and implements new high-priority demonstrations that will test new payment and health care delivery models for the future, it will continue to evaluate these programs and provide information to policymakers about the impacts of these alternatives. The new projects include the Medicare Choices Demonstrations, which will test the feasibility and desirability of new types of managed care plans for Medicare; demonstrations of Medicare payment for telemedicine services in rural areas; Operation Restore Trust, which is demonstrating improved methods for investigation and prosecution of health care fraud and abuse; a demonstration of Centers for Excellence that replace separate fee-for-service payments with capitated payments for the entire medical costs associated with certain expensive medical/surgical procedures; a competitive pricing demonstration to evaluate an array of cost-saving approaches of paying for managed care; and the continued monitoring and evaluation of State health reform demonstrations.
MISSION: Improve the health of the Nation by assuring quality health care to underserved and vulnerable populations and by promoting primary care education and practice.
The purposes of the HRSA evaluation program are to enhance strategic planning, budget decisions, and legislative planning, and to improve program management. Consequently, major emphases during fiscal 1996 and beyond will be performance measurement and assessment of program implementation and policy.
Performance measurement includes (1) technical and training activities to strengthen HRSA's capacity to assess program performance, and (2) studies to assess the outcomes of individual programs or groups of programs. Projects to enhance measurement capacity have assumed greater emphasis in relation to the Government Performance and Results Act. During fiscal 1995, HRSA established a performance measurement baseline for all operating programs; the ultimate objective is to ensure that HRSA establishes valid and useful indicators and measures for all programs by the end of fiscal 1998. Over the past year, HRSA has made significant progress in performance management that provides operational linkages between strategic planning, program activity, and the budget process--as intended by GPRA.
Assessment of program implementation and policy includes a broad range of efforts to assist HRSA during a time of significant change caused by new policy directions initiated by the Administration or Congress. Specifically, HRSA-funded programs are entering into new arrangements for delivering services, providing health professions education, and encouraging the development of systems reform within seven program priority areas. These priorities reflect HRSA's focus on underserved populations. They include academic and community partnerships to foster clinical training in community-based settings; new arrangements to bring poor, uninsured, rural, and chronically ill persons into the mainstream of managed care; assistance to communities in strengthening their health care infrastructure; and new activities with States. HRSA's studies relate to one or more of these priorities and include efforts to describe and assess the initial or later implementation of a program; compare alternative approaches to delivering services; assess the benefits of a current or potential policy; examine the effectiveness or efficiency of resources management; and conduct evaluation syntheses.
The objective of HRSA's evaluation work is to provide useful and timely information to the Administrator, the four bureaus, and the Office of Rural Health Policy. To ensure that this objective is met, all study proposals are reviewed first by a committee composed of the bureau directors and four other senior executives who consider the proposals' relevance to important policy, budgetary, or legislative issues; potential to answer questions about program effectiveness or impact; and degree of attention to crosscutting topics. This committee makes recommendations to the Administrator about study approval and in so doing establishes relative priorities for funding. Subsequently, expanded materials needed for developing contracts for approved studies are reviewed by a committee of senior analysts from HRSA, the Agency for Health Care Policy and Research (AHCPR), the National Center for Health Statistics (NCHS), and the Office of the Assistant Secretary for Planning and Evaluation (ASPE). The role of the latter group is to offer suggestions, as needed, on ways to strengthen study design and/or data sources.
HRSA places major emphasis on the effective dissemination of findings and information about the use of completed studies. Dissemination is done in a number of ways and is especially designed to ensure that evaluation information promptly reaches the Administrator and other agency executives. In addition, attention is given to a broader dissemination through articles in professional literature and presentations at professional conferences. An annual report on completed studies describes purpose, findings, and uses; a second annual volume provides brief summaries of studies initiated the preceding fiscal year.
HRSA completed an agencywide review of performance measurement capacity for all line programs last year through a project that is highlighted in chapter II, "Performance Indicators for GPRA: Initial Assessment of HRSA Programs." This work is the foundation for current component-specific efforts and for developing the fiscal 1998 budget submission. Three projects reflect differing aspects of health professions work force issues. A study of user satisfaction found that information from the National Practitioner Data Bank was clearly important in identifying practitioners who had not voluntarily revealed their "problem records" to hiring and certifying authorities. HRSA has used the study to develop the improved data bank system implemented in June 1995 and to strengthen the reporting process. Another health professions study, "Development of Integrated Requirements for PAs, NPs, CNMs, and Physicians (MDs and DOs)," has produced a model for estimating the national demands for primary care practitioners by year through 2020. HRSA advisory groups (the Council on Graduate Medical Education and the National Advisory Council on Nursing Education and Practice) are using the model to form recommendations for data and education program development. Another external user is the Utah State Health Department, which is adapting the model to project State-level needs.
A third example is a study that describes factors influencing decisions about the numbers and types of primary care personnel used by health maintenance organizations. The growth of managed care is altering the ways in which members of the health care work force are used and thus is modifying the demand for, and the costs of, various groups of health care professionals. Understanding how primary care staffing decisions are made in managed care organizations will improve health professions' forecasts by helping determine how to adjust to the influence of managed care. The study also investigates ways in which managers alter their primary care staffing patterns to accommodate the needs of Medicaid enrollees. It has given an initial insight into changes in the demand for primary care personnel that will accelerate as more States move into managed Medicaid.
Major priority is placed on improving access to high-quality services for prevention and treatment of HIV/AIDS. A study of Ryan White CARE Act Title I assessed grantee strategies used to reach African-Americans not in care in four metropolitan areas. Among the strategies described in the report are establishing and maintaining broad planning council representation from all populations affected by epidemic; enhancing the capacity of community-based agencies to successfully compete for local funding; adapting case management systems to changing priorities; recruiting African-American physicians as clinicians in community-oriented primary care; and building community awareness and participation in planning through African-American institutions, such as communities of faith. Two key findings of the report were that the methods used here also could be used to assess strategies for other underserved populations and that developing effective strategies requires the grantee, the planning body, and constituents of services to recognize that each acts on distinctive underlying values and principles.
Another study, "Impact of Ryan White CARE Act Title I on Capacity Building in Latino Community-Based Organizations," developed and pilot tested a methodology for evaluating whether Title I funds have influenced the development, expansion, or enhancement of HIV/AIDS services in San Diego and Boston. (This methodology also can be used to study capacity building in organizations serving other populations.)
A careful analysis was conducted of the cost-effectiveness of the Community Health Centers (CHCs) Program. It showed that AFDC recipients who used a CHC as their main source of primary care incurred lower Medicaid costs and used fewer hospital days than other recipients living in the service areas of selected CHCs in California and New York. Study findings, which reflect only care reimbursed on a fee-for-service basis, have been used in presentations to the Office of Personnel Management and Congress. Building on this study, HRSA has contracted for a national evaluation of the effectiveness and impact of CHCs, including examination of the experience of centers involved in managed care.
Representatives of the Substance Abuse and Mental Health Services Administration (SAMHSA) participated in planning and oversight of an additional CHC study, "Community and Migrant Health Centers and the Assessment of and Response to Mental Health and Developmental Needs in Primary Care Patients." The final report has been distributed to State primary care organizations (which administer the primary care cooperative agreements with States) and to primary care associations for use in advising member organizations on improving their mental health services.
Essential to achieving more effective program evaluation is the production of useful data. To help produce this data, a third primary care project developed a Uniform Data System (UDS) covering five programs, including Community and Migrant Health Centers and Health Care for the Homeless, and a user manual for the UDS. This new system provides uniformly defined data for related programs and eliminates duplication and inconsistency in reporting, thus reducing grantee burden.
Also in the services area, a maternal and child health study assessed the outcomes of grants that were intended to foster creation of a permanent infrastructure for child and adolescent injury prevention in seven States. This study, which analyzed and compared the State strategies, found that recipients in general exhibited a considerable increase in injury prevention capacity several years after the expiration of the special incentive grant funding. Results will be used by policymakers in considering future funding for injury prevention and will influence the design of such efforts.
Ongoing studies include a range of projects concerned with enhancing performance measurement capacity and assessing the effectiveness or implementation of specific programs. Illustrative studies in particular topic areas follow.
In the AIDS area, HRSA is continuing a longitudinal examination of the effects of Ryan White CARE Act Title I funding on services for active or recovering drug users with HIV. Staff of the National Institute of Drug Abuse are collaborating in the design and oversight of this study. Another AIDS-related project provides a synthesis of local evaluations sponsored by Ryan White grantees to facilitate dissemination nationwide of findings and experience with various methodologies as applied to locally identified issues.
A study concerning organ transplantation, titled "Reasons African-American and White Waiting List Patients Are Unavailable for an Organ Offer," reflects HRSA's concern with cultural competence in health service delivery. An Inspector General's report prepared in 1991 showed that African-American patients on the waiting list for a kidney transplant waited nearly twice as long as did Caucasian patients. Only part of this difference can be explained by biological and medical factors. HRSA, through the Organ Procurement and Transplantation Network, is responsible for ensuring that the U.S. organ allocation system operates equitably. This current study will help determine whether changes in Federal policy governing the organ procurement system are needed.
Continued major attention is directed toward community-based service programs. "Effectiveness of the National Health Service Corps" is a 3-year study to assess the Corps' performance by using such indicators as satisfaction of communities where Corps members are working; increases in numbers of people served because of placement of a Corps member in a site; and long-term retention of former Corps assignees in a primary care or related profession.
As noted, HRSA is conducting a national study of the effectiveness and impact of Community and Migrant Health Centers through a sample of 50 centers in 10 States. The data will be analyzed separately for users served under managed care arrangements. The "Community Health Center User and Visit Survey" involves interviews with 2,000 users and a review of 3,000 medical records for visits at the same centers to provide information about the demographics of CHC users, their reasons for seeking care, their diagnoses, services used, and outcomes of care. Another study is assessing the impact of Medicaid waivers on HRSA-funded, federally qualified health centers in States that have instituted mandatory managed care for Medicaid beneficiaries.
In addition, "The Future of Primary Care" was intended to define the place of primary care in the changing health care environment and to develop normative goals for primary care. Results will be used to assess the adequacy of the primary care system in the United States and strategies needed to influence the supply and distribution of primary care providers.
An emerging policy issue concerns the management of the J-1 visa program, a matter that cuts across health professions and primary care issues, therefore calling for a collaborative effort with the Educational Commission for Foreign Medical Graduates and the Appalachian Regional Commission. This project, "Tracking of J-1 Visa Exchange Students," is examining the postresidency experience of exchange students who have secured waivers (to the requirement to return home for 2 years following completion of residency) to remain in the United States. Experience of those remaining will be compared with experience of those who return to their home country. Results, expected in 1997, will provide information for guiding U.S. policy about such waivers for physicians.
The "National Evaluation of the Healthy Start Program" continues as a 5-year longitudinal study of the development, implementation, and outcomes of comprehensive, coordinated perinatal care systems in the initial 15 Healthy Start demonstration sites. This project, the largest study funded by HRSA, includes an assessment of changes in the health status of pregnant women and infants across the sites. The study is designed to answer four questions: Did the Healthy Start initiative succeed? If so, why? If not, why not? What would be required for a similar intervention to succeed in other settings?
Finally, with the continued and projected emphasis on technology, HRSA is asking whether investment in telemedicine improves the availability and quality of care to underserved populations and provides easier access to continuing education and consultation for providers in isolated settings. "Rural Applications of Telemedicine" is constructing a broad base of knowledge about telemedicine upon which further assessment of the HRSA telemedicine grant program will be built. The four main objectives are (1) to determine the current status of telemedicine in rural health; (2) to explore the effects of telemedicine on access to care, practitioner isolation, and the development of health care networks; (3) to explore the organizational factors that aid or impede the successful development and implementation of telemedicine systems; and (4) to develop, test, and refine data-collection instruments that can be used in subsequent evaluation efforts. Representatives of several other Federal departments and agencies are participating in the conduct of this study.
Major evaluation priorities in fiscal 1996 include managed care; such primary care programs as Community and Migrant Health Centers and the National Health Service Corps; care for mothers and children, exemplified by the Healthy Start initiative; HIV/AIDS services, including new approaches to delivering and financing services through the Special Projects of Regional and National Significance; health professions efforts to foster community-based training for primary care practitioners; and strengthening of the health care infrastructure at the community level, partly through collaboration with States and external organizations such as the American Hospital Association. Underlying the work in all of these topical areas will be continuing efforts to further strengthen HRSA's performance measurement capacity through projects targeted to the needs of particular components and programs and to the integration of performance management approaches agencywide in the context of budget decisionmaking and strategic planning. Finally, HRSA will continue efforts to broaden the dissemination of evaluation results and information about ongoing studies to the public health community and to Congress.
MISSION: To provide a quality health services delivery system for American Indians and Alaska Natives with opportunity for maximum tribal involvement in developing and managing programs to meet health needs.
The goal of IHS is to raise the health status of its principal beneficiary, American Indians and Alaska Natives to the highest level possible. The importance of evaluation in supporting this goal has increased significantly in recent years. The IHS evaluation process seeks to include American Indians and Alaska Natives as primary stakeholders in defining the purpose, design, and execution of evaluations. Stakeholders are the users of the end product of evaluations and typically are the population or groups most likely to be affected by evaluation findings. The principles of responsive evaluation practice have been adopted by IHS because they can best address these needs and concerns of American Indians and Alaska Natives.
The purpose of the Office of Planning, Evaluation, and Legislation (OPEL) is to advise the Director of IHS on policy formulation; conduct and manage program planning, operations research, program evaluation, health services research, legislative affairs, and programs statistics; develop the long- range program and financial plan for IHS in collaboration with appropriate agency staff; coordinate with HHS, Indian tribes, and organizations on matters that involve planning, evaluation, research, and legislation; and develop and implement long-range goals, objectives, and priorities for all activities related to resource requirements and allocation methodologies and models. OPEL serves as the principal advisory office to IHS on issues of national health policy. It also coordinates these four evaluation functions:
OPEL meets part of IHS evaluation needs with two types of short- term studies: policy or program assessments and evaluation studies. The policy study contributes to IHS decisionmaking about budget, legislation, and program modifications and includes background information to support IHS initiatives. Evaluation studies are carried out at the program level, or area offices, and focus on specific program needs.
Annually, OPEL identifies the high-priority health care and health management issues and concerns through the submission of headquarters and area office proposals for assessment or evaluation. IHS area and associate directors are asked to submit proposals for possible areas of evaluation study. These proposals are reviewed and rated by a panel of subject matter experts and evaluation experts. They are also reviewed by IHS staff for more specific concurrence with IHS strategic goals, objectives, and priority areas. The proposals are then prioritized and forwarded to the OPEL Associate Director for review and approval. The Director of IHS reviews the final proposals and decides the respective funding levels.
During fiscal 1995, OPEL completed four evaluation projects of several major Indian health topics.
"Case Study of Family Violence in Four Native American Communities: Final Report." Family violence on Indian reservations is devastating for individuals, families, and reservation communities. There are many families in American Indian communities who have experienced violent behaviors, who have coped with violent behaviors positively, or who wish to learn more about violent behaviors and their prevention. IHS sponsored this study to produce the information and data needed to guide program planning and development.
"Evaluation of Diabetes Services Provided by IHS Model Diabetes Program: Final Report." This evaluation of the IHS diabetes model projects used data from 634 patient medical records, four focus groups, and 20 informant interviews to describe these projects and examine their effect on two patient health outcomes--blood sugar control and hospitalizations. Two diabetes project sites (Winnebago, Nebraska, and Fort Totten, North Dakota) and one "usual care" site (Rosebud, South Dakota) were selected for the evaluation. Data from the 1993 Diabetes Program Audit were used to assess whether or not the findings from this evaluation of two diabetes projects could generally represent other diabetes team approaches in other IHS areas.
"Evaluation of IHS Midlevel Health Providers: Final Report." IHS must determine its needs for midlevel health providers (MLHPs), such as physician assistants, nurse practitioners, certified nurse midwives, and clinical nurse specialists through the year 2000 and address recruitment and retention of MLHPs. This study resulted in obtaining responses from 119 MLHPs and 14 primary care managers. The survey confirmed that MLHPs are making a major contribution to the IHS primary care program and are well utilized. However, IHS must begin immediately to address the shortfall of MLHPs that is projected to be approximately 51 percent over the next 6 years.
"Phase III Final Report: Child Abuse (CA) and Child Neglect (CN) in American Indian and Alaska Native Communities and the Role of the Indian Health Service." Indian child and adolescent abuse and neglect are issues of widespread concern; however, no reliable statistics exist on the prevalence of abuse or neglect. Recent data have indicated that more than 6,500 referrals for suspected child abuse and neglect were made to the Bureau of Indian Affairs (BIA), reflecting a minimum of 1 percent of Indian children in the BIA service area. This study provided comprehensive assessments of the effectiveness of IHS and tribal policies, procedures, and personnel in recognizing and treating CA/CN, and facilitated the design of an intervention program flexible enough to be used by American Indian and Alaska Native communities across the country.
During fiscal 1995, IHS funded more than 10 studies, including health program evaluations, policy analysis projects, health services research, and special studies and initiatives. Examples of these projects follow.
"Analyzing the Underreporting of American Indian and Alaska Native Deceased Persons on State Death Certificates, 1986- 1988." The fiscal 1990 phase of the project funded the matching of the IHS patient registration records with State death certificate records. The records of American Indians and Alaska Natives matched with certificates from the National Death Index (NDI) for 1986, 1987, and/or 1988. The output was a computer tape and hard copy printout of probable and possible matched records. The results from the NDI match are used to develop estimates for each IHS area and each Reservation State on the number and proportion of deaths occurring each year that are misreported by race. These data will indicate which States have significant problems in underreporting American Indians and Alaska Natives on death certificates.
"A Mental Health Service Delivery Model for Urban Native Americans: An Evaluation of Utilization Rates and Mental Health Treatment Factors in an Urban Setting." This study will investigate utilization rates for mental health services over a 9-year period in an urban Indian population so that a profile of service use can be assessed over time. The approach will include the use of historical time analysis to examine the patterns of use over a set period within the context of events taking place during the those years and the effect on the population being served. The study will also examine the interaction of a service delivery model with traditional and western approaches within the same operational framework.
"Evaluation of the IHS Adolescent Regional Treatment Centers." This study evaluates the effectiveness of the regional treatment centers (RTCs) that provide alcoholism rehabilitation for American Indian and Alaska Native youth. The study will define the issues facing RTCs and help establish the parameters of what these programs have accomplished and plan to accomplish over the next 10 years. It will also provide guidance on how the success rate of RTCs can be improved.
"Evaluation of the IHS-Supported Substance Abuse Treatment Program for American Indian/Alaska Native Women." This study assesses the current IHS substance abuse and treatment program. It seeks to improve program effectiveness by monitoring behaviors that enhance risk, because research is virtually nonexistent on the scope of the problem of alcohol and other substance use among American Indian and Alaska Native women. The study will also evaluate the effectiveness of treatment.
IHS is responding to dramatic changes taking place inside and outside the government. The causes of the many changes include a decrease in the funding level of discretionary Federal programs, greater involvement of tribal governments in the Indian health care system, technological innovations, the changing patterns of disease to more chronic conditions, and the transfer of many Federal programs and resources to individual States.
These changes will affect the IHS evaluation strategy in the coming years. Nevertheless, IHS remains committed to comprehensive community-based, preventive, and culturally sensitive projects that empower tribes and communities to overcome health issues. The Director of IHS has placed increased emphasis on several initiatives to focus attention on specific health areas and to serve as a management tool to prioritize IHS' workload. These initiatives include emphasis on women's health, youth, traditional medicine, and elder care and on establishing working relationships with Federal and State government agencies. The initiatives will undoubtedly affect new directions for evaluation.
Research and evaluation proposals to be considered in upcoming years include the following topics: childhood obesity; elderly wellness; program review of training the practitioner in the assessment and treatment of adolescent sexual perpetrators; impact of the Alaska tribal health compact on programs and services in the Anchorage Service Unit; Pueblo of Zuni end-stage renal disease quality of life; and village-based women's preventive health services delivered by community health aides/practitioners.
MISSION: To sponsor and conduct medical research that leads to better health for all Americans.
NIH develops scientific knowledge that leads to improved means to prevent illness, cure disease, and treat disability. It accomplishes its mission by conducting medical research in its own intramural laboratories and supporting research in universities, medical and health professional schools, hospitals, and other health research organizations. NIH fosters the widespread dissemination of the results of medical research, facilitates the training of research investigators, and ensures the viability of the research infrastructure. The NIH Evaluation Program is an integral part of how NIH manages its support and conduct of medical research.
NIH evaluation activities assess program performance (efficiency, effectiveness, responsiveness); analyze both quantitative and qualitative results based on those assessments; and use the resulting information in policymaking, strategic planning, budgeting, and program development and management.
The quantity and diversity of diseases, disorders, and biological systems in the NIH portfolio make strategic planning and evaluation at NIH a complex task that is continuously evolving. Priorities are set and research programs are planned and evaluated at two levels: the institutes and centers (ICs), and centrally by the NIH Director, with whom the ultimate responsibility lies for the course of NIH-funded medical research.
This two-level approach ensures that planning and priorities specific to the mission of each IC are fully developed and implemented with a clear vision and within the fiscal constraints set by the IC budget, and that there is central leadership for developing crosscutting initiatives and promoting collaboration among the ICs. The NIH Evaluation Program provides information to assist the NIH Director and IC directors in determining whether NIH goals and objectives are being achieved and to help guide policy development and program direction.
A distinguishing feature of the NIH Evaluation Program is the extent to which it employs a variety of evaluation strategies and inputs that include the use of national advisory councils, boards of scientific counselors, consensus development conferences, and ad hoc committees that help chart scientific directions and select the most promising research.
Program evaluations are funded through both a trans-NIH mechanism, the 1 percent set-aside authority, and by individual IC program funds. A two-tier system reviews project requests to use set-aside funding. The first tier involves review and recommendations by the NIH Technical Merit Review Committee (TMRC) on the technical aspects of project proposals and on whether a project fits within HHS guidelines for set-aside funding. The second tier involves the NIH Evaluation Policy Oversight Committee (EPOC) that considers TMRC recommendations and makes final funding recommendations to the NIH Director or designee. It also conducts policy-level concept reviews of proposals for NIH-wide evaluation studies, establishes the overall NIH set-aside budget, and oversees the process. EPOC recommendations are approved by the NIH Director or designee. Concurrently, the ICs fund program evaluations from their budgets; these evaluations are used by various committees, working groups, task forces, workshops, conferences, and symposia for program management and development.
An important characteristic of NIH's Evaluation Program is solicitation from a number of sources to provide information to the NIH Director and the IC directors. Discussions are continuously held with extramural grantees, intramural investigators and other NIH staff, members of Congress and the Administration, and members of the public, including professional societies and voluntary health organizations. These individuals and groups provide valuable input on pressing public health needs, important scientific opportunities, knowledge gaps, and the balance between patient-oriented and laboratory research.
The evaluations completed in fiscal 1995 addressed a cross- section of the NIH research program, as demonstrated by the following sample of studies.
"Fetal Alcohol Syndrome: Diagnosis, Epidemiology, Prevention, and Treatment" responded to the congressional mandate that a study of fetal alcohol syndrome (FAS) and related birth defects be carried out. A complete description of this study, funded by the National Institute of Alcohol Abuse and Alcoholism, is in chapter II of this report.
"Directions in Nursing Research Training" summarizes information on National Institute of Nursing Research (NINR) (1) funding that permits nurse scientists to follow research career paths; (2) recommendations for nursing research training in a National Research Council report; and (3) the views and recommendations of the scientific community. The report also discusses research training issues in the broad context of Federal and academic perspectives. The report recommends that NINR disseminate the report widely, analyze the research career paths of NINR trainees and fellows to determine the criteria for successful training experiences, emphasize the need to increase the number of nurses with doctoral degrees, and support research consortia and partners.
"Cancer at a Crossroads: A Report to Congress for the Nation" describes the results of an evaluation undertaken at the request of Congress to assess the achievements of the National Cancer Program, identify barriers to reducing the burden of cancer, and recommend future research and program directions. A subcommittee appointed to address these issues concluded that the strongest strategy for a renewed "war on cancer" should (1) apply currently available knowledge about cancer prevention and care to all segments of the population; (2) increase support for transnational research that develops basic cancer knowledge into preventive strategies, new technologies, and effective treatments; and (3) increase support for basic cancer research to ensure the continued flow of new discoveries that lead to better cancer prevention and care.
"Measuring Social Inequalities in Health" provides the results of a National Institute of Child Health Development conference held to develop recommendations for improving or changing measures of socioeconomic gradients in federally supported health data sets. The conferees recommended (1) the availability of existing social class data be publicized, and (2) researchers be encouraged to analyze the data. The conference also provided investigators wishing to apply for NIH funds with improved tools for implementing the revised guidelines for the inclusion of women and minorities in clinical research.
"Report on Outreach Activities of the National Library of Medicine" details a 5-year review of National Library of Medicine (NLM) outreach activities in response to congressional concerns that hospitals and health professionals in isolated areas lack access to recent scientific and biomedical information. The report found that (1) the number of outreach projects had increased from 16 in 1989 to almost 300 at the end of 1994; (2) approximately 30,000 user codes issued in 1989 had increased to nearly 100,000 by 1994; (3) 4 million searches of the NLM databases in 1989 increased to 7 million searches in 1994; and (4) similar changes have occurred in the use of the "Grateful Med" system.
"Support for Bioengineering Research" responded to a request from Congress that the HHS Secretary, acting through the NIH Director, conduct a detailed inventory of sources and amounts of public and private funding for basic bioengineering research in fiscal 1993. The report recommended (1) establishing an Interagency Bioengineering Coordinating Committee; (2) including basic bioengineering research within appropriate intramural programs; and (3) providing, through the Federal Register, a "comment period" notice to suggest research topics for inclusion in the annual Small Business Innovation Research Omnibus solicitation.
NIH currently supports more than 30 evaluations. They range from small- to large-scale assessments, from evaluative studies to comprehensive evaluations. Examples illustrating this range include the following:
"Evaluation of the National Research Service Award (NRSA) Research Training Program" is the first phase of a longer term evaluation effort directed at examining the extent to which the objectives of NIH/NRSA programs are being met. It is an update and extension of earlier work performed by the National Academy of Sciences published in a 1984 report titled "Career Achievements of NIH Predoctoral Trainees and Fellows." The study will address the major evaluation questions frequently posed by constituencies of NIH research training; make efficient use of extant data relevant to these evaluation questions; and identify gaps in existing databases, thus guiding the development of the data-collection efforts planned for the second and third stages of the evaluation.
"Implementation Phase for an Evaluation of the Minority Access to Research Careers (MARC) Honors Undergraduate Research Training Program" will evaluate the degree to which the intended goals of the MARC undergraduate program are being met and will result in the establishment of a general information database on the program and its trainees. Information is being collected on the programs implemented at MARC institutions, the trainees, the training experience, and former trainees' subsequent education and career paths.
"Evaluation of NIH Implementation of Section 491 of the PHS Act Mandating a Program of Protection of Research Subjects" is examining NIH's implementation of a program to ensure adequate protection to all individuals who are used as research subjects. The study is attempting (1) to determine the impact of the program on the administrative operations and research activities of awardee institutions and (2) to examine information on some financial costs, administrative burden, levels of effort, and selected program measures related to the protection of human subjects, the education of researchers, and the facilitation of research.
The Task Force on Genetic Testing is examining issues surrounding the use and regulation of genetic tests. The objectives of the task force are to review the state of the art of genetic testing; examine the strengths and weaknesses of current practices and policies relating to testing; and, if needed, recommend changes or policy options to ensure that the public is protected so that only the appropriate tests are done by qualified laboratories.
The study, "Methodology To Assess the Impact of National Heart, Lung and Blood Institute (NHLBI) Research," is examining alternative measures to demonstrate the impact of basic research in terms of reductions in disease mortality and morbidity. The objectives of this study are to introduce the concept of examining patients as an additional measure for assessing the impact of NIH-supported research; to develop a methodology to use patients in evaluation research; and to apply the methodology to an NHLBI project to compare the outcomes for individual research project grants funded under requests for applications with individual research grants funded from unsolicited applications.
The study, "Characterization of User Population and User Satisfaction of the Physician Data Query (PDQ) Database," is being conducted to obtain information on PDQ database users and their satisfaction with the comprehensive cancer database. The objectives of the study are to survey PDQ database users to characterize who is using the database, and to determine how the information is used and if users are satisfied with the information they receive and with the method of retrieval (e.g., CD-ROM, online, or hard copy).
The "Evaluation of Laboratory Animal Use, Facilities, and Resources" will acquire, analyze, aggregate, and report objective data for fiscal 1993 on the number and characteristics of animals used and on animal-related facilities and resources at those institutions and organizations that have animal welfare assurances currently on file with the NIH Office of Protection From Research Risks.
The "Survey of Academic Research Equipment" responds to a congressional directive under Public Law 96-44, Section 7, that the National Science Foundation, in cooperation with NIH, conducts an ongoing instrumentation survey and issue biennial reports. The purpose of the study is to assess national trends in medical research instrumentation.
NIH will continue to engage in a wide variety of evaluation activities. Priorities for future evaluation activities reflect input from a number of sources: the HHS Strategic Plan, the NIH Strategic Plan, and the NIH GPRA Performance Plan--all in the developmental phase; the NIH Director's Areas of Emphasis; the Administration's High Priority Areas; recommendations of NIH/IC directors; and recommendations of the Evaluation Policy and Oversight Committee. As a result, NIH will give priority funding to the following areas:
NIH evaluation studies provide a rational basis for managerial decisionmaking and responding to public concerns for accountability in government. In addition, it is through such studies that NIH is able to determine its progress in meeting scientific objectives, strengthening research and administrative activities, and contributing to its mission of sponsoring and conducting research that leads to better health for all Americans.
MISSION: To provide analytical support and advice to the Secretary on policy development and assist the Secretary with the development and coordination of HHS-wide program planning and evaluation activities.
ASPE functions as a principal advisor to the Secretary on policy development and conducts a variety of evaluation and policy research studies on issues of national importance. ASPE is also responsible for HHS-wide coordination of legislative, planning, and evaluation activities. In its evaluation coordination role, ASPE does the following:
Through the departmental evaluation planning process, ASPE has the capacity to identify crosscutting program or policy issues of particular concern to the Secretary and specific program and policy areas not covered by HHS agency evaluation plans. In these instances, ASPE initiates evaluations or collaborates with the agencies to conduct evaluations or policy assessments. For example, in recent years ASPE has initiated projects to develop cost estimates for health financing issues in general and specifically for Medicare and Medicaid programs; the effects of managed care expansion on public health infrastructure; welfare- to-work approaches; long-term care alternatives; and studies evaluating alternative services for children at risk of harm from drug abuse, crime, abuse, and other pathologies.
Another continuing ASPE evaluation objective is to support and promote the development and improvement of databases that HHS agencies and ASPE use to evaluate health care programs and health trends. ASPE provides support to the HHS Data Council charged with integrating key national surveys, such as linking health status indicators with indicators of well-being. HHS needs more comprehensive data sources to assess the anticipated transformation in health and human services.
Finally, ASPE uses evaluation funds to promote the effective use of evaluation-generated information in program management and policymaking. The latter is accomplished through the dissemination of evaluation findings and other activities such as providing technical assistance to agencies in the development of performance measures.
During 1995, ASPE completed 28 studies and reports on a number of issues that provided information useful to the Secretary and HHS divisions for purposes of program planning and budget and legislative development. A description of some of those completed studies that have broad potential application follow.
Several studies were aimed at providing the Department with an expanded understanding of the causes, impacts, and possible ways to address the pervasive national triad of adolescent sexual activity, pregnancy, and teen parenthood. One study examined prevention and intervention strategies designed to reduce adolescent pregnancy and parenthood. A second report summarized recent research on adolescent sex, contraception, and childbearing. The report summarized the factors that lead to teenage childbearing. The varied antecedents of sexual activity include biological factors, race and gender, family characteristics, use of alcohol or drugs, and other behavioral factors. The studies documented the increased rate of sexual intercourse among teens. The earlier the age of sexual initiation, the more likely the experience is coercive and the more likely the teen will become pregnant or contract a sexually transmitted disease (STD). Another study analyzing the increases in nonmarital births found that one-third of all births occur outside of marriage, with teenagers accounting for about one-half of all nonmarital births; and a smaller proportion of nonmarital pregnancies are ended in abortion. Most parents in nonmarital births are disadvantaged before the birth.
As the pressure on expanding data-collection systems continues and the ability to store, manipulate, and transmit such growing amounts of information accelerate, it becomes more important to protect the privacy of this information. ASPE supported a task force on the privacy of private sector health records. The task force examined the extent of the problems with the collection, storage, and use of health information in the private sector. The task force also examined the social, legal, and economic issues affecting the privacy of people who use the health care system. The report recommended a coordinated Federal policy on medical records, confidentiality, universal identifiers, effective security standards and guidance, the establishment of a data- protection entity, and an education program about the issue.
Substance abuse problems blight the lives of individuals served by a number of HHS programs. It is therefore important to increase understanding of the etiology of substance abuse and of effective modes of intervention. One study looked at patterns of substance use and substance-related impairment among recipients of Aid for Families with Dependent Children (AFDC). It found that while AFDC recipients have higher rates of substance use and substance-related impairments than the general population, the vast majority of substance users and impaired people do not receive AFDC. A second study of substance abuse among women and parents also examined the number of children potentially at risk because of parental drug abuse. The study also found that children of drug-using parents tend to be younger than children overall and women and men with young children in their homes tend to report one-half as much illicit drug use as those with no young children at home.
Another area of significant concern and interest is the movement of individuals from various dependent arrangements, including welfare, into employment. Three complementary studies were carried out to learn more about barriers to and models for facilitating transition of disabled individuals to work.
One study reviewed literature on barriers and incentives to improving the labor force participation of persons with significant disabilities. The second study evaluated the methods used in transition-to-work demonstration projects aimed at helping schools and adult providers obtain integrated employment, using natural supports for students with very severe disabilities. The report details the barriers encountered by the demonstration sites, such as difficulty locating jobs, difficult funding procedures, transportation problems, and staff turnover.
A third study examined approaches States have taken to move significant numbers of welfare recipients into work-related activities, with the goal of identifying operational lessons for program administrators and policymakers. The report documented strategies States used to increase participation in work, the role of child care in achieving this objective, and how States changed the culture of welfare to have a stronger focus on employment.
ASPE currently has 26 studies in progress on a broad range of program and policy areas. The major areas of study include the following:
ASPE has commissioned a national study of assisted living, or the residential settings that combine adapted housing, assistive technologies, personal assistance, and other supportive services for persons with disabilities. Assisted living is considered an important component of services to the increasingly aging American population and an effective response to the rising costs of nursing home care. The study will examine the role of assisted living from the perspective of consumers, owners and operators, workers, regulators, developers, investors, and others with a stake in the long-term care system.
As many as 1 million mostly elderly or disabled individuals are dependent on personal services and supervision provided by board and care homes. To examine one significant way in which public agencies seek to influence the care provided in these facilities, ASPE is carrying out an analysis of the effect of regulations on the quality of care provided in board and care homes. In addition to raising concerns about the unwillingness or inability of most homes to meet changing resident needs and inadequate staffing ratios, the study's preliminary findings show that extensive regulation did reduce the use of psychotropic drugs and increases operator training and the availability of social aids and supportive devices. However, regulation does not affect operators' requirements for staff training, the availability of licensed nurses, or the cleanliness of the home.
Two studies are looking at the costs of domestic violence to the health care system and the domestic violence policy and programs of selected communities. Together, the studies will provide information about the economic consequences of costs of domestic violence to the health care system and will develop an economic model for determining these costs. The studies will also examine how selected communities have built community-based comprehensive family violence programs.
ASPE and the Administration for Children and Families (ACF) are evaluating selected family preservation programs, including placement prevention services aimed at preventing children from entering substitute care, broader family preservation services that may be less intensive and of longer duration than placement prevention services, and reunification services to speed the return of children to their homes after entering substitute care. Measures of program success will include reduced placements of children into substitute care (for preplacement services), successful reunification (for reunification services), improved child psychological well-being, improved child behavior, improved family functioning, and reduced recurrences of child abuse and neglect.
ASPE is preparing current information on the demand for technology assessment. The study will examine the performers of technological assessment, the methods of conducting assessments, the uses of the results, and the unmet needs that might be met by further cooperation between the public and private sectors of health care.
The purpose of this study is to understand the relationship between the increasing amounts of information that are being produced for physicians and the types of information that physicians actually use. In particular, this study is assessing physician use of available computerized health and medical information sources, such as those on the World Wide Web and the Internet, and how access to such information influences medical practice.
It is difficult to estimate how often people who enter nursing homes down their assets and become eligible for Medicaid. Because some elderly people enter nursing homes more than once, a longitudinal study to help provide this information is being conducted.
Managed care can provide unique opportunities but also has potential pitfalls for people with disabilities. Research is under way to determine the impact of managed care on access, quality, and satisfaction for people of all ages who experience disabilities. Of particular interest are the experiences of children with disabilities in managed care, best practices in managed care for the disabled, and the impact of extending Medicaid managed care to the disabled population eligible for Supplemental Security Income (SSI).
HHS, with ASPE's assistance, is conducting a comprehensive, multiyear study of the government's principal program for moving people off welfare and into employment--the Job Opportunities and Basic Skills Training (JOBS) program. The evaluation addresses the long-term effects of different welfare reform approaches including whether any approach significantly improves the economic circumstances of the most disadvantaged recipients, the effects of welfare-to-work programs on the children of welfare recipients, and the cost effectiveness of different approaches.
The Federal grants relationships to States in public health are evolving into outcomes-based performance management. A health outcomes-based monitoring approach requires data system development, and several projects are being conducted for that purpose. One project, with the National Academy of Sciences, will work on identifying which results of a performance-based approach can be measured at the State and Federal level and will recommend specific steps that can be taken to improve these measurement capabilities. A related project will develop approaches to obtaining comprehensive baseline and trend data on public health infrastructure. Finally, a third study, building on a Robert Wood Johnson Foundation project, will assess the quality of data that States collect and determine whether a network can be established to share health data among the States and thus improve the health policy decisions they make.
The growing influence of managed care has had a profound influence on the pharmaceutical marketplace and has highlighted a number of concerns among pharmaceutical companies, third-party payers, and the Federal and State governments. ASPE is engaged in a project to develop a framework to assess the impact of managed care on the pharmaceutical marketplace, as well as on consumer access to newly developed drugs.
Three studies are looking at various aspects of father involvement, especially among fathers who are not living with their children. One study examines the relationship between various child custody, visitation, and support payment patterns and the effect that payment and contact have on child well-being. The second is developing a design for the evaluation of community-based programs for vulnerable fathers. Although programs to increase and enhance father involvement exist in some communities, very few have been evaluated. The third study will develop a theoretical framework to assist in understanding and implementing programs and activities to promote responsible fathering. The last two projects are being conducted in collaboration with ACF.
Subacute care is considered a cost-effective alternative to acute nursing home care services, that is, it can be provided in lower cost settings with no diminution of quality. ASPE has commissioned a study to identify and understand the definitions of subacute care; examine the provision of subacute care in select market areas by a variety of providers (e.g., hospital- based and freestanding skilled nursing facilities, rehabilitation and long-term care hospitals, and home health agencies); evaluate policy issues about who receives, provides, and pays for subacute care; and assess the cost, quality, and cost-effectiveness of this type of care.
In fiscal 1995, ASPE provided funds for creating the first annual report on the health and well-being of America's children. The report will be a single volume that shows up-to-date trends on how our Nation's children are faring, pulling together information on all available national trends in the lives of children and youth. The volume will fill a crucial gap and will become an annual series to keep the Nation apprised of the well- being of its children and youth.
Extensive new legislation, enacted during the current 104th Congress, will lead to the consideration of a range of new evaluations. First, understanding the effects of profound structural changes taking place in health care will be a majority priority for ASPE evaluation activity. The objective is to understand the changes on health status of families and individuals, access to quality health care, patterns of health care utilization and spending. Included is the need to evaluate the impact of Medical Savings Accounts on spending and savings and the connection between levels of income and amounts of savings under this option. A continuing and important evaluation objective is to support and promote the development and improvement of databases that ASPE and others can use to conduct evaluations of health care and human services programs and health and social trends.
Second, the welfare reform legislation also will require a varied evaluation response in order to assess the success of meeting the objectives of this legislation as well as to gather effective objective information on impacts on current and future recipients. It will also be critical to document and assess the impact of the recently enacted welfare legislation. The evaluation activities will focus on understanding the operation and organization of the new welfare system and assessing the impact of the changes on low-income families and children.
In other areas, ASPE will examine long-term initiatives that focus on the development and implementation of systems of acute care, subacute care, assisted living, long-term care, and personal assistance services for people with disabilities. ASPE will continue to develop effective indicators of the well-being of children in order to measure changes in the conditions of our children. ASPE will support the Departments' efforts to develop outcomes-based performance measures for health and social service programs, intended to provide the framework for new types of grant relationships with States.
Finally, one of ASPE's continuing and important evaluation objectives is to support and promote the development and improvement of databases that ASPE and others can use to conduct evaluations of health care and human services programs and health and social trends. All of the above activities will be supported with a combination of program appropriations, policy research, and evaluation funds.
MISSION: To provide advice on public health and science to the Secretary of HHS, executive direction to program offices within OPHS, and, at the direction of the Secretary, coordinate crosscutting public health and science initiatives in the Department.
OPHS provides advice, policy and program coordination, and leadership in the implementation, management, and development of activities related to public health and science, as directed by the Secretary of HHS. OPHS provides advice to ensure that the Department conducts broad-based public health assessments designed to better define public health problems and solutions to those problems. It assists other components of HHS in anticipating future public health issues and problems and provides assistance to ensure that HHS designs and implements appropriate approaches, interventions, and evaluations that will maintain, sustain, and improve the health of the Nation. OPHS provides a focus for leadership on matters including recommendations for policy on population-based public health and science and, at the Secretary's direction, leads and coordinates initiatives that cut across agencies and operating divisions. In addition, OPHS communicates and interacts, on behalf of the Secretary, with national professional and constituency organizations on matters of public health and science.
In keeping with its newly defined role within HHS, OPHS has specifically sought to develop an evaluation plan that avoids duplication of efforts that might more appropriately and effectively be undertaken by the operating divisions of the Department or by the Office of the Assistant Secretary for Planning and Evaluation. Thus, the direction of the fiscal 1996 evaluation strategy for OPHS is toward public health and science issues that cut across multiple interests of the operating divisions and that require a coordinated approach to achieve the most effective results. In addition, OPHS commits itself to carrying out every project proposed as part of this year's strategy in collaboration with relevant operating divisions.
OPHS specifically conducts evaluations requested of the Secretary by Congress that are most appropriately managed by staff with medical or health sciences credentials and that cannot be assigned to an operating division. Also, evaluations are conducted to support the Assistant Secretary for Health's role as senior advisor to the Secretary on matters of public health and science. This role includes the responsibility to "assist the Secretary in developing a policy agenda for the Department to address major population-based public health, prevention and science issues" and to provide "leadership and a focus for coordination of population-based health, clinical preventive services and science initiatives that cut across operating divisions." In addition, OPHS conducts evaluations specific to the needs of the program offices located within OPHS, such as Women's Health, Minority Health, Disease Prevention/Health Promotion, International Health, and Emergency Preparedness.
In fiscal 1995, OPHS's predecessor, the Office of the Assistant Secretary for Health (OASH), completed 12 evaluations in an effort to better inform policy decisions throughout the Public Health Service (PHS) as the health care sector went through a period of rapid change. Studies were in diverse areas of public health in an effort to strengthen the public health infrastructure, including primary health care, HIV/AIDS, work force training, women's health, and substance abuse. OASH efforts focused on issues that cut across PHS program areas, as follows.
In an effort to study and learn from past experiences, one study, performed by the Institute of Medicine (IoM), analyzed crisis decisionmaking by the Food and Drug Administration, the Centers for Disease Control and Prevention, and the National Institutes of Health in response to a threat to the blood supply that emerged in the early 1980s--HIV. These Federal entities, as well as the plasma fractionation industry, community blood banks, and other groups, have responsibilities to protect the supply of blood from communicable disease. The report concludes that the system did not deal well with blood safety issues, that strong leadership is required to counteract legal and competitive concerns that may inhibit effective agency action when a crisis is not well defined, and that agencies need to formulate a systematic approach to advisory committees and should not rely on the entities they regulate for data analysis or modeling. These lessons are being used by the Assistant Secretary for Health in a cross-departmental review on blood safety.
Another project, also performed by the IoM, examined current HHS standards for methadone programs and evaluated the effects of Federal regulations on the provision of methadone treatment services. Unlike other controlled substances, methadone use is controlled by a multitiered system of regulations, and Federal oversight of methadone programs is provided by FDA, the Drug Enforcement Administration, the National Institute on Drug Abuse, and the Substance Abuse and Mental Health Services Administration. The IoM report examines the underlying ideology of methadone regulations that societal risks of methadone outweigh its benefits to such an extent that its use must be extraordinarily regulated. The IoM committee concludes such an approach is not valid in the current environment, and the scope of Federal regulation should be scaled back to untie clinicians' hands in determining the best course of a patient's treatment.
A third report examined the clinical background for trials involving the experimental drug fialuridine (FIAU) and its parent drug fiacitabine to determine whether any rules governing the trial process should be changed, and if so, what burdens or costs these changes might place on future clinical trials. The study followed up on an NIH clinical drug trial in which several hepatitis B virus (HBV) outpatients who had been administered FIAU were hospitalized with liver failure; five patients died and two were saved only by liver transplantation. The report finds that the entire set of trials reviewed was an ethically sound clinical research project designed and carried out by highly competent investigators who frequently exceeded regulation requirements imposed by institutional review boards. The report discusses conclusions and recommendations issued by FDA and concurred with them without ascribing blame for the tragedy to any party.
Among projects in the Office of Disease Prevention and Health Promotion (ODPHP), one report obtained baseline data on the disease-prevention activities of 1,310 nurse practitioners as primary care providers and compared the findings to target goals from 17 objectives of Healthy People 2000: National Health Promotion and Disease Prevention Objectives. The 892 respondent reported spending 50 percent or more of their time providing primary care. In the areas of assessment services, screening and immunizations, cancer-screening services, obstetrics/gynecology, and family medicine, nurse practitioners generally met or exceeded the target percentages of the Healthy People 2000 objectives for selected services.
In an effort to document the relative investment in public health in the United States, nine States participated in another ODPHP project to develop a tool for collecting Federal, State, and local public health expenditures using the essential services of public health terminology. Data-collection instruments were distributed to State health officials who, in turn, coordinated the data collection for State and local public health, substance abuse, and environmental agencies. Mental health data collection was coordinated through the National Association of State Mental Health Program Directors. The report is intended to lay important groundwork for tracking these expenditures over time and across various agencies and levels of government.
An evaluation by the Office of Minority Health (OMH) was designed to document, describe, and assess the effectiveness of community- based projects that address HIV/AIDS prevention within minority communities. The study compares various community-based approaches to disease prevention in nine projects and will be used by OMH and other agencies to identify successful practices and common themes to apply as innovative community-based risk- reduction strategies when targeting minority communities and at- risk populations. The study found that viable linkages with community-based providers and social service agencies are critical to the provision of effective, coordinated, and comprehensive HIV/AIDS services in the community; that projects that build upon existing AIDS activities are better able to influence the local health care and social service agencies and are more likely to continue after Federal funding ends; and that projects must conduct and document internal formative/process evaluations.
Four OASH evaluation projects were concerned with the operation of programs in the PHS regions. One project evaluated the impact of factors that impede the implementation of breast and reproductive cancer prevention efforts at nine Community and Migrant Health Centers (C/MHCs) located in California. In reviewing screening, referral, diagnosis, treatment, and case management protocols, the report finds that the majority of C/MHCs have weaknesses in the areas of community education and outreach; designation of staff to perform specific duties; attention to ethnic or culturally specific barriers to health care; and overall lack of understanding about the importance of protocols. To correct these weaknesses, a Primary Care Effectiveness Review has been scheduled for each center.
After identifying a gap in reproductive health care information within the C/MHC setting, an interagency task force in PHS Region IX began an evaluation project to strengthen the regional public health information infrastructure. The study focused on a representative sample of women 20 to 44 years of age who received reproductive health care services from 1 of 36 study sites. Using a 220-item data-collection instrument designed to collect and analyze the frequency and extent of services provided, the study gives regional programs an information base that will help improve quality of care by disclosing demographic and utilization patterns of the typical client. The system provides immediate feedback on quality of care through documentation analysis; for example, preventive health screenings are highly documented, while services related to the report of abuse and violence are the most poorly documented. Also, prenatal documentation tends to be complete, and documentation of health promotion education and counseling appears low.
Copies of the PHS Region IX report were sent to all C/MHCs involved in the study and to executive and clinical staff in the region. Technical assistance in the areas of violence and abuse, reproductive health, and depression is being made available to the centers as part of a continuous quality-improvement plan. In addition, forms are being redesigned to improve staff use. A domestic violence task force has been organized in the region to address the issues raised in the report.
A report from Region IX assesses HBV education and outreach efforts among primary care centers serving Asians and Pacific Islanders, who constitute more than 50 percent of the infectious HBV carriers in the United States. The report summarizes programs at five centers, which completed and returned a self-assessment tool. It finds that program staffing is the strongest component of HBV education and outreach programs, and media-based outreach is the weakest. The report recommends sensitizing staff to cultural beliefs and attitudes impacting on HBV, using translated health education materials, integrating HBV activities with standard patient care protocols, using ethnic media, and developing linkages with school-based health programs and other community-based programs that help high-risk Asian/Pacific Islanders overcome the barriers identified by the study. The centers are now working to incorporate these findings into ongoing staff training and development.
From surveys mailed to nine C/MHCs in Region VI, another project assessed the level of interest in primary care, practice-based research. The report finds that significant barriers to research include lack of research skills and expertise, too many other clinical responsibilities, and lack of access to research consultation. The report recommended further evaluation of PHS and non-PHS clinic sites to determine topics of interest; encouragement of collaborative arrangements with academic researchers; and, where possible, the use of electronic forums for clinicians. As a result, the Region VI office convened a primary care research meeting in San Antonio to provide clinicians with an opportunity to discuss research interests and provided technical assistance to clinicians in accessing electronic bulletin boards as a means of acquiring research information.
Two studies were completed demonstrating the impact of the rapidly changing health care delivery system. One project evaluated responses to a Federal Register notice soliciting comments on the extent to which competitive health plans contract with academic medical centers (AMCs). The objective was to obtain insight into the potential impact of further growth in managed care organizations (MCOs) on the viability of AMCs and their teaching hospitals and to determine what additional research would be useful to further explore these issues. The study finds that the growth of managed care enrollment will change the mix of patients admitted to AMCs, thus affecting training opportunities for resident physicians and other health professionals. Both AMCs and MCOs suggest that the costs of teaching and research be separated from patient care costs and funded through separate mechanisms.
Another project analyzed issues raised by proposals for improving the supply, training, and distribution of primary care providers. Many proposals considered by Congress included provisions that would have altered the mode of funding graduate medical education, thus changing the numbers and types of new physicians. The report includes a chart book on the supply, training, and distribution of physicians; background on the need for an increased number of generalist physicians; and estimates of the number of advanced practice nurses and physicians' assistants that would be needed if the number of physician residents were reduced.
In reviewing evaluation projects for 1996, OPHS considered the following priority areas: congressional mandates, cross- departmental initiatives, OPHS strategic plan-related initiatives, program improvements, and the Government Performance and Results Act. OPHS selected 20 program and policy evaluations for funding for 1996, which are described below.
The Commission on Dietary Supplement Labels, appointed by the President, is evaluating factors relevant to possible FDA regulation of labels for dietary supplement regarding health claims. This project was requested by Congress.
An evaluation of the Cooperative Agreements for Demonstration Projects for Capacity Building at Historically Black Colleges and Universities (HBCUs) is under way. This cross-departmental initiative will examine the extent to which the overall program and individual projects have resulted in the establishment of offices of sponsored programs and the adoption of uniform processes. The evaluation will also assess how well the program has achieved increased funding for health-related research and training at the institutions and how effectively it has enhanced current research, training, and services.
The OPHS evaluation priorities related to strategic planning initiatives will focus on several areas. First, the organization of the Healthy People 2000 objectives reflects categorical funding streams and the concern of special interests in the field of health. To promote an integrated public health message, OPHS will reconsider the basic framework of national health goals and objectives in preparing for the Healthy People 2010 plan. The study will collect insights on the current objectives framework from the Healthy People 2000 consortium members, State health agencies, managed care industry representatives, and major Fortune 500 purchasers of health care plans.
School health programs, both comprehensive and categorical, continue to be developed and implemented throughout the Nation. OPHS will fund a study to identify evaluations of school health programs and make available an updated compilation of methodologically sound studies and their findings to assist school boards, administrators, health personnel, and health educators trying to maximize limited resources for effective health programs.
OPHS will continue to expand efforts to improve nationally available data on public health infrastructure, especially focusing on local public health capacity. In addition, funds will be used to support a similar effort conducted by several pilot States to categorize funding for population-based health activities, as distinct from funding of personal clinical services. The results will provide local health officials and planners with a validated instrument for estimating the level of support for essential public health functions.
In the area of health data and the environment and in continuing support for public health infrastructure revitalization, OPHS will fund development of two data-tracking systems: one related to sentinel public health indicators, related health outcomes, levels of health risks, and the health protection infrastructure, and the second on national and State-level data concerning environmental health outcomes and risks. OPHS will also fund an assessment of health plan involvement in community-based initiatives and a survey of clinicians to assess the level of provision of and the importance placed on preventive clinical services by primary care providers.
OPHS will support several projects to help improve program operations and management. These include a study of management alternatives for emergency preparedness and response, an evaluation of a new model of coordinating and integrating HIV- prevention messages and primary care in high-risk populations, an evaluation of the Minority Health Resource Center, and the assessment of the Adolescent Family Life Program.
OPHS will continue support for the work of the Panel on Cost- Effectiveness in Health and Medicine, a non-Federal expert panel appointed in 1993, which will publish its report this year. The report will be discussed at a conference to explore the panel's recommendations and conduct workshops on the application of cost- effectiveness analysis for specific medical and public health applications. OPHS will also provide support through the National Academy of Sciences for a critical reassessment and revision of dietary reference intakes (recommended dietary allowances).
The implementation of GPRA is a priority for OPHS. The development of performance measures for public health programs will be crucial for program planning, budgeting, and legislative development. OPHS will target efforts in three program areas to develop these measures. First, it will support completion of the consultation begun in fiscal 1995 with State and local health officials concerning the development of performance measurement systems that can be used to improve accountability for expenditure of Federal funds. OPHS will help States and local governments determine the impact Federal programs are having on improving the health status of Americans. OPHS will also discern the extent to which States and major American cities have current data available to measure their own year 2000 objectives and selected national objectives. Finally, the Office of Minority Health will examine its Bilingual/Bicultural Service Demonstration Grant Program, which intends to build the capacity of community-based organizations to address access to health services for limited English-speaking minority populations.
The next few years will be a time of enormous transformation for this Nation's health care system. The need to manage skyrocketing health care costs and rein in the Federal budget deficit--while trying to address the health needs of more than 40 million uninsured Americans and a general population that is becoming older as well as more linguistically and culturally diverse-- presents enormous challenges to HHS, OPHS, and their partners in the public health community.
The shifting emphasis on managed care presents new opportunities coupled with new dilemmas about how best to ensure that, in the quest to manage the cost of care, overall quality of care is not unduly compromised. It raises new questions of how to ensure a work force that is properly trained and fully capable of performing the essential services of public health for increasingly diverse populations in this increasingly dynamic health care environment. Furthermore, transformations in the health care and public health arena necessitate monitoring and assessing (and the data systems to do so) effects and impacts of these changes on the financing, organization, and availability of population- and clinic-based preventive services and medical treatment.
OPHS has given highest priority in its fiscal 1997 strategic plan and evaluation efforts to addressing these issues. The plan emphasizes the need for a strong foundation for public health in the 21st century. The priorities focus attention on (1) the information systems and work force that constitute the Nation's public health infrastructure and that are necessary to effectively provide the essential services of public health; and (2) the impact of managed care arrangements on the resources available for this infrastructure and on the health of all Americans, especially those most vulnerable.
Future OPHS evaluation planning efforts will reflect the OPHS strategic plan as well as the broader HHS programmatic priorities related to improving the health of all Americans and assessing the effects of health sector transformation. Planned investigations in support of the public health infrastructure include analyses of existing and needed data systems and data to adequately monitor health status and address potential threats to the public's health before they become actual threats; better assessments of public health expenditures at the Federal, State, and local level; and examinations of policies and programs that impact population-based services in the States and communities. Efforts related to managed care include evaluations of service delivery to Medicaid managed care beneficiaries; identification of deficiencies in the "safety net" (e.g., the uninsured) as a result of managed care; an assessment of outreach efforts to racial/ethnic minority populations in the implementation of Medicaid managed care; and collection of baseline data on the extent to which managed care organizations and State and local health agencies with linguistically and culturally diverse populations in their service areas are able to provide services.
Other planned activities will focus on addressing specific health needs of the most vulnerable populations. These include evaluations of successful models for overcoming system barriers to better integration of prevention and treatment services for populations at high risk of HIV infection and those with HIV disease; the effectiveness of demonstration projects intended to promote abstinence among teens who are not sexually active and a project to improve parenting skills among pregnant or parenting teens; guidelines and tools to enhance the development and assessment of health information and materials for linguistically and culturally diverse audiences; the impact of HBCUs consortium activities on reducing family and domestic violence; and the research base for post-traumatic stress disorders in women, children, and minorities.
Finally, in response to the increased need to measure the performance and effects of public health programs and activities, OPHS will support efforts to integrate national Healthy People 2000 objectives and performance measures in grants with program and department performance plans; develop and implement measures that assess the capacity of State and local health agencies to provide essential public health services and to meet OPHS goals; and ensure expansion of the public health knowledge base through scientific investigations in the behavioral and social sciences, preventive medicine, public health practice, nutrition, environmental and occupational health, and health systems and services.
MISSION: To improve the quality and availability of prevention, treatment, and rehabilitation services for substance abuse and mental illness.
SAMHSA is committed to evaluating its overall programs and individual grant projects in order to assess the effectiveness of prevention, treatment, and rehabilitation approaches and systems of care; the accountability of Federal funds; and the achievement of SAMHSA's programmatic and policy objectives.
To the greatest extent appropriate and feasible, SAMHSA will encourage the use of comparable data elements and instruments across its evaluations in pursuit of a comprehensive evaluation system and to minimize respondent burden.
SAMHSA conducts grant programs under a variety of legislative authorities. These authorities can generally be grouped into two types: services and demonstrations. The evaluation required for a particular grant program is dependent on the type and purpose of the program. SAMHSA will evaluate each of its service programs to provide information to program managers about the accountability of Federal funds. In addition, the evaluations of demonstration programs will generate new knowledge to lead the field in developing policies that improve services. Program and evaluation staff must work together to identify clearly the questions or goals each grant program will address and to propose appropriate evaluation strategies.
The two types of grant programs represent two facets of SAMHSA's mission: service delivery and knowledge development. SAMHSA's leadership in the field is dependent upon the successful interaction of these two facets. Through evaluation, SAMHSA must identify effective approaches to prevention, treatment, and rehabilitation. Through service delivery funds, SAMHSA must provide incentives to the field to implement effective approaches. Major emphases of SAMHSA's mission are the development, identification, and dissemination of effective strategies and systems for prevention, treatment, and rehabilitation.
SAMHSA is now implementing a new integrated model of evaluation and planning. Strategic planning will identify priorities, such as managed care, that drive the development of grant programs and evaluations. In compliance with the Government Performance and Results Act, SAMHSA is attempting to improve performance by identifying performance goals as part of its strategic planning process. The formulation of programmatic and evaluation priorities will include consultation with the SAMHSA and Center Advisory Councils, and with other experts in the fields of evaluation and service delivery. Early and continuous coordination of program planning and evaluation design will result in the articulation of program objectives that can be evaluated. Evaluations will show how well the overall grant programs have achieved their objectives. SAMHSA will translate these results into information that can be used for program and policy development. The strategic planning and policy development processes will then use the results to refine SAMHSA's priorities and objectives.
This evaluation policy will help SAMHSA achieve its goal of continually informing policy and program development with knowledge culled from past performance. In this way SAMHSA can best serve its customers by enhancing the quality of public substance abuse and mental health services.
In compliance with the PHS guidelines for the technical review of evaluations, SAMHSA has established a standing committee of PHS staff who are evaluation specialists. Representatives of the Office of the Assistant Secretary for Planning and Evaluation serve as ex officio members of the committee. The SAMHSA evaluation officer is the committee chair. The committee does not generally review the evaluation proposals of individual grantees. It does review proposals for broader, more comprehensive evaluations, such as the cross-project evaluations of grant programs.
Evaluation project proposals are generally prepared by SAMHSA program staff in the various centers. The standing committee reviews each proposal on the following criteria: clarity of evaluation objectives and research questions, appropriateness and feasibility of the specifications for evaluation design and methods, appropriateness of the plans for dissemination of results, and use of previous relevant evaluations and existing program data systems. Each proposal must clearly state the relationship of the evaluation to SAMHSA's overall policy and priorities and evaluation program.
During fiscal 1995, SAMHSA completed several reports on its evaluations in the Center for Mental Health Services. Some of these studies focused on youth, one of the population groups that SAMHSA has identified as in greatest need of services. In particular, services for children and adolescents with serious emotional disabilities were the subject of a series of research studies funded by the Center for Mental Health Services (CMHS). The evaluations were highlighted in a special issue of the Journal of Emotional and Behavioral Disorders.
In one CMHS research study, investigators developed, implemented, and evaluated an intensive, adolescent-centered case management approach to treating homeless adolescents in Washington State. Results of a 3-month followup showed a significant decrease in symptoms of depression, problem behavior, and substance abuse; significant increases in self-esteem; and reports from the youth of an increase in quality of life. This study is significant because it is the first to formally assess the effectiveness of mental health-related services to homeless youths.
A second research study examined the effectiveness of an individualized, intensive case-managed approach to improving adjustment outcomes for foster children with emotional and behavioral disorders. Results of this community-based, case- controlled experiment indicate that children who received the additional individualized services showed significantly greater short-term improvements in some behavioral and emotional adjustments than did children who received standard practice foster care services.
Another CMHS-funded study assessed the impact of multisystemic family preservation therapy (MFP) on family functioning and problem behavior of delinquent adolescents in rural and urban South Carolina communities. MFP is a nonoffice-based therapy. Interventions are directed toward individuals and families, peer relations, school relationships, academic performance, and any other social system believed to be involved in the problem behaviors. In general, the MFP group demonstrated improvements in the amount of problem behavior, level of mothers' psychological distress, and family functioning following MFP treatment.
SAMHSA has 12 major evaluations under way in the following general areas: program accountability, evaluation of demonstrations, and managed care. A description of each type follows, with some examples.
SAMHSA conducts evaluations for program accountability in compliance with GPRA. These evaluations are undertaken to inform program management and help managers refine program operations. This is the primary type of evaluation conducted on SAMHSA's service grant programs. For example, the children's mental health service program has an ongoing evaluation. The evaluation will yield continuous information on program implementation and on outcomes for children and families served. The information will be used for reports to Congress, feedback to grantees, program development, and performance improvement.
Evaluations of demonstrations are designed to generate new knowledge for policy development. The primary purpose of SAMHSA's demonstration programs is to generate new knowledge to lead the field in developing policies that improve substance abuse and mental health services. For example, evaluation results on substance abuse prevention for high-risk youth will allow policymakers to draw inferences about the effectiveness of certain interventions for this population. The Job Corps evaluation, which will be completed in fiscal 1996, is assessing an enriched substance abuse treatment program for adolescents. An evaluation of the program for Access to Community Care and Effective Services and Supports (ACCESS) will provide information for the design of ongoing service programs at the Federal, State, and local levels.
The National Treatment Improvement Evaluation Study (NTIES) is a cross-site evaluation study examining the effectiveness of demonstration grants funded by the Center for Substance Abuse Treatment (CSAT). The study will assess the extent to which treatment enhancements improve substance abuse treatment outcomes over time. Preliminary results from NTIES provide strong support for the efficacy of substance abuse treatment in reducing levels of substance use, reducing involvement in criminal activities, and increasing employment. The final report will be available in fiscal 1997.
In fiscal 1996, CSAT is initiating an important managed care evaluation to assess the impact of the States' managed care initiatives on substance abuse treatment in terms of access, cost, and quality. Currently, there is little information at the State, provider, or client levels about the impact of managed care on the provision and outcomes of substance abuse treatment services. Five States will be selected for rigorous, statistical evaluation, which will include measures of costs, access, quality, and treatment outcomes. Results will be used to improve the States' substance abuse managed care programs and for Federal policymaking related to managed care and health care reform.
SAMHSA is designing its evaluation activities to guide programmatic and policy decisions. The evaluation activities will complement the SAMHSA and HHS strategic plans and will respond to emerging trends such as managed care.
One SAMHSA evaluation priority is to evaluate demonstration programs and the impact of demonstration findings and knowledge- transfer activities on service delivery. In fiscal 1996, SAMHSA proposed a new demonstration program designed to answer specific, important policy-relevant questions. These questions will be designed to provide critical information to improve the Nation's mental health and substance abuse treatment and prevention services. Evaluations of the demonstrations planned for fiscal 1996 will generate knowledge on such topics as the relative effectiveness of alternative models of managed care for treatment and prevention of substance abuse and mental disorders, the relative effectiveness of alternate models for preventing homelessness among adults with serious mental illnesses, and the efficacy of a brief intervention for marijuana dependence.
Another of SAMHSA's evaluation priorities is to assess the impact of managed care on the availability of services to populations in need and to develop standards for measuring quality and outcome. SAMHSA's managed care evaluation strategy is designed to leverage important knowledge that directly impacts its mission. This knowledge will be acquired through focused applied health services research, evaluations, demonstrations, and epidemiological and service capacity studies. SAMHSA is currently considering an evaluation strategy that would first establish baseline information about the incidence, prevalence, functional disability, and negative consequences associated with mental health and substance abuse; characteristics of the service systems; and characteristics of individuals serviced by those systems. Evaluations would then assess the impact of changes in organization and financing services through managed care and health care system reforms.