Performance Improvement 1997

Chapter III

Agency Evaluation Activities

Administration for Children and Families

Administration on Aging

Evaluation Program

As the responsibilities of this nationwide network of State and Area Agencies on Aging continues to grow, it is essential that they have the necessary information to meet these responsibilities.

The overall evaluation priorities of AoA are to support studies that provide information on the following:

• the success of existing program implementation in meeting the goals of the Older Americans Act
• the design and operation of effective programs
• issues relevant to policy development, legislative planning, and program management.

Summary of Fiscal Year 1996 Evaluations

The ENP provides an average of one million meals per day to older Americans. These meals are targeted toward highly vulnerable elderly populations, including the very old, people living alone, people below or near the poverty line, minority populations, and individuals with significant health conditions or physical or mental impairments. On average, the meals provided easily meet the recommended daily allowance requirements and significantly increase the dietary intakes of ENP participants. The ENP also reduces the social isolation of older Americans in both the congregate and home-delivered programs and links participants with other needed services. Agencies at all levels have forged close links with other parts of America's emerging home and community-based long-term care system. Federal dollars are highly leveraged. Despite participant's low income levels, their contributions account for 20percent of both congregate and home-delivered meal costs. Local donations and volunteer time, often from program participants, account for 14 percent of costs.

Evaluations in Progress

New Directions for Evaluation

The changes occurring in the aging service-delivery network present an opportunity to learn critical lessons in program planning and system development. These changes include more systematic focus on home- and community-based long-term care, growing sophistication in addressing a variety of needs of the older population, the growth of managed care, and the implementation of sophisticated program information systems. Future evaluations will need to consider the impact of these and similar developments on the delivery of Older Americans Act services to our Nation's older persons.

Agency for Health Care Policy and Research

Evaluation Program

• the need for comprehensive information to assess the agency's effectiveness in meeting its major and long-term priorities and goals
• the need for information from "fast-track" or quick turnaround projects to respond to critical agency and departmental concerns
• the need for information from internal evaluations to improve the efficiency with which the agency performs its work

To address these needs, evaluation components are built into virtually all AHCPR activities. Among the evaluation mechanisms used by the agency are targeted evaluation studies undertaken through contracts or grants; peer review of grant applications and technical review of contract proposals for scientific rigor and integrity; efforts to obtain feedback from "customers" on the usefulness of AHCPR research efforts; feedback from AHCPR's User Liaison Program (which provides information and technical assistance to State policymakers, health departments, and officials); and other targeted efforts, such as focus groups and surveys to provide baseline information, inform the design of future agency work, and assess progress toward goals.

AHCPR received delegated authority to review evaluation projects in 1992. In carrying out that authority, it established a two-tier system for assessing proposed evaluation projects eligible for 1-percent set-aside funds. The first phase of the process, an executive-level review assessing the policy relevance and relative priority of proposed projects, is conducted by the administrator and senior staff. The second, a technical merit review, assesses policy-relevant proposals for technical feasibility, soundness of design, costs, potential importance of the findings, and relation to ongoing evaluation activities. This second review is conducted by the Task Force on AHCPR Evaluation Projects, a group consisting of one individual with evaluation expertise from each Office and Center in the agency.

Summary of Fiscal Year 1996 Evaluations

Evaluating Practice Guidelines and Developing Science-Based Measures of Clinical Quality

With the assistance of Medicare Peer Review Organizations (PRO's) from four States, the project developed guideline-based measures and tested them for use among the Medicare population. The PRO's also served as educational facilitators in a quasi-experimental design component of the project focusing on the introduction of the benign prostatic hyperplasia (BPH) guideline in four "continuum-of-care" settings (clinics and provider groups providing both ambulatory and inpatient care).

The project found that it is feasible to develop guideline-based performance measures that could be used to evaluate and improve clinical performance. The BPH guideline was favorably received by both physicians and patients, and improvements in clinical performance made for specific areas were the focus of educational interventions.

Enhancing the Science of Clinical Quality Measurement

In addition to information on clinical performance measures, the project summarizes information on 50 common clinical conditions into a "condition" data base. Information in the condition data base includes age groups affected by the condition; forms and prevalence of the condition; cost and utilization associated with the condition; comorbidities, risk factors, and potentially preventable adverse events associated with the condition; information on clinical services used to diagnose and treat the condition; and information on provider settings and health care professionals associated with the primary condition. A key feature of the condition data base is its inclusion of recommendations from clinical practice guidelines and findings from medical effectiveness research. Information from the condition data base can be used to identify performance measures for a given condition, facilitate the use of individual performance measures for broader quality measurement and improvement uses, and interpret findings from performance data relative to information from practice guidelines and medical effectiveness research.

Initial response to CONQUEST 1.0 has been extremely favorable; the agency has distributed over 3,000 copies of the first version of the software and user manual and is currently developing a "run time" version that will function in Windows 95. In addition, the agency is building on this initial effort by sponsoring a multiyear contract to develop a Quality Measurement Network, a clearinghouse and technical resource for those interested in identifying and using measures to evaluate and improve clinical performance.

Helping Consumers Understand Health Care Quality Information and Evaluate Health Care Plans

The project found that the information needs of consumers and health professionals differ markedly. While consumers judge health plans on such issues as how they gain access to specialty services for various acute or chronic conditions, health care professionals use population-based performance measures to judge quality of care. These population-based measures focus more extensively on clinical processes and outcomes; while these measures currently may have little meaning to most consumers, they have great potential in shaping the quality of health care and health plan accountability. The project found that consumers are interested in a variety of formats for reviewing health plan data and that the availability of a "personal guide" or a trained individual to assist consumers in understanding information would be highly beneficial. With respect to health plan information needs, the project found that health plans face a "data burden" that is costly and could be relieved by the establishment of uniform standards that are used by all health plans and purchasers.

In addition to providing technical and substantive assistance to the State of Oregon, the project develops the science base for statewide quality assessment and reporting, an effort which can be used to help other States evaluate quality of care.

Synthesizing Information From AHCPR-Sponsored Research

Evaluations in Progress

Assessing the Health Care Environment

One project builds on the work of the Department of Health and Human Services Data Council's Committee on Health Data Standards and the Information Infrastructure Task Force's Health Information Working Group to collect and summarize information on the information standards activities of various Federal agencies. This project will update a previous report, Current Activities of Selected Health Care Informatics Organizations. Another project will identify and describe the range of outcomes research activities conducted by various private-sector organizations and focus on approaches for identifying research topics, outcomes research methodologies, and translation of findings into clinical practice. A third project in this group will identify approaches to achieving productive public-private research collaborations. This effort will use publicly available data to identify examples of public-private collaborations and to examine such facets of the project as methods for identifying research topics, treatment of investment and financial records, and reporting research results.

Evaluating Ongoing Agency Work

Evaluation Designs and Planning Studies

Another project examines the feasibility of compiling an encounter-level managed-care research data base by identifying currently available public- and private-sector data and by identifying data analysis and data release problems. The third project in this category focuses on better understanding research needs in the area of appropriateness of care. This study will explore the published literature on the causes of diagnostic inefficiency in primary care, interventions designed to address these inefficiencies, and the effect of interventions on the quality of care. The project will focus on errors in selected high-cost, high-prevalence areas of health care for which there may be considerable uncertainty about diagnosis, but accepted treatments. The final study in this category examines published and unpublished studies on the effects of consumer health informatics on patient decision making.

New Directions for Evaluation

The first area emphasizes agencywide performance measures that can evaluate the quality of the agency's work and inform policy making, budget planning, and program management. Agencywide performance measures will help AHCPR address requirements of the Government Performance and Results Act, providing information that will improve the agency's performance in key program areas. Projects initiated in FY 1997 will produce and test measures that can be applied in FY 1998 and beyond. In addition to these efforts, the agency will be evaluating the effect of changes in programmatic operations. One example of this type of effort is a proposed evaluation of the effect of recent improvements to the AHCPR grants process.

The second area centers on building AHCPR's portfolio of research to translate research findings into forms of information that actively assist consumers, practitioners, payers, and others in making effective health care decisions. Of particular interest to the agency are needs assessment and formative evaluation efforts related to the new Evidence-Based Practice Centers program. This new initiative, which will replace AHCPR's guideline development program, creates research centers that will conduct systematic reviews and synthesize scientific evidence related to the effectiveness of various approaches, procedures, and technologies. These evidence syntheses can be used by practitioners and health care organizations to develop clinical practice guidelines and other quality improvement tools. Evaluation activities will first focus on assessing user needs in terms of topics, formats, and technical assistance. Subsequent evaluation activities will assess the usefulness of the new products for clinical quality improvement.

The third area will build on AHCPR's work to examine and evaluate changes in the health care system and how those changes affect access to care, use of services, quality of care, and patient outcomes. As the agency continues improving the Medical Expenditure Panel Survey, new evaluation questions and opportunities arise on such issues as the efficiency of the new design and its effectiveness in answering key health policy questions. The agency will be evaluating aspects of the survey process, including data support contracts and approaches to evaluating modeling capacities.

Centers for Disease Control and Prevention

Agency for Toxic Substances and Disease Registry

CDC Evaluation Program

• Monitor health.
• Detect and investigate health problems.
• Conduct research to enhance prevention.
• Develop and advocate sound public health policies.
• Implement prevention methods.
• Promote healthy behaviors.
• Foster safe and healthful environments.
• Provide leadership and training.

CDC's evaluation philosophy is consistent with its overall focus on the practice and application of sound public health strategies. This orientation is exemplified by the agency's "bottom up" nature of project development. Projects are conceptualized, developed, and monitored by the public health professionals at CDC who are most closely aligned with the everyday practice of public health--program staff.

On an annual basis, the Director of CDC provides guidance to the various Center, Institute, and Office Directors on 1-percent set-aside evaluation activities. This memorandum generally includes information about the types of studies to be carried out with 1-percent evaluation funds. Each proposal undergoes multiple levels of review. Initial review is conducted by the Office of Program Planning and Evaluation. Subsequent reviews are completed by staff within the Office of the Assistant Secretary for Planning and Evaluation. Study authors are provided with comments, questions, and recommendations made by reviewers. In addition to providing their responses, authors are given the opportunity to revise their proposals at this time.

A panel of CDC evaluators, scientists, and program managers meets to review and rank proposals. Review criteria include (1) relevance to prevention effectiveness; (2) relative importance of the public health problem being addressed; (3) probability that the proposed project will accomplish its objectives; and (4) the extent to which other CDC programs will derive benefit from the project. Results from this panel review are converted into a comprehensive ranking, which is provided to the Director of CDC. Final funding decisions are made at this time.

Finally, staff within the Office of Program Planning and Evaluation work closely with program staff to ensure development of a clear statement of work for selected projects. Prior to initiation of procurements, a final ad hoc review of the project statement of work is completed.

ATSDR Evaluation Program

Prominent issues addressed in the new planning system emphasize ATSDR's priority of improving the health of people affected by hazardous substances polluting the environment. Using exposure assessments and demographic data to identify people at risk and, more directly, assessing and addressing the concerns of ATSDR customers represent some of the improvements. The new planning system provides the basis for measuring ATSDR performance and making systematic improvements as part of its internal evaluation activities.

Summary of Fiscal Year 1996 CDC Evaluations

Data Policy and Surveillance Studies

Program Evaluations

Evaluations of Educational Interventions

Assessments of Specialized Aspects of Current or Proposed Programs

The two final studies of this type were related to immunization. One examined issues surrounding consent for adolescent immunization. As adolescent vaccination expands, particularly hepatitis B vaccination, issues related to need for parental consent for receipt of indicated vaccines become increasingly important. Similarly, operational issues surrounding the provision of immunization services is critically important. The final study evaluated how audits conducted in the State of Georgia from 1986 to 1994 affected immunization coverage levels in the preschool population and determined medical and management policies and practices that influence immunization rates.

CDC Evaluations in Progress

Evaluations of Data Policy and Surveillance Systems

In the case of the study of NETSS and PHLIS, determinations about how surveillance data transmitted through each of these systems are used at local, State, and Federal levels will be made. Similarly, the centerpiece of the NIS study is to design an evaluation system that will focus on three main questions: How well does the NIS meet the program needs of the National Immunization Program for estimates of immunization coverage rates for population groups of special interest? Is there a way to simplify the analytic methods currently used on the survey without jeopardizing the statistical integrity of the survey? Is there a way to make the survey data collection less costly without jeopardizing the statistical integrity of the survey?

Both the evaluation of racial and ethnic identification data and the redesign of the NHIS represent efforts to understand and refine survey operational issues. Finally, the study evaluating CDC's current surveillance system for drug-resistant Streptococcus pneumoniae will evaluate this system's hospital-based sentinel surveillance system for sensitivity and representativeness. Active population-based surveillance for invasive pneumococcal infections in two geographically distinct areas will be conducted. Surveillance areas for this evaluation are either adjacent to a sentinel surveillance hospital or in a community with one or more hospitals sufficiently large to serve as a comparison for the rest of the community.

Program Evaluations

The study of the Health Hazard Evaluation (HHE) Program is a process evaluation. As such, the overall purpose of the study is to develop a tool and procedure by which the NIOSH can conduct an ongoing evaluation of the effectiveness of the HHE Program, which responds to 400 to 500 requests for on-site health hazard evaluations each year from employers, employees, employee representatives, or other Federal, State, or local agencies.

By contrast, two of the other studies mentioned are outcome evaluations. The evaluation of the Field Epidemiology Training Program (FETP) will ascertain whether the program has achieved its objectives, which are to train public health professionals in applied epidemiologic skills, to promote the sustainability of autonomous FETP's, and to develop a global network of national programs.

The evaluation of the National Laboratory Training Network (NLTN) will assess the degree to which the NLTN achieves its goals and mission. Study questions fall into five general categories: offerings related to needs of laboratories and their staff; quality of the training provided; impact of the training provided; quality of outreach and marketing; and barriers to training.

Finally, the evaluation of the Prevention Centers Program will assess to what extent CDC-supported research is providing the public health community with workable strategies to address major public health problems. Prevention research issues related to innovation, relevancy, dissemination, application, and quality of research are the focus of this project.

Development of Performance Measures

Assessments of Specialized Aspects of Proposed or Existing Programs

Each of the studies mentioned has at its core a desire to understand particular human behaviors and motivations. A clear understanding of these issues must be present before wholehearted program and fiscal commitments are made. For example, the Advisory Committee for the Elimination of TB recommended increased funding for TB outreach and workers conducting the outreach. However, prior to committing resources to this strategy, it is incumbent upon CDC to have a clear understanding of how outreach work is currently conducted and what outreach workers do in their everyday work. It is through the evaluation of TB Outreach Workers that the National Center for HIV, Sexually Transmitted Disease, and TB Prevention will be provided with baseline data about how outreach workers function.

Similarly, a community context study is also being conducted that will refine CDC's understanding of the relationships between public policies prohibiting minors' access to tobacco, implementation and enforcement of such policies, tobacco vendor perceptions, actions that may influence the sale of tobacco products to minors and the use of tobacco products by minors.

Finally, two evaluations addressing violence-prevention issues are also under way. The first of these studies is being conducted jointly by CDC and the Indian Health Service. This study will evaluate surveillance systems, process indicators, and outcomes of four multifaceted suicide-prevention programs. The focus of the second violence-prevention study involves evaluating the effectiveness of a selected program for health care providers and battered womens' advocates. Specifically, the ability of this program to successfully diagnose, manage, refer, and otherwise assist female victims of intimate partner violence will be assessed.

New Directions for CDC Evaluation

CDC's evaluation priorities in upcoming years will be based on five cross-cutting strategies that address the programmatic needs of its 11 Centers, Institutes, and Offices.

Strengthen Core Public Health Functions

Enrich Our Capacity to Respond to Urgent Threats to Health

Develop Nationwide Prevention Strategies

Promote Women's Health

Invest in Our Nation's Youth

Food and Drug Administration

Evaluation Program

Performance Management

Customer Participation

An extension of both performance-based goals and customer participation also affects our response to small businesses.

Responding to the Needs of Small Business

In sum, the FDA's evaluation efforts are driven by the mandates of GPRA and its corollaries, are carried out by line managers rather than specialized evaluation staffs, and are focused in the areas of performance management and customer participation, with additional attention to the needs of small businesses. Because the FDA receives its funds from agriculture appropriations rather than Public Health Service Act appropriations, the FDA does not manage a 1-percent evaluation set-aside.

Summary of Fiscal Year 1996 Evaluations

Implementation of the Prescription Drug User Fee Act

The success of the fourth year of PDUFA is reported in the Fourth Annual Performance Report. Despite an increasing workload, the agency has exceeded every PDUFA performance goal, resulting in a record number of new product approvals and shorter approval times. Under PDUFA, during FY 1996, the FDA approved 131 new drug applications and product license applications, an increase of 56 percent over FY 1995's total (84) and a 96-percent increase over FY 1994's total (67). PDUFA has enhanced the working relationship between the FDA and its sponsors, resulting in higher quality applications that can be accepted immediately and reviewed more quickly. Ultimately, new products are reaching the market faster.

Consumer Use of Food Labels

• The data support the popular perception that Americans have embraced the new food label as an effective tool to enable them to make healthier food choices.
• Consumer confidence in nutrient content and health claim information also showed improvements.
• Information on food labels appears to be having a growing impact on consumer decision making.

For a small number of product categories in which fat-modified products are available, there has been an impressive, simultaneous increase in new product introductions and percentage of market share for these fat-modified products. While each product category has a unique history, these data mirror recent news reports that suggest both consumers and food manufacturers have responded to the impact of the nutrition facts label on food products. These preliminary market trend analyses are consistent with the views that increased availability of nutrition information, as mandated by the NLEA regulations, has contributed to a growing number of new fat-modified products in the marketplace and that these products are garnering an increasing share of their respective markets.

Food Labeling and Package Survey

• Nutrition-labeled products account for an estimated 96.1 percent of the annual sales of processed, packaged food across 58 major product groups.
• There was a 27.2-percent increase in the proportion of nutrition-labeled products between 1993 and 1995.
• For processed meat products, 100 percent are now nutrition labeled, an increase of 47.9 percent over the rate in 1993.
• In three product groups, less than 90 percent of the products sold have nutrition labels--tea, salt (including seasoning and spices), and coffee.

The current study focuses only on the percentage of packaged food products sold annually that bear quantitative nutrition labeling, as well as data on health claims, nutrient content claims, and ingredients.

Evaluations in Progress

The following projects are currently under way.

Implementation of the Prescription Drug User Fee Act

Assessment of the Mammography Quality Standards Act of 1992 (MQSA)

The FDA is concerned that recent trends by third-party payers to capitate payments for mammography services have severely limited the ability of facilities to absorb any cost increases caused by the quality standards. Studies have indicated that excess capacity exists within the industry. Therefore, more efficient provision of mammography services may result in low-volume facilities leaving the industry. The agency needs to assess these economic implications in order to ensure that patients will continue to have access to affordable, high-quality mammography services. This study addresses these issues in two ways. The first involves measuring the effect of standards on patient access to mammography, particularly in rural areas. Findings for this analysis were reported in the document Effects of Facility Closures on Access to Mammography. The second component is a cost-benefit analysis that will be used to predict the impact of proposed standards on the costs of providing mammography. These issues are addressed using data collected from facilities that provide mammography services. A draft report of the results of the cost-benefit analysis was presented in the spring of 1996, and comments are currently being reviewed. A final version will be available in the fall of 1997. Agency managers are using these results to develop a performance-oriented regulatory policy that will be coordinated with existing customers.

Food Labeling and Package Survey (FLAPS)

FDA Inquiries Assistance Program for Small Business

FDA Penalty-Reduction Program for Small Business

Influenza Virus Vaccines in High-Risk Patient Categories

Nationwide Evaluation of X-Ray Trends (NEXT)

New Directions for Evaluation

Health Care Financing Administration

Evaluation Program

• Monitoring and Evaluating Health System Performance: Access, Quality, Program Efficiency, and Costs--HCFA's research program produces information and descriptive statistics on the infrastructure of the health system, on populations of health care users, and service and expenditure patterns; examines differences in costs, quality, and access to care; and assesses the effects of HCFA programs on beneficiary health status.
• Improving Health Care Financing and Delivery Mechanisms: Current Programs and New Models--HCFA performs research and demonstrations to develop and test new payment and delivery models intended to increase the efficiency and effectiveness of Medicare. ORD's evaluations of these demonstration projects provide policymakers with information about their impact.
• Meeting the Needs of Vulnerable Populations--HCFA's research and evaluation program includes projects that develop new approaches to improve access to cost-efficient and appropriate health services for vulnerable populations.
• Information to Improve Consumer Choice and Health Status--Improving beneficiaries' knowledge and ability to make informed health care choices, both in the health plans they select and in the services they use, is part of HCFA's commitment to improving communication of information to beneficiaries. ORD examines variations in the use of services and treatments and the impact of new information tools and technologies in making health care decisions and improving beneficiaries' health status.

Summary of Fiscal Year 1996 Evaluations

Evaluation of the Ventilator-Dependent Unit Demonstration looked at the cost of the service for patients who are being weaned from ventilators, which often exceeds the present-day payment system under prospective payment. The project evaluated four demonstration sites that provided care for chronic ventilator-dependent patients. The evaluation comprised three major components: case studies of the demonstration sites, including a comparison of Medicare reimbursement for patient care under the Tax Equity and Fiscal Responsibility Act (TEFRA) compared with reimbursement for the same care under the prospective payment system rules; outcome measures such as comparing the utilization of services, patient health, hospital charges, and Medicare expenditures for individuals admitted to demonstration sites and for patients selected to serve as a control group; and estimation of the effects of implementing a national ventilator-dependent unit program, under TEFRA reimbursement, on utilization and Medicare expenditures.

Evaluation of the Arizona Health Care Cost-Containment System looked at the continuing operation of the Arizona Health Care Cost-Containment System (AHCCCS), with particular emphasis on the implementation and operation of the Arizona Long-Term Care System (ALTCS), a new component of AHCCCS, which began in December1988. AHCCCS is a unique, State-sponsored capitation demonstration that provides public assistance medical care to residents of Arizona who are eligible for Aid to Families with Dependent Children and Supplemental Security Income cash payments. The major research questions included the following:

• Does competitive bidding and selective contracting result in a lower per unit long-term care service cost?
• How effective is the preadmission screening instrument used by ALTCS in identifying individuals who are at risk of being institutionalized?
• Can home- and community-based services be substituted for long-term institutional care for individuals who pass preadmission screening, and are those services less expensive than institutional care?
• Does case management of long-term care services result in lower cost and better coordination of care?
• What are the effects of capitating long-term care services?
• Is the ALTCS more cost-effective than a comparable State's fee-for-service long-term care program?

The results of the quality-of-care analysis indicate that ALTCS nursing home residents are more likely to experience a decubitus ulcer, a fever, or a catheter insertion than nursing home residents covered by NewMexico Medicaid, suggesting a lower quality of care for ALTCS nursing home residents than for those in NewMexico. However, the lack of pre-ALTCS data precluded an analysis of the improvements in quality since ALTCS began. The cost of the ALTCS program during its first 3years was somewhat less than the cost of a traditional program in Arizona (6percent in FY 1990; 13 percent in FY 1991). The AHCCCS acute care program cost also continued to be less costly than of a traditional fee-for-service program.

Assessment of the Impact of Medicaid Drug Rebate Policy on Expenditures, Utilization, and Access used decomposition analysis to determine the change in total drug expenditures before and after implementation of the Medicaid drug rebate program. The role of covered population changes, intensity (utilization rate) changes, changes in efficiency (drug product prices), changes in dispensing fees, changes in rebates, and administrative costs were evaluated. The impact on recipient access was assessed by constructing a person-level file of prescription drug claims, both pre- and post-Omnibus Budget Reconciliation Actof1990 (the legislation that mandated the drug rebate program). During the first two fiscal years the drug rebate amounts accrued were 10.3 percent of the total Medicaid drug expenditures.

Sustainable Support System for Telemedicine Research and Evaluation created an ongoing mechanism by which the cost, effectiveness, and utility of telemedicine services could be systematically evaluated. This was done through formation of a Clinical Telemedicine Cooperative Group, which was modeled after a successful cooperative multicentered research organization. Its functions included (1) providing operational and statistical support for telemedicine research and evaluation; (2) maintaining a communication system to link geographically distant telemedicine projects to share information and perform telemedicine research; (3) creating easily adaptable electronic data collection and tabulation instruments for use in telemedicine research; and (4) building a comprehensive on-line telemedicine information clearinghouse for gathering, storing, and disseminating information about the utility, effectiveness, and suitability of telemedicine for a broad range of medical and social applications.

Evaluation of the Home Health Agency Prospective Payment Demonstration examined the first phase of a program designed to test the effectiveness of using prospective payment methods to reimburse Medicare-certified home health agencies. In this demonstration a per-visit payment method that sets a separate payment rate for each of six types of home health visits (skilled nursing, home health aide, physical therapy, occupational therapy, speech therapy, and medical social services) was tested. The study looked at the effects of this payment method on agency operations, service quality, and expenditures. It also analyzed the relationship between patient characteristics and the cost and utilization of home health services. The findings suggested that demonstration sites had not decreased their cost per visit, had increased their total revenues and net revenues, or had altered their behavior in ways that affect the quality of home health care.

Evaluating the Effects of Physician Payment Reform on Access: Time Series Analyses of Hospitalizations for Ambulatory-Care Sensitive Conditions looked at the effects of physician payment reform (PPR) on access to care in the Medicare population by studying patterns of hospitalization for ambulatory-care sensitive (ASC) conditions. This project analyzed the trend in rates of hospitalization for selected ASC conditions to see whether there is a discontinuity in the time series associated with the implementation of PPR. Analyses were compiled for the trend in hospitalizations for one ASC condition, congestive heart failure (CHF). No significant discontinuity was found in hospitalizations for CHF with the implementation of PPR.

Effects of Predetermined Payment Rates for Home Health Care is a study of the Home Health Per-Visit Prospective Payment Demonstration that pays home health care agencies a prospectively set rate for home health visits, thus providing an incentive to these agencies to control their costs of delivering Medicare home health visits. The study shows that this incentive was largely overwhelmed by the current home health care environment, which is characterized by diversity, change, and competitive pressures. Nonetheless, the opportunity to earn a profit and the increased possibility of losses may have slightly increased the level of attention agencies gave to cutting costs. Prospective rate setting had no discernable effect on the number of visits provided by agencies or on patients' other Medicare costs, quality of care, access to care, or use of services not covered by medicare.

Trends in Access to Health Care Services for Selected Segments of the Medicare Population were developed for the years prior to, during, and after implementation of PPR. The focus was on vulnerable subgroups of the Medicare population, such as persons with low income, persons without supplemental medical insurance, and persons with acute and chronic conditions. Geographic differences also were examined. These trend data were derived from the National Health Interview Survey (NHIS) conducted by the National Center for Health Statistics. The years 1984, 1986, 1989, 1990, and 1991 were used to develop pre-PPR baseline data. The years 1992 and 1993 were used to develop post-PPR data. Pre- and post-PPR data from the NHIS showed that health insurance and health status are both important determinants of the use of physician services.

Assessment of the Impact of Pharmacy Benefit Managers looked at companies that apply managed care principles to prescription drug programs. Their objective is to ensure optimal and cost-effective drug prescribing and use. The project characterized these firms, compared the costs and quality of care (pharmacy benefits) in Medicaid programs versus pharmacy benefit managed care for the privately insured and for Medicaid enrollees in managed care. The project found that these firms did provide both administrative functions and drug use control, that they were dispersed throughout the country and covered substantial numbers of beneficiaries, that this aspect of the health care industry is complex and rapidly changing, and that they offer various programs (e.g., Medicaid, extensive provider networks, favorable market reimbursement rates for pharmacies, and sophisticated claims processing and data management systems).

The Evaluation of Medicare SELECT Amendments looked at a pilot Medicare supplemental insurance product under which full Medigap benefits are paid only when services are provided by the plan's provider network. Case studies were conducted. The analytical portion of the project compared cost and use of Medicare and supplemental services, selection effects, beneficiary satisfaction, and physician practice patterns with other Medigap options.

Evaluations in Progress

Medicare Projects

The Evaluation of Medicare SELECT looks at a pilot Medicare supplemental insurance product under which full Medigap benefits are paid only when services are provided by the plan's provider network. Case studies are being conducted. The analytical portion of the project will compare cost and use of Medicare and supplemental services, selection effects, beneficiary satisfaction, and physician practice patterns with other Medigap options.

The Evaluation of the Medicare Case Management Demonstrations studies the appropriateness of providing case management services to beneficiaries with catastrophic illnesses and high medical costs. It will test case management as a way of controlling costs in the fee-for-service sector.

The Medicare Participating Heart Bypass Center Demonstration Extended Evaluation continues to study the feasibility of a negotiated all-inclusive pricing arrangement for coronary artery bypass graft surgery while maintaining high quality care. The project will look for any net cost savings to the Medicare program, any volume increases at the demonstration sites, the aspects of the demonstration that were attractive to beneficiaries and referring physicians, and whether the quality of care at the sites was equivalent to that provided prior to the demonstration.

Monitoring and Evaluation of the Medicare Cataract Surgery Alternate Payment Demonstration assists HCFA in tracking this set of demonstrations, which test the feasibility of an all-inclusive negotiated (bundled) price for cataract surgery. The price covers the physician, facility, and intraocular lens costs. The analysis portion of the project tests whether there were any net savings to the Medicare program, changes in the use of services included and excluded from the bundle, beneficiary satisfaction, and quality of care.

In the Impact of the Medicare Fee Schedule on Access to Physician Services, HCFA is evaluating the effect of the physician fee schedule on the beneficiary's access to care. Six different strata of beneficiaries are selected based on the relative size of the payment change under the fee schedule compared to the earlier payment scheme. The project is examining use of services, outcomes of services, and the change in the beneficiary's financial liability.

The Evaluation and Technical Assistance of the Medicare Alzheimer's Disease Demonstration assists with HCFA's projects that provide comprehensive in-home and community-based services to beneficiaries who have dementia. Two different models of care are involved in the demonstration, differing by the intensity of the case management and the amount of service costs covered each month. The analytical portion of the contract is attempting to identify the factors associated with cost-effectiveness, the services that appear to affect the health status and functioning of the patients, the effects on the caregiver (burden and stress), and whether the provision of the additional home care services delay or prevent institutionalization.

In later years, the following Medicare projects are expected to be completed.

The Evaluation of HMO Outlier Demonstration that examining the Outlier Pool Demonstration that is under way in the Seattle area. Participating plans are paid 97 percent of the adjusted average per capita cost, with 2 percent of the payments going into a pool. Plans with a higher than average incidence of high-cost cases will receive more from the pool than they paid in, and those with a lower incidence will receive less.

The Evaluation of the Medicare Choice Demonstration assesses the feasibility and desirability of new types of managed care plans for Medicare. These plans can be integrated delivery systems and preferred provider organizations.

The Medicare HMO Evaluation updates the findings of an earlier study of Medicare risk health maintenance organizations. That study found that 5.7 percent more was paid for plan enrollees than would have been spent under fee-for-service. The current study looks at disenrollment, beneficiary satisfaction, quality of care, and selection and savings.

HCFA is experimenting with paying skilled nursing facilities on a prospective basis. Currently, such facilities are reimbursed on a retrospective cost basis. This demonstration uses a case-mix classification, called Resource Utilization Groups, to classify patients. The Evaluation of the Nursing Home Case-Mix and Quality Demonstration seeks to estimate specific behavioral responses to the prospective payment and to test hypotheses about aspects of such responses. The main goal of the project is to estimate the effects on the health and functioning of the nursing home residents, their length of stay, and use of health care services; on the behavior of the facilities; and on the level and composition of Medicare expenditures.

Medicaid Projects

The State of Delaware is experimenting with its Medicaid program under the assumption that by enrolling children into a managed-care system, they will reap the benefits of a higher level of coordinated care while benefiting from lowered costs. The Evaluation of the Demonstration Entitled Delaware Health Care Partnership for Children is examining this real-time test of the hypothesis.

The Evaluation of Medicaid-Managed Care Programs With 1915(b) Waivers will provide information on the extent to which various features of the waiver projects contribute to the ability of the Medicaid program to deliver cost-effective care to eligible populations.

The Evaluation of the Utah Prepaid Mental Health Plan: Coordinated Care Systems as Alternatives to Traditional Fee for Service looks at Utah's project which has three mental health centers providing mental health services to all Medicaid beneficiaries in their catchment areas (these areas include over 50 percent of all Utah Medicaid beneficiaries). The State hopes this program will control the cost inflation and improve patient outcomes.

The Evaluation of the Iowa Implementation of Ambulatory Patient Groups (APG's) studies a Medicaid outpatient prospective payment system that groups patients for payment purposes rather than paying on a cost basis. It involves a case study on Iowa's implementation of the APG system and an analysis of the reimbursement methodology.

The Comparative Study of the Use of Early and Periodic Screening, Detection, and Treatment and Other Preventive and Curative Health Care Services by Children Enrolled in Medicaid is examining the effects of the 1989 changes to this portion of the program. It is looking at the process of providing health services and the appropriateness of expenditures for services in four States. It compares Medicaid children with other Medicaid-enrolled children in the same State who are not receiving these services, with emphasis on preventive services. Using national survey data, it also compares Medicaid-enrolled children with non-Medicaid-enrolled children, insured and uninsured, on the use of and expenditure for preventive and other health services.

The Evaluation of the Demonstration for Improving Access to Care for Pregnant Substance Abusers is assessing the effectiveness of projects that improve outreach and assessment; expand, integrate and coordinate program services; and improve client case management. The evaluator will look at access to prenatal care, substance abuse treatment services, and other relevant services. It will assess the effects of services on the health of the drug-addicted pregnant women, any prevention of reduction or short-term impairments to their infants, and the impact on birth outcomes.

The Evaluation of the Medicaid Uninsured Demonstrations covers projects in Maine, South Carolina, and Washington State. They test the effects of extending Medicaid coverage to low-income families. The evaluator will look at the ability of the programs to enroll significant numbers of eligible persons, the conditions under which these persons are willing to participate, the program's ability to induce adequate numbers of providers to participate, the effect on service utilization and health outcomes, their cost-effectiveness, and the extent to which these demonstrations' interventions could be applied nationally.

The Examination of the Medicaid Expansions for Children will use enrollment and expenditure trends to ascertain the impact of the 1989 program changes. It will look the penetration of the target population, and the impact of State policies (and the eligibility group) on enrollment, expenditures, and utilization of services.

The Department is required to report to Congress on the relative quality of care in the Medicaid program. HCFA's Medicaid Quality of Care Study examines the necessity, appropriateness, and effectiveness of selected medical treatments and surgical procedures for Medicaid patients. It is assessing the variation in the rate of performance of selected treatments and procedures on Medicaid beneficiaries for small areas within and among States. It is determining underutilized, medically necessary treatments and procedures for which failure to furnish them could have an adverse effect on their health status.

The Community-Supported Living Arrangements Program: Process Evaluation is designed to test the effectiveness of developing a continuum of care concept as an alternative to the Medicaid-funded residential services provided to individuals with mental retardation and related conditions. The program serves individuals who are living in the community--either independently, with their families, or in homes with three or fewer other individuals receiving the same services.

The Project Demonstrating and Evaluating Alternative Methods to Assure and Enhance the Quality of Long-Term Care Services for Persons With Developmental Disabilities Through Performance-Based Contracts With Service Providers tries to determine whether and how well the implementation of new approaches to quality assurance, with outcome-based definitions and measures of quality, will replace input and process measures of quality in this population group.

In addition, HCFA has other major Medicaid evaluations whose results are anticipated in several years.

HCFA is sponsoring a wide variety of waiver-based demonstrations that give States the opportunity to experiment with their Medicaid programs. As these demonstrations are begun, HCFA undertakes examinations of their impact. The Evaluation of the Oregon Medicaid Demonstration, the Evaluation of the State Medicaid Reform Demonstrations (in Hawaii, Rhode Island, and Tennessee), and the Evaluation of the State Medicaid Reform Demonstrations (in Ohio and Minnesota), are three such projects. More will be started as more State demonstrations are begun.

The Drug Utilization Review Evaluation is looking at the impacts of retrospective and prospective review, which include the payment of pharmacists for cognitive services. Data from demonstrations in Iowa and Washington State and information from programs in other States serve as the basis for this study. Maryland and Georgia will serve as coexperimental and comparison States.

Other Health Financing Policy and Demonstration Evaluation Projects

Evaluation of the Essential Access Community Hospital/Rural Primary Care Hospital Program examines the development, implementation, and early operating experiences of this program. Known as EACH/RPCH, the program is supposed to assist States in maintaining access to health care services in rural areas. This is done through the development of rural health plans, establishment of rural health networks, and creation of a limited service alternative for communities that can no longer support a full-service hospital.

The Evaluation of the Community Nursing Organization Demonstration looks at a set of projects mandated in 1987. The legislation directs projects in four or more sites to test a capitated, nurse-managed system of care. The two fundamental elements of these demonstrations are capitation payment and nurse case management. They are designed to promote timely and appropriate use of community health services and to reduce the use of costly acute-care services.

The Evaluation of the Program of All-Inclusive Care for the Elderly Demonstration, known as PACE, examines the replicates of a unique model of managed care service delivery for very frail community-dwelling elderly, most of whom are dually eligible for Medicare and Medicaid coverage and all of whom are assessed as being eligible for nursing home placement. The core services include adult day health care and multidisciplinary team care management through which all health and long-term care services are arranged. The evaluator is looking at the demonstration sites before and after assumption of financial risk to see if the replicates are cost effective relative to the current Medicare and Medicaid services. They are also looking at the decision to enroll to understand how PACE enrollees differ from the eligible beneficiaries who choose not to enroll.

The Inspector General of the Department is conducting a demonstration of improved methods for investigating and prosecuting fraud and abuse. The Evaluation of the Effectiveness of the Operation Restore Trust Demonstrations will determine whether the more concentrated effort rendered through the demonstration's partnership model has a relatively greater impact on industry fraudulent behavior.

The Evaluation of the Impact of Health Plan Report Cards on Consumer Knowledge, Attitudes, and Choice in a Managed Competition Setting seeks to determine whether the dissemination of information about health plans to consumers (who choose health plans within a managed-care competition framework) will influence their knowledge of plan characteristics, attitudes toward the plans, or choice of plan.

New Directions for Evaluation

HCFA also will continue to develop a wider array of evaluation and measurement tools. The agency's evaluation activities will continue to examine specific policy issues within the HCFA programs. For example, as Medicare and Medicaid continue to pursue managed care options, ongoing work will examine the cost-effectiveness of, quality of, and beneficiary satisfaction with managed care. HCFA plans to carry out projects to monitor and compare the health status or health risk behaviors of beneficiaries in various delivery systems and how these change over time.

Finally, as HCFA develops and implements new high priority demonstrations that will test new payment and health care delivery models for the future, the agency will evaluate these programs and provide information to policymakers about the impacts of these alternatives.

Health Resources and Services Administration

Evaluation Program

Performance measurement includes technical assistance and training activities to strengthen the agency's capacity to assess program performance, as well as studies to assess program outcomes. These activities, mandated by the Government Performance and Results Act (GPRA), will provide data for ongoing program monitoring and evaluation and for developing annual performance plans and budgets. A study completed in September 1995 established a performance measurement baseline for all operating programs, with the ultimate objective of ensuring that the HRSA specifies valid and useful indicators and measures for all programs by September 1998. Over the past two years, the HRSA has made significant progress in performance management, such as strengthening linkages among strategic planning, program activity, and the budget process.

Assessment of program implementation includes a broad range of efforts to assist the agency during a time of new directions from the Administration and Congress. HRSA programs are entering into new arrangements for delivering services, providing health professions education, and encouraging the development of systems reform within seven Program Priority areas. These priorities reflect the HRSA's focus on the communities where underserved populations live, and include academic and community partnerships to foster clinical training in community-based settings; new arrangements to bring poor, uninsured, rural, and chronically ill persons into the mainstream of managed care; assistance to communities in strengthening their health care infrastructure; and new activities with States. All agency studies relate to one or more of these Priorities, which also include improved comprehensiveness and integration of HIV/AIDS programs; school-based care for children and adolescents; and activities to enhance the health of people living along the U.S.-Mexican border.

Cross-cutting policy analysis and research includes efforts to build capacity and conduct studies to clarify the environmental shifts within which HRSA programs operate and to improve the HRSA's ability to document performance and impact through better measurement tools and data sources. Developing this capacity at the agency level and initiating some studies are special priorities for FY 1997; work in later years will build on these beginnings. "Cross-cutting" implies policy issues that bear on the programs of two or more HRSA Bureaus, such as trends in numbers and characteristics of the uninsured and the health of the safety net.

The main purpose of the HRSA's evaluation program is to provide accurate and relevant information on a timely basis to the administrator and other senior line managers. In working toward this objective, HRSA subjects study proposals to a formal review process, which begins with consideration by a committee comprised of senior line officials and is chaired by the Director of the Office of Planning, Evaluation and Legislation (OPEL). This committee's task is to assess the relevance of the proposals to important policy, budgetary, or legislative issues; its potential to answer questions about program effectiveness or impact; its degree of attention to cross-cutting topics; and its relative priority for funding. Following administrator decisions on approval of the recommended studies, a committee of senior analysts from the HRSA, the Agency for Health Care Policy and Research, the National Center for Health Statistics, and the Office of the Assistant Secretary for Planning and Evaluation (ASPE) reviews scopes of work and other substantive portions of documents needed for development of contracts. The role of this technical review group is to suggest ways to strengthen study design or data sources.

HRSA is continuing to place major emphasis on disseminating findings and information about use of study results. This is done through inclusion of abstracts in the ASPE evaluation data base (on the Internet), submission of articles to peer-reviewed professional journals, presentations by HRSA staff at professional conferences, and placement of final reports with the National Technical Information Service. In addition, the HRSA prepares annual volumes of summaries of studies begun or completed the preceding fiscal year. These are distributed within the department and their availability is noted on the HRSA OPEL web page at http://www.hrsa.dhhs.gov/oa.html#opel.

Summary of Fiscal Year 1996 Evaluations

Performance Measurement

Relevant to HRSA's continued efforts to identify appropriate health outcomes indicators, the agency conducted a Consensus Conference on Health Status Gaps of Low-Income and Minority Populations. The report recommends potential measures and data sources for eight conditions, sets of conditions, or preventive approaches: diabetes; hypertension, cardiovascular, and cerebrovascular conditions; breast cancer; cervical cancer; infectious diseases, including AIDS, sexually transmitted diseases, and tuberculosis; immunization; asthma; and pregnancy outcomes.

Another HRSA effort developed quality assurance procedures for the Uniform Reporting System for Titles I and II of the Ryan White Comprehensive AIDS Resources Emergency (CARE) Act, which provide grants to disproportionately affected metropolitan areas and States. The purpose of the funding is to improve the availability and coordination of services for persons with HIV/AIDS. Under the Uniform Reporting System, grantees submit provider-level administrative data and data on client demographics and services received. Uniform data are used to assess the extent to which grantees are achieving the goals of the Act and to help metropolitan planning councils, State agencies, and State consortia target and monitor the provision of services to specific population groups. Through this study, data quality targets were established and manuals were written to help grantees compile high-quality data. HRSA and grantee staff are using the results to improve the quality of data submissions, which will support national and local program monitoring and evaluation.

Managed Care

Additionally, tools were developed for monitoring the cultural competence of primary care providers practicing in managed care systems. The work was carried out through physician and patient panels in two California HMO's serving predominantly Latino or Chinese populations, and through a literature review. "Cultural competence" was defined as the level of knowledge-based skills required to provide effective clinical care to patients from a particular ethnic or racial group. A related concept, "cultural sensitivity," was defined as the psychological propensity to adjust one's practice style to the needs of different ethnic or racial groups. Study products include a patient satisfaction questionnaire, which a managed care plan could use to assess the degree to which enrollees with limited English believe their care is culturally competent; a provider self-assessment questionnaire; and behavioral ethnic identifiers that can be used in enrollment processes as a cue for designing culturally appropriate services.

Primary Care

Assessing the Impact of Public Spending on the Health of Vulnerable Populations: A Framework for Evaluating HRSA Programs provides a structure for describing the linkages among HRSA dollars and HRSA-funded services, HRSA-enabled resource- and infrastructure-building programs, and the populations affected in terms of health outcomes. The analytic approaches developed here could be used by the HRSA or other operating divisions to explain the difficulties in establishing causal factors, to highlight areas with the greatest need for surveillance and monitoring, and to develop data systems to provide the needed information.

A study entitled Impact of Case Management on Health Status in Community and Migrant Health Centers (CMHC's) examined case management practices and their impact on health status of participants in the Comprehensive Perinatal Care Program (CPCP). The study consisted of three components: a control-site study of paired CMHC's, intensively using and not intensively using case management; a model-site study of seven CMHC's serving high-risk perinatal populations; and a two-part mail and telephone survey of a sample of centers receiving CPCP funding. The study found that case management had a positive impact on the continuity and adequacy of care, on the content of care in some areas (psychological risk assessment and health promotion), and on users' perceptions of care. The HRSA is using the findings to integrate case management in all primary care projects serving perinatal patients. The study also developed a self-assessment tool that grantees can use to evaluate their case management activities.

Finally, Technical Support for Need Designation and State Primary Care Access Plan Initiatives assessed the process for designating areas of primary health care shortage. Results are being used in finalizing new designation criteria for shortage, underservice, and access barrier areas.

Care for Mothers and Children

Health Professions

Another study assessed the feasibility of increasing the production of advanced practice nurses by investigating the potential for nursing educational institutions to expand their capacity. "Advanced practice nurses" include nurse practitioners, clinical nurse specialists, nurse anesthetists, and certified nurse midwives. The study found that the single most significant factor in determining an institution's ability to expand capacity is the availability of sites for clinical education, although the availability of clinical faculty is also important. HRSA is using the results to guide future directions for advanced practice nursing programs and to provide technical assistance to grantees. The report has also been shared with schools of nursing with advanced practice programs.

A third health professions study was an exploratory effort to describe arrangements between primary care residency programs and ambulatory training sites affiliated with them. The most important findings are that (1) residency programs do not significantly offset the costs of ambulatory training experienced by the sites nor are there many written agreements in place; (2) residency programs are not aggressively seeking managed care training sites; and (3) cost pressures on ambulatory training sites may make decisionmakers less willing to assume or continue to absorb the cost of training. HRSA will use these findings, in conjunction with findings from two other studies on site-specific costs of ambulatory training, to expand understanding of the costs of ambulatory education and help HCFA in its deliberation on new approaches to GME funding.

Evaluations in Progress

Performance Measurement

Managed Care

Work will also continue to examine the extent to which CMHC's are using clinical outcome measures to assess quality of health care, to meet center needs in a managed care environment, and to improve and monitor the quality of care locally and nationally. Findings will be used to identify performance issues needing further research and development and to facilitate collaboration among private and governmental organizations focused on health outcome measures and underserved populations.

Two studies are examining staffing issues in a managed care context: HMO Staffing Strategies in Underserved Areas is describing strategies to ensure access to services for enrollees in underserved areas, as well as the implications of these strategies for overall access in these areas. The study involves interviews and other data collection at nine HMO's with significant Medicaid enrollment. Selected Aspects of Education in Managed Care is surveying managed care organizations to identify their practices and determine the potential for these organizations to provide high-quality learning experiences for medical students and residents.

Primary Care

Casemix Differences in Health Centers and Other Providers and Their Relationship to Cost will recommend a method for measuring casemix (severity of illness) in ambulatory populations; develop a methodology for creating "casemix profiles" of patient characteristics and expenditure data; and test the methodology with Medicaid data from several States. This study should also provide insights into the relative importance of casemix compared with other factors that may affect expenditures for primary care in the ambulatory setting.

HRSA is continuing to work with the American Hospital Association Hospital Research and Educational Trust (HRET) on evaluation of the HRET-sponsored Community Care Network, a series of demonstrations in 25 sites across the United states. HRSA support for this collaboration was begun through a project completed last year, National Demonstration of the Community Care Network Vision: Development of a Program Evaluation. HRSA will use the findings from the evaluation as a guide for selecting model types or community characteristics that deserve detailed exploration to form hypotheses about attributes that may lead to the successful launching of collaborative activities and to assess implications for cost-effective approaches to measurement and data collection in future evaluations of HRSA programs.

Another study concerns the ability of HRSA-supported community-based organizations to integrate services for individuals with multiple risk factors, such as HIV/AIDS, substance abuse, and homelessness. This project has explored the use of multiple funding sources to provide integrated care, barriers to integration of services supported by separate funding sources, successful integration strategies, and ways in which HRSA might reduce barriers created through multiple funding streams. The results will inform policy on primary care and AIDS programs and will be shared with other operating divisions.

Attention to bilingual and multilingual services is also continuing through studies directed to services at CMHC's, such as the project, An Assessment of Bi/Multicultural Services Offered at Community and Migrant Health Centers. Bilingual Assistance Program: Evaluation of Strategies for Reducing Cultural and Linguistic Barriers to Health Care for Hispanic and Asian Pacific Islander Populations concerns bilingual services in the context of small grants to State and local health departments.

Health Care for Mothers and Children

HIV/AIDS

Organ Transplantation

Telemedicine

Health Professions

The cost and location of training of physicians and others to provide primary care in ambulatory settings is a continuing concern. One project is obtaining quantitative data on the costs involved in training medical students and residents in over 30 ambulatory sites across the country.

A collaborative study with the Educational Commission on Foreign Medical Graduates and the Appalachian Regional Commission is tracking exchange students granted a J-1 visa to participate in graduate medical education in the United States. The HRSA portion of the project will examine the postresidency experience of exchange visitors who have secured waivers to remain in the United States following completion of training for the purpose of providing services in an underserved area.

New Directions for Evaluation

Indian Health Service

Evaluation Program

The purpose of the Office of Planning, Evaluation and Legislation (OPEL) is to advise the Director of the IHS on policy formulation; to conduct and manage program planning, operations research, program evaluation, health services researches, legislative affairs, and program statistics; to develop the long-range program and financial plan for the IHS in collaboration with appropriate agency staff; to coordinate with HHS, Indian Tribes, and organizations on matters that involve planning, evaluation, research and legislation; and to develop and implement long-range goals, objectives, and priorities for all activities related to resource requirements and allocation methodologies and models. OPEL serves as the principal advisory office to the IHS on issues of national health policy and coordinates these four evaluation functions.

• Health Program Evaluations--Collect and analyze information useful for assisting IHS officials in determining the need for improving existing programs or creating new programs to address health needs.
• Policy Analysis--Conduct analyses when a change in the IHS health service delivery system must be considered, when issues emerge in an area where no policy currently exists, or when current policies are perceived as inappropriate or ineffective.
• Health Services Research--Undertake analyses of the organization, financing, administration, effects, and other aspects of the IHS.
• Special Studies and Initiatives--Conduct studies and prepare special reports required by Congress in response to pending legislation or policies, often using a roundtable whenever an issue or a health problem requires immediate action and it is unclear what type of action should be taken.

OPEL meets part of the IHS evaluation needs with two major types of short-term studies: policy or program assessments and evaluation study. The policy study contributes to IHS decision making about budget, legislation, and program modifications and includes background information to support IHS initiatives. Evaluation studies are carried out at the program level, or area offices, and focus on specific program needs.

Annually, OPEL identifies the high-priority health care and health management issues and concerns through the submission of headquarters and area office proposals for assessment or evaluation. IHS area and associate directors submit proposals for possible areas of evaluation study. These proposals are reviewed and rated by a panel of subject-matter experts and evaluation experts and also reviewed by IHS staff for more specific concurrence with IHS strategic goals, objectives, and priority areas. The proposals are then ranked by priority and forwarded to the OPEL Associate Director for review and approval. The Director of the IHS reviews the final proposals and decides the respective funding levels.

Summary of Fiscal Year 1996 Evaluations

Methodology for Adjusting IHS Mortality Data for Inconsistent Classification of Race-Ethnicity of American Indian and Alaska Natives Between State Death Certificates and IHS Patient Registration Records

Evaluation of the Indian Health Service (IHS) Adolescent Regional Treatment Centers

Evaluating the Effectiveness of Alcohol and Substance Abuse Services for Native American and Alaska Native Women: Phase II Final Report

Evaluations in Progress

Elderly Wellness

Prior Trauma Care of Intoxicated Patients as a Predictor of Subsequently Fatal Injury

Resource Requirements Methodology Update

Development of a Health Services Research Agenda for American Indian and Alaska Native Populations

Evaluation of Indian Health Service-Supported Substance Abuse Treatment Program for American Indian/Alaska Native Women

New Directions for Evaluation

These changes will affect the IHS evaluation strategy in the coming years. Nevertheless, the IHS remains committed to comprehensively community-based, preventive, and culturally sensitive projects that empower tribes and communities to overcome health issues. The Director of the IHS has increased emphasis on several initiatives to focus attention on specific health areas and to serve as a management tool to prioritize the IHS's workload. These initiatives focus on women's health, youth, traditional medicine, elder care, and establishment of working relationships with Federal and State governmental agencies. The initiatives will undoubtedly affect new directions for evaluation.

Research and evaluation proposals to be considered in upcoming years include the following topics: evaluation of the effects of medical nutrition therapy on patient outcomes among Native Americans with newly diagnosed type II diabetes mellitus, evaluation of the elders clinic at the Zuni (New Mexico) Ramah Service Unit, evaluation of the impact of the Northern Cheyenne End-Stage Renal Disease Prevention Project, and evaluation of IHS aftercare services provided by IHS's regional treatment centers.

National Institutes of Health

Evaluation Program

Philosophy and Priorities

The quantity and diversity of diseases, disorders, and biological systems in the NIH portfolio make strategic planning and evaluation at NIH a complex task that is continuously evolving. Priorities are set and research programs are planned and evaluated at two levels: the Institutes and Centers (IC's); and centrally by the NIH Director, with whom the ultimate responsibility lies for the course of NIH-funded medical research.

This two-level approach ensures that planning and priority setting specific to the mission of each IC are fully developed and implemented with a clear vision and within the fiscal constraints set by the IC budget and that there is central leadership for developing cross-cutting initiatives and promoting collaboration among the IC's. The NIH Evaluation Program provides information to assist the NIH Director and IC Directors in determining whether NIH goals and objectives are being achieved and to help guide policy development and program direction.

Policies and Operations

Program evaluations are funded through both a trans-NIH mechanism, the 1-percent set-aside authority, and by individual IC program funds. A two-tier system is used to review project requests to use set-aside funding. The first tier involves a review and recommendations by the NIH Technical Merit Review Committee (TMRC) on the technical aspects of project proposals and whether a project fits within HHS guidelines for use of the set-aside fund. The second tier involves the NIH Evaluation Policy Oversight Committee (EPOC), which considers TMRC recommendations and makes final funding recommendations to the Director of NIH or his or her designee. It also conducts policy-level concept reviews of proposals for NIH-wide evaluation studies; establishes the overall NIH set-aside budget; and oversees the process. EPOC recommendations are approved by the NIH Director or his or her designee. Concurrently, the IC's fund program evaluations from their budgets that are used by various committees, working groups, task forces, workshops, conferences, and symposia to assist them and the NIH in program management and development.

One of the ways NIH sets priorities and evaluates its programs is discussion with extramural grantees, intramural investigators and other NIH staff, members of Congress and the Administration, and members of the public, including professional societies and voluntary health organizations. These individuals and groups provide valuable input on pressing public health needs, important scientific opportunities, knowledge gaps, and the balance between patient-oriented and laboratory research.

Summary of Fiscal Year 1996 Evaluations

Evaluation of NIH Shared Instrumentation Grant Programs: Reports From Users documents the results of surveys of users of costly, state-of-the-art instruments purchased through the National Center for Research Resources shared instrumentation program. The program provides equipment that is to be shared among mainly NIH-funded investigators (termed major users), and other researchers who do not have an NIH grant. The study found that 16,050 scientists nationwide used these instruments in 1993; that of these, 69 percent were NIH-supported investigators; and that three-quarters of the major users believed the instruments were essential to their research. The study also found that 70 percent of 11-year-old equipment was still in use, and that 4 out of 5 major users of this older equipment believed the equipment was adequate for their research.

Moving from bench science, the report Clinical Research in the National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) examined how much clinical research was being conducted in NIDDK's research portfolio, under two definitions of clinical research. The first definition required that the patient and the physician had a consultation; this was the narrow definition. The broader definition, adopted by the Advisory Committee to the NIH Director on Clinical Research, subsumed the narrow definition and also included areas of epidemiology, behavioral studies of patients, outcomes research, health services research, and technology development. The report found that in FY 1995, NIDDK spent 31 percent of its extramural budget on clinical research using the narrow definition and 37 percent of its budget using the broader definition; NIDDK spent 29 percent of the intramural budget on clinical research using the narrow definition and 33 percent using the broader definition.

Turning to evaluation of a demonstration project, the report Navajo Alcohol Rehabilitation Demonstration Program examined a federally funded program to provide culturally sensitive treatment for Navajo Native Americans who suffer from alcoholism and the effects of alcohol abuse. It is the only program to use Navajo-speaking counselors in the treatment. The evaluation found a higher level of participation in culturally sensitive therapies after completion of an inpatient treatment program, low use of aftercare services, a high program completion rate, and a low readmission rate. Data on long-term effects were not available.

Another way to treat chemical dependency is medication. The report Development of Medications for the Treatment of Opiate and Cocaine Addictions: Issues for the Government and Private Sector examines the National Institute of Drug Abuse's program to coordinate and encourage academic, private, and Federal regulatory involvement in developing and marketing new medications to treat drug abuse. The study found that pharmacotherapy for the treatment of drug addiction has received little attention; only two additional drugs beyond methadone have been approved, both of which were developed at least 20 years ago; and there is no approved medication for the treatment of cocaine addiction. The report noted major disincentives to pharmaceutical research and development for antiaddiction medications, among which is a lack of sustained Federal leadership. The report gives recommendations to help remove these barriers.

Evaluations in Progress

The study Career Status and Satisfaction With NIH Research Grant Application and Award Processes: A Sample Survey of FY 1992 R01 and R29 Applicants will survey individuals who applied to NIH for an R01 or R29 grant in FY 1992. The purpose is to obtain information on the current status of these individuals. Career progress and research productivity will be assessed by comparing: (1) funded principal investigators versus others supported on NIH grants; (2) principal investigators funded on grants from non-NIH sources; and (3) those who are unsuccessful in getting any support for their research. Respondents will also be asked to assess their satisfaction with the grants application and award process and to provide their opinions on specific modifications NIH is considering for improving policies and procedures.

The study Comprehensive School Health Programs in Grades Kþ12, conducted by the Institute of Medicine, will assess grade-school health education, school health promotion and disease prevention, and school-based health care services from kindergarten through high school. The study will identify and examine model programs to formulate strategies for successful implementation of school health programs at national, State, and local levels.

NIH is responsible for the nationwide infrastructure for animals in research, which includes not only laboratory animals, but also the resources and facilities required to ensure their availability, appropriate use, and humane care. The purpose of Evaluation of Laboratory Animal Use, Facilities and Resources--Phase II is to provide NIH and other government agencies with information and objective data to inform the current national discussion--involving the press, animal rights activists, scientists, and others--about laboratory animal use.

Congress requires NIH to report annually on clinical research it is sponsoring for rare diseases. Development of a Database for NIH-Supported Rare Disease Research will establish a data base of NIH-sponsored rare disease clinical research. The Oak Ridge Institute for Science and Education will provide the instrument for NIH to determine the presence of existing data bases and the nature and extent of data on rare disease studies funded by the NIH. The contractor will review the completed instruments and will develop and provide recommendations on estimated costs and on the best ways to make the data base accessible to the biomedical research community, practicing physicians and other health professionals, and the public.

New Directions for Evaluation

• Medical Research--This area focuses on issues such as the economic impact of medical research; clinical research including the impact of managed care; the health of special populations; and the development of measures for medical research accomplishments.
• Data Collection and Dissemination--This area focuses on issues such as expanded data collections to address emerging priorities; the health information dissemination infrastructure and strategy; and technology transfer.
• Research Training--This area focuses on issues involving national needs for medical research personnel; the quality of NIH training programs; and diversity among trainees.
• Research Infrastructure--This area focuses on issues involving the maintenance and construction of buildings and facilities; computers in biomedical research and medicine; and informatics in medical research.

NIH evaluation studies provide a rational basis for managerial decision making and responding to public concerns for accountability in government. Additionally, it is through such studies that the NIH is able to determine progress in meeting its scientific objectives, strengthening research and administrative activities, and contributing to the mission of NIH to sponsor and conduct research that leads to better health for all Americans.

Office of the Assistant Secretary for Planning and Evaluation

Evaluation Program

• Provide annual guidance to all HHS agencies and staff offices regarding evaluation priorities, procedures, and review requirements.
• Review evaluation priorities proposed by the HHS agencies, providing advice about the focus or method of proposed projects, identifying opportunities for collaboration, and ensuring effective use of resources.
• Prepare planning and summary reports on evaluation activities as required by Congress.

Through the departmental evaluation planning process, ASPE has the capacity to identify cross-cutting program or policy issues of particular concern to the Secretary and specific program and policy areas not covered by the HHS agency evaluation plans. In these instances, ASPE initiates evaluations or collaborates with the agencies to conduct evaluations or policy assessments. For example, in recent years ASPE has initiated projects to develop cost estimates for health financing issues in general, and specifically for the Medicare and Medicaid programs; the effects of managed care expansion on public health infrastructure; welfare-to-work approaches; long-term care alternatives; and studies evaluating alternative services for children at risk of harm from drug abuse, crime, child abuse, and other pathologies.

Another continuing ASPE evaluation objective is to support and promote the development and improvement of data bases that HHS agencies and ASPE use to evaluate health care programs and health trends. For example, ASPE has been the major initiator on collaboration with the National Center for Health Statistics at the Centers for Disease Control and Prevention (CDC) of the first comprehensive survey of people with disabilities in the United States. The first component of these new data was completed in FY 1996, and national prevalence data on disability are now available. ASPE provides support to the HHS Data Council, which is charged with integrating key national surveys, such as linking health status indicators with indicators of well-being. The Department needs more comprehensive data sources to assess anticipated transformations in health and human services.

Finally, ASPE uses evaluation funds to promote effective use of evaluation-generated information in program management and policy making. The latter is accomplished through the dissemination of evaluation findings and other activities, such as providing technical assistance to agencies in the development of performance measures.

Summary of Fiscal Year 1996 Evaluations

Health Policy

• Health Care Technology Assessment--This project, also described in chapter II, contains current information on the demand for technology assessment. The study examined the performers of technological assessment, the methods of conducting assessments, the uses of the results, and the unmet needs that might be met by further cooperation between the public and private sectors of health care. In the past 5 years, the evaluation of clinical practice for its cost-effectiveness has assumed greater market value. The use of technology assessment in the managed care sector is primarily to support coverage decision making. This study also examined the appropriate role of the Federal Government in technology assessment and the appropriate division of labor between the public and private sector.
• Information to Guide Physician Practice--The purpose of this study was to understand the relationship between the increasing amounts of information that are being produced for physicians and the types of information that physicians actually use. In particular, this study assessed physician use of computerized health and medical information sources available, such as those on the World Wide Web and the Internet, and how access to such information influences medical practice. Among the findings are the following: (1) The wealth of information available to physicians causes them to aggressively screen information sources; this screening process depends significantly on the characteristics of the physician, on the financial and legal practice environment, and on the source and type of information being provided; (2) technology advances are making it more feasible for rural physicians to communicate with colleagues, but financial constraints prevent them from taking full advantage of resources; (3) potential barriers to use of the On-line Medical Networked Information resources include the fact that information resources are not of enough value to justify the time needed to access them.
• Assumptions Underlying Insurance Participation Modeling--This report provides background information to analyze policies designed to expand health insurance coverage, especially models that seek to estimate the impact of incremental health care reforms on the size and composition of the uninsured population. The study found that the number of uninsured persons increased from 30.5 million in 1979 to 39.6 million in 1994 (about one-third due to increases in the population); disagreement over estimates of the numbers of uninsured persons can be traced to several factors, including different survey methods and disparities between point-in-time and longitudinal estimates. The report concludes that the major contributor to the problem of uninsured workers lies in the area of insurance market supply, but the lack of demand on the part of some workers also contributes to the problem.

Disability, Aging, and Long-Term Care Policy

• Board and Care Homes--Nearly one million elderly or disabled individuals are dependent on personal services and supervision provided by board and care homes. ASPE carried out an analysis of the effect of State regulation on the quality of care provided in board and care homes in 10 States. The study, also described in chapter II, found that board and care residents are significantly older and more frail than a decade ago, making it a more complex challenge for board and care operators and staff. The study findings indicate that States with extensive regulatory and licensing requirements are more able to cope with this challenge on a number of dimensions including greater availability of supportive services in licensed homes, more operator training, and lower use of psychotic drugs and inappropriate medications.
• Subacute Care: Market Analysis, Cost, and Quality--There has been a huge increase in health care services provided in postacute care settings, as providers struggle to reduce inpatient hospital care. This exploratory study, also presented in chapter II, examined the extent to which subacute care appears to be a low-cost, appropriate alternative to hospital care. It found that even "state-of-the-art" subacute care programs had difficulty documenting their success in reducing costs and improving outcomes compared with traditional nursing home care. The study concludes that systematic evaluation is needed before the cost-effectiveness of subacute care is demonstrated.
• Research and Demonstration Strategy to Test Models for Integrating Acute and Long-Term Care for Children With Disabilities--This study documented "state-of-the-art" practices in developing managed care for children with disabilities and how States are coordinating managed care strategies with their Title V programs for children with special needs. The study concludes that there is little operational experience to guide States in developing managed care approaches for children with special needs, particularly programs that integrate acute and long-term care services. It also found that Medicaid managed care providers could benefit from the experience of Title V providers in serving special needs children as they create new managed care models.
• Americans With Disabilities--This project provided funds to the Census Bureau to analyze the 1990 Census questions on disability and to create a national baseline on disability data. Data were collected for the population age 16 and over on several types of disabilities: limitations in work, ability to work, mobility limitations, and self-care limitations. The tabulations are unique disability data for States, counties, and large metropolitan areas, including persons with disabilities living in the community and in institutions. The results--which include tables, graphs and maps--are available on the HHS home page available through the Internet at http://www.hhs.gov.
• Interdisciplinary Education and Training of Professionals Caring for Persons With Disabilities--People with disabilities generally use more health care services than people without disabilities and often require a range of both health and human service professionals. If managed care arrangements are to serve such populations effectively, providers must learn to integrate the services of a number of different professional disciplines. This study identified a number of promising programs where professionals receive interdisciplinary instruction and practicum experience in providing care to people with disabilities. It also found that few training programs address the needs of managed care organizations and that managed health care providers often lack the specialized knowledge and skills to serve people with significant disability.

Human Services Policy

• Alcohol and Other Drugs Treatment for Parents and Welfare Recipients: Outcomes, Costs, and Benefits--While alcohol and drug treatment have been shown to be effective for reducing crime and health care costs, very little is known about this relationship for welfare clients. This study analyzed the outcomes, benefits, and costs of substance abuse treatment for two overlapping groups: parents of children under 18 years of age and recipients of public income support such as Aid to Families with Dependent Children (AFDC). The study found that many people in treatment are both parents and welfare recipients. The data documented that the treatment benefits for these individuals exceeded the treatment costs.
• Coordinated Community Responses to Domestic Violence in Six Communities: Beyond the Justice System--Awareness of domestic violence has extended past the justice and domestic violence service systems to health care providers, child welfare and substance abuse service agencies, churches, and business. This study examined the coordinated responses to domestic violence in six U.S. communities. The study found that a strong community response requires well-coordinated policies and consistent application among community criminal justice agencies. A need was found for health care providers to develop screening protocols to identify women who are battered and to develop a system for referring victims to the most appropriate agency.

Special Populations

The second project was an effort to synthesize knowledge about the demography of American Indian and Alaska Native populations, with a focus on the major health problems they face and their use of health care services. The report, Changing Numbers, Changing Needs: American Indian Demography and Public Health, documents the difficulties in tracing demographic and public health trends for Native American populations. The report synthesizes data on population growth since the turn of the century, including information on fertility rates; reduction of infant mortality; persistence of high death rates for American Indian youths; declines in infectious diseases; and increases in rates of chronic disease, injuries, and alcohol and drug abuse. One analysis presented in the report showed that although IHS facilities are well suited to serve rural Native American population, problems continue with adequate access to care in rural areas and sufficient levels of budgeted resources.

Evaluations in Progress

Health Policy

• Monitoring Changes in Health and Human Service Programs--With the rapid changes in the health and human services systems, ASPE is studying the effects of State Medicaid waivers and changes in the State welfare laws on HHS programs such as Medicaid, foster care, and child support enforcement. ASPE evaluations will help define key measurements with which to monitor the effects of both program changes and changes in the environment, such as the impact of managed care on nonelderly individuals with disabilities, a project being done in collaboration with the Health Care Financing Administration (HCFA), and the effect of welfare-to-work programs on the well-being of children, a project being done in collaboration with the Administration for Children and Families (ACF) and foundations.
• Understanding Private-Sector Changes--ASPE's objective is to understand the private-sector environments in which HHS programs operate and to be able to describe and monitor the ongoing changes. A joint project with HCFA on managed care outcome measures and collaborative efforts with some of the Public Health Service (PHS) agencies on health promotion, prevention services, and benefits coverage in a changing private-sector marketplace were initiated during FY 1996. ASPE also continues to analyze data and information on health insurance coverage on an intramural basis.
• ASPE Data Initiative--This is a series of projects directed at filling gaps in the data HHS needs to analyze and monitor the health and well-being of the population, particularly the potential policy changes in health and human service programs. These gaps include State-level data; improved data on providers and the supply of health services; data that allow the measurement of child well-being; and better linkages between health and human services data.
• Performance Data--The Federal grants relationships to States in public health are evolving into outcomes-based performance management. A health outcomes-based monitoring approach requires data system development. ASPE is coordinating three projects with HHS agencies to accomplish this data development. One, with the National Academy of Sciences, is working to identify options for a performance-based approach that States and others want to achieve. The project will recommend specific steps that can be taken to improve State and local community data capabilities. A related project is developing approaches to obtain comprehensive baseline and trend data on public health infrastructure. A third study is building on a Robert Wood Johnson Foundation project to assess the quality of data that States collect and determine whether there is a foundation on which to build a multistate health data system that can be shared among States to improve health policy decision making.
• HHS Agency-Related Priority Health Services Evaluations--During FY 1996 ASPE has collaborated with HHS agencies on several program evaluations and technical assistance projects in program areas such as CDC's domestic violence initiatives, Indian Health Service alcohol treatment programs, the FDA's food additive review process, Medicare and Medicaid payment policies, and the Minority Male Consortium for Violence Prevention with the Office of Minority Health.

Disability, Aging, and Long-Term Care Policy

• Managed Care and People With Disabilities--Managed care can provide unique opportunities, but it also has potential pitfalls for people with disabilities. ASPE has projects under way to determine the impact of managed care on access, quality, and satisfaction for people of all ages who experience disabilities. Of particular interest are two policy questions. First is a concern about the impact of extending Medicaid managed care to the population eligible for Supplemental Security Income. Second, the impact of managed care on elderly persons with physical and cognitive impairments needs to be examined, and a comprehensive program for capitating acute and long-term care services for children with disabilities must be initiated and evaluated.
• Assisted Living--ASPE has commissioned a national study of assisted living, or the residential settings that combine adapted housing, assistive technologies, personal assistance, and other supportive services for persons with disabilities. Assisted living is considered an important component of services to the aging American population and an effective response to the rising costs of nursing home care.

Human Services Policy

• Evaluation of Family Preservation Services--ASPE and ACF are now examining family preservation programs aimed at preventing children from entering substitute care and reunification services to speed the return of children to their homes after they enter substitute care. Measures of program success will include prevention of placement of children into substitute care (for preplacement services), successful reunification (for reunification services), reduction of the recurrence of child abuse and neglect, improved child psychological well-being, improved child behavior, and improved family functioning.
• Moving Welfare Recipients to Work--ASPE is also working with ACF to conduct a comprehensive, multiyear study of the Government's principal program for moving people off welfare and into employment--the Job Opportunities and Basic Skills Training (JOBS) Program. The evaluation addresses the long-term effects of different welfare reform approaches, including whether any approach significantly improves the economic circumstances of the most disadvantaged recipients; the effects of welfare-to-work programs on the children of welfare recipients; and the cost-effectiveness of different approaches.
• Promoting Father Involvement--Three evaluations are looking at various aspects of paternal involvement, especially among fathers who are not living with their children. One project is examining the relationship between various child custody, visitation, and support payment patterns and the effect that payment and contact have on child well-being. The second is developing a design for the evaluation of community-based programs for vulnerable fathers. The third project will develop a theoretical framework to assist in understanding and implementing programs and activities to promote responsible fathering. The last two projects are being conducted in collaboration with ACF.
• Trends in the Well-Being of America's Children and Youth--In FY 1995, ASPE provided funds for the first annual report on the health and well-being of America's children. The report presented up-to-date trends on how our Nation's children are faring, pulling together information on all available national trends in the lives of children and youth. This volume fills a crucial gap and is now an annual series to keep the Nation apprised of the well-being of its children and youth. ASPE is developing the second annual report, to be released in 1997.

In addition, human services policy assessments are under way to examine child welfare and domestic violence policies, the health and mental health of immigrant children, and factors in the growth and decline of AFDC caseloads.

New Directions for Evaluation

Impact of Program Transformation

Managed Care and Disability

Impact of Welfare Reform

Other ASPE efforts to study the combined effects of changes in health care and income support policies on the public health system and its clients will focus on developing better State and national data on outcomes and the impact of the changes on the health and well-being of children and youth. Studies that address teen pregnancy prevention will build on work already undertaken and will examine how communities are addressing this issue as required in the new welfare law.

Quality-Related Studies

Improving Evaluation Data and Analytic Capability

Office of Public Health and Science

Evaluation Program

In keeping with its newly defined role within the Department, OPHS has specifically sought to develop an evaluation plan that avoids duplication of efforts that might more appropriately and effectively be undertaken by the operating divisions of HHS or by the Office of the Assistant Secretary for Planning and Evaluation (ASPE). Thus, the direction of the FY 1997 evaluation strategy for OPHS is toward public health and science issues that cut across multiple interests of the operating divisions and that require a coordinated approach to achieve the most effective results. In addition, OPHS commits itself to seeking to carry out every project proposed as part of this year's strategy in collaboration with relevant operating divisions.

As to its specific responsibilities, OPHS conducts evaluations requested of the Secretary by Congress that are most appropriately managed by staff with medical or public health sciences credentials, and that cannot be assigned to an operating division. Also, evaluations are conducted to support the Assistant Secretary for Health in his or her role as senior advisor for public health and science to the Secretary. This role includes the responsibility to "assist the Secretary in developing a policy agenda for the Department to address major population-based public health, prevention and science issues" and to provide "leadership and a focus for coordination of population-based health, clinical preventive services and science initiatives that cut across operating divisions." In addition, OPHS conducts evaluations specific to the needs of the program offices located within OPHS, such as Women's Health, Minority Health, Disease Prevention and Health Promotion, International and Refugee Health, and Emergency Preparedness.

Summary of Fiscal Year 1996 Evaluations

In an effort to study and learn from past experiences, the cost-effectiveness analysis (CEA) report provides an assessment of the current state-of-the-science of CEA methodology and makes recommendations about ways to improve the quality, comparability, and utility of CEA's. CEA is a powerful tool for evaluating which strategies best serve programmatic and financial objectives. Unresolved methodological issues are discussed in the context of factors that limit the policy relevance of existing CEA's. The CEA report is described in more detail in Chapter II.

A second methodological report, Measuring Expenditures for Essential Public Health Services, was completed on quantification of State public health expenditures. Working with State and local public health, mental health, substance abuse, and environmental agencies in nine States and with Federal PHS agencies, a methodology was developed and tested for estimating investments in 10 essential public health service areas. For all nine participating States, the total expenditure for essential public health services was $8.8 billion in 1994-95, of which $2.7 billion is population based. Estimates of public health expenditures are provided for agencies within the States and by service areas. In addition to estimating the relative investment in public health in the United States, the report lays the groundwork for a system to track these expenditures over time, across agencies, and across levels of Government. The resulting financial data can be related public health outcomes, infrastructure, and workforce measurements. An article about this study was published in Morbidity and Mortality Weekly Report on February 21, 1997.

A report on the health needs of newly arriving refugees evaluated the health assessment and follow-up care available to refugees, Cuban and Haitian entrants, and Amerasian immigrants from Vietnam during the first 8 months following their arrival in the United States. During this 8-month period, the Federal Government provides funding for health care coverage to refugees to prevent the spread of health conditions that could affect the public health and to identify health conditions that could impede effective refugee resettlement. The findings indicate areas of success, as well as areas needing improvement. For example, the flexibility of Federal funding has enabled States to fill gaps in their refugee health programs. In addition, statewide infrastructure for refugee health care has been developed in States that contract with local providers to deliver health-screening services. Concerns are raised about the accuracy and completeness of screening conducted outside of refugee-specific clinics, the reporting of refugee health screening results to the Centers for Disease Control and Prevention, and the availability of follow-up care to refugees.

Scientific misconduct is a topic of great interest to the public health and science communities. In its report on the consequences of being accused of research misconduct, the Office of Research Integrity (ORI) examined the impact of misconduct investigations on exonerated researchers. About 70 percent of all cases of alleged scientific misconduct that come to the attention of ORI result in exoneration. Yet little was known about the adverse consequences for the accused and the extent to which institutions comply with confidentiality requirements. The findings raise questions about the confidentiality of ongoing investigations. In a survey of 54 individuals who were accused of scientific misconduct and exonerated (prior to 1995), 60 percent reported one or more negative consequences of being accused of scientific misconduct. Of these, 90 percent indicated that the negative actions began during the inquiry or investigation and 65 percent reported that these actions continued after they had been exonerated. Yet only 39 percent viewed the impact of accusation on their careers as negative; 39 percent believed there was a continuing stigma attached to having been accused of misconduct. A similar 38 percent of the respondents were dissatisfied with the efforts of their institutions to restore their reputations.

Evaluations in Progress

The Commission on Dietary Supplement Labels, appointed by the President, is evaluating factors relevant to Food and Drug Administration regulation and possible legislation of label claims and statements for dietary supplements, including health claims as requested by Congress. A second nutritional study, undertaken by the National Academy of Sciences, is evaluating human nutrient requirements. The study will provide reference intakes, guidance about these values for clinical and public health use, and a listing of research needs.

An evaluation of the Cooperative Agreements for Demonstration Projects for Capacity Building at Historically Black Colleges and Universities (HBCU's) is under way. The setting will examine the extent to which the overall program and individual projects have resulted in the establishment of offices of sponsored programs and the adoption of uniform processes in participating schools. The evaluation will also assess how well the program has achieved increased funding for health-related research and training at the institutions and how effectively it has enhanced current research, training, and services.

The OPHS evaluation priorities related to strategic planning initiatives will focus on several areas. First is the organization of the Healthy People 2000 objectives. To promote an integrated public health message, OPHS is consulting a broad range of members of the Healthy People 2000 Consortium and using focus groups to evaluate the basic framework of national health goals and objectives in preparing for the Healthy People 2010 Plan. The study will collect insights on the current functions of the objectives framework and explore opportunities for making them more relevant to the Healthy People 2000 Consortium members, State and local agencies, managed care industry representatives, and Fortune 500 major purchasers of health care plans.

School health programs, both comprehensive and categorical, continue to be developed and implemented throughout the Nation. OPHS has funded a study to identify evaluations of school health programs and make available an updated compilation of methodologically sound studies and their findings to assist school boards, administrators, health personnel, and health educators trying to maximize limited resources for effective health programs.

OPHS will continue to expand efforts to improve nationally available data on the public health infrastructure, focusing on local public health capacity and workforce issues. The results will provide local health officials and planners with a validated instrument for estimating the level of support for essential public health functions. This local study of expenditures represents the community analog to the State study funded in FY 1996.

In the area of health data and the environment and in continuance of support for public health infrastructure revitalization, OPHS funded development of two data-tracking systems, one related to sentinel public health indicators, health outcomes, levels of health risks, and the health protection infrastructure; and the second on State and local environmental health data needs and sources of State and local environmental information. OPHS also funded an assessment of health plan involvement in community-based initiatives and a survey of clinicians to assess the level of provision of and the importance placed on preventive clinical services by primary care providers,

OPHS supports several projects to assist in improving program operations and management. These include an evaluation of a new model of coordinating and integrating HIV-prevention messages and primary care in high risk populations, an evaluation of the Minority Health Resource Center, and the assessment of the Adolescent Family Life Program.

The implementation of the GPRA of 1992 is a priority for OPHS. The development of performance measures for public health programs will be crucial for program planning, budgeting, and legislative development. OPHS, in collaboration with the Office of the Assistant Secretary for Planning and Evaluation, has targeted efforts in three program areas to develop these measures. First, it is supporting completion of the consultation begun in FY1995 with State and local health officials concerning the development of performance measurement systems that can be used to improve accountability for expenditure of Federal funds. OPHS will assist States and local governments in determining the impact Federal programs are having on improving the health status of Americans. OPHS will also discern the extent to which States and selected counties and cities have current data available to measure their own State year 2000 objectives and selected national objectives.

Two communication studies are also under way. The first focuses on the design and evaluation of interactive communication applications for consumer health information. A second study examines the dissemination of public health information from PHS agencies.

Finally, the Office of Minority Health is examining the efficiency and effectiveness of its Bilingual/Bicultural Service Demonstration Grant Program, which is intended to build the capacity of community-based organizations to address access to health services for limited-English-speaking minority populations.

New Directions for Evaluation

The shifting emphasis on managed care presents new opportunities coupled with new dilemmas regarding how best to ensure that, in the quest to manage the cost of care, overall quality of care is not unduly compromised. It raises new questions of how to ensure a workforce that is properly trained and fully capable of performing the essential services of public health to increasingly diverse populations in this dynamic health care environment. Furthermore, transformations in the health care and public health arena necessitate the monitoring and assessment (and the data systems to do so) of the effects and impacts of these changes on the financing, organization, and availability of population- and clinic-based preventive services and medical treatment.

OPHS has given highest priority in its FY 1997 strategic plan and evaluation efforts to addressing these issues. The FY 1997 OPHS strategic plan emphasizes the need for a strong foundation for public health in the 21st century. The priorities focus attention on (1) the information systems and workforce that comprise the Nation's public health infrastructure and that are necessary to effectively provide the essential services of public health; and (2) the impact of managed care arrangements on the resources available for this infrastructure and on the health of all Americans, especially those most vulnerable.

Future OPHS evaluation planning efforts will reflect the OPHS strategic plan, as well as the broader HHS programmatic priorities related to improving the health of all Americans and assessing the effects of health-sector transformation. Planned investigations in support of the public health infrastructure include an assessment of response to health and medical consequences of chemical and biological agent incidents; an evaluation of summary measures of health status; an evaluation of the electronic gateway to consumer health information; analyses of international year 2000 plans to inform the United States effort and the WHO Renewing Health for All strategy; and an examination of outreach efforts and potential uses of Healthy People Objectives for the Year 2010.

Other planned evaluation activities will focus on addressing specific health needs of our most vulnerable populations. These include evaluations of strategies and approaches for addressing the health of racial and ethnic minorities in the United States as compared with the United Kingdom, cultural competence in health care to address issues of measurement and linkages to health and patient outcomes, an inventory of State data on women's health, an inventory and assessment of linguistically and culturally appropriate services in managed care outreach organizations and an assessment of minority consumer experiences with State outreach activities related to Medicaid managed care.

Last, in response to the increased need to measure the performance and effects of public health programs and activities, OPHS will support efforts to enable performance measurement activities among local, State, and Federal health agencies.

Substance Abuse and Mental Health Services Administration

Evaluation Program

To the greatest extent appropriate and feasible, SAMHSA will encourage the use of comparable data elements and instruments across its evaluations in order to work toward a comprehensive evaluation system and to minimize respondent burden.

SAMHSA conducts grant programs under a variety of legislative authorities. These authorities can generally be grouped into two types: (1) services and (2) knowledge development and application (KDA). The evaluation required for a particular grant program is dependent on the type and purpose of the program. SAMHSA will evaluate each of its service programs and provide information to program managers about the accountability of Federal funds. In addition, the evaluations of KDA programs will generate new knowledge to lead the field in the development of policies that improve services. Program and evaluation staff must work together to identify clearly the questions or goals each grant program will address and to propose appropriate evaluation strategies.

The two types of grant programs (service delivery and KDA) represent the two facets of SAMHSA's mission. SAMHSA's leadership in the field depends on the successful interaction of these two facets of its mission. Through evaluation, SAMHSA must identify effective approaches to prevention, treatment, and rehabilitation. Through service delivery funds, SAMHSA must provide incentives to the field to implement effective approaches. Major emphases of SAMHSA's mission are to develop, identify, and disseminate effective strategies and systems for prevention, treatment, and rehabilitation.

SAMHSA is implementing an integrated model of evaluation and planning. Strategic planning identifies priorities, such as managed care, that drive the development of grant programs and evaluations. In compliance with the Government Performance and Results Act (GPRA), SAMHSA is attempting to improve performance by identifying performance goals associated with its strategic plan. The formulation of programmatic and evaluation priorities includes consultation with the SAMHSA and Center Advisory Councils and with other experts in the fields of evaluation and service delivery. Early and continuous coordination of program planning and evaluation design will result in the articulation of program objectives that can be evaluated. Evaluations will show how well the overall grant programs have achieved their objectives, and SAMHSA will translate these results into information that can be used for program and policy development. The strategic planning and policy development processes will then use these results to refine SAMHSA's priorities and objectives.

This evaluation policy will help SAMHSA achieve its goal of continually informing policy and program development with knowledge culled from past performance. In this way SAMHSA can best serve its customers by enhancing the quality of public substance abuse and mental health services.

In compliance with Public Health Service (PHS) guidelines for the technical review of evaluations, SAMHSA has established a standing committee of PHS staff who are evaluation specialists. Representatives of the Office of the Assistant Secretary for Planning and Evaluation serve as ex officio members of the committee. The SAMHSA evaluation officer is the committee chair. The committee does not generally review the evaluation proposals of individual grantees. It does review proposals for broader, more comprehensive evaluations, such as the cross-project evaluations of grant programs.

Evaluation project proposals are generally prepared by SAMHSA program staff in the various Centers. The standing committee reviews each proposal on the following criteria: clarity of evaluation objectives and research questions; appropriateness and feasibility of the specifications for evaluation design and methods; appropriateness of the plans for dissemination of results; and use of previous relevant evaluations and existing program data systems. Each proposal must clearly state the relationship of the evaluation to SAMHSA's overall policies, priorities, and evaluation program.

Summary of Fiscal Year 1996 Evaluations

Also during FY 1996, SAMHSA completed a report entitled Employment Outcomes of Indigent Clients Receiving Alcohol and Drug Treatment in Washington State. This report was prepared by researchers at the University of Washington and the Washington State Division of Alcohol and Substance Abuse. It reports results from a project that evaluated substance abuse treatment outcomes for indigent clients served in Washington State. The treatment group included 499 individuals in various treatment modalities. The comparison group included 168 clients who were assessed but did not initiate treatment. Analyses were conducted on employment outcomes, using data obtained from official State employment records. The employment data cover a 12-month period before treatment and an 18-month period after treatment.

The study found that the duration of treatment is an important predictor of employment outcomes. Clients staying in treatment longer experience better outcomes, as do clients who complete treatment. A second finding was that the benefits of treatment, in enhanced earnings, compare favorably with the costs of treatment.

The policy implication of the study is that less treatment may not necessarily be less costly in the long run. Limiting coverage for substance abuse treatment for public clients will reduce short-run treatment costs, but may compromise important benefits of treatment, such as enhanced employment and increased earnings.

Evaluations in Progress

Program Accountability

Knowledge Development and Application

The National Treatment Improvement Evaluation Study is a cross-site evaluation study examining the effectiveness of demonstration grants funded by the Center for Substance Abuse Treatment. The purpose of the study is to assess the extent to which treatment enhancements improve substance abuse treatment outcomes over time. Preliminary results from this study provide strong support for the efficacy of substance abuse treatment in reducing levels of substance use, reducing involvement in criminal activities, and increasing employment. The final report will be available in FY 1997.

Managed Care

New Directions for Evaluation

One SAMHSA evaluation priority is knowledge development and application. SAMHSA's KDA program is designed to answer specific, important, policy-relevant questions. These questions will be designed to provide critical information to improve the Nation's mental health and substance abuse treatment and prevention services. Evaluations of the current KDA program will generate knowledge on such topics as the relative effectiveness of alternative models of managed care for treatment and prevention of substance abuse and mental disorders, the relative effectiveness of alternate models for preventing homelessness among adults with serious mental illnesses, and the efficacy of a brief intervention for marijuana dependence.

Another of SAMHSA's evaluation priorities is to assess the impact of managed care on the availability of services to populations in need, and to develop standards for measuring quality and outcome. SAMHSA's managed care evaluation strategy is designed to leverage important knowledge that directly affects the agency mission. This knowledge will be acquired through focused applied health services research, evaluations, demonstrations, and epidemiological and service capacity studies.

Return to Table of Contents.