| |
Office of the Assistant Secretary |
Policy Information Center |
Each of the 11 agencies of the Department of Health and Human Services (HHS), the Office of the Assistant Secretary for Planning and Evaluation (OASPE), and the Office of Public Health and Science (OPHS) in the Office of the Secretary maintains its own evaluation program--including the functions of evaluation planning and policy review, quality assurance through technical review, project coordination and management, dissemination of reports, and utilization of results. This Chapter provides an agency-by-agency summary of the evaluation activities. Each summary begins with an overview of the relevant evaluation program, describing its philosophy, policies, and procedures. Next is an overview of the major evaluations completed in fiscal year (FY) 1997 and the evaluations that are currently in progress. Abstracts and contact persons for all evaluations completed in FY 1997, organized by agency, are presented in Appendix A. Listings of all evaluations in progress (by agency) are located in Appendix B.
Chapter III Divisions
Administration for Children and Families
Agency for Health Care Policy and Research
Agency for Toxic Substances and Disease Registry
Centers for Disease Control and Prevention
Health Care Financing Administration
Health Resources and Services Administration
Office of the Assistant Secretary for Planning and Evaluation
Office of Public Health and Science
Substance Abuse and Mental Health Services Administration
ADMINISTRATION FOR CHILDREN AND FAMILIES
MISSION: To promote the economic and social well-being of families, children, individuals, and communities.
Evaluation Program
The Administration for Children and Families (ACF) administers a broad range of entitlement and discretionary programs, including income maintenance (Temporary Assistance for Needy Families [TANF]); child support; children and family services (Head Start, Child Welfare, Family Preservation and Support, and youth programs); four block grants; and special programs for targeted populations, such as the developmentally disabled, immigrants, and Native Americans.
The objectives of ACF's evaluations are to furnish information on designing and operating effective programs; to test new service delivery approaches capitalizing on the success of completed demonstrations; to apply evaluation data to policy development, legislative planning, budget decisions, program management, and strategic planning and performance measures development; and to disseminate findings of completed studies and promote application of results by State and local governments.
ACF actively engages with other Federal agencies, State and local policy and program officials, national organizations, foundations, professional groups and practitioners, and consumers to stay current on emerging issues affecting its programs and to identify questions for evaluation studies. Systems changes and how they affect vulnerable populations, particularly children, are of primary concern. The movement toward devolving responsibility for health and human services to State and local organizations--in particular, the Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA) of 1996--offer both tremendous opportunities and unprecedented challenges in redefining and implementing services for families. Evaluation study designs are negotiated carefully with the States and other interest groups. Studies often are funded as joint ventures with Office of the Assistant Secretary for Planning and Evaluation and other Federal agencies and foundations. Such collaborations permit large-scale efforts that are better informed and more representative of varying perspectives. Proposals are reviewed by multidisciplinary experts. Work groups of various kinds are used to monitor the progress of projects and to advise on design refinements and the presentation of findings.
Summary of Fiscal Year 1997 Evaluations
ACF's evaluations are linked closely to its two strategic goals:
During FY 1997, ACF completed 19 evaluations--4 pertaining to the economic independence and productivity of families, and 15 related to the healthy development of children and families.
Economic Independence and Productivity of Families
LEAP: Final Report on Ohio's Welfare Initiative to Improve School Attendance Among Teenage Parents--Ohio's Learning, Earning, and Parenting Program (6668), highlighted in Chapter II, found that (1) even the initial implementation of LEAP was a considerable achievement as an unprecedented statewide effort requiring new linkages between county welfare and education systems; (2) the program achieved smooth operations after implementing a sophisticated statewide public assistance computer system; and (3) to a large extent, LEAP met its initial goals by significantly increasing school enrollment and attendance for in-school teens and dropouts (these outcomes were directly linked to the incentives and sanctions provided); (4) in general, the impacts on school completion and employment were more positive for teens who were still in school when they were identified as eligible for the program (initially-enrolled participants) than for dropouts who returned to school; and (6) LEAP increased the employment rates for initially-enrolled teens and had positive impacts on their earnings during the first 2 years of the 4-year followup.
Responsible Fatherhood: An Overview and Conceptual Framework (5981) found that fathering, even more than mothering, is influenced by contextual forces in the family and the community. A father in an adversarial relationship with the mother is at risk of becoming an irresponsible father, as is a father who lacks adequate employment and income. Responsible fathering can be fostered by positive changes in cultural, economic, institutional, and interpersonal influences. The report contends that fathering programs should involve a wide range of interventions, including (1) involving the mother where feasible and, for unmarried fathers, the families of origin; (2) promoting collaborative co-parenting; (3) emphasizing critical transitions, such as the birth of a child and divorce of the parents; (4) dealing with employment, economic issues, and community systems; (5) promoting father-to-father learning; and (6) promoting the visibility of committed and collaborative marriage. By developing a theoretical underpinning to guide empirical research, program development, and program evaluation, this project is intended to help inform policymakers about what is necessary to enable a father to support and nurture a child.
The Self-Sufficiency Project Implementation Manual: Lessons Learned from Eight Years of Office of Community Services Demonstration Partnership Programs (4336.4), featured in Chapter II, summarizes lessons learned during eight years of Demonstration Partnership Program (DPP) projects. DPP's purposes have been to stimulate the development of innovative approaches to increasing the self-sufficiency of the poor, to test and evaluate these new approaches, and to encourage their replication through dissemination of project results and findings. These projects were designed to strengthen the ability of grantees to integrate, coordinate, and redirect activities through community partnerships that promote maximum self-sufficiency among low-income individuals and families who rely on or are at risk of relying on public assistance. The report divides the projects into five program areas: (1) case management and family development, (2) micro-enterprise and self-employment, (3) minority males, (4) homelessness, and (5) youth at risk. Generic models for establishing effective community-based programs are presented, and materials for program evaluation are elaborated. For each of the five program areas, general lessons learned are presented, followed by specific lessons learned pertaining to project design, project start-up, project operations, and project evaluation. The project implementation manual was designed as a step-by-step guide to the successful design, implementation, and evaluation of self-sufficiency projects by community action agencies, community-based organizations, and local community program planners. It is also meant to serve as a critical reality check for legislators and policymakers as they strive to design laws and programs that will use scarce resources more effectively to foster self-sufficiency and build sustaining capacity within low-income families and communities.
The Arkansas Prenatal and Postnatal Paternity Acknowledgment Project (6221) used five basic models: (1) basic in-hospital paternity acknowledgment with a genetic testing option; (2) prenatal and multiple postnatal opportunities to acknowledge paternity; (3) interstate paternity acknowledgment transfer project (i.e., parents from one State traveled to another State, gave birth, and signed paternity acknowledgments in that State); (4) home-visiting nurse program; and (5) alternative parental support program, which abates or reduces child support payments for young fathers who participate in opportunities to increase parental responsibility. First, the study found that mothers, identified by child support staff to be uncooperative in establishing paternity, stated that they did not want to establish paternity because either the child's father was not involved and that the father gave the mother money already, or that the mother did not know where the father was living. By the end of the project, 13,688 paternity acknowledgments had been received, and the ratio of paternity acknowledgments to unmarried births ranged from 48 percent in 1995 to 56 percent for the first 3 months of 1996. The study also found that 38 percent of the acknowledgments were matched to children in the Arkansas child support data base, and 27 percent of these had closed by September 1996. Payments had been received on only 16 percent of matched cases.
Healthy Development of Children and Families
The Descriptive Study of the Head Start Health Component (5849) describes health screening, examination, referral, treatment, and followup procedures across the four health domains of medical, dental, nutrition, and mental health. The researchers reviewed the case records of 1,200 4-year-old children enrolled in a national sample of 40 randomly selected Head Start programs and interviewed parents regarding their child's health status and health service utilization patterns. Through interviews with appropriate Head Start personnel, descriptions were obtained on Head Start program staffing patterns and staff training, the use of community health resources, and barriers to providing health services.
The First Progress Report on the Head Start Program Performance Measures (6693) is the first program assessment using performance measures to promote accountability through the assessment of program quality and outcomes. It describes the development of the performance measures, noting their focus on outcomes rather than on processes. The outcomes that will be measured concern how well the children in the program do, rather than on such things as teacher credentials. The Head Start program also has in place a series of Program Performance Standards-- which define program activities--while the Program Performance Measures define program results. The report describes the background of the Program Performance Measures development, including the conceptual model and data sources. The report includes an account of how the Head Start program will gather data for the measures and establish a "feedback loop" for policy and resource decisionmaking. The report concludes that the performance measures will provide a snapshot of the Head Start's program performance at a given point in time and will allow the program to compare its performance to the past. At the national level, it will allow the program to determine how well it is doing through the production of periodic national Head Start progress reports. This first report will serve as a benchmark against which future performance can be measured and furnishes program accountability information to be used in appropriate decisions.
The National Study of Protective, Preventive, and Reunification Services Delivered to Children and Their Families (3874) assessed the numbers of children and families in the child welfare system who are receiving protective, preventive reunification, out-of-home care, and/or aftercare services. It also obtained national data on the number, types, and dynamics of the services provided. Case record abstracts were completed on a nationally representative sample of 3,000 children and their families served by public child welfare agencies. A subsample was followed for a 9-month period. These findings were compared with findings from a previous study, the 1977 National Study of Social Services to Children and Their Families. The study found that (1) between 1977 and 1994, there was a dramatic decline in the number of children receiving child welfare services, reflecting the evolution of the child welfare system from a broad-based social services system to one primarily serving abused and neglected children and their families; (2) the child welfare system has not evolved into an in-home family-based system from a foster care system, as had been envisioned by the Adoption Assistance and Child Welfare Act of 1980; (3) foster care drift remains a problem, with more than one-third of the children in foster care remaining there for more than 18 months; (4) minority children, particularly African-American children, are more likely to be in foster care placement than to receive in-home services, even when they have the same characteristics and problems as white children; and (5) kinship care does not explain the dramatically longer stays in foster care for African-American and Hispanic children compared to white children.
The Examination of Special Needs Adoption in New York State--Phase III Report: Subsidized Adoptions in New York State, 1989-1993 (4380.1) was initiated in order to pilot a statewide data collection system for special needs adoptions. The study analyzed information on the characteristics of adopted children and their adoptive families, looking at procedural issues and the nature and amount of subsidy payments provided to the adopted child. This was done for the entire population of children being placed in New York by public and private agencies. The Phase III report, one of five phase reports, contains a descriptive analysis of statistics on the number of children placed between 1989 and 1993 in New York State, the level of this support, and the nature of the children's special needs. The report found that (1) 12,858 new adoption subsidies were approved in New York State during the study period; (2) 62 percent of children receiving an adoption subsidy in the State qualified because they were hard to place; (3) personality or behavioral problems were the most prevalent reason for a child's classification as handicapped (about one-fourth of children qualified for a subsidy had physical handicaps or severe medical conditions); (4) more male than female children were adopted with a subsidy in New York State; and (5) about one-half of all special needs children are placed in adoptive homes headed by a single mother.
Eleven evaluations addressed the issue of child maltreatment. First, Child Maltreatment 1995: Reports from the States to the National Child Abuse and Neglect Data System (5387.4) found that (1) in 1995, more than 1 million children were identified as victims of abuse or neglect (a rate of 15 per 1,000 children); (2) about 80 percent of the perpetrators of child maltreatment were the parents of the victims, while another 10 percent were other relatives and about 2 percent were people in other caretaking roles; (3) 52 percent of victims suffered from neglect, about twice as many as were subjected to physical abuse, the next most frequent kind of maltreatment,(25 percent); (3) about 13 percent of victims were sexually abused; (4) more than half of all victims were 7 years of age or younger, while about 26 percent younger than 4 years old (about 21 percent of victims were teenagers); (5) the majority of victims of neglect and medical neglect were younger than 8 years old, while most victims of other forms of maltreatment were older than 8 years old; and (6) 45 States reported that 996 children were known by the child protection services (CPS) agency to have died as a result of abuse or neglect--most of these deaths were children 3 years of age or younger. The report also found that (1) CPS agencies investigated nearly 2 million reports alleging maltreatment of about 3 million children (a rate of 43 per 1,000 children); (2) reports were received from professionals (53 percent), family members (19 percent), friends and neighbors (9 percent), and anonymous sources (19 percent); and (3) nationally, about 36 percent of investigations resulted in a disposition of either substantiated or indicated maltreatment, while 58 percent of allegations were not substantiated.
Evaluation of Nine Comprehensive Community-Based Child Abuse and Neglect Prevention Programs: Cross-Site Evaluation Report (5851) is a synthesis of the site evaluations that were part of ACF's comprehensive community-based child abuse and neglect prevention programs. The purposes of the study were (1) to design and implement a process and impact evaluation of nine comprehensive community-based child abuse and neglect prevention projects; (2) to provide technical assistance to the nine projects in meeting the requirements of the evaluation; and (3) to aid the programs in designing and implementing their own internal program evaluations. The evaluation was conducted in three phases over a 3-year period. The nine projects had up to ten service components, many of which differed across the programs. Therefore, individual experimental designs were developed for each service component. Process and impact data were collected across programs and through a series of studies conducted in each site. All grantees were aided in refining their evaluation and research plans. The report provides a context for understanding the experiences of the nine projects. A literature review, an examination of study methodology, the projects' implementation experiences, an accounting of the study findings, and policy recommendations are offered. The report found several program elements vital to the success of the projects, including (1) an emphasis on community involvement and ownerships, (2) employing a positive approach, (3) starting on a small scale, and (4) implementing a strong evaluation and using it as a program management tool. The report recommends that the National Center on Child Abuse and Neglect (NCCAN) implement several strategies to enhance the success of these kinds of projects, including (1) focusing future grant programs on more narrowly defined target populations, (2) stressing the importance of community involvement, and (3) providing grantees with a research framework and priorities delineating key research questions on child maltreatment and requiring grantees to implement appropriate process and outcome evaluation designs.
Described below are the results of the individual evaluations of the nine comprehensive community-based child abuse and neglect prevention programs. Preventing Child Abuse and Neglect: A Case Study of Family Care Connection (5851.3) describes a replication in several high-risk communities of a preexisting program in Allegheny County, Pennsylvania, using community-based parent education to improve parent-child relationships. Family Care Connection (FCC) expanded services at an existing drop-in center and established four centers in three more at-risk communities. The centers provided such services as respite care, child development, recreation, education, and counseling. An evaluation found that (1) 279 parents had graduated from the parenting classes from September 1989 through June 1991, and that these graduates reported that they would be less likely to use physical punishment with their children; (2) the rate of low birthweight babies decreased after program implementation in two communities; and (3) FCC was successfully established as an institution in the communities through aggressive pursuit of funding and by using staff that were hired and paid by "partner agencies." The report concludes that FCC achieved notable success in institutionalizing its services in the communities in which it established drop-in centers under the NCCAN grant. FCC carefully selected community agencies as partners, and established personal contacts and networking connections with agencies and organizations in its target communities.
The report of Preventing Child Abuse and Neglect: A Case Study of Families First in Fairfax (5851.1) found that (1) the program being operated in Fairfax County, Virginia, at the end of the grant period differed substantially from the one originally proposed: a number of child abuse prevention strategies attempted during the first two years were ultimately discontinued; (2) the greatest barrier to program implementation during the first two years was a lack of specificity on how to achieve the program's mission; (3) during the last two years, program development was dynamic and yielded many effective strategies that were made permanent at the end of the demonstration period; (4) the demonstration program fostered a growth in parenting programs, family and early infant health care programs, neighborhood resource centers, and directories of services made available to ethnic minority populations; (5) the program also encouraged increased collaboration and information-sharing between the agencies and organizations working in the area of child abuse and neglect prevention; and (6) the biggest problems during the 5-year demonstration period were frequent leadership changes and staff turnover.
The report, Preventing Child Abuse and Neglect: A Case Study of Community Coalition Acting for Positive Parenting (5851.2), developed by Temple University's Center for Social Policy and Community Development, found that the project (1) affected three different groups: the community, child abuse professionals, and the University, allowing members of each group to learn from members of other groups and fostering an expanded community cooperation; (2) sparked the institutionalization of community-based efforts to support families and inform the community about its role in preventing child abuse; (3) created a community-based planning and development council or community board through which key leaders can both receive and disseminate information; (4) shifted its attention away from child abuse and neglect to a more positive, family-support approach by fostering Family Life Festivals, library programs, and parent-child activities, which reduced parental isolation; (5) using community-based providers as agents for prevention activities enabled the program to attract more of the target population; and (6) universities that have experience working with social issues facilitate program implementation and lend legitimacy to community-based efforts.
Preventing Child Abuse and Neglect: A Case Study of Community Lifelines Program (5851.4) describes a program developed by the Family Life Development Center of Cornell University in collaboration with the government and human service agencies of Chemung County, New York. The program was designed to alleviate community conditions that lead to isolation, poor self-image, and economic stress, thereby reducing some underlying causes of child abuse and neglect. Community Lifelines Programs (CLPs) were responsible for various program initiatives in the city of Elmira and the rural Van Etten/Spencer school district. The initiatives included Parent Partner Programs (PPPs), parent support groups, activities encouraging better parent-child communication, and family support programs. The report describes the community, the grantee agency, the program design and changes over time, barriers to program implementation, and strategies used to overcome these barriers. It also describes the program effects and efforts to institutionalize CLP components. The report finds that the Elmira PPP was very successful, as was the CLP in Van Etten/Spencer. In Elmira, the PPP fostered better relationships between schools and parents, between parents, and between parents and children. In Van Etten/Spencer, several initiatives succeeded, including a program for parents of teenagers (Parent Lifelines), an after-school program, and a playgroup. However, several other initiatives had little impact or started too late in the demonstration period to be measured. These included the Family Portraits component, the Family Connections Room, and Food Stamps Outreach. The public awareness campaign launched in conjunction with CLP was successful and useful, according to a reader survey.
The report of the project Preventing Child Abuse and Neglect: A Case Study of PARE (Physical Abuse and Neglect Reduction Effort) (5851.5), developed by the Exchange Club Center for the Prevention of Child Abuse in Carolina, Puerto Rico, discusses the environmental factors and indicators of social disruption associated with increased risk for child abuse and neglect in Puerto Rico. It provides a comprehensive overview of all aspects of the PARE program design and operations, as well as evaluation findings. PARE provided intervention strategies in an effort to stop future child abuse while preserving the family. PARE was designed to be family-oriented and to emphasize self-help and volunteer action. The demonstration model emphasized interventions directed towards the individual, the family, and the community, as well as cultural factors. The report describes several interventions, including (1) an interagency task force, (2) a public awareness campaign, (3) respite centers, (4) a life skills curriculum, (5) an educational curriculum for prenatal clinics, (6) a parent aid program, and (7) a parent laboratory. The report finds that (1) the development and implementation of three respite centers pioneered new concepts and approaches to providing secondary prevention; (2) 702 volunteers provided support services and served as role models; (3) the program gained the support of the community, government officials, public and private agencies, the business community, and the media; and (4) the public awareness campaign was broadcast throughout Puerto Rico and reached a large segment of the population. The report concludes that PARE increased child abuse awareness in Puerto Rico, but that more work needs to be done before child abuse prevention becomes a priority for leaders and politicians on the island.
Preventing Child Abuse and Neglect: A Case Study of North Lawndale Family Support Initiative (5851.6) describes a project developed by the Greater Chicago Council of the National Committee to Prevent Child Abuse to test the efficacy of a model that advocates comprehensive prevention strategies aimed at both the community and its residents. The model includes community education, parent education, parent support groups, school-based prevention services for children, and therapeutic services for abused children and their families. The evaluation found that (1) the most successful components of the North Lawndale Family Support Initiative (NLFSI) were the advisory council, the community needs assessment, public awareness, and life skills training; (2) the least effective components were the parent education program, the parent support groups, and therapeutic care--this ineffectiveness was due to inadequate program design, local implementation difficulties, and the grantee's perception that NCCAN required more direct services; (3) the NLFSI benefitted from stable and credible staff that had already established themselves in the community before the program's implementation; (4) although staff possessed strong outreach skills which served them well in the public awareness campaign, but lacked the skills and background needed to implement direct services; and (5) the NLFSI lacked a clearly defined or effectively developed relationship with its grantee Agency and had difficulty in being acknowledged as a full-fledged member of the community, due to its situation on the campus of a community college located at the edge of the target community. The report also discusses barriers to program implementation and efforts to institutionalize the program, and includes several recommendations in key areas.
The report, Preventing Child Abuse and Neglect: A Case Study of Dorchester CARES (5851.7), describes a collaborative project of the Massachusetts Committee for Children and Youth, Inc., Federated Dorchester Neighborhood House, and, by the end of the demonstration period, several other collaborators. The model was designed to demonstrate that through collaboration, advocacy, resource development, education, and services, a community can create opportunities for families to increase their social networks and become more independent and self-sustaining. The report found that (1) Dorchester CARES staff stressed that it was not an agency or a program, but rather a collaborative comprised of member agencies and staff who thought of themselves as a process; (2) the project reflected an ecological approach that combined a psychiatric model and a sociological model (the first was directed toward a "sick parent" and stressed therapy and education, while the second was aimed at the "sick society" and stressed educational and political strategies); (3) over the demonstration period, the program improved family functioning to provide a more nurturing environment for children's development; (4) the program created opportunities for family members to become more independent and self-sustaining and for community members to work together to achieve common goals; (5) the CARES family support program worked by proactively encouraging social networking, increasing nurturing values and skills, and empowering parents to carry out their caregiving roles; and (6) the program was successfully duplicated in several other neighborhoods and was institutionalized in Dorchester.
Preventing Child Abuse and Neglect: A Case Study of Maine Families (5851.8) describes a collaboration between the urban Cumberland County Child Abuse and Neglect Council (CCCANC) and the rural Franklin County Children's Task Force (FCCTF). Other local community, school, and service organizations joined the two county organizations to identify and address the needs of children and their families. CCCANC implemented a school family center, a media program, a public library discussion series, a teenage parent day care and support center, a drop-in laundry program, a parents' speaker and support group, and a parent education program in a low-income housing development. FCCTF implemented a summer reading program, a resource directory, a parent education program for Head Start-eligible families, parent cooperative support and education groups, and a parent mentoring and home visitation program. In both counties, the project supported community events to reduce the isolation of families and to encourage a sense of support and togetherness. The Maine Families Project evaluation was primarily a qualitative assessment of the reactions of program participants and leaders. Major findings were that (1) the project was guided by an effort to structure interventions based upon what parents said they needed, rather than on what project staff or other agency staff believed they needed; (2) focus groups revealed that few parents believed that they were doing an effective parenting job, and most felt isolated from support and in need of information and assistance; (3) the project emphasized involving business and industry in designing and implementing community programming; and (4) collaboration is enhanced by using a businesslike approach.
Preventing Child Abuse and Neglect: A Case Study of I CARE (5851.9) describes a project developed by the Ohio Research Institute on Child Abuse and Prevention to model a prevention program in direct response to the needs and existing resources of the target community in Columbus, Ohio. The conclusions and recommendations were that (1) a prevention model containing all the components specified for this demonstration would be difficult, if not impossible, notwithstanding an annual budget of $200,000; (2) although community-based programs cannot correct all the societal problems leading to child abuse and neglect, it should be aware of their effects; (3) an interdisciplinary task force from the community should be considered mandatory; and (4) the uncertainty of future funding is a major obstacle for demonstration or pilot programs.
Evaluations in Progress
ACF has 71 evaluations in progress, 43 of which relate to the economic independence and productivity of families and 28 to the healthy development of children and families.
Economic Independence and Productivity of Families
Policymakers and program operators recognize that job loss among newly employed welfare recipients is a major barrier to the more general goal of increasing the rate at which recipients obtain employment and leave welfare. A high proportion of welfare recipients who become employed lose their jobs and return to public assistance, often within a few months. The four-site Post-Employment Services Demonstration (5974) addresses this important aspect of a comprehensive welfare-to-work strategy. Demonstrations in Riverside, California; Chicago, Illinois; Portland, Oregon; and San Antonio, Texas; are providing job retention and re-employment services to recently employed Job Opportunities and Basic Skills Training (JOBS) program participants randomly assigned to receive additional services, which will be provided regardless of continued AFDC receipt. Each site will enroll approximately 500 participants.
The Home Visiting Services Demonstration (5980) evaluation is assessing the effectiveness of the Home Visiting Services Demonstration in three sites--Chicago, Illinois; Dayton, Ohio; and Portland, Oregon--in which first-time teenaged parents on AFDC are required to participate in the JOBS program. Those randomly assigned to receive the treatment are assigned a paraprofessional home visitor. The demonstrations are testing whether adding weekly home visitor services to mandatory JOBS programs will result in a substantial strengthening of the effectiveness of JOBS programs in helping young mothers better support themselves and their children while promoting positive parenting and reductions in repeat childbearing.
The Prenatal and Early Childhood Nurse Home Visitation Program--Replication/Dissemination (6757) project allows for data collection and evaluation of an initiative to replicate and disseminate the nurse home visiting model. The model has been tested through randomized trials and found to be effective in improving outcomes for mothers and their children on a number of important outcome measures, including (1) educational attainment, (2) employment, (3) welfare dependency, (4) parenting attitudes, and (5) subsequent pregnancies. The Department of Justice's Office of Juvenile Justice and Delinquency Prevention funded a grant to support technical assistance and training to replicate the model in selected Weed and Seed sites. This project supports the evaluation component of the initiative to test the effectiveness of the replication process within normal operating environments.
The New Hope Project (5982) is assessing the effect of subsidizing work for individuals and families that are currently poor. The project offers participants assistance in finding a job, or finding a community service job if they are unable to obtain employment in the private sector after 8 weeks. Wage subsidies are provided to assure an income above the poverty level. Participants are also offered access to affordable health insurance and child care. The offer is designed to be flexible according to individual circumstances. Benefits are phased out as earnings increase to make a smooth transition.
Capitalizing the Bridge from Welfare to Independence (CBWI) (6755) is designed to expand the employment and training services provided by Goodwill Industries by constructing new centers in four counties in southwest Florida and six parishes in southeast Louisiana. Once constructed, the centers are expected to be self-supporting (i.e., no need for continuing public support) by utilizing the business activity and payroll generated by the donated goods business.
Analyzing the Employment and Wage Patterns of Welfare Recipients (6756) will describe the employment and wage patterns of welfare recipients using data from the National Longitudinal Survey of Youth and from the evaluation of the Post-Employment Services Demonstration, which operated in California, Illinois, Oregon, and Texas. While many studies have looked at the dynamics of welfare receipt and the reasons people leave welfare, few studies have focused on employment patterns among welfare recipients or wage profiles of welfare recipients who obtain employment. The project will address the circumstances of welfare recipients who find jobs, the employment and wage patterns of welfare recipients who find jobs, and what accounts for welfare recipients' employment and wage patterns.
The Ohio Works First Evaluation (6758) is an assessment of the Ohio Works First Program, which is designed to help families obtain employment and become self-sufficient. It includes provisions for completing a self-sufficiency contract, more generous earned income disregards, a 36-out-of-any-60-month time limit, and whole family sanctions. The evaluation focuses on the following topics: implementation of the core set of services; variations across counties in program design and service delivery; policy changes affecting recipients' attitudes and behaviors toward work participation; the response and reaction of local officials to the new policies and governance structure under welfare reform; the effects on recipients' participation in work activities; the effectiveness of work participation activities in increasing employment levels and self-sufficiency; the most effective components of work participation activities; differences among population groups affected by the program; the groups most at risk of exceeding the time limit; and the impact of specific policy provisions (e.g., time limits and sanctions).
The Examination of State Diversion Programs (6759) project is gathering information from all States on the policies and practices that constitute diversion programs and/or activities designed to divert Temporary Assistance for Needy Families (TANF) applications. More detailed case studies will be conducted in five to seven States in order to more fully document actual implementation and operation of the diversion policies and activities. A major focus of the study will be on examining linkages between Medicaid enrollment and diversion programs/activities. The study also will examine whether and how local communities and institutions, particularly traditional safety net providers, are affected by diversion programs or activities. The project will address the following issues: (1) how State diversion programs or activities are being conceived, structured, and implemented; (2) the effects of these programs/activities on participants, particularly with respect to Medicaid enrollment; (3) how local community institutions are affected, particularly by changes in Medicaid enrollment rates; and (4) whether strategies for monitoring changes in Medicaid enrollment rates can be developed based on existing data.
The Evaluation of Community-Based Job Retention Programs (6760) is divided into two phases. The first phase consists of a detailed implementation analysis and short-term outcome findings for approximately 700 participants (employed TANF recipients) receiving various mixes of job retention and post-employment services through five community-based neighborhood service organizations in Pittsburgh that received funding from the Pittsburgh Foundation (referred to as the GAPS program). A common set of services will be provided, but within the context of somewhat different existing service delivery systems among the community-based organizations. The second phase, in the second and third years of the project, will include State-funded job retention programs in addition to the GAPS projects. Issues addressed will include the post-employment services most instrumental in helping newly employed TANF recipients maintain employment, the organizational features important to successful delivery of job retention services, and the appropriate duration of services.
The Family Investment Program (FIP) (6761) project continues the evaluation of a welfare reform demonstration combining program changes designed to ease a family's transition from welfare to work with strict requirements that recipients participate in developing and executing a social contract, the Family Investment Agreement (FIA), under Iowa's employment program. The FIA details the steps a family will take to become self-sufficient and sets a time frame for doing so. Families opting not to develop an FIA or not following through with the self-sufficiency plan outlined in the agreement are placed on a 6-month Limited Benefit Plan, which leads to the complete loss of cash assistance for the following 6-month period.
The State Welfare Reform Evaluation Track 2 (6762), complementing a separately funded evaluation being completed on Iowa's Family Investment Program (FIP), consists of a study of repeat Limited Benefit Plan (LBP) assignments and a study of post-employment services. The study of repeat LBP assignments will gather information on why cases are reassigned, how reassigned cases are affected by the immediate cessation of cash assistance in a second LBP, and the time pattern of their return to cash assistance. The study of post-employment services will describe and compare standard and enhanced services and assess their contribution to a client's progress toward self-sufficiency and off cash assistance. It will also document processes for developing, implementing, and delivering post-employment services.
Neighborhood variation in the availability of public and private social services throughout Los Angeles County early in the process of welfare reform will be examined in the Neighbors, Service Providers, and Welfare Reform in Los Angeles County (6763) project. The project will also study how agencies are adapting to the current and anticipated changes in demand for their services as a result of welfare reform. The focus of the inquiry will be specifically on services directed toward children and families.
The Welfare Restructuring Project (WRP) (6764) continues the evaluation of a welfare reform demonstration in Vermont. A key feature of the demonstration is a time limit that requires some adult recipients to participate in community work experience after 15 months (for two-parent families) or 30 months (for single-parent families) of receiving cash assistance. Other policies of the project include asset and disregard changes, which require minor parents with temporary disabilities to participate in rehabilitation and training programs.
Achieving Change for Texans (6765) is the evaluation of Texas' original welfare reform demonstration. It consists of a policy on recipient responsibilities for immunization of children, school attendance, and adhering to a personal responsibility agreement; a policy on differential benefit time limits based on work experience and the need for education; several policy options for individual development accounts and fill-the-gap budgeting; and a one-county pilot offering a check for $1,000 in lieu of regular TANF with no reapplication for benefits for one year. The major study questions are as follows: How does the differential time limit work, taking into consideration work experience and education for setting the limit's duration? Is TANF-One-Time an effective way to help people with temporary needs without making them recipients? Will personal responsibility measures help TANF recipients become self-sufficient faster? Can policies designed to promote and reward work help the process of self-sufficiency?
The Evaluation of Arizona Employing and Moving People Off Welfare and Encouraging Responsibility Program (EMPOWER) (6818) project will address whether there are significant differences between experimental and control groups with respect to: (1) income and employment, (2) program participation, (3) program duration, (4) family structure (including fertility) and stability, (5) children's well-being, and (6) employer health benefit provisions.
The State Welfare Reform Evaluation Project (Jobs First) (6819), initiated in Connecticut, will look at the following program outcomes: work and earnings; welfare receipt and costs; job-retention and recidivism; family composition and stability; the well-being of children; child support and other sources of income; family income and poverty status; the use and perceived quality of program services and child care; material hardship; and attitudes toward work, welfare, and the Jobs First program. The implementation and operation of the program will be explored to determine how, and how well, it was integrated with other agencies, including staff training, changes in office culture, and changes in case management.
The Florida Family Transition Program (FTP) Evaluation Project (6820) will address the following questions: Is the Family Transition Program (FTP) program being implemented as intended? How do program staff across all components of FTP achieve the multiple goals implicit in the program model? How do welfare recipients respond to the new package of rules and services? Who reaches the time limits? What happens prior to and at the point when time limits are reached, and after benefits are terminated? Does FTP increase the rate at which welfare recipients participate in and complete various education and employment activities? Does FTP lead to increases in employment and earnings? Does FTP reduce welfare receipt? Is FTP cost-effective?
In Illinois, the Youth Employment and Training Initiative (YETI) (6821) is studying whether interventions that focus on inner-city youth from welfare families (1,000 students in 3 high schools randomly assigned to a treatment or a control group) will help participants complete high school, obtain and hold regular jobs, avoid the pitfalls of substance abuse or early parenthood, and achieve economic self-sufficiency. The process study centers on whether such a program can be effectively implemented and operated in inner-city schools. A cost-benefit analysis will determine the cost-effectiveness of YETI for government, participants, and the general public.
The Indiana Welfare Reform Evaluation Project (6869) continues the evaluation of the Indiana Manpower Placement and Comprehensive Training Program waiver demonstration now operating under TANF. The study will examine the following questions: Was the reform successfully implemented? In what ways does the new program differ from its predecessor? What organizational characteristics, service features, and aspects of the demonstration environment determine its impacts? How do these reforms impact benefit payments, income, self-sufficiency, children's well-being, and family stability? What are the consequences of the changes on the size and composition of the welfare caseload under TANF? What trends should the State expect in levels of benefit payments and services needed? What happens to individuals after they leave welfare, especially those who reach the time limit? What are the impacts of time limits? How has welfare reform affected welfare entry?
The Maryland Family Investment Program Evaluation (6823) is a statewide process study to examine and document front-line assessment and allocation practices in the State's 24 local jurisdictions. The research questions to be addressed are as follows: How much discretion do county-level workers have in assessing clients and directing them to services that are likely to address their needs? How does worker discretion vary from one county to another? How do county management differences influence assessment practices? How do county front-line assessment practices and differences affect welfare service delivery? Do assessment practices appear to be appropriate to clients' situations and clearly directed to desired TANF outcomes (e.g., successful employment placement before a time limit is reached) and other measures of family well-being?
An eight-county (four urban and four rural), 17,000-case (randomly assigned to treatment or control group) evaluation of the Minnesota Family Investment Program (MFIP) (6824) is attempting to answer the following major research questions: What are the effects of financial incentives in employment and welfare receipt? What is the effect of mandatory case management in combination with increased financial incentives? What is the effect of case management services timing? What is the extent of client understanding and participation in MFIP? What is the nature of services being provided and the interactions between clients and staff? What are the net costs of the program from a variety of perspectives?
The Minnesota WorkFIRST Program (Track 2) (6825) project is a comparative evaluation of Minnesota's WorkFIRST with the Minnesota Family Investment Program and STRIDE services (MFIP-S). WorkFIRST uses a rapid labor force attachment strategy versus MFIP-S (the State's TANF program), which represents a progressive labor force attachment model. The project will address the following questions: Will the WorkFIRST program have a greater impact on moving first-time applicants into employment and promoting long-term self-sufficiency or will the concept of "make work pay" strategy of the MFIP-S programs prove more effective in moving people off welfare? Is WorkFIRST more effective in preventing first-time applicants from becoming dependent on public assistance? Do tough sanctions and vendor payment reinforce the WorkFIRST message of personal responsibility?
The Evaluation of the Employment First Program (6826) project in Nebraska includes an impact study featuring a random assignment experiment in Omaha of 4,800 recipient and 2,250 applicant cases, and a subsample of 2,000 randomly assigned cases that will be interviewed in a survey. A process study will be conducted in four sites, including Omaha and Lincoln and will include interviews with a subsample of 180 cases 3 months from the date of initial assignment to determine understanding of the program. Another subsample of 350 cases will be used to collect data not available from the State's data system. The case manager study will begin with a case manager background information questionnaire supported by client outcomes data. High-performing case managers will be identified and interviewed to determine the components of their success.
The New Hampshire Employment and Training Program Process and Outcome Study (6827) will examine the planning, funding, and implementation of the State's welfare reform program and its different components at the State and local levels. This study will examine the following questions: What are the allocated funding levels for planning and ongoing program operation? What are the reactions to, and effects of, the various sorts of interagency collaborations around which the program is designed? What activities has the State conducted to promote culture change among the staff, clients, and community partners? What has been the reaction to the up-front diversion and employment aspects of the program? How are staff implementing assessment and program assignment procedures? Overall, what do the assessment procedures help discover about client barriers and needs? What are the outcomes of measures such as employment rates, length of employment, total earned income, child support collection, total family income, and savings?
The Impact Study of the New Hampshire Employment Program (6828) will involve a pre-post non-experimental design consisting of comparisons between baseline and post-implementation cohorts, caseload modeling techniques, and a pre-post time series analysis of trends on selected welfare-related aggregate measures. Answers will be sought to the following questions: Does New Hampshire's welfare reform program have an impact on State welfare case dynamics, including exit rates, length of benefit receipt, and recidivism? Does the program have an impact on welfare caseloads compared to forecasted caseload values such new welfare case openings, case closings, denials and withdrawals, "child only" cases, earned income cases, and benefit payments? Does the program affect welfare-related measures such as foster care placements, child abuse and neglect, homelessness, paternity establishments, and child support collections?
Data gathered in eight counties in the Evaluation of the North Carolina Work First Program (6829) will be used to address several questions: To what extent can the observed differences in client outcomes between counties be attributable to variations in local economic conditions and unemployment rates? To what extent are the intercounty differences attributable to other contextual factors such as variations in health coverage or demographics of the local population? What political and organizational factors may have affected the results for each county? What are the outcomes of Work First in terms of indicators for self-sufficiency, welfare costs, levels of work activity participation, family structure and stability, and child health and well-being?
In North Dakota, the Training, Education, Employment, and Management (TEEM) Project (6830) will continue the evaluation of the TEEM project, which consolidates TANF and the Low-Income Home Energy Assistance Program (LIHEAP) into a single cash assistance program. The TEEM project includes social contracts, increased work incentives, sanctions, raised asset limits, and incentives for family stability and marriage. The study will look at the following questions: How are policies, services, and program rules implemented? How are families and county staff affected by the new policies and programs, and does this vary by subpopulation (e.g., ethnicity) within and between the counties? What are the components of the TEEM screening system? What barriers to employment are addressed? How do client participation rates in welfare programs, including employment/training programs and the Child Support Enforcement program, change with the implementation of TEEM? How well does TEEM serve clients with special needs (such as disabled clients, clients with disabled children, and clients with substance abuse or domestic violence problems)?
The Virginia Independence Program (6831) project, an evaluation of TANF implementation, will study the following policy topics: diversionary assistance, family caps, compulsory school attendance, paternity establishment, minor parent residency, required immunizations, savings accounts, 2-year time limits, personal responsibility agreements, expanded earned income disregards, transitional Medicaid, and child care benefits. The major research questions are as follows: What happens to families after they reach the time limit? What are the effects of post-employment and job-retention services offered by the State? What are the implications of the growing number of child-only cases the State is finding on its welfare rolls? What are the issues around players and localities where unique innovations have been introduced in the State? How does diversionary assistance work in a limited number of localities? What are the dynamics of the Virginia caseload over time, under the TANF regime? Are there social health indicators for TANF recipients other than those required by law that would help the State keep abreast of developments as they affect the caseload?
Welfare-to-Work: Monitoring the Impact of Welfare Reform on American Indian Families with Children (6832) is being conducted in Arizona. The State was selected because it has the largest population of American Indians living on reservations and because the 21 reservations differ on a variety of dimensions. The research questions that will be addressed are as follows: What are the demographic, social, and economic characteristics of Indian families with children on welfare? What are the reservation-based or individual barriers to raising the skills and employment potential of Indian parents on welfare? How have the tribal councils prepared themselves to deal with the consequences of welfare reform? How are tribes who plan to administer TANF independently positioning themselves to undertake this task? Will the benefits and outcomes to families vary significantly depending on whether TANF is administered by the State or the tribe? What strategies are time-limited parents using to attain independence? What proportion of those losing eligibility shift to Indian general assistance programs?
The Assessing Effective Welfare-to-Work Strategies for Domestic Violence Victims and Survivors in the Options/Opciones Project (6833) will document the needs of battered girls and women on welfare and will identify those strategies successfully employed to eliminate violence and to provide alternatives to welfare. The research will focus on services provided through Options/Opciones in partnership with the Illinois Department of Public Aid in one community of North Lawndale. Questions to be addressed are as follows. What are the successful strategies that battered women on welfare use to become economically self-sufficient? What are the service needs of battered girls and women on welfare? Which problems serve as the major welfare-to-work barriers for battered girls and women? What combination of services is needed to address the domestic violence and welfare-to-work issues, and in what order? Did these services lead to labor market participation, and for what percentage of women and under what time frame? What are the characteristics of girls and women who successfully remove themselves from situations containing domestic violence? What are the characteristics of the girls and women who successfully enter the labor force versus those who do not?
The purpose of the Welfare Reform Studies and Analyses (Rural TANF) (6834) project is to conduct an in-depth process evaluation of family response to the implementation of welfare reform in three remote rural counties of northeastern Washington State: Stevens, Ferry, and Pend Oreille. Eighty families (40 experimental and 40 comparison) will be interviewed 5 times within 16 months. The families will be selected from the 340 households currently participating in the Comprehensive Child Development Program and who are also enrolled in TANF and eligible to participate in Washington's Work First program. This study will look at several questions: How do families formulate and act upon strategies to attain economic self-sufficiency in an environment of scarcity? What is the relationship of a family's social network to its effectiveness in formulating strategies and successfully carrying out a plan to implement them? How do the actions of community agencies and organizations carrying out the tasks of welfare reform respond to family strategies?
The Jobs-Plus: Community Revitalization Initiative for Public Housing Families (6835), a 7½-year demonstration program, is aimed at dramatically increasing employment, earnings, and job retention among working-age residents, a large percentage of whom are on public welfare or are at risk of dependency. Aside from Housing and Urban Development (HUD) and ACF, the demonstration is supported by the Rockefeller Foundation and three other foundations. The program supports the planning, development, implementation and evaluation of locally-based approaches to providing saturation-level employment opportunities and supportive services for working-age residents of family housing developments in seven cities. It incorporates a combination of state-of-the-art employment, training and supportive services; financial incentives (occasioned by the welfare reform and new public housing policies); and rigorous efforts to rebuild and strengthen the community in support of work.
The purpose of the Management-Focused Welfare Reform Evaluation Data Access and Policy Manual Development (6836) is to develop (1) a software package that organizes public assistance evaluation data into readily accessible formats that can be continually accessed and used for timely and responsive decisionmaking; (2) a policy analysis manual that assists users in understanding how to read and apply continually accessed outcome or impact data to program or policy decisions; and (3) technical documentation for systems personnel and regular users. The manual will assist the user in bridging the gaps between what outcomes and impacts may appear in the sample, what this information suggests, caveats in interpreting the results, and ultimately, what program or policy responses should be considered.
The Welfare Policy Typology Project (6837) is laying the groundwork for a data base-- containing key information about State welfare policies (including TANF), State-funded maintenance-of-effort programs, and federally-funded child care assistance programs--that would be available to the public, probably through the Internet. With the enactment of the Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA) of 1996, there are few, if any, strict Federal requirements for welfare programs, and States are not required to report their policies in any detail, let alone in a uniform format. This will make research on the effects of welfare reform somewhat problematic, because researchers will not have a ready source of information on State policies. Without a standard compilation and classification of State welfare policies, different researchers are likely to classify policies in different ways, making cross-study analysis virtually impossible.
The Wisconsin Pay for Performance/Self-Sufficiency First Evaluation (6838) project will complete an impact evaluation of a statewide work-focused welfare reform demonstration. Within a research site, current public assistance recipients and new applicants were randomly assigned to either an experimental group subject to the Pay for Performance/Self-Sufficiency First (PFP/SSF) provisions, or a control group subject to the regular program rules according to the State's approved AFDC/Medicaid State Plans and approved Food Stamp Plan of Operations. Several questions will be addressed: Do the program policies improve employment rates, length of employment, amount of earned income, hours worked per month, child support collections, total family income, and accumulated savings? Do these policies affect public assistance participation and program costs? Does the program affect participation in employment and training activities? Do these program rules and policies affect marriage and separation rates and use of foster care? Does the demonstration affect the incidence of reported child abuse and neglect and health insurance status of children?
The Partner and Father Involvement in the Lives of Low-Income First-Time Mothers and Their Children (6839) project is investigating the role that fathers and partners play in improving the material, emotional, and developmental well-being of low-income women and children. The study consists of a set of intensive secondary analyses using data from three longitudinal randomized experiments of a prenatal and infancy home visitation program serving first-time mothers from various ethnic and racial groups. Research questions to be addressed are as follows: What is the natural history of father and partner involvement in the lives of low-income women and children? What is the impact of a prenatal and early childhood home visitation program on partner and father involvement? What is the impact of partner and father involvement on women's life course development (e.g., welfare, dependence, subsequent pregnancies, and mental health)? What is the impact of father and partner involvement on children's health and development (e.g., child abuse and neglect, intellectual development, and antisocial behavior)?
The Evaluation of Los Angeles Jobs-First GAIN (6840) project will examine the results of changing Los Angeles GAIN, a human resource-focused welfare-to-work program, into Los Angeles Jobs-First GAIN, a labor force attachment strategy. Data collection will be primarily from administrative records, with a 2-year survey of approximately 250 experimental and control group cases. The following research hypotheses are expected to be confirmed. It will be possible to convert a large, diverse, mature, education-focused welfare-to-work program to a Work First program. The new program will produce earnings and recipient changes of greater magnitude than the original program. The new program will produce earnings increases and welfare reductions with much smaller expenditures per enrollee than its predecessor. Having profited by experience, Jobs-First GAIN will produce equal or better results than previously evaluated Work First programs. Jobs-First GAIN can effectively operate on a large scale, serving a diverse welfare population. The program will quickly move many recipients into jobs, thereby minimizing the number who reach the TANF time limit unemployed. Finally, the programs' economies of scale will achieve savings for government budgets.
The principal objective of the Front-Line Management and Practice Study (6841) is to evaluate whether front-line workers are implementing the welfare goals and policies established by the States. Using the local TANF office as the unit of analysis, researchers will execute an in-depth process/implementation study relying on observable variables such as assessment processes, allocation of client services, allocation of paid and unpaid work, enforcement of work-related policies, delivery of information, orientation of individual worker practice, office level practices, and implicit messages or signals.
The ACF Office of Community Services (OCS) is supporting evaluations of grantees funded under the Job Opportunities for Low-Income Individuals (JOLI) Program (6430). Assistance is provided to grantees for developing project designs and evaluation plans. For example, the Bridgeport Artisan Center is creating jobs and enterprise opportunities for low-income artisans in the inner-city and surrounding neighborhoods of Bridgeport, Connecticut, an Enterprise Community. Another example is Avenues (Avenidas), a project of the Mi Casa Resource Center for Women in Denver, Colorado, which trains low-income persons--primarily women--for jobs and apprenticeships in highway construction and maintenance. OCS will evaluate the impact of the demonstrations and disseminate the project results to Congress and other interested parties.
During FY 1997, OCS initiated several reports on evaluation results of projects funded under the Demonstration Partnership Program (6883). One report will contain summaries of eight Demonstration Partnership Program (DPP) projects focused on various facets of case management in the delivery of social services. A second report will focus on youth at risk and minority males. A third report will summarize projects in the area of micro-enterprise and self-employment, and homeless individuals and families. In addition, OCS will be examining 21 DPP projects, funded in FY 1994, that are in the final stages of their evaluations.
ACF also has three projects addressing child support enforcement. The first project, the Study of the Impact of the Child Support Enforcement (CSE) Program on Avoiding Costs to Public Programs (6842) will include (1) a literature review on CSE and cost avoidance and an annotated bibliography; (2) a small conference on cost avoidance consisting of experts from State and Federal government, academia, and the private sector; (3) an assessment of national micro-simulation models to study the impacts of the CSE program; and (4) an assessment of State administrative data capacity for measuring cost avoidance.
The second project, Evaluation and Reporting on State Access and Visitation Grant Programs (6843), will help define data elements to report on and evaluate State Access and Visitation Grant Programs. The project will also assist States in implementing data elements and reporting as part of the monitoring requirement.
The third project, Partners for Fragile Families (6844), is an assessment of a program to assist low-income young fathers to work with the mothers of their children to share the legal, financial, and emotional responsibilities of parenthood and improve the child support system's interaction with the fathers of fragile families. The National Center for Strategic Non-Profit Planning and Community Leadership in Washington, D.C., will offer technical assistance to service providers in public agencies and community-based organizations, facilitate dialogue and information exchange among public and private partnerships; and identify models and promote public and private partnerships.
Healthy Development of Children and Families
ACF has eight studies under way that address Head Start issues. The Early Head Start Research and Evaluation National Study (3570) is evaluating the effectiveness of the Early Head Start program in 15 diverse communities in 13 States and the District of Columbia. The study is examining child, family, staff, and community outcomes in a sample of 3,400 children and their families who were randomly selected for program and comparison groups while the mothers are pregnant or until the children are 12 months old. Assessments of the children, families, and child-care environments will made when children are 14, 24 and 36 months of age.
The Profiles of Head Start Families: Development of a Head Start Family Information System (3869) project will (1) provide a national description of Head Start family and child characteristics using family-level data, (2) identify Head Start child and family service needs and services received within a given program year, and (3) provide for the regular reporting of data over time so as to permit analyses of changes in Head Start family and child characteristics.
The Evaluation of the Head Start/Public School Early Childhood Transition Demonstration (4393) project is assessing the effectiveness of providing comprehensive, continuous, and coordinated services to Head Start families and children from the time of Head Start enrollment through the third grade in public school. Working in concert with local evaluators, the contractor is developing a set of common data collection instruments to be used across all sites. The project is providing data regarding the effectiveness of the Transition Project models in maintaining gains by children and families while in Head Start. The evaluation and demonstrations were mandated by the Head Start reauthorization legislation. Other studies completed during the ten years prior to 1992--notably the Head Start Transition Study and Developmental Continuity--did not provide information on services extended to children after they left Head Start.
All 41 Family Service Center Demonstrations are being evaluated through a consortium of local evaluators under the Evaluation of the Head Start Family Service Center Demonstrations (4394) project. The contractor is providing coordination, technical assistance, and analysis on common data elements across sites to yield a coordinated and integrated summary of process and impact evaluations conducted by local evaluators. This consortium approach is ensuring a consistent and technically sound method for evaluating these demonstrations of how Head Start can collaborate with community programs to meet the needs of Head Start families dealing with problems such as illiteracy, substance abuse, and unemployment.
The Study of the Characteristics of Families Served by Head Start Migrant Programs (4974) is documenting and describing the implementation of 26 Head Start Migrant programs, providing a profile of Head Start migrant families in the main migratory streams and generating information on unique issues related to serving migrant families through Head Start programs. It is also documenting the availability and coordination of services for Head Start families during their migration. Finally, it will provide a national estimate of the number of eligible migrant children versus the number of those children being served by the Migrant Head Start programs. Findings from the study will be used to inform future policy decisions on Head Start migrant programs, as well as the new Early Head Start program for infants and toddlers.
The Evaluation of Head Start Family Child Care Homes (4975) project is assessing the effectiveness of the 18 Head Start Family Child Care (FCC) Homes demonstration projects funded by the Administration on Children, Youth and Families (ACYF) in FY 1992. The project is assessing the quality of Head Start services provided in FCC homes with regard to Head Start Program Performance Standards. The project is comparing services delivered in FCC homes to those delivered in Head Start centers. Children are randomly assigned to the experimental and control groups. Data is being collected on the cognitive, socio-emotional, and physical development of the two cohorts of children participating in the study.
The Descriptive Study of Head Start Bilingual/Multicultural Program Services (5845) aims to identify the number, geographic distribution, and sociodemographic characteristics of the Head Start-eligible population for different cultural and linguistic groups by region and nationally. It will describe the bilingual and multicultural children currently served by the Head Start program. The range of bilingual and multicultural services provided by the Head Start program nationally will be compiled using existing data and a survey of all Head Start programs. Finally, the project will assess the service models, staff training approaches, community partnerships, and administrative plans and processes of 30 Head Start programs that have developed innovative methods for addressing the unique program needs of one or more of the diverse cultural and linguistic groups.
A 3-year Descriptive Study of Families Served by Head Start (6331) will provide crucial policy-relevant information on a nationally representative sample of families served by Head Start in 40 programs across the country. Through a survey and more intensive case study methods, the project will chart family demographics, strengths, needs, expectations, and experiences within Head Start programs, as well as programmatic efforts to join in partnership with families.
The Evaluation of the Comprehensive Child Development Programs (3868) is an assessment of the effectiveness of the first group of 21 Comprehensive Child Development Program grantees. In particular, the study will examine (1) the longitudinal impact of each program model on the development of the participating children and their parents, (2) the effectiveness of the programs in achieving their stated objectives, and (3) the impact of related programs on the delivery of services.
The National Study of Low-Income Child Care (6845) is studying the low-income child care market in 25 communities in 5-7 States with a substudy to examine the license-exempt family care market in 5-7 neighborhoods drawn from these communities. It will provide essential information to help inform the issues surrounding subsidized child care and its implementation by the States, with particular reference to the provisions in the Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA) of 1996 over time. It will further examine how significant shifts in welfare policy and programs affect the child care market for welfare recipients and the working poor at the community level. It will study the impact of subsidies on the types, amounts, and quality of child care available; children's child care placements; and family decisionmaking. Additionally, it will report on the largely unknown license-exempt family care market for both preschool and after-school cohorts.
In the Role of Child Care in Low-Income Families' Labor Market Participation (6846), optional research designs to identify and address child care services needed by parents to succeed at work will be developed. A series of stand-alone working papers will be prepared that critically evaluate relevant research related to child care and labor force attachment and that develop the rationale for the factors included in the research designs. Priority outcomes to be incorporated in the research designs include employment, welfare participation, children's development, family well-being, and child-care quality.
The Child Impact Studies (6847) project will augment welfare reform demonstration evaluations in five States (Connecticut, Iowa, Minnesota, Indiana, and Florida) to assess the effects of different welfare approaches on children's well-being. The project adds detailed data on children to these evaluations. At each site, treatment groups subject to welfare reform policies will be compared with control groups subject to former AFDC policies. Child outcome data, focusing on ages 5-12, will be collected through surveys and administrative records. The study will address two questions. What are the effects of alternative approaches to welfare reform on child well-being, including school achievement, behavioral problems, and health status? What intervening mechanisms--such as quality and regularity of the home environment, child care arrangements, and parental employment and income-- will affect these outcomes?
Three projects address the issue of homelessness. The first, Evaluation of the Impact of Homelessness on ACYF Programs (4396), will conduct site visits and collect information from service providers for the homeless, administrators, and representatives of local service delivery networks in 40 communities. The project will examine programs supported by ACYF, but will also review non-ACYF-sponsored programs. In addition, the project will conduct a small-scale longitudinal study of homeless people and the impact of services upon them. Five local programs serve as case studies. The results of the review will focus on key strategies for increasing the effectiveness of ACYF programs and on measures to help reduce people's risk of homelessness.
The second, Evaluation of the Transitional Living Program for Homeless Youth (4397), will examine the number and characteristics of homeless youth served by this program. The project will examine the effectiveness of the transitional living program in alleviating the immediate problems of homeless youth and its success in preparing homeless youth for self-sufficiency and in helping them decide upon future education, employment, and independent living. The ability of the program to strengthen family relationships and to encourage intrafamilial problem solving through counseling and developing self-sufficient skills will be assessed.
The third, Evaluation of Runaway and Homeless Youth Programs--A Follow-Up Study (4361), will examine the impact and effect of the Runaway and Homeless Youth (RHY) shelter program on youth and review the policy, program, and service delivery issues that impede or facilitate RHY shelter goals. The study will obtain information from 23 sites on runaway and homeless youth shelters and services and assess the impact and effect of the RHY program from the perspective of program administrators and service providers. It will identify the policy, program, and service delivery issues that influence whether or not RHY shelter goals are met.
Factors Related to Gang Membership Resistance (5042.1) is studying gangs from two contrasting Los Angeles communities--one with higher-than-average Hispanic and African-American gang activity, and the other with lower-than-average gang activity. The project is designed to increase understanding of how youth in urban areas with high levels of street gang activity avoid gang involvement. The project will compare gang members with youth who do not belong to a gang in different urban settings, such as a community with high gang activity versus one with low gang activity. Individuals and community variables and characteristics in the contrasting sites will be analyzed.
In Prevention of Foster Care Placement of Children at Risk for Domestic Violence (4378), the Family Advocate Project (FAP) is building on existing programs operated by the Family Intervention Center of the Children's Hospital of Pittsburgh, which provide medical and psychosocial evaluations to battered women and their children. The project is studying 150 mothers assigned to either a study or a control group. Study-group mothers receive intensive services in the areas of counseling, housing assistance, and legal, health care, and social services. The project is looking at ways to alleviate the emotional trauma of battered women and explore whether the provision of support services to increase self-esteem and coping skills can prevent out-of-home placement or reduce the length of time children spend in foster care.
The National Child Welfare Research Center: University of Chicago (4390.1), located in the Chapin Hall Center for Children, is conducting research on the following issues: (1) research and management uses of computerized child welfare data; (2) service and policy issues concerning in-home and community-based care; (3) the uses, consequences, and costs of in-home care; and (4) ways to apply knowledge and research about child development to children's policies. The Center holds conferences and sponsors a child development fellowship in conjunction with the Society for Research on Adolescence. Publications will include research syntheses, a training manual, and a report on national data issues.
The National Child Welfare Research Center: University of California at Berkeley (4390.2), located in the School of Social Welfare, will conduct projects to examine (1) child abuse and child welfare; (2) family preservation and maintenance; (3) foster care and adoption; (4) drug and AIDS-affected children; and (5) the organization, financing, and evaluation of child welfare services. The center serves as a knowledge-building and information-disseminating resource for improved child welfare services.
The National Study of Outcomes for Children Placed in Foster Care with Relatives (5846) is examining the outcomes for and associated costs of children and families in the various configurations of relative foster care compared to the configurations of non-relative foster care. Information obtained in an initial nine-State survey is being used to design a national study on the outcomes of children placed in relative foster care compared to the outcomes for those placed in non-relative foster care.
The aim of How Decisions to Change the Case Plan Goal Are Initiated (6848) is to conduct a 3-year study of children entering State custody as infants that will identify factors that facilitate or delay changes in the case plan goal leading to permanency. Data will be collected through in-depth interviews with caseworkers responsible for selected cases. Products of this study will include three major reports and policy and practice recommendations for facilitating permanence for children entering State custody as infants.
The Factors Related to the Quality of Family Foster Care (6849) study seeks to determine the quality of the foster care experience and the factors that influence it, based on data collected on a sample of 500 families whose children spent time in foster care in Wayne County (Detroit) Michigan during 1993. The study examines the relationship between agency and service characteristics (e.g., private versus public agency, caseload size, staff turnover, continuity of case services, provision of family preservation services, and use of kinship care) and the quality of foster care. Five aspects of foster care will be assessed: the health and well-being of the child while in care, incidence of maltreatment in care, family continuity, number of placements by type, and duration of care and recidivism.
The Assessing the Quality of Foster Family Care: An Initiative for the Integration of Research and Practice (6850) project aims to fill the gap in the literature on foster family care by examining the individual characteristics of foster parents across domains, as well as the environmental characteristics of the home. Participants (240 families) will be drawn from the pool of foster parents and children in Maryland's Prince George's and Montgomery counties. The project will examine the following questions: (1) Does the current pool of foster parents possess the characteristics to meet the needs of the foster child population? (2) What is the appropriate role of relative foster homes? (3) How does ethnicity relate to other characteristics of foster family homes?
Project REFRESH: Research and Evaluation of Foster Children's Reception into Environmentally Supportive Homes (6851) seeks to discover the effect of the everyday occurrences in foster family homes. Research indicates that foster children aged 9-18 may perceive themselves as outsiders to their foster families. Some birth children in foster families perceive a foster child as an intruder. Many foster caregivers have difficulties balancing parenting and caregiving roles, which impacts quality, duration, and level of birth and foster children's care. Assessment tools and evaluation protocols are needed to examine factors affecting quality and satisfaction with care in kinship/non-kinship placements. The major research goal of this 3-year study by Oregon State University is to develop a model to assess a child's integration into a foster family.
The purpose of the Evaluating Quality of Out-of-Home Care in Kinship Foster Families (6852) project is to identify criteria for assessing the quality of out-of-home care provided to children in kinship care homes and to develop objective items that use these criteria to effectively measure the quality of kinship care. Instruments to be developed by the University of Illinois-Urbana will measure the quality of the kinship family home, contextual factors related to kinship family functioning, and indicators of child functioning in the home.
Two studies address issues of family preservation and family support. The first, the National Evaluation of Family Support Programs (5848), is a comprehensive review of what is currently known about family support programs and their effects. The project will include several studies to fill gaps in understanding about family support programs and their effects, and to synthesize the results with the results of the earlier review. The review conducted in the first year will guide the formulation of a research strategy for the remaining four years of the project. Up to ten program evaluations will be designed and implemented.
The second, the Family Preservation and Family Support Services (FP/FS) Implementation Study (5975), examines the FP/FS implementation process across States and communities and among different stakeholders over time. It examines the process broadly over all 50 States, and in depth in 10 States and 20 communities. The first component of the study analyzes and synthesizes first-year applications and examines State planning and use of FP/FS funds, e.g., the nature of the planning process, the involvement of various stakeholders, and decisions about fund allocation. The study will assist ACYF in its joint planning efforts with States by providing feedback on States' progress in implementing FP/FS legislation.
The purpose of the National Longitudinal Study of Children and Families in the Child Welfare System (6748) is to describe the outcomes experienced by children and families who come to the attention of the child welfare system, and to gain an understanding of the factors, including system-level and service factors as well as child and family characteristics, that contribute to those outcomes. A nationally representative sample of 6,000 children will be sampled upon entry into the child welfare system. Data will be collected during the first year after entry and at three annual followup interviews. Information will be obtained from the children and their caregivers, caseworkers, and other child welfare agency personnel and service providers.
MISSION: To foster the development of services to help older persons maintain their independence.
Evaluation Program
The Administration on Aging (AoA) is the Federal focal point and advocate agency for older persons and their concerns. The AoA administers key Federal programs mandated under various titles of the Older Americans Act. These programs help vulnerable older persons remain in their own homes by providing supportive services. Other programs offer opportunities for older Americans to enhance their health and to be active contributors to their families, communities, and the Nation through employment and volunteer programs. The AoA works closely with its nationwide network of regional offices and State and Area Agencies on Aging to plan, coordinate, and develop community-level systems of services that meet the unique needs of individual older persons and their caregivers. The AoA collaborates with Federal agencies, national organizations, and representatives of business to ensure that, whenever possible, their programs and resources are targeted to the elderly and coordinated with those of the network on aging.
As the responsibilities of this nationwide network of State and Area Agencies on Aging continue to grow, it is essential that they have the necessary information to meet these responsibilities.
The overall evaluation priorities of the AoA are to support studies that provide information on the following:
Summary of Fiscal Year 1997 Evaluations
Although there are no major evaluations to report during FY 1997, AoA initiated an evaluation of the Supportive Services and Senior Centers provisions of the Older Americans Act (Title III-B). These provisions provide home- and community-based services (HCBS) for which all elderly persons are eligible, with particular attention to older persons in greatest economic or social need. Most home- and community-based services fall under three broad categories: access services, in-home services, and other community-based services. Access services include care coordination, transportation, outreach, and information and assistance. In-home services include personal caregivers, homemakers, chores, and telephone reassurance. States and local communities decide on the mix of services these funds provide through State and area plans for which public participation is required. In FY 1995, the last year for which complete data are available, the Title III-B Program ($306,711,000) provided services to approximately 6,050,000 persons.
A basic operating principle of the Older Americans Act is to provide funds through States to local governments and community organizations with Federal assistance and leadership, but which is flexible enough to address local community needs. This approach is consistent with current legislative trends in establishing intergovernmental funding relationships. There is substantial anecdotal evidence that this flexible decentralized approach has worked very well in fostering the development of HCBS appropriate to diverse States and communities. However, the diversity of the States, communities, service programs, and resource inputs, the long time frame for development, and the general nature of the outcomes sought (comprehensive coordinated HCBS systems) make it difficult to evaluate progress related to Title III-B. State Program Report (SPR) data are a necessary starting point in describing results from Title III-B. But the SPR shows only the output of the services funded and does not illuminate the developmental effects of the aging network funded through Title III-B. Under provisions of the Government Performance and Results Act (GPRA) of 1993, it is appropriate for AoA to evaluate the outcome results from Title III-B despite the conceptual and methodological difficulties.
Evaluations in Progress
Given the evolving roles of AoA and State and Area Agencies on Aging and the continued (and projected) growth of the Nation's elderly population, AoA's evaluation efforts will continue to focus on effective program planning and service delivery, as well as on continued monitoring of the program's effectiveness in addressing the goals of the Older Americans Act. During FY 1997, AoA initiated a study to plan an evaluation of the Supportive Services and Senior Centers provisions of the Older Americans Act (Title III-B) (6855). Through this project AoA will identify the conceptual and methodological issues in the evaluation and will develop an approach to this evaluation. The evaluation is expected to develop outcome measures with broad applicability, which can be used repeatedly over time. It is anticipated that the actual evaluation will begin early in FY 1999.
Evaluation is an important part of the AoA program, and a number of evaluation activities are currently under way, although not on a national level. Many State and Area Agencies on Aging are conducting evaluations of their Older Americans Act service programs. These studies generally focus on such issues as the needs of the target population, the quantity and quality of services delivered, and the impact of the services on the older person. Results are used to better design and target Older Americans Act services. Regional offices of the AoA also conduct assessments of the States' needs for training and technical assistance. Finally, demonstration projects funded by the AoA have evaluation components to enable them to assess their progress in meeting the objectives of their demonstrations.
AGENCY FOR HEALTH CARE POLICY AND RESEARCH
MISSION: To generate and disseminate information that improves the health care system.
Evaluation Program
The Evaluation Program within the Agency for Health Care Policy and Research (AHCPR) is used to respond to three types of information needs:
To address these needs, evaluation components are built into virtually all major AHCPR programmatic activities and a wide variety of "freestanding" projects are undertaken as needs are identified. Among the evaluation mechanisms used by the Agency are targeted evaluation studies undertaken through contracts or grants; efforts to obtain feedback from "customers" on the usefulness of AHCPR research efforts, including such mechanisms as focus groups and surveys; and feedback from AHCPR's User Liaison program (which provides information and technical assistance to State policymakers, health departments, and officials). Information gathered from AHCPR's evaluation-related activities will be used in the annual performance plans developed in response to the Government Performance and Results Act starting in FY 1999.
All "freestanding" evaluation activities undergo two levels of review. First, brief evaluation proposals are developed by staff and undergo review by affected senior managers. Those proposals approved by senior managers are submitted for review by the administrator. The administrator evaluates proposals for policy relevance, priority, cost effectiveness, and timeliness. Those that receive the administrator's approval are then methodologically developed and centrally reviewed for technical merit, including technical feasibility, costs, and relation to ongoing evaluation activities.
Summary of Fiscal Year 1997 Evaluations
Outcomes and Effectiveness Research in the Private Sector
To help clarify AHCPR's future priorities in outcomes and effectiveness research (OER) and to provide insights regarding future opportunities for public-private partnerships in this area, AHCPR undertook an analysis and evaluation of private-sector OER activities (6385). The study found that a wide variety of private-sector organizations report performing OER, often for the purposes of monitoring health care performance against certain standards, e.g., accreditation standards. In addition, private-sector organizations conduct OER primarily to improve their strategic and financial positions. More specifically, health product companies usually focus their OER efforts on new drugs and devices, while less attention is given to other health care interventions, particularly those that are not new or those that are used primarily for traditionally underserved or vulnerable populations. AHCPR has supported the development and validation of many of the research tools used in private-sector OER. The report found little evidence of a redundant effort between publicly and privately supported OER and reinforced the need for public support for continued methodologic development in OER.
Informatics
Drawing upon research previously performed in the U.S., Canada, and the United Kingdom, a recent study reported that informatics tools and decision aids affect patient decisionmaking about medical screening and treatment (6376). More specifically, researchers found that patients who use informatics tools are more satisfied with the decisionmaking process and have more knowledge of treatment alternatives than patients who do not use the tools. The researchers recommend that the effects of informatics tools on a full range of outcomes be assessed (e.g., patient-provider communication, treatment selection, health behavior, clinical and health status outcomes, and health care costs). It is important that factors influencing patient use of informatics tools be identified as well as the cost-effectiveness of different types of patient informatics tools.
Quality Measurement
As part of its effort to measure and improve quality of care, AHCPR completed two related evaluations of CONQUEST--the Computerized Needs-Oriented Quality Measurement System (5961.1 and 5961.2). CONQUEST is a prototype software system for collecting and evaluating clinical performance measures, and organizing them into a classification scheme that presents essential information in a standard format. The two evaluations identified short-term, intermediate, and long-range strategies for CONQUEST's continued development. While finding CONQUEST 1.0 to be a good starting point, the evaluations recommended that future versions of CONQUEST be expanded to include more measures, more conditions, and more detail on measure construction/specifications. In addition, the evaluations indicated that CONQUEST should be modified to make it more intuitive and user-friendly. Many of the recommendations were incorporated into CONQUEST 2.0.
Managed Care Organizations
To help determine the extent to which AHCPR and other organizations will be able to analyze data from managed care organizations (MCOs), AHCPR completed work on a project to determine the feasibility of creating a managed care encounter-level data base (6374). The rapid growth of managed care is changing the ability of health services researchers and policy analysts to answer fundamental questions about access, utilization, cost, and quality of care. Data that were previously produced as a "by-product" of the fee-for-service system are either no longer available or vary in comprehensiveness and accuracy. One major finding of this study was that MCO data bases vary substantially in the availability and comprehensiveness of encounter-level data and in the ability to link these data with administrative and financial data. MCOs increasingly view their information systems, and the data they store, as proprietary, and there are a number of disincentives for MCOs to share their data. In addition, there are no industrywide definitions and reporting standards. Despite these barriers, MCOs are interested in exploring ways to work with Federal agencies to facilitate the availability of data for the purposes of research.
AHCPR has used the study findings to 1) identify future opportunities for public-private partnerships; 2) solicit feedback from multiple constituents on models that the Agency could employ to facilitate improvements in, and accessibility of, encounter-level data; 3) assess current AHCPR supply-side data initiatives (e.g., Healthcare Cost and Utilization Project); 4) formulate a longer-term encounter-level data strategy; and 5) inform and support related activities in the Secretary's Quality Initiative and the President's Commission on Consumer Protection and Quality.
Dissemination Evaluation
AHCPR conducted a customer survey to determine satisfaction with its publications and their dissemination (6375). The effort found that AHCPR's monthly newsletter, Research Activities, is well-received as it is currently written, presented, and distributed. A large percentage of readers keep the publication as a reference source and would prefer to continue to receive the publication in hard copy format, rather than via the World Wide Web (where it is also currently available). Overall, the survey results showed that the publication has a relatively loyal following. The study also found that AHCPR's patient/consumer guides are most frequently requested by phone and delivered in a timely manner. In addition, recipients are pleased with the services of AHCPR's publications clearinghouse and had very positive reactions to many of the guides' different features. This information will be used as performance data in AHCPR's Government Performance and Results Act Plan and will serve as a baseline for any future surveys fielded to determine satisfaction with changes made to Research Activities. During FY 1997, AHCPR also completed projects on evaluating minority health services research training activities (6384), identifying private sector organizations as potential research partners (6386), developing a workplan for preparing an inventory of performance measures currently being used or considered by HHS agencies (6691), and evaluating AHCPR's small conference grant program (6692).
Evaluations in Progress
Grant Application and Review Processes
AHCPR is conducting an evaluation of its grants (large and small) application receipt and review processes (6696). The evaluation includes a combination of qualitative and quantitative techniques that will document the extent to which changes made to the grants application submission and peer review processes have resulted in improved efficiency and effectiveness from the perspectives of applicants, reviewers, and Agency staff.
Data Collection Evaluation
AHCPR is undertaking a project to assess the usefulness to policymakers and researchers of AHCPR-supported major data collection efforts, such as the Medical Expenditure Panel Survey, the Health Care Cost and Utilization project, and the HIV Cost and Services Utilization Survey. More specifically, it will examine methods to improve survey operations and timely data release. The study will examine how other Federal agencies, as well as private foundations, conduct various aspects of their large-scale surveys (6383).
AGENCY FOR TOXIC SUBSTANCES AND DISEASE REGISTRY
MISSION: To prevent exposure and adverse human health effects and diminished quality of life associated with exposure to hazardous substances from waste sites, unplanned releases, and other sources of pollution present in the environment.
Evaluation Program
The Agency for Toxic Substances and Disease Registry (ATSDR) was created as a Federal agency by the Comprehensive Environmental Response, Compensation, and Liability Act (CERCLA), more commonly known as Superfund legislation. ATSDR was created to implement the health-related sections of CERCLA and other laws that protect the public from hazardous waste and environmental spills of hazardous substances. ATSDR relies heavily on strong working relationships with tribal, State, and local organizations; private, nonprofit, and community-based organizations; and other Federal agencies to implement critical environmental health programs. As part of the public health assessment process, ATSDR recommends actions that can be taken to reduce or mitigate the risks of adverse health effects from exposures to toxic substances to the Environmental Protection Agency (EPA) and State and local governments.
The ATSDR evaluation program is coordinated with the agencywide strategic planning process, which was started in March 1996 to implement requirements of the Government Performance and Results Act (GPRA). ATSDR's strategic goals and its annual performance plan are the result of an interactive process that reflects a long-term commitment by Agency staff to develop stronger relationships among external clients and stakeholders, to assess products and services using relevant data, and to improve our processes and systems for more efficient accomplishment of ATSDR's mission. The following strategic questions are important to the Agency:
‚ How should ATSDR evaluate the public health needs of its stakeholders and involve those stakeholders in the Agency's program planning, implementation, and evaluation?
Effectively focusing ATSDR's programs on these issues enables the Agency to understand stakeholders' needs, address its mandates, and evaluate its programs so that ATSDR operations, management, products, and services can be improved.
ATSDR receives its funds from the Hazardous Substance Trust Fund/Superfund appropriations rather than Public Health Service appropriations; therefore, ATSDR does not receive a 1-percent evaluation set-aside. Nevertheless, the new planning system provides the basis for measuring ATSDR performance and for making systematic improvements a part of its internal evaluation activities.
An example of such "in-house" evaluation activities through ATSDR resources is the Evaluation Working Group. This group was formed to develop a long-term strategy for site-specific evaluations to better assess the outcomes and impact of ATSDR activities at hazardous waste sites. The Methyl Parathion Evaluation Workshop was established to allow coordination between and continue work being accomplished by ATSDR, the EPA, and State and local agencies when responding to the illegal spraying of homes and businesses with methyl parathion, an agricultural pesticide. The workshop's intended purpose is to discuss approaches and reach a consensus on the best way to document the activities in the methyl parathion program.
Summary of Fiscal Year 1997 Evaluations
In FY 1997, ATSDR (1) developed an agencywide inventory of evaluation activities, (2) conducted an assessment of its site-specific evaluation needs, and (3) developed a guidance document for State cooperative agreement evaluation activities. These activities and information enabled ATSDR to assess why certain public health actions or decisions may or may not be working effectively. These activities also allowed for the identification of program strengths and weaknesses and suggestions to improve the efficiency and effectiveness of public health actions.
A number of States collaborated with ATSDR to develop a guideline for assessing the impacts and outcomes of site-specific health education activities. The documents described in the following paragraphs are the results of this collaborative effort. These tools will allow cooperative agreement participants to assess the overall improvement in public health as a result of site-specific health education activities and demonstrate the cooperative agreement program's effectiveness.
Site-Specific Outcome Measurement Guidelines for ATSDR Cooperative Agreement States (6853.1) provides guidelines for assessing the impacts and outcomes of site-specific health education activities. Evaluation of State Capacity Building--Program Announcement #607, published in the Federal Register, was also developed collaboratively by ATSDR staff members and State partners to assist in the overall evaluation of the effectiveness of the capacity building effort occurring in participating States. The evaluation also includes an assessment of ATSDR activities by State cooperative agreement personnel.
Next, ATSDR developed the document, Did We Make a Difference? ATSDR's Framework for Community-Focused Evaluation (6853) to provide guidance and information on how to conduct evaluations of ATSDR's site-specific programs and activities.
Evaluations in Progress
ATSDR has a number of ongoing evaluation activities. In addition to conducting traditional evaluation studies, ATSDR's commitment to viewing evaluation as an integral part of program planning is exemplified in the following activity now in progress.
Hazardous Substances Emergency Events Surveillance (6854). This evaluation of ATSDR's Hazardous Substances Emergency Events Surveillance (HSEES) program is based on the ability to measure the sensitivity and reliability of the system. Sensitivity is measured by the ability of the system to receive notification of events involving hazardous substances meeting the HSEES case definition. Reliability is measured by the ability of the system to receive uniformly recorded data from the 13 State health department partners.
To address the sensitivity of HSEES, ATSDR provides to each participating State health department a list of events involving hazardous substance releases that occurred in their State during the previous quarter and that were reported to one of two national data bases--the Department of Transportation's Hazardous Materials Information System or the EPA's Emergency Response Notification System. This activity provides the State partners with a mechanism to detect whether there were (1) incidents reported to one of the national data bases that were not reported to HSEES, (2) incidents reported to HSEES but not reported to one of the national data bases, or (3) incidents reported to both HSEES and a national data base.
To address the reliability of HSEES, ATSDR mails to participating State health departments each quarter a report describing a case study involving the release or threatened release of one or more hazardous substances. State health department partners review the case study and complete a hard-copy data collection form, which is returned to ATSDR. Then ATSDR compares the responses on these forms to each other and to ATSDR's preferred responses.
CENTERS FOR DISEASE CONTROL AND PREVENTION
MISSION: To promote health and quality of life by preventing and controlling disease, injury, and disability.
Evaluation Program
The Centers for Disease Control and Prevention (CDC) conducts evaluation studies designed to provide information essential for CDC's 11 strategies to achieve its mission:
CDC places high priority on evaluations to answer policy, program, and strategic planning questions related to the goals and objectives of Healthy People 2000. Performance improvement studies, such as those focusing upon the development of indicators consistent with the Government Performance and Results Act (GPRA), are of particular interest and import to the Agency. With the support of 1-percent evaluation funds, the GPRA planning process began at CDC in FY 1995 and is continuing. CDC's commitment to performance measurement is exemplified by this year's requirement that each project proposal be linked to one of CDC's strategic goals. These goals were identified through the GPRA strategic planning process.
On an annual basis, the Director of CDC provides guidance to the various Center, Institute, and Office Directors on 1-percent evaluation activities. This memorandum generally includes information about (1) types of studies to be carried out with 1-percent evaluation funds, (2) the format for submitting proposals, and (3) a timeline for selection. Each proposal undergoes multiple levels of review. Initial review is conducted by staff within the Office of Program Planning and Evaluation. Subsequent reviews are completed by the Office of the Assistant Secretary for Planning and Evaluation. Study authors are provided with comments, questions, and recommendations made by reviewers. In addition to providing their responses, authors are given the opportunity to revise their proposals at this time.
A panel of CDC evaluators, scientists, and program managers is convened to review and rank proposals. Review criteria include (1) relevance to prevention effectiveness, (2) relative importance of the public health problem being addressed, (3) probability that the proposed project will accomplish its objectives, and (4) extent to which other CDC programs will benefit from this project. Results from the review panel are presented to the Director of CDC for final funding decisions.
Finally, staff within the Office of Program Planning and Evaluation work closely with program staff to ensure development of a clear statement of work for selected projects. Prior to initiation of procurements, a final ad hoc review of the project's statement of work is completed.
Summary of Fiscal Year 1997 Evaluations
A total of 22 evaluation projects were completed in FY 1997. These studies are of four types: program evaluations, data policy and surveillance studies, development of performance measures, and assessments of specialized aspects of proposed or existing programs.
Program Evaluations
Six program evaluations were completed in FY 1997. These included an evaluation of the programmatic impact of CDC's 1-Percent Evaluation Program (6581 and 6581.1), an evaluation of suicide interventions in three Native American communities (5504), an evaluation design for the Business Responds to AIDS (BRTA) Program (6277), and an evaluation of the National Laboratory Training Network (NLTN) (5507). These studies represented three types of evaluations: informational, process, and outcome. For example, one of the studies looked at the design of BRTA, which was initially undertaken to help businesses create and implement comprehensive HIV/AIDS workplace programs, but also included questions that could help track trends in worksite policies, screening programs, and health promotion activities. In addition, one evaluability assessment that focused on CDC's Traumatic Brain Injury/Spinal Cord Injury program (6338) was completed.
Data Policy/Surveillance Studies
Four data policy/surveillance studies were completed in FY 1997. Two of the studies addressed reporting issues. For example, one study was conducted to ascertain whether the codes for International Classification of Disease (ICD)-10 should be modified to allow for greater specificity for morbidity use (5503). A second study in this reporting category evaluated data collected on birth records completed by multiracial and Hispanic women to see how they interpret the race question on the birth certificate (5918). Finally, two other studies addressed the area of surveillance, looking at the effectiveness of CDC surveillance for drug-resistant Streptococcus pneumoniae and the feasibility of developing a public health surveillance plan to assess current community public health. (6580, 6580.1)
Development of Performance Measures
In keeping with the congressional intent of the Government Performance and Results Act (GPRA) of 1993, one study was conducted in FY 1997 to support development of strategic planning at CDC (5928). This report offers a record of CDC's experience in its early implementation of GPRA by documenting the process, summarizing key findings, and making recommendations for strategies and approaches for the ongoing implementation of GPRA at CDC.
Assessments of Specialized Aspects of Proposed or Existing Programs
Eleven studies relating to discrete aspects of current or proposed programs were completed in FY 1997. Two of these projects were assessments of the Sexually Transmitted Disease (STD) prevention programs supported by the CDC (6579, 5348). One project was a comparative case study of local-level syphilis prevention efforts, while the other assessed the feasibility of parallel STD surveillance systems currently operated by CDC. Additionally, two studies were conducted to assess the effects of HIV prevention activities: (1) through HIV Community Planning to learn how community planning is confirming, enhancing, and changing HIV prevention programs (5927); and (2) through the Prevention Marketing Initiative, which is aimed to influence behaviors that contribute to the sexual transmission of HIV among young people below 25 years of age (6337). Another study looked at CDC's efforts to link health promotion and disease prevention research to public health practice (6535).
Three studies focused on issues related to women. The first of these studies identified and investigated factors that place women at risk for an abusive relationship and, in turn, how women can protect themselves from further violence during and after the relationship (6515). The second study focused on presenting successful strategies used by grantees of the National Breast and Cervical Cancer Early Detection Program (NBCCEDP) to reach women in need of mammography screening, especially low-income, underserved, and uninsured women (6273, 6273.1). The information will be shared with grantees in order to describe how various programs and interventions have reached marginalized populations. This information will also be useful for private sector and other organizations that are working to provide prevention services to hard-to-reach groups.
Two additional studies measuring the effectiveness of particular programs and activities were completed during FY 1997. These studies involved an evaluation of the National Immunization Survey (6477) and the Fatality Assessment and Control Evaluation (FACE) Program (5506.1). Finally, another activity focused on the first phase of a national study designed to profile the relationship between youths' access to tobacco and the contextual variables that define communities' policies, attitudes, and activities regarding youth and tobacco (6474).
Evaluations in Progress
CDC's evaluation studies in progress during FY 1997 consisted of four types: program evaluations, data policy and surveillance studies, development of performance measures, and assessments of specialized aspects of proposed or existing programs.
Program Evaluations
Nine program evaluations are currently in progress. Included are evaluations of the C. Everett Koop Community Health Center (CHIC) (6709), diabetes control programs (6584), the Field Epidemiology Training Program (FETP) (5923, 5923.1), evaluation of teen pregnancy interventions (6274), lead poisoning prevention training (6702), sexually transmitted disease (STD) partner notification (6711), impact of the Clinical Laboratory Improvement Amendments (CLIA) (6335), and CDC's Prevention Centers Program (5919). The study of the C. Everett Koop Community Health Center is a process evaluation that will look at the effectiveness of the center to serve as a national physician-based model. This will be accomplished by assessing (1) the effectiveness of the CHIC products and services from the users' perspective; (2) the use of the products and services, as well as barriers to their use; (3) the effectiveness of collaborative arrangements with nonprofit organizations, corporations, libraries, government agencies, and voluntary health agencies; and (4) the effectiveness of the products and services, such as quality of resources.
By contrast, the remaining studies in this category are outcome evaluations. The evaluation of CDC-supported, State-based diabetes control programs seeks to develop and implement an evaluation strategy to improve capacity for uniform assessment of States' progress in reducing the burden of diabetes on an evolving health system.
Similarly, the evaluation of CDC's Field Epidemiology Training Program (FETP) will ascertain whether the program has achieved its objectives, which are to train public health professionals in applied epidemiological skills, to promote the sustainability of autonomous FETPs, and to develop a global network of national programs.
The evaluation of teen pregnancy interventions will assess and test the adolescent pregnancy prevention initiatives in 13 cities within the United States. This project will involve the development and testing of efficient, systematic, and durable approaches to translating evaluation findings and program experiences of the 13 communities into changes in program interventions in demonstration and replication communities.
The last of the outcome evaluations involves the development of a systematic evaluation of notification to sexual partners of individuals who are diagnosed with sexually transmitted diseases. This project seeks to ascertain whether the substantial investment of State, Federal, and local health care dollars is decreasing disease prevalence or incidence when partners are notified.
Evaluation of Data Policy and Surveillance Systems
Ten studies are under way that relate to data management or reporting systems. Seven studies include evaluations of the U.S. Standard Certificates (6699), medical certification process for death certificates (4230), pregnancy-related violence (6712), Assisted Reproductive Technology (ART) embryo lab procedures (6716), use of data from Immunization Information Systems (IIS) (6713), evaluation of data uses of the National Electronic Telecommunications System for Surveillance (NETSS) and the Public Health Laboratory Information System (PHLIS) (6332), and the Urban Health Systems Sentinel Network (6276). In addition, three surveys are being conducted to evaluate guidelines for preventing perinatal HIV infection (6279), assess immunization knowledge and practice of primary care providers in the U.S. (5930.1), and inform the public about skin cancer (6710). The Skin Cancer National Survey examines the knowledge, attitudes, and behaviors of parents regarding skin cancer and the protection of their children from the sun. Results from this survey will enable CDC to design and distribute pertinent health messages about sun exposure.
The overall objective of the U.S. Standard Certificates evaluation is to examine the birth, death, and fetal-death information currently obtained from State vital registration programs to determine if relevant and high-quality data are being collected for State and national decisionmaking. The assessment will also consider what changes should be made to the standard certificates to improve their utility.
The other two evaluations are examining the use of survey instruments in data collection. In the area of pregnancy-related violence, a national population-based survey will be conducted of administrators and clinical providers in family planning programs. Information gained in the survey will be used to develop recommendations for the development of clinical guidelines. Similarly, since CDC is responsible for developing a model certification program for ART embryo laboratories in the United States, a survey will be conducted on the laboratories currently using ART to provide an enumeration of those procedures and practices.
Finally, at this point in the development cycle of the Immunization Information Systems (IIS), an assessment is being conducted on how the IIS data are used. The objectives of this project are to (1) assess how IIS data are used to assess public clinic and private provider performance and to identify structural and functional barriers to using the data; (2) assess how IIS data are used to assess community-based immunization coverage; (3) assess how IIS data are used to monitor the impact of changing recommendations in immunization schedule and the health care delivery system; and (4) promote the development of immunization registries.
Development of Performance Measures
Four ongoing projects involve the development of performance measurement systems. One of these projects, Development and Implementation of Strategic Planning at CDC, seeks to gather and analyze information that supports a comprehensive, unified, and data-driven strategic and performance planning framework for the CDC (6275). Three other projects focus on the development of indicators that will be used to assess HIV prevention efforts in the community, management and operational aspects of HIV community planning, and the coordination of multiple HIV prevention programs (5518, 6278, 6707).
Assessment of Specialized Aspects of Proposed or Existing Programs
Fifteen specialized studies relating to discrete aspects of current or proposed programs are in progress. These wide-ranging studies include evaluations of the development of a public health surveillance plan (5925), a feasibility study of Pap screening in STD clinics (6705), dental treatment and health status differences attributable to water fluoridation (6334), and the effectiveness of Group B streptococcal disease prevention guidelines (6333), still other studies assess the public health impact of the National Institute for Occupational Safety and Health (NIOSH) health hazard evaluation program (5922), tuberculosis outreach worker activities (5502), elder care groups (6708), small business interventions (6336), the Morbidity and Mortality Weekly Report (MMWR) recommendations for the use of bike helmets (6700), coordinated tobacco control efforts (6714), CDC's technical assistance to local health departments on violence prevention (5929), the efficacy of different types of vancomycin-resistant enterococci (6703), and HIV-related projects (6701, 6704, 6715, and 6706).
With the goal of complete eradication of tuberculosis by the year 2000, one of the studies focuses on an evaluation of tuberculosis outreach worker activities. Specifically, the project will (1) identify how these workers accomplish their tasks, (2) assess the quality of outreach workers' interactions with clients and supervisors, and (3) evaluate the amount of time spent on various research activities.
Many of the studies being conducted hope to provide the public with better information for all age groups. For instance, the elder care focus group study was undertaken in order to obtain data that will be used to produce an elder care health and safety handbook in the United States. One study focuses on the safety of our youth, as well as adults, with a desire to develop updated recommendations and guidelines for the use of bicycle helmets. In addition, an evaluation of tobacco control efforts will continue by looking at the impact and the relative cost effectiveness of tobacco control programs.
Several HIV prevention-related projects are in progress. Projects are being conducted to develop community indicators for HIV, to administer PCP prophylaxis for women and persons in underserved communities, to evaluate the current HIV technical assistance (TA) network, and to determine how the next iteration of TA could be improved. Finally, a study to determine the feasibility of developing a demonstration project to assess identification and screening of women at high risk of HIV infection is in progress. The community indicators study will identify (1) cultural and social-structural characteristics of communities that are associated with HIV/AIDS risk behavior and prevention efforts; (2) theories, assumptions, or hypotheses that explain the causal role of community characteristics; and (3) measures or indicators that could be used to assess changes in those characteristics that support sustained individual behavior change and community prevention efforts. This information will be used to support community-level research on HIV/AIDS prevention.
The goal of evaluating the CDC HIV Technical Assistance Network is to provide CDC staff who are responsible for planning the next iteration of a TA network and stakeholders of the current CDC-supported TA network with valuable information that can be used to develop and strengthen the existing TA model.
MISSION: To protect and promote public health through food, drug, medical device, and cosmetic regulation.
Evaluation Program
The evaluation program of the Food and Drug Administration (FDA) continues to be aligned with the Agency's policy, its program and strategic planning initiatives, and the goals of the Government Performance and Results Act (GPRA). Since GPRA and the reorientation of all government managers toward performance management, the responsibility for program evaluation has shifted from traditional specialized staff offices and contractor studies to managers of all FDA programs. In addition, FDA managers are identifying further opportunities to involve their customers in the design and testing of alternative ways of doing business. Both performance-based goals and customer participation also impact FDA's response to the needs of small businesses and enhance the participation and performance of small businesses.
One element of FDA's strategic framework is called external leveraging. The goal here is to augment the ability of external stakeholders to manage FDA-regulated risk. FDA's evaluation program illustrates the implementation of this goal with a number of studies under way or completed during FY 1997 that illustrate the strategies that FDA is using to accomplish this goal. One strategy is the fostering of an industry quality assurance program, where FDA will establish protocols to guide firms, certify their approaches, and audit industry practices. A current example of this quality assurance is the Mammography Quality and Standards Act (MQSA), which monitors the administration of mammography practices. The following reports are based on evaluations of mammography activities: the Mammography Quality Standards Act: Second Annual Government Entity Declaration Program Audit, and the Economic Impact Analysis of Regulations Under the Mammography Quality Standards Act of 1992. In addition, the Center for Safety and Applied Nutrition (CFSAN) has implemented the Survey of Current Hazard Assessment Quality Control Points (HAACP) Practices of the Food Industry. While excluding seafood, this survey originated as a result of the Seafood HAACP Program, another FDA-fostered quality assurance program.
Other CFSAN evaluations, both completed and in progress, are clearly in line with FDA's objective of communicating risk associated with FDA-regulated products. These studies are directly linked to the President's National Food Safety Initiative, which will involve a collaboration between FDA, the U.S. Department of Agriculture (USDA), the Environmental Protection Agency (EPA), and the Centers for Disease Control and Prevention (CDC) in order to assure safety and quality of our food supply. Special emphasis will be placed on microbial hazards, which account for 90 percent of the confirmed food-borne outbreaks, 6.5 to 33 million illnesses, and 9,000 deaths in the U.S. annually. Since food-borne chemical hazards often present chronic rather than acute health threats, specific estimates of their impact on health and the economy are not readily available. Finally, data collected from the FDA's Safe Alert Survey will demonstrate how well the Agency is communicating risk associated with medical devices to the medical community.
Summary of Fiscal Year 1997 Evaluations
Prescription Drug User Fee Act
The Prescription Drug User Fee Act (PDUFA) has demonstrated that the Agency and industry, by combining their resources and working together, can achieve the levels of performance the American people deserve. The fifth and final report completed in November 1997 demonstrates that the successes reported in FY 1996 have been confirmed and surpassed. PDUFA has resulted in more and better applications filed which can be accepted immediately, reviewed, and approved more quickly (6079.2). Nearly all applications now go directly into the review process, with "applications refused" approaching the vanishing point. There are substantially more approvals on first review, which demonstrates that submission quality is a key factor in achieving timely approvals. Another indication of improved submission quality is the dramatic increase in the percentage of submissions that are ultimately approved faster. By working to meet the PDUFA's stringent review goals, new products are getting on the market at an unprecedented pace without compromising safety.
Product Review Performance Report
In FY 1997, FDA completed its first annual report analyzing performance in reviewing applications for human drugs, biological products, medical devices, food and color additives, and veterinary drugs (6731). Responding to legislation (such as PDUFA) as well as to requirements under the Federal Food, Drug, and Cosmetic Act, FDA has developed clear performance goals for reviewing new product applications. Setting these goals has provided a valuable management tool for identifying performance expectations, assessing achievements, and meeting the requirements of GPRA. The report finds that (1) for most categories of products, FDA performance has improved since FY 1993; (2) the greatest improvements in performance have occurred for product reviews covered by PDUFA; (3) significant progress has been made in reviewing applications for medical devices; and (4) on-time performance for reviews of non-PDUFA applications should be improved.
Assessment of the Mammography Quality Standards Act of 1992 (MQSA)
Under this statute, FDA is charged with ensuring that accessibility to quality mammography services is maintained for all facilities performing mammographies. This program is FDA's second major user-fee, performance-oriented, GPRA-style initiative. During FY 1997, Eastern Research Group, Inc. completed an economic analysis of FDA's final rule under MQSA working under contract to the Office of Planning and Evaluation (OPE), Economics Staff. The purpose of the cost-benefit analysis was to predict the impact of the proposed standards on the costs of providing mammography. The results of their analysis are presented in the report entitled, Economic Impact Analysis of Regulations Under the Mammography Quality Standards Act of 1992 (6080). The results of this analysis will be used by FDA's managers to develop a performance-oriented regulatory policy that will be coordinated with existing customers. OPE's Economics Staff developed a model for estimating health outcomes to a patient population at various levels of mammography quality.
Quality Standards Act: Second Annual Government Entity Declaration Program Audit and the Economic Impact Analysis of Regulations Under the Mammography Quality Standards Act of 1992
FDA's Division of Mammography Quality and Radiation Programs requested an audit of the MQSA Government Entity (GE) Declaration Program (6730). The audit was undertaken by the Division of Planning, Analysis, and Finance. The goals of the audit were to notify facilities that they will be held accountable for their self-certification of fee exemption and to determine the rate of compliance with the program. Based on the results, the auditors concluded that the GE Declaration Program appears to be quite effective. Several recommendations were made based on the findings of the audit. First, the results should be published to allow the impact of the audit to reach most of the facilities. This would discourage any future misrepresentations and demonstrate that FDA will exercise its authority. Second, the sample size of future audits should be smaller and should contain a larger portion of those facilities claiming fee exemption based solely on funding from the CDC. Finally, the MQSA contractor should continue to examine each GE Declaration form to ensure that it identifies the body operating the facility.
Evaluation of Device Performance in the Determination of Substantial Equivalence
Many diagnostic medical devices that reach the market are based on diagnostic agreement with some device already on the market that targets the same attribute. Agreement is often assessed by side-by-side testing of the same subjects or specimens, even if the true disease status is unknown. If each new test is compared with the most recent predicate, what would be the effect on device performance over time? Would it deteriorate, remain the same, or improve? FDA produced an analysis that addressed these questions using statistical measures of sensitivity, specificity, and predictive values (6734). When a new test was found to be equivalent to the current predicate, it became the new predicate. After 50 new predicates were found, a forecast procedure was applied to the sequence of 50 performance parameters in order to evaluate the long-term trend.
Center for Food Safety and Applied Nutrition (CFSAN) Evaluation
During FY 1997, three evaluations relating to food issues were completed by the Economics Branch in CFSAN's Division of Market Studies. The first, Preliminary Investigation into the Morbidity and Mortality Associated with the Consumption of Fruit and Vegetable Juices (6858), assessed the population risk from the consumption of fruit and vegetable juices. The main risks studied were microbial pathogens. Two studies evaluated regulation of the seafood industry: Cost of Restrictions on Gulf Oyster Harvesting for Control of Vibrio vulnificus-Caused Disease (6856), an economic impact analysis of alternative remedies for controlling Vibrio vulnificus-related illnesses in the Gulf States oyster industry, and Health Transfers: An Application of Health-Health Analysis to Assess Food Safety Regulation (6857), an evaluation tool based on the first study's methodology that can be used to measure economic versus health costs of public health decisions.
Evaluations in Progress
FDA currently has a broad range of evaluations in progress which, as in previous years, are based upon the two driving forces relating to GPRA goals: performance management and customer participation (with particular focus on small businesses). The following are examples of FDA evaluations in progress during FY 1997.
Although PDUFA ends in FY 1997, FDA and the pharmaceutical and biological prescription drug industry have negotiated re-enactment beyond FY 1997. Reauthorization of the collection of user fees under the Food and Drug Administration Modernization Act (enacted in November 1997) will continue to enhance the review process of new human drugs and biological products (6079.3). The PDUFA program serves as a model for reinventing government, with the Congress, FDA, the industry, and consumer groups working together to provide the necessary resources, set performance goals, and hold the Agency accountable. Revenues generated from the fees paid by the pharmaceutical and biological prescription drug industries will continue to be dedicated to improving and expediting the prescription and application review and approval process. The legislation reauthorizing PDUFA for FY 1998 was critical in ensuring uninterrupted service to U.S. customers.
Evaluation of Influenza Virus Vaccines in High-Risk Patient Categories (4355) is intended to ensure public health by investigating the reactogenicity and immunogenicity of influenza virus vaccines. The information will be critical to making accurate recommendations for the selection of new strains of the influenza virus vaccine to be used in the United States.
Nationwide Evaluation of X-Ray Trends (NEXT) (4984) is a collaborative program conducted jointly between FDA and State radiation control agencies. The study estimates the radiation dosage from diagnostic x-ray examinations using data collected from previous NEXT surveys of different types of x-ray examinations. The data are used by public health and professional organizations in setting policy.
Food Labeling and Package Survey (FLAPS) (5711) was implemented by the FDA in 1977 to produce a data base containing label and package information obtained from a sample of food products from the processed packaged food industry. FLAPS data enable FDA personnel to keep abreast of market responses to food labeling rules via changes in product package labels.
FDA Penalty Reduction Program for Small Businesses (6461) reviews pre-existing and revised policies related to monetary penalties for noncompliance with FDA regulations, evaluates the scope of small businesses that qualified (or failed to qualify) for the program, and assesses the total amount of penalty reductions or waivers. The Small Business Regulatory Enforcement Fairness Act (SBREFA) required FDA to create a program or policy that specifies the circumstances under which the Agency may reduce or waive the penalties levied against small business entities for noncompliance. The final report for this project is anticipated in March 1998.
FDA Inquiries Assistance Program for Small Businesses (6462) is FDA's response to the SBREFA requirement to establish a program to respond to inquiries and to help small businesses and other small entities apply FDA rules. The report on this project is anticipated in March 1998.
During FY 1998, FDA will continue to implement the requirements of the Mammography Quality Standards Act of 1992 (MQSA) (6080.1). FDA requested an increase in MQSA authorized inspection fees over the FY 1997 level. Continuing implementation of the MQSA requirements is essential to ensure that women receive quality mammography from facilities that maintain a high standard of safety and accuracy. The primary emphasis of this program for FY 1998 will be to continue to ensure that all facilities meet the quality standards and that deficiencies previously identified through annual inspections are corrected. In FY 1998, Federal and State personnel will continue to conduct annual inspections of 10,000 facilities and certifications of 6,000 facilities, and to provide training for new inspectors. The fees will be used to pay for the costs of the inspections.
Review of Therapeutic Agents as an Antimicrobial Adjunct in Treatment of Sepsis (6736) is a technique used in the drug approval process. No therapeutic agent has yet been identified that shows either a clinical benefit or an alternative that is superior to standard care. FDA faces many challenges in designing clinical trials that demonstrate efficacy. These are due to the heterogeneity of patient populations, poor understanding of the pathophysiology of acute sepsis, and no uniformly-accepted standard of care in these international trials. Intensive efforts have been undertaken by FDA to propose the optimal trial design and prospective analysis plans. This analysis of failed sepsis trials is being conducted by FDA to achieve better clinical trial results for the evaluation of adjunctive sepsis therapy in the future.
The Real-Time PMA Supplement Program Evaluation (6732), implemented by the Office of Device Evaluation (ODE), has evolved as a result of the Medical Device Review of the Premarket Approval (PMA) Applications program. FDA has reduced review times and backlogs for medical device applications. But, in order to achieve mandated goals, continued improvement is needed. FDA requested an increase in funds for FY 1998 in order to improve the quality and timeliness of its review process for Class III PMA applications and PMA supplements. These applications involve the highest-risk devices that require full clinical investigations of safety and efficacy. An increase in funding will facilitate FDA in allocating more resources to the PMA process without sacrificing performance gains made in the 510(k) program. It is projected that FY 1998 will be the first year of this enhancement effort, and FDA expects a 15 percent increase in PMA workload during FY 1997 and FY 1998 due to requests for reclassifications and submissions of required preamendment PMAs. Given these additional resources, FDA will attempt to limit the increase in pending PMAs at the end of FY 1998 to 5 percent. FDA has defined specific review performance goals for Class III applications that should be achieved by the end of FY 1998 with the additional resources.
The Safe Alert Satisfaction Survey (6733) will determine the impact of the safety alert and public health advisory on the behavior and knowledge of the public. The collection of these data is an important step in determining how effectively FDA is communicating risk to the public.
Survey of Current Hazard Analysis and Critical Control Point (HACCP) Practices of the Food Industry (6866) is a two-part survey that will yield information that will be used to estimate the HACCP-related practices of the food industry (except seafood, meat, and poultry), and a model that will be used to estimate the marginal costs of firms to comply with HACCP.
Economic Characterization of the Dietary Supplement Industry (6867) will provide a detailed analysis of the dietary supplement industry, including a description of the types of firms producing and marketing dietary supplements, and will create a relational data base that will link firms to products.
Estimating the Benefits of Various Kinds of Food Standards (6865) will categorize food standards by type of food and function and to estimate consumers' valuations of those functions.
Harmonizing Nutritional Goals and Food Processing (6864) is a paper to be given at the "Towards Convergence on the Dietary Guidelines, Research and Policy Needs in the Nutrition, Health, and Agricultural Sectors" conference on April 30, 1998.
The Impact of Tobacco Initiative on Youth (6890) is a project in which Office of Planning and Evaluation staff are currently designing a sample questionnaire to facilitate data collection on tobacco use. The survey responder will determine whether the tobacco initiative has had any positive impact on the propensity of youth to use these products. In August 1996, President Clinton approved FDA's final rule for regulating tobacco products. The President's Tobacco Initiative demonstrates a collaborative effort to thwart major public health risk with the objective of decreasing tobacco use over the next 7 years. FDA is working with the States to conduct unannounced compliance checks of retail establishments that sell tobacco products. FDA will rely on Federal and State agencies to monitor retailer compliance with the new regulations and track the degree of success in decreasing tobacco use by youth.
HEALTH CARE FINANCING ADMINISTRATION
MISSION: To promote the timely delivery of appropriate, quality health care to the Nation's aged, disabled, and poor through administration of the Medicare and Medicaid programs.
Evaluation Program
The research arm of the Health Care Financing Administration (HCFA)--the Office of Strategic Planning (OSP)--performs, coordinates, and supports research and demonstration projects (through intramural studies, contracts, grants, and waivers) to develop and implement new health care financing policies and to provide information on the impact of HCFA's programs. The scope of HCFA's research and demonstration activities embraces all areas of health care: cost, access, quality, service delivery models, and financing approaches. OSP's research responsibilities include evaluations both of the ongoing Medicare and Medicaid programs and of demonstration projects testing new health care financing and delivery approaches. These projects address four major themes.
Summary of Fiscal Year 1997 Evaluations
During FY 1997, HCFA completed 21 evaluations, which are briefly described below.
In, Access to Physicians' Services for Vulnerable Beneficiaries: Impact of the Medicare Fee Schedule (6301), HCFA evaluated the effect of the physician fee schedule on a beneficiary's access to care. Six different strata of beneficiaries were selected based on the relative size of the payment change under the fee schedule compared to the earlier payment scheme. The project examined the use of services, outcomes of services, and the change in the beneficiary's financial liability.
The Impact of the Medicare Fee Schedule on Teaching Physicians (6666) was started because of four concerns: (1) teaching physicians perform more high-technology procedures and less primary care, which could result in disproportionate net loss for these practices; (2) teaching hospitals and medical schools tend to be located in large urban areas, where the Geographic Practice Cost Index portion of the payment is lower; (3) customary charges for teaching physicians may be higher than the areawide historical payments calculated for fee schedule transition; and (4) a number of academic medical practices had not signed Medicare participation agreements. For this study, 720 hospitals were selected, including all U.S. academic medical centers. The results indicated that there is no evidence that teaching physicians have been adversely affected by the Medicare Fee Schedule.
1995 Influenza Immunizations Paid for by Medicare: State and County Rates (6615) is a data book tied to the program begun in 1993 to stimulate the provision of flu vaccine to Medicare beneficiaries. It provides information about National, State, and county rates of vaccination in fee-for-service settings (managed care beneficiaries could not be counted because their encounter data are not reported).
The Department was legislatively required to report to the Congress on the relative quality of the care in the Medicaid program. HCFA's A Study of the Appropriateness, Process of Care, and Outcomes of Medical Care for Medicaid Patients (6302) examined the necessity, appropriateness, and effectiveness of selected medical treatments and surgical procedures for Medicaid patients. It assessed the variation that exists in the rate of performance of selected treatments and procedures on Medicaid beneficiaries for small areas within and among States. It determined that there are underutilized, medically necessary treatments and procedures for which failure to furnish could have an adverse effect on health status. The study had two phases. The first used extant data in HCFA to examine patterns of utilization for over 160 different conditions and procedures. It found that, for all conditions combined, utilization was higher in the Medicaid population and that the differences were higher for medical than for surgical conditions. Within Medicaid, medical conditions showed somewhat more small-area geographic variation than did surgical conditions. For all conditions combined, the Medicaid population tended to exhibit more small-area geographic variation than did the non-Medicaid population, but the pattern was not consistent across States. In the second phase, data were abstracted from medical records at 118 hospitals in California, Georgia, and Michigan for a selected series of index diagnoses: pediatric asthma, hysterectomy, and complicated labor and delivery. Five indicators were developed to assess the level of quality. In general, Medicaid women and children were treated in public hospitals, teaching hospitals, and hospitals with large bed capacities. The appropriateness of the decision to hospitalize was comparable between Medicaid and the privately-insured. And, once hospitalized, they received a similar level of care. Most importantly, the outcomes of care were essentially similar between the two payer groups, although the Medicaid patients were more severely ill. As a result, there were differences in outcomes that were not related to the inpatient quality of care provided. The inpatient care for the Medicaid patients was considered adequate and comparable with care received by privately-insured patients.
Evaluation of the Medicaid Uninsured Demonstrations (6298) assesses three demonstrations conducted in the States of Maine, South Carolina, and Washington. These demonstrations, implemented in response to a Congressional mandate under section 4745 of the Omnibus Budget Reconciliation Act of 1990, tested the effect of allowing States to extend Medicaid coverage to low-income families. The project examined the ability of the Medicaid programs to enroll significant numbers of eligible persons, looking particularly at the conditions under which eligible persons and their families are willing to participate in such programs, given their scarce financial resources. The evaluation also looked at the ability of the programs to induce adequate levels and the willingness of employers to participate in the programs.
The Inspector General of the Department conducted a demonstration of improved methods for the investigation and prosecution of fraud and abuse. The Evaluation of the Effectiveness of the Operation Restore Trust Demonstrations (6311) examined whether this more concentrated effort, rendered through the demonstration's partnership model, had a relatively greater impact on health industry fraudulent behavior than the same efforts conducted separately. Operation Restore Trust (ORT) involved a targeted set of initiatives that focused on a few States, a few areas of the Medicare benefit, and a few standard operating procedures (especially, failures to coordinate among the most central agencies and to exploit available data resources). The evaluation concluded that ORT turned around certain troubling trends of the early 1990's and has brought new vigor to government efforts to combat health care fraud and abuse. ORT had important effects across the full spectrum of organizations it sought to affect, and even had some important effects on organizations over the boundaries of the demonstration (e.g., Medicare contractors) that were not so directly orchestrated by the ORT initiative.
Assessing the Viability of All-Payer Systems for Inpatient Hospital Services (6594) explored the feasibility and possible structure of alternative approaches to developing such a system. It looked at the practicality of adapting or extending Medicare's Prospective Payment System to other payers. Two analyses were conducted to determine (1) whether diagnosis-related group-based payment rates for Medicare and non-Medicare patients should be adjusted based on hospital characteristics, and (2) alternative approaches in defining the Medicare standardized prospective payment rate. Current Medicare standardized prospective payment rates are based on the mean adjusted cost for a given category of hospital; high-cost hospitals are penalized under the assumption that their costs reflect presumed inefficiency. It developed hospital-specific measures of inefficiency and then excluded the costs of inefficient hospitals from the calculation of standardized payment rates. The conclusion was that a frontier cost function model approach is superior to the current system and can result in substantial program savings.
Monitoring and Evaluation of the Medicare Cataract Surgery Alternate Payment Demonstration: Final Evaluation Report (6295) assisted HCFA in tracking the 3-year demonstration that tested the concept of a single global Medicare payment for outpatient cataract procedures. The payment amount represented the cost for physician, facility, and intraocular lenses. The evaluation found that the project was reasonably successful in encouraging provider flexibility and in managing the bundle of services. There was ample anecdotal evidence that the cost-effectiveness incentives were working; e.g., cost reduction efforts by surgeons in the techniques and time required to perform the surgery and actions to standardize the intraocular lenses and other supplies and materials. Once operations began, government activities related to pricing were greatly reduced and limited to simple verification and payment of the global fee. The evaluator was not able to measure appropriateness, but in terms of quality control and utilization review, the demonstration was successful in that the outcomes were unchanged. The project offered some insights into the strengths and weaknesses of particular incentives in encouraging providers to participate. However, it provided only limited new information on the factors that influenced beneficiaries to participate.
HCFA completed an evaluation on the implementation of ambulatory patient groups (APGs) under the title, Toward a Prospective Payment System for Outpatient Services: Implementation for Outpatient Services-- Implementation of APGs by State Medicaid Agencies and Private Payers. (6320) The focus of this study was the Medicaid outpatient prospective payment system that groups patients for payment purposes rather than paying on a cost basis. These groups are similar in concept to diagnosis-related groups, which form the heart of Medicare's prospective payment system for inpatient care. It involved a case study of Iowa's implementation of the APG system and an analysis of the reimbursement methodology. All six payers that had operational experience tailored their APG systems to their own priorities and markets. They all implemented the system without major incident. The payers reported success in reducing outpatient costs where that was the immediate goal, and they believe the system encourages higher-cost facilities to reduce costs and rewards lower-cost facilities. The views of providers were more mixed. Though they reported generally adequate overall payment, they also viewed the system as complex, could not calculate expected payment, and often let their computer systems group related claims rather than consolidating them prior to submission, as was the intent of the system. No evidence was found of much behavioral response by providers, but that could change if the system is implemented by a large payer such as HCFA.
The Second Update of the Geographic Practice Cost Index: Final Report (6611) was produced and is scheduled to be fully phased in by January 1, 1999. One-half of the total change will be implemented in 1998. A comparison between the 1997 and 1999 Geographic Practice Cost Indexes (GPCIs) shows that (1) there is no change between the physician work GPCI or the non-physician employee wage index of the practice expense GPCI, except that they are reweighted with 1994 county relative value units; (2) the office rental index of the practice expense GPCI is updated with FY 1996 Fair Market Rents (from the Department of Housing and Urban Development); (3) county rental adjustment factors are eliminated for all metropolitan areas (except for the New York City Primary Metropolitan Statistical Area); (4) the malpractice GPCI is updated with 1992, 1993, and 1994 premium data, and specialty premiums are weighted by national relative value unit proportions rather than allowed charges; (5) there is no change in the GPCI cost shares, which measure the share of the various inputs in total practice revenues; and (6) the county relative value units used to weight averages to county input prices for each Fee Schedule Area are updated from 1992 to 1994. The project determined that the changes in the Medicare Fee Schedule resulting from implementation of this GPCI update will be less than 2.5 percent for each Fee Schedule Area.
The Payment of Pharmacists for Cognitive Services: Results of the Washington State C.A.R.E. Demonstration Project (6528) is a report on a demonstration project called Cognitive Activities and Reimbursement Effectiveness (C.A.R.E.). It is operated under the premise that (1) direct reimbursement for cognitive services would remove financial barriers associated with the pharmacists' provision of these services; and (2) that this change would result in more consultations, with a subsequent impact on costs and outcomes from drug therapy. Data were drawn from three groups of approximately 100 pharmacies in the State of Washington. The project found that (1) pharmacists who received a financial incentive consistently reported higher cognitive service intervention rates than did those who did not receive financial incentives; (2) about half of all documented service problems were for patient-related matters (about one-third were for drug-related problems, about 17 percent were for prescription-related problems, and the residual 1.4 percent were for non-drug problems); (3) a drug therapy change occurred as a result of 28 percent of all pharmacist cognitive services; (4) for each cognitive service that was associated with any type of drug therapy change, the average downstream drug cost savings was about $13.05 (but these savings varied by type of drug therapy); and (5) the demonstration generated savings of about $78,000 in Medicaid prescription drug expenditure over the 18-month period.
The Consequences of Paying Medicare HMOs and Health Care Prepayment Plans (HCPPs) (6314) compared the cost effectiveness of such plans to fee-for-service (FFS) and health maintenance organizations (HMOs), which operate on a risk basis. The main question examined in this study was whether HCFA payments would have increased or decreased had these organizations remained risk contractors. The evaluator found that costs to HCFA increased substantially under the cost programs, HMO, and Health Care Prepayment Plan (HCPP). Furthermore, most of the cost-based plans were found to have favorable selection. Total dollar losses to HCFA for 1993 due to the cost, HMO, and HCPP programs were estimated to be $185 million relative to fee-for-service, and $4 million relative to risk-based payments. Dropping the seven plans that converted to risk or dropped out of Medicare since 1993 would have resulted in estimated losses of $210 million compared to FFS, and $90 million compared to risk contracting.
The Costs and Consequences of Case Management for Medicare Beneficiaries: Final Report (6291) studied the appropriateness of providing case management services to beneficiaries with catastrophic illnesses and high medical costs. It tested case management as a way of controlling costs in the fee-for-service sector. The target conditions and case management protocols differed in each site, though all three generally focused on increased education regarding proper patient monitoring and management of the target chronic condition. All three sites anticipated reduced hospitalizations and medical costs compared to the beneficiary control groups. The evaluation found that (1) the projects successfully identified and enrolled populations of Medicare beneficiaries likely to have much higher than average Medicare reimbursements during the demonstration period; (2) each project met with unexpectedly low levels of enthusiasm for the demonstration from beneficiaries and their physicians; and (3) despite high levels of satisfaction among the high-cost, chronically ill beneficiaries who eventually participated, the projects failed to improve client self-care or health, or reduce Medicare spending. Comparisons of health status, functional status, and expenditures between the control and intervention groups showed no improvements resulting from the case management intervention.
The Validation of Nursing Home Quality Indicators Study (6612) developed a system by which a series of indicators could be used, in conjunction with claims data, to monitor quality of care in nursing facilities. Fifty such indicators were developed and validated. The indicators focused on three broad problem areas: adverse outcomes, lack of therapy, and inappropriate pharmaceutical treatments. The indicators were designed to be used in an automated system to continuously monitor the quality of care provided. The study found that (1) it is possible to base such indicators on Medicaid and Medicare claims data; (2) as a whole, the indicators were better at predicting the absence of a quality issue (suggesting that the negative finding represented the lack of a quality concern in a specific nursing home); and (3) using covariant diagnoses was useful to adjust for the risk of a resident receiving an indicator tag.
The Evaluation of the Community Nursing Organization Demonstration Interim Evaluation Report (6306) examined a set of demonstration projects that tested a capitated, nurse-managed system of care. It was designed to promote timely and appropriate use of community health services and to reduce the use of costly acute care services. More than 5,400 beneficiaries were enrolled in the four Community Nursing Organization (CNO) demonstration sites. The evaluation report covers the first 15 months of operations. The applicants have been healthier and more independent, on average, than the general Medicare population in their respective market areas. Analyses of measures of death, physical functioning, and satisfaction with care detected no statistically significant differences. Analysis of a subsample of persons who were impaired in Activities of Daily Living indicated that members of the CNO treatment group were more likely to improve (or less likely to deteriorate further) than members of the control group. An even stronger result, in the same direction, was found for individuals who were limited in Instrumental Activities of Daily Living.
The Evaluation of the Utah Prepaid Mental Health Plan (6293) focuses on a demonstration project in which three mental health centers provided services to all Medicaid beneficiaries in their catchment areas (these areas included over 50 percent of all Utah Medicaid beneficiaries). It was expected that this program would control the cost inflation and improve patient outcomes in mental health services. The study examined (1) the organizational and financial characteristics and their evolution over time, (2) the impact on the organization of service delivery and use of services, and (3) the financial impact on providers and the Medicaid program in Utah. The development of the demonstration proceeded fairly smoothly, albeit somewhat more slowly than planned. Three centers chose to enter the demonstration and eight chose to not join. Several changes in the program's environment affected the development during the initial 3 years. There were general expansions in children's programs, day treatment programs, and case management. The community mental health centers began to focus staff attention on ways to increase the efficiency with which discharges from the hospital were managed and on the way outpatient services were delivered--all while maintaining quality of care. The results suggest that the demonstration reduced expenditures on acute inpatient care by reducing admissions, without significant effects on outpatient expenditures and visits. Financial impact was possible in that the demonstration altered the way in which Medicaid funds flowed to the participating community mental health centers and it redefined their responsibilities with respect to the provision of services. In addition, there were relatively few financial differences between contracting and non-contracting community mental health centers. Thus, the decision to participate appears to have had a null effect on the site's financial performance. The payments to the participating sites were less than estimated. In the second method of looking at payments to the sites, there were savings on inpatient care, which were partially offset by increases in outpatient care for a net overall savings. The overall estimates of total program savings showed this same pattern of inpatient savings, diminished by outpatient increases. However, these estimates are very sensitive to the methods used to project trends.
The Limited-Service Hospital Pioneers: Challenges and Successes of the Essential Access Community Hospital/Rural Primary Care Hospital (EACH-RPCH) Program and Medical Assistance Facility (MAF) Demonstration (5200.1) concerned a legislatively mandated attempt to assist States in maintaining access to health care services in rural areas. The program awarded grants to several States to develop rural health plans, establish rural health networks, and create a limited service alternative for communities that could no longer support a full-service hospital. These limited service providers and other health care providers were organized into rural health networks. This evaluation examined the development, implementation, and early operating experience of the program. In addition, it looked at a forerunner program, the Montana Medical Assistance Facility Demonstration. It found that these limited service hospital models occupy a niche in rural health care system by permitting an alternative for outright closure. However, there were considerable limitations to participation. The financial benefits varied by facility, depending on such things as cost structure, provider supply, and patterns of use in the specific community. It found that program flexibility and the linkage of primary care services to developing networks was essential.
The Comparative Study of the Use of EPSDT and Other Preventive and Curative Health Care Services by Children Enrolled in Medicaid: Final Project Synthesis Report (6236.7) included (1) a description of the Medicaid program in the four study States (Michigan, Georgia, Tennessee, and California) and these States' responses to the 1989 legislative changes relating to children's and Medicaid coverage and also to the Early Periodic Screening, Diagnosis, and Treatment (EPSDT) program; (2) an analysis of the impacts of the State's responses on participating providers and enrolled children; and (3) an assessment of the national context. The States' efforts to increase provider participation in Medicaid and EPSDT were successful in increasing provider participation and average caseloads among all types of providers (except for dentists in Michigan). However, the increased supply may not have been adequate to meet the increased demand because the ratios of Medicaid children to participating providers and percentages of counties with provider shortages rose from 1989 to 1992 in all four States. Only 42 to 54 percent of Medicaid children were recommended to have preventive care visits and, of these, only 36 to 59 percent made such visits. Compared with other low-income children, Medicaid coverage did increase service use and improved access to illness-related care. However, Medicaid children had very low completion rates for age-appropriate immunizations.
The Evaluation of the Demonstration to Improve Access to Care for Pregnant Substance Abusers (6297) sought to evaluate project effectiveness in (1) outreach and assessment; (2) expansion, integration, and coordination of program services; and (3) improvement of client case management. The evaluator examined the effects of access to prenatal care and substance abuse treatment services on the health of drug-addicted pregnant women and birth outcomes of their infants. The evaluation showed that the number of abusers enrolled in the demonstrations was low relative to all pregnant substance abusers in the area, since women were reluctant to be identified. The project found higher enrollment rates in States that implemented broad-based outreach efforts, and that higher levels of participation and greater retention in substance abuse treatment resulted in higher-birthweight infants.
An Evaluation of Implementation of the Medicaid Community Supported Living Arrangements (CLSA) Program in Eight States (6304) was designed to test the effectiveness of developing a continuum of care concept as an alternative to the Medicaid-funded residential services provided to individuals with mental retardation and related conditions. The program served individuals who are either living in the community independently, with their families, on in homes with three or fewer other individuals receiving the same services. These services were provided in eight States between fiscal years 1991 and 1995. By the end of the program, only 73 percent of the expected number of individuals were receiving services in the Community Supported Living Arrangement (CSLA), and the expenditures were only 58 percent ($38.4 million) of the amount authorized.
The Impact of Report Cards on Employees: A Natural Experiment (6312) sought to determine whether the dissemination of information about health plans to consumers (who choose health plans within a managed care competition framework) would influence their knowledge of plan characteristics, attitudes toward the plans, or choice of plan. Investigators found that report cards had no effect on employees' knowledge of health plans, their attitude about quality of health plans, or their choices in selecting a managed care health plan. The study did find that health plan characteristics (such as price) were strongly related to health plan choice. The investigators suggest that consumers do not seem to be influenced by the information in any of the ways measured in this study.
Evaluations in Progress
HCFA currently supports 19 evaluation projects in progress, 13 of which are scheduled to be completed in FY 1998. These projects provide information for continued monitoring of the Medicare and Medicaid programs and assess the impacts of HCFA's Medicare and Medicaid demonstration projects.
The Evaluation of Medicare SELECT (5966.3) examines a pilot Medicare supplemental insurance product under which full Medigap benefits are paid only when services are provided by the plan's provider network. Case studies, primary surveys, and administrative data (claims) analyses are conducted. The analytical portion of the project compares cost and use of Medicare and supplemental services, selection effects, beneficiary satisfaction, and physician practice patterns to other Medigap options. To date, the evaluation has found no health status differences or quality of care received by SELECT and non-SELECT beneficiaries. In addition, there were no differences in overall satisfaction levels between these populations. Premiums for SELECT policies were almost always cheaper than the same company's own standard Medigap policy for any age group. In two out of three years examined, SELECT policies were generally more expensive for older beneficiaries when compared to community rated policies. Medicare costs generally increased as a result of Medicare SELECT. The costs increased in five States, decreased in three States; and were not affected in three other States. Since the States studied were not chosen to be representative of the Nation, their experience is not necessarily indicative of a national program. Consequently, the effect of SELECT on Medicare program expenditures appears to be mixed at best.
The Medicare Participating Heart Bypass Center Demonstration Extended Evaluation (5985.5) continues to study the feasibility of a negotiated, all-inclusive pricing arrangement for coronary artery bypass graft surgery while maintaining high-quality care. The project looks for any net cost savings to the Medicare program, any volume increases at the demonstration sites, the aspects of the demonstration that were attractive to beneficiaries and referring physicians, and whether the quality of care at the sites was equivalent to that provided prior to the demonstration.
The Medicare HMO Evaluation (6294) updates the findings of an earlier study of Medicare risk health maintenance organizations. That study found that Medicare paid 5.7 percent more for plan enrollees than would have been spent under fee-for-service. The current study looks at disenrollment, beneficiary satisfaction, quality of care, and selection and savings.
The Evaluation of Medicaid-Managed Care Programs with 1915(b) Waivers (6290) will provide information on the extent to which various features of the managed care waiver projects contribute to the ability of the Medicaid program to deliver cost-effective care to eligible populations.
Maximizing the Effective Use of Telemedicine: A Study of the Effects, Cost Effectiveness, and Utilization Patterns of Consultation via Telemedicine (6303) is a comprehensive evaluation of HCFA's telemedicine demonstration projects. The project assesses alternative payment options for the providers of telemedicine services, such as fee-for-service, facility payments, or capitation. Objective measures of the cost of telemedicine services in different clinical settings are used, both from the payer's and the physician's perspective. Provider and patient satisfaction, utilization measures (e.g., physician visits, hospitalizations or rehospitilizations, and frequencies of complications or comorbidities), and the effect of telemedicine on practice patterns are included.
The Drug Utilization Review Evaluation Contract (6296) is examining the impacts of retrospective and prospective review, which included the payment of pharmacists for cognitive services. Demonstrations of on-line prospective drug utilization review (DUR) in Iowa and of cognitive service by Washington State pharmacists provided the primary data for evaluation. This evaluation looks at the two types of DUR program effects in preventing adverse drug-related events by incorporating and analyzing the program participants' pharmaceutical and medical information.
The Evaluation and Technical Assistance of the Medicare Alzheimer's Disease Demonstration (6305) is being conducted to determine the effectiveness, cost, and impact on health status and functioning of providing comprehensive in-home and community-based services to beneficiaries with dementia. Two models of care will be studied. Both include case management and an extensive package of services not presently covered under Medicare, such as adult day care, homemaker/personal care services, companion services, and caregiver education and training. The models are differentiated by the intensity of the case management provided to beneficiaries and their families and by the monthly amount of Medicare payments for project services. The purpose of the evaluation is to measure the cost and effect on the beneficiaries' functional level and institutionalization rate, as well as its impact on caregiver burden and stress.
Evaluation of Phase Two of the Home Health Agency Prospective Payment Demonstration (6308) examines the second phase of a program designed to test the effectiveness of using prospective payment methods to reimburse Medicare-certified home health agencies (HHAs). In this demonstration a per-encounter payment method is being tested. The study looks at the effects of this payment method on HHAs operations, service quality, and expenditures. It also analyzes the relationship between patient characteristics and the cost and utilization of home health services.
The Examination of the Medicaid Expansions for Children (6300) uses enrollment and expenditure trends to ascertain the impact of the 1989 program changes. It will look at target population penetration, and the impact of State policies (and the eligibility group) on enrollment, expenditures, and utilization of services.
The Project Demonstrating and Evaluating Alternative Methods to Assure and Enhance the Quality of Long-Term Care Services for Persons with Developmental Disabilities through Performance-Based Contracts with Service Providers (6310) tries to determine whether and how well the implementation of new approaches to quality assurance--with outcome-based definitions and measures of quality--will replace input and process measures of quality in this population group.
In future years, several other evaluations are expected to be completed.
The Evaluation of the Medicare Choice Demonstration (6292) assesses the feasibility and desirability of new types of managed care plans for Medicare. These plans can be integrated delivery systems and preferred provider organizations.
HCFA is conducting demonstrations testing the payment of skilled nursing facilities on a prospective basis for Medicare and Medicaid services. Medicare currently reimburses such facilities on a retrospective cost basis. This demonstration uses a case-mix classification system (called Resource Utilization Groups) to classify patients and establish payment rates. The Evaluation of the Nursing Home Case-Mix and Quality Demonstration (6307) seeks to estimate specific behavioral responses to the prospective payment and to test hypotheses about aspects of such responses. The main goal of the project is to estimate the effects on the health and functioning of the nursing home residents, their length of stay, and use of health care services; on the behavior of the facilities; and on the level and composition of Medicare expenditures.
The State of Delaware is experimenting with its Medicaid program. The State believes that by enrolling Medicaid children in a managed care system, it will benefit from a higher level of coordinated care and reduced costs. The Evaluation of the Demonstration Entitled Delaware Health Care Partnership for Children (6288) is examining this project.
HCFA is sponsoring a wide variety of waiver-based demonstrations that give States the opportunity to experiment with their Medicaid programs. As these demonstrations begin, HCFA will undertake examinations of their impact. The Evaluation of the Oregon Medicaid Reform Demonstration (6166), the Evaluation of the State Medicaid Reform Demonstrations (in Hawaii, Rhode Island, and Tennessee) (6289), and the Evaluation of the State Medicaid Reform Demonstrations (in Kentucky; Los Angeles County, California; Minnesota; New York; and Vermont) (6289.1) are three such projects. More evaluations will be initiated as more State demonstrations are begun.
In 1977, the Congress authorized a new type of provider, the Rural Health Clinic, which must be located in a rural health professional shortage area, medically underserved area, or Governor-designated shortage area and must make use of mid-level practitioners. Rural Health Clinics are reimbursed on a cost basis by the Medicare and Medicaid programs. Their numbers have been growing at a relatively rapid rate. The Evaluation of Rural Health Clinics (6299) examines the reasons for this growth, the impact on access to care for these rural populations, and the costs to the Federal government and States.
The Evaluation of the Program of All-Inclusive Care for the Elderly Demonstration (6309), known as PACE, examines replications of a unique model of managed care service delivery for very frail community-dwelling elderly, most of whom are dually eligible for Medicare and Medicaid and all of whom are assessed as being eligible for nursing home placement. The core services include adult day health care and multidisciplinary team care management through which all health and long-term care services are arranged. The evaluator is looking at the demonstration sites before and after assumption of financial risk to see if the replicates are cost effective relative the current Medicare and Medicaid services. They are also looking at the decision to enroll in order to understand how PACE enrollees differ from the eligible beneficiaries who choose not to enroll.
HEALTH RESOURCES AND SERVICES ADMINISTRATION
MISSION: To improve the Nation's health by assuring equitable access to comprehensive, quality health care for all.
Evaluation Program
The purposes of the Health Resources and Services Administration (HRSA) evaluation program are to enhance strategic and performance planning, strengthen budget and legislative development, and improve program management. The Agency's evaluation priorities are to develop and support performance measurement, assess program implementation, and conduct crosscutting policy analysis and research.
Performance measurement includes technical and training activities to strengthen the Agency's capacity to assess program performance, as well as studies to assess program outcomes. These activities, mandated by the Government Performance and Results Act (GPRA), will provide data for ongoing program monitoring and evaluation and for developing annual performance plans and reports. HRSA completed a study in September 1995 to establish a performance measurement baseline for all operating programs, with the intent of specifying valid and useful indicators and measures for all programs by September 1998. Over the past three years, HRSA has made significant progress in performance management, such as in strengthened linkages among strategic planning, evaluation, and budget decisionmaking.
Assessment of program implementation includes a broad range of efforts to assist the Agency in adjusting program development and management as the priorities of the Executive Branch and the Congress change. HRSA programs are entering into new arrangements for delivering services, providing health professions education, and encouraging the development of systems reform.
Crosscutting policy analysis and research includes efforts to build capacity and conduct studies to clarify the environmental shifts affecting HRSA programs and to improve HRSA's ability to document performance and impact through better measurement tools and data sources. The term, "crosscutting," refers to issues that bear on the programs of two or more HRSA Bureaus, such as trends in numbers and characteristics of the uninsured, and the viability of the health safety net. Development of this capacity at the Agency level began in FY 1997; work in FY 1998 and beyond will build on this foundation.
HRSA is striving to improve the quality of evaluation products. Potential approaches for this initiative include (1) an increase in the level and amount of technical assistance available to staff across the Agency; (2) expansion in the proportion of work done intramurally or with short-term help from individual experts; (3) encouragement of HRSA staff to participate in evaluation courses offered by local universities; and (4) establishment of an external evaluation advisory committee to consult on evaluation strategies in terms of content, approaches, and dissemination. Evaluation capacity is being strengthened first in the central evaluation unit, with the intent of making similar improvements in evaluation resources of the Bureaus in the future.
HRSA is continuing to place a major emphasis on dissemination of findings and information about the utilization of study results. A broader dissemination strategy, including an assessment of opportunities for using electronic media more effectively, will also be an important component of the quality improvement activities now under development.
Summary of Fiscal Year 1997 Evaluations
Studies related to the three key areas of HRSA's mission are described first, followed by crosscutting projects.
Eliminating Barriers to Care
One significant study, Exploratory Evaluation of Rural Applications of Telemedicine (5749), provides national baseline data on the status of telemedicine in health care for rural populations. The project included a mail survey of all non-Federal hospitals located outside metropolitan areas in the summer of 1995, plus a short survey of hospitals doing only teleradiology and a longer survey of hospitals applying telemedicine for purposes beyond radiology. The major findings were that (1) rural telemedicine is in the early stages of development but is rapidly expanding; (2) telemedicine networks are complex, with an average of four "spoke sites," two hubs, and four facilities that provided and received consults; and (3) Federal and State grants and support from hospitals are common sources of direct funding, but fewer than 25 percent of hub facilities had negotiated payment with insurance carriers and many had not begun negotiations. Study results have been used in forming the FY 1997 guidelines for the Rural Telemedicine Grant Program, in part to encourage the use of telemedicine in community settings. As requested, early findings were shared with congressional staff, who then specified priority areas in the authorizing legislation. The report was distributed to Federal agencies funding telemedicine, to State Offices of Rural Health, and through the World Wide Web. Finally, findings are informing the design of a common telemedicine evaluation instrument for HRSA grantees and for consideration by other Federal sponsors.
Another study relating to access is Institutional Factors That Influence the Likelihood of Living-Related Kidney Transplant Operations--Phase II (4917.1). The purpose of this project was to determine (1) the willingness to conduct, and frequency with which the transplant community does conduct, kidney transplants from living-related donors; (2) the medical and non-medical institutional factors influencing the likelihood that transplant professionals will suggest and conduct such transplants; and (3) the circumstances and conditions under which a transplant team that does not encourage these transplants would refer a patient to another transplant center for evaluation. One finding was that over half of kidney transplant centers lacked patient education materials about live kidney donors. A summary report has been sent to all kidney transplant centers. In addition, discussions have been held with groups such as the National Kidney Foundation about ways to apply the findings, such as through development of educational materials for national distribution.
In the health professions area, a major statistical study produced a technical and an analytic report; the title of the latter is The Registered Nurse Population, March 1996: Findings from the National Sample of Registered Nurses (6367). A contractor prepared the sampling design and conducted the survey, which is the most comprehensive source of statistics on registered nurses (RNs) with current licenses to practice in the United States. The survey resulted in a data base of information on 29,950 nurses, which was then analyzed by HRSA staff. The report provides information on the number of nurses; their educational background and specialty areas; their employment status, including type of employment setting, position level and salary; their geographic distribution; and their personal characteristics, including gender, race and ethnicity, age, and family status. HRSA is using the study products to guide nursing education programs, to prepare biennial reports to Congress on characteristics of the RN population, and for projecting the supply and demand for nurses at a national level. The data base and analytic report have been shared with the nursing community, which looks to the Department to maintain and update this data base.
Eliminating Disparities in Health Status
Another major quantitative effort was the Community Health Center User and Visit Study: Survey Methodology Report (5737). This project gathered, for the first time, nationally representative data about Community Health Center (CHC) users and the services provided to them. The study had two components: (1) personal interviews with 1,932 patients selected randomly from a representative sample of 48 CHCs, and (2) data extraction from 2,878 medical records concerning randomly selected encounters during 1994 at the same centers. Data on users were gathered in a manner to permit comparison with results from the National Health Interview Survey. Similarly, information from the visit (encounter) survey will be compared with results from the National Hospital Ambulatory Medical Care Survey. The CHC data are being analyzed to answer questions on such topics as the race and ethnicity of users, health-risk behaviors presented, most prevalent diagnoses, services utilized, and the extent of monitoring of compliance with treatment prescribed for chronic conditions. In addition, the data are being reviewed to form a profile of CHC visits in comparison with studies of the general population in terms of conditions, diagnoses, type of provider seen, and disposition. Analysis of data from the initial surveys continues intramurally, as well as through a current study noted below.
Assuring Quality of Care
A major concern in recent years has been the spread of tuberculosis (TB) among persons with HIV/AIDS. Examining Relationships Between HIV/AIDS and Tuberculosis Programs in Title I Eligible Metropolitan Areas (6359) was designed (1) to determine the extent to which Ryan White Comprehensive AIDS Resources Emergency (CARE) Act grantees were giving guidance to providers on TB screening and prophylactic and therapeutic regimes, and (2) to describe models of collaboration between HIV and TB control programs within Eligible Metropolitan Areas (EMAs). The study found that TB services varied across EMAs and across care settings within them. As of 1995/1996, Federal and State funding policies and staffing patterns hampered collaboration between HIV and TB programs. However, the study identified mechanisms to promote TB-HIV services collaboration, fostered in some places by outbreaks of TB and multidrug-resistant TB. HRSA has used the results to inform Title I program guidance and to develop the training and technical assistance plan for FY 1998. The report has been sent to Title I grantees and to the Centers for Disease Control and Prevention (CDC) TB control grantees. Results were also presented at the 1995 International AIDS conference in Vancouver, Canada, and in September 1997 at the CDC conference entitled, "Issues for HIV and TB Co-Infected People."
The increasing adoption of managed care by public insurance programs, such as Medicaid, has raised concerns about whether capitated reimbursement affects quality of care provided to people with chronic illnesses such as HIV/AIDS, or with other special health care needs. One way to address these concerns is to create risk-adjusted capitation rates to provide equitable reimbursement for the care of patients with costly illnesses, an approach several States had proposed or implemented by 1997. In May of last year, HRSA convened an invited conference to examine issues surrounding risk adjustment for HIV/AIDS, in collaboration with other HHS organizations (the Office of the Assistant Secretary of Planning and Evaluation and the Health Care Financing Administration), the Henry J. Kaiser Family Foundation, and the National Academy of State Health Policy. The HIV Capitation Risk Adjustment Conference Report (6894) presents perspectives of participants (representing persons living with HIV/AIDS, providers, managed care organizations, and States); methodological issues; and research needs. Conference findings have been used in technical assistance to States on coordinating CARE Act programs with managed care. HRSA has distributed the report to all CARE Act grantees and State Medicaid directors.
Crosscutting Issues
One recently completed study concerned managed care and its relevance for vulnerable populations. The Availability of Primary Care Services Under Medicaid Managed Care: How 14 Health Plans Provide Access and the Experience of 23 Safety Net Providers and Their Communities (6366) examined strategies used by health maintenance organizations (HMOs) to generate access to health care services for Medicaid enrollees, who often live in inner-city and rural, underserved areas. The project also explored the implications of these strategies for overall access in underserved areas and for existing care providers. Third, the study considered how these strategies affect traditional Medicaid providers in the community. The study concluded that Medicaid managed care has had an overall positive effect on the availability of primary care to low-income residents in the 14 communities, and that traditional safety net providers have not had to reduce services. But the future viability and range of services of public sector-supported providers might be endangered in a more competitive health system. The findings will be useful to policymakers concerned with the provision of care for vulnerable populations, providers in underserved areas, and those attempting to assure an adequate supply of health personnel. The study has been noted in the professional press and will be referenced in a journal article now under preparation.
Evaluations in Progress
Projects in progress are presented first according to HRSA's three mission goals; crosscutting projects follow.
Eliminating Barriers to Care
Begun in September 1994, the Evaluation of the Effectiveness of Community Health Centers: Implementation Phase (5736) is describing CHC delivery model(s), measuring the impacts of the model(s), and identifying factors that contribute to the observed impacts. The study is examining the experience of CHC users and non-users in both fee-for-service and managed care arrangements though a sample of 50 CHCs in 10 States. The project has three components: (1) an analysis of utilization and expenditures, based on HCFA State Medicaid Research Files for eight States; (2) an analysis of outcomes and content of care for four conditions (otitis media, asthma, hypertension, and diabetes) based on CHC medical records and the literature on medical treatment norms; and (3) a description of CHC delivery systems using primary data collection (a mail survey) and secondary sources. The analysis of utilization and expenditures is addressing casemix and differences between exclusive and regular CHC users.
Effectiveness of the National Health Service Corps (6357) is examining the various mechanisms for training and recruiting providers, placing them in underserved areas, and retaining them in primary care and service to vulnerable populations. The project includes mail surveys of three random samples: (1) National Health Services Corps (NHSC) alumni clinicians; (2) current NHSC clinicians; and (3) administrators of sites staffed with NHSC clinicians. (Some of the sites are CHCs and others are free-standing.) The study also involves review of secondary data on individual NHSC participants, sites that are eligible for NHSC assignees, characteristics of designated shortage areas, and site vacancies available for assignment of obligated scholarship providers.
In another critical policy area, HRSA is conducting a study entitled Enrollment, Utilization and Cost-Forecasting Models for Use in State AIDS Drug Assistance Programs (6806). This study is exploring the effect of the introduction of protease inhibitors on medical service utilization for use in State AIDS Drug Assistance Programs (ADAP). During 1996, five protease inhibitors became available; this class of drugs has been shown to have superior outcomes in treating HIV. However, although pharmaceutical interventions represent a sizeable portion of the cost of care for HIV-infected patients, the efficacy of these drugs has not been shown outside of controlled clinical trials. This study is using client-specific data from a geographically diverse population to identify personal factors that may modify the link between the introduction of protease inhibitors and services use.
Eliminating Disparities in Health Status
The National Evaluation of the Healthy Start Program (5610) continues as a longitudinal study of the development, implementation, and outcomes of the comprehensive maternal and child health delivery systems at the original 15 Healthy Start sites, which were funded initially as 5-year demonstration projects. A sixth year was added in FY 1996. This study includes an assessment of changes in the health status of pregnant women and infants across the sites. The study was designed to answer four questions: Did the Healthy Start initiative succeed? If so, why? If not, why not? What would be required for a similar intervention to succeed in other settings? The study includes both process and outcome components. Findings from both will contribute to analyses bearing on replication of successful approaches for reducing infant mortality, and on improving program effectiveness in general. Study results will be used to guide the development and implementation of Healthy Start projects funded subsequently, and will be widely shared with the public health community for application in settings funded under other auspices.
Analysis of CHC User Survey: Selected Conditions (6805) is assessing portions of the survey data from the Community Health Center User and Visit Study (see PIC ID No. 5737). The purpose is to describe the self-reported process of care and outcome measures for CHC users with certain conditions: hypertension, high serum cholesterol, diabetes, asthma, and pregnancy. The analysis includes the demographic characteristics, income and educational levels, health status, and activity limitations of these patients as compared with (1) CHC users without the selected conditions, and (2) the general population. Data on the general U.S. population will be from the National Health Interview Survey and the National Hospital Ambulatory Medical Care Survey. Findings will be used, in part, to determine whether care provided at CHCs is appropriate and conforms to the specifications of Healthy People 2000.
Assuring Quality of Care
One of HRSA's critical concerns is cultural competence of care. A study begun in FY 1997, Development and Refinement of Tools for Monitoring Cultural Competence in Managed Care (6817), is analyzing successful experiences of managed care organizations in California in meeting the needs of culturally and linguistically diverse Medicaid populations. A second purpose of this study is to plan for the validation of tools for measuring cultural competence among individual providers. The new study builds on Tools for Monitoring Cultural Competence in Health Care (6352), which was completed in FY 1996.
Direct Support for Conduct of Evaluation Studies at Selected CARE Act Pilot Sites (6807) is providing assistance to six Title I and Title II grantees in analyzing data sets and developing models to assure the effectiveness of their primary care and supportive services. These local studies constitute a first step in developing future "sentinel" studies to create a picture of the national impact of these CARE Act programs. These initial projects include three types: (1) examining the association of CARE Act-funded services with clinical and other outcomes for patients with HIV; (2) developing models to assess the effects of managed care on fund allocations and service delivery; and (3) developing and pilot testing models to analyze the effectiveness of Title II care consortia in evaluating needs, setting priorities, and delivering services.
In addition to studies relating mainly to one mission area, HRSA has ongoing projects focused on the capacity of two HRSA Bureaus to measure performance across the organization. The first of these projects, which concerns the Maternal and Child Health Bureau (MCHB), is the MCHB GPRA Maturity Project (6811). This study, building on a series of program-specific efforts, is providing coordination of performance measurement activities to produce coherent performance data for the Bureau as a whole. The project also provides technical assistance in working with grantees and others to promote acceptance and implementation of performance measures throughout the maternal and child health community, and training for MCHB program staff in the ongoing use of performance measures. The intent is to complete the development of a fully operational performance management system.
A Bureau of Health Professions (BHPr) project also illustrates the use of evaluation to develop performance measures that encompass all Bureau programs. Development of BHPr's Comprehensive Performance Monitoring System (6800) is helping the Bureau implement its monitoring system. In preparation for meeting GPRA requirements, BHPr has developed goals, outcomes, and indicators with respect to workforce quality, supply, diversity, and distribution. This project, begun in FY 1997, is using a data collection instrument that was tested through an earlier pilot study to begin nationwide implementation of the data collection. The study includes technical assistance to applicants and grantees in gathering the required data and completing the data collection instrument. This project will involve grantees for the first time in submitting data electronically using the new instrument. The study includes development of software, training in its use for Bureau staff and grantees, and a help desk to provide ongoing technical assistance during implementation of the reporting system. The study will also include an evaluation of the data collection instrument and electronic implementation, the training and technical assistance, and the quality of data collected. Results of the project will be used to determine whether changes are needed in the current approach for collecting and analyzing program data, and will be shared with the public through an article in a professional journal.
Crosscutting Issues
Through Managed Care and the Safety Net Providers (6815), HRSA is examining the impact of Medicaid managed care on the fiscal and operational viability of safety net providers in primarily ambulatory and primary care settings. Among the types of providers to be included are Community and Migrant Health Centers, Maternal and Child Health programs, Ryan White CARE Act grantees, and rural health centers. A key topic being explored is the effect of a more competitive, market-oriented health care system on capacity to serve vulnerable populations. Policy and program interventions to deal with possible shortfalls will be identified.
A collaborative effort with NCHS, Person-Place Approximations and Utilization vs. Health Status Analysis of the National Health Interview Survey (5614) is addressing three questions: (1) What common characteristics are shared by communities with problems in access to health care? (2) Can these characteristics/
problems be measured directly, or through good proxies from data that can be easily obtained? (3) What is the appropriate aggregation of geographical units to use in describing small areas? The intent is to gain a better understanding of the ways in which the characteristics of particular areas influence the behavior of respondents to the National Health Interview Survey in order to use the knowledge for predicting access problems in small areas for the entire U.S. population.
Finally, HRSA Crosscutting Community Case Studies (6814) is examining the nature, extent, and impact of the interaction of HRSA programs in selected "sentinel" communities in order to determine whether inflows of HRSA resources at the local level combine in a mutually reinforcing way, operate independently, or work at cross-purposes. This project builds on two ongoing studies sponsored by other organizations. The Robert Wood Johnson Foundation is developing baseline information for tracking changes in the health care market in 12 metropolitan statistical areas. The other project, sponsored by the HHS Office of Disease Prevention and Health Promotion, is addressing how market-driven changes are affecting the level of effort of public health departments in providing ten essential public health services. Sites for the HRSA study are three urban communities (Boston, Cleveland, and Phoenix) in the Johnson Foundation study. This project is HRSA's first attempt to test a methodology for measuring the interactions and collective impacts of the Agency's programs on communities, which can then to used for conducting prospective and long-term evaluations.
MISSION: The mission of the Indian Health Service, in partnership with American Indian and Alaska Native people, is to raise their physical, mental, social, and spiritual health to the highest level.
Evaluation Program
The goal of the Indian Health Service (IHS) is to assure that comprehensive, culturally acceptable, personal and public health services are available and accessible to American Indian and Alaska Native people. The importance of evaluation in supporting this goal has increased significantly in recent years and includes American Indians and Alaska Natives as the primary stakeholders in defining the purpose, design, and execution of evaluations. The stakeholders use the end product of the evaluations, and are typically the population or groups most likely to be affected by the evaluation findings. The IHS has formally adopted the principles of a responsive evaluation practice to address the needs and concerns of American Indians and Alaska Natives.
The evaluation program of the IHS is managed by the Office of Public Health, Staff Office of Planning, Evaluation, and Research (OPH-SOPER), which provides national leadership and consultation for IHS and Area Offices on strategic and tactical planning, program evaluation and assessment, public health and medical services, research agendas, and special public health initiatives for the Agency. The OPH-SOPER serves as the principal advisory office on issues of national health policy and coordinates the following four evaluation functions.
The evaluation needs of the IHS service components are coordinated by OPH-SOPER using two major types of short-term studies: policy assessments and program evaluation studies. The IHS policy assessments contribute to decisionmaking about budget, legislation, and program modifications and include background information to support the Agency's initiatives. Evaluation studies are focused at the program level, or Area Offices, and focus on specific program needs.
Each year OPH-SOPER selects high-priority health care and management studies for funding through the submission of proposals to headquarters and Area Offices. These proposals are reviewed and rated by a panel of subject-matter experts, evaluation experts, and IHS staff for concurrence with IHS strategic goals, objectives, and priority areas. The proposals are then prioritized and forwarded to the IHS Director, who reviews the projects that are recommended for funding and determines the respective funding levels.
Summary of Fiscal Year 1997 Evaluations
During FY 1997, IHS completed two major evaluation projects.
The Evaluation of the Impact of the Alaska Tribal Health Compact on Programs and Services in the Anchorage Service Unit, Alaska Area (6407) project is the first research to systematically evaluate the perceptions of a wide range of beneficiaries, officials, and employees of the IHS and the tribal system regarding the Alaska Tribal Health Compact, Title III Self-Governance Demonstration Project. There are almost 200 federally recognized tribes in a single Areawide Compact, organized via 13 separate consigners, each of which has its own Annual Funding Agreement. Seven of these consigners are in the Anchorage Service Unit, which is one of nine Service Units in the Alaska Area. The evaluation concludes that more village representatives have seen either no change or have seen a negative change in in-village services than those who have seen a positive change. The perceptions of the officials and employees of the Alaska Native Medical Center and the Alaska Area Native Health Service have revealed problems in staff morale and shifts in workload resulting from Compact consigners transferring patients to the Alaska Native Medical Center who were previously treated outside the Anchorage Service Unit. These changes are considered as having a negative impact on the Anchorage Service Unit. The "microcosmic" view from the Ahtna Region indicates that there is a distinct difference in the perception of those who are participating in the Compact and those who are not. The analysis of the village respondent data, comparing the information that came from the Title III villages with the information from the Title I villages, further solidified the conclusion that individuals in villages that are not served by a Compact consigner hold a consistently more negative view than those who live in villages that are served by a Compact consigner.
The Prior Trauma Care of Intoxicated Patients as a Predictor of Subsequently Fatal Injury (6451) study was undertaken to identify intervention opportunities that are the result of nonfatal, alcohol-related injury contacts in emergency departments and clinics that could, over time, decrease alcohol-related injury deaths in the IHS area in Billings, Montana. The concept was to determine the frequency of injury treatment recidivism, both alcohol- and non-alcohol related, and further to determine if patterns can be established to suggest that injury death is predictable based on those previous nonfatal contacts. The findings indicate a high degree of recidivism and a strong correlation between nonfatal incidents and eventual injury death. This study further reveals that there is a high prevalence of alcohol in nonfatal injury-related incidents among this cohort. There appears to be a strong relationship between repeated episodes of nonfatal injury with alcohol as a contributing factor and subsequently fatal events among this cohort. These nonfatal episodes should serve as warning signals and opportunities for alcohol intervention strategies to take place.
Evaluations in Progress
The IHS currently has several major evaluations under way in the following general areas: the IHS Resource Requirement Methodology (RRM), aftercare services provided by IHS regional treatment centers, and effectiveness of alcohol and substance abuse services for American Indian and Alaska Native women. The IHS also has a number of smaller evaluations in progress at several area offices.
The Elderly Wellness (6452) study focuses on collecting data to determine if providing health care and health promotion services specially designed for the elderly will increase their use of these services and ultimately improve their health status. Comprehensive care remains an unmet need for the elderly. Recent reviews show that less that 50 percent of the elderly population residing in the Fort Peck Service Unit (Billings, Montana, Area Office) has had a health care visit of any kind in the last two fiscal years.
The Cross Sectional Evaluation of the Prevalence of Childhood Obesity Among American Indians in the Aberdeen Area (6859) studies the trend in the prevalence of diabetes among American Indians in the Aberdeen, South Dakota area over a 10-year period. Findings suggest that not only are diabetes prevalence rates increasing, but the gap is widening between Aberdeen, South Dakota area rates and national IHS rates. This study will establish baseline rates of childhood obesity and will facilitate early intervention efforts among 100 schools attended by Indian children. Obesity is the most important risk factor for diabetes in American Indian communities. The study results will be used by service unit directors and their staff (diabetes coordinators, public health nurses, and nutritionists) to assess their progress toward the Healthy People 2000 objectives and the Aberdeen Area Clinical Objectives to determine childhood obesity prevalence and plan primary prevention programs.
The Evaluating the Effects of Medical Nutrition Therapy on Patient Outcomes Among Native Americans with Newly Diagnosed Type II Diabetes Mellitus (6746) study will evaluate the effectiveness of medical nutrition therapy (MNT) on clinical/behavioral outcomes of American Indians/Alaska Natives with newly diagnosed Type II diabetes. The evaluation will include the creation of a uniform methodology for collecting nutrition-oriented outcome data that is consistent among participating tribes, IHS service units, and urban programs. The outcome data will be analyzed to determine differences in the type and amount of MNT provided with regard to patient outcomes. They will also be linked with current clinical data from the IHS Diabetes Program that identifies continuous quality and performance improvement and measures. The analysis will also describe individual cases in which MNT has contributed to improved patients perceived quality of life.
The purpose of the Evaluation of IHS Capacity for Epidemiologic Surveillance (6745) is to determine the epidemiologic surveillance capacity of the Indian/Tribal/Urban (I/T/U) health system and to make recommendations for improving the capacity to meet the needs of the IHS and its stakeholders. The IHS health care environment is undergoing unprecedented change; reorganizing, consolidating, and downsizing are part of the overall restructuring of services to its customers. Limited resources will be available to conduct epidemiologic surveillance, and the I/T/U health system will become more reliant on local capacity. The evaluation will be designed to collect data from a representative sample of IHS direct service providers, P.L. 93-638 contracts, self-governance compacts, and urban programs to determine the availability of epidemiological useful data and the capacity to perform disease surveillance and epidemiologic analyses of data at the local level.
The objective of the Evaluation of the Effectiveness of the Indian Health Service Alcohol and Substance Abuse Treatment Programs, Phase I (6744) is to evaluate the effectiveness of aftercare and continuing care services provided to youth and adult patients who have received treatment for alcohol and substance abuse at a primary treatment facility, a regional treatment center, or other IHS-funded program. The study will examine the availability, accessibility, acceptability, and cost of community-based followup services, as well as the potential for coordinating and integrating these services with the other aspects of alcohol and substance abuse treatment, such as detoxification, residential treatment, and halfway houses. The purpose of Phase I is to develop a methodology to assess the effectiveness of the IHS-sponsored aftercare/continuing care program to meet the needs of tribal and urban programs. Phase II will include a national assessment to measure the outcomes of aftercare and continuing care services attained by patients/clients of alcohol/substance abuse treatment programs in a representative sample of Indian communities that have IHS-sponsored alcohol and substance abuse programs. The study will include recommendations and an implementation plan to improve aftercare/continuing care services in Indian communities.
In the early 1970s, the IHS formulated the Resource Requirements Methodology (RRM) as a management tool to provide a comprehensive, systematic, and uniform process for estimating the level of resource requirements necessary to provide adequate health care to IHS customers and to assist in the allocation of unearmarked resources. To reaffirm the purpose of the RRM, a 1995 study determined the validity and accuracy of the present methodology for use in today's health care environment. The study findings support the need to update the current methodology to meet the future program demands of the IHS. The Resource Requirement Methodology Update (6450) will include a complete revision and documentation of the RRM and will consist of the following phases: (1) Update Staffing Criteria and Modules, (2) Formulate Needs Assessment Cost Model, and (3) Needs Assessment Model Training.
The Evaluation of Indian Health Service-Supported Alcohol and Substance Abuse Treatment Program for American Indian/Alaska Native Women (6201) is part of a four-phase evaluation of alcohol and substance abuse services for adult American Indian and Alaska Native women in alcohol and other drug treatment centers that are funded by the IHS. Phases I and II of this study were completed; a report was prepared in December 1995. The primary objectives of this project are to assess and measure the treatment services provided; to describe the organization and provision of substance abuse treatment and aftercare services available for adult American Indian and Alaska Native women; and to identify common strengths, problems, and recommendations for improvement. Phase I of this effort focused on designing the study, and Phase II collected descriptive data on the women who use IHS-sponsored substance abuse treatment centers. Phase III will be a prospective client outcomes study, and Phase IV will consist of an evaluation of the efficacy of IHS-funded treatment centers.
Development of a Health Services Research Agenda for American Indian and Alaska Native Populations (6153). The IHS and the Agency for Health Care Policy and Research co-sponsored a health services research conference as a first step in a long-term agenda-setting process to identify the most important health services research issues facing American Indian and Alaska Native communities and their health care systems over the next 5 to 10 years. The health services research agenda is intended to promote collaboration between American Indians and Alaska Native organizations, tribal and urban health systems, medical communities, foundations, and government agencies to increase communications and produce research information on health program services for American Indian and Alaska Native patients. The health services research agenda is also intended to provide a forum for discussing health care reform changes that are creating new directions in the Indian health care system.
Malignant neoplasm survival rates for well-defined American Indian populations are scarce. Although a national cancer surveillance system is in place, information relevant to Native Americans in the Northern Plains is not available. The Evaluation of Available Cancer Surveillance Systems for Montana Native Americans and Correlates of Decreased Cancer Survival Among Native Americans in Montana (5944) will examine the utility of the IHS Resource and Patient Management System and the Montana Central Tumor Registry as a cancer surveillance system for the Native Americans in Montana. The results of this evaluation may determine the current level of cancer detection and treatment for Montana Native Americans compared to non-Native Americans in the same region. The development of health promotion and disease prevention activities requires an efficient surveillance system and knowledge of the current deficiencies and strengths of the current system. A better understanding of the sites of late stage cancer at diagnosis will help providers direct education and early detection efforts to Montana Native Americans.
The Study of the Interaction of Persons Treated for Alcoholism with the Health Care Delivery System (6742) will evaluate the results of alcohol treatment for American Indians who abuse alcohol, focusing on the overall health care delivery system. The evaluation design and methodology will be developed and applied to the population served by the Crow Agency Service Unit. Data retrieval, audit instruments, and analytical strategies will be piloted for two years.
MISSION: To sponsor and conduct medical research that leads to better health for all Americans.
Evaluation Program
The National Institutes of Health (NIH) generates scientific knowledge that leads to improved health. This is done by conducting medical research in its intramural laboratories and by supporting research in universities, medical and health professional schools, and other health research organizations. NIH fosters the widespread dissemination of the results of medical research, facilitates the training of research investigators, and ensures the viability of the research infrastructure. The NIH Evaluation Program is an integral part of how NIH sponsors and conducts medical research.
The complexities and challenges of evaluating and assessing fundamental science have become more widely recognized as science agencies have begun implementation of the Government Performance and Results Act (GPRA). The GPRA stresses the use of evaluation to develop measures for reporting on program results. However, measures for evaluating the results of fundamental science have limitations. These are discussed in the March 1997 General Accounting Office report, Measuring Performance: Strengths and Limitations of Research Indicators. The report compares private versus public research and discusses the strengths and limitations of indicators used to measure research. The NIH evaluation program is addressing these challenges as part of its strategy for implementing the GPRA.
Philosophy and Priorities
The NIH Evaluation Program provides information to assist the NIH Director and Institute, Center, and Division (ICD) Directors in determining whether NIH goals and objectives are being achieved and to help guide policy development and program direction. Evaluations are planned and conducted from two sources of funds: 1-percent set-aside evaluation funds used to fund trans-NIH projects, and ICD program funds used for program evaluations for use by various committees, working groups, task forces, workshops, conferences, and symposia to assist the ICD in program management and development. This approach ensures that planning and priority setting specific to the mission of each ICD are fully developed and implemented and that there is central leadership for developing crosscutting initiatives and promoting collaboration among the ICDs.
NIH's major evaluation priority areas fall within three broad program areas: basic research, research training and career development, and facilities maintenance. These broad program areas form the basis for the strategies and activities that NIH undertakes to achieve the goals identified in the GPRA planning process.
Policies and Operations
A distinguishing feature of the NIH Evaluation Program is the utilization of a variety of evaluation strategies that include the use of national advisory councils, boards of scientific counselors, consensus development conferences, and ad hoc committees that help to chart scientific directions and select the most promising research to support.
A two-tier system is used to review project requests that will use 1-percent evaluation set-aside funding. The first tier involves a review and recommendations by the NIH Technical Merit Review Committee (TMRC) on the technical aspects of project proposals and whether a project fits within HHS guidelines for use of the set-aside fund. The second tier involves the NIH Evaluation Policy Oversight Committee (EPOC), which considers TMRC recommendations, conducts policy level reviews, and makes final funding recommendations to the NIH Director or his designee.
Summary of Fiscal Year 1997 Evaluations
Completed evaluations addressed a cross-section of the NIH research program, training, and career development. Three of the NIH evaluations completed are highlighted below.
A blue-ribbon panel of experts, the National Institute of Mental Health (NIMH) Intramural Research Program Planning Committee (IRPPC), was charged with evaluating and, as necessary, redefining the role of the Intramural Research Program (IRP) in fulfilling the mission of the NIMH. The IRPPC report, Finding the Balance: Report of the National Institute of Mental Health Intramural Research Program (IRP) Planning Committee (6697), provided the NIMH Director and Scientific Director with tools for fundamental renewal of the IRP. For example, the IRPPC recommended the consolidation of the Neuroscience Center of St. Elizabeth's Hospital with the NIMH IRP program at the Bethesda campus. Such a consolidation will provide better access to the special facilities the Bethesda campus offers, more opportunities for collaboration, and better access to patients.
Insuring a strong scientific infrastructure--
with a suitable talent base of highly qualified, highly trained, and diverse investigators capable of yielding the scientific discoveries of the future--and excellent research facilities is essential for medical progress. To increase the number of underrepresented minorities entering careers in biomedical research, the National Institute of General Medical Sciences (NIGMS) established two "bridges" programs to encourage minority students to go beyond the associate degree to obtain the bachelor's degree, or beyond the masters degree to obtain a doctorate. NIGMS established an Electronic Student Tracking and Reporting system to capture the information needed to assess the educational outcomes of students participating in the Bridges Program. The results from this evaluation, Evaluation of the Bridges to the Future Program (6093), indicated relatively high rates of transfer from 2-year to 4-year institutions, or from M.S. to Ph.D. programs, when compared to national norms. As a result, NIGMS is continuing support of this program.
To examine how research results translate into practice, A Study to Determine the Changes of Physician Prescribing of Antimicrobial Agents, Omeprazole, and H2-Receptor Antagonists for Peptic Ulcer Disease Two Years Following the NIH Consensus Conference on Helicobacter Pylori (6284) was designed to evaluate the effect of the consensus conference in prescribing patterns of physicians for the treatment of peptic ulcer disease (PUD), focusing on prescriptions for antimicrobial and antisecretory agents and H2-receptor antagonists. In particular, the study focused on whether the recommendations of a February 1994 NIH Consensus Development Conference advocating the use of antimicrobial agents for PUD patients with H. pylori infection were coincident with a change in physician prescribing patterns. The study revealed that two years after the highly publicized conference on the eradication of H. pylori, antimicrobial agents were not widely prescribed for the treatment of PUD within the Pennsylvania Medicaid population. The need to eradicate H. pylori to reduce and, ultimately, eliminate the recurrence of PUD in association with the findings of this research suggest that physicians in this specific population do not appear to be using national recommendations based on state-of-the-art information and clinical studies pertaining to PUD, as presented by the NIH. Restricting the study to Medicare patients limits its generalizability and could be related to issues of equity and quality of care for a disadvantaged population. If findings indicate significant cost savings associated with the use of the NIH recommended drug regimens in the treatment of PUD, this approach could be applicable to other populations.
Other NIH evaluations were completed on the following topics: the status of biomedical research facilities (3562.2); National Research Service Award Research Training Programs (5571); the National Heart, Lung and Blood Institute (NHLBI) article citation and patent references study (5577); the Rare Disease Clinical Research Database (6043); laboratory animal use, facilities, and resources (6048); promoting safe and effective genetic testing (6090); user satisfaction of the Physician Data Query data base (6092); research instrumentation needs (6121); toxicology and environmental health information resources (6534); evaluation of the international cooperative bio-diversity groups (6641); and the Federal Interagency Forum on Aging-Related Statistics (6652).
Evaluations in Progress
NIH has over two dozen evaluation projects in progress, as listed in Appendix B. Several of the major evaluations, soon to be completed, are summarized below.
An ongoing study that will provide information in the research program area is the Surgeon General's Report on Oral Health (6861). This study will provide information on how the public health elements of education, daily preventive self-care, and collaboration between individuals and their care providers can help produce a high level of health in a particular area. In spite of tremendous oral health progress throughout the last five decades, data demonstrate that results are distributed unevenly across populations. There still remain populations and communities with devastating levels of oral, dental, and craniofacial diseases. This report will identify specific steps that individuals and communities can take to improve their oral health status.
Another study under way in the research program area is a project titled, Evaluation of Simplification of the Human Resources Management System at the National Institutes of Health (6862). The Secretary has designated NIH as a pilot project for a period of 5 years to simplify personnel management and personnel administration policies and procedures. The main platform for the evaluation is the delegation of human resources management requirements and responsibilities; the long-range thrust is to strengthen NIH management and integrate human resources management into strategic planning. The evaluation seeks to determine the linkage between the impact of the human resources management delegations and the achievement of major NIH goals, such as advancing research to improve health and prevent disease.
An ongoing study that will provide information in the research training and career development area is, Study of National Needs for Biomedical and Behavioral Research Personnel (6860). This congressionally-mandated study determines the overall national need for research personnel in the biomedical, behavioral, and clinical sciences; and assesses the efforts of the National Research Service Awards (NRSA) programs in ensuring an adequate supply of qualified investigators. Earlier studies have included recommendations concerning both the number of NRSA research training positions needed in the future and the importance of systematic program monitoring and assessment. These recommendations have been used by NIH in formulating program priorities and budget requests, and by the Congress during the appropriations process. For example, it is not uncommon for congressional appropriations reports to cite the number of positions recommended by this study and to appropriate funds accordingly.
In the facilities area, NIH collaborates with the National Science Foundation (NSF) to assess the condition of existing facilities and identify needs for new and refurbished research facilities nationwide. Since its inception in 1988, the biennial Survey of Scientific and Engineering Research Facilities at Colleges and Universities (6863), prepared jointly by NIH and NSF, has provided considerable data on the amount, condition, and capital project activity by the Nation's research-performing institutions. For example, the 1996 Status of Biomedical Facilities reports that 47 percent of all biomedical research-performing institutions classified the amount of biological science research space as inadequate, and 51 percent indicated that they had an inadequate amount of medical science research space.
OFFICE OF THE ASSISTANT SECRETARY FOR PLANNING AND EVALUATION
MISSION: To provide analytical support and advice to the Secretary on policy development and assist the Secretary with the development and coordination of departmentwide program planning and evaluation activities.
Evaluation Program
Through the departmental evaluation planning process, ASPE has the capacity to identify crosscutting program or policy issues of particular concern to the Secretary and specific program and policy areas not covered by the HHS Agency evaluation plans. In these instances, ASPE initiates evaluations or collaborates with the agencies to conduct evaluations or policy assessments. For example, in recent years, ASPE has initiated projects to develop cost estimates for health financing issues in general, and specifically for the Medicare and Medicaid programs; the effects of managed care expansion on public health infrastructure; welfare-to-work approaches; long-term care alternatives; and studies evaluating alternative services for children at risk of harm from drug abuse, crime, child abuse, and other pathologies.
Another continuing evaluation objective of ASPE is to support and promote the development and improvement of data bases that HHS agencies and ASPE use to evaluate health care programs and health trends. For example, ASPE has been the major initiator in collaboration with the National Center for Health Statistics at the Centers for Disease Control and Prevention of the first comprehensive survey of people with disabilities in the United States. The first component of these new data was completed in FY 1996, and national prevalence data on disability are now available. The ASPE co-chairs and provides support to the HHS Data Council, which is charged with integrating key national surveys, such as linking health status indicators with indicators of well-being. The Department needs more comprehensive data sources to assess anticipated transformations in health and human services.
Finally, ASPE uses evaluation funds to promote effective use of evaluation-generated information in program management and policymaking. The latter is accomplished through the dissemination of evaluation findings and other activities, such as providing technical assistance to agencies in the development of performance measures.
Summary of Fiscal Year 1997 Evaluations
During FY 1997, ASPE completed a wide range of studies and reports on many issues, providing information useful to the Secretary and the HHS divisions for program planning and budget and legislative development.
Health Policy
Market Barriers to the Development of Pharmacotherapies for the Treatment of Cocaine Abuse and Addiction Final Report (6694) presents an analysis of the market barriers to development of pharmacotherapies for substance abuse and addiction, particularly for abuse of and addiction to cocaine. The study examined the development of pharmacotherapies for cocaine abusers in light of the size of the potential market, market penetration estimates, and the basic relationships between price, market size, and revenues. The report outlines several critical market barriers that must be taken into account. The substance abuse treatment market relies heavily on State and Federal reimbursement. Most substance abuse treatment services are subsumed under the mental health benefits of entitlement programs, and drug companies are reluctant to rely upon this kind of reimbursement in an age of shrinking budgets for mental health services.
Public Health Laboratories and Health Systems Change (6458) explored how the relationships among public health laboratories, managed care organizations, and other stakeholders in the laboratory services market have recently changed and the implications of these changes for Federal and State governments. The dynamic health care environment is posing many new challenges for public health laboratories. Changes in both the public and private sectors have made it increasingly difficult for public health laboratories to fulfill their mission. In the private sector, managed care and independent laboratories are expanding; hospital-based laboratories are consolidating; and private sector laboratories are experiencing rapid technological change, both in a clinical sense and in information processing. In the public sector, public health laboratories are experiencing a redefinition of the public health safety net, with greater reliance on managed care to address public health needs and shrinking budgets as State coffers have decreased.
Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health (6177). ASPE formed the Panel on Performance Measures and Data for Public Health Performance Partnership Grants (PPGs) to examine the state-of-the-art in performance measurement for public health and to recommend measures that could be used to monitor the PPG agreements to be negotiated between each State and the Federal Government. Performance measures were identified in ten public health areas: chronic diseases, sexually transmitted diseases (STDs), human immunodeficiency virus (HIV) infection, tuberculosis (TB), mental health, immunization, substance abuse, sexual assault, disabilities, and emergency medical services. More than 3,200 measures were proposed to the panel through various outreach efforts. The panel used the following four guidelines to assess the measures: (1) the measure should be specific and result oriented; (2) the measure should be meaningful and understandable; (3) data should be adequate to support the measure; and (4) the measure should be valid, reliable, and responsive. The panel found that many Federal efforts to collect health-related data provide national rates, but many do not collect data that provide State-level rates, and that much State-level data may not be comparable.
Information Needs Associated with the Changing Organization and Delivery of Health Care: Summary of Perceptions, Activities, Key Gaps, and Priorities (6388) reports on unmet needs for supply-side information (information provided by health care suppliers and insurers) on the health system, focusing on helping HHS identify how information needs associated with a changing health care system are perceived by a wide variety of user groups and constituencies. Based on information gathered from over 50 individuals and from a review of relevant literature and Web sites, the report examines perceived gaps, their causes, and the activities generated in response to those perceptions. The report identifies 23 ongoing efforts within the private sector, States, and foundations to address these gaps, and studies 11 of these more thoroughly. The study concludes that private sector stakeholders and States cannot alone sufficiently address supply-side information needs, giving importance to Federal leadership and convening functions.
State Regulatory Experience with Provider-Sponsored Organizations: Final Report (6628) describes the experience of selected State governments in regulating provider-owned health care delivery systems that accept insurance risk for the provision or arrangement of health care services (provider-sponsored organizations, or PSOs). The States included in the study are California, Colorado, Illinois, Iowa, Minnesota, Ohio, Pennsylvania, Texas, and Washington. The report finds that the two models that States use to regulate the activities of PSOs contracting directly with purchasers of health care (e.g., employers) are (1) licensing the organizations as they would a nonprovider-owned entity, and (2) creating a separate licensing category for risk-assuming PSOs. The study found that the States vary considerably in their regulation of PSOs that assume risk from self-funded, ERISA-exempt employer plans, and concludes that since State policy toward PSOs is still in the early stages of development, the lessons to be learned by Federal policymakers are limited.
Establishing an Analytical Framework for Measuring the Role of Reinsurance in the Health Insurance Market (6573). Reinsurance is widely used by underwriters of health benefits to limit their risk exposure. Reinsurance contracts can be structured in many ways, and insurers can use reinsurance to limit their risk for an individual contract or exposure, for a group of contracts or exposures, or for a whole portion of the insurer's business. This report presents a limited analytical study of the level of risk borne by reinsurers in three markets: the indemnity insurance market, the managed care market, and the self-insured employer benefit plan market. The purpose of the study is to determine to what extent these data can be used to measure the role of reinsurance in the various markets and to suggest ways whereby better assessments of reinsurance might be made. The report was generally unable to identify either data sources or existing research allowing quantification of the risk being ceded to reinsurers in the three markets. While a substantial amount of information is available about reinsurance in indemnity (non-HMO) health insurers, the data are aggregated in ways that make it difficult to specifically identify transactions related to primary health insurance offered to groups and individuals.
A Framework for Assessing Insurer Responses to Health Care Market Changes (6572). Where once a few large commercial carriers dominated the market with fee-for-service plans, there are now dozens of plans in most markets. In highly competitive markets, plans and carriers develop more sophisticated strategies and arrangements to identify and retain target markets. To better
integrate government programs and develop public policy, the variations and rationales for insurers' business strategies need to be better understood. This project developed an analytical framework for analyzing insurer behavior, responses to insurance market reforms, and the likely impact of insurer behavior on the availability and cost of insurance.
Deriving State-Level Estimates From Three National Surveys: A Statistical Assessment and State Tabulations (6380). HHS programs are rapidly changing their focus from the national to the State level. Federal roles and responsibilities traditionally played by HHS will be dramatically altered. Recent events bear this out. Landmark legislation eliminating entitlement to Aid to Families with Dependent Children (AFDC) benefits and block granting it to the States was just passed. States have vast new flexibility to administer Medicaid as the result of the 1115 waivers. The basic data needed to address departmental policy and research issues have typically come from major, national, Federally-sponsored population-based surveys, which are designed to provide national (not State) estimates. This project focused on three major Federal surveys--the Survey of Income and Program Participation (SIPP), the Current Population Survey (CPS), and the National Health Interview Survey (NHIS)--selected for their widespread applicability, use, and potential for analyzing policy issues. The purpose of this project was to (1) understand the statistical issues involved in addressing policy and research questions at the State level with national survey data; (2) assess the current capabilities of these three major surveys to provide answers to these questions at the State level; and (3) present options for obtaining valid reliable State-level data from relatively minimal enhancements and changes to survey design.
Improving Health in the Community: A Role for Performance Monitoring (6583) draws on lessons from a number of current community health efforts to outline the elements of a community health improvement process, discusses the role that performance monitoring can play in this process, and proposes tools to help communities develop performance indicators. The report finds that a community health improvement process (CHIP) can be an important tool for developing a shared vision and supporting a planned and integrated approach to improving community health. The report suggests that a CHIP should include two principal interacting cycles based on analysis, action, and measurement. The first is a problem identification and prioritization cycle; the second is an analysis and implementation cycle.
Privacy and Health Research: A Report to the U.S. Secretary of Health and Human Services (6605) identifies privacy issues surrounding research on personally identifiable health data, paying special attention to the international aspects; reviews the ethical, legal, and general social context surrounding the privacy and confidentiality of health data; describes relevant privacy-protection practices and problems, and identifies emerging issues; analyzes the implications of the new European Union Data Privacy Directive and related policy and legal changes; and recommends policy approaches and technical processes for ensuring that the privacy of individuals is respected. The report examines four major groups of issues that are growing rapidly in scale and complexity, and that must be urgently attended to: (1) secondary uses of data and data linking, (2) research on private-sector health data, (3) cybersecurity, and (4) genetic privacy. The international flow of data, new health data privacy laws in Europe, the United States' new Health Insurance Portability and Accountability Act, and the need for dialogue between the United States and Europe are also examined in detail.
Disability, Aging and Long-Term Care Policy
State Assisted Living Policy: 1996 (4719.3) describes assisted living facilities as an expanding source of supportive housing and services that represent a new model of long-term care. Consumer choice and control are central to assisted living models, which seek to allow consumers to control key features of their living environments and to direct their own receipt of services. This study reviews policies on assisted living and board and care in each of the 50 States. One of the major difficulties associated with assisted living is the lack of a common or standard definition of this kind of care. The report provides specific information about policies in each State.
The Role of Home and Community-Based Services in Meeting the Health Care Needs of People with AIDS: Literature Review (6418.1) provides an overview of the literature (published since 1991) describing the delivery and financing of home- and community-based services for people with AIDS. The review draws inferences from literature on the provision of these services to these elderly for their potential application to the AIDS population.
Consumer-Directed Personal Assistance Services: Key Operational Issues for State CD-PAS Programs Using Intermediary Service Organizations (6728). This project examined approaches and solutions for implementing consumer-directed service models for the delivery of home and community-based personal assistance services (PAS) for persons with disabilities. It addressed issues in this area arising from regulatory compliance requirements as well as issues concerning potential tort liability faced by payers, consumers, and providers of these services. The project included case studies of up to 20 programs that use a variety of intermediaries and the services they provide; meetings with Federal officials to identify and discuss tax and labor regulations affecting PAS providers and consumers; and development of model contracts of Federal regulations and requirements in clear, simple, and accurate terms.
Human Services Policy
Evaluating Two Welfare-to-Work Program Approaches: Two-Year Findings on the Labor Force Attachment and Human Capital Development Programs in Three Sites (6576.1) examines the relative strengths and limitations of particular versions of the labor force attachment (LFA) strategy and human capital development (HCD) strategies. It includes the findings from one part of a multi-year, 7-site evaluation and draws on the advantages of a unique experimental design implemented in three sites: Atlanta, Georgia; Grand Rapids, Michigan; and Riverside, California. The evaluation had its origins in the Family Support Act of 1988, which marked a major shift in the philosophy of welfare by establishing a system of mutual obligation--between government and recipients--within the AFDC entitlement structure. The report is part of a larger study called the National Evaluation of Welfare-to-Work Strategies (formerly known as the JOBS Evaluation). It contains an analysis of the implementation, participation patterns, and costs of the two types of programs operated in each site and an assessment of the effectiveness of the LFA and HCD program approaches in promoting employment and reducing welfare.
Setting the Baseline: A Report on State Welfare Waivers (6651). The Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA) of 1996 eliminated the entitlement to Aid to Families with Dependent Children (AFDC) and replaced it with Temporary Assistance for Needy Families (TANF), a block grant that States can use to provide cash and services to low-income families with children. This report is one in a series designed to provide baseline information on the AFDC program as it existed prior to enactment of the new law. The report focuses on the waivers of AFDC requirements that were granted to States in the years preceding enactment of PRWORA. It provides an overview of the types of waivers that States requested and received, and provides details on specific State policies.
Approaches to Evaluating Welfare Reform: Lessons from Five State Demonstrations (6050). Prior to the enactment of PRWORA, many States obtained waivers from AFDC and Food Stamp Program rules in order to implement welfare reform. Between 1993 and 1996, the Clinton administration approved waivers for 43 States. This project examined waiver demonstration projects in five States (California, Colorado, Michigan, Minnesota, and Wisconsin). Four of the five States used an experimental evaluation design, while the fifth State used a quasi-experimental evaluation design. The report addresses issues in five areas: (1) the choice between an experimental and quasi-experimental evaluation design, (2) sample design, (3) implementation of experimental evaluations, (4) data collection, and (5) analytical methods.
Implementing Welfare Reform Requirements for Teenage Parents: Lessons from Experience in Four States (6228). Federal welfare policy requires minor custodial parents receiving cash assistance to attend school and to live with their parents or in an adult-supervised setting. These requirements were established as part of the PRWORA, which created the program for Temporary Assistance for Needy Families (TANF) and abolished the Aid to Families with Dependent Children (AFDC) program. This report summarizes lessons learned from the operational experiences in four States (Arizona, California, Massachusetts, and Virginia) that implemented waivers requiring school attendance and certain living arrangements under the AFDC program.
Determinants of AFDC Caseload Growth: Final Report (5953). This project examined the dynamics of Aid to Families with Dependent Children (AFDC) caseload growth and analyzed the effects of economic, demographic, and programmatic changes at the State level on States' AFDC caseload from the early 1980's to the present. The project addressed the interaction of these factors by analyzing the economic, demographic, and programmatic factors simultaneously. Compared to earlier studies of AFDC caseload determinants, many of which are outdated, this study focused much more on State-level indicators, and included a more comprehensive set of economic, demographic, and programmatic factors.
Indicators of Welfare Dependence and Well-Being: Interim Report to Congress (6634). The Welfare Indicators Act of 1994 directed the Secretary of Health and Human Services to conduct a study to determine which statistics would be most useful in tracking and predicting dependence on three means-tested cash and nutritional assistance programs: Aid to Families with Dependent Children (AFDC), Food Stamps, and Supplemental Security Income (SSI). Enactment of the PRWORA of 1996 makes this task even more important. The report recommends two kinds of indicators: indicators of dependence, self-sufficiency, and family conditions; and indicators of child achievement, health, and well-being. In most cases, the recommended indicators should be tabulated separately for children, male non-elderly adults, female non-elderly adults, and other important population subgroups wherever possible. To measure dependence, self-sufficiency, and family conditions, the report recommends that indicators reflect points on a continuum, from total dependence to complete self-sufficiency.
Iowa's Limited Benefit Plan (6390). Iowa's Family Investment Program (FIP) replaced Aid to Families with Dependent Children (AFDC) in 1993. FIP participants who are able-bodied and not caring for young children--yet who do not develop and carry out a Family Investment Agreement (FIA)--are sanctioned with assignment to the Limited Benefit Plan (LBP). Some clients may also choose LBP over FIP. LBP originally provided 3 months of benefits at FIP levels, followed by 3 months of reduced levels and 6 months of no cash benefits; the program was later changed to eliminate the original 3-month period. At the end of the 6-month period, the client can reapply for FIP, but must meet its requirements. This report presents findings from a study of the original LBP, showing how families enter the LBP and how their well-being changes after benefits are cut. It examines records for over 4,200 cases assigned to LBP records during 6 months in 1994 and 1995, a survey of 137 cases whose cash benefits were terminated, and case studies of 12 LBP families. The authors note that there are several features of the LBP program that mitigate against extreme deprivation when cash assistance has been terminated, and that the absence of these safeguards in a national welfare reform program could result in extreme hardship for many families.
Family and Children's Social Services Policy
Informal and Formal Kinship Care (6016). A growing number of States prefer to place children who have been removed from their homes of origin in the home of a relative. This practice is known as kinship foster care and it is a growing component of family foster care. It comprises between one-third and one-half of all foster care days provided in the States with the largest foster care populations. This study uses several data sources to examine the characteristics of informal kinship care arrangements and how they compare with those initiated through or subsidized by the State child welfare system. It considers national patterns, examines living arrangement patterns by State for 1990, compares formal and informal kinship care in four States, and uses data from Illinois to provide a more focused picture of kinship care.
Trends in the Well-Being of America's Children and Youth: 1997 (6170.1). This is the second edition of an annual report on the well-being of children and youth in the United States. The report's purpose is to provide the policy community, the media, and all interested citizens with a statistical overview describing the condition of children. The report focuses on trends in the well-being of children and youth in 5 areas: (1) population, family, and neighborhood; (2) economic security; (3) health conditions and health care; (4) social development, behavioral health, and teen fertility; and (5) education and achievement.
Efforts by Child Welfare Agencies to Address Domestic Violence: The Experiences of Five Communities (6155). Historically, child protective services and domestic violence services have not been closely linked. However, child welfare agencies across the country are beginning to consider how families in their child protection caseload are affected by domestic violence and are re-examining their policies accordingly. This report documents how child welfare agencies in five communities (Massachusetts; Michigan; San Diego County, California; Hilo, Hawaii; and Oregon) are attempting to integrate domestic violence concerns into their services. The communities were chosen for study because they are extending their services well beyond educating staff in domestic violence concerns. The report includes a discussion and evaluation of approaches to addressing domestic violence within child welfare agencies; the role of the larger community in preventing domestic violence; and more complex policy questions, such as whether a child's witnessing of domestic violence is itself abusive.
Evaluability Assessment of Responsible Fatherhood Programs (6159). Responsible fatherhood programs provide fathers, particularly those who are young, unmarried, and unemployed, with the support and skills to become active participants in their children's lives. An increased interest in fatherhood programs and a lack of information on their accessibility and effectiveness led to this evaluability assessment. The study visited five fatherhood program sites to assess the readiness of the programs for outcome evaluation and to determine the evaluation process and mechanisms necessary for a full evaluation.
Chapin Hall Center for Children at the University of Chicago: Administrative Data Guide (6389). This report is a guide to administrative data resources on health, social services, and related programs available through the Chapin Hall Center for Children at the University of Chicago. Each data section in the guide provides an overview of a particular data set. Included is information to assist the researcher in interpreting the data sections. The guide was designed for use by an audience of informed policy analysts in order to consider what data are available to answer particular policy questions. ASPE will use the guide to better design short-turnaround analyses on a variety of topics of current policy interest.
Evaluations in Progress
ASPE's evaluations in progress cover a broad range of program and policy areas. In health policy, ASPE is concentrating its efforts in the four areas described below.
Studying Changes in Health and Human Services Programs. America's health care system is undergoing rapid and dramatic change. States are using a variety of laws (e.g., the Balanced Budget Act of 1997 and PRWORA of 1996), as well as Medicaid and welfare policy waivers to make substantial changes to their welfare and health financing and delivery programs. In this area, ASPE's goal is to put into place mechanisms to assess the impact of such legislative changes, the interactive effects of waivers and changes in the law, and the effects of the rapidly changing private sector environment. ASPE is supplementing ongoing private sector and foundation efforts to create a baseline of health information at the State level with projects that will help define key measurements and themes with which to monitor the effect of both program changes and changes in the environment. For example, ASPE has projects under way to examine child health insurance patterns (6721) and innovative State strategies (6729) to insure children.
Understanding Private Sector Changes. In recent years, rapid changes to health care in the private sector have transformed the landscape of health financing and service delivery systems. ASPE's goal is to understand the private sector environment in which HHS programs operate and to be able to describe and monitor ongoing changes. A better understanding of how the private sector works and the dynamics of these changes are essential to developing Federal policy. For example, ASPE is currently managing projects to evaluate the effects of "direct-to-consumer" advertising on consumer knowledge, attitudes, and behaviors concerning prescription drugs (6343) and innovative coverage of health promotion, clinical preventive services, and other coverage issues in the private sector (6347).
.Improving Data and Analytic Capability. Current projects that focus on describing, evaluating, and analyzing information to develop estimates for the decisionmaking process require resources for data collection and analytic capability. An actuarial services contract, for example, supports a wide range of analyses, including cost estimates related to legislation. In addition, the rapid evolution of the delivery system has created information gaps that need to be addressed in a comprehensive way. ASPE has several projects that are part of the HHS "Data Initiative," an effort to fill gaps in the data needed to analyze and monitor the health and well-being of the population, particularly given recent policy changes in HHS programs. This effort focuses on the need for State-level data, better linkages between health data and human service data, data on the impact of policy changes on the well-being of low-income children, and data on health care providers and the supply of employer-provided fringe benefits. For example, ASPE is exploring the linkage of data files from the National Health Interview Survey and the Survey of Income and Program Participation (6419) to be used in program evaluation. This effort emphasizes making better use of existing data.
Agency Related Priority Studies. ASPE continues to collaborate with various HHS agencies on program evaluation and technical assistance. As resources continue to shrink, the need to develop and share information on program efficiency and effectiveness assumes even greater importance. For example, ASPE is working on a number of projects with the Public Health Service (PHS) agencies regarding the food additive review process (6342), Indian health programs (6345), and the role of enabling services in the provision of health care (6750). These continue a tradition of joint projects with both the PHS agencies, as well as with HCFA on Medicare and Medicaid payment policies such as a study of managed delivery systems for Medicare beneficiaries with disabilities and chronic illnesses (6391).
In the area of disability, aging, and long-term care issues, ASPE is focusing its efforts in several areas.
Disability, Aging, and Long-Term Care Trends. ASPE has current projects to assess the profound demographic changes brought about by the aging population and the implications of these changes for active aging and health, long-term care, and retirement policies for the next century. ASPE is looking at the implications of disability for the Nation's health, long-term care and personal assistance, welfare, income security, housing, transportation, educational, civil rights, and employment policies. For example, one project is examining two large claims data sets to assess the health and long-term care expenditure patterns of children with disabilities (5758). Another project is reviewing information on how health insurance coverage and the particular provisions of health care policies are related to employment by people with disabilities. The review will consist of an examination of the available research literature on the relationship of health care coverage and employment among people with disabilities (6240).
Promoting Active Lifestyles and Maximum Independence. More seniors and individuals with disabilities can and should be able to expect to lead active and productive lifestyles of their own choosing and direction than ever before in history. There is increasing evidence that disability rates among the elderly are falling. New advances in technology, the availability of personal assistance services in the community, and the stimulus of new civil rights are empowering individuals with disabilities to live independent lives. Through policy analysis and research and demonstration functions, ASPE is identifying what is known and what remains to be learned about how to assist children and adults with the full range of disabilities to become contributing members of their families, communities, and the Nation (6719).
Managed Care and People with Disabilities. Managed care can provide unique opportunities, but it also has potential pitfalls for people with disabilities. ASPE has projects in progress to determine the impact of managed care on access, quality, and satisfaction for people of all ages who experience disabilities (6398). Of particular interest are two policy issues: (1) there is a concern about the impact of extending Medicaid managed care to the SSI-eligible disabled population, (2) the impact of managed care on elderly persons with physical and cognitive impairments needs to be examined. Evaluation of a comprehensive program for capitating acute and long-term care services for children with disabilities will be initiated.
Assisted Living. ASPE has commissioned a national study of assisted living, or the residential settings that combine adapted housing, assistive technologies, personal assistance, and other supportive services for persons with disabilities (4719.5). Assisted living is considered an important component of services to the aging American population and an effective response to the rising costs of nursing home care.
The following projects are focused on human services issues.
Moving Welfare Recipients to Work. ASPE continues to work with the Administration for Children and Families (ACF) to conduct a comprehensive, multi-year study of the different strategies designed for moving people off welfare and into employment, looking particularly at the cost-effectiveness of different approaches. The evaluation will address questions about the long-term effects of different welfare reform approaches, including whether any approach significantly improves the economic circumstances of the most disadvantaged recipients (6576.3) and the effects of welfare-to-work programs on the children of welfare recipients (6576.4). ASPE has also funded a project to look at one-stop shopping models for welfare-to-work programs (6891) and a project to examine the impact of local labor markets on welfare-to-work transitions (6892).
Measuring the Outcomes of Welfare Reform. ASPE recently initiated a project to look at the impact of employment on AFDC recipients in Wisconsin (6727), comparing the characteristics and outcomes of two groups of women who have left the AFDC program. In collaboration with ACF, HCFA, the Immigration and Naturalization Service, and the Department of Agriculture, ASPE is now studying the impact of welfare reform on the economic and health status of immigrants, their communities, and the organizations that serve them (6747). ASPE has also funded a study in New Jersey to look at changes in family well-being and family coping strategies following benefit sanctions (6893).
Welfare Reform Implementation Issues. ASPE has several projects examining human service delivery policy questions related to the implementation of welfare reform. As welfare programs require employment or participation in work-related activities, a better understanding of the low wage labor market is critical. ASPE has commissioned a series of papers to summarize and interpret the results of previous research on the low-wage labor market (6717). ASPE will examine the relationship between domestic violence and welfare reform (6724) by looking at implementation of the TANF Family Violence Option, changes in child support enforcement programs, and the use of good cause waivers for battered women. Evidence suggests that some battered women seek employment and child support while others need specialized assistance in addressing the abuse of partners. A third example is ASPE's study of child health and development programs in the context of welfare reform (6754). This project will identify and present profiles of promising Federal, State and/or community-based health and human services programs believed to be enhancing the health and development of children in the context of welfare reform.
Child Support. ASPE will examine the impact of a change in child support review and modification policy under PRWORA (6749). The law prior to the enactment of PRWORA required that all AFDC child support cases be reviewed every 3 years and adjusted if the child support award was inadequate. PRWORA made review and modification in TANF cases optional. A number of States may discontinue review and modification in TANF cases. This project will provide estimates of the financial impacts to the Federal and State governments, or how such a discontinuance would impact on child support offsets to cash assistance payments. In addition, the financial impacts on families are to be estimated, paying particular attention to those who have left the welfare rolls.
Family Preservation Services. ASPE and ACF are now examining the following types of family preservation programs: placement prevention services aimed at preventing children from entering substitute care; broader family preservation services that may be less intensive and of longer duration than placement prevention services; and reunification services to speed the return of children to their homes after entering substitute care (5337.2). Measures of program success will include prevention of placement of children into substitute care (for pre-placement services), successful reunification (for reunification services), improved child behavior, improved family functioning, and reduction of the recurrence of child abuse and neglect.
Teen Pregnancy. ASPE is coordinating several efforts with HHS agencies to develop and evaluate strategies to reduce teen pregnancy. The first project is part of the National Strategy to Prevent Teen Pregnancy, aimed at furthering our knowledge about teen pregnancy prevention programs and the process of building and strengthening partnerships to reduce teen pregnancy and provide a positive future for youth (6725). ASPE will provide information and tools to help communities develop teen pregnancy prevention programs and partnerships. Next, ASPE will conduct a project on understanding and preventing abusive intimate relationships among adolescents (6751). It will examine the issue through a literature review; analysis of newly available data from the National Longitudinal Study of Adolescent Health, the National Survey of Family Growth, and other surveys; identification of data needs; focused discussion groups; a summary of promising approaches to prevention and intervention; and recommendations for further research, policy, and discussion. Finally, ASPE has two teen pregnancy prevention efforts that target young men. First is a project to develop a research and policy information strategy to inform regional, State, and local policy officials and community-based organizations serving children and youth of the program models--primarily directed at boys and young men-- that can be implemented to help teenagers avoid premature sexual activity and unintended pregnancies (6766). Second, there is a project is to identify abstinence-based pregnancy prevention programs that target boys or both boys and girls (6726). It will use a framework to assess and summarize the state of the field and provide the information in a format useful to State, local, and community policymakers. States and communities have shown interest in providing services and programs for boys and young men, but have also expressed frustration that they do not have access to information about what strategies have been tried, if they have been successful, and how they can access funding for the development and operation of these programs.
OFFICE OF PUBLIC HEALTH AND SCIENCE
MISSION: To provide advice on public health and science to the Secretary of Health and Human Services, to provide executive direction to program offices within the Office of Public Health and Science (OPHS), and, at the direction of the Secretary, to coordinate crosscutting public health and science initiatives in the Department.
Evaluation Program
The Office of Public Health and Science (OPHS) provides advice, policy and program coordination, and leadership in the implementation, management, and development of activities related to public health and science, as directed by the Secretary. OPHS helps HHS conduct broad-based public health assessments to better define public health problems and solutions. It assists other components of HHS in anticipating future public health issues and problems and helps ensure that HHS designs and implements appropriate approaches, interventions, and evaluations that will maintain, sustain, and improve the health of the Nation. OPHS provides leadership and policy recommendations on population-based public health and science and, at the Secretary's direction, leads or coordinates initiatives that cut across agencies and operating divisions. In addition, OPHS communicates and interacts, on behalf of the Secretary, with national and international professional and constituency organizations on matters of public health and science. Finally, OPHS's unique role allows it to use its resources to link important HHS programs or fill gaps in areas needing better policy formulation and coordination.
In keeping with its role within the Department, OPHS has developed an evaluation plan that avoids duplication of efforts that might more appropriately and effectively be undertaken by operating divisions of HHS or by the Assistant Secretary for Planning and Evaluation (ASPE). Thus, the FY 1998 evaluation strategy for OPHS focuses on public health and science issues that cut across multiple interests of the operating divisions, and requires a coordinated approach to achieve the most effective results. In addition, OPHS will continue its commitment to carry out every project proposed as part of this year's strategy in collaboration with relevant operating divisions.
OPHS conducts evaluations requested of the Secretary by Congress that are most appropriately managed by staff with medical or health science credentials, and that cannot be assigned to an operating division. Further, evaluations will be conducted to support the Surgeon General and the Assistant Secretary for Health in their respective roles as the Nation's top doctor and senior advisor for public health and science to the Secretary. These roles include the responsibility to assist the Secretary in developing a policy agenda for the Department to address major population-based public health, prevention, and science issues and to provide leadership and a focus for coordination of population-based health, clinical preventive services and science initiatives that cut across operating divisions. In addition, OPHS will conduct evaluations specific to the needs of the programs operated from the offices located within OPHS, such as Women's Health, Minority Health, Disease Prevention and Health Promotion, International Health, and Emergency Preparedness. A portion of the evaluation funds will be made available to the ten HHS Regional Health Administrators.
Summary of Fiscal Year 1997 Evaluations
In FY 1997, OPHS completed five evaluations in an effort to track the impact of the transformation in the Nation's health care system on underserved and vulnerable populations or to improve OPHS program performance. Studies completed this year will better inform the public health and science communities concerning the return on their investment of scarce public health resources and will help shape future budget decisions.
In an effort to quantify investments made by the various levels of government in essential public health services--both personal health services and population-based services--OPHS worked with State and local public health, mental health, substance abuse, and environmental agencies in nine States to define, measure, and monitor pubic health expenditures (6194). Using the essential services framework, the study concluded that expenditures could indeed be determined. Some rather surprising conclusions were reached concerning the source of the largest portion of public health expenditures, indicating that more than 2 out of every 3 dollars spent on essential public health services went for personal health services and that only 1 percent of the total health care expenditures went for population-based health services.
OPHS funded development of a study to identify the types of information and data necessary to successfully protect and monitor the human health hazards of environmental incidences (6322). The study concluded that the Federal government could help State and local public health agencies in environmental surveillance by assuring integration of public health, environmental health, and environmental protection agencies' information. However, any information system developed must be evaluated based on its usefulness to State and local agencies that respond to adverse effects to human health arising from exposure to environmental hazards.
In FY 1997, OPHS asked stakeholders in the Healthy People 2000 process to evaluate the framework of national health goals and disease prevention objectives in anticipation of the development of the Healthy People 2010 plan (6491). By reviewing the successes and failures of the Healthy People 2000 benchmarks, OPHS hoped to make this national framework of performance indicators on health status more results-oriented. In general, participants in the various forums liked the overall structure and content of the Healthy People 2000 document, although some recommended reducing the size of the document for Heathy People 2010 and encouraged the use of new communication avenues for this valuable assessment tool, making it available to a wider range of professionals and community leaders. Both health care purchasers and managed care providers agreed that the number of objectives should be reduced to permit adequate focus on a limited number of problem areas.
Using resources made available through the HHS Regional Health Administrators, one study examined the health and well-being of women in the State of Washington (6684). Through a broad coalition of health care providers, researchers, and activists, as well as focus groups of women from diverse geographical areas, economic levels, and social strata, the report concluded that in general, women in Washington State are living longer and enjoy a better standard of living than women elsewhere in the United States. Although access to health care overall was relatively good, the report indicated that elderly women, those with less than a high school education, and poor women were less likely to use mammography and have regular Pap smears. The report will be used to provide recommendations for addressing several problem areas for Washington women's health.
Finally, in a continuing effort to evaluate program effectiveness, the Office of Minority Health concluded an analysis of sponsored program offices at Historically Black Colleges and Universities (HBCUs) (6246). The evaluation determined that grant funds and technical assistance had achieved the program aims begun in 1992 to increase the involvement of HBCUs in health and social service programs funded by the Public Health Service and other Federal agencies. On campuses where sponsored program offices were funded, there was an increase in proposal submissions and awards. The report indicated a continuing need to invest in both programmatic and administrative infrastructure.
Evaluations in Progress
In reviewing evaluation projects for FY 1997, OPHS gave highest priority to efforts aimed at tracking the enormous transformation occurring in the Nation's health care system. The shifting emphasis on managed care presents new dilemmas about how to ensure that, in the quest to control cost, overall quality of care is not unduly compromised. Ensuring a workforce that is properly trained and fully capable of performing essential public health services in this dynamic health care environment is a task that OPHS must, at a minimum, also monitor. Monitoring this entire process presents new challenges to HHS, OPHS, and their partners in the public health community.
The FY 1997 OPHS evaluation plan emphasized the need for a strong foundation for public health in the 21st century. Priorities focused attention on (1) the information systems and workforce that comprise the Nation's public health infrastructure and that are necessary to effectively provide the essential services of public health; and (2) the impact of managed care arrangements on the resources available for this infrastructure and on the health of all Americans, especially those most vulnerable. In support of broader HHS efforts to improve departmental programs, OPHS evaluations also reflect the assessment of various programs and activities funded through PHS.
Several studies were continued from previous years. The Commission on Dietary Supplement Labels, appointed by the President, continued its evaluation of factors relevant to Food and Drug Administration regulation and possible legislation of label claims and statements for dietary supplements as requested by Congress (6193). The National Academy of Sciences also continued its study of dietary reference intakes (6323). In the area of program improvement, OPHS continued supporting an evaluation of a new model of coordinating and integrating HIV prevention and primary care services among high-risk populations (6321), an evaluation of the Minority Health Resource Center (6244.1), and the assessment of the efficiency and effectiveness of the Office of Minority Health's Bilingual/Bicultural Service Demonstration Grant Program (6247). In the area of communication, OPHS is designing and evaluating applications of interactive communications for consumer health information and examining the utility and impact of timed dissemination of public health information from PHS agencies (6327). OPHS continued its support of a study to identify evaluations of school health programs and make available to school boards, administrators, health personnel, and health educators an updated compilation of methodologically sound studies of these programs (6198).
In an effort to track the impact of managed care arrangements on the health care system, OPHS collaborated in a broader study of health systems change (being funded by the Robert Wood Johnson Foundation) to track changes in the public health sector in 12 randomly selected communities over a 4 year period (6325). OPHS also developed a research agenda to address gaps in scientific knowledge related to cultural competence in health care and linking measures to outcomes (6675), and assessed managed care organizations serving racially/ethnically diverse communities to determine the extent and nature of linguistically and culturally appropriate services (6674). When completed, the study will analyze and report the range of services being provided, identify best practices and model approaches, and explain the community and organizational factors conducive to providing linguistically and culturally appropriate services.
In FY 1997, OPHS undertook a project in cooperation with the United Kingdom that examines efforts to address the health concerns of racial and ethnic minorities in the two countries and, for six issue areas, conducts comparative analyses of strategies and approaches to identify strengths and limitations, best practices, lessons learned, and future areas for collaboration. For each of the six issues, a paper incorporating these comparative analyses into an integrated piece suitable for publication is forthcoming (6676).
In a continuing effort to improve the usefulness of Healthy People 2000 objectives, OPHS worked with States and localities to determine their ability to assess health trends in their communities (6488). Although Healthy People 2000 has driven the development of some surveillance and data systems at the national level, the usefulness of this information is dependent on State and local ability to measure objectives locally. Information from this study will be used to develop objectives for Healthy People 2010, in an effort to make them more usable for local performance and health assessment. In this area, OPHS is also assessing linguistically and culturally appropriate community health promotion programs in local health departments to gather the baseline data necessary to support the review of Healthy People 2000 objectives (6798).
The balance of the ongoing evaluation projects funded by OPHS were conducted on a regional basis, examining programmatic impact in significant Presidential or departmental priority areas or in vulnerable or special populations. Specifically, four studies examined effectiveness of services in the area of women's health: preventive health screening among older women (6677), breast and cervical cancer screening in Region X (6682), averted pregnancies and associated cost savings in Region X (6683), and a retrospective study of the preventive health practices of former Title X (Family Planning) clients (6685). Other studies are looking at the impact on public health services of an increased migrant population in northwest Arkansas (6687) and evaluating the impact of the loss of Federal funds in the AoA and ACF programs on the Freely Associated States (6688). Studies reviewing activities concerning Presidential or departmental priorities include an evaluation of an immunization education program for child care centers (6680), a study of males who father children born to teenagers (6681), an assessment of Region IX programs to promote positive images in girls ages 9-14 (6689), and an analysis of local capacity to respond to the health and medical consequences of welfare reform (6686). One project is evaluating the community-based public health practice training program (6678).
SUBSTANCE ABUSE AND MENTAL HEALTH SERVICES ADMINISTRATION
MISSION: To improve the quality and availability of prevention, treatment, and rehabilitation services for substance abuse and mental illness.
Evaluation Program
The Substance Abuse and Mental Health Services Administration (SAMHSA) is committed to evaluating its overall programs and individual grant projects to assess the effectiveness of prevention, treatment, and rehabilitation approaches and systems of care; the accountability of Federal funds; and the achievement of SAMHSA's programmatic and policy objectives.
To the greatest extent appropriate and feasible, SAMHSA encourages the use of comparable data elements and instruments across its evaluations in order to work toward a comprehensive evaluation system and to minimize respondent burden.
SAMHSA conducts grant programs under a variety of legislative authorities. These authorities can generally be grouped into two types: (1) services and (2) knowledge development and application (KDA). The evaluation required for a particular grant program is dependent on the type and purpose of the program. SAMHSA evaluates each of its service programs so as to provide information to program managers about the accountability of Federal funds. In addition, the evaluations of KDA programs will generate new knowledge to lead the field in the development of policies that improve services.
The two types of grant programs (service delivery and KDA) represent the two facets of SAMHSA's mission. SAMHSA's leadership in the field depends on the successful interaction of these two facets. Through KDAs, SAMHSA must identify effective approaches to prevention, treatment, and rehabilitation. Through service delivery funds, SAMHSA must provide incentives to the field to implement effective approaches. Major emphases of SAMHSA's mission are to develop, identify, and disseminate effective strategies and systems for prevention, treatment, and rehabilitation.
SAMHSA is implementing an integrated model of evaluation and planning. Strategic planning identifies priorities, such as managed care, that drive the development of grant programs and evaluations. In compliance with the Government Performance and Results Act (GPRA), SAMHSA is improving performance management by identifying annual performance objectives and measures. The formulation of programmatic and evaluation priorities includes consultation with SAMHSA, Center Advisory Councils, and with other experts in the fields of evaluation and service delivery. Early and continuous coordination of program planning and evaluation design will result in the articulation of program objectives that may be evaluated. Evaluations will demonstrate the extent to which the grant programs have achieved their overall objectives, and SAMHSA will translate these results into information that can be used for program and policy development. The strategic planning and policy development processes will then use these results to refine SAMHSA's priorities and performance objectives.
This evaluation policy will help SAMHSA achieve its goal of continually informing policy and program development with knowledge culled from past performance. In this way, SAMHSA can best serve its customers by enhancing the quality of publicly-funded substance abuse and mental health services.
In compliance with the Public Health Service (PHS) guidelines for the technical review of evaluations, SAMHSA has established a standing committee of PHS staff who are evaluation specialists. Representatives of the Office of the Assistant Secretary for Planning and Evaluation serve as ex officio members of the committee. The SAMHSA evaluation officer is the committee chair. The committee does not generally review the evaluation proposals of individual grantees; rather, it reviews proposals for broader, more comprehensive evaluations, such as the cross-project evaluations of grant programs.
Evaluation project proposals are generally prepared by SAMHSA program staff in the various Centers. The standing committee reviews each proposal on the following criteria: clarity of evaluation objectives and research questions, appropriateness and feasibility of the specifications for evaluation design and methods, appropriateness of the plans for dissemination of results, and use of previous relevant evaluations and existing program data systems. Each proposal must clearly state the relationship of the evaluation to SAMHSA's overall policies, priorities, and evaluation program.
Summary of Fiscal Year 1997 Evaluations
During FY 1997, SAMHSA completed six evaluations. A summary of the resulting reports follows.
The NTIES: National Treatment Improvement Evaluation Study: Final Report (5346.1), highlighted in Chapter II, was a comprehensive assessment of the results of 157 three-year demonstration grants and cooperative agreements funded by the Center for Substance Abuse Treatment (CSAT) beginning in 1990-91. These grants were intended to improve alcohol and drug treatment in selected target cities with severe substance abuse problems, among critical populations such as minority groups and women, and among persons in the criminal justice system. Based on a final sample of 6,600 clients who entered treatment, the results showed a substantial reduction in the selling of drugs and the commission of violent crimes, high-risk sexual behaviors related to HIV transmission, medical visits, homelessness, and inpatient mental health visits. Results also showed a substantial increase in employment.
The CSAT Demonstration: Evaluation of Job Corps Drug Treatment Enrichment Program (4523) was a 4-year demonstration project sponsored by CSAT and put in place within the Department of Labor's Job Corps program. The Drug Treatment Enrichment Program (DTEP) was implemented in four experimental Job Corps Centers. These were matched with four control centers that provided the standard Job Corps Alcohol and Other Drugs of Abuse program. The evaluation found that DTEP students reduced marijuana and crack or cocaine use following their Job Corps experience significantly more than did the students in the control groups. DTEP students were less likely to engage in moderate or extensive use of marijuana or to use other drugs, but no differences were found for alcohol abuse. DTEP students were less likely to report selling or helping to sell drugs after Job Corps. No differences between DTEP and controls were found for other criminal activities. DTEP improved students' mental health. DTEP students with mental health problems experienced a significantly higher job placement rate after Job Corps than did control students, although the program had no effect on job placement rates of students without mental health problems. DTEP appeared to have no significant positive effect on duration of stay in Job Corps or on overall educational and vocational gains. The final report makes recommendations regarding the integration of DTEP into Job Corps and other related policy implications.
The Center for Substance Abuse Prevention (CSAP) released the findings of a demonstration evaluation, Evaluation of Model Projects for Pregnant and Postpartum Women and Their Infants (5674). The evaluation assessed the effectiveness of the demonstration program in enhancing services coordination and in increasing the availability and accessibility of substance abuse prevention services delivered to clients. The study also documented the effectiveness of program models in decreasing alcohol and drug use among substance-abusing women, and enhancing the healthy development of their children. Ten projects funded by CSAP in 1992 were selected to participate in the evaluation. Data collected by the grantees were used for cross-site analyses. The major evaluation finding was that from intake to delivery, the treatment group members reported significantly lower use of marijuana, crack, and other illicit substances than comparison group members. These differences, however, were not significant by the 6-month postpartum followup assessment. Overall, treatment and comparison group members gave birth to infants with similar birth-related outcome characteristics. However, when the history of the mother's use of crack was taken into account, treatment babies had significantly higher birth weights than the babies in the comparison group.
CSAP also conducted a Report on the CTS National Evaluation (4514.1). The training system included the development and delivery of more than 30 curricula to more than 10,000 participants. Recipients included community teams, health professionals, State administrators, and volunteers engaged in prevention activities. The report found that the training system was innovative in several important respects. It focused on community coalitions and associations of health professionals that are responsible for planning and delivering prevention services, rather than concentrating on target populations. Curricula were developed quickly in response to emerging issues in the field. The training system had a continuing commitment to quality control and continual curricular improvement.
CSAP has completed Findings from the Evaluation of the Faculty Development Program, Volume I: Final Report (4510). This program offers training to faculty members from schools of medicine, nursing, and social work, and from graduate psychology programs. The purpose of the development program was to enhance research, teaching, and clinical practices on issues of alcohol, tobacco, and other drugs (ATOD). The evaluation found that faculty fellows, after participating in the program, reported (1) spending significantly more time on ATOD-related activities, (2) conducting more ATOD research, (3) incorporating more ATOD information in courses, (4) joining ATOD professional organizations, (5) increasing ATOD teaching or clinical skills, (6) producing more ATOD publications, and (7) making more ATOD presentations at conferences and other events than they had prior to their participation in the program.
The Center for Mental Health Services (CMHS) produced Final Report: Review of the CMHS Mental Health Care Provider Education in HIV/AIDS (6695). The program supports the provision of state-of-the-art training targeted to (1) traditional mental health care providers (psychiatrists, psychologists, nurses, and social workers); (2) other first-line providers of mental health care services (primary care physicians and medical students); and (3) nontraditional providers of such services (clergy or alternative health care workers). The goal of the program was to improve access to appropriate and adequate mental health services for people living with HIV/AIDS and for their families and partners. The evaluation found that training participants increased their knowledge about the mental health aspects of the disease, their willingness to serve HIV-infected individuals, and their confidence that they could do so effectively. The recommendations of the report stressed the need for training in the mental health aspects of HIV/AIDS throughout the country, especially as affected areas and populations change over time.
Evaluations in Progress
SAMHSA currently has major evaluation projects under way in two areas. First, CSAT's National Evaluation Database Service (5994) is providing centralized data management and analysis for the evaluation of several large demonstrations targeted to special populations, including female, adolescent, criminal justice, culturally distinct, and rural populations. The service represents part of an evaluation strategy that builds upon prior findings, and which seeks to identify a set of consistent evaluation questions that apply across similar substance abuse treatment programs targeted to special populations. The service has identified data elements to provide uniform information across sites so that comparisons of effectiveness could be made.
Second, CSAP is conducting a National Evaluation of the Community Partnership Demonstration Project (4997). This project was designed to identify successful prevention and partnership strategies and common inhibitors to forming successful partnerships and prevention programs. This impact evaluation has focused on assessments of short-term outcomes specific to each interview site, long-term outcomes reliant upon uniform surveys, and community-wide indicators of alcohol and drug abuse across several sites. Data have been collected through a questionnaire administered in schools and a telephone survey for adults.
Other evaluations in progress are looking at services integration for homeless persons with chronic mental illness (4980.1), effects of managed care on substance abuse treatment outcomes (6454), evaluation of substance abuse treatment improvement protocols (6737), long-term effects of drug abuse treatment (6738), and an evaluation of opioid treatment program accreditation (6739).