|
Office of the Assistant Secretary for Planning and Evaluation |
| |
|
Policy Information Center |
MISSION: To sponsor and conduct medical research that leads to better health for all Americans.
Evaluation Program
The National Institutes of Health (NIH) generates scientific knowledge that leads to improved health. This is done by conducting medical research in its intramural laboratories and by supporting research in universities, medical and health professional schools, and other health research organizations. NIH fosters the widespread dissemination of the results of medical research, facilitates the training of research investigators, and ensures the viability of the research infrastructure. The NIH Evaluation Program is an integral part of how NIH sponsors and conducts medical research.
Results based management is recognized as a basic principle for the sound and productive operation of government agencies and their programs. This is evidenced most notably by passage of the Government Performance and Results Act (GPRA) and by the considerable effort across the federal government to implement results based management mechanisms. With GPRA and other initiatives aimed at increasing public sector accountability (such as the Chief Financial Officers Act and the Government Management Reform Act), interest in the use of evaluation has increased steadily among NIH administrators and others, such as officials within the General Accounting Office, the Office of Management and Budget and the Department of Health and Human Services.
Philosophy and Priorities. The NIH Evaluation Program provides information to assist the NIH Director and the NIH Institute and Center (IC) Directors in determining whether NIH goals and objectives are being achieved and to help guide policy development and program direction. Evaluations are planned and conducted from two sources of funds: 1 percent evaluation set-aside funds used to fund trans-NIH projects, and IC program funds used for program evaluations for use by various committees, working groups, task forces, workshops, conferences, and symposia to assist the ICs in program management and development. This approach ensures that planning and priority setting specific to the mission of each IC are fully developed and implemented and that there is central leadership for developing crosscutting initiatives and promoting collaboration among the ICs.
NIH's major evaluation priority areas fall within three broad program areas: basic research, research training and career development, and facilities. NIH conducts evaluations in these areas to assess strategies and goals, develop performance measures and improve operations.
Policies and Operations. A distinguishing feature of the NIH Evaluation Program is the utilization of a variety of evaluation strategies that include the use of national advisory councils, boards of scientific counselors, consensus development conferences, and ad hoc committees that help to chart scientific directions and select the most promising research to support.
A two-tier system is used to review project requests that will use 1-percent evaluation set-aside funding. The first tier involves a review and recommendations by the NIH Technical Merit Review Committee (TMRC) on the technical aspects of project proposals and whether a project fits within HHS guidelines for use of the set-aside fund. The second tier involves the NIH Evaluation Policy Oversight Committee, which considers TMRC recommendations, conducts policy level reviews, and makes final funding recommendations to the NIH Director or his designee.
This report, the eleventh in a series that began in 1975, reviews the recent production and current supply of scientists. Unlike earlier studies, it also considers research training mechanisms other than National Research Service Awards (NRSA) training grants and fellowships. A demographic analysis was conducted by the Committee on National Needs for Biomedical and Behavioral Scientists and it conducted a review of such indicators of short-term demand as trends in faculty and industry hiring and perceptions of the job market by recent Ph.D.s. It found that the number of new Ph.D.s awarded annually in the basic biomedical sciences is well above that needed to keep pace with growth in the U.S. economy and that the number of new PH.D.s awarded annually in the behavioral and social sciences is also sufficient and recommended that overall Ph.D. production in these fields should not be increased; they found a decline in the number of M.D.s conducting research while Ph.D.s awarded in clinical science fields have increased at a rate faster than in the biomedical or behavioral sciences. The recommendation to NRSA is to intensify efforts to train and retain physicians until the trend is reversed. There was also a finding for increasing the supply of minority scientists in biomedical and behavioral research and more minority and nonminority investigators turning their attention to disparities in health. The committee recommends improving opportunities for minority studies in secondary schools, an education level outside the scope of the existing NRSA program.
AGENCY SPONSOR: Office of the Director
FEDERAL CONTACT: Dr. Walter Schaffer, 301-435-2770
PIC ID: 4664.1
PERFORMER: National Academy of Sciences, Washington, DC
The Baltimore City Health Department (BCHD) established a Comprehensive School Health Services Program (CSHSP) in collaboration with the Baltimore City Public Schools. The goal was to create a single comprehensive health model for adolescents attending selected secondary schools. This evaluation describes CSHSP success in engaging students and in preventing student child bearing and dropouts. For students who did bear a child, the study describes CSHSP success in promoting adequate prenatal care and favorable birth outcomes. The research applies time series and cohort analyses to measure program coverage and impact in the CSHSP's first ten years, from 1985-86 through 1994-95. Study data were derived by matching existing computer files of school registration data, CSHSP registration and utilization data and birth certificates. The CSHSP was successful in engaging a majority of students in both middle and high schools. During the ten year study period, school birthrates rose then fell back to levels similar to those when the CSHSP was first initiated. The CSHSP was closely linked to prevention of school dropout. Within CSHSP schools, age-for-grade appropriate ninth graders who used the CSHSP were significantly more likely to stay in school. The study demonstrated the feasibility of evaluating the CSHSP through matching and secondary analysis of existing data sources. The CSHSP did not influence school-wide birthrates, but for students who did use the school-based clinics, birthrates during the high school years decreased by 5% and school dropout rates decreased by over a third.
AGENCY SPONSOR: National Institute of Child Health and Human Development
FEDERAL CONTACT: Susan Newcomer, 301-435-6981
PIC ID: 4467
PERFORMER: Johns Hopkins University School of Medicine, Baltimore, MD
The primary goal of the Research Centers in Minority Institutions (RCMI) Program has been to enable predominantly minority institutions to become more competitive in obtaining support for the conduct of biomedical and/or behavioral research relevant to the mission of the U.S. Public Health Service (PHS). To date, the National Center for Research Resources (NCRR) has funded 21 academic institutions. In 1995 NCRR began a two-phase evaluation of the RCMI Program to assess the extent to which the goals and objectives of the Program have been achieved during its first 10 years. The present Phase II study, which was conducted during 1998-1999, incorporated the evaluation design developed in Phase I and focused on the 15 institutions that had received RCMI funding for at least 10 years. As a group, the 15 institutions showed greater improvement in competing for research grants after 10 years of RCMI support than a comparable group of non-RCMI institutions, increasing their average grant funding by 139 percent compared with 56 percent for the comparison institutions. Also, for the group as a whole, correlation analysis indicated that scientific leadership, administrative leadership, and good management and communications systems were the factors most highly related to overall success. Recommendations were provided by the expert advisory panel, which consisted of six distinguished scientists. They recommended NCRR consider converting the RCMI grants to cooperative agreements, thus ensuring that the centers receive the technical assistance they need in order to achieve success.
AGENCY SPONSOR: National Center for Research Resources
FEDERAL CONTACT: Patricia A. Vorndran, 301-435-0866
PIC ID: 5580.1
PERFORMER: National Institutes of Health, Bethesda, MD
The National Center for Research Resources (NCRR) contracted with the Center for Health Policy Studies (CHPS) Consulting to conduct a midcourse assessment of the Research infrastructure in Minority Institutions (RIMI) grant program. In 1996, seven institutions were awarded RIMI funding to build their research capacity over a five year period. This report documents the assessment of the RIMI Program, which was performed at the end of the third year of funding. It was found that all seven of the RIMI institutions increased their capacity to conduct biomedical and behavioral research, as evidenced by numbers of research faculty, facility improvements and research activities. To varying extents, all schools have active collaboration with research intensive institutions; and schools are better prepared to compete for future research funding than they were three years ago. However, there is not enough evidence to definitively indicate whether schools are developing a critical mass of faculty investigators and research activity that will be self-sustaining beyond RIMI grant funding. They are on the track of doing so, but it is not clear that five years of RIMI funding will be enough for schools to stand on their own without it. Recommendations include decrease teaching loads for faculty members so they can be allowed time to conduct research; facilitate faculty interaction with other scientists; resolve procurement difficulties; and provide adequate staffing in laboratories.
AGENCY SPONSOR: National Center for Research Resources
FEDERAL CONTACT: Patricia A. Vorndran, 301-435-0866
PIC ID: 5580
PERFORMER: Center for Health Policy Studies, Columbia, MD
The major message of this Surgeon General's report is that oral health is essential to the general health and well-being of all Americans and can be achieved by all Americans. Yet, not all Americans are achieving the same degree of oral health. While the Surgeon General's report is a comprehensive review of oral health as a public health issue, it is also in part a systematic scientific review of oral health research literature and an assessment of future research questions and priorities. Based on the oral health research review, the need for behavioral and clinical research, clinical trials, health services research, and community-based demonstration research is focused in several key areas. Development of risk assessment procedures for individuals and communities and of diagnostic markers to indicate whether an individual is more or less susceptible to a given oral disease can provide the basis for formulating risk profiles and tailoring treatment and program options accordingly. Progress is needed for a better understanding of the etiology and distribution of oral disease. The report states that epidemiologic and surveillance databases for oral health and disease, health services, utilization of care, and expenditures are limited or lacking at the national, state, and local levels. Such data are essential in conducting health services research, generating research hypotheses, planning and evaluating programs, and identifying emerging public health problems. Future data collection must address differences among the sub- populations making up racial and ethnic groups. More attention should be paid to demographic variables such as age, sex, sexual orientation, and socioeconomic factors in determining health status.
AGENCY SPONSOR: National Institute of Dental and Craniofacial Research
FEDERAL CONTACT: Dr. Dushanka Kleinman, 301-496-7716
PIC ID: 6861
PERFORMER: National Institutes of Health, Bethesda, MD
The Cost of Cancer Trials Study (CCTS) is a study of cancer patients throughout the U.S. being conducted by RAND, a private, non-profit research institution based in Santa Monica, California, with principal funding and scientific guidance from the National Cancer Institute (NCI). This study will estimate the incremental costs of medical treatment provided as part of NCI-sponsored protocols. Incremental costs refers to the costs of additional medical resources, if any, provided to patients on protocols above and beyond those that would have been received in the absence of trial participation. The results from this study should be of interest to policymakers, insurers and healthcare decision makers trying to determine appropriate reimbursement for clinical trials. As secondary endpoints, patient satisfaction and health outcomes of patients in trials will be compared with those not in trials. The study will use a multi-stage study design to select 1,600 patients at 50-60 study sites among all of the institutions, clinics or practices in the U.S. that are participating in NCI-sponsored Phase II or III clinical trials. The patient sample will be obtained by randomly selecting patients who have enrolled in a clinical trial during a specified period of time at these study sites. Using cancer registries and chart reviews, a matched control group of cancer patients who did not enroll in a clinical trial will be sampled and followed. Using economic models of costs, the two groups will be compared to estimate the cost of trial participation.
EXPECTED DATE OF COMPLETION:12/28/2001
AGENCY SPONSOR: National Cancer Institute
FEDERAL CONTACT: Mary S. McCabe, 301-496-6404
PIC ID: 7116
PERFORMER: Rand Corporation, Santa Monica, CA
This project is an outgrowth of a comprehensive review of NCI's clinical trials program for cancer treatment. Based on this review, NCI has decided to restructure the way in which the Cooperative Groups, the largest single trials program sponsored by NCI, carry out large, randomized clinical trials. The restructuring will occur in a set-wise manner by focusing on a series of demonstration projects, all of which depend heavily on the use of automated applications distributed via the Internet. The evaluation project will carefully compare the new Internet-based methods with the previous methods using descriptive and inferential analyses.
EXPECTED DATE OF COMPLETION:09/30/2001
AGENCY SPONSOR: National Cancer Institute
FEDERAL CONTACT: Jeffrey Abrams, 301-496-2522
PIC ID: 7482
PERFORMER: Research Triangle Institute, Research Triangle Park, NC
This study is part of a long-term evaluation effort directed at examining the extent to which the objectives of the National Institutes of Health (NIH) National Research Service Award (NRSA) pre-doctoral and post- doctoral training programs are being achieved. Using extant data, the focus of this project is to develop a profile of NRSA post-doctoral award recipients and determine what portion of former NRSA post-doctoral trainees and fellows successfully pursue health-related research or training careers.
EXPECTED DATE OF COMPLETION:12/03/2001
AGENCY SPONSOR: Office of the Director
FEDERAL CONTACT: Dr. Walter Schaffer, 301-435-2770
PIC ID: 6285
PERFORMER: Vanderbilt University, Institute for Public Policy Studies, Nashville, TN
The NIH will engage the services of an expert organization to develop criteria, conduct evaluations and provide program development support. The Secretary, DHHS, and the director, NIH, will jointly appoint a panel of distinguished scientists and administrators to review the contractor evaluations and the impact of these delegations on the NIH research programs.
EXPECTED DATE OF COMPLETION:10/30/2001
AGENCY SPONSOR: Office of the Director
FEDERAL CONTACT: Stephen C. Benowitz, 301-496-3592
PIC ID: 6862
PERFORMER: National Academy of Public Administration, Washington, DC
The Regional Primate Research Center (RPRC) program represents a commitment by the National Institutes of Health (NIH) to the development and support of regional and National non-human primate research resources for biomedical and behavioral studies. This evaluation is intended to determine the optimum configuration of resources for non-human primate research programs.
EXPECTED DATE OF COMPLETION:09/30/2001
AGENCY SPONSOR: National Center for Research Resources
FEDERAL CONTACT: Barbara Perrone, 301-435-0871
PIC ID: 6045.1
PERFORMER: James Bell Associates, Inc., Arlington, VA
The Early Childhood Longitudinal Study, Year 2000 Birth Cohort (ECLS-B) is sponsored by the U.S. Department of Education, National Center for Education Statistics, in collaboration with the National Center for Health Statistics, the National Institutes of Health, the Administration on Children, Youth & Families, the Department of Agriculture, and other Federal agencies. It is designed to provide detailed information about children's early life experiences, including children's health, development and care, and education from birth through first grade. Beginning in September 2000, over 12,000 nine-month-olds nationwide will be recruited into the ECLS-B. At this time, parents will be interviewed and infants will be observed to learn about their health and development. The study will follow the children and their families for at least six years, by which time most of the children will have reached first grade. Evaluation funds will help shape the ECLS-B into an important health planning, evaluation and research tool. Specifically, the funds will be used to supplement questions and assessments, including costs associated with question development, assessment of data collection methodologies, and field testing. Funds have been committed to support an oversample of twins, an oversample of very low birth weight (VLBW) infants, a self- administered questionnaire for resident fathers, and a self-administered pilot questionnaire for non- resident fathers who have some contact with the child. Data from the ECLS-B are expected to inform many government analyses and policy reviews, including program evaluation and research efforts to focus on infants, and variables that mediate the impact of service or intervention programs, or otherwise influence children's health in this country.
EXPECTED DATE OF COMPLETION:04/30/2002
AGENCY SPONSOR: National Institute of Child Health and Human Development
FEDERAL CONTACT: Rose Maria Li, 301-496-1175
PIC ID: 7115
PERFORMER: Department of Education, National Center for Educ Statistics, Washington, DC
Since 1986 and every two years subsequently, the National Science Foundation (NSF) and the National Institutes of Health (NIH) have conducted the Survey of Scientific and Engineering Research Facilities. The availability and condition of biomedical research space directly affects the scope and quality of the biomedical research conducted at the Nation's colleges, universities, medical schools, hospitals and other research organizations. To address the need for information on the amount and quality of S&E research space, Congress mandated that the NSF gather this information and report it to Congress.
EXPECTED DATE OF COMPLETION:09/03/2001
AGENCY SPONSOR: National Center for Research Resources
FEDERAL CONTACT: Fred W. Taylor, 301-435-0766
PIC ID: 6863
PERFORMER: National Science Foundation, Arlington, VA