|
|
Office of the Assistant Secretary for Planning and Evaluation |
Policy Information Center |
MISSION: To sponsor and conduct medical research that leads to better health for all Americans.
Evaluation Program
The National Institutes of Health (NIH) generates scientific knowledge that leads to improved health. This is done by conducting medical research in its intramural laboratories and by supporting research in universities, medical and health professional schools, and other health research organizations. NIH fosters the widespread dissemination of the results of medical research, facilitates the training of research investigators, and ensures the viability of the research infrastructure. The NIH Evaluation Program is an integral part of how NIH sponsors and conducts medical research.
Results based management is recognized as a basic principle for the sound and productive operation of government agencies and their programs. This is evidenced most notably by passage of the Government Performance and Results Act (GPRA) and by the considerable effort across the federal government to implement results based management mechanisms. With GPRA and other initiatives aimed at increasing public sector accountability (such as the Chief Financial Officers Act and the Government Management Reform Act), interest in the use of evaluation has increased steadily among NIH administrators and others, such as officials within the General Accounting Office, the Office of Management and Budget and the Department of Health and Human Services.
Philosophy and Priorities. The NIH Evaluation Program provides information to assist the NIH Director and the NIH Institute and Center (IC) Directors in determining whether NIH goals and objectives are being achieved and to help guide policy development and program direction. Evaluations are planned and conducted from two sources of funds: 1 percent evaluation set-aside funds used to fund trans-NIH projects, and IC program funds used for program evaluations for use by various committees, working groups, task forces, workshops, conferences, and symposia to assist the ICs in program management and development. This approach ensures that planning and priority setting specific to the mission of each IC are fully developed and implemented and that there is central leadership for developing crosscutting initiatives and promoting collaboration among the ICs.
NIH's major evaluation priority areas fall within three broad program areas: basic research, research training and career development, and facilities. NIH conducts evaluations in these areas to assess strategies and goals, develop performance measures and improve operations.
Policies and Operations. A distinguishing feature of the NIH Evaluation Program is the utilization of a variety of evaluation strategies that include the use of national advisory councils, boards of scientific counselors, consensus development conferences, and ad hoc committees that help to chart scientific directions and select the most promising research to support.
A two-tier system is used to review project requests that will use 1-percent evaluation set-aside funding. The first tier involves a review and recommendations by the NIH Technical Merit Review Committee (TMRC) on the technical aspects of project proposals and whether a project fits within HHS guidelines for use of the set-aside fund. The second tier involves the NIH Evaluation Policy Oversight Committee (EPOC), which considers TMRC recommendations, conducts policy level reviews, and makes final funding recommendations to the NIH Director or his designee.
This report was prepared by the Office of Extramural Research at the National Institutes of Health (NIH) and depicts a stratified random sample of 2,694 individuals who had applied to the NIH for an R01 or R29 grant in 1994. These applicants were surveyed to assess their satisfaction with the NIH's grant application and review process. Over 78 percent of respondents held PhDs or other research doctorates, the majority of which were in the biomedical sciences. Ninety-three percent of the respondents were working full-time in academic institutions, and 54 percent had received some type of NIH research funding by June 1997. Based on their experiences in applying for NIH grants since 1994, about 41 percent of the respondents were satisfied with the handling of their application to the NIH, 35 percent expressed mixed feelings, and 24 percent were dissatisfied. Not surprisingly, unfunded applicants showed greater dissatisfaction with the overall application and review process than funded applicants. The appropriateness of the scientific review group yielded the highest percentage of satisfied responses (47 percent) and timely receipt of the 'pink sheet' and notification of the NIH's funding decision elicited the strongest reactions from applicants. Approximately 54 percent of the applicants offered observations and suggestions for improvement that included: (1) modification of forms and submission practices; (2) increased use of electronic submissions and availability of forms in word processing formats; (3) shortened length of the application and reduction of the amount of information required; (4) use of more expert reviewers; (5) reduction of the length of time between application submission and the receipt of funding.
AGENCY SPONSOR: Office of the Director
FEDERAL CONTACT: Schaffer, Dr. Walter
301-435-2770
PIC ID: 6280
PERFORMER: Macro International, Inc.
Calverton, MD
This project originally involved the design and pilot-test of a process for developing an inventory of cancer-related research activities across federal agencies. The goal was to assess both the feasibility and usefulness of developing the inventory. Anticipated benefits were the possibility for improved utilization and coordination of resources and the means to identify opportunities for collaborations and promising areas of research. In the course of assessing the availability of existing database information systems for cancer research, the project learned of a database under development for the federal Office of Science and Technology Policy. Known as RaDiUS, this data system is intended to be a comprehensive source of basic information on all research and development supported by the U.S. government. The project shifted direction to evaluate RaDiUS in terms of its success in its ability to identify a comprehensive set of relevant project records within specific cancer research areas (lung cancer, cancer and genetics). The accuracy and completeness of data supplied by RaDiUS within identified project records was also evaluated. RaDiUS data were compared to data acquired directly from several agencies to evaluate accuracy and comprehensiveness. Overall, findings of the research indicate that RaDiUS is currently the most comprehensive database available that lists U.S. Federal R&D efforts, and is a potentially useful tool in identifying cancer research across the Federal government.
AGENCY SPONSOR: National Cancer Institute
FEDERAL CONTACT: Middleswarth, Anne
301-496-5515
PIC ID: 6091
PERFORMER: NOVA Research Company
Bethesda MD
This report extends the analyses of previous reports by examining the changes that have occurred over the last 30 years in graduate and postgraduate training of life scientists and the nature of their employment on completion of training. It also suggests reasons for the decrease in the number of young scientists applying for NIH grants. Findings indicate that the training and career prospects of a graduate student or postdoctoral fellow in the life sciences in 1998 are very different from what they were in the 1960s or 1970s. Today's life scientist will start graduate school when slightly older and take more than two years longer to obtain the PhD degree. Today's life science PhD recipient will be an average of 32 years old. Furthermore, the new PhD today is twice as likely as in earlier years to take a postdoctoral fellowship and thus join an ever-growing pool of postdoctoral fellows--now estimated to number about 20,000--who engage in research while obtaining further training and waiting to obtain permanent positions. It is not unusual for a trainee to spend five years--some more than five years--as a postdoctoral fellow. As a consequence of that long preparation, the average life scientist is likely to be 35-40 years old before obtaining his or her first permanent job. The median age of a tenured or tenure track faculty member is now about eight years more than that of the faculty member of the 1970s.
AGENCY SPONSOR: Office of the Director
FEDERAL CONTACT: Schaffer, Dr. Walter
301-435-2770
PIC ID: 6098
PERFORMER: National Academy of Sciences, National Research
Council
Washington, DC
The Regional Primate Research Center (RPRC) program represents a commitment by the National Institutes of Health (NIH) to the development and support of regional and National non-human primate research resources for biomedical and behavioral studies. This evaluation is intended to determine the optimum configuration of resources for non-human primate research programs.
AGENCY SPONSOR: National Center for Research Resources
FEDERAL CONTACT: Perrone, Barbara
301-435-0871
PIC ID: 6045.1
PERFORMER: James Bell Associates, Inc.
Arlington, VA
EXPECTED DATE OF COMPLETION: 12/20/99
Since 1986 and every two years subsequently, the National Science Foundation (NSF) and the National Institutes of Health (NIH) have conducted the Survey of Scientific and Engineering Research Facilities. The availability and condition of biomedical research space directly affects the scope and quality of the biomedical research conducted at the Nation's colleges, universities, medical schools, hospitals and other research organizations. To address the need for information on the amount and quality of S&E research space, Congress mandated that the NSF gather this information and report it to Congress.
AGENCY SPONSOR: National Center for Research Resources
FEDERAL CONTACT: Taylor, Fred W.
301-435-0766
PIC ID: 6863
PERFORMER: National Science Foundation
Arlington, VA
EXPECTED DATE OF COMPLETION: 12/20/99
The Cost of Cancer Trials Study (CCTS) is a study of cancer patients throughout the U.S. being conducted by RAND, a private, non-profit research institution based in Santa Monica, California, with principal funding and scientific guidance from the National Cancer Institute. This study will estimate the incremental costs of medical treatment provided as part of NCI- sponsored protocols. Incremental costs refers to the costs of additional medical resources, if any, provided to patients on protocols above and beyond those that would have been received in the absence of trial participation. The results from this study should be of interest to policymakers, insurers and healthcare decision makers trying to determine appropriate reimbursement for clinical trials. As secondary endpoints, patient satisfaction and health outcomes of patients in trials will be compared with those not in trials. The study will use a multi-stage study design to select 1,600 patients at 50-60 study sites among all of the institutions, clinics or practices in the U.S. that are participating in NCI-sponsored Phase II or III clinical trials. The patient sample will be obtained by randomly selecting patients who have enrolled in a clinical trial during a specified period of time at these study sites. Using cancer registries and chart reviews, a matched control group of cancer patients who did not enroll in a clinical trial will be sampled and followed. Using economic models of costs, the two groups will be compared to estimate the cost of trial participation.
AGENCY SPONSOR: National Cancer Institute
FEDERAL CONTACT: McCabe, Mary S.
301-496-6404
PIC ID: 7116
PERFORMER: Rand Corporation
Santa Monica, CA
EXPECTED DATE OF COMPLETION: 12/28/01
The Early Childhood Longitudinal Study, Year 2000 Birth Cohort (ECLS-B) is sponsored by the U.S. Department of Education, National Center for Education Statistics, in collaboration with the National Center for Health Statistics, the National Institutes of Health, the Administration on Children, Youth & Families, the Department of Agriculture, and other Federal agencies. It is designed to provide detailed information about children's early life experiences, including children's health, development and care, and education from birth through first grade. Beginning in September 2000, over 12,000 nine- month-olds nationwide will be recruited into the ECLS-B. At this time, parents will be interviewed and infants will be observed to learn about their health and development. The study will follow the children and their families for at least six years, by which time most of the children will have reached first grade. Evaluation funds will help shape the ECLS-B into an important health planning, evaluation and research tool. Specifically, the funds will be used to supplement questions and assessments, including costs associated with question development, assessment of data collection methodologies, and field testing. Funds have been committed to support an oversample of twins, an oversample of very low birth weight (VLBW) infants, a self-administered questionnaire for resident fathers, and a self-administered pilot questionnaire for non-resident fathers who have some contact with the child. Data from the ECLS-B are expected to inform many government analyses and policy reviews, including program evaluation and research efforts to focus on infants, and variables that mediate the impact of service or intervention programs, or otherwise influence children's health in this country.
AGENCY SPONSOR: National Institute of Child Health and Human Development
FEDERAL CONTACT: Li, Rose Maria, Ph.D.
301-496-1175
PIC ID: 7115
PERFORMER: Department of Education, National Center for Educ
Statistics
Washington, DC
EXPECTED DATE OF COMPLETION: 4/30/02
This project developed a study design for an evaluation of the National Institute of Dental Research (NIDR) center program.
AGENCY SPONSOR: National Institute of Dental and Craniofacial Research
FEDERAL CONTACT: Corrigan, Dr. James G.
301-496-6705
PIC ID: 5578
PERFORMER: National Institute of Dental Research
Bethesda, MD
EXPECTED DATE OF COMPLETION: 12/20/99
For over three decades, the Demographic and Behavioral Sciences Branch (DBSB) has supported the Population Centers Program (PCP), which has provided infrastructure support for population research. Currently, 12 centers are supported nationwide. While the PCP has been critical in supporting the development of population research over these 30 years, it is clear that the needs and institutional context of the discipline have changed significantly over time. In response, the DBSB is evaluating how the PCP is meeting the needs of population researchers today, and whether there are different ways of structuring and competing the program to better serve the future of the science. The DBSB is recruiting a group of five eminent scientists to serve as consultants, and prepare background and summary papers, as well as recommendations, to assist the Branch in conducting the evaluation. Using a variety of data, these scientists will perform a needs assessment, evaluate the current status of the PCP, and prepare papers that summarize their findings and suggestions for modifying the program. The DBSB will convene two or three meetings over the course of a year to monitor progress and assess the need for additional data. The DBSB will also design a procedure for inviting public comment from those who want to express their views on the Center's program, and these comments will be shared with the consultants. Based on the recommendations made by individual consultants, and discussions at meetings of the consultants and other scientists, the DBSB will work with Institute staff to translate the suggestions into appropriate NICHD mechanisms and policies.
AGENCY SPONSOR: National Institute of Child Health and Human Development
FEDERAL CONTACT: Bachrach, Christine A., Ph.D.
301-496-1174
PIC ID: 7119
PERFORMER: National Institute of Child Health and Human
Development
Bethesda, MD
EXPECTED DATE OF COMPLETION: 7/1/00
The Surgeon General's (SG's) report will provide an opportunity to describe oral health and provide the backdrop to present the case for the critical inter-relationship between oral health and general health.
AGENCY SPONSOR: National Institute of Dental and Craniofacial Research
FEDERAL CONTACT: Kleinman, Dr. Dushanka
301-496-7716
PIC ID: 6861
PERFORMER: National Institutes of Health
Bethesda, MD
EXPECTED DATE OF COMPLETION: 1/20/00
This evaluation will undertake a detailed assessment of vaccine research and development needs and priorities for the upcoming decade. The committee will then apply the model to general priorities for domestic vaccine development for the next ten to twenty years.
AGENCY SPONSOR: National Institute of Allergy and Infectious Diseases
FEDERAL CONTACT: Rabinovich, D. Regina, M.D.
301-402-2126
PIC ID: 5984
PERFORMER: National Academy of Sciences
Washington, DC
EXPECTED DATE OF COMPLETION: 12/31/99
This study is part of a long-term evaluation effort directed at examining the extent to which the objectives of the National Institutes of Health (NIH) National Research Service Award (NRSA) pre-doctoral and post-doctoral training programs are being achieved. Using extant data, the focus of this project is to develop a profile of NRSA post-doctoral award recipients and determine what portion of former NRSA post-doctoral trainees and fellows successfully pursue health-related research or training careers.
AGENCY SPONSOR: Office of the Director
FEDERAL CONTACT: Schaffer, Dr. Walter
301-435-2770
PIC ID: 6285
PERFORMER: Vanderbilt University, Institute for Public Policy
Studies
Nashville, TN
EXPECTED DATE OF COMPLETION: 12/20/99
The NIH will engage the services of an expert organization to develop criteria, conduct evaluations and provide program development support. The Secretary, DHHS, and the director, NIH, will jointly appoint a panel of distinguished scientists and administrators to review the contractor evaluations and the impact of these delegations on the NIH research programs.
AGENCY SPONSOR: Office of the Director
FEDERAL CONTACT: Benowitz, Stephen C.
301-496-3592
PIC ID: 6862
PERFORMER: National Academy of Public
Administration
Washington, DC
EXPECTED DATE OF COMPLETION: 10/30/01
To address research training needs related to the fields of mental health, neuroscience, alcohol abuse, and drug abuse, the Alcohol, Drug Abuse, and Mental Health Administration established a National Research Service Award (NRSA) individual fellowship mechanism in 1989 for training physicians and scientists. After rejoining the National Institutes of Health (NIH) in 1993, the NIMH, along with the National Institute on Alcohol Abuse and Alcoholism (NIAAA) and the National Institute on Drug Abuse (NIDA), elected to continue this mechanism. The objectives of this fellowship program are to increase the numbers of physicians and scientists trained in, and conducting, biomedical and behavioral research in areas related to mental illness and addictions. Because this fellowship mechanism is young, and most participants are still in training, this is an optimal time to establish a tracking system which builds a framework for measurement of program performance and future periodic assessments of the program. The objectives of this project are to assess the impact of this support mechanism on: (1) changes in the numbers of M.D.s, and Ph.D.s conducting clinical and basic research in neuroscience, mental health, and behavioral sciences, and mental health services research; and (2) the specific areas of science in which fellowship graduates conduct research.
AGENCY SPONSOR: National Institute of Mental Health
FEDERAL CONTACT: Babich, Karen
301-443-4335
PIC ID: 6096
PERFORMER: National Institute of Mental Health
Rockville, MD
EXPECTED DATE OF COMPLETION: 9/30/00
The National Academy of Sciences (NAS) is establishing a study committee which will direct activities related to the development of estimates of future needs for biomedical and behavioral research personnel. Using estimates of future needs, the committee will make recommendations concerning the size and composition of the National Research Service Awards (NRSA) research training programs.
AGENCY SPONSOR: Office of the Director
FEDERAL CONTACT: Schaffer, Dr. Walter
301-435-2770
PIC ID: 4664.1
PERFORMER: National Academy of Sciences
Washington, DC
EXPECTED DATE OF COMPLETION: 1/20/00