Appendix C

HHS Evaluation Review Panel

The following individuals served on the evaluation review panel that made recommendations on the reports highlighted in Chapter I of the Performance Improvement 2002: Evaluation Activities of the U.S. Department of Health and Human Services:

Douglas Barnett, Ph.D.
Department of Psychology
Wayne State University

Heather Becker, Ph.D.
University of Texas-Austin
School of Nursing
Austin, TX

Leslie J. Cooksy, Ph.D.
Center for Community Development & Family Policy
College of Human Services, Education, and Public Policy
University of Delaware
Newark, DE

Holly Korda, Ph.D.
Consulting Principal
Health Systems Research Associates
Chevy Chase, MD

John Kralewski, Ph.D.
University of Minnesota
School of Public Health
Minneapolis, MN

Kenneth McLeroy, Ph.D.
Texas A & M
Associate Dean for Academic Affairs
Bryan, TX

Marv Mandell, Ph.D.
Policy Sciences Graduate Program
University of Maryland, Baltimore County
Baltimore, MD

Melvin Mark, Ph.D.
Professor of Psychology
Penn State University
University Park, PA

Donna Mertens, Ph.D.
Gallaudet University
Washington, DC

Joy Quill
C. J. Quill & Associates, Inc.
Annapolis, MD

Contractors for Review Panel Process:

Cheryl J. McDonnell, Ph.D.
James Bell Associates
Arlington, VA 22201

Norma I. Gavin, Ph.D.
Research Triangle Institute
Research Triangle Park, NC

Project Manager: Vijaya ChannahSorah, Ph.D.
Office of the Assistant Secretary for Planning and Evaluation, HHS

Project Assistant: Elsie Quinones
Executive Potential Program
On Rotation with the Assistant Secretary for Planning and Evaluation, HHS

Program Evaluation Review Criteria

The following criteria were used by the HHS Evaluation Review Panel to select evaluation reports highlighted in Chapter I of Performance Improvement 2001:

Overall Significance

• The study addresses a significant issue of policy relevance.
• Evaluation findings are likely to be useful.

Conceptual Criteria

Conceptual Foundations 

• A literature review is included.
• The project is shown to be logically, based on previous findings; the report uses either theory, or models, or both.
• The program assumptions are stated.
• The evaluation draws from any previous evaluation.
• The report is linked with a program and describes the program.
• The report presents multiple perspectives.
• Multiple relevant stakeholders are consulted and involved.
• The timing is appropriate because the program is ready for evaluation.

Questions for Evaluation

• The aims of the evaluation are clear, well-specified, and testable.

• The questions are feasible, significant, linked to the program, appropriate for the  resources and audience, and derive logically from the conceptual foundations.

• The questions show ingenuity and creativity.

  Findings and Interpretation

  • The conclusions are justified by the analyses.
  • The summary does not go beyond what the data will support.
  • The appropriate qualifiers are stated.
  • The conclusions fit the entire analysis.
  • Equivocal findings are handled appropriately.
  • The initial questions are answered.
  • The interpretation ties in with the conceptual foundation.
  • The report notes that the findings are either consistent with or deviate from the relevant literature.
  • The presentation is understandable.
  • The results have practical significance.
  • The extent of program implication is assessed.

  Recommendations

  • The recommendations follow from findings, are worth carrying out, and are affordable, timely, feasible, useful, and appropriate.
  • The recommendations are shown to be relevant to the questions asked.
  • The breadth of specificity of the recommendations is addressed.
  • Any recommendations for either future evaluations, or improvements, or both are clearly presented.

  Methods

  Evaluation Design

• Design considerations include overall appropriateness, soundness, feasibility, funding and time constraints, generalizability, applicability for cultural diversity, assessment of the extent of program delivery, validity, feasibility for data collection, reliability of selected measurements, use of multiple measures of key concepts, and appropriateness of the sample.
• Variables are clearly specified and fit with the questions and concepts.

Data Collection

• Data are collected using appropriate units of measurement for analysis, controls for participant selection and assignment bias, and proper handling of missing data and attrition.
• Data collection is characterized by use of an appropriate comparison group of control; adequate sample size, response rate, and information about the sample; a data collection plan; data collection that is faithful to the plan; attention to and cooperation with the relevant community; project confidentiality; and consistency.
• The quality of the data (including the quality of any extant data sets used in the study) and the efficiency of sampling are addressed.
• The data collection is appropriate to evaluation questions.

Data Analysis

• The data analysis addresses the handling of attrition, the matching of the analysis to the design, the use of appropriate statistical controls, the use of methodology and levels of measurement appropriate to the type of data, and estimation of effect size.
• The analysis shows sensitivity to cultural categories.
• The analysis makes appropriate generalizability of inferences.
• The chosen analysis type is simple and efficient.

Cross-Cutting Factors

Cross-cutting factors that are likely to be important at all stages of a report include: clarity, presentation, operation at a state-of-the-art level, appropriateness, understandability, innovation, generalizability, efficiency of approach, logical relationships, and discussion of the report's limitations. The report should also address ethical issues, possible perceptual bias, cultural diversity, and any gaps in study execution.