Performance Improvement 1997
Chapter II
Highlights of Selected Evaluations Completed During Fiscal Year 1996
In this chapter, the U.S. Department of Health and Human Services (HHS) highlights evaluations of
general interest to the public health and human services community and illustrates the diversity of HHS
evaluations completed in fiscal year (FY) 1996. Included are summaries of 11 evaluation projects selected
by the Senior Editorial Advisors on the basis of criteria identified in appendix C and applied to 34 reports
nominated by HHS agencies. These criteria are as follows:
- Is the report important? Does it address a significant issue or problem for which evaluation would
help confirm or change program direction? Does it measure program impact? Are the findings likely to
be useful and generalizable?
- Is the report methodologically sound? Are its concepts, designs, data collection, and analyses
conducted and reported in a competent manner?
- Is the report faithful to the data? Do the conclusions and recommendations logically follow from
the data and analyses, and are they relevant to the questions asked?
Each summary includes a brief abstract; a description of the study, including its purpose, background,
methods, findings, and use of results; the names of any publications that resulted; and the name and
phone number of the person to contact for additional information.
Highlights
This study evaluated the results of eight Child Access Demonstration Projects from two waves of
demonstrations testing the effectiveness of mediation, counseling, education, and visitation monitoring
programs designed to facilitate noncustodial parents' access to their children following divorce and
separation. The incidence of access problems ranged from 31 percent to 4 percent of cases. The nature
of access disputes varied widely; such disputes were reported by both the custodial and noncustodial
parents. Making both parents attend mediation sessions is seen as critical and difficult. Where both parties
attended, mediation visitation increased, relitigation was low, and there was increased child support
compliance for the experimental vis-a-vis the control groups. Other interventions (e.g., parenting classes,
counseling) which were done for severely conflicted parties or at a distance from the divorce/separation
were not seen as making an additional impact in these areas. Also, the timing of access disputes was
unaffected by the demonstrations.
Purpose
As set forth in the Family Support Act of 1988, this evaluation explored the effect of two waves of Child
Access Demonstration projects on the amount of time required to resolve access disputes; reductions in
litigation related to access disputes; improvements in compliance with court-ordered child support
amounts; and promotion of the emotional adjustment of children. It also assessed the extent and nature
of child access disputes as well as parental satisfaction with the demonstrations.
Background
Recent research in child psychology shows generally that close, frequent, and positive contact with the
father following divorce and separation is beneficial for the child.
Child access is also important for child support enforcement. Recent Census data and research studies
have indicated that where noncustodial parents have visitation rights or joint custody they tend to be more
compliant with child support orders, although it is difficult to show cause and effect since the parents
wanting to see the child may also be the better payers. Desire for increased child contact may follow child
support payment rather than vice versa. Moreover, denial of visitation is seen as the major reason for
nonpayment of child support for noncustodial parents who have money to pay child support.
There has been considerable pressure for the system to give support to the needs of noncustodial as well
as custodial parents. Over 43 States authorize joint custody. There are currently over 200 court-based
divorce mediation programs and over 280 fathers' rights groups organized throughout the country to
facilitate child access by noncustodial parents.
Congress responded to the continuing public debate about the problem of noninvolvement by noncustodial
parents and resulting litigation by directing HHS to conduct State demonstration projects relating to a
variety of means of facilitating continuing involvement by the noncustodial parent.
In 1996 a new Federal grant program for child access and visitation programs was established nationwide.
Methods
Projects involved control and experimental group testing of different interventions (e.g., mediation,
parenting education, community services for the noncustodial parent, group counseling, telephone
monitoring of visitation) in a variety of public and private, court and administrative settings in Idaho,
Florida, Indiana, Massachusetts, Arizona, and Iowa. Evaluation was provided for randomly selected
experimental and control groups on such issues as relitigation, relitigation timing, visitation, and child
support compliance. There were over 2,400 cases in the baseline and 1,500 cases in follow-up interviews.
Findings
The incidence of child access problems for divorced and separated parents ranged from a low of 4 percent
to almost 30 percent of the cases by site. This is consistent with findings of other studies. The nature of
child access problems are varied and may be longstanding for both parents. Some frequent problems
include insufficient amount of visitation time, being cut out of a child's life, scheduling visitation, fights
during pick-up and drop-off, concerns for the child's safety, concerns about the other parent's parenting
style and negativity, and continuing problems with the parents' relationship.
There is a big problem of nonattendance by one or the other spouse at mediation or related interventions;
and some effort should be made to compel attendance. However, where both parents attend mediation
there is a high (65 percent to 70 percent) rate of arriving at parenting plans.
Mediation did resolve conflict in many, but not all, cases, and problems decreased after interventions.
Visitation days increased for most sites and child support compliance increased to over 20 percent. A
majority of both parents indicate satisfaction with interventions. However, the speed of dispute resolution
and incidence of relitigation were not affected. In addition, nonmediation experiments did not seem to
have an impact. It was difficult to assess improvements in emotional child development where tested.
Use of Results
The results should assist State, local, and private agencies in establishing programs to improve the
continued involvement by the noncustodial parent. Mediation interventions at the time of the
divorce--when both parents can be made to attend--are most effective. When disputes are high charged
or considerable time has elapsed, it is more difficult to make an impact. Scarce resources can be focused
on cases with problems before they become intractable. The results of these demonstrations are more
relevant for divorce and formal separation cases than for unwed situations.
Agency sponsor:
Office of Child Support Enforcement
Federal contact:
David Arnaudo
202/401-5364
PIC ID: 5972.2
Performer organization:
Policy Studies Inc., Denver, CO
Highlights
This study evaluated the implementation of State child support guidelines mandated by the Family
Support Act of 1988. It concluded that States need to increase the consistency of guideline application,
including income verification procedures, extended custody and visitation arrangements, and consideration
of multiple family situations, health insurance, and day care expenses in the calculation of child support
awards. Adoption of a standardized support order would improve documentation of departures from
guideline award calculations. State reviews of guidelines could be more thorough if carried out with
sufficient resources, case surveys, and analysis of the cost of raising children. The study did not
recommend the adoption of a national child support guideline.
Purpose
This evaluation explored three issues: how State child support guidelines are applied; the extent, amount,
direction, and causes of deviations from State guideline award formulas; and how States account for
special family circumstances in determining child support awards. Research focused on the process of
applying guidelines; the extent, cause, and documentation of deviation from guideline formulas; the
consideration of special family circumstances (such as child day care, health expenses, and multiple
families) in adjusting awards; and actions taken as a result of the mandated guideline reviews prepared
by the States.
Background
In 1984, in order to ensure appropriate and equitable child support awards by courts and other
decisionmakers, Congress required every State seeking Federal funding for public welfare programs to
establish child support guidelines. These initial guidelines were only advisory. However, the Family
Support Act of 1988 requires that the State guidelines be used in determining the proper amount of
support. Deviation from the presumptive guideline amount of award requires written justification.
Several Federal requirements help States develop child support guidelines. These include need for
uniformity within each State; use of numerical formulas that consider all of a payer's income and provide
for health care needs; use of guidelines to determine rebuttable, presumptive award amounts in judicial
or administrative proceedings; use of guidelines for subsequent award modification; application of
guidelines to all cases; and consideration of the child's best interests in State criteria for deviating from
a guideline. The Family Support Act also requires that States reevaluate their guidelines at least once
every 4 years to ensure the appropriateness of child support awards. States are to consider the cost of
raising children and to sample case data on deviations from the guidelines. States must review child
support orders for cases handled by a State child support agency at least once every 3 years.
Methods
This evaluation employed various sources of data--child support case records, stakeholder interviews,
analysis of State guideline review studies, analysis of the Current Population Survey (CPS), and an expert
panel. A sample of case records was drawn from 2 counties in each of 11 States. Although not
representative of all States, over 4,000 cases were selected to reflect diverse guideline model types and
child support formulas. The study conducted 215 unstructured telephone interviews with State and local
officials, parents, and other stakeholders to gain a real-life perspective of the overall operation of State
guidelines. Case studies of child support guidelines in 20 States were analyzed to assess the review
process and the extent of required documentation for deviation from the guideline award amount. Over
2,000 cases in the March-April Matched CPS file with established child support awards in 1992 were
analyzed to determine if presumptive State guidelines affected child support award amounts. Finally, an
expert panel was assembled to provide advice on the evaluation and to make recommendations based on
study findings.
Findings
Analysis of sample case record data indicated that only 17 percent of cases involved deviations from the
guideline award amount, although the percentage varied considerably across the counties of 11 States.
The four most common reasons for deviation were (1) agreement between the parties; (2) second
households or multiple families; (3) extensive or extraordinary visitation or custody expenses; and (4) low
income of the payer. The majority involved downward deviations--the average award decreased 34
percent. The ordered amount of support differed from the calculated guideline amount of support in a
significant percentage of cases. Differences were found both in cases that followed the guideline
calculations and in deviation cases. However, reasons for these differences were not documented in the
case records. The consistency of the guideline application varied considerably. While some factors (such
as income and health care expenses) were considered in nearly all orders, the consideration of other
factors (such as multiple families and child care expenses) was quite variable across counties and within
the same State.
Unstructured interviews with decisionmakers and other stakeholders in 21 study counties found a
consensus that child support issues involving the impact of multiple families and second marriages,
income imputation, and support for postsecondary education required further investigation.
Examination of State guideline reviews indicated that only 20 States collected and analyzed case data.
Fewer than half of the States consider economic data on the cost of raising children. The extent to which
these States took action as result of the data is unclear. When a particular factor, such as multifamily
situations, is mandated by a numerical formula in the guidelines, there generally is a more uniform and
frequent consideration of that factor in child support actions by the award decisionmaker. Verification
of parental income, which is important in calculating accurate child support, is not frequent across
counties; parties are often not prepared with the proper forms at hearings. When health insurance was
ordered for one or both of the parents, the cost was rarely included in the child support calculation in
States that have a mandatory numerical adjustment for such costs. Child day care expenses were included
in the child support calculations for the cases that mentioned these expenses. There was no clear
consensus in State reviews about the provision of postsecondary education support. Tax exemptions are
commonly allocated in a flexible and equitable manner. The States handle extended custody and visitation
arrangements without consistency or equity. These arrangements were commonly used reasons for
deviation in child support awards.
Analysis of CPS data indicated that support awards may have increased by a small amount after the shift
from voluntary to presumptive guidelines in 1988. However, the demographic composition of custodial
parents changed over the period and the income of the noncustodial parent was unavailable, preventing
any definitive assessment of the impact of mandatory guidelines.
The panel of experts reached the following conclusions:
- No steps should be taken at this time to adopt a national child support guideline because States are
still in an experimental phase.
- Presumptive State guidelines should continue, with emphasis on the conduct of required State
guideline reviews every 4 years (in accordance with Federal regulations).
- The Federal Government should provide technical assistance and point out areas in the guideline
review process for States to consider.
Use of Results
The findings and recommendations of this evaluation should benefit those States considering the revision
or improved consistency of guidelines for awarding child support payments. Other States should be able
to use this study to accelerate the implementation of guidelines, periodic case record reviews, and other
related actions required by Federal regulations. Examples of specific State actions that can be taken on
the basis of this study include (1) more consistent application of guidelines across cases; (2) improved
case documentation of departures from the guideline award amount and reasons for the deviations; (3)
improved independent verification of parental income; (4) adoption of a standardized support order to
ensure that factors common to the majority of child support actions are addressed consistently; (5) use
of a mandated numerical formula to adjust for various factors; and (6) coordination and consistency of
policy within and between agencies and courts with regard to procedures and documentation.
Agency sponsor:
Office of Child Support Enforcement
Federal contact:
David Arnaudo
202/401-5364
PIC ID: 5983
Performer organization:
CSR, Incorporated, Washington, DC
Highlights
The Elderly Nutrition Program provides grants to State units on aging to subsidize the provision
of daily meals to people 60 years of age or older. This evaluation study describes participant
characteristics, compares the characteristics of recipients of home-delivered meals to those receiving meals
in congregate settings, assesses how well the program reaches disabled and poor elderly, estimates the
program's impact on the nutritional intake and social contacts of participants, and examines program costs
and other sources of funding. The study showed that participants have higher daily intakes of nutrients
and more social contacts per month than a comparable group of nonparticipants. The Elderly Nutrition
Program is successful in targeting older people who are poor, live alone, are nutritionally "at
risk" because they are overweight or underweight, or are more functionally disabled than their
age-group peers. Federal expenditures are highly leveraged with State, local, and private funds.
Purpose
This study was conducted to inform policymakers about the effectiveness of the Elderly Nutrition
Program (ENP), the largest Federal program aimed at meeting the nutrition service needs of the elderly.
Four major issues were evaluated: (1) the ENP's effects on participants' nutrition and socialization, (2)
who is using the ENP and how effectively the program serves targeted groups most in need of its
services, (3) how efficiently and effectively the ENP is administered and delivers services, and (4) the
range of ENP funding sources and the allocation of ENP funds among its components.
Background
America will face critical challenges in the coming decades as it attempts to provide long-term care
services to the nation's elderly population. As the large group of individuals born after World War II
ages, a much higher proportion of Americans will be elderly and will require more health services and
long-term care. In this context, it is important to examine whether existing long-term care service
programs are producing their intended outcomes and whether their services are directed to those who need
them the most.
One very important component of the overall package of home- and community-based services available
for elderly people is nutrition services. A critical step toward meeting this need was taken in 1972 with
the creation of the Elderly Nutrition Program under Titles III and VI of the Older Americans Act.
Through Title III, State Units on Aging and Area Agencies on Aging implement a system of coordinated,
community-based services targeted to older individuals. The ENP is administered by the Administration
on Aging (AoA) of the U.S. Department of Health and Human Services. The AoA awards funds to State
Units on Aging, which in turn distribute the funds to Area Agencies on Aging on the basis of
State-determined formulas that reflect the proportion of older people in their areas and other factors. The
area agencies provide services directly or award grants to and contract with nutrition projects that provide
nutritional and supportive services. In addition, area agencies receive financial support from State and
local governments, in-kind contributions, private donations, and voluntary contributions from participants.
Title III of the Older Americans Act authorizes the provision of nutrition and supportive services, such
as meals, nutrition education, transportation, personal and homemaker services, and information and
referral. Congregate meals and supportive services are provided at nutrition projects' meal sites (e.g.,
senior centers, religious facilities, schools, public or low-income housing sites, or residential care
facilities). Alternatively, home-delivered meals are provided to homebound clients, either by the
congregate meal sites and affiliated central kitchens or by nonaffiliated food service organizations. ENP
meals are targeted to persons aged 60 or older. While there is no means test for program qualification,
services are targeted at older persons with the greatest economic or social need. Similar nutrition and
supportive services for American Indians, Alaskan Natives and Native Hawaiians are authorized
separately under Title VI. Indian Tribal Organizations and agencies serving Native Hawaiians receive
grant awards directly from the AoA.
Methods
Mathematica Policy Research, Inc., undertook this study for the AoA. A comprehensive research design
was used, including in-person surveys of participants and eligible nonparticipants; in-person surveys of
staff of congregate meal sites; in-person observation of sanitation and health practices at congregate meal
sites, central kitchens, caterers, and home-delivered meal distribution sites; telephone surveys of staff at
State Units on Aging, Area Agencies on Aging, Indian Tribal Organizations, and nutrition projects;
review of program records on amounts of funding transfers; review of scientific literature related to
nutritional requirements for the elderly population; extraction of relevant census data; and extraction of
results from the previous national ENP program evaluation published in 1981.
All State Units on Aging were sampled. The sample of Area Agencies on Aging and nutrition projects
was selected as a geographically clustered sample to maximize efficiency in conducting surveys. For Title
VI programs, the agency sampling was simplified by the fact that each ENP grant is awarded directly to
the Indian Tribal Organization, which in turn consists of a single nutrition project, often operating from
a single site. In all, data were collected at 197 randomly selected meal sites. Approximately 2,500
interviews were conducted with elderly ENP participants (both home-delivered meal recipients and
congregate site meal recipients) and nonparticipants. The nonparticipant sample used for comparison with
the Title III program participant sample was selected from lists of Medicare recipients in the same ZIP
Code areas as the participant sample, screening for income and disability status to ensure that the samples
were matched as closely as possible.
Findings
The evidence collected in this evaluation suggests that the ENP has substantial positive effects on
participants. Title III participants have higher daily intakes of key nutrients and more social contacts per
day than a comparable group of nonparticipants. The average ENP meal meets the program's
requirements to provide at least one-third of the relevant Recommended Daily Allowances. Both
congregate and home-delivered participants have about 14 more social contacts per month than the
comparison group; this finding and interviews of participants suggest that the program increases
socialization opportunities for participants. Over 60 percent of all Title III and Title VI participants are
satisfied with program services. Approximately 40 percent of Title III ENP service providers have waiting
lists for home-delivered meals, suggesting a significant unmet need for these meals.
The evaluation shows that the program has achieved considerable success in efforts to target ENP services
to populations in the greatest need. Compared with the overall population in the United States age 60 and
older, ENP participants tend to be older, poorer, more likely to be members of racial or ethnic
minorities, and more likely to live alone. About one-third of Title III congregate participants and one-half
of Title III home-delivered participants have incomes at or below the poverty threshold. More than
one-half of Title VI meal participants have incomes at or below this level. The comparable figure for the
overall population age 60 and older is 15 percent. Nearly four times as many Title III participants and
nine times as many Title VI participants are low-income minorities, compared with the overall population
age 60 and older.
Agencies at the various administrative levels of the program have forged close links with other parts of
America's emerging home- and community-based long-term care system, primarily through cross-referrals
and coordination of service delivery at all levels within the aging network. The ENP provides a
continuum of services, including nutrition screening, assessment, education, and counseling. Title III
funds support 37 percent of congregate costs and 23 percent of home-delivered costs and are highly
leveraged by other funding sources, including U.S. Department of Agriculture commodities or cash in
lieu of commodities, State, local, and private funds. Participant contributions support another 20 percent
of the program costs. The leveraging rate for Title VI is considerably lower, with over 60 percent of the
costs supported by Title VI grants.
Use of Results
This report will provide policymakers with the information they need to determine the future scope of
activities of the ENP, as well as future funding levels. The information in this report will also assist
service providers in their efforts to improve the operation of their ENP projects and to obtain additional
State, local, and private funding to supplement Federal grants.
Agency sponsor:
Office of Program Operations and Development
Federal contact:
Jean Lloyd
202/619-2005
PIC ID: 6175
Performer organization:
Mathematica Policy Research, Inc., Plainsboro, NJ
Highlights
The final report on the Oregon Consumer Scorecard (OCS) Project describes the development, pilot
testing, and revision of a user-friendly guidebook, "A Consumer Guide to Selecting a Health
Plan," by the Oregon Consumer Scorecard Consortium. The prototype scorecard is intended to help
consumers choose a health plan consistent with their individual needs and health care service delivery
preferences. The model also may help purchasers and State policymakers develop their own comparative
reports about health plan performance.
Purpose
A contract from the Agency for Health Care Policy and Research was awarded in January 1995 to the
Oregon Health Policy Institute through the University of Washington for this Federal-State collaboration
in policy making and knowledge-building. There were five project objectives:
- Identify information consumers want and need to make better choices.
- Identify different information needs, based on geographic location and health conditions.
- Identify relevant health plan performance measures.
- Pilot test various formats for displaying quality, access, and satisfaction measures to consumers.
- Make recommendations for designing a broader statewide demonstration to evaluate a preferred
scorecard format for statewide implementation.
The model scorecard was intended to aid consumer decision making in two primary ways. First, it was
to provide comparative health plan descriptions based on the expressed information preferences of Oregon
consumers. Second, it was designed to educate consumers about health plan function, management, and
performance. The project was guided by the presumption that people will make more prudent choices
among health plans if they have access to objective, reliable measures of technical quality, health plan
performance, and consumer satisfaction.
Background
Oregon has one of the most extensive managed care markets in the United States. After Oregon received
approval for a Federal Medicaid waiver, the State legislature enacted a series of reforms known as the
Oregon Health Plan (OHP). Since its implementation, OHP has extended coverage to approximately
130,000 individuals, including many previously without health insurance coverage who must now navigate
an unfamiliar health care delivery system.
These reforms led to the establishment of the Oregon Office of the Health Plan Administrator (OHPA),
with responsibilities that included "to assist OHP consumers in selecting a health care provider or
a health care plan" (SB 5530, Oregon Statutes of 1993). The OHPA facilitated the creation of the
Oregon Consumer Scorecard Consortium, a public-private partnership committed to developing a
high-quality, user-friendly health plan guide.
Methods
A series of activities was undertaken to develop and pilot test stimulus materials that would subsequently
inform the design of a prototype consumer scorecard to be evaluated on a statewide basis. The first step
was the formation of the Oregon Consumer Scorecard Consortium, comprising more than 50 stakeholders
from the public and private sectors that may benefit from or be affected by the scorecard.
Second, a work plan was established for reviewing and synthesizing policy-related information,
conducting focus groups, and reporting on all other Consortium-sponsored data collection activities. The
information review and synthesis included analyzing the Oregon Office of Medical Assistance Programs'
annual consumer satisfaction survey, designed during the project year in consultation with the OCS
Project staff and the Consortium's Technical Committee; pilot testing measures from Medicaid HEDIS
(Health Plan Employer Data and Information Set), a widely used health plan performance measurement
tool; and gathering objective, comparative, and descriptive navigational information to help consumers
understand how a health plan really works.
Through a subcontract to Oregon Health Decisions, Inc., two rounds of four focus groups were conducted
with OHP-Medicaid consumers. Targeted at consumers living in rural areas and persons with chronic
health care conditions and disabilities, the first-round focus groups were designed to gain understanding
of consumers' expressed information needs and preferences for selecting a health plan. In the second
round, consumers who had a choice between two or more competing health plans in their counties were
given a model guidebook and a set of alternative formats and media presentations to use and critique.
Third, the first set of prototype scorecard materials was produced. Finally, using real health plan data
but fictitious health plan names, the guidebook was presented to Oregon consumers in a series of focus
groups for their review and assessment.
Findings
Consumers expressed a desire for more accountability from their health plans. They wanted objective
information on satisfaction and health plan performanceþinformation that is understandable, reliable,
meaningful, and geographically sensitive. There is a need for unambiguous descriptive data, free of
marketing bias, that show real differences in how health plans manage care.
Materials developed during this project were well received. Consumers were interested in multimedia
presentations of choice information, using interactive computer kiosks, videos, and telephone advice. The
availability of a trained, live individual was seen as an important backup to the scorecard.
The report noted that consumers and health professionals think differently about quality. Plan-level
information was less important to consumers than provider-level information. Consumers preferred
information that was relevant to their individual experience, including how they personally use health care
services. Condition-specific indicators in population-based performance measures (such as HEDIS) were
found to be less well understood, and therefore not as useful to consumers in selecting a health plan.
Despite the advantages of this voluntary collaboration, producing a reliable and uniform consumer
scorecard in a timely fashion proved difficult. The report recommended the establishment of uniform
standards and data specifications that all health plans and purchasers agree to abide by, with an
independent "audit" to ensure compliance.
The report also noted that the political nature and the potential economic consequences of the comparative
health plan information made it critically important to have an objective, disinterested third party
responsible for producing a scorecard. In addition, because scorecard development is politically and
methodologically complex, the report advised that future efforts should proceed in stages, with more
modest goals and an examination of the considerable costs of producing a scorecard.
Use of Results
This project provides a unique model for integrating major types of health plan performance information
into a consumer-oriented guidebook and adds to the emerging body of knowledge on consumer
information needs and preference modeling. This contribution to the science base for statewide quality
assessment and reporting can be used by other States in their efforts to evaluate quality of care.
Major policy and methodological issues to pursue in the future are addressed in the report, including the
need to address differences in how professionals and consumers think about quality; the need to balance
the focus on health plans as the unit of comparison for scorecard purposes with the interest of most
consumers in the attributes and performance of individual clinicians and the facilities in which they
practice; the need to reconcile population-based measures and personalized, consumer-relevant
information; the need to tailor information to consumers with special concerns; and the need to critically
evaluate the costs and benefits of various forms of presentation. The report recommends as a top public
policy goal the development of a real and meaningful quality feedback loop that extends from the State
to health plans to consumers and back to the State.
Agency sponsor:
Office of Planning and Evaluation
Federal contact:
Sandra Robinson
301/594-1349
PIC ID: 5960
Performer organization:
Oregon Health Policy Institute, Portland, OR
Highlights
Between 1986 and 1994, the State of Georgia implemented a program of audits of the immunization
practices of all State public health immunization clinics (227 in 1994) in an effort to improve child
immunization rates. During this time period, the immunization rate for preschoolers increased from 31
percent to 90 percent. This evaluation found that the performance of these audits was significantly
associated with higher immunization rates for preschool children in the clinic catchment areas.
Other management and medical practices were found to be significantly associated with increased
immunization rates. Greater accessibility, informed vaccine administration practices (such as following
only true contraindications), adequate clinic resources, and staff participation in leadership were also
associated with higher immunization rates. The report lists 11 recommendations associated with the
findings; in particular, the audits of the clinics should continue, and efforts to improve immunization rates
should be comprehensive in scope and continuous in application. Findings from the evaluation of this
highly successful program formed the basis of Centers for Disease Control and Prevention (CDC)
recommendations to all States.
Purpose
The purpose of this study was to evaluate how audits conducted by the State of Georgia affected
immunization rates among the preschool population, to assess management and organization factors
associated with immunization coverage, to identify aspects of immunization clinic practice and
organizational or managerial processes associated with changes in immunization coverage levels, and to
make recommendations regarding the implementation of similar programs in other projects in the United
States.
Background
The Public Health Service (PHS) objectives for the year 2000 include 90 percent immunization coverage
levels for children up to the age of 24 months. The diseases against which immunization is sought are
diphtheria, tetanus, pertussis, poliomyelitis, measles, mumps, rubella, Hemophilus influenza type B, and
hepatitis B. The CDC's Comprehensive Childhood Immunization Initiative was developed to address this
objective through an intensified strategy that includes service delivery, assessment, information and
education, operational research, and surveillance.
To improve its child immunization rates, the State of Georgia conducted a program of annual audits
between 1986 and 1994. The program allowed and encouraged district and clinic immunization staff at
the State's public health clinics to implement interventions to increase clinic-specific coverage levels.
Various approaches and combinations of interventions to change clinic practices were undertaken and
evaluated.
Methods
Data for the study were drawn from clinic audit data, 1987-94, and from a self-administered mail survey
of key immunization personnel in all public health clinics in the State of Georgia (227 in 1994).
Individuals surveyed included district health directors or program managers, district immunization
coordinators, and clinic immunization coordinators or clinic lead nurses. The survey response rate, due
to intensive follow-up procedures, was 100 percent. Univariate and multivariate methods were used for
analyses relating information on clinic practices from the surveys to information on audit rates from
medical files.
Findings
The study found substantial improvement in coverage rates following the audit program, with median
immunization rates at the clinics improving from 31 percent to 90 percent. Staff participation in the audit
process resulted in their increased awareness of the performance levels of their offices, and of how their
clinic compared with state averages; this, in turn, appeared to prompt improved immunization rates at
the clinics.
Accessibility factorsþsuch as short waiting times, fixed fees, and convenient hours of serviceþwere
associated with higher immunization rates at the clinics. Clinics with higher immunization rates also used
improved vaccine administration practices, suggesting an association between appropriate clinical
knowledge by key staff and higher coverage rates. Early outreach, aggressive reminder and recall, and
close coordination with Women, Infants, and Children (WIC) and other programs were also associated
with higher immunization rates.
Clinics with adequate resources had higher coverage rates. Computer technology, appropriately supported
and implemented, appeared to benefit clinic rates. Lower coverage rates were more likely to be found
at sites in the process of computerization than in those that were fully automated, although clinics without
computers had even lower immunization rates. These findings suggest the importance of support and
training to the process of computerization, as well as the importance of computerization in the clinic
setting.
The study also found that staff participation and management leadership were associated with higher
immunization rates. The investigators conclude that sustained efforts to improve immunization rates can
make a difference.
Use of Results
Eleven recommendations based on study findings are included in the report, supporting continuing efforts
to improve immunization rates through strategies that are long term, comprehensive in scope, and
continuous in application. The report recommends continuing the audits, which were a critical element
helping to improve these rates, and furnishing audit results to clinic providers and managers.
Recommendations also target ready accessibility to clinics, early outreach, and aggressive reminder and
recall. Staff training and development were encouraged to provide accurate information about
contraindications and to minimize missed opportunities for immunization. Clear management practices
that support and recognize the importance of immunization, that foster staff participation in decisions
about improving immunization rates, and that provide adequate staff and financial resources were also
recommended. The report recognizes the important role that programs such as WIC can play in
improving immunization rates, and it calls for ongoing study of the role of computerization and of
improvement efforts in other settings as they relate to coverage rates.
These recommendations have had far-reaching application, providing a national model for immunization
practice. Georgia staff report that the evaluation findings have stimulated new staff thinking about
immunization practice, as well as changes in policies to carry out the study's recommendations. Georgia's
successful strategyþusing routine assessment and feedback to achieve high sustainable coverageþwas
adopted and produced successful results in public health clinics in Colorado, Illinois, Iowa, Kansas, and
South Carolina. It also worked in private and managed care provider settings in Arizona, Massachusetts,
New York, and Washington.
Several reports of early study findings have appeared in the Morbidity and Mortality Weekly Report and
CDC information sheets. Workshops and presentations on the project have been conducted throughout
the term of the project, resulting in replication of this approach in other States. Furthermore, CDC
reports that the recommendations of this study now provide the basis for their national immunization
program: Since 1995, all States and localities receiving Federal funds for vaccine programs have been
required to conduct annual assessments of vaccine rates in public health clinics.
Agency sponsor:
National Immunization Program
Federal contact:
Andrew Baughman
404/639-8239
PIC ID: 5354
Performer organization:
Battelle Centers for Public Health Research and Evaluation, Arlington, VA
Highlights
Chronic ventilator patients experience high mortality, and their care is exceedingly expensive. The
purpose of this evaluation was to determine whether specialized units serving chronic ventilator-dependent
patients deliver better clinical outcomes at a reasonable cost to Medicare. The evaluation compared
chronic ventilator patients treated in ventilator-dependent units (VDU's) at four demonstration sites with
similar patients treated in conventional hospital settings, usually in intensive care units. The evaluation
found that patients treated in VDU's have improved clinical outcomes and lower hospital daily costs, but
higher overall costs to Medicare because of greater longevity. For example, while the patients' median
survival from the time of admission was about 5 months longer, and their functional status superior at
discharge, the overall cost to Medicare was increased by 35 percent, due to longer life expectancy. The
findings across the four demonstration sites varied greatly, however. Because of the variability in clinical
outcomes and the difficulty of screening patients for placement in VDU'sþboth of which can lead to
greater costs without commensurate clinical benefitsþthe evaluation concluded that nationwide
implementation of VDU's is not recommended. However, the evaluation did find a small number of
special VDU centers of excellence to be warranted. The analysis of costs and outcomes provides
important information to policymakers and health providers about a seriously ill population with intensive
resource needs.
Purpose
The purpose of the evaluation was to assess whether chronic ventilator patients covered by Medicare were
better served at a reasonable cost by specialized hospital units called ventilator-dependent units. These
units, studied at four Medicare demonstration sites, have a rehabilitation focus and are staffed by a highly
trained multidisciplinary team of health professionals. The daily cost of care in a VDU is about $500 to
$700 less than that of an intensive care unit, where most hospitals care for chronic ventilator patients.
The evaluation compared costs and clinical outcomes of 211 chronic ventilator patients treated in VDU's
with 401 comparison cases treated in conventional hospital settings. The evaluation also projected the
overall costs of providing care in VDU's to Medicare patients nationwide.
Background
Mechanical ventilation is a life-sustaining technology for patients suffering from acute respiratory failure.
The ventilator is an apparatus that delivers air to the lungs of patients unable to breathe on their own,
usually because of failure of other organ systems or severe chronic illnesses. Because of the severity of
their underlying illnesses, 49 percent of Medicare ventilator patients died in FY 1988. The 133,500
Medicare patients discharged in FY 1994 after an episode of mechanical ventilation had an average length
of stay of 23.1 days, with total Medicare expenditures amounting to $3.5 billion. Thus, ventilator
patients' high mortality, coupled with the high cost of treatment, warranted further scrutiny by the Health
Care Financing Administration (HCFA), the agency that administers Medicare.
The focus of the evaluation was on chronic ventilator patients, that is, the small subset of ventilator
patients who need ventilation for extended periods. Chronic ventilator patients are especially important
to study because they are even more seriously ill and their care is correspondingly more costly. In this
evaluation, a chronic ventilator patient was defined as one receiving at least 20 days of ventilation.
This evaluation was conducted under an exemption to Medicare's prospective payment system to allow
demonstration hospitals to be paid for VDU care on a cost-reimbursement basis, with some incentives
to control costs. Under Medicare's prospective payment system, Medicare normally pays a flat fee for
chronic ventilator patients. Because of earlier reimbursement problems, patients are now grouped under
one of three distinct diagnosis-related groups specifically tied to their receipt of chronic ventilation, rather
than to their underlying illnesses. Yet problems still exist with the revised groupings, because hospitals
and patients continue to have strong incentives to discharge patients to another setting. Understanding
costs and effective treatments therefore permits further refinement to the prospective payment system for
chronic ventilator patients.
Methods
The evaluation of costs and outcomes relied upon a constellation of data bases, including Medicare's
enrollment and claims data and special clinical data sets. The four VDU demonstration sites were the
Mayo Clinic; Temple University Hospital; RMS Health Providers in Hinsdale, Illinois; and Sinai Hospital
of Detroit. New clinical instruments were developed for these sites to obtain a comprehensive portrait
of the course of ventilator episodes. Claims data were used to track the VDU and the comparison group
through 18 months following hospital admission.
The comparison group was drawn from a HCFA pilot data base known as the Uniform Clinical Data Set
System, which is no longer in place. The data on chronic ventilator patients were derived from hospital
records in five States. A major difference between VDU and comparison cases was that the former were
screened for entry into VDU's, while the latter were not. Screening according to criteria set by HCFA
excluded medically unstable patients and those with poor prospects for rehabilitation. Multivariate models
were employed to control for differences between the VDU and comparison cases.
Findings
Three sets of findings were generated by the study: admissions, clinical and cost outcomes, and national
implementation findings. The admissions findings revealed that, in many instances, patients were
improperly admitted to VDU's. In other words, it was difficult for VDU sites to meet all of HCFA's
complex criteria for entry, which relate to medical stability and reasonable potential for rehabilitation.
Across VDU sites, there was great variability in the application of admission criteria.
With respect to outcomes, the evaluation analyzed 15 clinical outcomes and 17 expenditure outcomes.
On many of the clinical outcomes, VDU cases fared better than comparison cases. Among the most
important findings was that median survival from the point of hospital admission was 258 days for VDU
cases and 106 days for comparison cases, a statistically significant increase of about five months. The
mortality rate within the hospital was 34 percent for VDU cases and 48 percent for comparison cases.
When patients were discharged, VDU cases were in better condition: 34 percent of VDU cases were
discharged to their home, compared with 27 percent of comparison cases. Their functional status at
discharge was superior, using a special index based on activities like bed mobility, locomotion, toileting,
and eating. About 18 percent of VDU cases at discharge scored in the highest functional group, compared
with 11 percent of comparisons. At two of the demonstration sites, however, some clinical outcomes were
not significantly improved relative to the comparison group.
While daily Medicare expenditures for VDU cases were lower, overall expenditures were about 35
percent higher because patients lived longer. Total hospital expenditures averaged $123,000 per VDU
case and $91,000 per comparison case. The mean daily spending within the hospital was $1,468 for VDU
cases and $1,740 for comparison cases, because VDU's were generally less costly than alternatives.
In the national implementation analysis, costs were projected for covering VDU's for all eligible Medicare
patients. The evaluation found that 67 percent to 80 percent of comparison cases would have been
admitted to a VDU if one had been available, yielding an estimate of between 24,000 and 41,000 total
admissions nationwide. Total Medicare expenditures for their care were estimated between $0.6 and $2.2
billion, depending on assumptions and duration of coverage. In light of the difficulty of controlling
admissions and the variability in clinical outcomes, the evaluation concluded that nationwide
implementation of VDU's would be very costly and of questionable benefit.
Use of Results
The evaluation recommends continuation of VDU's at selected centers of excellence to refine
rehabilitative treatment for chronic ventilator patients. It also recommends further exploration of financing
mechanisms that could encourage integration of care, rather than moving patients from one setting to
another. Managed care, particularly through case management, is seen as holding promise for such
integration. The evaluation also pointed to research questions that warrant further attention. Finally, by
providing detailed information about costs and clinical outcomes for a population at high risk of death,
the evaluation raises for policymakers and health care providers the difficult question of whether and how
resources should be allocated to a seriously ill population in a manner that optimizes duration and quality
of life, without being economically wasteful and prolonging suffering.
Agency sponsor:
Office of Research and Demonstrations
Federal contact:
Michael Henesch
410/786-6685
PIC ID: 4590
Performer organization:
The Lewin Group, Fairfax, VA
Highlights
This evaluation reports on Phase II of a three-phase effort to develop a fully integrated computerized
data system to facilitate planning and evaluation of the programs administered by the Bureau of Health
Professions. Phase I assessed indicators of the Bureau's progress in meeting the goals and objectives
defined in its strategic plan. Phase II analyzed the feasibility and appropriateness of the goals, outcomes,
and indicators developed by the Bureau and identified strategies the agency can use in refining them with
periodic input from grantees in the field. A performance monitoring system was proposed to facilitate
ongoing program management. The system described in this study builds on the Bureau of Health
Professions Grants Management Application System. The study recommended that the proposed
performance monitoring system be piloted with several programs before it is fully implemented.
Purpose
The overall purpose of this project was to help the Bureau of Health Professions develop a set of
outcome-based performance measures and design a performance monitoring system, both to measure
whether supported programs are meeting national health workforce objectives and to identify necessary
program course corrections. The following issues were addressed in the study:
- Can the outcomes and indicators identified to monitor program progress in meeting the agency's
goals be reliably collected and analyzed?
- Do grantees believe that the identified goals, outcomes, and key indicators reflect the most
valuable contributions of their projects?
- What data sources and collection tools are currently available?
- What kind of system should be developed to support the requirements of the Government
Performance Results Act (GPRA) of 1993 and of other performance monitoring and reporting measures?
Background
The Bureau of Health Professions provides both policy leadership and support for enhancement of the
health professions workforce and development of its educational infrastructure. This evaluation follows
earlier work to assess and refine a set of goals with respect to workforce quality, supply, diversity, and
distribution, as well as the outcomes and indicators of performance identified to measure and monitor
progress toward those goals. This strategic planning process was undertaken by the Bureau in response
to internal and external pressures for more effective targeting of scarce Federal resources toward those
programs and activities that support and have a demonstrable effect on national workforce priorities. The
development of a more explicit outcome-oriented system that identifies measures of performance related
to Bureau-funded efforts will help address such concerns and will be very useful in future planning and
program management.
The cross-cutting goals developed by the Bureau of Health Professions address (1) the development of
a health care workforce that has the mix of competencies and skills needed to deliver cost-effective,
quality care; (2) the need for educational programs that will yield professionals who can meet the needs
of vulnerable populations; (3) the need for cultural diversity in the health professions; and (4) the need
to stimulate and monitor the education system's ability to respond to the changing demands of the health
care marketplace. Expected outcomes and indicators of success were developed and refined for each goal
as part of phaseI of this evaluation effort. This evaluation was undertaken to address the next set of
questions that needed to be examined by the Bureau in order to develop a performance measurement and
management system that could link individual grantee level information in all of the Bureau's programs
to one or more of the cross-cutting goals, associated outcomes, indicators, and performance-related
functions.
Methods
The investigators employed several approaches to achieve the evaluation objectives. Among these were
ongoing discussions among evaluators, Bureau of Health Professions program administrators, and other
agency officials and committees whose input would inform the design of the performance evaluation
system. The evaluators also consulted outside experts representing a range of health workforce training
perspectives on feasibility issues related to data collection from grantees for program monitoring and
evaluation purposes. An initial survey of data sources was then undertaken to explore existing systems
that may be applicable to the ongoing data collection needs of the Bureau's performance monitoring
system.
Findings
This report focuses on the design of a Comprehensive Performance Monitoring System (CPMS). The
study calls for a system that will provide information to answer a basic performance question: Can the
Bureau of Health Professionsþwith available funding and guiding legislation and through planned and
funded grant activitiesþmeet national health workforce objectives for targeted populations? In capturing
the information necessary to answer this question, the system would track essential inputs, processes,
outputs, and outcomes for Bureau programs. The system would provide that information in regular
reports to internal decisionmakers and to the Bureau's external customers, including Congress and
grantees. The CPMS should support the following functions: monitoring and measuring performance
goals; analyzing and assessing indicator data; identifying successes and problems; reviewing key program
processes; and identifying opportunities.
A detailed set of recommended steps was developed for the Bureau's preparation for implementing the
CPMS. These include completing final specifications of cross-cutting indicator definitions; specifying
performance benchmarks and a process for assessment; identifying external indicators that must be
tracked (e.g., market interest, availability); identifying sources of data and pilot testing new data
collection; specifying hardware and software requirements of the system; and developing a pilot version
of the system for beta testing.
The processes essential for maintaining the functions of the CPMS will require data, hardware, software,
and human resources. The analytic and reporting functions of the system would be largely automatic once
designed and implemented. Obtaining key information resources will require collecting data that are
readily available and developing data collection instruments or surveys to collect data that are not
currently reported by all grantees on a systematic basis. Other data collection resources managed outside
of the Bureau may also be integrated into the system. The CPMS could exist in a variety of forms,
ranging from paper files and reports to computer data base systems. The study recommended that an
add-on module linked to the Grants Management Application System may yield the most efficient system.
The system could be built as a relational data base, with linkage variables for cross-referencing data. The
types of resources that would be needed for a computer-based performance monitoring system could
include computer hardware; system software, including data base system software; and trained software
and hardware systems support staff to develop, maintain, and update the capabilities of the system as
needed. The data base could be on a stand-alone computer or could reside on a network.
The study concluded that in the final phase of this effort, the Bureau of Health Professions must address
the questions necessary to perform key functions of the CPMS for each of the cross-cutting goals,
outcomes, and indicators. Furthermore, the Bureau must decide what data elements should be collected
from grantees to support the CPMS. Many data elements may already be collected and captured in the
Grants Management Application System; thus, the CPMS should be integrated with this system and the
staff with these responsibilities should coordinate their efforts.
Use of Results
This Bureau of Health Professions study on performance monitoring to support program management and
Bureau-wide planning also fits into the context of broader efforts within the Health Resources and
Services Administration and HHS to conduct strategic planning for resource investment and compliance
with the GPRA. Policymakers and program staff in other Federal agencies within and outside HHS can
benefit from the efforts described in this evaluation.
Agency sponsor:
Bureau of Health Professions
Federal contact:
Jennifer Burke
301/443-1590
PIC ID: 5497
Performer organization:
The Lewin Group, Fairfax, VA
Highlights
This study assessed the effects of State board and care home regulations on the quality of care
provided by those institutions. The investigators also attempted to identify the characteristics of the board
and care environments in the States selected for the study and to determine whether licensed and
unlicensed homes differed in terms of those characteristics or in the quality of care provided. The
evaluation concluded that extensive State regulation and licensure can improve the quality of care
provided by board and care homes.
Purpose
This investigation examined the quality of care in board and care homes in 10 Statesþ5 with extensive
regulatory systems and 5 with more limited regulatory systems. For the purposes of this study, board and
care refers to nonmedical, community-based residential settings that house two or more unrelated adults
and provide such services as meals, medication supervision or reminders, organized activities,
transportation, or help with activities of daily living.
The principal purpose of the study was to analyze and compare State board and care regulations and their
effects on the quality of care received by board and care residents. Investigators attempted to do the
following:
- Identify the characteristics of the board and care environments in the selected States, including the
characteristics of the homes, operators, staff, and residents.
- Determine the effect of State regulation on the quality of care and the experience of board and care
residents.
- Explore differences between licensed and unlicensed homes, particularly with respect to quality
of care.
To accomplish these objectives, the study analyzed data on State regulatory systems, constructed a sample
frame of licensed and unlicensed homes, and implemented a complex multistage sample design. In
addition, the investigation compared data with the findings of earlier studies of board and care and with
current data on the characteristics of other residential long-term care settings to clarify the role played
by board and care homes.
Background
Changes in population demographics and a number of policy initiatives have increased demand for
residential facilities that offer support services for the aged and disabled. Chief among these are a rapidly
growing elderly population with significant levels of physical disability and mental impairment, an almost
universal rejection of nursing home care by younger persons with disabilities and their advocates, and
a strong preference among the elderly for in-home and community-based services as opposed to traditional
nursing home care. While family efforts continue to be the primary source of long-term care for elderly
and disabled loved ones, different types of residential settings with support services have emerged to
supplement the efforts put forth by families.
There are approximately 34,000 licensed board and care homes in the United States, with more than
613,000 beds. These homes fall into one of three basic types of licensed facilities: those serving mentally
retarded or developmentally disabled persons, those serving mentally ill persons, and those serving a
mixed population of physically frail elderly, cognitively impaired elderly, and persons with mental health
problems. Not all board and care homes are licensed, however, and by some estimates, unlicensed homes
are as numerous as licensed facilities. Thus, the total number of persons living and receiving care in all
types of board and care homes may be as high as one million.
While the Federal Government has traditionally played only a limited role in monitoring or regulating
the quality of services provided by board and care homes (leaving primary oversight to the States), the
Office of the Assistant Secretary for Planning and Evaluation (ASPE) in HHS has maintained a
long-standing interest in the potential of board and care homes to meet the needs of elderly and disabled
persons for residential services. This becomes particularly important in light of the strengthened Federal
oversight of nursing home quality required by the Omnibus Budget Reconciliation Act of 1987.
The present analysis stems from Federal concerns regarding the lack of systematic information on board
and care residents, changing levels of disability among the population served, adequacy of care,
protection from health and safety risks, and the significant numbers of unlicensed and unregulated homes
in the United States.
Methods
To accomplish the goals of the investigation, the study design incorporated several activities, including
a major collection of new data. However, all activities focused on the main study goals of facilitating
cross-sectional comparisons among facilities and residents based on the licensure status and regulatory
environment under which the homes operated.
After reviewing current State regulatory approaches, the investigators selected 10 States for inclusion in
the study; these States represented the extremes on a continuum of regulatory systems, ranging from very
extensive regulation (California, Florida, New Jersey, Oklahoma, and Oregon) to very limited regulation
(Arkansas, Georgia, Illinois, Kentucky, and Texas). A stratified, multistage, cluster design was used to
select probability-based samples of homes, staff, and residents in each State. A sampling frame of eligible
unlicensed homes was created using the Social Security Administration's State Data Exchange (SDX)
tapes and network sampling of State and local agencies.
Primary data collection was then undertaken in 385 licensed and 129 unlicensed board and care homes,
including interviews with 490 operators, 1,138 staff, and 3,257 residents. Site visits were conducted at
each facility and extensive information was collected on characteristics of the home, including patient and
payer mix; characteristics of the staff, including knowledge of aging and caregiving; resident
demographics, health status, and satisfaction; and quality of care, including the physical environment,
adequacy of staff, unmet health needs, and patient satisfaction. Data were analyzed using descriptive
statistics and logistic and linear regression models.
Findings
This inquiry found substantial and widespread positive effects of both regulation and licensure on the
quality of care in board and care homes. States with extensive regulatory systems had a significantly
smaller proportion of unlicensed facilities than States with limited regulation (7 percent versus 25
percent). Extensive regulation also had a positive effect on several quality-of-care and quality-of-life
indicators, such as lower use of psychotropic drugs and medications contraindicated for the elderly, more
operator training, and greater availability of social aids (e.g., reading materials and community rooms)
and supportive devices (e.g., grab bars in showers and call buttons in bathrooms).
Licensure also had a positive effect on many of the quality-of-care measures explored. For example,
licensed homes were more likely to have operators with training and to make more social aids and
supportive devices available to residents. Licensure also enhanced the availability of key services and the
prevalence of safety features. A significant finding is that licensure was effective in raising homes above
a minimum level of acceptable performance, or "preventing" the worst performance. Thus,
licensed homes were less likely to have the lowest scores on such measures as availability of social aids,
physical amenities, safety features, and an environment with little diversity and a very institutional
atmosphere.
The study also found that neither extensive regulation nor licensure had a positive effect on some aspects
of quality, including a requirement for preservice training of staff and staff knowledge of care,
monitoring, and medication management. Regulation also had no significant effect on the cleanliness of
homes and the availability of amenities or the likelihood that a home would have any licensed nurses
(registered nurses or licensed practical nurses) on staff. Furthermore, there was little variation among
homes on such issues as unmet health care needs, residents' rights, and indicators of resident satisfaction.
Finally, the report noted that the niche for board and care homes appears to be different in States with
extensive regulatory systems compared with States with limited systems. States with extensive regulatory
systems have higher-than-average board and care bed supply and lower nursing home bed supply. Homes
in States with extensive regulation have residents with higher levels of disability than homes in States with
limited regulatory systems. Thus, it appears that States with extensive regulatory systems were using
board and care beds as substitutes for nursing home beds, particularly in comparison with States with
limited regulatory systems.
Use of Results
The study findings have several important implications for key participants in the board and care sector.
Findings point to a board and care population that is considerably more frail and disabled than it was 10
years ago. Furthermore, the mix of physically frail elderly, cognitively impaired elderly, and residents
with mental illness and developmental disabilities (some of whom are nonelderly) presents a complex
caregiving challenge. These factors should prompt a reexamination of the health and safety issues that
confront board and care providers and the States' systems for regulating the industry. Of primary
importance are the range of services, staffing patterns, and staff training and knowledge needed to meet
the needs of today's residents.
This study confirms that there is a well-defined role for board and care homes in the provision of
long-term care. The findings specifically suggest the following:
- States can improve the quality of care in board and care homes through appropriate regulation.
- States can improve other aspects of quality by requiring licensure of board and care homes.
- The Federal Government can support State and provider efforts to improve the quality of care by
developing and disseminating information.
- The board and care industry should work closely with State Governments to improve the quality
of care.
Agency sponsor:
Office of Disability, Aging, and Long-Term Care Policy
Federal contact:
Floyd Brown
202/690-6613
PIC ID: 4720
Performer organization:
Research Triangle Institute, Research Triangle Park, NC
Highlights
Medical technology embraces innovations in medicine, such as new drugs, biologics, medical
devices, and procedures, as well as existing therapeutic and diagnostic capabilities. Medical technology
has been identified as a major factor driving the increase in national expenditures for health care services.
Thus, the evaluation of the clinical and cost-effectiveness of medical technology is of substantial interest
to many parties. Technology assessment is the term most often applied to such evaluations. A strong
national system of technology assessment has not emerged in the wake of a series of failed efforts by the
Federal Government. With rapid changes in health care, particularly the rise of managed care, the
demand for technology assessment has increased in the private sector. It is not clear if technology
assessment activities in the private sector are replacing the assessment functions once expected of
Government.
This report briefly describes a number of private-sector health organizations that are engaged in
technology assessment. Technology assessment is one of a family of evaluation activities occurring within
these health care organizations. Private-sector technology assessment is highly evidence based. Rigorous
evaluation of clinical effectiveness based on a systematic review of scientific and clinical evidence has
become the norm among private organizations conducting technology assessment. Technology assessment
in this sector has increasingly become "full service," encompassing drugs, medical devices,
and clinical procedures.
Purpose
This report considers the use of technology assessment by managed care organizations, which constitute
the most significant change in the financing, organization, and delivery of health care. Managed care
organizations have strong incentives to hold down the costs of care, and one way to do so is to avoid
providing unnecessary, inappropriate, or inadequately tested medical technologies and procedures. This
study aims to answer the following questions: What is the demand for technology assessment? Who are
the performers of technology assessment? What characterizes the conduct of technology assessment? How
are the results of technology assessment used? What unmet needs for technology assessment might be met
by cooperation between the public and private sectors of health care?
Background
Much attention has been given in the past two decades to a strong governmental role in technology
assessment, a term used to describe the evaluation of the clinical and cost-effectiveness of medical
technology. Yet the Government bodies established to conduct such evaluations have not survived the
budget cuts and political environment of the 1990's. At the same time, large corporate purchasers of
health care have become more active in seeking to rein in the growth of health care expenditures.
Managed care organizations have responded to market opportunities by enrolling an increasing proportion
of patients, reducing excess hospital capacity, and shifting care away from physician specialists to primary
caregivers. In this context, the evaluation of clinical practice for cost-effectiveness has assumed greater
market value, and the demand for technology assessment has increased. There are two main sources of
increased demand for technology assessment: the press for cost containment and performance evaluation
and the need to be responsive to a changing health care marketplace.
With the failure of health care reform efforts in the early 1990's, and with the emergence of managed
care, new questions have emerged about the extent of technology assessment activity in the private sector
and whether these activities fulfill the functions that were once expected of the Federal Government. This
study briefly describes a number of private-sector health organizations engaged in technology assessment.
Methods
The scope of this study was narrowed to focus on managed care. Data collection was accomplished
through site visits, interviews, attendance at meetings, document review, and follow-up interviews. Since
the Blue Cross/Blue Shield Association and The Emergency Care Research Institute had long been
involved in technology assessment, they were the primary contacts for data collection. In total, 46
interviews were conducted with 56 individuals in 29 different organizations, including performers of
technology assessment, third-party insurers, managed care organizations, professional organizations, and
other settings. This report is not a comprehensive enumeration of all the organizations engaged in
technology assessment, but is a selective picture of the major players; thus, the extent to which these
practices are diffused is unknown.
Findings
In the past decade, a strong, decentralized technology assessment capability has developed in the private
health care sector of the United States, consisting of a small number of national organizations and a
growing number of larger health plans. Private organizations are conducting technology assessment in
response to private-sector demands. These assessments are often designed to serve special clientele and
require a significant number of internal staff resources.
Some national insurers and managed care organizations conduct centralized technology assessment efforts
that serve member plans. Several membership organizations have constituency-oriented technology
assessment programs, and some individual managed care organizations have significant technology
assessment capabilities. In short, the private-sector technology assessment system is a mix of distributed
centralized (national subscription, plan, and constituency efforts) and decentralized (regional and local
health plans) efforts. Managed care has strong incentives to support and conduct technology assessment.
Several issues in technology assessment appear to have been settled in the past decade, including the
organizational decisions supported by such assessments, the evaluative criteria for assessing a technology,
the means of setting priorities for conducting assessments, and the stage of technology that is the object
of the assessment. However, issues of how technology assessment applies to clinical trials and the
development and maintenance of data bases are yet to be resolved.
In general, the evaluative context within which technology assessment is conducted has changed, focusing
more on effectiveness, outcomes, appropriateness, and clinical practice guidelines. There has also been
a shift toward evidence-based assessment, that is, a deemphasis on intuition and more emphasis on
systematic clinical experience as sufficient grounds for clinical decision making. Although consensus
processes are used to interpret and apply the results of technology assessments, the evidence-based
movement discounts very heavily the use of consensus to assess the scientific bases of clinical practice.
Driven by cost-containment and quality-assurance objectives, health plans are increasingly examining the
clinical effectiveness of interventions in relation to their financial implications. The use of technology
assessment in the managed care sector is primarily to support coverage decision making. However, a
number of innovative developments go beyond this narrow range of issues. These include benchmarking
for member organizations of evaluative activities believed to be needed for survival in the current
marketplace and providing a forum for reviewing the evidence related to competing technologies. They
also include educational efforts in some plans, directed at changing physician behavior toward
evidence-based medicine. Finally, the working limits of technology assessment have been broadened to
include the evaluation of drugs after Food and Drug Administration approval for marketing.
Use of Results
An examination of private-sector technology assessment leads back to questions about the appropriate role
of the Federal Government in technology assessment and the appropriate division of labor between the
public and private sectors. One strong implication is that the Federal Government, as purchaser of health
care services, should be no less competent and effective than the private sector in supporting and
conducting technology assessment. However, this does not necessarily imply a centralized Federal
Government leadership role, which does not appear feasible at present. Although the Federal Government
has not shown a leadership role in technology assessment, it can play a significant role as sponsor of
technology assessment-related research.
Agency sponsor:
Office of the Assistant Secretary for Planning and Evaluation
Federal contact:
Cheryl Austein-Casnoff
202/690-6102
PIC ID: 6180
Performer organization:
RAND Corporation, Santa Monica, CA
Highlights
This investigation examined policy-relevant issues pertaining to the development of more effective
and efficient health care delivery systems. The evaluation report provides an overview of and historical
perspective on subacute care, which generally refers to skilled care for patients with complex needs
outside of the acute care, or hospital, setting. In addition to describing the current state of the art in
subacute care and how it varies across selected market areas, the evaluation examines the evidence about
its cost and effectiveness. Implications of the study's findings are discussed, and key policy issues related
to the future of subacute care are presented. The study concludes that the emerging concept of subacute
care holds promise in terms of better care and lower cost, but that it also poses many challenges for the
public and private sectors.
Purpose
This study was initiated by ASPE to examine policy-relevant issues pertaining to the development of more
effective and efficient health care delivery systems and involving substantial public and private
expenditures. These issues encompass the definition of subacute care; variations in subacute care patients,
providers, and settings; incentives and barriers to the growth of subacute care; and the current and
potential size of the market. The primary purpose of the study was to provide useful information to both
the public and the Department, including a synthesis of subacute care research, practices, and trends and
an assessment of public policies related to subacute care. The way in which the term "subacute
care" has evolved and the extent to which the ideal vision is currently implemented in practice are
key subjects of the study.
Background
Public spending for post-acute care services increased in the past decade from $3.4 billion to over $12.1
billion, an average annual growth rate of nearly 25 percent. In contrast, Medicare spending for acute care
services grew by only 6 percent per year during the same period. If the current system is unchanged,
Medicare expenditures for post-acute care and acute care are expected to rise even more rapidly as the
number and proportion of the older adult population increases. Escalating costs and shifting demographics
have prompted some providers to advance the concept of "subacute care" as a cost-effective
alternative to inpatient acute hospital care. Proponents believe that subacute care has the potential to
generate savings and improve patient outcomes.
At the time of this study, there were many definitions of subacute care. Historically,
"subacute" described care provided for hospitalized patients who did not meet established
criteria for a medically necessary acute stay. Currently, the term refers almost exclusively to patients
treated somewhere other than in acute-care beds. The concept now focuses on shorter stays that result
in improved rehabilitation for lower cost, using specific procedures designed to achieve better outcomes
for patients whose service needs fall somewhere between those traditionally provided by acute care
hospitals and nursing facilities.
Little reliable information is available about the likely impact of subacute care on costs, quality, and
access. This lack of data may be related both to differing definitions of "subacute care" and
to the recent and rapid expansion of the industry.
Methods
Initiated in August 1994, this study had several methodologies: a comprehensive literature search and
review of published and unpublished documents; the establishment of an advisory group comprising
public- and private-sector experts; interviews conducted with additional national experts from a variety
of fields and settings; site visits to 19 institutional providers in four market areas (Los Angeles, Miami,
Boston, and Columbus, Ohio); interviews with seven home health care providers; telephone interviews
with eight home care firms; and a series of stakeholder interviews.
Findings
The study team concluded that insight into the different ways in which "subacute care" is
defined is critical to understanding the subacute care phenomenon. The term commonly refers to skilled
care for patients with complex needs that some nursing facilities, home care providers, and others have
been providing for years under a variety of names. There is, in addition, a growing movement to create
a new type of subacute care. The core elements of this idealized, prototypical form of subacute care
include a program organized around particular disease categories, specific interventions, or homogeneous
patient characteristics; a focus on achieving measurable outcomes in a more efficient and lower cost
manner; special resources, such as certain physical plant characteristics and more and better trained staff;
and a set of techniquesþfor example, the use of interdisciplinary teams, case managers, care maps, or
critical pathway protocols; evaluation of outcomes; and an emphasis on continuous quality improvement.
Estimates of the number of days of subacute care provided each year in this country range from 1.2
million to 8.1 million. It was not possible to develop a reliable estimate of the current amount of subacute
care provided. Distinguishing characteristics of subacute care are not captured in any large national data
base. In addition, the study team encountered difficulties in identifying and comparing subacute care
providers, in finding subacute care providers in reportedly better developed markets, and in obtaining data
on subacute patients.
Factors shaping the development of subacute care nationally include the growth of managed care; efforts
of managed care providers to find more cost-effective services; the implementation of new Medicare
payment policies applicable to acute and post-acute care providers; changes in patient preferences, such
as a desire for a homelike environment for patients with minimal nursing needs; and the growth of
publicly owned, for-profit, post-acute care companies. Industry growth is also being driven by
facility-based subacute care providersþskilled nursing facilities, freestanding and hospital-based nursing
facilities, rehabilitation hospitals, distinct-part rehabilitation units, and long-term hospitals. The
establishment of minimum standards and broadly defined quality guidelines by the Joint Commission on
Accreditation of Health Care Organizations and the Rehabilitation Accreditation Commission has also
influenced the subacute care industry.
The study team found that home infusion therapy and full-service home health agencies were providing
a product with many of the elements of the prototypical subacute care facility. In response to the growing
potential for substituting home health care for facility-based subacute care, many interviewees expressed
concern about the quality of care, staffing qualifications, the role of managed care, access issues for
lower-income patients, the perception of a "no-care zone" for patients with lower acuity, and
the Medicare reimbursement policy for home infusion therapy.
Little evidence was discovered to support the premise that shifting patients earlier from hospitals to
subacute care will save money. It was noted that a national study of potential subacute care savings for
Medicare was based on several questionable assumptions. There are only a few empirical studies on
quality in subacute care facilities; these have generally found poorer outcomes for patients in skilled
nursing facilities than for those in rehabilitation hospitals. The four available studies comparing functional
status outcomes for rehabilitation patients in hospitals and skilled nursing facilities involved a limited
number of facilities; none was nationally representative.
In conclusion, the study team determined that the emerging concept of subacute care is appealing because
of the new attention it has focused on some types of patients and the programs envisioned in the ideal.
While site visits identified some subacute care providers that are successfully applying elements of the
concept, many other services labeled "subacute care" fell short of the ideal. Despite
aggressive marketing, the study team found that much that is called "subacute care" is little
more than a new name for care provided to higher acuity, medically complex patients or to those
requiring more intensive therapies. The idealized approach to subacute care promises similar or better
care for lower cost but poses major challenges to both the public and private sectors.
Use of Results
Significant proposed legislative changes to Medicare and Medicaid have heightened the need for accurate
and reliable information about subacute care, including a rigorous examination of the potential savings.
This kind of information can assist Federal and State Government policymakers in determining their role
in the evolution of subacute care and in understanding the implications of their actions on public and
private expenditures, quality of care, patient access, and provider equity. This study underscores the
importance of finding replicable technologies for producing better value, not just cheaper care, and of
developing appropriate payment policies that achieve both savings and value.
Agency sponsor:
Office of Disability, Aging, and Long-Term Care Policy
Federal contact:
Jennie Harvell
202/690-6613
PIC ID: 5951
Performer organization:
The Lewin Group, Fairfax, VA
Highlights
Cost-effectiveness analysis (CEA) is one tool available to decisionmakers to help determine the
relative value of different approaches to improving health or life expectancy. For a variety of reasons,
however, CEA has not been widely used in health policy settings. This study represents a review of the
methodological and theoretical aspects of CEA, its effectiveness, and its expected outcomes and uses. The
resulting report is a consensus among many of the top researchers in the field and provides guidance to
the conduct of CEA. The report sets forth several overlapping recommendations: some intended to
improve the overall quality of CEA and some intended to set a research agenda for further improvement
of CEA. Recommendations are made in the areas of framing and designing the study, identifying and
valuing outcomes, assessing effectiveness, estimating costs, discounting, reflecting analytical uncertainty,
and reporting analyses. It also discusses practical constraints to CEA's policy uses, carefully taking into
account the needs of its varied audiences.
Purpose
The purpose of this study was to assess the current state of the art of the use of CEA in health care and
to recommend steps to improve its quality and comparability. The Panel on Cost-Effectiveness in Health
and Medicine, working closely with methodologists and liaisons from the Public Health Service and the
Health Care Financing Administration attempted to identify a number of methodological problems in
existing CEA's that limit the analyses' policy relevance and usefulness.
Background
Society is confronting many difficult choices regarding the provision of health care services and the
funding of public health programs to improve the Nation's well-being. The aging of the population and
the costs of expanding medical technologies have placed growing pressure on health care resources. The
private sector, which has long paid for health insurance for a significant proportion of the population, is
confronted by competitive challenges and shifting nationwide economic trends. In the public sector, there
are competing demands for tax dollars to fund health programs that range from the clinical care supported
by Medicare and Medicaid to the population-based approaches employed by public health programs. The
Nation is faced with an apparent series of tradeoffs among priorities, as we seek to make wise investment
decisions to improve health.
There are many ways to resolve these choices, none of them straightforward and none without
compromise. To make the most informed decisions, information is needed about the impact of programs
and interventions, their costs, and what is given up when one program is implemented at the cost of
another. This is true for Government, managed care organizations, providers, employers, and consumers.
As the pressures to control U.S. health care spending have accelerated, the term
"cost-effective" has found an increasingly broad and interested audience. But this term often
has different meanings to groups as disparate as the U.S. Congress, business leaders, managed care
organizations, the pharmaceutical industry, and the public. A more systematic and reliable approach to
determining relative values through CEA is required.
Methods
This report summarizes the work of the Panel on Cost-Effectiveness in Health and Medicine, a
non-Federal expert panel appointed in 1993 by PHS. The panel consisted of 13 scientists and scholars
with expertise in CEA and related methods. The panel was charged with assessing the current state of
the science of CEA and with providing recommendations for the conduct of studies to improve its policy
relevance and utility. The panel met from 1993 through 1995 in collaboration with methodologists from
Public Health Service agencies and the Health Care Financing Administration.
The panel organized itself into nine working groups focusing on specific topics: (1) the role of CEA in
decision making; (2) CEA's theoretical foundations; (3) the appropriate framing of a study; (4) measuring
effectiveness; (5) valuing outcomes; (6) measuring costs; (7) discounting considerations; (8) handling
uncertainty within a study; and (9) guidelines for reporting analyses. Panel members and staff drafted
papers on topics addressing the major controversies in each area. The papers contained recommendations
that were debated by panel members in successive meetings until consensus was reached and formal
recommendations could be issued or, in a few cases, until it became clear that consensus could not be
reached. In areas where consensus was elusive, the report describes the full range of arguments presented.
Findings
CEA furnishes a framework for evaluating the economic and health impacts of different types of
investments and can inform a wide range of policy decisions. The results of a CEA are typically
summarized as a cost-effective ratio, for example, $138,000 per quality-adjusted life-year, in which the
denominator reflects the gain in health from a specific intervention (e.g., years of healthy life gained,
premature births averted, or sight-years gained) and the numerator reflects the cost in dollars of obtaining
that gain.
A well-constructed CEA provides information that helps decisionmakers identify which strategies will best
serve their programmatic and financial objectives. Depending on the type of analysis conducted, a CEA
can compare similar or very different approaches to a given problem, such as strategies for decreasing
cardiovascular disease. For example, State and Federal decisionmakers with responsibility for Medicaid
programs, as well as for public health systems, may wish to know where investments of public dollars
will have the largest impact in decreasing premature death from heart disease. In the private sector, a
managed care organization might wish to know how extending its coverage to include cardiovascular
disease prevention programs would affect the organization's expenditures.
Some decisionmakers may also wish to make comparisons among health conditions and programs; for
example, they may wish to know the efficiencies of investments in interventions to improve neonatal
outcomes relative to investments targeted at decreasing cardiovascular disease in adults. Ideally, CEA
should allow its users to assess the relative value of dissimilar health-producing programs or treatments.
Unfortunately many CEA's are inadequate to these tasks. Reviews of CEA's chronicle a troublesome
failure of CEA practitioners to adhere to basic analytic principles; there also is wide variation in sources
and the quality of information on costs and effectiveness. Concerns about the discretionary nature of the
application of analytic methods and the economic incentives of commercial sponsors of some analyses
have resulted in a decision by at least one medical journal to restrict publication of any CEA where
authors may have a financial conflict of interest.
Methodological inconsistency has often stymied the application of CEA in important public policy
decisions. For example, in the Oregon Medicaid experiment, in which the State legislature attempted to
expand coverage to uninsured Oregonians by basing inclusion of services on the cost-effectiveness of
different treatment-condition pairs, deficiencies in CEA technique and knowledge were partly responsible
for the fact that the initial list of included services lacked credibility. In a more narrow application to
policy needs, a review by the National Cancer Institute of an extensive literature on the cost-effectiveness
of screening mammography found that the study results ranged from a finding that mammography would
be cost saving to a finding that it would cost nearly $84,000 per year of life saved. Confusing results can
lead to confusing policy.
The imprecision attached to the term "cost-effective" stems in part from the variety of masters
the concept serves: purchasers of health care, who use the term to convey a careful assessment of the
value of different health care services; producers of health care technologies and programs, who use the
idea to support marketing claims; and advocates for particular constituencies or illnesses, who use the
term to garner resource investments. But the imprecision also comes from the manner in which methods
of performing CEA have evolved over the past three decades. CEA is an analytic tool whose fundamental
purpose is to provide information to decisionmakers about the relative value of different approaches to
improving health, life expectancy, or both. Architects of the field and analysts who apply CEA methods
come from a number of academic disciplines, including economics, medicine, operations research,
medical sociology, psychology, public health, and ethics. Each discipline brings a particular set of
concepts and a unique language that have been melded in the building of the CEA technique.
Use of Results
The panel has produced a book, Cost-Effectiveness in Health and Medicine, summarizing CEA
methodology with the objective of improving its usefulness to policymakers. The report will be distributed
to Federal decisionmakers and analysts to inform them about the interpretation of existing CEA and ways
to improve studies conducted by and for the Government. The interdepartmental group is planning
broader dissemination and discussion of the panel's work through an international conference. The
conference will explore the implications of the panel's recommendations and will identify next steps in
moving toward the production and use of high-quality comparable CEA.
Agency sponsor:
Office of Disease Prevention and Health Promotion
Federal contact:
Kristine McCoy
202/205-8180
PIC ID: 5619
Performer organization:
Office of Disease Prevention and Health Promotion, Washington, DC
Return to Table of Contents.