The federal welfare reform legislation enacted in August 1996 (the Personal Responsibility and Work Opportunity Reconciliation Act of 1996--P.L. 104-193) eliminates the Aid to Families with Dependent Children (AFDC) entitlement program and replaces it with a block grant to the states. Under the block grant, states will design, implement (by July 1997), and administer their own programs to aid families with dependent children. These programs, however, must satisfy requirements in the new federal law concerning such matters as work requirements for program participants and limits on the amount of time that a family can receive assistance.
Although the former AFDC program was federally administered and imposed requirements on the states, it did offer the states considerable flexibility to deviate from those requirements through a formal process of applying for and receiving federal waivers to implement welfare reform demonstration programs. Provision for waivers of AFDC regulations is included in Section 1115 of the Social Security Act, which was added to the original 1935 act as part of the Social Security Amendments of 1962. Section 17(b) of the Food Stamp Act provides for similar waivers of federal regulations governing the Food Stamp Program. Beginning in 1992, the Bush administration actively encouraged states to initiate reforms to their welfare systems through these waiver provisions. The Clinton administration continued this encouragement and had approved AFDC waivers for 43 states as of August 1996. The new law limits the DHHS waiver authority, but expands Food Stamp waiver authority by giving states more flexibility to experiment with Food Stamp rules under waivers.(1) In addition, the act allows states to keep their waiver programs in place for the duration of the waivers, if the waivers are inconsistent with the requirements of the new law. Because this report is being prepared in September and October 1996, it is not clear how this provision will be interpreted or how many states will keep their waiver programs (and evaluations) in place.
In granting a state's request for waivers of regulations governing the AFDC and Food Stamp programs, the federal government required that the state satisfy explicit terms and conditions. The terms and conditions, which the federal government negotiated with each state, (1) defined the new policies under which the demonstration programs would be run, (2) required the demonstrations to be cost neutral to the federal government, (3) specified requirements for evaluations of the demonstrations, and (4) detailed the cost neutrality methodology and reporting requirements. Accompanying the terms and conditions was a list citing the specific provisions of the Social Security Act and the Food Stamp Act that were being waived and the nature of the waivers. The terms and conditions also required the state to contract with a third party to evaluate the effects of the waivers. In general, the evaluation was to include impact, cost-benefit, and process/implementation analyses.
Several divisions within the U.S. Department of Health and Human Services (DHHS) were responsible for reviewing and approving state Section 1115 waiver applications along with the Office of Management and Budget (OMB).(2) DHHS project officers were responsible for reviewing the evaluation plans that states submitted as part of their waiver application packages and monitoring the conduct of the evaluations. Although they shared certain design features, the evaluations used widely divergent methods to achieve similar research objectives. DHHS determined that it would be useful to obtain a broader perspective, from a source outside of the day-to-day activities of reviewing and monitoring the evaluations, to identify common research issues and to assess the appropriateness of the approaches that the states and the third-party evaluators had adopted. DHHS contracted with Mathematica Policy Research, Inc., (MPR) to conduct this study of evaluations. This report presents our findings and recommendations.
In recent years, the federal government has required most evaluations of welfare reform demonstrations to be based on an experimental design.(3) In welfare reform demonstrations, an experimental design means that some welfare cases are randomly assigned to the demonstration program (these are referred to as experimental group cases) and others are randomly assigned to the pre-reform program (these are referred to as control group cases). A comparison of individuals randomly assigned to the experimental and control groups provides an estimate of the impact of the reforms that controls for factors external to the demonstration, such as changes in the economy. In addition to requiring a common design for the demonstration evaluations, the federal government also specified a core set of outcome measures that must be included in the evaluations, including the employment and earnings of welfare recipients and their rates of marriage and separation.
Because the welfare reform demonstration evaluations generally share a common research design and core set of outcome measures, they repeatedly confront the same research issues.(4) These issues must be appropriately resolved if the evaluations are to produce reliable research results that are useful for assessing and making welfare policy. Given the similarity in many of the programs and in the overall research designs, the diversity of approaches that the evaluators take to specific research issues is somewhat surprising. Some evaluators do not address a particular issue, either because they fail to recognize it as a potential problem or because they recognize it but judge that doing nothing is the most appropriate course of action. Among those who actively respond to an issue, a variety of technical approaches may be used.
DHHS, as it observed the welfare reform demonstration evaluators struggling with a common set of research issues, recognized the need to systematically identify the issues and assess the approaches to them. This led to discussions between DHHS and MPR during fall 1995 that resulted in two conclusions that provided structure for the study of waiver evaluations:
These observations implied two broad objectives for this study. One objective was to identify for the federal government, the states, and the evaluators the principal research issues surrounding the design and execution of the waiver demonstrations and evaluations and to assess the appropriateness of various technical approaches to those issues. The study's other objective was to document those issues and approaches in a sufficiently general way that the information would be useful in designing and conducting welfare demonstrations and evaluations outside of the federal waiver context.
To achieve these objectives, DHHS and MPR agreed that the study would be based on the actual experiences and decisions of states and evaluators conducting waiver demonstrations and evaluations. They also agreed that its focus would be on the estimation of impacts of the demonstrations, rather than the cost neutrality calculations or other federally mandated components of the evaluations. MPR would obtain this information primarily by reviewing documents prepared by the federal government, the states, and the evaluators pertaining to selected waiver demonstrations and evaluations. These documents included the following:
When this study was being conducted in 1996, most of the waiver evaluations had not been under way long enough to have produced interim reports; this limited what could be learned by reviewing project documents. As a supplemental source of information on the evaluations, MPR staff members spoke by telephone with the state officials in charge of the selected evaluations, the directors of the evaluation contracts and (in some states) other researchers working on the evaluations.(5) These calls were made to fill in gaps in information obtained from documents and to obtain the latest information about the status, methods, and findings of the evaluations.
An advisory panel of experts on welfare evaluations was a final source of information for this project.(6) The advisory panel reviewed MPR memos identifying the key issues surrounding the evaluations, reviewed MPR summaries of state demonstrations and evaluations, and met with DHHS and MPR staff to comment on these items and to provide additional information about the evaluations and guidance regarding the course of the study.
Because of budget limitations, the number of welfare reform demonstration evaluations considered in this study was restricted to five. DHHS identified 30 states that were evaluating waiver demonstrations. Several of these states had implemented more than one set of reforms and, consequently, were conducting multiple evaluations. The following criteria guided the selection of five evaluations to study:
We sought evaluations that demonstrated strength in criteria 1 through 3 and diversity in criteria 4 through 6. In making our selections, we also sought to achieve geographic diversity and gave preference to states with larger populations or welfare caseloads; however, we were more willing to compromise on the latter criteria than on the six listed here.
On the basis of these criteria, we selected the following five states and associated waiver demonstrations and evaluations:
Appendix A provides detailed descriptions of the five welfare reform demonstration evaluations that provided the information base for this study. This section summarizes that material.(8)
California's Assistance Payments Demonstration Project (APDP) and Work Pays Demonstration Project (WPDP) were implemented statewide in December 1992 and March 1994, respectively. Four of the state's 57 counties are serving as research counties, where ongoing welfare cases and newly approved applicants for assistance are randomly assigned to experimental or control groups. APDP/WPDP combine cuts in the maximum cash grant (the amount paid to families with no other cash income) with incentives for recipients to obtain jobs and increase their market labor. The incentives, which are common features of welfare reform demonstrations in many states, include removing the time limits on the AFDC $30 and one-third earnings disregards, raising the limit on the assets that a welfare recipient or qualified applicant may own, and eliminating the 100-hour limit on the number of hours per month that the principal earner in a two-parent family may work while the family qualifies for cash assistance (the "100-hour rule").(9) APDP reduces the maximum cash grant by about 11 percent.
To comply with the standard federal terms and conditions for welfare reform waiver demonstrations, California is conducting a random-assignment evaluation of APDP/WPDP. Although California implemented its waivers in phases beginning in December 1992, a single research sample (including both newly approved applicants and recipients) is being used to evaluate the waivers. The subsample of experimental cases is subject to all of the waivers as they are phased in; the subsample of control cases is subject to none of them. Because California's AFDC and Food Stamp programs are administered by the counties, there is no statewide welfare data system, which has presented a major challenge to the evaluation. The state Medicaid data system has been used extensively, and procedures for extracting standardized data from the county data systems have been developed. The Center for Child and Family Policy Studies at UCLA and UC DATA, a research unit at UC Berkeley, are conducting the APDP/WPDP evaluation.
Unlike many of the welfare reform waiver demonstrations, the Colorado Personal Responsibility and Employment Program (CPREP) has not been implemented statewide; instead, it has been operating in 5 of Colorado's 62 counties since June 1994. CPREP is designed to encourage ongoing and new cases who do not have earnings to obtain jobs. It does this primarily by altering the financial incentives imbedded in the AFDC eligibility and benefit formulas. In particular, CPREP expands earned-income disregards, eliminates time limits on those disregards, and raises the eligibility limits on income and asset holdings. In addition, AFDC and child care benefits are combined in a single check and food stamp benefits are cashed out in a second check. Changes in the Job Opportunities and Basic Skills (JOBS) training program are relatively minor.(10)
The CPREP evaluation is being conducted in the five demonstration counties, based on an experimental design. Random assignment of applicant cases to experimental or control status occurs after the determination of their eligibility and benefit levels according to traditional AFDC program (control group) rules, although these rules impose tighter asset limits than under welfare reform. This timing of random assignment minimizes the burden on income maintenance workers. It should not introduce significant bias into the evaluation findings, because most of the waivers (other than those concerning assets) involve the treatment of earned income. CPREP is restricted to cases without initial earnings; therefore, the changes in the treatment of earned income are irrelevant for these cases at the time of application. The evaluation is being based on data from administrative files plus three waves of client survey data. A research unit within the University of Colorado at Denver is evaluating CPREP.
Michigan's welfare reform demonstration, To Strengthen Michigan Families (TSMF), was implemented statewide in October 1992. Significant amendments to the demonstration were implemented two years later. TSMF combines many of the financial incentives of the California and Colorado demonstrations (for example, expansion of earned-income disregards, raising of limits on asset holdings, and elimination of the 100-hour rule for two-parent families) with a stiffening of requirements for participation in the JOBS program. The provisions added in October 1994 include elimination of most JOBS exemptions and an increase in the severity of sanctions for noncompliance with JOBS requirements. Another provision of the 1994 TSMF amendments requires that preschool children be immunized, with a penalty for noncompliance. (This provision is common to the waiver demonstrations in many states.)
A random-assignment evaluation of TSMF is being conducted in four welfare offices located in 3 of Michigan's 83 counties. The research sample includes both ongoing recipients and new applicants for assistance. Unlike the California APDP/WPDP and Colorado CPREP evaluations (but like many other waiver demonstration evaluations), applicant cases in the TSMF evaluation undergo random assignment before determination of eligibility. This permits eligibility to be determined according to the provisions specific to an applicant's treatment or control status. Denied applicants who did not qualify for the alternative State Family Assistance (SFA) program were excluded from the research sample because of data limitations.(11) The TSMF evaluation is based on data from administrative files; a single client survey is also planned. The relatively early implementation date of TSMF means that this evaluation has generated more results than the other four evaluations that make up the information base for this report. Abt Associates, Inc. is the TSMF evaluator.
The Minnesota Family Investment Program (MFIP) was implemented in April 1994 in 7 of the state's 87 counties. The same seven counties are serving as research sites for the evaluation. MFIP consists of three major elements: (1) consolidation of AFDC, Food Stamps, and Family General Assistance into a single benefit check with one set of rules governing eligibility and benefits; (2) expansion of financial incentives to work; and (3) stiffening of requirements for participation in the JOBS program. The work incentives are those found in many state waiver demonstrations: expansion of earned-income disregards, an increase in the limit on asset holdings, and elimination of the 100-hour rule for two-parent families. The new JOBS requirements include fewer JOBS exemptions and a 10 percent reduction in benefits for failure to participate in JOBS when participation is mandatory.
The MFIP evaluation has an experimental design with a unique feature that allows the impacts of distinct subsets of the waiver provisions to be estimated separately. (This contrasts with the typical experimental design for waiver demonstration evaluations, which only supports estimation of the overall impact of the full set of waiver provisions.) The unique feature of the MFIP experimental design is the random assignment of applicant and ongoing cases to four experimental/control groups. The four groups are (1) a full experimental group subject to all MFIP reforms (E1), (2) a partial experimental group subject to all MFIP reforms except those pertaining to the JOBS program (E2), (3) a traditional control group subject to the pre-reform AFDC program and the pre-reform JOBS program (C1), and (4) a second control group subject to the pre-reform AFDC program but with no access to the JOBS program (C2). By comparing outcomes for different pairs of groups, impacts of different program combinations may be estimated. These estimates will be based on administrative data, baseline data forms, and two rounds of follow-up surveys. The baseline data forms were completed by applicants/ongoing cases and program staff just before random assignment. The Manpower Demonstration Research Corporation (MDRC) is evaluating MFIP; Research Triangle Institute (RTI) is conducting the follow-up surveys.
Wisconsin has implemented a number of welfare demonstration programs, authorized by distinct sets of waiver packages. Some of these demonstrations are operating statewide, while others have been implemented only in selected sites. Work Not Welfare (WNW), one of the latter type of demonstrations, has been operating in 2 of the state's 72 counties since January 1995. WNW is meant to radically alter the culture of welfare. Its intent is to affect the administration of welfare and how current and potential recipients and the community at large perceive it. The program seeks the active involvement of local businesses in creating jobs and of community organizations in supporting children and families. It imposes a two-year time limit on the receipt of cash assistance during any four-year period; however, exemptions to this requirement are granted. The program places a strong emphasis on encouraging applicants to enroll only in Food Stamps and Medicaid, thus avoiding the start of the two-year clock limiting the receipt of cash assistance. Those who receive cash grants must participate the requisite number of hours per week in employment or in approved education or training programs, or have their grants reduced in proportion to the shortfall in hours.
DHHS and the state of Wisconsin agreed that an experimental evaluation of WNW was inconsistent with the demonstration's emphasis on communitywide changes in the culture of welfare; it was feared that random assignment would dilute the cultural change sought. Consequently, Wisconsin received approval from the federal government for a quasi-experimental "matched comparison county" design for the WNW evaluation. Each of the two demonstration counties was matched with two comparison counties (for a total of four comparison counties) that have similar economic, demographic, and welfare caseload characteristics but in which WNW is not operating. Impacts on applications, enrollments, caseloads, and other outcomes will be estimated on the basis of aggregate time series data and case-level longitudinal data for the demonstration counties and matched comparison counties. MAXIMUS, a policy research firm, is conducting the WNW evaluation.
(1)A Medicaid waiver process also exists, and this is administered by the Health Care Financing Administration. In some instances Medicaid waivers were part of a comprehensive welfare reform package. In other cases, the Medicaid program has granted waivers for comprehensive Medicaid demonstrations. The new law does not eliminate or restrict the Medicaid waiver process.
(2)Because states typically submitted requests for Food Stamp waivers and AFDC waivers as part of an integrated package, DHHS worked closely with the U.S. Department of Agriculture (USDA) to review and process those requests. The evaluations of these waivers also tended to be highly integrated, with DHHS and USDA again cooperatively monitoring the evaluations.
(3)Here and in the rest of this report (unless otherwise noted), federal government refers to DHHS, OMB, and USDA, and demonstrations refer to AFDC, Medicaid, and Food Stamp reform demonstrations approved under Section 1115 of the Social Security Act and Section 17(b) of the Food Stamp Act.
(4)This report uses the present tense to describe the welfare reform demonstration evaluations because, despite the passage and signing of the new welfare law, most of the evaluations of the previously approved waiver demonstrations were ongoing at the time of the writing of this report in September and October 1996.
(5)These data were collected in the first half of 1996, and reflect the status of the evaluations through June 1996.
(6)The advisory panel members were Daniel Friedlander of the Manpower Demonstration Research Corporation, Robert Moffitt of Johns Hopkins University, Larry Orr of Abt Associates Inc. (on leave at the Office of the Assistant Secretary for Planning and Evaluation, DHHS), and Michael Wiseman of the University of Wisconsin.
(7)This objective was somewhat constrained by Objective 1, however. For instance, only one program testing time-limited welfare was included.
(8)Appendix B describes the material reviewed concerning each evaluation.
(9)The AFDC disregard of the first $90 in earnings per month (to offset work expenses) was not time limited. However, there were time limits of 12 months on the disregard of the next $30 of earnings and 4 months on the disregard of one-third of earnings in excess of $120.
(10)AFDC rules required all nonexempt adult AFDC recipients to participate in the state-administered JOBS employment and training program.
(11)Since most of the TSMF applicants who would be denied AFDC under control group rules but not under reform rules qualified for SFA (and therefore were included in the research sample), this exclusion is not expected to create serious problems for the evaluation.
Home Pages:
Human Services Policy
Assistant Secretary for Planning and
Evaluation
U.S. Department of Health and Human Services
Updated 09/24/01