Evaluation of Family Preservation and Reunification Programs:

Chapter 7:
The Services

[ Main Page of Report | Contents of Report ]

Contents

• 7.1 Caseworker Activities
• 7.2 Social Program Participation
• 7.3 Caretakers’ Reports of Services
• 7.4 Relationship with Caseworker
• 7.5 Caseworkers’ Reports of Services
• 7.6 Contact Data
• 7.7 Summary of Services
• 7.8 Services during the follow-up period
• EndNotes

In this Chapter, we describe the services provided to families in both the family preservation and control groups. We are concerned with describing the experiences of these families in these programs and with making comparisons between the experimental and control groups. We must determine whether the experimental group received more services and more intense services in order to assess the extent to which the tended experimental conditions held. The interpretation of outcome information depends on a demonstration that experimental services were more extensive than “regular” services. We also attempt to compare the services received by the experimental group to the Homebuilders model, to get at questions of the extent to which the model was implemented. Finally, we describe how the families experienced these programs.

Most of the data come from the second interview with caretakers and caseworkers in which we asked questions about services offered and received during the period since random assignment and from the contact forms completed by workers serving both groups. In the follow-up interview a year after random assignment, caretakers were also asked about services received since the post-treatment interview and we report on analyses of those data at the end of this chapter. Comparisons were made between experimental and control group families as they were initially randomly assigned (the "primary analysis";). "Secondary"; analyses, where violations of random assignment and cases receiving minimal service are dropped, were also performed. Tables showing secondary analyses are provided in Appendix H. In most cases, secondary analyses show similar results to those of the primary analyses. Differences are highlighted in footnotes to the following text.

[ Go to Contents ]

7.1 Caseworker Activities

Caretakers were asked to indicate whether the caseworker provided help with a number of specific problems. Table 7-1 shows the number of affirmative responses in each group.

Kentucky. According to caretakers, the most common activities in which workers engaged were discussing discipline and anger management and telling caretakers about other agencies that offer services. On the 19 items on which caretakers were questioned, never did the control group workers reportedly engage in an activity more than the experimental group workers. In the primary analysis, for 7 of the 19 items, experimental group workers reportedly engaged in the activity significantly more often than control group workers (all at p = .01 or less). One additional item showed significant differences in the same direction at p = .05 or lower.⁽¹⁾ A total count of the number of these 19 caseworker activities reported by caretakers also shows significant differences between the experimental and control groups. Caretakers in the experimental group reported an average of 3.9 caseworker activities (n = 148) while caretakers in the control group reported an average of 2.2 caseworker activities (n = 146) (p = .001).⁽²⁾ Caretakers were asked which of the caseworker activities were especially helpful. Experimental group caretakers judged significantly more activities to be helpful than did control group caretakers (1.7 vs. 1.0, p = .001). The services most often cited as helpful by experimental group caretakers were, in order, "the caseworker talked with you about discipline,"; "the caseworker talked with you about how to handle anger,"; "the caseworker told you about other agencies,"; and "the caseworker helped you with money for other things [other than rent, electricity, or phone]."; For control group caretakers the most often cited helpful items were "the caseworker told you about other agencies,"; "the caseworker talking with you about discipline,"; and "the caseworker talked with you about how to handle anger."

New Jersey. The most common activities of workers (according to the caretakers) were discussions of discipline and the handling of anger and referrals to other agencies. In the primary analysis, in 2 of the 19 items control group workers more often engaged in the activity (ignoring those items with 1% differences): discussing getting a better place to live (p = .055) and discussing child care (not significant). For 4 of the 19 items, experimental group workers significantly more often engaged in the activity: discussion of discipline, transportation, discussion of how to handle anger, and discussion of other agencies. A fifth item, discussion of proper feeding of the child, was nearly significant at p = .06.

There were significant differences between the experimental and control groups in the average number of activities reported by caretakers, 3.25 for the experimental group (n = 210) and 2.31 for the control group (n = 134) (p = .001).⁽³⁾ When asked which of these activities were especially helpful, experimental group respondents judged significantly more activities to be helpful than did the control group respondents (1.97 vs. 1.11, p = .0001). The items cited most often as helpful were remarkably similar to those in Kentucky. The services most often cited as helpful by experimental group caretakers were, in order, "the caseworker talked with you about discipline,"; "the caseworker talked with you about how to handle anger,"; "the caseworker told you about other agencies,"; "the caseworker provided transportation,"; and "the caseworker helped you with money for other things [other than rent, electricity, or phone]."; For control group caretakers the most often cited helpful items were exactly the same as in Kentucky: "the caseworker told you about other agencies,"; "the caseworker talking with you about discipline,"; and "the caseworker talked with you about how to handle anger.";

Tennessee. Activities most often engaged in were similar to those in Kentucky and New Jersey: talk about discipline and talk about handling of anger. In 15 of the 19 items, experimental group workers were reported to have engaged in the activity more often than control group workers, although the differences were significant on only 3 of these items (talk about discipline, talk about handling anger, and talk about relations with spouse). For the four items control group workers more often engaged in, differences between the groups were small and not significant.

As in Kentucky and New Jersey, there was a significant difference between the groups in the average number of activities reported by caretakers, 4.6 for the experimental group (n = 80) vs. 2.89 for the control group (n = 37) (p = .02).⁽⁴⁾ Experimental group respondents also judged more activities as especially helpful, an average of 1.34 vs. .84 (p = .04). Again, the items cited most often as helpful were similar to those in Kentucky and New Jersey. Both the experimental and control groups most often listed talk about discipline, talk about how to handle anger, and transportation as most helpful, although experimental group respondents cited these activities far more often.

[ Go to Contents ]

7.2 Social Program Participation

In the second interview, at the completion of family preservation services for the experimental group, caretakers were asked about their participation in the same set of social programs they were asked about in the initial interview (see Chapter 6), except this time they were asked to report their participation since the time of the first interview (Table 7-2).

Kentucky. The proportions of involvement were remarkably similar to those in the first interview with less than a 2 percent change for most programs in the participation of both groups combined. Exceptions to this were a 2.6 percent decrease in participation in WIC services (from 41% to 38%), a 4 percent decrease in the proportion of respondents receiving food stamps (from 67% to 63%), and a 6.3 percent decrease for participation in Head Start or Pre-school programs (from 30% to 24%). Similar to results of the first interview, there were significant differences in experimental and control group participation in the WIC program, with 45 percent of the experimental group reporting participation at the post-treatment interview (n = 148) compared to 32 percent of the control group (n = 146) (p = .021). Differences between experimental and control groups were also found with respect to participation in job training and marriage counseling. For job training, 8.1 percent of the experimental group reported participation (n = 148) compared to 3.4 percent of the control group (n = 146) (p = .085). Seven percent of the experimental group (n = 102) but none of the control group respondents (n = 105) reported participation in marriage counseling (Fisher’s exact p = .006). No significant differences were found with respect to the total number of income support programs or treatment programs in which respondents participated since the time of the first interview.

New Jersey. As in Kentucky, the proportions of involvement were similar to those in the first interview except for community mental health programs (26% in the second interview compared to 31% in the first) and using Head Start or another pre-school program (32% vs. 42%). There were no significant differences between the experimental and control groups. There were no significant differences between groups in the number of income support programs and treatment programs.

Tennessee. Participation in social programs at the post-treatment interview was similar to that at the initial interview, except for declines in use of food stamps (from 72% to 64%), AFDC (from 61% to 50%) and head start/preschool (38% to 34%). There were no significant differences between the experimental and control groups in participation in any program post-treatment, nor were there significant differences in the average number of income support or treatment programs.

[ Go to Contents ]

7.3 Caretakers' Reports of Services

In the second interviews, caretakers were asked if they had received any of a set of specific services in the time since the first interview. Results are shown in Table 7-3.

Kentucky. A significantly greater proportion of caretakers from the experimental group reported receiving such services as day care (19% vs. 5%), transportation (16% vs. 7%), and counseling (52% vs. 35%) (all are significant at p = .05 or less). Reported receipt of medical or dental care was also a higher for the experimental group than for the control group (15% vs. 8%) ( p = .07).⁽⁵⁾ In a separate question, caretakers were asked whether the agency provided homemaker services or the assistance of parent aide. Approximately 2 percent of all caretakers reported having a homemaker and about 3 percent reported receiving assistance from a parent aide, with slightly greater but not significantly different percentages reported in the experimental group as compared to the control group. When caretakers were asked whether they did not receive any services they felt were needed, 27 percent of the control group responded affirmatively compared to 19 percent of the experimental group, a difference that was not statistically significant in either the primary or secondary analyses.

New Jersey. There were no significant differences between the experimental and control groups in receipt of any of these services in the primary analyses.⁽⁶⁾ About 4 percent of the caretakers reported having a homemaker, with no significant difference between the experimental and control groups. Control group caretakers significantly more often reported they did not receive services that were needed (56% vs. 42%, p = .01).

Tennessee. On most of the services items, control group caretakers more often reported receiving the service, these differences being significant at .06 or lower for five of the items listed in Table 7-3. Experimental group percentages were higher for only two items, counseling (reported by far fewer caretakers in Tennessee than in Kentucky and New Jersey) and transportation, neither difference being significant. These rather surprising results, indicating more services for the control group, contradict data on caseworker activities presented above and data from the Tennessee caseworkers (discussed in section 7.5). Control group caretakers more often reported they did not receive services that were needed (39% vs. 24%, p = .1).

[ Go to Contents ]

7.4 Relationship with Caseworker

Table 7-4 shows results from a number of questions in which caretakers were asked about their relationships with caseworkers. In all three states, for most of these questions, caretakers in the experimental group rated their workers significantly more positively than did caretakers in the control group. A greater proportion of experimental group caretakers felt their workers listened to their concerns "most of the time"; other responses were "some of the time"; and "not very often";. Also, a greater proportion of experimental group caretakers felt their workers understood their situation "very well"; as compared to "not very well."; A greater proportion of caretakers in the experimental group reported reaching agreement with their workers on goals "most of the time."

In all three states, experimental group caretakers significantly more often than control group caretakers reported that workers talked with them about problems that were not easy to talk about, helped caretakers to "see your problems"; (p = .1 in Kentucky), and helped them see their good qualities. With regard to the frequency of contact with the workers, in Kentucky, approximately 20 percent of caretakers from both the experimental and control groups indicated they did not see their caseworkers often enough. A greater proportion of caretakers in the experimental group indicated they saw their workers "more often than [they] wanted"; (18% vs. 9%) and a greater proportion of caretakers in the control group indicated they saw their workers "as often as [they] wanted"; (70% vs. 62%). In New Jersey, a greater proportion of caretakers in the experimental group responded that they saw their workers "as often as [they] wanted"; (59% vs. 43%) and a greater proportion of caretakers in the control group responded that they saw their workers "not often enough"; (45% vs. 27%). In Tennessee, more experimental group caretakers said they saw their workers more often than they wanted (27% vs. 19%) and more control group caretakers said they did not see their workers often enough (36% vs. 25%), but the differences between the groups on this item were not significant.

In none of the three states did the groups differ in the extent to which they called workers when they had problems.

[ Go to Contents ]

7.5 Caseworkers' Reports of Services

In the second interview, caseworkers were asked whether they had made referrals to any of 25 services, such as childcare, homemaker services, income programs, treatment programs of various sorts, and health care. Results from these 25 items are shown in Table 7-5.

Kentucky. Caseworkers for the experimental group reported helping their clients with an average of 5.0 of these services, while caseworkers from the control group reported helping their clients with an average of 3.2 of these services (p = .001).⁽⁷⁾ In the primary analyses, 7 specific services were provided significantly more often to the experimental group than to the control group (significance levels were all at p = .05 or less). These services include health care, health assessment, housing financial assistance, other housing assistance, emergency financial assistance, household management, and homemaker services. A greater proportion of caseworkers from the experimental group also reported providing outpatient mental health counseling (p = .07) and selected the unspecified category of "other"; services provided (p = .06). No services were provided significantly more often to the control group than the experimental group.⁽⁸⁾

New Jersey. Experimental group caseworkers reported that on average their clients were provided 3.2 of these services, while control group families were provided 2.3, a difference significant at .001.⁽⁹⁾ When individual services are examined, there were six services that were provided significantly more often to the experimental group (education services, parent training, outpatient mental health, recreational services, family planning, and household management). Three services were provided significantly more often to the control group: drug treatment, emergency shelter, and homemakers.⁽¹⁰⁾

Tennessee. Experimental group caseworkers reported providing an average of 3.2 services, compared with 1.6 for the control group, significantly different at .001. Six individual services were significantly more often provided to experimental group families (drug treatment, help with education, parent training, health care, emergency financial assistance, and recreational services).⁽¹¹⁾ No services were significantly more often provided to the control group.

[ Go to Contents ]

7.6 Contact Data

Workers serving clients in both the experimental and control groups were asked to complete a one-page contact report following each in-person contact with a family member (see Appendix K). The report was a simple check-off form, asking about who was present in the visit and about the content of the conversation. Although these forms were quite simple and easy to fill out, it proved difficult to get workers to complete them. We implored workers who did not fill out these forms to do so, and we have at least one on a fair proportion of the cases. However, it is likely that for at least some cases on which we have forms that we do not have them for all of the contacts. We are unable to determine how many contacts occurred for which we have no forms. Furthermore, the quality of information may be affected by the fact that some of the forms were submitted after many calls from our office and after long delay. The following analyses were limited to those families with contact reports. Only "primary"; analyses are reported for contact reports.

Some data on contact forms are presented in Table 7-6. Forms were received on between 71 percent and 91 percent of the experimental groups and between 51 percent and 71 percent of the control groups. It should be noted that the lower rate for control group cases is partially due to the fact that there was no contact in the four weeks after the date of random assignment, the period of time for which we requested contact forms for the control group (a period comparable to the 4 week period of services for the experimental group. On average, more contact forms were submitted for the experimental group than for the control group. In addition to the overall number of reports submitted, in all three states the experimental group received significantly more home visits, visits with caretakers, visits with the other parent, and visits with children. The experimental group workers were more likely to involve other adults in the family, non-family members, and other workers. As experimental group families received significantly more contacts than the control group families, they also received significantly more individual activities. For both experimental and control families, in all three states the most common concrete service was the provision of transportation. Purchasing food, child care, and providing clothing, furnishings, and supplies were also common forms of concrete services.

Contact forms also captured general information about the topic of discussion, counseling, or instruction. In all three states, for both experimental and control families, the most common topics of discussion were the discipline of children, goals, and the caretaker’s interaction with the children. Other common topics were the child’s anger management and supervision of children.

Experimental group contacts. We examined further the contact forms for the experimental group to explore some issues in the adherence of programs to the Homebuilders model of service, subscribed to in all three states (see Table 7-7). In addition to other critical elements of family preservation, the Homebuilders model specifies that workers should provide an in-home contact within 72 hours of referral, and family preservation workers should be available 7 days per week. Substantial contact should take place within the first week; Kinney, Haapala, and Booth suggest that the typical case receive 11 hours of service in that time.⁽¹²⁾ Concrete services are also an important component of service, particularly early in the case.

In Kentucky, of the 124 experimental families with submitted contact forms, 55 (44%) received an in-home contact within 72 hours, 97 (78%) had contact in the first week. Those 97 families had an average of 5.1 hours of face-to-face contact in the first week. Regarding availability of worker, 18 (1%) of contacts occurred on either Saturday or Sunday. Finally, 34 (27%) of the experimental families received some type of concrete service within the first seven days.

In New Jersey, of the 250 experimental families with submitted contact forms, 73 percent received an in-home contact within 72 hours, 219 (88%) in the first week, and those families had an average of 6.5 hours of face-to-face contact in the first week. Regarding availability of the worker, only 196 (6%) of submitted contacts occurred on Saturday or Sunday. Finally, 38 percent of the experimental families received some type of concrete service within the first seven days.

In Tennessee, of the 73 experimental families with submitted contact forms, 42 (57%) received an in-home contact within 72 hours, 53 (73%) had contact in the first week. We are able to calculate hours of contact for 45 of these 53 cases and these cases had an average of 8.3 hours of face-to-face contact in the first week. Regarding availability of worker, 60 (9%) contacts occurred on either Saturday or Sunday. Finally, 21 (29%) of the experimental families received some type of concrete service within the first seven days.

These data seem to indicate that some "structural"; aspects of the Homebuilders model (contact within 72 hours of referral, amount of contact in the first week, services provided at all hours, including weekends, and concrete services early in the case) are not always upheld in these states. However, it is not possible to draw firm conclusions about this, because of issues in the quality of the contact form data discussed earlier.

[ Go to Contents ]

7.7 Summary of Services

In all three states, the caretaker interview, the caseworker interview, and the contacts data generally confirmed the expectation that the experimental group would receive more services and more intensive services that the control group. An exception is the caretaker reports of services received in Tennessee. Table 7-8 shows a summary of those items on which there were significant differences between experimental and control groups on the primary analyses in any state.

In all three states, the number of experimental group caseworker activities reported by caretakers was greater than that reported by control group respondents, and this was also true of "helpful"; caseworker activities. As for specific caseworker activities, experimental group workers in all three states were more likely to provide transportation, talk about discipline, and talk about how to handle anger. In all three states, the number of specific services received by experimental group families was greater than the number received by control group families. Contact from data confirmed that there was far more contact with experimental group families. The most common concrete service reported on contact forms was transportation; the most common topics of discussion were discipline of children, goals, and caretaker’s interaction with children.

It is of interest that transportation is a theme in a number of sources of information about services. We do not have information on where workers were transporting parents and children, but it is evident that needs for transportation are common in these families, needs that workers are able to respond to. This is a concrete service that provides immediate help and builds relationships. Furthermore, workers told us that they often use the time in the car to good advantage in discussing problems of the family.

The most common subject of counseling, interaction with children and in particular their discipline, reflect central problems in these families, problems of paramount concern to the child protective system. It is, therefore, not surprising that workers were focused on altering parent-child interaction patterns.

Experimental group caseworkers in all three states were more often reported to have talked about difficult issues, to have helped the caretaker to see her/his good qualities and problems, and to have "understood your situation.";

Insofar as there are differences between groups, we can be reasonably sure that the experimental conditions held. Conclusions regarding adherence to the Homebuilders model are less clear cut. Families did not always receive contact within 72 hours, fewer than expected contacts occurred in the first week of the program, and few contacts occurred on weekends. There was relatively little provision of concrete services early on. These results are not entirely surprising. Social programs are never implemented precisely as they are designed. Perhaps the test of a program conception is that it achieves desired outcomes even when it is not implemented exactly as intended.

[ Go to Contents ]

7.8 Services during the follow-up period

When caretakers were interviewed a year after random assignment, they were asked some of the same questions about services received, this time since the last interview (since the end of family preservation services for the experimental group and during a comparable period for the control group). Tables 7-9, 7-10, and 7-11 show analyses of these questions.

Caseworker Activities. Caretaker reports of caseworker activities since the post-treatment interview are shown in Table 7-9. In all three states, the experimental group respondents reported more caseworker activities than did control group respondents. In Kentucky, there were five activities the experimental group caretakers significantly more often reported: help with money for rent, electricity, or phone; help with money for other things; transportation; advice on getting medical care; and information about other agencies. In New Jersey, there were two such activities, help in cleaning the house and talk about how to handle anger, with a third item nearly significant, information about other agencies (p = .06). In Tennessee, there were seven activities significantly more often reported by experimental group caretakers: help with money for other things, transportation, talk about discipline, advice on substance abuse, help with relations with spouse, talk about how to handle anger, and information about other agencies. Differences between the groups were not as great as those reported for the treatment period, as is to be expected, since the treatment did not continue during this period.

Participation in Social Programs. As indicated in Table 7-10, there were no significant differences between the experimental and control groups in any state in involvement in social programs during the post-treatment period.

Caretaker report of services. Table 7-11 indicates that there was only one service in the three states on which there was a significant difference between groups in receipt post-treatment; in Tennessee more control group respondents reported having a parent aide. In Kentucky and New Jersey, the proportions of the two groups receiving each service are remarkably similar. Control group families in Tennessee more often received a couple of other services, but the differences were not significant. For the most part, the superiority of the control group in Tennessee in receipt of services observed at the post-treatment interview dissipated at the time of the follow-up interview.

Summary of Post-treatment Services. A summary of the significant differences between experimental and control groups on report of services at the follow-up caretaker interview is shown in Table 7-12. In the questions about caseworker activities, there is some indication that experimental group families received more services during the post-treatment period. Since caretakers were asked about the period of time following the last interview, we assume that for experimental group respondents the activities were undertaken by workers other than family preservation workers, perhaps workers in the public agency or workers in other private agency programs to which they might have been referred. Hence, the data may be taken as indicating receipt of somewhat more services by the experimental group families after the end of family preservation services, in accordance with the goal of these programs to connect families with ongoing services. However, this finding was not confirmed by data on social programs or services. It is possible that the finding also reflects something that we have often heard from public agency workers working with family preservation programs, that the family preservation involvement gave them more information about the family and enabled them to plan better for services after family preservation.

[ Go to Contents ]

Endnotes

(1)  The results of the secondary analyses show slightly greater differences between the experimental and control groups. Here, 8 of the 19 items show significant differences in favor of the experimental group at p = .01 or lower, and an additional 2 items show significant differences in the same direction at p = .05 or lower. See Appendix H-1. [Back To Text]

(2)  These differences were even larger when violations and minimal service cases were excluded from the analyses (4.6 vs. 2.1, ns of 138 and 110, p = .001). [Back To Text]

(3) Differences were even greater when the violations and minimal service cases were excluded (3.57 vs. 1.90, ns of 181 and 115, p = .0001). [Back To Text]

(4) In the secondary analysis, the experimental group had an average of 4.99 activities, compared to 2.88 for the control group. [Back To Text]

(5) When violations and minimal service cases are excluded from the analyses, the differences remained significant and most p-values decreased even further. The secondary analyses showed a significantly greater proportion of the experimental group caretakers report attendance at parent education classes (p = .05). See Appendix H-3. [Back To Text]

(6) In the secondary analysis (dropping violations and minimal service cases) there was a difference on one item: experimental group caretakers more often received counseling (59% vs. 46%, p = .03). See Appendix H-3. [Back To Text]

(7) When violations and minimal services cases were excluded, the difference between the groups was even larger (5.8 vs. 3.1, p = .001). [Back To Text]

(8)  In the secondary analyses, excluding violations and minimal service cases, 14 services were provided significantly more often to the experimental group than to the control group (again, significance levels were all at p = .05 or less). In addition to the 9 primary analysis items showing differences in favor of the experimental group (seven significant items plus outpatient mental health counseling and the category of "other";), these were: food stamps (p = .02), parent training (p = .002), and recreational services (p = .002). Again, in the secondary analyses, no services were provided significantly more often to the control group than the experimental group. See Appendix H-5. [Back To Text]

(9) The difference between groups was even larger when violations and minimal service cases were excluded: 3.47 vs. 1.94 (p < .0001).[Back To Text]

(10) Excluding violations and minimal service cases, only one service was provided significantly more often to the control group, emergency shelter, while nine services were significantly more often provided to the experimental group, the above six plus legal aid, emergency financial assistance, and self help groups. [Back To Text]

(11)  In the secondary analysis, there was one additional service provided significantly more often to the experimental group: household management. [Back To Text]

(12) Jill Kinney, David Haapala, and Charlotte Booth. (1991). Keeping families together: The Homebuilders model. New York: Aldine de Gruyter. [Back To Text]

Where to?

Top of Page
Contents
Main Page of Report

Home Pages:
Human Services Policy (HSP)
Assistant Secretary for Planning and Evaluation (ASPE)
U.S. Department of Health and Human Services (HHS)

Updated: 2/22/01