Multiple Methods of Assessing Family Therapy Outcomes Using the CAFAS

Multiple Methods of Assessing Family Therapy Outcomes Using the CAFAS
Jeanne M. Slattery, Whitney Smith, Erin Buchenauer, Marci Krapf (Clarion University)
and Terry Bean (Family Links of Clarion County)
The Child and Adolescent Functional Assessment Scale (CAFAS) is a short, parental report of child functioning that describes child functioning at home, in school, and in the community. It has been extensively studied with several clinical populations including a low-functioning population receiving in-home family therapy (Hodges, 1999; Hodges, Doucette-Gates, & Liao, 1999; Hodges & Kim, 2000; Motter, Slattery & Bean, 1999). Hodges reports good test-retest reliability and predictive and criterion-related validities for the CAFAS. It has also been useful for predicting level of service utilization, and acting out behaviors (Hodges & Kim, 2000). Clinical scales have generally been used to identify the nature of problems, while CAFAS totals have been used to identify their severity (Hodges & Kim, 2000).

With treatment costs rising and mental health budgets dropping, intensive psychological treatments must be considered carefully and used only when they are seen as effective. While we are interested in whether change across treatment is statistically significant (Hodges et al., 1999; Motter et al., 1999), we are also interested in whether this change is clinically significant. To answer this question, we compared three approaches to assessing outcomes using family means on CAFAS scales: assessing the statistical significance of change on outcome measures, identifying the magnitude of its change, and determining the percentage of families below clinical cut-offs at termination from therapy.

Method

Participants

Thirty-eight families seen in the Family Links of Clarion County program since May of 1996 with both intake and exit scores on the CAFAS participated in this study. Slattery, Hollis, Bean and Graham (1997) reported most families from this population were under the official poverty level (57% earning less than $10,000 a year and 85% earning below $20,000 a year). Pfendler and his colleagues (1997) reported that at least one parent frequently has a psychiatric diagnosis (36%), or a criminal history (37%). The families are often headed by a single parent (48%), but also include blended (38%) and intact (14%) families. Children in this population have a moderately high rate of prior psychiatric hospitalizations (M = 1.55) or foster care placement (M = .35), and were still judged to be at risk of being removed from their homes. All identified patients ranged between two and 17 years of age.

Identified patients and their families were seen at Family Links of Clarion County, an in-home family therapy program. Teams of professional and paraprofessional therapists saw families at least weekly. In general, families were seen for no more than six to eight months before being discharged or transferred to a less intensive service. Therapy tended to focus on family structures, rules, roles and communication, although a range of techniques - both traditional and less traditional mental health services were used.

Procedure

Assessments from the first 160 children and teens through Family Links were reviewed to identify those with both intake and termination CAFAS scores. The 38 complete cases were entered into StatView. Successful change on the CAFAS over the course of therapy was assessed in three ways. (1) Intake and exit scale scores were compared to determine whether a statistically significant change occurred. (2) Scale scores were compared to clinical descriptors used with the CAFAS (e.g., No Problem). (3) Changes between intake and exit were categorized according to magnitude. All statistical differences were compared to p values of .05.

Results

Using paired t-tests, statistically significant changes between intake and exit were observed on CAFAS totals and six of its eight scales (Role Performance: School/Work, Role Performance: Home, Role Performance: Community, Behavior towards Others, Moods & Emotions, Self-Harm). See Table 1.

Medians on four scales (Role Performance: Community; Self-Harm; Substance Use; and Thinking) were in the nonproblematic range at intake. See Table 2. The median for another scale (Role Performance: School/Work) was in the minimal problem range. By termination, five medians were in the nonproblematic range and the median for all scales was in the minimal problem range. In fact, on all but one scale (Role Performance: Home), at least 75% of the children in this "difficult" population had no more than minimal problems by termination. On this single scale, 68.42% had no more than minimal problems by the end of treatment. The greatest percentage of identified patients moving into the problem-free range was on the Self-harm scale: On this scale, an additional 31.58% of children were problem-free at termination. The most intransigent of the problematic scales was the Role Performance: Community scale, where only an additional 13.15% of identified patients were problem-free at termination.

When averaged across CAFAS scales, 20.72% of identified patients improved at least 20 points. See Table 3. The greatest observed change was on Moods and Emotions (31.58%), the smallest observed change on scales with initial problems was Role Performance: Community (7.89%). On the other hand, an average of 8.22% of children got worse (10+ points). The largest percentage of identified patients got worse on Role Performance: School/Work (15.79%). On the Moods and Emotions and Self-harm scales, few identified patients got worse (5.26%).

Discussion

These results suggest that this type of family therapy was successful even with this difficult population and, further, support the use of the CAFAS in assessing this change. We used three ways of evaluating change: statistical significance of changes observed, percentages dropping below clinical cut-offs, and the magnitude of change between intake and termination. Each of these measures identifies different views of the nature of success. While we certainly hoped that change was statistically significant, this change is meaningless unless it is also clinically significant (Hodges et al., 1999; Motter et al., 1999). That most observed change is in a positive direction and that the 75th%ile on most scales was now in the "Minimal Problem" (10) range, supports this conclusion.

It must be noted, however, that this sample was not a random sample of the children seen in therapy at Family Links. This paper only includes data from families who completed the CAFAS at both intake and exit since May of 1996. Some families dropped out of treatment or were unavailable for assessments. In other cases, therapists failed to administer the CAFAS at either intake or exit. While there may be some bias affecting this sample, superficial inspection of these data suggests that there were no consistent differences between these groups. No group did consistently better or worse than the other.

References

Hodges, K. (1999). Child and Adolescent Functional Assessment Scale CAFAS). In M.E. Maruish (Ed.), Use of psychological testing for treatment planning and outcome assessment (2nd ed.). Mahwah, NJ: Lawrence Erlbaum.

Hodges, K., Doucette-Gates, A., Liao, Q. (1999). The relationship between the Child and Adolescent Functional Assessment Scale (CAFAS) and indicators of functioning. Journal of Child and Family Studies, 8, 109-122.

Hodges, K., & Kim, C. (2000). Psychometric study of the Child and Adolescent Functional Assessment Scale: Prediction of contact with the law and poor school attendance. Journal of Abnormal Child Psychology, 28, 287-297.

Motter, T. A., Slattery, J.M., & Bean, T. (1999, April). Assessment of in-home family therapy outcomes. Poster presented at the annual meetings of the Eastern Psychological Association, Providence.

Pfendler, B. A., Sharrow, W.W., Slattery, J. M., & Bean, T. (1997, April). Validation of the McMaster Family Assessment Device (FAD) in an intensive family therapy program. Poster presented at the annual meetings of the Eastern Psychological Association, Washington, D.C.

Slattery, J. M., Hollis, M. L., Bean, T., & Graham, A. (1997). [Predictors of family therapy outcome.] Unpublished data.

Reference:

Slattery, J. M., Smith, W. N., Krapf, M. L., Buchenauer, E. L., & Bean, T. (2001, August). Multiple methods of assessing family therapy outcomes using the CAFAS. Poster presented at the annual meetings of the American Psychological Association, San Francisco.