Functional Assessment Systems : CAFAS® Overview of Reliability and Validity

Assessments

Software

Technical Requirements

Training and Consultation

Home > CAFAS® Overview of Reliability and Validity

CAFAS® Overview of Reliability and Validity

Kay Hodges, Ph.D.

The Child and Adolescent Functional Assessment Scale (CAFAS: Hodges, 2000a; 2000b), assesses the degree of impairment in youth with emotional, behavioral, psychiatric, or substance use problems. The CAFAS provides an objective, comprehensive assessment of a youth’s needs that is sensitive to change to over time, making it the most widely used outcome measure available. Using information collected during a routine clinical interview, the practitioner selects items that describe the youth’s problematic behaviors, as well as strengths and goals. This is done for eight life domains: At School, At Home, in the Community (delinquency), Behavior Toward Others, Moods/emotions, Self Harm, Substance Use, and Thinking (assessing irrationality). A Total Score and subscale scores are generated, with higher scores indicating greater impairment in day-to-day functioning. Therefore, as treatment progresses, lower CAFAS total and subscale scores indicate improvement.

Common applications of the CAFAS include assessing the needs of youth who access services across the System of Care (mental health, child welfare and social services, youth & adolescent justice, education, prevention, and community-based programs) and evaluating outcomes for programs, Evidence-Based Treatments (EBTs) and Evidence Informed Practices (EIPs). The CAFAS has been translated into French, Spanish, and Dutch.
Originally developed in 1989 and supported by over 20 years of research and 80 published articles, the CAFAS is a robust, psychometrically sound measure. Reliability studies have demonstrated that the CAFAS has satisfactory internal consistency and interrater reliability (Hodges & Wong, 1996), as well as test-retest reliability (Hodges, 1995). Studies investigating the validity of the CAFAS have included evaluations of concurrent and predictive validity.

Concurrent Validity. Studies show that the CAFAS is able to differentiate between youth being served at varying levels of intensity of care (e.g., youths in inpatient facilities scored significantly higher than youth receiving home-based and out-patient services) (Hodges & Wong, 1996); youths in different living arrangements (e.g., youths living at home or in foster care were less impaired than youths in residential treatment facilities) (Hodges, Doucette-Gates & Liao, 1999); youth with varying severity of psychiatric diagnoses, (e.g., youths with more serious psychiatric disorders were more impaired than youth diagnosed with less serious disorders [e.g., adjustment, anxiety]) (Hodges, Doucette-Gates, & Liao, 1999); and youths with varying number of psychiatric diagnoses (e.g., youth with a psychiatric disorder were more impaired than youth with no diagnosis and youth with more than one disorder were more impaired, with an incremental increase in the CAFAS total score observed for each additional disorder) (Ezpeleta, Granero, de la Osa, Domenech, & Bonillo, 2006). In addition, higher impairment scores on the CAFAS have been associated with problems in social relationships (Hodges & Wong, 1996), involvement with youth & adolescent justice (Doucette-Gates, Hodges, & Liao, 1998; Hodges, Doucette-Gates & Liao, 1999; Hodges & Wong, 1996), school related problems (Doucette-Gates, Hodges, & Liao, 1998; Hodges, Doucette-Gates, & Liao, 1999; Hodges & Wong, 1996), and child and family risk factors (Manteuffel, Stephens, & Santiago, 2002; Walrath, Mandell, Liao, Holden, DeCarolis, Santiago, & Leaf, 2001).

Predictive Validity. CAFAS scores at intake predicted subsequent episodes of care (Hodges, Doucette-Gates, & Kim, 2000), care that is more restrictive (Doucette, Hodges, & Laio, 1998; Hodges, Doucette-Gates, & Kim, 2000), and cost of services (Hodges & Wong, 1997; Doucette, Hodges, & Laio, 1998). In addition, intake CAFAS score was predictive of future contacts with the law and school attendance (Hodges & Kim, 2000). In a study with youth in a youth & adolescent justice residential center, higher CAFAS scores at discharge predicted recidivism during the year after discharge (Quist & Matshazi, 2000).

Assessing Outcomes. Demonstrating sensitivity to change over time is a requirement of all outcomes measures. Studies have established that the CAFAS is sensitive to assessing the degree and rate of change over time. In a large evaluation study conducted at Fort Bragg, statistically significant reduction in impairment was observed from intake to both 6 and 12 months, with large to moderate effect sizes (Hodges & Wong, 1996; Hodges, Wong, & Latessa, 1998; Hodges, 1999). A second large evaluation study of over 60 communities, which were part of the Center for Mental Health Services (CMHS) System of Care Initiative, found significant improvement in functioning from intake to 6 months (Hodges, Doucette-Gates, & Liao, 1999), and from intake to 2 years post intake (Manteuffel, Stephens, & Santiago, 2002). Outcome results for individual CMHS-funded sites have been reported, revealing significant reduction in youth impairment (Resendez, Quist, & Matshazi, 2000; Rosenblatt & Furlong, 1998). Additionally several smaller evaluations studies have also demonstrated that the CAFAS is sensitive to assessing outcomes. An evaluation of a school-based intensive mental health program found significant pre to post differences, with large to moderate effect sizes (Vernberg, Jacobs, Nyre, Puddy, & Roberts, 2004). In an outcomes study of youth with Serious Emotional Disturbance (SED) served by public mental health in Hawaii, a statistically significant improvement in functioning was observed over a three-year period. The authors were also able to demonstrate that this improvement was achieved in less time (i.e., the average length of time to achieve the same results was reduced by 40% to 60%) (Daleiden, Chorpita, Donkervoet, Arensdorf, & Brogan, 2006). In an evaluation of a home-based model for treating youth with SED, the authors found that youth significantly improved every four months, over the full course of 24 months of treatment (Williams & Sherr, 2008).

Generalizability. The CAFAS has been shown to be a robust indicator of youth’s functional impairment in a wide array of service settings, cultural contexts, and among youth with diverse backgrounds. A sample of the settings and populations that routinely use the CAFAS include: at-risk youth who are simultaneously served by multiple agencies, (e.g., mental health, schools, youth & adolescent justice, and child welfare) (Walrath, dosReis, Miech, Liao, Holden, DeCarolis, Santiago, & Leaf, 2001; Walrath, Nickerson, Crowel, & Leaf, 1998; Walrath, Sharp, Zuber, & Leaf, 2001; Walrath, Mandell, & Leaf, 2001); delinquent youth and youth served by youth & adolescent justice (Abram, Choe, Washburn, Romero, & Teplin, 2009; Quist & Matshazi, 2000; Rosenblatt, Rosenblatt, & Biggs, 2000; Timmons-Mitchell, Bener, Kishna, & Mitchell, 2006); youth placed in foster care, residential settings, and youth removed from their homes due to abuse and neglect (Reifsteck, 2005; Walrath, Ybarra, Holden, Liao, Santiago, & Leaf, 2003; Zima, Bussing, Crecelius, Kaufman, & Berlin, 1999), and youth referred for specialized educational services to assess the impact of school-based mental health programs (Rosenblatt & Rosenblatt, 1999; Roy, Roberts, Vernberg, & Randall, 2008; Vernberg, Jacobs, Nyre, Puddy, & Roberts, 2004).

Effectiveness of Evidence Informed Practices. The CAFAS has been used to evaluate Evidence Based Treatments (EBTs) to determine if the EBT implemented in a local community is effective. The CAFAS has also been used to evaluate locally-derived interventions, which may then become Evidence Informed Practices (EIPs). The CAFAS has formally been used to evaluate the implementation and effectiveness of several EBTs, including Multisystemic Therapy, where a randomized controlled trial showed greater improvement for the treatment group (Timmons-Mitchell, Bener, Kishna, & Mitchell, 2006) and Parent Management Training – Oregon, where cases receiving PMTO made significant improvement on all subscales (Hodges, Wotring, Forgatch, Lyon, & Spangler, 2008). In addition, the CAFAS was used to demonstrate the effectiveness of well-defined clinical practices aimed at addressing the needs of youth with SED. A home-based program, developed by the Family Guidance Service in Michigan, resulted insignificantly greater improvement, compared to other comparable youth served at other sites, using propensity analysis (Hodges & Grunwald, 2005). The CAFAS has been used to evaluate Children’s Psychosocial Rehabilitation (CPSR), which is an intensive community-based treatment for children with SED. CPSR implemented at a large Idaho agency demonstrated significant improvement on CAFAS total and subscale scores (Williams, 2009a). The authors also determined that there was a dosage effect (i.e., more service hours per week were associated with greater improvement) and studied the rate of improvement over time (Williams, 2009b).

In summary, the psychometric properties of the CAFAS are robust; with published accounts substantiating that is has evidence of reliability, concurrent validity, predictive validity, and sensitivity to change. Furthermore, the widespread use of the CAFAS supports its generalizability, demonstrating that the CAFAS can be used for a wide variety of youth and cultural contexts. It is an effective tool for evaluating EBTs and EIPs. The CAFAS remains the gold standard for assessing and tracking outcomes for youth.

Key References

Abram, K. M., Choe, J. Y., Washburn, J. J., Romero, E. G., & Teplin, L. A. (2009). Functional Impairment in Youth Three Years after Detention. Journal of Adolescent Health, 44(6), 528-535.

Barwick, M. A., Schmidt, J., & Hodges, K. (2004). Culturally competent evaluation: Clinical considerations for rating the Child and Adolescent Functional Assessment Scale (CAFAS) with Aboriginal children and youth. Toronto, ON: The Hospital for Sick Children.

Boydell, K. M., Barwick, M., Ferguson, H. B., Haines, R . (2005). A feasibility study to assess service providers’ perspectives regarding the use of the child and adolescent functional assessment scale in Ontario. The Journal of Behavioral Health Services & Research, 32(1), 105-109.

Breda, C.S. (1996). Methodological issues in evaluating mental health outcomes of a children’s mental health managed care demonstration. The Journal of Mental Health Administration, 23(l), 40-50.

CAFAS in Ontario at the Hospital for Sick Kids. (2008). Level of Functioning Outcomes for Children and Youth Receiving Mental Health Treatment. Ontario’s Children with Mental Health Needs 2008 Report.

Daleiden, E. L.& Chorpita, B. F. (2005). From Data to Wisdom: Quality Improvement Strategies Supporting Large-scale Implementation of Evidence-Based Services. Child and Adolescent Psychiatric Clinics of North America, 14(2), 329 – 349.

Daleiden, E. L., Chorpita, B. F., Donkervoet, C., Arensdorf, A. M., & Brogan, M. (2006). Getting Better at Getting Them Better: Health Outcomes and Evidence-Based Practice Within a System of Care. Journal of American Academy of Child and Adolescent Psychiatry, 45(6), 749-756.

Doucette-Gates, A., Hodges, K., & Liao, Q. (1998, March). Using the CAFAS: Examining child outcomes and service use patterns. Paper presented at the 11th Annual Research Conference on a System of Care for Children’s Mental Health: Expanding the Research Base, Tampa, FL.

Ezpeleta, L., & Toro, J. (2009, September). Associations among anxiety disorders and non-anxiety disorders, functional impairment and medication in children and adolescents. Journal of Psychopathology and Behavioral Assessment, 31(3), 168-177.

Ezpeleta, L., Granero, R., de la Osa, N., Domenech, J. M., & Bonilla, A. B. (2006). Assessment of Functional Impairment in Spanish Children. Applied Psychology: An International Review, 55(1), 130-143.

Forgatch, M.S & DeGarmo, D.S. (1999). Parenting through change: An effective prevention program for single mothers. Journal of Consulting and Clinical Psychology,67(5), 711-724.

Grimbos, T. & Granic, I. (2009).Changes in maternal depression are associated with MST outcomes for adolescents with co-occurring externalizing and internalizing problems. Journal of Adolescence, 32(6), 1415-23.

Henggeler, S. W., Schoenwald, S. K., Borduin, C. M., Rowland, M. D., & Cunningham, P. B. (1998). Multisystemic treatment for antisocial behavior in children and adolescents. New York: Guilford Press.

Hodges, K. (1995). Psychometric study of a telephone interview for the CAFAS using an expanded version of the scale. Paper presented at the 8^th annual research conference: A System of Care for Children’s Mental Health: Expanding the Research Base, University of South Florida, Tampa, FL.

Hodges, K. (1999). Child and Adolescent Functional Assessment Scale (CAFAS). In M. E. Maruish (Ed.), The Use of Psychological Testing for Treatment Planning and Outcome Assessment. (pp. 631-664). Mahwah, NJ: Lawrence Erlbaum Associates.

Hodges, K. (2000a). Child and Adolescent Functional Assessment Scale. Ypsilanti, MI: Eastern Michigan University.

Hodges, K. (2000b). Child and Adolescent Functional Assessment Scale Self-Training Manual. Ypsilanti, MI: Eastern Michigan University.

Hodges, K. (2004a). Child and Adolescent Functional Assessment Scale (CAFAS). In M. E. Maruish (Ed.), The use of psychological testing for treatment planning and outcome assessment. (3^rd ed., pp. 405-441). Mahwah, NJ: Lawrence Erlbaum Associates.

Hodges, K. (2004b). Using assessment in everyday practice for the benefit of families and practitioners. Professional Psychology: Research and Practice, 35(5), 449-456.

Hodges, K. (2005). Child and Adolescent Functional Assessment Scale (CAFAS). In T. Grisso, G. Vincent, & D. Seagrave (Eds.), Mental health screening and assessment for youth & adolescent justice. (pp. 123-136). New York, NY: Guilford Publications, Inc.

Hodges, K., Doucette-Gates, A., & Kim, C. S. (2000). Predicting service utilization with the Child and Adolescent Functional Assessment Scale in a sample of youths with serious emotional disturbance served by Center for Mental Health Services-funded demonstrations. The Journal of Behavioral Health Services & Research, 27(1), 47-59.

Hodges, K., Doucette-Gates, A., & Liao, Q. (1999). The relationship between the Child and Adolescent Functional Assessment Scale (CAFAS) and indicators of functioning. Journal of Child and Family Studies, 8(1), 109-122.

Hodges, K., & Grunwald, H. (2005). The use of propensity scores to evaluate outcome for community clinics: Identification of an exceptional home-based program. Journal of Behavioral Health Services & Research, 32, 292-303.

Hodges, K., & Gust, J. (1995). Measures of impairment for children and adolescents. Journal of Mental Health Administration, 22, 403-413.

Hodges, K., & Kim, C. S. (2000). Psychometric study of the Child and Adolescent Functional Assessment Scale: Prediction of contact with the law and poor school attendance. Journal of Abnormal Child Psychology, 28(3), 287-297.

Hodges, K., & Wong, M. M. (1996). Psychometric characteristics of a multidimensional measure to assess impairment: The Child and Adolescent Functional Assessment Scale. Journal of Child and Family Studies, 5(4), 445-467.

Hodges, K., & Wong, M. M. (1997). Use of the Child and Adolescent Functional Assessment Scale to predict service utilization and cost. The Journal of Mental Health Administration, 24(3), 278-290.

Hodges, K., Wong, M. M., & Latessa, M. (1998). Use of the Child and Adolescent Functional Assessment Scale (CAFAS) as an outcome measure in clinical settings. The Journal of Behavioral Health Services & Research, 25(3), 325-336.

Hodges, K., & Wotring, J. (2000). Client typology based on functioning across domains using the CAFAS: Implications for service planning. The Journal of Behavioral Health Services & Research, 27(3), 257-270.

Hodges, K., & Wotring, J. (2004). Role of monitoring outcomes in initiating implementation of evidence-based treatments at the state level. Psychiatric Services, 55(4), 396-400.

Hodges, K., Wotring, J., Forgatch, M. S., Lyon, A., Spangler, J. (2008, March). Outcome indicators for youth’s functioning and parent’s child management skills: Results from PMTO training. In C. Newman, C. J. Liberton, K. Kutash, & R. M. Friedman (Eds.), The 21^st annual research conference proceedings, A system of care for children’s mental health: Expanding the research base (pp. 55-56). Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute, Research and Training Center for Children’s Mental Health.

Hodges, K., Xue, Y., & Wotring, J. (2004). Use of the CAFAS to evaluate outcome for youths with severe emotional disturbance served by public mental health. Journal of Child and Family Studies, 13(3), 325-339.

Hodges, K., Xue, Y., & Wotring, J. (2004). Outcomes for children with problematic behavior in school and at home served by public mental health. Journal of Emotional and Behavioral Disorders, 12(2), 109-119.

Holden, E. W., Friedman, R. M., & Santiago, R. L. (2001). Overview of the national evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program. Journal of Emotional and Behavioral Disorders, 9(1), 4-12.

Manteuffel, B., Stephens, R., & Santiago, R. (2002). Overview of the national evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program and summary of current findings. Children’s Services: Social Policy, Research, and Practice, 5(1), 3-20.

Quist, R., & Matshazi, D. (2000). The Child and Adolescent Functional Assessment Scale (CAFAS): A dynamic predictor of youth & adolescent recidivism. Adolescence, 35(137), 181-192.

Rosenblatt, J., & Furlong, M. (1998). Outcomes in a system of care for youths with emotional and behavioral disorders: An examination of differential change across clinical profiles. Journal of Child and Family Studies, 7(2), 217-232.

Rosenblatt, J.A., & Rosenblatt, A. (1999). Youth functional status and academic achievement in collaborative mental health and education programs: Two California care systems. Journal of Emotional and Behavioral Disorders, 7(1), 21-30.

Rosenblatt, J.A., Rosenblatt, A., & Biggs, E.E. (2000). Criminal behavior and emotional disorder: Comparing youth served by the mental health and youth & adolescent justice systems. The Journal of Behavioral Health Services & Research, 27(2), 227-237.

Roy, K., Roberts, M., Vernberg, E., & Randall, C. (2008). Measuring Treatment Outcome for Children with Serious Emotional Disturbances: Discriminant Validity and Clinical Significance of the Child and Adolescent Functioning Assessment Scale. Journal of Child and Family Studies, 17(2), 232-240.

Timmons-Mitchell, J., Bender, M. B., Kishna, M. A., Mitchell, C. C. (2006). An Independent Effectiveness Trial of Multisystemic Therapy with Juvenile Justice Youth. Journal of Clinical Child and Adolescent Psychology, 35(2), 227-236.

Vernberg, E. M., Jacobs, A. K., Nyre, J. E., Puddy, R. W., & Roberts, M. C. (2004). Innovative Treatment for Children with Serious Emotional Disturbance: Preliminary Outcomes for a School-Based Intensive Mental Health Program. Journal of Clinical Child and Adolescent Psychology, 33(2), 359-365.

Walrath, C., dosReis, S., Miech, R., Liao, Q., Holden, W., DeCarolis, G., et al. (2001). Referral source differences in functional impairment levels for children served in the Comprehensive Community Mental Health Services for Children and Their Families Program. Journal of Child & Family Studies, 10(3), 385-397.

Walrath, C., Mandell, D., & Leaf, P. (2001). Responses of children with different intake profiles to mental health treatment. Psychiatric Services, 52(2), 196-201.

Walrath, C. M., Mandell, D. S., Liao, Q., Holden, E. W., DeCarolis, G., Santiago, R. L., et al. (2001). Suicide attempts in the “comprehensive community mental health services for children and their families” program. Journal of the American Academy of Child and Adolescent Psychiatry, 40(10), 1197-1205.

Walrath, C.M., Nickerson, K.J., Crowel, R.L., & Leaf, P.J. (1998). Serving children with serious emotional disturbance in a system of care: Do mental health and non-mental health agency referrals look the same? (1998). Journal of Emotional and Behavioral Disorders,6(4), 205-213.

Walrath, C.M., Sharp, M.J., Zuber, M., & Leaf, P. (2001). Serving children with SED in urban systems of care: Referral agency differences in child characteristics in Baltimore and the Bronx. Journal of Emotional and Behavioral Disorders, 9(2), 94-105.

Williams, N. J. (2009a). Preliminary Evaluation of Children's Psychosocial Rehabilitation for Youth with Serious Emotional Disturbance. Research on Social Work Practice, 19(1), 5.

Williams, N. (2009b). Dose-Effect of Children’s Psychosocial Rehabilitation on the Daily Functioning of Youth with Serious Emotional Disturbance. Child and Youth Care Forum, 38(6), 273-286.

Wotring, J., Hodges, K., Xue, Y., & Forgatch, M. (2005). Critical ingredients for improving mental health services: Use of outcome, data, stakeholder involvement, and evidence-based practices. The Behavior Therapist, 28, 150-158.

Xue, Y., Hodges, K., & Wotring, J. (2004). Predictors of outcome for children with behavior problems served in public mental health. Journal of Clinical Child & Adolescent Psychology, 33(3), 516-523.

Zima, B.T., Bussing, R., Crecelius, G.M., Kaufman, A., & Belin, T.R. (1999). Psychotropic medication treatment patterns among school-aged children in foster care. Journal of Child and Adolescent Psychopharmacology, 9(3), 135-147.