Evaluation methodologies in multisector community change initiatives: The missing role of indigenous knowledge systems
Abstract
Comprehensive or multisector community change initiatives (CCIs) represent a promising approach to address the overrepresentation of Indigenous children and families in Canada’s child welfare systems. However, such initiatives are difficult to establish and sustain – in part due to the difficulty of evaluating their impacts and outcomes using standardized Western evaluation methodologies. Consequently, over the past 20 years there have been extensive efforts to develop evaluation principles, methodologies, methods, and tools that are more able to illustrate the benefits of these kinds of initiatives. A systematic review of the CCI evaluation literature found that while many of these principles, methodologies, methods and tools show considerable promise, there has been limited attention to or incorporation of Indigenous ways of knowing or approaches to research with Indigenous peoples. This paper presents two examples of Indigenous led multisector community change initiatives to enhance Indigenous well-being and notes the importance of evaluating their impacts. It argues that collaborative research is needed with participants in Indigenous led multisector collaborations to advance knowledge of culturally relevant approaches to their evaluation.
References
Abel, M. & Gillespie, J. (2015). Network Analysis in co-productive research with a multi-sector community network. Community Development Journal, 50, 327-344. doi:10.1093/cdj/bsu040
Auspos, P. & Kubisch, A. (2012). Performance management in complex, place-based work: What it is, what it isn’t, and why it matters. Washington, DC: The Aspen Institute. Retrieved from http://www.aspeninstitute.org/sites/default/files/content/images/rcc/Aspen_Performance_Management.pdf
Australian Institute of Health and Welfare. (2010). Child Protection Australia 2008-09. Available at: http://www.aihw.gov.au/publications/cws/cpa07-08/cpa07-08.pdf
Bennett, M. & Auger, A. (2013). First Nations and non-Aboriginal children in child protection services. Prince George, BC: National Collaborating Centre for Aboriginal Health. Available at: http://www.nccah-cnsa.ca/Publications/Lists/Publications/Attachments/7/protective_services_EN_web.pdf
Blackstock, C. & Trocmé, N. (2005). Community-based child welfare for Aboriginal children: Supporting resilience through structural change. In M. Ungar, ed. Handbook for working with children and youth: Pathways to resilience across cultures and contexts, (pp. 105-120). Thousand Oaks: Sage.
Blackstock, C., Trocmé, N. & Bennett, M. (2004). Child maltreatment investigations among Aboriginal and non-Aboriginal families in Canada. Violence Against Women, 10(8), 901-916.
Bradford, N. (2005). Place-based public policy: Towards a new urban and community agenda for Canada. Family Network Research Report F|5. Ottawa, ON: Canadian Policy Research Networks.
Berkowitz, B. (2001). Studying the outcomes of community based coalitions. American Journal of Community Psychology, 29, 213-227.
Bradford, N., & Chouinard, J. A. (2009). Learning through evaluation? Reflections on two federal community-building initiatives. The Canadian Journal of Program Evaluation, 24(1), 51-77.
Brown, (2010). Evaluating and learning from community change. In A. Kubisch, P. Auspos, P. Brown & T. Dewar, Voices from the Field III: Lessons and Challenges from Two Decades of Community Change Efforts (pp. 95-105). Washington, DC: The Aspen Institute.
Cabaj, M. (2014). Evaluating collective impact: Five simple rules. The Philanthropist, 26(1), 109-124.
Carter, V. (2010). Factors predicting placement of urban American Indian/Alaskan natives into out-of-home care. Children and Youth Services Review, 32(5), 657-663.
Carter, V. (2009). Prediction of placement into out-of-home care for American Indian/Alaskan natives compared to non-Indians. Children and Youth Services Review, 31(8), 840-846.
Canadian Centre for Policy Alternatives (CCPA) Manitoba. (2018). Na-gah mo Waabishkizi Okijaak Bimise Keetwaatino: Singing White Crane Flying North: Gathering a bundle for Indigenous evaluation. Winnipeg, MA.
Connell, W., Kubisch, A., Schorr, L. & Weiss, (1995). New approaches to evaluating community initiatives: Concepts, methods, and contexts. New York: The Aspen Institute.
Delfabbro, P., Hirte, C., Rogers, N., & Wilson, R. (2010). The over-representation of young Aboriginal or Torres Strait Islander people in the South Australian child system: A longitudinal analysis. Children and Youth Services Review, 32, 1418-1425.
Duran, E. (2006). Healing the soul wound: Counseling with American Indians and other Native peoples. New York: Teachers College Press.
Evans, M., Hole, R., Berg, L., Hutchinson, P., & Sookraj, D. (2009). Common insights, differing methodologies: Towards a fusion of Indigenous methodologies, participatory action research, and white studies in an urban Aboriginal research agenda. Qualitative Inquiry, 15, 893-910.
Farrow, F. & Schorr, L. (2011). Expanding the evidence universe: Doing better by knowing more. Washington, Center for the Study of Social Policy.
Fiester, L. (2010). Measuring change while changing measures: Learning in, and from, the evaluation of Making Connections. The Annie E. Casey Foundation. Available at: http://www.evaluationroundtable.org/documents/cs-measuring-change.pdf
Filbert, K. & Flynn, R. (2010). Developmental and cultural assets and resilient outcomes in First Nations young people in care: An initial test of an explanatory model. Children and Youth Services Review, 32(4), pp. 560-564.
Gillespie, J., Supernault, G, & Abel, M. (2014). Community networking: A promising practice for Aboriginal child welfare. In D. Badry, M. Montgomery, D. Fuchs, & S. McKay (Eds.), Reinvesting in Families, Securing a Brighter Future: Voices from the Prairies (pp. 201-219). Edmonton, AB: Prairie Child Welfare Consortium/Alberta Centre for Child, Family and Community Research.
Gillespie, J. & Albert, J. (2017). Promising approaches in evaluating comprehensive community initiatives to promote Aboriginal child and family well-being: The role of Indigenous ways of knowing and approaches to research. A Knowledge Synthesis submitted to the Social Sciences and Humanities Research Council of Canada. 33 pp.
Harris, M. & Hackett, W. (2008). Decision points in child welfare: An action research model to address disproportionality. Child and Youth Services Review, 30, 199-215.
Hart, M. (2010). Indigenous worldviews, knowledge, and research: The development of an indigenous research paradigm. Indigenous Voices in Social Work, 1(1), 1-16.
Hudson, P. (1999). Community development and child protection: A case for integration. Community Development Journal, 34, 346-355.
Kania, J. & Kramer, M. (2011). Collective impact. Stanford Social Innovation Review, (Winter), 36-41. Retrieved from: http://www.ssireview.org/articles/entry/collective_impact
Kelly, T. (2010). Five simple rules for evaluating complex community initiatives. Community Investments, 22(1), 19-22. Available at: http://www.frbsf.org/community-development/files/Spring_CI_2010a.pdf
Kovach, M. (2009). Indigenous methodologies: Characteristics, conversations, and contexts. Toronto: University of Toronto Press.
LaFrance, J. & Nicols, R. (2010). Reframing evaluation: Defining an indigenous evaluation framework. The Canadian Journal of Program Evaluation, 23(2), 13-31.
Lalonde, C. (2005). Identity formation and cultural resilience in Aboriginal communities. In R. Flynn, P. Dudding & J. Barber (Eds.), Promoting resilience in child welfare (pp. 52-71). Ottawa, ON: University of Ottawa Press.
Lavergne, C., Dufour, S., Trocmé, N. & Larrivée, M. (2008). Visible minority, Aboriginal and Caucasian children investigated by Canadian child protective services. Child Welfare, 87(2), 59-76.
Lonne, B., Parton, M., Thomson, J. & Harries, M. (2009). Reforming child protection. London: Routledge.
Martin, K. & Mirraboopa, B. (2009). Ways of knowing, being and doing: A theoretical framework and methods for indigenous and indigenist research. Journal of Australian Studies, 27(76), 203-214. DOI: 10.1080/14443050309387838
McKenzie, B. (2002). Community organizing in child welfare: Changing local environments and developing community capacity. In B. Wharf (Ed.) Community work approaches to child welfare (pp. 83-92). Peterborough, ON: Broadview Press.
Native Women’s Association of Canada. (2012). SIS vigils. Retrieved from http://www.nwac.ca/programs/sis-vigils
Provan, K., Veazie, M., Staten, L. & Teufel-Shone, N. (2005). The use of network analysis to strengthen community partnerships. Public Administration Review, 65(5), 603-613.
Public Health Agency of Canada. (2010). Canadian incidence study of reported child abuse and neglect – 2008: Major findings. Ottawa, ON.
Reason, P. & Bradbury, H. (2008). The Sage handbook of action research. London: Sage.
Royal Commission on Aboriginal Peoples. (1996). Report of the Royal Commission on Aboriginal Peoples. Ottawa, ON. Retrieved from: http://data2.archives.ca/e/e448/e011188230-01.pdf
Sinha, V., Trocmé, N., Fallon, B., MacLaurin, B., Fast, E., Thomas-Prokop, S. et al. (2011). Kiskisik Awasisak: Remember the children: Understanding the overrepresentation of First Nations children in the child welfare system. Ottawa, ON: Assembly of First Nations. Available at: http://cwrp.ca/sites/default/files/publications/en/FNCIS-2008_March2012_RevisedFinal.pdf
Standing Committee on the Status of Women (2011). Call into the night: An overview of violence against Aboriginal women. Ottawa, ON: House of Commons Canada. Retrieved from: http://www.parl.gc.ca/content/hoc/Committee/403/FEWO/Reports/RP5056509/feworp14/feworp14-e.pdf
Thoburn, J. (2007). Globalisation and child welfare: Some lessons from a cross-national study of children in out-of-home care. Social Work Monographs, UEA, Norwich.
Tilbury, C. (2009). The over-representation of indigenous children in the Australian child welfare system. International Journal of Social Welfare, 18(1), 57-64.
Torjman, S. & Leviton-Reid, E. (2003). Comprehensive community initiatives. Ottawa, ON: Caledon Institute of Social Policy.
Trocmé, N., Knoke, D. & Blackstock, C. (2004). Pathways to the over-representation of Aboriginal children in Canada's child welfare system. Social Service Review, 78(4), 577-600.
Truth and Reconciliation Commission of Canada. (2015a). Calls to action. http://nctr.ca/assets/reports/Calls_to_Action_English2.pdf
Truth and Reconciliation Commission of Canada. (2015b). Canada’s residential schools: The legacy. The final report of the Truth and Reconciliation Commission of Canada. Retrieved from: http://nctr.ca/assets/reports/Final%20Reports/Volume_5_Legacy_English_Web.pdf
Truth and Reconciliation Commission of Canada. (2015c). The survivors speak: A report of the Truth and Reconciliation Commission of Canada. Retrieved from: http://nctr.ca/assets/reports/Final%20Reports/Survivors_Speak_English_Web.pdf
Tuhiwai Smith, L. (2012). Decolonizing methodologies: Research and indigenous peoples. (2nd ed.). New York: Zed Books.
U.S. Department of Health and Human Services, Administration for Children and Families, Administration on Children, Youth and Families, Children's Bureau. (2009). The AFCARS report: Preliminary estimates for FY 2008 as of October 2009 (Report #16).
Wilson, S. (2008). Research is ceremony: Indigenous research methods. Halifax, NS, Canada: Fernwood.
