We know shockingly little about the effectiveness of programs for treating young people with mental illness or addictions, writes Karen Minden. We wouldn’t accept this kind of failure with any other disease.
By Karen Minden, May 11, 2021
In 1998, our family was in trouble. We were on the brink of losing a child – a daughter who was in deep trouble as a result of acute addictions and mental health problems. Living in Winnipeg at the time, we were on a desperate mission to find help. It wasn’t there. It wasn’t anywhere in Canada. We bounced between psychiatrists, social workers, community agencies and group homes. Yet every door was the wrong door.
At that time, there were no residential addiction treatment programs for teens in Canada demonstrating that they would be able to help, and none even existed for children under 16. We were stunned to discover how little help there was until we were in crisis. In its midst, we could barely think straight to figure out how to get help. Health care professionals told us, “there’s nothing you can do,” “let it run its course,” “most kids don’t die.” The bar for success was shockingly low.
There was treatment in the United States, but it was expensive, unregulated, and far from home. We struggled with these hurdles until we finally found a program in the US that worked. We were among the few lucky ones. Our child’s life was saved. Our first-hand experience on the impact of a treatment program for youth in the US with wonderful, life-changing, immediate and sustained outcomes for our family motivated us to act – specifically to commit ourselves to building a treatment centre in Canada. (In the more than two decades since we started this journey, I have heard from hundreds of families who experienced the same sense of shock and hopelessness when they needed help for their child.)
Sadly, too many of our societal problems like mental illness and addiction are considered intractable. Efforts usually involve more of the same and when they fail, we bemoan the lack of adequate resources to do things right. Those fighting the good fight against tremendous odds deserve our thanks and our empathy.
However, we need to look at problems in a different light, be determined not to accept and perpetuate futility, learn from our mistakes and be resolute and relentless in our pursuit of success. We know all too well that hope is not a strategy.
A decade ago, I co-authored a paper in the Child Health Quarterly titled “Waiting for What?” It asked two fundamental questions about adolescent mental health: What are we treating? And what works? The article proposed that the low hanging fruit to transform adolescent mental health and addictions treatment was to measure outcomes. Faster access and more integrated services to treatment that has no validation is not the answer.
Most mental health treatment programs, and specifically those for youth, still do not track client outcomes. Funders for these programs do not require outcome evaluation but rather are focused on outputs such as the number of clients and staff, number of hours of treatment, food costs, and length of stay. All of these are cost-related factors and not one has anything to do with the health of clients or the extent to which they were helped. Health care reforms and initiatives tinker around the edges – better access, better integration, co-design, but they still do not ask the fundamental question: Do treatments work?
We determined that at our treatment centre, we would know whether we helped kids, and our knowing would be intentional and core to our program. We developed a “logic model” outlining what we wanted to achieve and determining how we would know if we were successful. We had a detailed plan of how we would measure success. After raising the funds to start the treatment centre, we applied our logic model and measured client health from the very first phone call, even before we opened the doors.
In 2006, Pine River Institute opened in Ontario. From day one, the institute has tracked its impact on the troubled young people and their desperate parents who are nurtured back to health and wholeness. It consistently reports approximately 80 percent success in helping young people return to functional living – attending school, staying out of hospital or jail, pursuing healthy family relationships, and a quality of life where they can see a future for themselves. The current Director of Research and Evaluation at Pine River, Dr. Laura Mills, holds to the conviction that outcome measurement is a core commitment, stating “no program should dare touch a kid unless they can demonstrate they don’t hurt their clients.”
Building a treatment centre in Canada was fraught with stumbling blocks including a professional community that accepted the status quo, a dependence on government to set the standards, provide and pay for care, and very low expectations of client success. In total, the mental health care culture was built on an acceptance of hopelessness.
The barriers to effective treatment: Lack of data
2007. At a pivotal meeting with then Attorney General Michael Bryant, my husband Dr. Harvey Schipper and I presented the case for the Government of Ontario to support Pine River. It had been open less than a year with no government support and was running out of funds. We provided the preliminary data we had been carefully collecting that showed the program worked. Bryant asked about success rates in other programs that government was already funding. We told him they had no outcome data. He looked abashed, and said he didn’t mean to insult us, but he had to get his staff to verify this claim. Two weeks later, dumbfounded, Minister Bryant went all out to make the case for the Ministry of Health and Long-Term Care to fund Pine River Institute.
2013. Pine River was named the lead institution of a collaborative to develop and implement outcome evaluation in the youth addiction sector. Half a million dollars of Trillium Foundation and Ministry of Health funding led to the plan and launch of a province-wide evaluation program. Today the project sits in an office on Bay Street (Addiction and Mental Health Ontario) waiting for uptake.
2016. The Office of the Auditor General of Ontario’s Report stated the following: “Consistent with our audit of 2003, the Ministry (of Children and Youth Services) is still not effectively monitoring and administering the program to ensure children and youth in need of mental health services are receiving timely, appropriate, and effective services” (2016 Value-for-Money Audit Summary).
2018. The Office of the Auditor General of Ontario Follow-up study reported: “As of July 5, 2018…the Ministry and the agencies have made little progress toward implementing 76% of the actions we recommended in our 2016 Annual Report…Effective October 29, 2018, policy and financial accountability for child and youth mental health transferred to the Ministry of Health and Long-Term Care” (p.15).
2019. The Office of the Auditor General of Ontario Addictions Treatment Programs Value-for-Money Audit revealed the following:
- The Ministry of Health (Ministry) allocates funding for addictions treatment services without determining the need for each type of service across the province. Funding for addictions treatment programs grew approximately 25% from about $191 million to about $239 million between 2014/15 and 2018/19. It primarily went to non-residential counselling services despite clients presenting with more complex addictions needs, which may require more intensive services.
- The Ministry funds addictions treatment service providers without evaluating the effectiveness of their programs. While some service providers identified ways to assess the effectiveness of their programs (such as interviewing clients or conducting client surveys), the Ministry has never asked for this information.
Today, there are woefully few Ontario agencies that can demonstrate the impact they have on the young people they serve. There is almost no core funding for outcome evaluation. (Some of the organizations that are leaders in this field fund outcome evaluation through their philanthropic foundations, or research grants.) Yet addictions and mental health programs continue, year after year, to get funded without knowing or demonstrating their impact. The bar is still abysmally low. When will evaluation be a line item in core funding for all health care providers?
We are in the vortex of a mental health and addictions epidemic. The cost of not acting prolongs suffering and delays the real possibility that we can do better. If our collective belief that people with mental health issues – be it addictive behaviour, depression, or schizophrenia – are a lost cause, our low expectations will continue to affect everything up the chain.
We can raise the bar, and each of us, whether health care clients, providers, or funders, can make it better. Each of us can make sure that children and youth, and their families, get the best available treatment for mental health and addictions.
Here is what agencies can do to make this happen. First, they need to determine what they are trying to accomplish, and how they will know whether they are successful. Second, they need to set up a data collection and analysis system. Additional resources may be needed to provide expertise and service. This can be in-house or contracted with experts. And third, they must use that data to report regularly to themselves, their clients, and their funders. Roles and responsibilities must be clear and dedicated. Such an initiative requires strong and consistent leadership support and commitment. This is not rocket science. And it does not have to be expensive.
How does an agency actually measure success? By knowing how they hope they help their clients and validating that hope with evaluation. Do they offer treatment to reduce depression or anxiety? Do they reduce suicidal thoughts or behaviours? Do they help clients become more globally ‘functional’ across vocation or education, relationships, and mental well-being with reduced hospital visits and criminality?
Process indicators do not measure success. Knowing that you serve 100 meals a day or 300 people per year or spend 30 minutes per admission are process indicators and help understand treatment flow, human resources, and costing. None of these help us understand whether we are helping.
Imagine if funders allocated core funding dollars to every mental health and addiction agency for capacity to measure their clients’ outcomes, and mandated published evaluation findings. This simply means they would require accountability for the impact of their investments.
Cancer patients ask for data on the most effective treatment, and evidence about best practices based on outcome evaluation. They get that information to help them choose their best option. This is a fundamental expectation and is met with accountability and transparency. Imagine if mental health and addiction clients and their advocates demanded the same standard of care. Imagine if we acknowledged that our adolescents are not getting what they need to grow up strong and healthy.
Imagine if parents asked what the treatment approach is when a child loses his or her way; and asked to see outcome data on the success rate of treatment programs; demanded that those who provide programs – both the funders and the agencies that deliver treatment – provided data on their outcomes, showing that they are accountable to their patients and the ultimate funders – Canadian taxpayers.
So, this is my wake-up call to funders – including clients. You can raise the bar for all mental health providers by offering incentives and support to measure their impact on the people they serve, and improve the outcomes for all.
We have the power to drive innovation. We have the power to support a well-intentioned system to do better. A world-wide effort created several effective COVID-19 vaccines by making them a priority, applying brain power and funding for this urgent journey. We can achieve much more in many fields that are in desperate need, including mental health and addictions.
Treating youth mental illness and addictions is a problem we can solve. We need to think differently and act smarter. What we really need to focus on is “what works?” What treatments are effective in alleviating the suffering of our young people and their families? Let’s go higher than the established path of giving money away and then forgetting about how it is spent. The “system” will respond if funders require, facilitate, and fund accountability. We can do better. Our children and their families deserve better.
Karen Minden, CM, PhD, is Founder of Pine River Institute and Pine River Foundation, now retired. She has a broad foundation in research, public policy, philanthropy and international relations. As Vice President of the Asia Pacific Foundation, she helped to create a national think tank on Canada-Asia relations. She is a member of the Order of Canada.
