[BoulderCouncilHotline] Re: Questions re: Community Assistance Response Team pilot

[Schwartz, Wendy]

Dear Nicole,

Thank you for your questions. I have included our responses in blue below, and we are also happy to discuss these issues tonight at the study session.

Good morning, everyone,

I have some questions for the CART discussion and wanted to send them in advance to give staff some time to think about this issue of how we define and measure success.

My general question/comment is around what "refining" program goals means: How much more specific will our goals get so that we make sure council, staff, and the community are crystal clear on what success looks like before we launch this program?

The staff working on this project started by developing draft program goals, based on:

  *   what we had heard to date from council and community members about their hopes for the program;
  *   related goals in current approved city master plans, specifically the Fire Rescue master plan;
  *   review of goals and metrics for similar programs in other cities; and
  *   goals that seemed reasonable for the community given the nature of the program.

These draft goals were also shared as part of our discussions with our safety net health/mental health care providers, Boulder County and the Human Relations Commission (HRC).

We then created draft metrics that have a relationship to each goal, and would potentially be trackable. Similar to the process for goals above, we borrowed ideas from other jurisdictions where possible.

In your question, you reference refining goals. Our study session this week is part of that process, as our questions for council ask about its support for the goals as proposed. If council does not support the goals, or has modifications to the goals, they will be refined through our discussion.

If you are asking about our reference in the memo to refining success metrics, we envision that process happening with the assistance of a professional evaluator and input from council. Based on council direction tonight, we will proceed with the procurement process for selecting the contractor - in alignment with city procurement procedures - early in 2023. While our city team includes excellent data/analytics staff, we believe that we need someone with more specific expertise (as well as the time to do the project) advising us on this matter. That consultant may be able to suggest other metrics that better capture the intent of our goals, or advise us on why some of our proposed metrics may be less practical than they currently appear.

Finally, as an overall comment, there are limited metric examples and benchmarks for these types of programs nationally, so our staff is charting some new territory. Data available on STAR and CAHOOTS focus primarily on operational outputs - number of responses, service referrals made, whether police had to be called, types of calls the program responded to, etc. Currently both programs have evaluations in progress with external evaluators, and we may be able to get additional ideas from those efforts; as well as some work in the beginning stages with the Justice, Health & Democracy Impact Initiative (JHD) Mayors meeting, a joint project of Harvard University and the US Conference of Mayors that our mayor referred us to.

As some examples of what I mean, here are the types of questions that came to mind for me regarding the proposed pilot program goals and objectives on page 5:

  1.  What percentage of calls diverted from BPD and Fire will we consider a success? (1)
     *   There is not a specific national benchmark or best practice for this measure. STAR and CAHOOTS report a range of 3%-8% diversion, and that will be a starting place for comparison.
     *   Fire's long-term goal is a 20% reduction in low-acuity calls. Currently, we have a 5% reduction since the implementation of Nurse Navigator and a 3% reduction due to participation in the ET3<https://innovation.cms.gov/innovation-models/et3/faq> program. CART might involve a 5% reduction.


  1.  How much of an improved response times for higher-level or more critical emergencies will we need to see to think the program has successfully achieved this goal? (1a)
     *   In the core of the city, the goal would be improvement by up to 60 seconds. There are a variety of factors that impact this estimate that Chief Calderazzo can explain as needed at the study session.

  2.  Are we able to partner with the necessary groups that will generate the data we need to determine whether we are improving health outcomes in emergencies (e.g., with BCH)? And, what does "improve health outcomes" mean in terms of a measurable goal? (1b)
     *   We are hoping that we can establish some level of data sharing with necessary partners through Releases of Information (ROI). This is likely more feasible for clients that also participate in follow-up case management, as a crisis situation is not always the best time to get an ROI signed with an individual able to truly give informed consent. In some cases, we may also use self-reported data from follow up calls. For example, whether the individual in question reports making an appointment with a health care provider for ongoing care.
     *   We might need a little more information to better understand and answer your question about the "improve health outcomes" measure - so we don't just repeat memo text - and can discuss this at the study session.


  1.  How will we track positive health impacts for individuals and successful diversion from emergency services, especially in a time when community services are already stretched thin? (2)
     *   Similar to the previous item, we may need more information to better understand and respond to your question about tracking impacts and diversion and are happy to discuss with you tonight.
     *   You are correct that community resources are stretched thin, and part of the work with our community partners and our evaluation consultant is to consider not only what can be tracked, but what can be tracked without placing undue burden on staff and systems that are already being asked to do a great deal.

  1.  Who will realize the cost savings from reduced emergency services, and how much of a cost savings will we consider a success? (3)
     *   We do not have a specific estimate of cost savings at this time. EMS measures effective productivity by metrics such as Unit Hour Utilization. Simply, it is a ratio of hours on task (responding, transporting, repositioning etc.) to available hours. This gives us a measurement of crew "workload." As our system grows, we can calculate the decrease in workload for each unit due to the alternative response models.  This represents another form of "cost" to the system.
     *   Some of our local healthcare providers may experience reduction in emergency services costs. While there is not a direct financial benefit that we can attribute to the city in this scenario, our residents benefit if reduced emergency services visits free up healthcare capacity to serve community members.
  2.  How many unnecessary ambulance rides are we looking to reduce? What counts as unnecessary? (3a)

This is a nuanced issue and will take more work to refine. Chief Calderazzo can explain more as necessary at the study session.



  1.  Can we track connections to services and referrals and what data will tell us if they are increased? How will we define success (again, especially with community services stretched thin)?
     *   We can track referrals to services via the database staff will use for the program. Connections to services will likely be either self-reported or via ROIs as described in the answer to question 3 above. You are correct that we do not necessarily have a baseline for this data, though we do hope that if people are connected to services, we will not see as much repeat utilization of emergency services for those individuals.

Other general questions:

  *   Are we able to use any metrics that have existing data, so we have a baseline? (e.g., can we look at something like "a 5% decrease in complaints to the Police Oversight Panel")?
     *   We have existing data for some metrics. For example, we know how much police time is currently spent on welfare checks and similar call categories, response times on priority calls, number of ambulance rides, and 911 calls for city high utilizers.
     *   Regarding your specific mention of the Police Oversight Panel, we can discuss that idea further at the study session. Part of that discussion would be alignment of metrics with goals.
  *   How will we define success in cases where we do not have a baseline (e.g., to what will we compare "feeling respected during a call to initiate a CART response," p. 9)?
     *   You are correct that with a new program there are some metrics for which we will be establishing a baseline rather than mapping to an existing baseline. We will seek input from our evaluation consultant on this matter, but there will likely be some cases where there is no objective definition of success. In the example you mention in this question, we would certainly hope that most people feel respected during a call to initiate a CART response, but different people will have different ideas about what an acceptable percentage of positive response is for that question.
     *   Colorado state crisis centers are planning to collect data on patient satisfaction as part of their program evaluation - specifically client perception of 'feeling respected during interaction' and client perception of 'feeling equipped to handle crises in the future', and we may be able to gain additional insights in alignment with their work.
  *   What recommendations does the evaluator have to improve our ability to attribute changes to the CART program vs. to "other community efforts" (p. 8)?
     *   We will ask this question as we hire the evaluator. It may not be possible to entirely separate which changes can be attributed to CART alone versus a combination of community efforts.
  *   What types of demographics will we collect beyond housing status?
     *   To the best of our ability - age, gender and race/ethnicity. This information is sometimes challenging to collect. For reference, the Colorado state crisis centers only consistently collect age and gender during initial intake and staff mentioned it is not feasible for them to collect additional demographics during most interactions. At the time of interaction, many people are not in a position to answer numerous questions about themselves. Although the team will have a protocol for follow up calls with residents after the initial crisis, some people are difficult to reach on follow up or have a variety of reasons they do not wish to offer further information about themselves.

I know we won't have all these answers until the evaluator is on board, and I don't mean to suggest all these things would be measured. Rather, I am interested in hearing more about how we will ensure our goals are set in such a way that there is minimal room for interpretation on whether or not the pilot program achieved its goals.

Thanks,
Nicole

Nicole Speer, Ph.D.
Boulder City Councilmember
Pronouns: she, her(s), ella

Phone: 303-519-9068

Web: bouldercolorado.gov/person/nicole-speer<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbouldercolorado.gov%2Fperson%2Fnicole-speer&data=05%7C01%7CSchwW1%40cityofboulder.mail.onmicrosoft.com%7C45d90f0d977c4c5ef95508dad791f409%7C0a7f94bb40af4edcafad2c1af27bc0f3%7C0%7C0%7C638059319141897455%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=sXVVrXWdAViq95CzY%2B09a5HGyho%2F1FdYZC8z1o0EW3c%3D&reserved=0>
Twitter: @SpeerBldrCC<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2FSpeerBldrCC&data=05%7C01%7CSchwW1%40cityofboulder.mail.onmicrosoft.com%7C45d90f0d977c4c5ef95508dad791f409%7C0a7f94bb40af4edcafad2c1af27bc0f3%7C0%7C0%7C638059319141897455%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=DnnsuLlts6rYL9ujknJysTSGXOEOz4NB%2BOOAEqzKbzc%3D&reserved=0>



From: Speer, Nicole <SpeerN at bouldercolorado.gov>
Sent: Tuesday, December 6, 2022 6:55 AM
To: HOTLINE <HOTLINE at bouldercolorado.gov>
Subject: [BoulderCouncilHotline] Questions re: Community Assistance Response Team pilot

Good morning, everyone,

I have some questions for the CART discussion and wanted to send them in advance to give staff some time to think about this issue of how we define and measure success.

My general question/comment is around what "refining" program goals means: How much more specific will our goals get so that we make sure council, staff, and the community are crystal clear on what success looks like before we launch this program?

As some examples of what I mean, here are the types of questions that came to mind for me regarding the proposed pilot program goals and objectives on page 5:

  1.  What percentage of calls diverted from BPD and Fire will we consider a success? (1)
  2.  How much of an improved response times for higher-level or more critical emergencies will we need to see to think the program has successfully achieved this goal? (1a)
  3.  Are we able to partner with the necessary groups that will generate the data we need to determine whether we are improving health outcomes in emergencies (e.g., with BCH)? And, what does "improve health outcomes" mean in terms of a measurable goal? (1b)
  4.  How will we track positive health impacts for individuals and successful diversion from emergency services, especially in a time when community services are already stretched thin? (2)
  5.  Who will realize the cost savings from reduced emergency services, and how much of a cost savings will we consider a success? (3)
  6.  How many unnecessary ambulance rides are we looking to reduce? What counts as unnecessary? (3a)
  7.  Can we track connections to services and referrals and what data will tell us if they are increased? How will we define success (again, especially with community services stretched thin)?

Other general questions:

  *   Are we able to use any metrics that have existing data, so we have a baseline? (e.g., can we look at something like "a 5% decrease in complaints to the Police Oversight Panel")?
  *   How will we define success in cases where we do not have a baseline (e.g., to what will we compare "feeling respected during a call to initiate a CART response," p. 9)?
  *   What recommendations does the evaluator have to improve our ability to attribute changes to the CART program vs. to "other community efforts" (p. 8)?
  *   What types of demographics will we collect beyond housing status?

I know we won't have all these answers until the evaluator is on board, and I don't mean to suggest all these things would be measured. Rather, I am interested in hearing more about how we will ensure our goals are set in such a way that there is minimal room for interpretation on whether or not the pilot program achieved its goals.

Thanks,
Nicole

Nicole Speer, Ph.D.
Boulder City Councilmember
Pronouns: she, her(s), ella

Phone: 303-519-9068

Web: bouldercolorado.gov/person/nicole-speer<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbouldercolorado.gov%2Fperson%2Fnicole-speer&data=05%7C01%7CSchwW1%40cityofboulder.mail.onmicrosoft.com%7C45d90f0d977c4c5ef95508dad791f409%7C0a7f94bb40af4edcafad2c1af27bc0f3%7C0%7C0%7C638059319141897455%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=sXVVrXWdAViq95CzY%2B09a5HGyho%2F1FdYZC8z1o0EW3c%3D&reserved=0>
Twitter: @SpeerBldrCC<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2FSpeerBldrCC&data=05%7C01%7CSchwW1%40cityofboulder.mail.onmicrosoft.com%7C45d90f0d977c4c5ef95508dad791f409%7C0a7f94bb40af4edcafad2c1af27bc0f3%7C0%7C0%7C638059319141897455%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=DnnsuLlts6rYL9ujknJysTSGXOEOz4NB%2BOOAEqzKbzc%3D&reserved=0>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://webappsprod.bouldercolorado.gov/mailing-lists/mailman-archive/bouldercouncilhotline/attachments/20221208/8e0a7040/attachment.html