The Sandblog

< Back to Blog

Put Your Work to the Test

by Dr. Jeff Johnson | September 22 2016 | 1 Comment

How did we do? Did the program get us where we needed to go? What did our audience take away from the project? It’s important to dig deep when the subject is program evaluation. Oftentimes, evaluation is a last-minute thought rather than a critical part of our planning process. When we can’t answer, “Was it worth it?” in a tangible way, it’s virtually impossible to make a change or take a different course with any confidence.

Rouse Some Rigour

I’ve seen public funding poured into initiatives with the potential to have significant impacts on patient health, but whose outcomes remain a mystery due to the lack of any formal evaluation. The immense volume of work, funding and potential for important positive change for patients is rendered null with no sense of the extent to which the project succeeded—or not.

Planning any new initiative should also include a plan to evaluate it. The degree of confidence in the results of your evaluation will depend on how well you design it—that is, its rigour. This is the same concept used in research studies and, just like program evaluations, not all research (published or not/ethics approval or not) inspires confidence if it’s poorly designed.

While randomized wellness trials aren’t likely an option at the school jurisdiction level, and after-the-fact evaluations shouldn’t be relied upon as the sole piece of measurement for a program either, it’s likely that somewhere in between the two is, as Goldilocks says, juuuuuuust right.

Steps to Confident Planning

With this in mind, here are three steps you can use to design your evaluation methods and have confidence their results:

  1. Plan ahead. The goal here is to design the evaluation before implementing your new program or initiative.
    • Clearly state the purpose of the program. What problem are you trying to solve?
    • Identify goals and objectives that tie back to the purpose of your program. Why is this program important? What do I want my audience to take away?
    • Clarify the links between how the program is resourced and delivered to achieve your outcomes. What do you need to accomplish your goal?
  2. Use evaluation tools. Logic models and data matrices are tools you can use to help you plan and implement your evaluation.
    • Create a logic model. Start with a clear statement of the program’s purpose (you’re way ahead as you’ve done this already in Step 1!). Next, map out all of the other elements you require to follow through with your idea. This includes all of the resources you need, key activities to make your program happen, how you’ll deliver your program and what you’re hoping your audience will gain from it. Basically, you’re answering, “so what?”
    • Develop a data matrix. Identify questions for each stage of your logic model based on what data needs to be collected, from where and when to answer those questions can and should be answered.
  3. Align your evaluation with the program purpose. The questions you ask and the items you identify in your data matrix will help you decide what to evaluate and how to do it. If you’re going a little cross-eyed, don’t worry. It’s often an iterative process and you’ll work back and forth between the purpose of the program and its intended outcomes versus what/how you’re going to collect the data and measure those outcomes. This step can take time but is well worth the effort.
    • Here’s an example: if the purpose of your program is to improve quality of life for participants, quality of life will be highlighted as an outcome in your logic model. It’ll be the big piece you want to measure before and after you deliver your program. Knowing this, you may plan to use something like participant surveys before and after the program, so you know whether quality of life changed or not.

Confident planning takes time and attention. Done well, it should force you to think carefully about what you’re trying to accomplish and will invariably improve your confidence in the results. No one wants to feel like they’re running in circles. Mapping out where you want to go will help you and your team thoughtfully develop memorable programs for any group now and into the future.

Dr. Jeff Johnson

With a long list of credentials to his name (including pharmacist, PhD, professor and health services researcher), “Dr. Jeff” is a leader in diabetes and public health policy and research. In his current role of professor in the School of Public Health at the University of Alberta, and supported by a senior health scholar award from Alberta Innovates Health Solutions, Dr. Jeff is making a big impact in the health and wellness arena in Alberta and beyond. Despite this impressive array of talents, his inability to whistle has served a crushing blow, preventing him from realizing his dream of competing at the World Whistling Championship. Join the club, Dr. Jeff. Join the club.

Stéphanie C. ASEBP | September 28 2016 9:15 AM

Thank you for this great blog Jeff! Evaluation is so important and like you said, it should be designed before implementing a program. I find that this worksheet Setting SMARTY Goals is a great tool to use when setting goals and objectives with your wellness team or committee. An evaluation plan can be captured under the "Measurable" section.

Are there other links or easy to use resources that you would suggest that can help build an evaluation plan?