GallopNYC is an organization that provides therapeutic horseback riding to children and adults with disabilities. ReLearning Curve started working with GallopNYC as one of the first recipients of our Evaluation Within Reach service grant. Their goals were to: 1) show how their riding program helped people to lead more independent and productive lives; and 2) build a sustainable internal evaluation process with buy-in from its staff – riding instructors who were not “paper people”.
Evaluation Support Strategies
ReLearning Curve conducted the following activities to build GallopNYC’s evaluation capacity.
- Using a Theory of Change process ReLearning Curve worked with GallopNYC’s staff and stakeholders (i.e., participants, board members, and colleagues) to conceptualize and map how program activities led to long-term changes in participants’ lives.
- We collaboratively designed a comprehensive evaluation plan with multi-year priorities by assigning indicators to each outcome, identifying data collection instruments, and establishing a timeline for data collection.
- We conducted a data inventory to assess what instruments were currently being used, how effectively they were used, and identified data collection gaps and needs.
- We revised data collection instruments and processes and developed new ones.
- We reviewed changes in GallopNYC’s CRM system to make sure they were aligned with new instruments.
- GallopNYC piloted new instruments and we reviewed preliminary data and made recommendations for analysis and reporting.
- GallopNYC debriefed with staff who recommended more modifications which they intend to make.
The Results of Evaluation Capacity Building Support
According to the Executive Director of GallopNYC, the support provided by ReLearning Curve had the following impacts.
- Using the ToC map improved staff understanding of the relationship between riding session activities and organization’s broader mission of improving lives.
- Using the ToC Map helped the Executive Director to communicate their broader mission to prospective funders more effectively. They have been able to attract more measurement-oriented funders and feel more confident driving the outcomes conversation with funders who may not know what results to ask for.
- Having “user-friendly” evaluation tools and processes increased buy-in from staff. Initially, in addition to riding skills, instructors were evaluating riders on 23 potential therapeutic goals each week of an 11-week session, using observation categories that were vague and included a limited range of development. ReLearning Curve helped to streamline data collection so instructors evaluated riders only twice per session, on 3 goals, which they chose based on riders’ therapeutic needs and how they planned to work with that rider over the session. Observation categories were better defined. The instructors completed the new observation forms more often and completely. They said they were more useful because they were connected to their actual lessons and more specific therapeutic goals.
- Having new data improved reporting to stakeholders. Previously, riders or their parents received a general letter describing the riders’ accomplishments at the end of the 11-week session. Subsequently, riders and parents received a letter with the riders’ growth ratings on the three selected therapeutic goals. Most participants preferred the new method of reporting.
- Having a comprehensive evaluation plan and tested data collection instruments increased GallopNYC staff ability and confidence to make ongoing modifications and build on their evaluation over time.
Attainment of Evaluation Capacity Goals
ReLearning Curve successfully helped GallopNYC to complete the goals they set out to achieve. By the end of the consultancy GallopNYC had an outcome map and evaluation plan that could show how their riding program helped people to lead more independent and productive lives.
I always thought we were seeing results way beyond what we saw minute-by-minute, but it was hard to figure out how to capture those, how to express those, and how to connect those to what we were able to measure. It was really useful to be able to think through how that worked and to capture it. (Alicia Kershaw, Executive Director)
By the end of our engagement there was more staff support for evaluation and the Executive Director felt confident and empowered to make modifications as needed. In terms of next steps, much of the evaluation process resides with the Executive Director. That will eventually need to change for the organization to be able to grow and sustain internal evaluation.
I would say our evaluative culture was pretty top down, pretty much limited to me. I think the staff that participated in it with me are able to understand it more clearly now. Making it more manageable for the line staff has helped them be more positive about it. We have all gone over it at every retreat and recently at an instructor clinic. It is a lot more accessible to them now. It’s huge that they are more comfortable with it, and they are more willing. (Alicia Kershaw, Executive Director)