Evaluating the effectiveness of diversity and inclusion efforts

But before measuring begins, when you start paying attention to something, it’s importance in your thinking increases. Let’s say you want to buy a car for example. Suddenly you notice things about cars you never noticed before and your conversations with others seem to regularly come back to the topic of cars. When you pay attention to diversity and inclusion, your awareness of it’s presence or lack of presence grows too.

Last summer my daughter, who is white, was reading a newspaper with my daughter-in-law, who is black, to find a shopping destination for the day. When they both saw an advertisement for one of their favorite stores showcasing a picture of all-white models they both exclaimed, “Monoculture!” and immediately burst out laughing at their simultaneous reaction. In the end, they decided not to go that that store. Two other visitors to our home that same day only saw the word “sale.” Why? It all depends what you pay attention to.

Getting the growth process started

It all starts with paying attention, but after that there is a process that needs to happen before realistic numerical targets can be set and measured. Let’s look at a few basics:

First, to see diversity and inclusion there must be open communication. Start with conversations between a group of people from diverse backgrounds, colours, experiences, ages, genders, abilities, orientations and persuasions, who are all working together on a common goal. Some training directly related to diversity and inclusion might be involved. People responsible for these types of training initiatives should work with key decision makers to develop training that meets specific organizational objectives. These objectives should tie into a broader strategic outline, and there should be ongoing feedback between upper management, trainers, and trainees to ensure the programming offered is “on course” to deliver forecasted results. Of course, the final critical piece is to actually measure these results and make adjustments. Remember: what you measure grows and what grows should uncover something unexpected.

Consider an example. Many Canadian cities have diversity and inclusion plans and they all look pretty good on paper. Here are Edmonton’s key diversity and inclusion elements:

1. Informed and committed leadership
2. Comprehensive scope of goals and activities
3. Integration of objectives within business plans throughout the organization
4. Dedicated resources
5. Focused education and training opportunities
6. Policy review and development
7. Shared responsibility and individual accountability
8. Measurement and evaluation

Effectiveness principles: The proof is in the pudding

But how do we know these elements are effective? What if the progress reports are just propaganda from the people who liked the decision? I was their idea after all…

To start, we know initiatives are effective if we have a good measurement process and tools in place to find out. It is critical to measure and evaluate initiatives otherwise there is no way to find out if they have been effective or if money has been well spent. Without measurements there is also no way to prove whether good programs should be kept or expanded.

On the other hand, there are issues with setting numerical targets. There could be concern about measuring the wrong thing or if targets are not met, numerical goals could be discouraging. Further, finding measurement tools that are right for the job and getting accurate demographic data on which to base goals is often harder than it seems. It could also be a problem to measure something that does not have enough variety of tangible, intangible, financial and non-financial indicators to provide a real-life snap shot of the situation.

But remember Principle #1: What you pay attention to grows. Measurement matters and it’s worth putting effort into.

Moving forward, you can always check whether or not a measurement is true – that is valid, and reliable – when the end users of the program or initiative are consulted, and when something new or surprising occurs.

Here we have Principle #2: if you are involving the end users, there will be surprises and changes.

Let’s look again at Edmonton’s diversity and inclusion elements on a more local level.

This preliminary work has enabled the EAC to establish a benchmark. The have records of where they started from and can now look at two key indicators of growth and change: who applies and who gets funding and support. They have a measuring stick to evaluate how well their efforts are working.

When the EAC Community Liaison, Kristina DeGuzman started to go to the communities to get to know the “end users,” she discovered some surprises that are leading to changes: People from different cultures have different ideas about what art is and how to participate in it. Collective expressions were often more common than individual expressions. Topics and reasons for artistic expression, places where they occurred, and the value attributed them also varied greatly. These discoveries about community art are starting to shape what the Edmonton Arts Council sees as emerging artistic expression and how to support and encourage it in diverse communities.

As they track process, the EAC will see numerical growth in their target areas as Principle #1: “What you pay attention to and what you measure grows” predicts. And as they listen to the community and learn from its input, the EAC will be on track to progress further as per Principle #2: “If you are talking to the end users, you should learn new things and let this learning propel changes for the organization.”

Example #2: Edmonton Transit Services

A few years ago, ETS made some attempts to increase the diversity of their driver workforce. They found that applicants from certain cultural communities had the skills and qualities they needed and should have been passing the entrance exams, but for some reason they were not. So one of the drivers, Ali Abdi, who himself was from a minority community, suggested that all applicants do a “ride along” before the exam where they would talk with the driver, ask questions, and find out what the daily job requirements were. The result was that all applicants who did the ride along performed better in the exam, and those minority candidates who had been failing the exam were now passing it. As a result of the unexpected learning from this experience, ETS decision makers decided to try a Heritage Days recruitment campaign to attract a wider variety of diverse candidates, which proved highly successful.

With respect to Principle #1, ETS first paid attention to the diversity of their workforce. They realized that some part of their process was not working for certain diverse candidates who had all the necessary skills and qualifications. Further investigation tied the problem to passing the entrance exam. A driver from a minority background who had passed the exam suggested the “ride along” prior to the exam instead of after. The results of listening to the target user showed better exam scores all around and increased success in the diversity hiring target group.

Onto Principle #2: As a result of learning from this experience, ETS decision makers realized they could leverage Heritage Days, a festival celebrating diversity, to target the demographics they were seeking to hire. This led to new recruiting campaigns at Heritage Days that have proven very successful. The numbers of applicants from this process and the number of new hires from diverse populations has increased as a direct result.

Capturing the learning: 6 Ideas to get you started

Let’s pull from these processes to capture some key takeaways:

• To evaluate the success of any diversity and inclusion initiative, you have to actually pay attention to diversity and inclusion
• Decision makers need to believe these initiatives matter and must actively show their support for them
• Make your planning group as diverse as possible – these people are where the clues to success lie
• Careful consideration of the context and data gathering is critical so you can establish relevant benchmarks
• Use your data to set intervention targets, with an eye to measure what grows. Remember, establishing realistic numerical targets that will show real, rather than ideal or imagined, growth
• Ensure the people running the objective learn from their target populations directly – this should change the way the project is perceived, run, and how it ultimately emerges in its context

We all know that numbers matter. But they will matter more if they are embedded in an evaluation process that is going to lead to actual correlations between the findings and the actions taken. And they will be even more valuable if they have been built on solid input from the targeted end users.

Post script: Thank you to Ali Abdi for sharing his ETS story, and to Kristina DeGuzman for sharing the EAC process and results to date.

References:

City of Edmonton Diversity and Inclusion framework and implementation plan. http://www.edmonton.ca/city_government/city_organization/office-of-diversity-and-inclusion.aspx

Gallagher-Louisy, Cathy. (July 2013). The ROI of D&I: Why municipalities must measure their efforts in equity, diversity and inclusion. http://www.cidi-icdi.ca/whats-happening/in-the-news/mw-roi/

McGee, Sherry, “Understanding Best Practices for Community Engagement in Municipal Contexts” (2009). Theses and Dissertations (Comprehensive). Paper 943.

Peel Newcomer Strategy Group: Immigrant Civic Engagement: Resources and Best

Practices Tool for Institutions and Organizations. www.peelnewcomer.org/site/peel…/resource-tool-for-institutions.pdf

Welcoming and inclusive communities toolktit. Human Rights Citizenship and Immigration Education Fund. https://web.archive.org/web/20160925202721/http://www.albertahumanrights.ab.ca/publications/AHRIS/2014/June_25.asp