Why Your ‘Objective’ Screening Rubric Produced Biased Results

Full vitae allison feb.16 gears 5688996 1280 copy

Image by Mohamed Hassan From Pixabay

By Allison M. Vaillancourt

When you were invited to serve on the search committee for a high-profile leadership post, you thought it would be a chance to diversify an executive team that had no diversity at all. Here, finally, was a tangible way to respond to equity demands and disrupt previous hiring practices that seemed to yield the same leadership phenotype over and over again at your institution.

At the first search-committee meeting, you were both pleased and surprised when the chair announced that the group had two primary goals: (1) Recommend at least three finalists who would work well with the president’s cabinet, and (2) ensure that diverse candidates were both identified and fully considered.

To guide this work, the search chair said he wanted to apply some new theories about unconscious bias in hiring that he had gleaned by reading Jennifer L. Eberhardt’s Biased and Abigail J. Stewart and Virginia Valian’s An Inclusive Academy. Both books, he said, had helped him realize how much “gut feelings” had guided candidate selections in the past. This time, he was committed to using a highly structured assessment process that would be both equitable and methodical. Everyone on the committee applauded his good intentions. “Finally,” you said to yourself, “a search committee that knows what it is doing and is committed to a fair process.”

The committee created a rubric to evaluate candidate dossiers, and the assessment tool produced a diverse slate of candidates for first-round, one-hour video interviews. Soon after those interviews, however, things took a wrong turn. The committee’s score sheets revealed that white male candidates had outperformed women and people of color on almost every criterion of the carefully crafted rubric, including “grasp of emerging trends in higher education,” “strong public-speaking skills,” and “comfort with conflict.”

The committee was puzzled. How was that possible? How did diverse candidates whose impressive dossiers indicated such promise fail so miserably during a short on-camera interview? Was there a problem with the evaluation rubric? No; it worked just as it was intended. And that was the problem.

Let me pause here to acknowledge that the scenario I have described is fictional. While it would be amazing to have a search-committee chair kick things off by citing the work of Eberhardt, Stewart, and Valian, I’ve never actually seen that happen. That said, in my 30 years in higher-education administration and now as an organizational consultant, I have served on and observed several hiring committees that relied on detailed assessment rubrics to evaluate candidates. Such rubrics tend to yield a diverse pool of candidates when the dossiers are reviewed, but things start to fall apart once the interviewing begins. Why? Because of tightly held perceptions of how leadership behavior should be demonstrated in higher education.

Ijeoma Oluo — in her 2020 bookMediocre: The Dangerous Legacy of White Male America — wrote that she and her racial-justice colleagues often utter the words “works according to design” in response to actions or decisions that so obviously benefit white men at the expense of people of color. “Although the phrase may seem alarmingly cold-hearted,” she wrote, “it is our way of reminding ourselves that the greatest evil we face is not ignorant individuals, but our ignorant systems.”

What that means in leadership searches is that committee members often rely on narrow visions and demonstrations of leadership to assess candidates. Candidates who do not look or sound like the leaders we have come to expect end up being evaluated less favorably. White male candidates score well against the evaluation criteria because they act in accordance with the visual and auditory expectations that come to mind when we think about the majority of higher-education leaders we have seen for as long as we can remember.

So, what is the solution? Are we doomed to homogenous leadership teams until the end of time? Of course not, but achieving different results will take more than good intentions. It will require some different goals, including a commitment to reimagining what a campus leader looks and sounds like.

Here are five things that search committees can do to move more women and people of color forward in the executive-hiring process:

  • Don’t be swayed by the math. Using rubric scores works only if the assessment instrument and the evaluators are completely unbiased and the measures are easy to quantify. Leadership attributes are, of course, highly subjective. If certain types of people seem more likely to be screened out by your assessment tool, consider the possibility that there is something wrong with it or the way it is being used. While initial assessment scores can be useful in getting a sense of how candidates compare against one another, be sure to take time to discuss why each candidate scored well or poorly. Slowing down to consider the various factors that explain a candidate’s rubric score can reveal a reliance on faulty models of what a campus leader must look and sound like to be successful.
  • Avoid the “likeability” trap. Most of us gravitate to people who are like us. If you hear comments like “It would be fun to work with her” or “I felt an immediate connection,” be open to the possibility that affinity bias is at play. It is normal to prefer people who are like you and to be wary of those who are not. Make it a practice to both name and analyze how candidates make you feel.
  • Challenge the charisma requirement. In his 2019 bookWhy Do So Many Incompetent Men Become Leaders?, Tomas Chamorro-Premuzic notes that we are drawn to people who have a strong sense of self, make bold declarations, and discuss visionary plans. He adds, however: “There is a world of difference between the personality traits and behaviors it takes to be chosen as a leader and the traits and skills you need to be able to lead effectively.” Too often, charisma gets people their leadership roles, but it does not help them to succeed on the job. Charisma can blind search committees to a candidate’s lack of other critical leadership attributes.
  • Consider that perceptions of “professionalism” and “gravitas” are often based on white male characteristics. Have you ever been on a search committee when a candidate’s passion was cited as evidence of a lack of emotional control? How about when a quiet and reserved candidate was labeled “uninspiring” or “obviously not interested” in the position? I have observed introverted thinkers — who paused a beat before responding — get tagged as “too tentative” or “less prepared” than other candidates. Be open to the possibility that you and your search-committee colleagues are evaluating a candidate’s style based on what makes you comfortable rather than what is essential for the job you are seeking to fill.
  • Think about which candidates might offer new ways of thinking about old problems. Give greater consideration to candidates who are most likely to challenge assumptions, interrupt default ways of thinking, force difficult debates over complex issues, and ask “Why?” over and over again. While it may not be easy or even pleasant, constructive conflict typically yields better analysis and results than comfortable conversations do.

Spending time in search committees exploring the criteria to be used in evaluating candidates is a valuable strategy for interrupting decision-making patterns that give some groups advantages over others. These are not easy conversations, but they can reveal that we tend to favor candidates for leadership roles based on our default vision of what a leader looks like rather than being open to the possibility that we might benefit from something entirely different.

Allison M. Vaillancourt provides organizational consulting services as a vice president in Segal’s organizational-effectiveness practice. She retired in December 2019 as vice president for business affairs and human resources at the University of Arizona, after three decades as an administrator and faculty member at large public research universities. Browse her previous columns in the Management Corner series on administrative-career issues.

Join the Conversation

1 Comment

Log In or Sign Up to leave a comment.