Psychometrics Schmychometrics: Why everyone need to care about the numbers - Notes from the Nerd Desk
- By Elana Duffy, Pathfinder CEO
Since the ol' Nerd Desk can get a little cluttered with, well, nerd talk, here's why you should read this month's blog:
The team at Pathfinder Labs talks (or listens to their CEO talk) an awful lot about impact: what is the lasting effect of a benefit or program on the lives of those the provider serves? How do potential program attendees figure out which organization will give them the best results when solving their problem? If organizations are generally limited to annual, self-report "how do you feel" surveys and testimonials, how can a potential funder know their donation is effective? To answer each of these questions (and more!), we need to establish measurable, comparable standards for assessing impact: we need psychometrics.
What makes the metrics psycho, and what do they have to do with impact?
Well, psychometrics are a systematic deduction of measured criteria that place statistically consistent values on a variety of potential attributes defining aspects of psychology, produced through a series of verified methodologies which create as little deviation as possible during repetitive trials.
Got it?
Yeah, you are not alone if you don't. Check this out:
Experimental and applied psychology - the branch of psychology conducting most studies on psychometrics - focus nearly all of their efforts on research. These scientists generally spend their days conducting controlled experiments to record a numeric measurement of how much a person has of this personality trait versus that one, or displaying a behavior or showing an emotion. The experiments' results are compared with results of other studies and methods for validation, put on scales and charts to check for important math words like "normal distribution" and "minimized outliers," and then the experiments are repeated. When the same outcomes occur again and again, the scales can be defined as statistically accurate and the scientists can publish the study results as metrics (standardized, numerical measures). Psycho(logical) metrics. Psychometrics. Boom.
The study of impact (how much of an effect something has on something else) is generally rooted in three aspects of psychology: personality, behavior, and emotion. These aspects influence how we interact with our environment, and how we act and react from experience to experience. So if something influences a change in one or more of those aspects from one interaction to the next, that influencing event had an impact.
Putting it all together, psychometrics help us put our emotional responses and likely behavioral motivations - and the personality traits that caused them - on a scale so we can look for changes, thereby identifying an experience's impact.
Why understanding personality is so critical to understanding impact
For services and support programs, measuring influence on participant personality is perhaps the most important psychological metric an organization can provide when reporting impact to stakeholders. While we can't see it or control it, personality is the underlying influence behind everything we do.
Imagine driving along a stretch of straight, flat, open road. There are no speed traps, but also you have nowhere pressing to be. You start out going a consistent speed. After awhile, you look down at the speedometer. You may be going faster (excitement-seeking), slower (laid-back), or the same speed (diligent) as when you started. Your personality is what drove (pun somewhat intended) your foot to press or release the gas - the behavior - and allows you to enjoy the ride or just the comfort of a consistent pace - the sentiment.
Personality forms the base of the psychology layer cake, so it is critical to get an objective assessment before we can add the behavioral fillings and emotional icing. While most of us think the least about the base, it is what gives us a structure for complimentary flavors. If your base is graham cracker, for instance, chances are good that your middle is cheesecake and not carob and the topping is strawberry sauce instead of glaze. Similarly, if we have a diligent personality we are unlikely to show behaviors of speeding or emotions of exuberance, and we would get little benefit from test driving a Ferrari. For people seeking services, knowing the mix of dominant personality traits could tell us if a person would get more benefit from music therapy or equine therapy.
The base is also the most consistent part of the cake; is it very hard to change its flavor after the ingredients are baked. It can be done, though: consider a black forest cake where the sponge is soaked in potent rum. Changing the particular mix of traits in a personality is also difficult, and requires a similarly strong and persistent influence. But if you can get under the behaviors and emotions and increase someone's predisposition towards being optimistic - psychologically soaking their sponge cake in rum - you've now affected their behavior and emotions in a more permanent way than seeing if a program made them feel happy in the moment.
Bottom line? Lasting impact comes from a program affecting personality more than emotion or behavior.
Great! So let's get measuring, nerds!
There's a catch, though: personality is subconscious, and most means of measuring it - either surveys or behavioral observation - involve our conscious minds. That makes it really hard to get an objective, standardized metric.
Emotion and behavior are pretty easy to gauge: we can observe our behavior (how we act) and feel our emotions (how we feel). On the other hand, personality is innate; it's who you are most comfortable being. There's no conscious decision with personality; you don't wake up saying, "I think I'll be outgoing with a touch of adventurous today." It is based a bit on chemicals like serotonin and dopamine and other natural compounds, and on learning from your experiences.
Look at the car example: why don't we just conduct the open road experiment over and over to measure of personality? Because of bias: your conscious mind can self-censor and give a false result. The first time you drive the road, your body reacts naturally. The next time you drive the stretch of road, however, you know what to expect. Or you might have an emotional response, maybe you get bored on the same stretch and speed up to get it over with. And for personality, if you are simply self-conscious of being perceived in a particular way for having sped up or slowed down you may react to the discomfort. In this experiment, too much is conscious: there is no way to determine what is causing the behaviors from test to test.
This is where computers come in. Psychometric scales are created using experiments to measure subconscious traits without relying on observable behaviors or sentiment (we have other nerd desk blogs on this). This makes the values more reliable because the same outcomes are able to be produced over and over without the subject influencing results through their behavior. And because psychometrics are standardized, we can examine a trait over time and see if there are statistically significant differences before and after a program. If there are, we have a big indicator that person has experienced a fundamental change: their whole cake is going to be a little different going forward.
That program had real impact, and they got the numbers to prove it.
In summary, psychometrics are cool enough to care about
So what are psychometrics, really? They are the scientifically-bound, mathematical values given to aspects of psychology that seldom change, allowing us to measure and compare what makes us who we are. They tell us how we are likely to behave on an open road, and allow us to recognize what it is about ourselves we might want to work on so we can eventually slow down or speed up. They also provide an unbiased, nonjudgmental way to look at why we do the things we do and to track any improvements we choose to make.
And best of all, they give us a way to talk about cake and computers and have it make sense.