If you’ve read my blog posts on employee listening, design thinking and culture, you know I’m all for creating unique employee experiences that support business and talent strategies and foster high performance. Take the latest in human resources technologies: There’s certainly appeal in feedback platforms, gamification and apps that can predict employee behavior. But there’s also risk in adapting some of these trends in place of proven approaches, without using good, evidence-based HR practices.
So, should HR be proposing these tools to the business?
It’s a tough call. One way to help make the decision is to look at the predictive validity — the extent to which the tool or test can predict a certain behavior or outcome. Here are four key areas where I think the validity lens should be applied:
Strength of evidence
You need evidence to ensure the test or intervention will lead to the desired outcome (e.g., enhanced performance, productivity, retention or customer experience). Some examples I see gaining attention include:
- Surveys. There’s a trend to collect more regular employee feedback via do-it-yourself technology and “always on” surveys that may ask just one question. But the predictive validity of single-item scales is extremely low and using technology for more regular surveys could detract from the employee experience, especially if those surveys aren’t part of a well-designed employee listening strategy. And if the measures used aren’t validated, there’s a risk of poor data.
Sentiment or mood measures (often the focus of “always on” surveys) don’t have strong predictive validity either, though they may relate to performance. Is that a sufficient reason to use them? Depending on the context, and how the data are used, maybe, but keep in mind the relatively weak predictive validity. When possible, use other forms of validated data and feedback to validate findings. Again, I don’t think regularly asking employees if they’re happy is necessarily going to improve their employee experience. If an employee or a team receives feedback that they aren’t happy, what will the impact be on their experience?
- Gamification. It’s mainly used for recruitment or development. And while it can offer a great user experience, there’s little evidence it can predict performance.
- Technology for behavior change. There are numerous apps that purport to change behavior, via the individual logging their desired behavior change and tracking progress through reminders, even using virtual assistants to assess progress and intervene when the user is off course. But the their long-term impact on behaviour isn’t clear. Consider also whether being sent reminders by an app or virtual assistant enhances the employee experience.
Cost vs risk
There’s the direct cost (purchasing the psychometric test or licensing the software or app), but there are indirect costs as well. For example, consider apps that enable a manager to survey employees weekly on mood or work experience as a trigger to one-on-one follow-up discussions. In an organization with 500 managers with five direct reports each, this could amount to up to 1,250 hours per week of conversations (2,500 direct reports at 30 minutes each). That’s a lot of time to commit to a measure that may not have sufficiently proven links to performance.
Would managers invest the time and effort to regularly follow-up? Not doing so may actually detract from the employee experience. We often see engagement and views of leadership decline when employees are asked to provide feedback and that feedback isn’t acted on. There’s also the risk of false positives and negatives. If a psychometric tool is used for assessment of key sales leadership roles, but has poor predictive validity, the cost of a “poor” hiring decision may be significant.
Should the employee experience of the tool (e.g., gamification) be put ahead of predicting performance? Understanding the relationship between the performance and the value contributed to the organization is important when assessing risk in HR. My colleague, Ravin Jesuthasan, and his co-authors have written extensively about this in their book Transformative HR.
Long-term sustainability of performance gains
Will employee usage or compliance with an app or technology – and importantly the expected new behavior – be maintained over the long term? Will the technology usage be sustained if managers don’t quickly or sufficiently respond to employee feedback, or if the app reminders don’t result in behavior change? Consider the negative impacts on employee experience if they don’t.
“Fit for purpose” context
The final consideration is the context for the test or technology. Perhaps gamification has a role (without stringent evidence of validity) when used as part of a multiple assessments and for certain roles where the cost or risk of a “poor” decision is less? Or perhaps the user experience is important enough that it’s given greater weight than it otherwise might be, especially when used in conjunction with other proven measures. Perhaps the business is willing to take the risk because of other organizational considerations.
How will the data be used? Will greater focus on “HR Big Data” and analytics help answer broader organizational? If so, the standard of validity may need to be high to avoid “garbage in, garbage out.” In other words, validity is likely to be important when the data are relied on as input into other organizational decisions.
As I have posted previously, high-performance tends to come with the ability to improve the employee experience. The standard for validity should be high for measuring the more enduring aspects such as leadership effectiveness or culture. Otherwise, the cost and risk associated with low validity could be significant.