Engagement Research: Extensive Effort, Limited Insight
Doing employee engagement research is hard. I did it for a little while and grew to understand that data collection and research design is much easier in academic settings where the lead researcher has significant control over most of the critical research parameters. Research by committee is the fastest road to an ulcer. However, the challenge of non-academic personnel research pales in comparison to the task of drawing meaningful conclusions from that data. And that’s the entire point of the whole thing, is it not?
The path to insight and meaningful interventions is littered with obstacles, none of which are insurmountable, daunting though they may seem. Doing so requires more courage, creativity and strategic thinking than organizational agents can afford to employ. I hope to talk about a few of these obstacles and potential solutions in future posts (promises, promises, I know), but for now I wish to highlight a discussion on this topic from the Engagement Factor blog.
Summarizing a bit, the author indicates that the types of studies we normally undertake in the engagement research industry have an unfortunate tendency of not being particularly informative (I may be taking some liberties here, so read the article yourself in the link above). Much of this research takes place within a single organization. In my own experience, limited as it may be, we do this because normally a single organization is paying the bills. Of course, it is also much easier to obtain data from single organizations. I have found clients can be very skittish of cross-organizational comparisons as these have the potential of generating less-than positive results. Apparently no one wants to be the bearer of bad news; quite frankly, I can’t blame them.
So how does this degrade the utility of expensive engagement research projects? If we study a single organization we are blind to the influence of higher-order organizational phenomena like executive leadership, performance management systems, compensation strategies, organizational culture and most of those variables described in organizational justice literature. Each of these critically important variables are shared by the organization as a whole (yes, we often see differences across departments and organizational units- that’s a related can of worms just waiting to be opened). We cannot find an effect for differences in downward communication practices, for example, because they are largely shared by the entire organization. In psychological/statistical/research parlance, there simply isn’t enough variance in the independent variables (one could think of it as range restriction, sadly I couldn’t find an accessible definition of this concept on the web. You’re just going to have to trust me). What we are left with then, is the impact that individual supervisors have on the engagement of their charges. That is about the only thing that is left to vary within an organization, so that’s where engagement research usually points.
So, how do we solve this issue? How do we design engagement research such that we can identify the impact of higher-order organizational features on the affect, cognitions and behavior of its employees? The answer lies not in improving your survey (though those often need a lot of work too). The answer lies not in the research instrument, but in the research design. For the purposes of this discussion, multi-organizational research is probably the best option. Though longitudinal research designs could help a bit, it is often difficult to isolate the effects of macroeconomic factors on employee attitudes. The key lies in multi-organizational research. In my time one of my previous employers, I became aware of at least two such industry specific data consortium (one for global manufacturers and another for a group of smaller financial institutions)and we tried to establish at least one more (I’m not sure how that went after I “left”). This is a fantastic first step. However, at the moment data-sharing tends to be too shallow to glean meaningful insights. Organizations tend to compare their survey results against collective benchmarks, rather than engaging in the type of statistical analysis required to determining if they are performing differently than the rest of the industry. While this is better than nothing in that it makes us feel like we are doing something, I am not confident that it actually is different from doing nothing at all (from a statistical perspective, of course).