There are lots of organizational phishing awareness programs dedicated to giving employees the tools they need to recognize phishing scams. To prove their program’s effectiveness, companies often highlight their click rate, or the amount of people visiting webpages via links in emails. But how much is the click-rate alone telling you about your organizational phishing risks?
Let’s look at it another way — How do you rate the achievement and ability of a Major League baseball player?
Back in the old days, it was up to franchise managers to trade, hire and fire players based on little more than intuition, arm strength and RBIs. Despite the billions of dollars on the table, for over 100 years, rating players was almost a guessing game.
If you’re familiar with Michael Lewis’s 2003 best selling book Moneyball, you’ll know that today, rating a player is more of an empirical science than an artful guessing game. His exploration of Major League Baseball’s early use of the statisticl method known as Sabermetrics proves that to truly measure a player’s performance, piles of hardcore data must be analyzed and pulled apart. While this method isn’t without its flaws, it’s much more of a “sure thing”, a critical factor when a whole lot of money is involved.
Measuring Success in Phishing Awareness- What Metrics Should You Rely On?
So how do you rate the success of a phishing awareness program?
Sure, the click rate is used as the benchmark to understand if a training program is working or not and what the current organizational threat level is. But how much is it really telling you? Is the click rate alone a metric you can rely on to give an accurate picture of the risks imposed by your employees’ behavior? Or, are we falling for the same bias as decades of baseball teams? By narrowing our perspective to one lone measurement in hopes that it will provide simple, at-a-glance insights, we fail to see the entire, and far more accurate, picture.
The Fallacy of the Click Rate Explained
To illustrate, let’s look at two organizations, Companies A and B, both of which have 100 employees each. Each company has 10 employees falling for phishing simulations every month, which yields a constant 10 percent click rate year round at both companies. This rate implies no improvement at all at either company.
But here’s the catch; At Company A, it’s always the same 10 employees clicking links and falling for the simulation. This means that 90 employees were aware and ready for phishing emails in the first place; 10 have not improved at all, which can be seen by looking at the click rate. But over at Company B, it’s always different people falling for simulations, and here is the differentiating factor: no employee falls for the simulations more than once.
Here at company B, we have a great learning achievement, but it’s one that cannot be deduced by looking at the click rate alone. Though the click rate remained the same at both companies, the level of learning and internalizing the message of the program was entirely different.
So how can you determine with accuracy whether or not your phishing awareness program is working?
Imagine the following; It’s the year 1988 and as the manager for a Major League Baseball team, it’s your job to pick your team’s draft. You want the guys with the most home runs, the strongest arms. In your narrow-sightedness, you fail to notice Mike Piazza. As the 1390th pick of the season, Piazza was overlooked due to a lousy pitching record. But this bottom-of-the-barrel pick went on to become of one the greatest hitting catchers in baseball history.
Had you assessed his performance with a whole slew of parameters rather than just one limited metric, his greatness may have been much more apparent from the start. The problem with looking at just one stat is that it’s often meaningless. Whether it’s RBIs or click rates, using just one metric leads people astray and unaware of what’s really taking place.
Looking Beyond the Click Rate
So if you can’t trust your click rate alone, how can you accurately measure success? What metrics should you be using in combination to create one complete and accurate picture of whether your training program is meeting its goals or not? How can you determine if your employees are really becoming ever-more successful in avoiding phishing attempts with each iteration?
To make an accurate determination, you’ll need to assess the time period, variety of emails and the population receiving the emails to create an accurate overall picture. Let’s look at how using each of these metrics provides a much more accurate understanding of how your own team is doing.
Time period:
Compare the results of, for example, the seventh simulation with the results of the third simulation. This helps determine if your employees are internalizing the messages of the training more and more over time.
Variety:
Since in reality, no two phishing scams are exactly the same, you need to determine how employees perform on a wide range of emails rather than with the same templates over and over. For example, they may know to be wary when it comes to emails messages with perceived benefits, i.e. “Click here to redeem your free trip to Cancun!” Threatening emails, on the other hand, that appear to come from the CEO, IT department or law enforcement agencies may elicit a different responses and reactions.
Population:
Employees come and go, so determine if you’re looking at the same group of people or if there are new employees to take into account. Newer employees don’t have the same amount of learning under their belt. Therefore, it’s not fair nor realistic to expect the same level of maturity in learning as employees who have been going through the simulation process for a longer period of time.
Using The Right Metrics to Turn Your Game Around
These metrics are just the beginning; there are more that need to be tossed and turned over to really understand your organizational risks and successes. But knowing that the click rate alone isn’t enough is the first step to getting that clarity.
Employing sabermetrics helped turn baseball rankings from intuition into a science, changing the face of the sport in the process. When it comes to helping your organization avoid phishing attacks, you too need science rather than intuition to guide your efforts. Now is the time to abandon your click rate fixation and begin to understand if your phishing awareness efforts are really working or not.
