Phishing is prevalent because it provides big rewards for relatively little effort on the part of a hacker. In phishing scams, recipients get an email requesting that they perform a seemingly benign act that is actually quite malicious in nature. The past decade of security research shows that employees regularly fall victim to attacks, and that phishing is considered one of the easiest ways to access corporate infrastructure. Of course, the majority of corporate employees won’t fall victim to a phishing attack–but if your organization employs 1,000 employees, there’s a good chance that at some point, at least 100 will. When they do, they’ll inadvertently involve themselves in one of these malicious activities:
– Click a link that leads to a download website
– Post sensitive information to websites
– Open high-risk attachments
Measuring the effectiveness of any program should be based on performance outcomes. For security awareness programs, this entails charting secure vs. insecure practices and measuring them–both prior to the program and afterwards. If your company has a clean desk policy, your first step would be to measure compliance with this policy (say for example that 62% comply with the policy) and then decide what would satisfy you in terms of security (for instance, 90% compliance). A program is effective if your goal is met, and is only deemed highly effective if it is sustained over time. What does not count as an effectiveness measurement is measuring employee satisfaction, or how many people have passed its tests. Though frequently cited by managers, these figures measure niceties and wishful thinking, whereas you’re interested in security.
Merriam-Webster dictionary defines awareness as “knowing that something (such as a situation, condition, or problem) exists”. Phishing awareness is when people say “Hmmm….phishing? I know what that is; it’s a fraudulent email.” Awareness does not drive action, nor is it necessarily related to it. For example, most smokers are aware of the risks related to smoking, as are those driving across an intersection during a red light. As security professionals, we’re interested in secure practices rather than convictions. From a corporate security perspective, if Jane does not open a phishing email without knowing what phishing is, it’s much better than if Dave opens that same phishing email, but can define what phishing is. Most awareness programs are focused on making employees aware and are measured accordingly (that is why they are called awareness programs). If phishing concerns you, you should take the required steps to reduce the chances employees will actually fall prey to it.
When measuring the costs of security training programs we should factor in three elements: cost of training preparation, time lost on employee training and engagement costs. Let’s break these into subcategories:
Cost of training preparation (direct costs): These should include money paid to external parties, such as consultants, software licenses and time-related costs – such as costs associated with running the program, monitoring the learning software performance and time running around hoping everything works.
Time lost on employee training (indirect costs): This is straightforward to calculate: it’s the time of the training + travel costs. Travel costs are the time it takes get everyone in the training. (If it is led by a trainer, you can add in an extra 30 minutes for gathering people in the office; if it’s computer-based in workstations, then there’s only time spent on actual training.)
Engagement costs: These are the subtle costs. Do employees think favorably of security? Did they feel you’ve wasted their time? A good training program brings employees onboard, but a bad one pushes them away.
When looking to purchase training-related content or technologies, most organizations only examine direct costs; even with those, the focus is mainly on money spent. But the actual bill lies ahead, once the program is deployed.
One of the biggest challenges for any security professional is to adapt content into laymen’s terms. As a security professional, you may think to yourself: “If I write an email about a famous hack or link to a news report, employees will certainly make the required inferences.”
However, the truth is that as relevant as your content may seem, the challenge for the employee is to translate the lessons learned into day-to-day operations. So if you’d like to have your content not only read but actually implemented, you need to include a lessons learned section with key takeaways based on actual interorganizational situations that can be used as a guide to proper implementation. There’s a big difference in the lessons learned from a USB drive infection, for example, if USB drives are or are not in prevalent use across your organization.
On-the-job (OJT) training is a training paradigm that calls for training new and veteran employees through performance in real life scenarios and immediate feedback. Some categorize OJT with the concept of 70-20-10 in training, which claims that 70% of knowledge is gained through experience, 20% through peers, and only 10% through formal training. OJT and experience-led training is especially important for the implementation of practices, rather than theoretical knowledge. As phishing requires the implementation of secure practices, training methods that emphasize practice over memorisation have higher chances to succeed. OJT and 70-20-10 based methods are such approaches.
Measuring any training program is complicated; as such, we’d like to suggest metrics from different perspectives:
Readiness – has the susceptibility to phishing declined? Measuring internal infection rates can be a good indication of that
Smart learning – Are employees adapting to change/can they identify new attacks? This can be measured through control group simulations.
Engagement – Are employees satisfied? Is management satisfied? Engagement in this context means that employees are becoming more and more active with security, looking for ways to improve your organization.
Return On Investment – Were you able to reduce costs in other parts of your security infrastructure because of this program? Are you improving employees’ time utilization (or not)? When factoring in all the unforeseen costs, is the pricing still competitive?
Deployment – Is every technical aspect working properly? Are you registering the correct clicks? Is the content live and operational, or are the servers down (again)?
It’s true that if we scheduled all phishing simulations according to predefined times of day, it would allow help desk personnel to better prepare and reduce the overall cost. However, when performed this way, it might invoke a social desirability bias in which employees over-report to help desk, because they’re now aware of an ongoing exercise and their willingness to receive a good score. Minimizing help desk calls is a combination of simulation scheduling, email content variation and proper training of the help desk team prior to the simulation.