5 Reasons Phishing Simulations Tests Fail

We’ve been running successful phishing simulations for years now, developing proven steps toward employee engagement and behavioral change. Over the course of time, we’ve seen customers come to us, bruised by phishing campaigns gone awry. In our experience, most of these catastrophes come down to a handful of errors we’ve seen repeated time and time again.

Here are five of the most common mistakes to avoid to help you avoid the pitfalls of less-than-optimal security tests:

1. Choosing phishing simulations that are too difficult

A security team may make incorrect assumptions about what employees already know about phishing. They come to the mistaken belief that other members of the organization possess the same level of familiarity with cybersecurity as they do. This ‘Curse of Knowledge’ creates a lot of negative noise while producing few, if any, positive results.     

This also happens when security teams become concerned with establishing a definitive security risk worthy of their organization’s attention and resources. This leads to security teams creating and deploying phishing simulations that are too challenging for those without subject-matter expertise. When setting the bar too high and creating simulations most employees fall for, employees start to question the reasoning behind the simulations themselves. They may ask, “Are they meant to entrap us or teach us?” or “Should I really have detected this, or was I meant to fail?”

Learning has emotional roots. It’s best facilitated when people are eased into the process. This enables an employee to understand it’s their mistake instead of the result of an uneven trial by security.

2. Targeting only part of the organization

Some groups or departments are more vulnerable to risk than others. Maybe their usage of email differs or their employees have elevated permissions. This may lead to a decision to target only parts of the organization. But what we, as a security community, have learned from past breaches is that time and time again they prove that attackers will start wherever they can before moving laterally throughout an organization.

More importantly, this selective targeting has two downsides coupled with it:

• It signals to employees and management that phishing is a threat to only the part of the organization that is being tested and trained. When creating a security culture, we want each and every employee on board and this cannot be achieved when only some are being trained.
• It takes the discussion away from the real issue at hand—phishing—and leads to individual employees or departments worrying about why they are being targeted with anti-phishing simulation. Ultimately, it fosters a culture of distrust.

3. Failing to create engaging content

We’ve learned through experience that employees will tune out potential lessons learned from a phishing simulation when the content does not match or reflect the situation. If a security team gets too focused on providing exhaustively comprehensive training materials, they will facilitate cognitive overload rather than learning experiences. And if the content is too general (e.g., not localized, not concise enough), employees simply won’t consume it. As the saying goes, you don’t have a second chance to make a first impression–and those first few seconds in which employees see the materials produced for them will impact their willingness to engage with it.

This is, at its heart, a resource issue, as developing dynamic content requires both expertise in areas related to organizational learning and development, and the resources to create multiple versions. Many times security teams are unable or do not want to invest departmental resources into the creation of dynamic content. It leads to a one-size-fits-all approach that provides too much information and does not focus on what employees need to do: learn from their mistakes.  

4. Sending phishing simulations to everyone at the same time

There are many aspects related to phishing simulations that aren’t training-related—for example, monitoring the process, making sure emails are sent, and getting the help desk on board so they don’t collapse under the burden. Important things indeed, which lead many security teams to the almost intuitive belief that sending a phishing simulation to all employees on the same day and at the same time is a good thing. What better way to ensure improved monitoring of the process and lessen the burden for IT?

Unfortunately, this creates the opposite effect. The first employees to identify the simulation email or fall for the trick alert others. Their coworkers are now prepared for the exercise when they see it—prepared, that is, to call the IT helpdesk to report the phishing incident without actually going through the simulation or the training. This is actually a case of social desirability bias—employees will now over-report incidents to IT without personally having spotted them in order to be viewed more favorably. The result? System overload, a swamped IT department, and employees who’ve missed out on an effective training opportunity.

Ultimately, this leads to click rates that are unrealistically low and report rates that are too high; now everyone thinks these tests are low level or unimportant, and the metrics do not show the actual risk or issues to be addressed. This can both hurt secondary training efforts and credibility with top management.

5. Focusing on failures rather than results

There is a common belief that if we tell employees about the phishing simulations, they’ll be alert, and therefore, springing a surprise is the way to go. As a result, companies forget to communicate about phishing simulations with relevant stakeholders, such as the HR department, or the employees themselves. As mentioned earlier, there are a lot of emotions involved in learning, and failure to communicate can backlash.

We cannot overemphasize the importance of clear communications. It is critical not only for rolling out the simulations and getting backed up by senior stakeholders but also to the essence of phishing simulations—changing employee behavior. We need the employees with us, as their goal is our goal: the organization’s prosperity.

And eventually, we shouldn’t care so much about the failures themselves; rather, we should focus on the trends, for example:

• Do people click twice, or three times?
• Are employees engaged?
• Have we moved the needle when it comes to embracing an employee security culture?

Getting employees engaged means driving learning effectiveness.

Let’s not lose sight of the goal: behavioral change, not proving people wrong. Phishing training is about engendering a long-term shift, not a single ‘gotcha!’ moment.

Prepare for success from day one

If you set the right challenge level, focus on organizational learning, and engage rather than alienate your employees and stakeholders, you’ll be well-positioned to make your next security test a resounding success. In our next post, we’ll provide some simple steps that will help you do just that—and save time in the process.