One of the easiest statistics to understand your company’s security awareness program is also one that is easily misinterpreted. I am talking about overall phish percentage. Overall phish percentage is most easily defined as the percentage of users in your company that opened a phishing assessment and took the action that it requested. For some real phishes, this is buying prepaid visa gift cards, for others it is logging into your “microsoft” account.

Overall phish percentage attempts to distill the effectiveness of a security awareness program into one easily understood piece of data. While it is able to give you some insight into your company’s performance, like any statistic, there are many ways to misinterpret and draw incorrect conclusions. Here are 4 common mistakes I see companies make when they are measuring the health of their security awareness program

1. One piece of information is not enough

In the same way you can’t judge your own health with just your heart rate, blood sugar, or blood pressure, you can’t judge the health of your security awareness program with only one piece of information. When judging the performance of your company and the health of your program, you need to take into account things like the previous training your employees have completed, recent events that might drastically increase susceptibility to certain phishing attacks, and turnover at your company that changes how your employees work and who they work with. Changes in the way people think and act will also change the phishes to which they are vulnerable. A Great Security Awareness Program will account for these when measuring performance

2. Easy to detect phishing assessments. 

If you judge a fish by its ability to breathe underwater…… well, it’s always going to perform well. If the phishing assessments for every employee are delivered at the same time it is very easy for them to send it around or have others identify the phishing threat for them. In this case, employees are not identifying a threat because it is a threat, they are identifying the “threat” because of the way it was administered. Almost all realistic threats won’t be emails or texts blasted to every employee at the same time, malicious actors usually select groups of people and try different threats. Remember, the goal of training employees to recognize phishing threats is that they will recognize the real threats they will likely encounter while working at home or in the office.

3. One size fits all phishing tests. 

Every human being is unique, and your employees are no exception to this. Every employee is uniquely vulnerable to social engineering attacks. For one employee it could be an urgent text from “Amazon” saying their package might not be delivered and they should log in now. For another it might be an email from their “boss” late on a Friday saying they need $500 in pre-paid gift cards for a client. Employee’s vulnerabilities are unique and when you train employees to recognize threats, you should be able to take that into account individual vulnerability and adjust.

4. Poorly interpreted results: 

There are many ways to interpret results to see the outcome you want. We can’t possibly go over every way this could happen, but be careful not to do the following. When you are measuring the Overall Phish Percentage of your employees month over month, you may see a 4% phish rate for 3 months in a row and immediately celebrate! However, if in every month a different 4% of your employees fell for a phishing threat, in reality you have a 12% phishing rate for that quarter. Be careful of this and make sure to inspect your data. Many things can be automated, but sometimes a human touch is necessary.