Cyber Deception: Evaluation Metrics for Honeypot Research and Development

I have been researching honeypots for some time and needed to come up with some evaluation metrics. However, it is difficult to find effective metrics in current literature.

So, I started going through all the honeypot-related papers, documents, and whitepapers. I also asked chatGPT. And then I compiled all the evaluation metrics I have found out there.

General Metrics

There are several ways to measure the effectiveness of a honeypot:

1. Detection rate: This measures the percentage of attacks that are detected by the honeypot system. A higher detection rate indicates that the honeypot is more effective in identifying and capturing malicious activity.

2. False positive rate: This measures the percentage of non-malicious activity that is incorrectly identified as an attack by the honeypot system. A lower false positive rate indicates that the honeypot is better able to distinguish between legitimate and malicious activity.

3. Time to detection: This measures the amount of time it takes for the honeypot to detect an attack. A shorter time to detection indicates that the honeypot is more responsive to identifying malicious activity. Longer means less effective.

4. Type of attacks captured: This measures the types of attacks that the honeypot is able to detect. A honeypot that can detect a wide variety of attacks is considered to be more effective. Usually, honeypots are target-oriented and this metric is often ignored thereby.

5. Luring capability: This measures the honeypot’s ability to attract attackers by appearing to be a legitimate target. A honeypot that can deceive attackers into thinking it is a real system is considered to be more effective.

Overall, the effectiveness of a honeypot can be determined by evaluating a combination of these metrics.

After Redirection

If the honeypot does not attract, rather the defender redirects the attacker to a honeypot, researchers can use the following metrics for evaluation purpose:

1. Engagement rate: This measures the percentage of attackers that interact with the honeypot server. A higher engagement rate indicates that the attacker is more likely to be deceived.

2. Time spent on the honeypot: This measures the amount of time that the attacker spends interacting with the honeypot. A longer time spent on the honeypot indicates that the attacker is more likely to be deceived.

3. Type of actions taken by the attacker: This measures the types of actions that the attacker takes while interacting with the honeypot. An attacker who takes actions that are typically associated with a real target is more likely to be deceived.

4. Attack vector: This measures the specific method(s) used by the attacker to interact with the honeypot, such as exploiting vulnerabilities, collecting information, and trying to gain access to the honeypot. If the honeypot is able to deceive the attacker into thinking it is a real target, the attacker will likely use the same attack vector as they would use on a real target.

5. Logs and data collection : This measures the data that is collected from the attacker’s interactions with the honeypot, such as IP addresses, user agents, payloads, and commands, this data can be used to identify the attacker and track their activities, which can help in the understanding of the attack and the attackers’ behavior.

Human Subject

If conducted human experiments on honeypots, we researchers can measure the following:

1. Subject’s success rate: This measures the percentage of subjects that are able to successfully identify the honeypot. A lower success rate can indicate that the honeypot is more effective at deceiving the subjects.

2. Subject’s time to detect: This measures the amount of time that it takes for the subjects to identify the honeypot. A longer time to detect can indicate that the honeypot is more effective at deceiving the subjects.

3. Subject’s confidence level: This measures the subjects’ confidence level in their ability to identify the honeypot. A lower confidence level can indicate that the honeypot is more effective at deceiving the subjects.

4. Subject’s feedback: This measures the subjects’ feedback on the honeypot, such as how realistic it appeared, how difficult it was to identify, etc. Positive feedback can indicate that the honeypot is effective at deceiving the subjects.

5. Subject’s behavior: This measures the subjects’ behavior while interacting with the honeypot, such as the number of clicks, the number of pages visited, the number of attempts, etc. If the honeypot is able to deceive the subjects, their behavior should be similar to when they interact with real systems.

6. Subject’s attack vector: This measures the specific method(s) used by the subjects to interact with the honeypot, such as exploiting vulnerabilities, collecting information, and trying to gain access to the honeypot. If the honeypot is able to deceive the subjects, they will likely use the same attack vector as they would use on a real target.

7. Subject’s report: This measures the subjects’ report of the honeypot, which can give insight on how the honeypot was perceived, if it was effective in simulating a real target and if the subjects have any suggestions for improvements.

8. Subject’s level of expertise: This measures the level of expertise of the subjects, meaning how much they know about the field and how experienced they are. If the honeypot is able to deceive experts, it is considered more effective.

Please let me know in the comments if you have further to add! Best wishes for your research on Honeypots!