Thanks very much for considering this, @rgonzalez. I worry about “fidelity” too, but as described in the Rules Severities, “severity” has more to do with how badly your infrastructure is affected by the thing the rule detects. When I’m trying to assess whether I think a rule will give me false positives, I’m looking at tells that give me confidence that the rule is going to match on something very specific. A rule that’s very specific has a higher chance of being “accurate”, in my experience, and I regard those of high accuracy and Critical severity as the highest “fidelity”.
Could expanding the use of the “Confidence” tag help drive us toward a “fidelity” signal? That is, if something is of Critical severity and “High” confidence, it has high “fidelity”? I’ve been designing a workflow for assessing rules automatically that makes multi-field comparisons like this outside of surciata-update, generating a disable.conf file full of rules I want to turn off. Having the Confidence field available for more rules would really help.
And for the record, I’m not going to go around demanding that all rules be of “Critical” severity and “High” confidence. There are times where I’m less concerned about a rule’s accuracy, and more concerned about getting a detection for a suspected case of something bad. For example, I’ve been impressed by how quickly you all have gotten signatures into the rule set when there’s a Zero Day exploit. When that happens, I don’t mind alerting on something that’s of medium or low accuracy; if it’s an emerging threat and is of Critical severity, I want to err on the side of alerting on it. If they can’t start out as being highly accurate, I can use the updated_at field to help me tune out lower-confidence signatures as they age using the process I’m developing.