Three defensive controls that applied successfully and enforced nothing

Security controls are usually evaluated twice.

First, we ask whether the control was applied. Did the password reset command succeed? Does the firewall rule exist? Did the detection stack load the ruleset?

Then, often implicitly, we assume the system is safer.

We have started treating those as separate measurements.

While testing defensive interventions against live attacks, we found three cases where a control reported the expected state but did not produce the expected outcome:

1. A password reset succeeded, but the old NTLM credential still authenticated.
2. A firewall block rule existed, but the firewall enforcing it was disabled.
3. Authentication was logged correctly, but no detection rule generated an alert.

All three controls looked healthy if inspected at the configuration layer. Only end-to-end attack measurement showed that they were not doing what we expected.

The common failure is simple:

The interesting part is why.

01Credential rotation and the old-password window

Our first intervention was straightforward. Reset the password of the account used by the attacker with Set-ADAccountPassword -Reset, then rerun the same attack with the previous credential.

The reset succeeded. We expected attack success to fall. It did not.

Across 24 matched pre/post attack pairs, the measured change in success was zero:

The explanation was Windows' old-password behavior for NTLM authentication. The relevant setting is:

HKLM\SYSTEM\CurrentControlSet\Control\Lsa\OldPasswordAllowedPeriod

The default period in the environment was 60 minutes. During that window, the previous password remained usable for NTLM authentication.

The password reset had worked exactly as requested. The directory state had changed. The attacker could still authenticate with the credential we expected the intervention to invalidate.

Setting OldPasswordAllowedPeriod to 0 before rotation changed the measured behavior immediately. The old password began returning STATUS_LOGON_FAILURE, and the intervention finally produced the expected measurable effect.

The lesson is not that credential rotation is ineffective. It is that the semantics of "password changed" and "old credential immediately rejected everywhere" are not identical.

For an intervention benchmark, this is enough to reverse the result. A rotate-and-retest workflow performed inside the grace period can conclude that password rotation has no effect. An attacker holding the previous NTLM credential receives the same window.

02A firewall rule in a machine with the firewall disabled

The second failure was even simpler.

We created a Windows Firewall rule named Block-Attacker-10.0.0.4 with protocol Any. We verified that the rule existed. The intervention harness reported success.

Then we reran the attacks. 38 of 42 still succeeded.

The resulting diagnostic statistic even showed a negative correlation, \(\rho = -0.458\). Taken literally, the data suggested that segmentation might be associated with more exposure.

There was no interesting security phenomenon underneath that number. The Windows Firewall was disabled.

The range was intentionally vulnerable and shipped in that state. We had successfully installed a rule into an enforcement system that was not enforcing rules.

The fix was to move the control one layer outward and apply segmentation through an Azure Network Security Group. That exposed a second implementation detail.

We first attempted to create the deny rule at priority 90. Azure rejected it with SecurityRuleInvalidPriority: the minimum priority is 100, and priority 100 was already occupied by an existing allow rule.

The working intervention was therefore not to add the new rule. It was to change the existing priority-100 rule from Allow to Deny.

Once again, configuration intent and network effect were different objects.

This is an unusually clean example of a broader measurement problem.

If our verification query had been "does a rule named Block-Attacker-10.0.0.4 exist?", the intervention would pass.

If our verification query was "can a packet from 10.0.0.4 still reach the target?", it failed.

The second question is the one we care about.

03The credential use was logged. It was not detected.

The third case inverted the problem. This time, the underlying system generated exactly the telemetry we needed.

A real NTLMv2 network logon against the Samba domain controller produced:

Auth: [SMB2,NTLMSSP] ... status [NT_STATUS_OK]

The authentication was visible in the logs. We ran a real Sigma ruleset and Suricata over the activity. No rule fired.

We then tested the same signature stack against credential theft activity: an RC4 Kerberos service-ticket request.

This asymmetry is useful. The logging pipeline was not broken. The signature engine was not globally broken. The ruleset could recognize related credential-theft activity. It simply did not contain useful content for the credential-use event we were testing.

Across the inspected ruleset, the only credential-use signature was a low-severity rule for successful SSH password authentication. That rule has a different problem: successful password authentication is also normal behavior.

In other words, the telemetry existed, but the signature layer could not turn it into a high-quality alert.

This is a different kind of missing control. More logging would not solve it.

The system needs context about how the credential is being used: what principal is authenticating, from where, to which system, at what time, following what preceding events, and whether that behavior deviates from expected activity. That is closer to behavioral detection than to a static signature.

04Three mechanisms, one measurement error

The failures have almost nothing in common at the implementation level.

Credential rotation failed our immediate-effect expectation because the old NTLM password remained valid during a grace period. Segmentation failed because the configured enforcement mechanism was disabled. Credential-use detection failed because the event was logged but no relevant detection logic converted it into an alert.

Yet a state-based validation system would report all three interventions as healthy.

Each intervention passed a plausible control-state check. None initially produced its expected end-to-end security effect.

This suggests a stricter definition of an intervention.

A defensive control should not be considered successfully applied merely because its desired state is present. It should be considered applied when we can observe the intended change in the system it is supposed to affect.

The verification should occur at the output of the mechanism, not at its configuration interface.

05Why this matters for automated security systems

This becomes more important as security testing becomes automated.

An agent can run a PowerShell command and receive exit code 0. It can call a cloud API and receive a successful resource update. It can load a detection ruleset without error.

Those are easy signals for an automated system to optimize. They are also incomplete.

If the objective is to predict how defensive changes alter real attack exposure, the relevant variable is not control configured but something closer to control effect verified:

That requires interacting with the system after the change. Try the old credential. Send the packet. Repeat the attack. Inspect the alert stream.

This is more expensive than reading configuration state. It is also much harder to fool accidentally.

06Compliance and security are asking different questions

State checks are not useless. They are often exactly what configuration management and compliance systems are designed to perform.

A policy may reasonably require the host to contain a firewall rule or an account to have been rotated within a specific interval.

But those checks answer "is the desired configuration present?" They do not necessarily answer "did exposure change?"

Confusing those questions is particularly dangerous when configuration data is used as ground truth for training or evaluating predictive security systems. The label can be wrong even though the configuration database is correct.

07Limitations

These examples therefore do not imply that these defensive controls are generally ineffective. They demonstrate something narrower and, for our purposes, more important: the control's configuration state did not tell us its measured effect.

08What's next

We are making effect verification a first-class component of our intervention harness. Each defensive action will increasingly be paired with a postcondition at the level it is intended to change.

We also plan to repeat the credential-use experiment against the alert stream of a production behavioral EDR, where richer context should make this class of detection more tractable.

The principle is simple. A password was rotated. Test the old password. A firewall rule was installed. Send the packet. A detection rule was deployed. Run the behavior.