A breach gets discovered four months after it started. The security team goes to pull the logs that would show exactly how the attacker got in and what they touched - and the logs from that period are already gone, purged by a retention policy nobody had really examined since it was set, years earlier, mostly to control storage cost.
Why Retention Policies Usually Get Set for the Wrong Reason
Log retention periods frequently get decided primarily by storage cost considerations - keep logs for thirty or ninety days, then purge to control the ongoing bill - without much deliberate connection to how long it actually takes to detect the kinds of incidents an organization is realistically exposed to. This is backwards in a way that only becomes obvious after the fact: the retention window should be driven by realistic detection timelines for the threats an organization actually faces, with storage cost as a real constraint to manage within that requirement, not the primary variable setting the requirement itself.
Why the Real Dwell Time Numbers Should Set the Floor
Industry breach investigation data consistently shows real median dwell time - the gap between initial compromise and actual detection - running well into months for a meaningful share of breaches, not days. A retention policy shorter than that realistic detection window guarantees that a genuinely significant share of incidents will be discovered only after the very logs that would explain what happened have already been purged, leaving the investigation with no way to reconstruct the actual attack timeline, scope, or root cause with any real confidence.
Why Different Log Types Reasonably Need Different Retention
Not every log type needs to be retained for the same duration, and a uniform retention policy applied blindly across all log types is usually either needlessly expensive for low-value logs or dangerously short for the specific logs that matter most during a real investigation. Authentication logs, privileged access logs, and network flow data connected to genuinely sensitive systems generally warrant longer retention than routine application debug logs, which have much lower forensic value in a typical security investigation and can reasonably be purged sooner without meaningfully weakening the organization’s real investigative capability.
Why Compressed Cold Storage Changes the Economics
Extended retention doesn’t have to mean extended cost at the same rate - tiered storage architectures that move older logs to cheaper, compressed cold storage after an initial hot-storage window meaningfully reduce the cost of longer retention while still preserving the actual data for the rare but genuinely critical case where it’s needed for a delayed-discovery investigation. Organizations reluctant to extend retention purely on cost grounds are often comparing extended hot-storage cost against the status quo, without seriously evaluating tiered storage as the actual, materially cheaper way to get meaningfully longer retention without paying full hot-storage rates the whole way through.
