On this page
Review update: The example collection commands now set their own placeholder variables, named as in the runbook. The earlier example read a private state file that was never distributed and that no script creates.
What This Lab Demonstrated
A two-replica Deployment changed its pod template to set readOnlyRootFilesystem: true. Both pods were replaced. A subsequent Defender registry recommendation kept its exact assessment and resource identities while naming a verified post-rollout pod. The root-filesystem recommendation disappeared from complete ARM and Resource Graph exports; a direct read of its old address returned 404.
Registry identity stability is observed for this rollout. Root-filesystem absence is observed; an explicit Healthy or resolved assessment is not. The companion article explains the operational implications of Microsoftโs October 5 controller-scope announcement.
Downloads and Provenance
- Evidence-only ZIP: 10 sanitized result files, an evidence README and a hash manifest. It contains no source code.
- Post-rollout result JSON.
- Complete source ZIP: the exact 43-file lab source tree, including the runbook, manifests, collectors, helpers, tests and results. It contains no credentials or private captures.
- Source archive manifest: commit identity, archive checksum and per-file hashes.
The source ZIP matches published commit 7678f84 exactly. The archive manifest records the commit and file hashes. Evidence files retain dated review snapshots, including historical pre-publication flags; the operational and cleanup records carry their own timestamps.
Raw tenant/resource identifiers, full service responses, pod UIDs, and kubeconfig stay private. Shareable excerpts use documented pod aliases. Microsoft recommendation identifiers are retained because they identify the checks, not the tenant. SHA-256 sidecars detect changes to captured bytes; they do not independently authenticate a fileโs provider origin.
Run 2: Product Evidence
Dates use UTC. Rows follow the order of observations; the downloadable evidence retains exact capture and service timestamps.
Run 2 observation timelineSwipe across for columns
| Date | Observation | What it establishes |
|---|---|---|
| October 7 | Two-pod writable-root baseline captured | Initial Kubernetes state |
| October 8 | Baseline: matching Unhealthy root-filesystem and registry recommendations in ARM and Resource Graph | Actual service findings with KSPM-Agentless and Deployment attribution |
| October 8 | Pod replacement: deletion of one baseline pod accepted | Kubernetes replacement; isolated Defender follow-up remained inconclusive |
| October 8 | Template fix: only root-filesystem field patched | Controlled remediation intervention |
| October 8 | Rollout: same Deployment, generation 2, new ReplicaSet and two new ready pods | Completed Kubernetes rollout with read-only roots and unchanged image |
| October 9 | Earlier capture: ARM still returned the root-filesystem finding | Last captured presence |
| October 9 | Later captures: ARG / ARM no longer returned rootfs; registry referenced a rollout pod with unchanged assessment/resource IDs | Rootfs absence and independently corroborated registry identity stability |
| October 9 | Address check: exact old rootfs address returned 404; registry address returned 200 | Address-level corroboration, not a returned Healthy status |
The resource type remains k8s-container. Both baseline findings contain additionalData.topLevelKind: Deployment and the selected Deployment name. The assessed-resource identity incorporates namespace, controller kind/name, and container; a separate additionalData.podName carries supporting pod detail.
The keys observed in these captures are:
Observed recommendation outcomesSwipe across for columns
| Check | Assessment name | Observed outcome |
|---|---|---|
| Immutable (read-only) root filesystem should be enforced for containers | b4262879-2a96-468f-bacd-6ed4d567bbdd | Unhealthy baseline, then absent; no explicit Healthy/resolved record |
| Container images should be deployed from trusted registries only | 12ceffa0-bd66-4ae9-abbf-58f88b0a7bc0 | Unhealthy before and after; exact assessment/resource identity preserved with fresh supporting pod |
The registry finding’s first-evaluation and status-change fields kept their original October 8 values even when the pod detail changed. They are not last-scan clocks. The first KSPM service timestamp was roughly six and a half hours after workload creation; the first successful local observation was later that day. Neither establishes a fixed delivery interval.
The rootfs transition window falls on October 9, between the earlier capture that still contained the finding and the later capture that did not, about 24 to 27 hours after the fix. It is bounded by observations, not a precise clearance time. The new registry pod proves freshness for that recommendation; it does not independently prove when the separate rootfs evaluator ran.
Reproduce the Evidence Checks
Use an explicitly selected disposable Azure subscription, cluster and namespace. Read the bundled runbook before applying a manifest. The tested environment used AKS 1.35.8, two Azure Linux nodes, a digest-pinned pause image, no application ingress, and a restricted workload namespace. This is a configuration and identity test, not an application compatibility or filesystem-write test.
- Verify resource ownership, the selected account, immutable cluster UID, Deployment UID, and the controlling owner chain. A reused ARM resource ID alone cannot distinguish a recreated cluster.
- Record Defender plans, extensions, the Policy add-on state, and Trusted Access bindings. Configuration and successful permissions establish access, not completed discovery.
- Capture an actual baseline service finding before changing the workload. Retain complete ARM and ARG exports and their hashes. Identify candidates independently by recommendation identity, title, resource type and workload fields.
- Change one chosen variable. For this rollout, the guarded JSON Patch tested the expected Deployment UID, resource version, generation, replica count, container name and old field value before changing
readOnlyRootFilesystem. - Capture the resulting Kubernetes state: desired/ready replicas, Deployment generation and UID, active ReplicaSet, each pod UID, image, and security context.
- Preserve that state while collecting subsequent service observations. Compare exact finding/resource IDs. Attribute freshness using a post-intervention pod or another independently verified marker, not collection time alone.
- Record absence separately from an explicit Healthy/resolved result. Check pagination, scope, transport errors, and direct addresses before interpreting a missing row.
- Retain only the authorized lab resources until review is complete, then perform ownership-checked cleanup and verify deletion.
The collectors write raw responses into the ignored private/ directory. A failed ARM page now preserves a distinctly named incomplete export, safe error category and attempt count, and exits unsuccessfully. An incomplete export is never an empty result.
Example read-only collection. The variable names match the runbook’s setup section; substitute your own verified scope:
$sub = '<subscription-guid>'
$clusterId = "/subscriptions/$sub/resourceGroups/<lab-rg>/providers/Microsoft.ContainerService/managedClusters/<lab-cluster>"
$namespace = 'nls-kspm-controller-lab'
$deployment = 'kspm-scope-proof-v2'
python scripts/collect_assessments.py `
--subscription $sub --cluster-id $clusterId `
--namespace $namespace --workload $deployment --phase remediated
if ($LASTEXITCODE -ne 0) { throw 'ARM export incomplete or failed' }
python scripts/collect_arg.py `
--subscription $sub --cluster-id $clusterId `
--namespace $namespace --workload $deployment --phase remediated
if ($LASTEXITCODE -ne 0) { throw 'Resource Graph export incomplete or failed' }
Both collectors reject a placeholder subscription or cluster ID before they call Azure. The root-filesystem recommendation key is a built-in candidate; add --assessment-key 12ceffa0-bd66-4ae9-abbf-58f88b0a7bc0 to match the trusted-registry recommendation by key as well. Supply your own verified scope using the runbookโs setup procedure; never substitute another personโs raw identifiers. The label remediated records the experiment phase and does not assert Defender clearance.
Earlier Phases and Their Limits
Run 1, on October 7, established Kubernetes behavior only. The workload lived about 65 minutes and supplied no matching Defender result before deletion. Its four measured comparisons remain preserved:
Earlier Kubernetes-only comparisonsSwipe across for columns
| Run-1 intervention | Kubernetes result |
|---|---|
| Replace one of two pods | One new pod UID; same Deployment/ReplicaSet; writable-root setting retained |
| Scale two โ four | Two additional pods; same controller/template |
| Set template root filesystem read-only | Four new pods and a new active ReplicaSet; all read-only |
| Replace one hardened pod | One new UID; same active ReplicaSet; read-only setting retained |
Those observations do not prove Defender behavior. Run 2 did not perform the scale test or replacement-after-hardening test. Its isolated single-pod follow-up ended inconclusively before the template patch. The later registry result establishes identity continuity across the observed sequence and rollout; it cannot retrospectively complete the isolated pod test.
Prerequisites, Costs and Cleanup
The bundled runbook requires Python 3.12+, Azure CLI, Bicep, native kubectl, kubelogin, and permissions to create an AKS cluster and its scoped role assignments. Its PowerShell examples require PowerShell 7.5+.
Defender CSPM with agentless Kubernetes discovery and Defender for Containers were already enabled. The operator manually created the documented Trusted Access binding. Sensor components were present; โagentless findingโ does not mean there were no Defender agents in this lab.
The Azure Policy add-on was not installed by the operator. Defender’s own missing-add-on recommendation was Unhealthy alongside the two KSPM-Agentless findings. That documents how these two checks ran in this environment. For each recommendation’s prerequisites, check Microsoft’s architecture, support matrix, and troubleshooting guidance.
Cleanup complete, October 12: after I authorized teardown, successful Azure reads confirmed both the experiment resource group and its AKS-managed node group absent. The managed group remained after the experiment group was deleted, so it required a separate ownership review and guarded deletion. The scope was limited to those lab resources; shared Defender settings and other labs were not changed.
The saved October 8 and 9 evidence was preserved before teardown and remains the basis for the results above. The cluster was already deallocated, so no final live Kubernetes capture was available and it was not restarted. This October 12 resource cleanup is separate from the October 9 finding disappearance.
The approved USD 50 allowance is cumulative across both runs and is not an Azure-enforced cap. A final invoice was not measured, and resource deletion does not establish the total billed cost. Set your own allowance and cleanup plan before reproducing the lab. Microsoft’s discovery refresh guidance helps plan observation windows for the surfaces it covers. Keep that planning separate from your budget and verify fresh results; the guidance supplies neither a billing limit nor a guaranteed refresh or on-demand scan mechanism.
The current source checks comprise 91 collector/comparison/safety tests plus 8 manifest tests. They verify the tools and fixture, not the live product claim; that claim rests on the captured service and Kubernetes evidence above.
