Skip to main content
On this page

Review update: The example collection commands now set their own placeholder variables, named as in the runbook. The earlier example read a private state file that was never distributed and that no script creates.

What This Lab Demonstrated

A two-replica Deployment changed its pod template to set readOnlyRootFilesystem: true. Both pods were replaced. A subsequent Defender registry recommendation kept its exact assessment and resource identities while naming a verified post-rollout pod. The root-filesystem recommendation disappeared from complete ARM and Resource Graph exports; a direct read of its old address returned 404.

Registry identity stability is observed for this rollout. Root-filesystem absence is observed; an explicit Healthy or resolved assessment is not. The companion article explains the operational implications of Microsoftโ€™s October 5 controller-scope announcement.

Downloads and Provenance

The source ZIP matches published commit 7678f84 exactly. The archive manifest records the commit and file hashes. Evidence files retain dated review snapshots, including historical pre-publication flags; the operational and cleanup records carry their own timestamps.

Raw tenant/resource identifiers, full service responses, pod UIDs, and kubeconfig stay private. Shareable excerpts use documented pod aliases. Microsoft recommendation identifiers are retained because they identify the checks, not the tenant. SHA-256 sidecars detect changes to captured bytes; they do not independently authenticate a fileโ€™s provider origin.

Run 2: Product Evidence

Dates use UTC. Rows follow the order of observations; the downloadable evidence retains exact capture and service timestamps.

Run 2 observation timelineSwipe across for columns

DateObservationWhat it establishes
October 7Two-pod writable-root baseline capturedInitial Kubernetes state
October 8Baseline: matching Unhealthy root-filesystem and registry recommendations in ARM and Resource GraphActual service findings with KSPM-Agentless and Deployment attribution
October 8Pod replacement: deletion of one baseline pod acceptedKubernetes replacement; isolated Defender follow-up remained inconclusive
October 8Template fix: only root-filesystem field patchedControlled remediation intervention
October 8Rollout: same Deployment, generation 2, new ReplicaSet and two new ready podsCompleted Kubernetes rollout with read-only roots and unchanged image
October 9Earlier capture: ARM still returned the root-filesystem findingLast captured presence
October 9Later captures: ARG / ARM no longer returned rootfs; registry referenced a rollout pod with unchanged assessment/resource IDsRootfs absence and independently corroborated registry identity stability
October 9Address check: exact old rootfs address returned 404; registry address returned 200Address-level corroboration, not a returned Healthy status

The resource type remains k8s-container. Both baseline findings contain additionalData.topLevelKind: Deployment and the selected Deployment name. The assessed-resource identity incorporates namespace, controller kind/name, and container; a separate additionalData.podName carries supporting pod detail.

The keys observed in these captures are:

Observed recommendation outcomesSwipe across for columns

CheckAssessment nameObserved outcome
Immutable (read-only) root filesystem should be enforced for containersb4262879-2a96-468f-bacd-6ed4d567bbddUnhealthy baseline, then absent; no explicit Healthy/resolved record
Container images should be deployed from trusted registries only12ceffa0-bd66-4ae9-abbf-58f88b0a7bc0Unhealthy before and after; exact assessment/resource identity preserved with fresh supporting pod

The registry finding’s first-evaluation and status-change fields kept their original October 8 values even when the pod detail changed. They are not last-scan clocks. The first KSPM service timestamp was roughly six and a half hours after workload creation; the first successful local observation was later that day. Neither establishes a fixed delivery interval.

The rootfs transition window falls on October 9, between the earlier capture that still contained the finding and the later capture that did not, about 24 to 27 hours after the fix. It is bounded by observations, not a precise clearance time. The new registry pod proves freshness for that recommendation; it does not independently prove when the separate rootfs evaluator ran.

Reproduce the Evidence Checks

Use an explicitly selected disposable Azure subscription, cluster and namespace. Read the bundled runbook before applying a manifest. The tested environment used AKS 1.35.8, two Azure Linux nodes, a digest-pinned pause image, no application ingress, and a restricted workload namespace. This is a configuration and identity test, not an application compatibility or filesystem-write test.

  1. Verify resource ownership, the selected account, immutable cluster UID, Deployment UID, and the controlling owner chain. A reused ARM resource ID alone cannot distinguish a recreated cluster.
  2. Record Defender plans, extensions, the Policy add-on state, and Trusted Access bindings. Configuration and successful permissions establish access, not completed discovery.
  3. Capture an actual baseline service finding before changing the workload. Retain complete ARM and ARG exports and their hashes. Identify candidates independently by recommendation identity, title, resource type and workload fields.
  4. Change one chosen variable. For this rollout, the guarded JSON Patch tested the expected Deployment UID, resource version, generation, replica count, container name and old field value before changing readOnlyRootFilesystem.
  5. Capture the resulting Kubernetes state: desired/ready replicas, Deployment generation and UID, active ReplicaSet, each pod UID, image, and security context.
  6. Preserve that state while collecting subsequent service observations. Compare exact finding/resource IDs. Attribute freshness using a post-intervention pod or another independently verified marker, not collection time alone.
  7. Record absence separately from an explicit Healthy/resolved result. Check pagination, scope, transport errors, and direct addresses before interpreting a missing row.
  8. Retain only the authorized lab resources until review is complete, then perform ownership-checked cleanup and verify deletion.

The collectors write raw responses into the ignored private/ directory. A failed ARM page now preserves a distinctly named incomplete export, safe error category and attempt count, and exits unsuccessfully. An incomplete export is never an empty result.

Example read-only collection. The variable names match the runbook’s setup section; substitute your own verified scope:

$sub = '<subscription-guid>'
$clusterId = "/subscriptions/$sub/resourceGroups/<lab-rg>/providers/Microsoft.ContainerService/managedClusters/<lab-cluster>"
$namespace = 'nls-kspm-controller-lab'
$deployment = 'kspm-scope-proof-v2'

python scripts/collect_assessments.py `
  --subscription $sub --cluster-id $clusterId `
  --namespace $namespace --workload $deployment --phase remediated
if ($LASTEXITCODE -ne 0) { throw 'ARM export incomplete or failed' }

python scripts/collect_arg.py `
  --subscription $sub --cluster-id $clusterId `
  --namespace $namespace --workload $deployment --phase remediated
if ($LASTEXITCODE -ne 0) { throw 'Resource Graph export incomplete or failed' }

Both collectors reject a placeholder subscription or cluster ID before they call Azure. The root-filesystem recommendation key is a built-in candidate; add --assessment-key 12ceffa0-bd66-4ae9-abbf-58f88b0a7bc0 to match the trusted-registry recommendation by key as well. Supply your own verified scope using the runbookโ€™s setup procedure; never substitute another personโ€™s raw identifiers. The label remediated records the experiment phase and does not assert Defender clearance.

Earlier Phases and Their Limits

Run 1, on October 7, established Kubernetes behavior only. The workload lived about 65 minutes and supplied no matching Defender result before deletion. Its four measured comparisons remain preserved:

Earlier Kubernetes-only comparisonsSwipe across for columns

Run-1 interventionKubernetes result
Replace one of two podsOne new pod UID; same Deployment/ReplicaSet; writable-root setting retained
Scale two โ†’ fourTwo additional pods; same controller/template
Set template root filesystem read-onlyFour new pods and a new active ReplicaSet; all read-only
Replace one hardened podOne new UID; same active ReplicaSet; read-only setting retained

Those observations do not prove Defender behavior. Run 2 did not perform the scale test or replacement-after-hardening test. Its isolated single-pod follow-up ended inconclusively before the template patch. The later registry result establishes identity continuity across the observed sequence and rollout; it cannot retrospectively complete the isolated pod test.

Prerequisites, Costs and Cleanup

The bundled runbook requires Python 3.12+, Azure CLI, Bicep, native kubectl, kubelogin, and permissions to create an AKS cluster and its scoped role assignments. Its PowerShell examples require PowerShell 7.5+.

Defender CSPM with agentless Kubernetes discovery and Defender for Containers were already enabled. The operator manually created the documented Trusted Access binding. Sensor components were present; โ€œagentless findingโ€ does not mean there were no Defender agents in this lab.

The Azure Policy add-on was not installed by the operator. Defender’s own missing-add-on recommendation was Unhealthy alongside the two KSPM-Agentless findings. That documents how these two checks ran in this environment. For each recommendation’s prerequisites, check Microsoft’s architecture, support matrix, and troubleshooting guidance.

Cleanup complete, October 12: after I authorized teardown, successful Azure reads confirmed both the experiment resource group and its AKS-managed node group absent. The managed group remained after the experiment group was deleted, so it required a separate ownership review and guarded deletion. The scope was limited to those lab resources; shared Defender settings and other labs were not changed.

The saved October 8 and 9 evidence was preserved before teardown and remains the basis for the results above. The cluster was already deallocated, so no final live Kubernetes capture was available and it was not restarted. This October 12 resource cleanup is separate from the October 9 finding disappearance.

The approved USD 50 allowance is cumulative across both runs and is not an Azure-enforced cap. A final invoice was not measured, and resource deletion does not establish the total billed cost. Set your own allowance and cleanup plan before reproducing the lab. Microsoft’s discovery refresh guidance helps plan observation windows for the surfaces it covers. Keep that planning separate from your budget and verify fresh results; the guidance supplies neither a billing limit nor a guaranteed refresh or on-demand scan mechanism.

The current source checks comprise 91 collector/comparison/safety tests plus 8 manifest tests. They verify the tools and fixture, not the live product claim; that claim rests on the captured service and Kubernetes evidence above.