Governance-immune storage — Private Endpoint

This document treats "how to keep an automation job running even when a cloud governance policy auto-locks your storage" as a single Azure architecture case study, explaining which services were combined and why, from the perspective of an engineer who wants to rebuild it. (As of 2026-06-16 · Region: East US 2 · Target: the two digest agents behind moontaeyang.com)

🎯 What this demo shows: Many enterprise/regulated environments run a central governance automation that periodically enforces security baselines like "block public storage access (publicNetworkAccess=Disabled)" silently. Even if you turn it back on, it flips off again within minutes to a day. In such an environment, an automation job that relied on public access suddenly breaks one day. The fix is not to fight it but to route around it via a private path (Private Endpoint) — a design that coexists with the policy while keeping the job unbroken.

---

At a glance

• Problem: Central governance (MCAPSGov-AutomationApp) periodically reverts storage publicNetworkAccess to Disabled → the Container Apps Job that read/wrote blobs over the public path fails with AuthorizationFailure.
• Key insight: pna=Disabled blocks only the public internet; Private Endpoint (private IP) traffic is always allowed. So if you put the job inside a VNet and give the storage a private path (PE), the job keeps working even when the policy locks public access.
• Zero impact on public users: Public serving is handled by Static Web Apps (moontaeyang.com), not the storage. The storage is just the warehouse the job reads/writes — locking the warehouse door doesn't affect visitors.
• Hidden trap: The two jobs read each other's warehouse to reassemble the full portal → locking only one breaks the other. So we peered the two VNets + cross-registered DNS so each can reach the other's warehouse privately too.

Keywords: Private Endpoint · VNet-injected Container Apps · Private DNS · VNet Peering · Managed Identity (keyless) · coexisting with policy

---

Business view — why it matters

• Operational stability: "An automation that worked yesterday broke today with nobody touching it" is the most expensive class of outage. Because the cause is invisible policy enforcement, debugging burns days. The PE design structurally eliminates this class of failure.
• Security and productivity together: It does not fight the security team's "block public access" mandate (in fact it isolates data more strongly), while the dev team's automation keeps running. Instead of fighting the policy, it converges to the policy's desired end-state (public closed).
• Regulation / data residency: In finance, public sector, semiconductors, etc., "data accessible only within a private network" is a common requirement. This pattern is the standard building block that satisfies it (the data plane is never exposed to the internet).

---

Architecture

  [Public users]                        ※ Unaffected even when storage public access (pna) is off
      │ HTTPS
      ▼
  Azure Static Web Apps  ──  moontaeyang.com  (engit-daily · azure-daily sections)
      ▲                                   SWA owns public serving
      │ swa deploy (outbound)
      │
  ┌─────────────────────────── rg-english-daily ───────────────────────────┐
  │  VNet 10.60.0.0/16                                                       │
  │   ├ snet-aca /23 (delegation: Microsoft.App/environments)                │
  │   │    └ Container Apps Env (VNet-injected, workload profile)            │
  │   │         └ engitdaily-job ── UAMI (keyless) ─┐                         │
  │   └ snet-pe /24                                  │ blob R/W               │
  │        └ Private Endpoint(blob) 10.60.2.4 ◀──────┘                        │
  │             │ private path                                               │
  │             ▼                                                            │
  │        Storage engitdailyst…  (PE passes even when pna=Disabled)         │
  │   Private DNS: privatelink.blob.core.windows.net                        │
  │        engitdailyst… → 10.60.2.4   (+ azwhatsnewst… → 10.61.2.4 x-reg)   │
  └──────────────────────────────┬──────────────────────────────────────────┘
                                  │  VNet Peering (bidirectional, Connected)
  ┌──────────────────────────────┴──── rg-azwhatsnew ───────────────────────┐
  │  VNet 10.61.0.0/16  (snet-aca /23, snet-pe /24)                          │
  │    Container Apps Env (VNet-injected) └ azwhatsnew-job ─ UAMI ─ PE 10.61.2.4 │
  │    Storage azwhatsnewst…   ·   Private DNS (same, engit record x-registered) │
  └─────────────────────────────────────────────────────────────────────────┘

  Governance (MCAPSGov-AutomationApp): periodically forces both storages' pna to Disabled → jobs still work via PE

---

Components and design decisions

1) Make the Container Apps env "VNet-injected" — why recreation is unavoidable

• To use a PE, the client (job) must be inside a VNet so the storage name resolves to the PE's private IP.
• Trap: The existing env was Consumption-only (no VNet), and you cannot add a VNet to an existing ACA env later. A VNet can only be set at env creation time → you must create a new env, move the job, then delete the old env.
• Implementation: change bicep caeName to …-caev2-… to create a new env (workload profile + vnetConfiguration.infrastructureSubnetId). Because environmentId is an immutable job property, delete the job first (az containerapp job delete) before redeploying to avoid a conflict.
• Reuse the UAMI (user-assigned managed identity) → no need to re-grant roles (e.g., Storage Blob Data Contributor) when recreating the env. (Stays keyless.)

2) Storage Private Endpoint (blob) + Private DNS

• Create Microsoft.Network/privateEndpoints (groupIds=blob) for the storage in snet-pe.
• A privatelink.blob.core.windows.net Private DNS zone + VNet link + dnsZoneGroup → the PE auto-registers the A record.
• Result: inside the VNet, engitdailyst….blob.core.windows.net → 10.60.2.4 (private) → passes even when pna=Disabled.
• The job reads/writes via the blob container URL, so the single blob sub-resource is enough (the static-website $web also goes through the blob endpoint).

3) Cross-dependency — the most important lesson of this demo

• To prevent portal clobbering, each job reassembles the full portal, i.e., engit reads azure-daily's warehouse, and azwn reads engit's warehouse.
• So locking only one storage breaks the other job's mirroring. To lock both, each needs private access to the other's warehouse: 1. VNet Peering (bidirectional, Connected) — so packets route to the other VNet's PE private IP. 2. DNS cross-registration — since a VNet links to only one zone of a given name, put both storages' A records in each zone (engit zone += azwhatsnewst…→10.61.2.4, azwn zone += engitdailyst…→10.60.2.4).
• Alternatives rejected: ① inject both envs into one shared VNet (too much cross-RG coupling) ② fetch the peer section over HTTP from the public SWA (fragile, depends on already-deployed content). → Peering + cross DNS keeps each agent self-contained while being the most robust.

4) SWA decouples public serving — lock the warehouse, visitors unaffected

• When pna=Disabled, the storage static-web endpoint (z##.web.core.windows.net) returns 404. But visitors arrive via Static Web Apps (moontaeyang.com), so the public site is up 24/7.
• This lets "lock storage" (security) and "site must stay alive" coexist without conflict.
• The deploy script's --static-website step fails on the data plane when pna=Disabled, so guard it with || echo skip.

---

Verification (actually locked it and ran)

With both storages locked (publicNetworkAccess=Disabled + defaultAction=Deny), both jobs ran:

Check	Result
engit job / azwn job	both Succeeded
own blob write	Blob upload complete … ✅
cross-mirror (read peer warehouse)	mirrored engit-daily:4 / azure-daily:8 ✅ (via PE)
auth errors	0
live sites	moontaeyang.com /·/engit-daily/·/azure-daily/ all 200
notification	Telegram 200

→ Empirically confirmed: automation keeps running even in the steady-state where the policy has locked public access.

---

Cost / trade-offs

Item	Value
Private Endpoint	2 ≈ $15/mo + small data processing
Private DNS / VNet / Peering	effectively free (same-region peering, tiny queries)
Env recreation	one-time effort (downtime is short, between cron runs)

• vs Option A (runtime self-heal): flipping pna=Enabled at job start is cheaper and needs no infra, but it keeps "tug-of-war" with the policy and briefly reopens public access. Option B (PE) works with public closed forever → cleaner security and satisfies the regulated "no data-plane exposure" requirement.
• Governance resilience: A PE is Microsoft.Network/privateEndpoints, separate from the governance's storage/NSG rewrites, and pna=Disabled doesn't block PEs → durably safe.

---

Build-it-yourself checklist

1. Separate storage used for the data plane (job R/W) vs public serving (serve via SWA/CDN).
2. Put the job in a VNet-injected Container Apps env (existing env needs recreation; job environmentId is immutable).
3. Reuse a UAMI for keyless + minimal role re-granting.
4. Add a PE (blob) + privatelink.blob.core.windows.net Private DNS to the storage.
5. If multiple jobs read each other's storage → VNet Peering + cross-register every storage's A record into each zone.
6. Make the deploy script's data-plane calls (--static-website, etc.) tolerant of pna=Disabled.
7. Actually lock it (--public-network-access Disabled) and run the job to verify the steady-state.

---

Reference code / locations

• IaC: agents/english-it-daily/infra/main.bicep, agents/azure-whats-new-digest/infra/main.bicep (VNet · PE · Private DNS · VNet-injected env)
• Deploy: each agents/*/deploy.sh (static-web step tolerant of pna)
• Governance background: reference/mcaps-governance-체크리스트.md (§3 storage pna)
• Detailed know-how: playbook/CHANGELOG.md 2026-06-16