Bỏ qua đến nội dung
DevOps Lab

Flux Operator + Kyverno

Hướng dẫn platform — overview, bảng so sánh, Gitless pipeline diagram.

Flux OperatorGitless GitOpsKyverno PoliciesEphemeral EnvsD2 Reference Arch

Flux Operator + Kyverno
Production Platform Guide

Deep dive: Gitless GitOps với OCI Artifacts, Helm Chart production-ready, Ephemeral Environments từ GitHub PR, và Kyverno policies tích hợp native với Flux Operator APIs — dùng Kind để test local.

D2 Reference ArchitectureEKS + GKE multi-cloudFluxInstance / ResourceSet / ResourceSetInputProviderKind local dev
00 — OVERVIEW

D2 Architecture: Tại sao Flux Operator thay đổi mọi thứ

Flux D2 Reference Architecture là sự tiến hóa lớn nhất của GitOps. Thay vì Flux trỏ vào Git repository để sync, toàn bộ configuration được đóng gói thành signed OCI Artifacts và push lên container registry. Flux Operator sau đó pull artifact này xuống cluster — Git không còn trong delivery path.

🏗️ FluxInstance
Lifecycle management cho tất cả Flux controllers. Auto-upgrade, multi-tenancy lockdown, OCI sync config.
📦 ResourceSet
Template engine cho tenant/app provisioning. Thay thế hoàn toàn Kustomize overlays và Kyverno generate.
🔌 ResourceSetInputProvider
Dynamic inputs từ GitHub PR, GitLab MR, ConfigMap, Secret. Tạo ephemeral envs tự động.
📊 FluxReport
Built-in observability. Status của tất cả components, sync health, drift detection — không cần tool ngoài.
🔑
Core insight — Với Flux Operator, bạn không cần Kyverno để generate resources hay clone secrets hay inject labels — những việc đó Flux Operator làm native. Kyverno chỉ làm những gì nó giỏi nhất: validate, enforce security policy, và block bad actors.

So sánh: Trước vs Sau Flux Operator

ConcernTrước (Flux CLI + Kustomize overlays)Sau (Flux Operator D2)
Install Fluxflux bootstrap CLI, commit vào GitFluxInstance CR — declarative, self-managing
Multi-env configKustomize base/overlays per clusterFluxInstance per cluster, sync từ OCI tag
Onboard tenantCopy overlay folder, Kyverno generate SA/RB/NPResourceSet: thêm 5 dòng input
Secret syncKyverno clone policycopyFrom annotation native
Label injectionKyverno mutate policyspec.commonMetadata.labels trong FluxInstance
Multi-tenancy lockdownKyverno validate policymultitenant: true — built-in
Delivery sourceGitRepository — Flux cần network tới GitOCIRepository — pull từ registry, Gitless
Ephemeral envsManual script hoặc không cóResourceSetInputProvider + GitHub PR label
Flux upgradeflux bootstrap lại, commit lên GitSửa version trong FluxInstance, Flux tự upgrade
Observabilitykubectl get kustomization -AFluxReport CR — aggregated cluster health
01 — D2 REPOSITORIES

D2 Reference Architecture — 3 Repos, 3 Roles

Đây là điều quan trọng nhất cần hiểu đúng: D2 không phải một Helm chart cho tất cả. D2 tách ra 3 Git repositories độc lập, mỗi repo có ownership, permission, và delivery pipeline riêng.

D2 REFERENCE ARCHITECTURE — FULL OVERVIEW
D2 reference architecture — staging vs production clustersSOURCE REPOSITORIES — 3 repos · 3 ownership boundariesd2-fleetPlatform Team · Cluster Admin· FluxInstance per cluster· ResourceSet templates· InputProvider configsd2-infraPlatform Team · Cluster Admin· Kyverno + ClusterPolicies· cert-manager · ingress-nginx· kube-prometheus-stackd2-appsDev Teams · Namespace Admin· HelmRelease per service· values per environment· generic-web/jvm/worker chartgit push / git tag → CI builds OCI artifact + signs with Cosignflux push artifactflux push artifacthelm package + pushOCI Registry (GHCR / ECR / GAR)all artifacts signed with Cosign keyless · Flux verifies before apply:latest tag → staging:latest-stable tag → productionSTAGING CLUSTERFluxInstancesync: OCIRepository · ref: latestKyvernoAudit modeResourceSettenant namespacesEphemeral environmentsInputProvider polls GitHub PRsdeploy/flux-preview → auto HelmReleasePR+ labelMicroservicesgeneric-*-chart · pre-release semver · 1 replicaFluxReport · PolicyReportPRODUCTION CLUSTERFluxInstancesync: OCIRepository · ref: latest-stableKyvernoEnforce modeResourceSettenant namespacesTime-based deliveryInputProvider · cron schedule windowGitRepository pinned to approved SHAMicroservicesgeneric-*-chart · stable semver · HPA · PDBFluxReport · PolicyReport · Slack alertsgit tagv1.x.xOBSERVABILITY LOOPFluxReportcomponent healthPolicyReportKyverno violationsFlux NotificationAlert → Slack / PagerDutyDrift detectauto-remediateLEGENDd2-fleetd2-infra / Kyvernod2-appsGeneric Helm ChartsTime-basedSolid arrow = OCI pull & apply · Dashed arrow = admission validate / observability loop · Green arrow = promote via git tag

Pipeline Gitless (OCI) rút gọn — cùng luồng với diagram trên: build artifact, ký Cosign, Flux verify trước apply.

Gitless GitOps — OCI deliveryCI / Buildpush artifactOCI Registrysigned bundleno Git in pathFlux OperatorreconcileClusterworkloads
01b — D2-INFRA

d2-infra — Kyverno as Cluster Add-on

Kyverno sống trong d2-infra, không phải d2-apps hay d2-fleet. Platform team sở hữu repo này, reconcile với cluster admin permission. Kyverno policies là cluster-scoped — chỉ platform team mới được sửa.

Kustomization cho policies phụ thuộc HelmRelease Kyverno (dependsOn) để tránh apply policy trước khi admission webhook sẵn sàng.

d2-infra/components/kyverno/helmrelease.yaml
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: kyverno
  namespace: kyverno
spec:
  interval: 1h
  chart:
    spec:
      chart: kyverno
      version: "3.3.x"
      sourceRef:
        kind: HelmRepository
        name: kyverno
        namespace: flux-system
  install:
    crds: CreateReplace
  valuesFrom:
    - kind: ConfigMap
      name: kyverno-values
    - kind: ConfigMap
      name: kyverno-values-env
      optional: true
  values:
    admissionController:
      replicas: 3
d2-infra/components/kyverno/kustomization.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: kyverno-policies
  namespace: flux-system
spec:
  interval: 10m
  dependsOn:
    - name: kyverno
  sourceRef:
    kind: OCIRepository
    name: d2-infra
  path: "./components/kyverno/policies"
  prune: true
  wait: true
01c — D2-APPS

d2-apps — ResourceSet delivers apps, NOT manual HelmRelease

Pattern thực tế: d2-fleet dùng ResourceSet để deliver toàn bộ d2-apps. Dev không tạo HelmRelease trực tiếp trên cluster — chỉ commit vào d2-apps; CI đóng gói OCI artifact per tenant; ResourceSet trong d2-fleet tạo OCIRepository + Kustomization per tenant.

Mỗi tenant có OCI artifact riêng (oci://ghcr.io/org/d2-apps/<tenant>). Kustomization trỏ ./staging hoặc ./production trong artifact.

d2-fleet/tenants/apps.yaml (rút gọn)
apiVersion: fluxcd.controlplane.io/v1
kind: ResourceSet
metadata:
  name: apps
  namespace: flux-system
spec:
  dependsOn:
    - apiVersion: kustomize.toolkit.fluxcd.io/v1
      kind: Kustomization
      name: infra-configs
      namespace: monitoring
      ready: true
  inputs:
    - tenant: "frontend"
      tag: "${ARTIFACT_TAG}"
      environment: "${ENVIRONMENT}"
  resources:
    - apiVersion: v1
      kind: Namespace
      metadata:
        name: "<< inputs.tenant >>"
02 — KIND

Kind Cluster — Local Dev & CI Bootstrap

Dùng Kind để test stack locally trước khi push EKS/GKE. Config dưới đây mô phỏng production với port mapping cho ingress.

ci/kind-config.yaml
apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
name: platform-local
nodes:
  - role: control-plane
    kubeadmConfigPatches:
      - |
        kind: InitConfiguration
        nodeRegistration:
          kubeletExtraArgs:
            node-labels: "ingress-ready=true"
    extraPortMappings:
      - containerPort: 80
        hostPort: 8080
      - containerPort: 443
        hostPort: 8443

  - role: worker
  - role: worker

featureGates:
  ValidatingAdmissionPolicy: true

networking:
  podSubnet: "10.244.0.0/16"
  serviceSubnet: "10.96.0.0/16"
ci/bootstrap.sh
#!/usr/bin/env bash
set -euo pipefail

echo "==> Creating Kind cluster..."
kind create cluster --config=ci/kind-config.yaml --wait 60s

echo "==> Installing Flux Operator (via Helm)..."
helm repo add controlplane https://controlplaneio-fluxcd.github.io/charts
helm repo update

helm upgrade --install flux-operator controlplane/flux-operator \
  --namespace flux-system \
  --create-namespace \
  --wait

echo "==> Deploying platform chart (staging mode)..."
helm upgrade --install platform ./platform-chart \
  --namespace flux-system \
  --create-namespace \
  -f ./platform-chart/values.yaml \
  -f ./platform-chart/values-staging.yaml \
  --set global.environment=local \
  --set fluxOperator.instance.sync.kind=GitRepository \
  --set fluxOperator.instance.sync.url=https://github.com/your-org/fleet.git \
  --set policies.validationFailureAction=Audit \
  --wait --timeout=10m

echo "==> Waiting for Kyverno to be ready..."
kubectl wait --for=condition=Available \
  deployment/kyverno-admission-controller \
  -n kyverno --timeout=300s

echo "==> Checking FluxReport..."
kubectl get fluxreport -n flux-system -o yaml

echo "✅ Platform stack is ready!"
03 — FLUXINSTANCE

FluxInstance — Lifecycle Management

FluxInstance CR thay thế flux bootstrap. Flux Operator deploy controllers, cấu hình sync source; Flux tự quản lý chính nó.

Staging dùng tag latest; production dùng latest-stable — không cần Kustomize overlays cho hai môi trường.

templates/flux-operator/fluxinstance.yaml (rút gọn)
apiVersion: fluxcd.controlplane.io/v1
kind: FluxInstance
metadata:
  name: flux
  namespace: flux-system
spec:
  distribution:
    version: "2.x.x"
    artifact: "oci://ghcr.io/controlplaneio-fluxcd/flux-operator-manifests"
  components:
    - source-controller
    - kustomize-controller
    - helm-controller
  cluster:
    type: kubernetes
    multitenant: true
  sync:
    kind: OCIRepository
    url: oci://ghcr.io/org/d2-fleet
    ref: latest-stable
    path: "./clusters/prod"
  commonMetadata:
    labels:
      managed-by: "flux-operator"
04 — GITLESS

OCI Artifacts — Delivery Pipeline

Git vẫn là source of truth cho developer, nhưng delivery path không đi qua Git: CI build và push OCI artifacts — Flux pull từ registry. Git outage không chặn deployment.

Sơ đồ rút gọn ở trên (Gitless pipeline) bổ sung cho full D2 diagram — cùng semantics: push artifact → verify → apply.

05 — RESOURCESET

ResourceSet — Tenant provisioning

ResourceSet là “Helm cho Flux resources”: template + inputs. Thay Kustomize overlay copy-paste hoặc Kyverno generate hàng loạt object — thêm tenant trong values, Helm upgrade, ResourceSet tạo namespace, SA, NetworkPolicy, OCIRepository, Kustomization.

copyFrom annotation sao chép Secret/ConfigMap từ flux-system sang tenant — native Flux Operator, không cần Kyverno clone.

06 — EPHEMERAL

Ephemeral environments từ PR

Gắn label deploy/flux-preview trên GitHub PR: ResourceSetInputProvider tạo preview namespace, HelmRelease với image tag = commit SHA. PR merge/close → dọn resource.

Kyverno cho preview thường Audit thay vì Enforce để image mới nhất chưa ký đủ vẫn deploy được trong khi PolicyReport ghi nhận violation.

07 — KYVERNO

Kyverno layers — validate / enforce

Với Flux Operator multitenant: cross-namespace reference bị chặn native. Kyverno tập trung validate image registry, require labels, và PolicyException cho preview.

policy ví dụ — require tenant label
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-tenant-label
spec:
  validationFailureAction: Enforce
  rules:
    - name: check-labels
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        message: "tenant label required"
        pattern:
          metadata:
            labels:
              app.kubernetes.io/part-of: "?*"
templates/policies/deployment-audit.yaml
# Kustomization phải có SHA annotation — audit trail sau time-gate
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-deployment-sha
  annotations:
    policies.kyverno.io/category: Compliance
spec:
  validationFailureAction: Audit
  background: true
  rules:
    - name: require-sha-annotation
      match:
        any:
          - resources:
              kinds: ["Kustomization"]
              namespaces: ["team-*"]
      validate:
        message: "Missing deployment SHA — was it deployed via time-gate?"
        pattern:
          metadata:
            annotations:
              deployment.company.com/sha: "?*"
              deployment.company.com/deployed-at: "?*"
08 — FLUXREPORT

FluxReport — Aggregated health

FluxReport CR gom sync status, components, drift — quan sát tập trung. Kết hợp PolicyReport (Kyverno) và notification để alert Slack/PagerDuty.

09 — CI/CD

Full Pipeline — From PR to Production

Luồng đầy đủ: PR mở label deploy/flux-preview → preview env; review PolicyReport Audit; merge → CI đẩy OCI :latest + Cosign; staging FluxInstance sync; git tag → :latest-stable → production Kyverno Enforce + FluxReport Slack.

DEV
PR opened → label deploy/flux-preview
CI build image theo commit SHA → GHCR. ResourceSetInputProvider tạo preview namespace + HelmRelease. URL dạng pr-42.preview.example.com.
REVIEW
Code review + preview validation
Kyverno PolicyReport (Audit) trên preview; mỗi push nâng HelmRelease.
MERGE
PR merge → OCI :latest
flux push artifact …:latest; Cosign keyless sign; ResourceSet prune preview.
STAGING
Staging FluxInstance sync :latest
OCIRepository detect tag; verify Cosign; Kyverno Audit; FluxReport xác nhận healthy.
RELEASE
git tag v* → :latest-stable
Production FluxInstance sync :latest-stable; Kyverno Enforce; thông báo Slack từ FluxReport/Alert.
.github/workflows/flux-artifact-cd.yml (pattern)
name: flux-artifact-cd
on:
  push:
    branches: [main]
  pull_request:
    types: [opened, synchronize, labeled]

permissions:
  contents: read
  id-token: write
  packages: write

jobs:
  build-push-sign:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Login to GHCR
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Push Flux OCI artifact (path = fleet manifests)
        run: |
          flux push artifact oci://ghcr.io/${{ github.repository_owner }}/fleet:${{ github.sha }} \
            --path=./clusters/staging \
            --source=${{ github.repository }} \
            --revision=${{ github.sha }}

      - name: Cosign sign (keyless)
        uses: sigstore/cosign-installer@v3
      - run: |
          DIGEST=$(crane digest ghcr.io/${{ github.repository_owner }}/fleet:${{ github.sha }})
          cosign sign --yes ghcr.io/${{ github.repository_owner }}/fleet@$DIGEST
One-shot platform install (wiki callout)
helm upgrade --install platform ./platform-chart \
  -f values-production.yaml \
  --namespace flux-system --create-namespace
10 — TIME

ResourceSet + Cron — Controlled deployment windows

Trong môi trường highly regulated (fintech, banking, healthcare), deployment phải tuân CAB windows, tránh peak business, và có audit trail đầy đủ.

Flux Operator: ResourceSetInputProvider chỉ fetch commit SHA mới trong cửa sổ được phép. Ngoài window, GitRepository vẫn pin SHA cũ — cluster không đổi dù code đã merge (khác suspend/resume thủ công).

Core mechanism: ResourceSet sinh GitRepository + Kustomization động; InputProvider reconcile theo schedule — ref.commit = inputs.sha. Ngoài window InputProvider không lấy SHA mới.

Nhiều mục trong schedule[] là OR: primary window + emergency window trong cùng một InputProvider.

Hành vi ngoài window: InputProvider không fetch SHA mới; GitRepository giữ pin — cluster bất biến.

IndustryWindowCronDuration
Fintech / BankingNgoài giờ giao dịch0 22 * * 1-54h (22:00–02:00)
E-commerceLow-traffic hours0 3 * * *3h (03:00–06:00)
HealthcareWeekly maintenance0 2 * * 06h (Sun 02:00)
SRE / PlatformBusiness hours only0 9 * * 1-58h (09:00–17:00)
Multi-regionFollow-the-sun0 8 * * 1-58h per TZ
templates/time-delivery/branch-input-provider.yaml
apiVersion: fluxcd.controlplane.io/v1
kind: ResourceSetInputProvider
metadata:
  name: payments-app-main
  namespace: team-payments
  labels:
    app.kubernetes.io/name: payments-app
  annotations:
    fluxcd.controlplane.io/reconcileEvery: "10m"
    fluxcd.controlplane.io/reconcileTimeout: "1m"
spec:
  schedule:
    - cron: "0 9 * * 1-5"
      timeZone: "Asia/Ho_Chi_Minh"
      window: 8h
    - cron: "0 10 * * 6"
      timeZone: "Asia/Ho_Chi_Minh"
      window: 4h
  type: GitHubBranch
  url: https://github.com/your-org/payments-app
  secretRef:
    name: gh-app-auth
  filter:
    includeBranch: "^main$"
  defaultValues:
    env: production
templates/time-delivery/tag-input-provider.yaml
apiVersion: fluxcd.controlplane.io/v1
kind: ResourceSetInputProvider
metadata:
  name: payments-app-release
  namespace: team-payments
  labels:
    app.kubernetes.io/name: payments-app
spec:
  schedule:
    - cron: "0 22 * * 2,4"
      timeZone: "Asia/Ho_Chi_Minh"
      window: 4h
  type: GitHubTag
  url: https://github.com/your-org/payments-app
  secretRef:
    name: gh-app-auth
  filter:
    semver: ">=1.0.0"
    limit: 1
templates/time-delivery/oci-input-provider.yaml
apiVersion: fluxcd.controlplane.io/v1
kind: ResourceSetInputProvider
metadata:
  name: payments-chart-release
  namespace: team-payments
  labels:
    delivery-mode: oci-time-based
spec:
  schedule:
    - cron: "0 22 * * 2,4"
      timeZone: "Asia/Ho_Chi_Minh"
      window: 4h
  type: OCITag
  url: oci://ghcr.io/your-org/payments-chart
  secretRef:
    name: oci-auth
  filter:
    semver: ">=1.0.0"
    limit: 1
templates/time-delivery/resourceset-pinned.yaml
apiVersion: fluxcd.controlplane.io/v1
kind: ResourceSet
metadata:
  name: payments-app
  namespace: team-payments
spec:
  inputsFrom:
    - kind: ResourceSetInputProvider
      selector:
        matchLabels:
          app.kubernetes.io/name: payments-app
  resources:
    - apiVersion: source.toolkit.fluxcd.io/v1
      kind: GitRepository
      metadata:
        name: payments-app
        namespace: "<< inputs.provider.namespace >>"
      spec:
        interval: 12h
        url: https://github.com/your-org/payments-app
        ref:
          commit: "<< inputs.sha >>"
        secretRef:
          name: gh-app-auth
        sparseCheckout:
          - deploy
    - apiVersion: kustomize.toolkit.fluxcd.io/v1
      kind: Kustomization
      metadata:
        name: payments-app
        namespace: "<< inputs.provider.namespace >>"
      spec:
        interval: 30m
        retryInterval: 5m
        serviceAccountName: flux-payments
        prune: true
        wait: true
        timeout: 5m
        sourceRef:
          kind: GitRepository
          name: payments-app
        path: deploy/<< inputs.env >>
        commonMetadata:
          annotations:
            deployment.company.com/sha: "<< inputs.sha >>"
            deployment.company.com/branch: "<< inputs.branch >>"
            deployment.company.com/deployed-at: "<< inputs.provider.lastUpdated >>"
Flux Operator + Kyverno · React port · DevOps Lab