Flux Operator + Kyverno
Hướng dẫn platform — overview, bảng so sánh, Gitless pipeline diagram.
Flux Operator + Kyverno
Production Platform Guide
Deep dive: Gitless GitOps với OCI Artifacts, Helm Chart production-ready, Ephemeral Environments từ GitHub PR, và Kyverno policies tích hợp native với Flux Operator APIs — dùng Kind để test local.
D2 Architecture: Tại sao Flux Operator thay đổi mọi thứ
Flux D2 Reference Architecture là sự tiến hóa lớn nhất của GitOps. Thay vì Flux trỏ vào Git repository để sync, toàn bộ configuration được đóng gói thành signed OCI Artifacts và push lên container registry. Flux Operator sau đó pull artifact này xuống cluster — Git không còn trong delivery path.
So sánh: Trước vs Sau Flux Operator
| Concern | Trước (Flux CLI + Kustomize overlays) | Sau (Flux Operator D2) |
|---|---|---|
| Install Flux | flux bootstrap CLI, commit vào Git | FluxInstance CR — declarative, self-managing |
| Multi-env config | Kustomize base/overlays per cluster | FluxInstance per cluster, sync từ OCI tag |
| Onboard tenant | Copy overlay folder, Kyverno generate SA/RB/NP | ResourceSet: thêm 5 dòng input |
| Secret sync | Kyverno clone policy | copyFrom annotation native |
| Label injection | Kyverno mutate policy | spec.commonMetadata.labels trong FluxInstance |
| Multi-tenancy lockdown | Kyverno validate policy | multitenant: true — built-in |
| Delivery source | GitRepository — Flux cần network tới Git | OCIRepository — pull từ registry, Gitless |
| Ephemeral envs | Manual script hoặc không có | ResourceSetInputProvider + GitHub PR label |
| Flux upgrade | flux bootstrap lại, commit lên Git | Sửa version trong FluxInstance, Flux tự upgrade |
| Observability | kubectl get kustomization -A | FluxReport CR — aggregated cluster health |
D2 Reference Architecture — 3 Repos, 3 Roles
Đây là điều quan trọng nhất cần hiểu đúng: D2 không phải một Helm chart cho tất cả. D2 tách ra 3 Git repositories độc lập, mỗi repo có ownership, permission, và delivery pipeline riêng.
Pipeline Gitless (OCI) rút gọn — cùng luồng với diagram trên: build artifact, ký Cosign, Flux verify trước apply.
d2-infra — Kyverno as Cluster Add-on
Kyverno sống trong d2-infra, không phải d2-apps hay d2-fleet. Platform team sở hữu repo này, reconcile với cluster admin permission. Kyverno policies là cluster-scoped — chỉ platform team mới được sửa.
Kustomization cho policies phụ thuộc HelmRelease Kyverno (dependsOn) để tránh apply policy trước khi admission webhook sẵn sàng.
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: kyverno
namespace: kyverno
spec:
interval: 1h
chart:
spec:
chart: kyverno
version: "3.3.x"
sourceRef:
kind: HelmRepository
name: kyverno
namespace: flux-system
install:
crds: CreateReplace
valuesFrom:
- kind: ConfigMap
name: kyverno-values
- kind: ConfigMap
name: kyverno-values-env
optional: true
values:
admissionController:
replicas: 3apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: kyverno-policies
namespace: flux-system
spec:
interval: 10m
dependsOn:
- name: kyverno
sourceRef:
kind: OCIRepository
name: d2-infra
path: "./components/kyverno/policies"
prune: true
wait: trued2-apps — ResourceSet delivers apps, NOT manual HelmRelease
Pattern thực tế: d2-fleet dùng ResourceSet để deliver toàn bộ d2-apps. Dev không tạo HelmRelease trực tiếp trên cluster — chỉ commit vào d2-apps; CI đóng gói OCI artifact per tenant; ResourceSet trong d2-fleet tạo OCIRepository + Kustomization per tenant.
Mỗi tenant có OCI artifact riêng (oci://ghcr.io/org/d2-apps/<tenant>). Kustomization trỏ ./staging hoặc ./production trong artifact.
apiVersion: fluxcd.controlplane.io/v1
kind: ResourceSet
metadata:
name: apps
namespace: flux-system
spec:
dependsOn:
- apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
name: infra-configs
namespace: monitoring
ready: true
inputs:
- tenant: "frontend"
tag: "${ARTIFACT_TAG}"
environment: "${ENVIRONMENT}"
resources:
- apiVersion: v1
kind: Namespace
metadata:
name: "<< inputs.tenant >>"Kind Cluster — Local Dev & CI Bootstrap
Dùng Kind để test stack locally trước khi push EKS/GKE. Config dưới đây mô phỏng production với port mapping cho ingress.
apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
name: platform-local
nodes:
- role: control-plane
kubeadmConfigPatches:
- |
kind: InitConfiguration
nodeRegistration:
kubeletExtraArgs:
node-labels: "ingress-ready=true"
extraPortMappings:
- containerPort: 80
hostPort: 8080
- containerPort: 443
hostPort: 8443
- role: worker
- role: worker
featureGates:
ValidatingAdmissionPolicy: true
networking:
podSubnet: "10.244.0.0/16"
serviceSubnet: "10.96.0.0/16"#!/usr/bin/env bash
set -euo pipefail
echo "==> Creating Kind cluster..."
kind create cluster --config=ci/kind-config.yaml --wait 60s
echo "==> Installing Flux Operator (via Helm)..."
helm repo add controlplane https://controlplaneio-fluxcd.github.io/charts
helm repo update
helm upgrade --install flux-operator controlplane/flux-operator \
--namespace flux-system \
--create-namespace \
--wait
echo "==> Deploying platform chart (staging mode)..."
helm upgrade --install platform ./platform-chart \
--namespace flux-system \
--create-namespace \
-f ./platform-chart/values.yaml \
-f ./platform-chart/values-staging.yaml \
--set global.environment=local \
--set fluxOperator.instance.sync.kind=GitRepository \
--set fluxOperator.instance.sync.url=https://github.com/your-org/fleet.git \
--set policies.validationFailureAction=Audit \
--wait --timeout=10m
echo "==> Waiting for Kyverno to be ready..."
kubectl wait --for=condition=Available \
deployment/kyverno-admission-controller \
-n kyverno --timeout=300s
echo "==> Checking FluxReport..."
kubectl get fluxreport -n flux-system -o yaml
echo "✅ Platform stack is ready!"FluxInstance — Lifecycle Management
FluxInstance CR thay thế flux bootstrap. Flux Operator deploy controllers, cấu hình sync source; Flux tự quản lý chính nó.
Staging dùng tag latest; production dùng latest-stable — không cần Kustomize overlays cho hai môi trường.
apiVersion: fluxcd.controlplane.io/v1
kind: FluxInstance
metadata:
name: flux
namespace: flux-system
spec:
distribution:
version: "2.x.x"
artifact: "oci://ghcr.io/controlplaneio-fluxcd/flux-operator-manifests"
components:
- source-controller
- kustomize-controller
- helm-controller
cluster:
type: kubernetes
multitenant: true
sync:
kind: OCIRepository
url: oci://ghcr.io/org/d2-fleet
ref: latest-stable
path: "./clusters/prod"
commonMetadata:
labels:
managed-by: "flux-operator"OCI Artifacts — Delivery Pipeline
Git vẫn là source of truth cho developer, nhưng delivery path không đi qua Git: CI build và push OCI artifacts — Flux pull từ registry. Git outage không chặn deployment.
Sơ đồ rút gọn ở trên (Gitless pipeline) bổ sung cho full D2 diagram — cùng semantics: push artifact → verify → apply.
ResourceSet — Tenant provisioning
ResourceSet là “Helm cho Flux resources”: template + inputs. Thay Kustomize overlay copy-paste hoặc Kyverno generate hàng loạt object — thêm tenant trong values, Helm upgrade, ResourceSet tạo namespace, SA, NetworkPolicy, OCIRepository, Kustomization.
copyFrom annotation sao chép Secret/ConfigMap từ flux-system sang tenant — native Flux Operator, không cần Kyverno clone.
Ephemeral environments từ PR
Gắn label deploy/flux-preview trên GitHub PR: ResourceSetInputProvider tạo preview namespace, HelmRelease với image tag = commit SHA. PR merge/close → dọn resource.
Kyverno cho preview thường Audit thay vì Enforce để image mới nhất chưa ký đủ vẫn deploy được trong khi PolicyReport ghi nhận violation.
Kyverno layers — validate / enforce
Với Flux Operator multitenant: cross-namespace reference bị chặn native. Kyverno tập trung validate image registry, require labels, và PolicyException cho preview.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-tenant-label
spec:
validationFailureAction: Enforce
rules:
- name: check-labels
match:
any:
- resources:
kinds:
- Pod
validate:
message: "tenant label required"
pattern:
metadata:
labels:
app.kubernetes.io/part-of: "?*"# Kustomization phải có SHA annotation — audit trail sau time-gate
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-deployment-sha
annotations:
policies.kyverno.io/category: Compliance
spec:
validationFailureAction: Audit
background: true
rules:
- name: require-sha-annotation
match:
any:
- resources:
kinds: ["Kustomization"]
namespaces: ["team-*"]
validate:
message: "Missing deployment SHA — was it deployed via time-gate?"
pattern:
metadata:
annotations:
deployment.company.com/sha: "?*"
deployment.company.com/deployed-at: "?*"FluxReport — Aggregated health
FluxReport CR gom sync status, components, drift — quan sát tập trung. Kết hợp PolicyReport (Kyverno) và notification để alert Slack/PagerDuty.
Full Pipeline — From PR to Production
Luồng đầy đủ: PR mở label deploy/flux-preview → preview env; review PolicyReport Audit; merge → CI đẩy OCI :latest + Cosign; staging FluxInstance sync; git tag → :latest-stable → production Kyverno Enforce + FluxReport Slack.
name: flux-artifact-cd
on:
push:
branches: [main]
pull_request:
types: [opened, synchronize, labeled]
permissions:
contents: read
id-token: write
packages: write
jobs:
build-push-sign:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Login to GHCR
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Push Flux OCI artifact (path = fleet manifests)
run: |
flux push artifact oci://ghcr.io/${{ github.repository_owner }}/fleet:${{ github.sha }} \
--path=./clusters/staging \
--source=${{ github.repository }} \
--revision=${{ github.sha }}
- name: Cosign sign (keyless)
uses: sigstore/cosign-installer@v3
- run: |
DIGEST=$(crane digest ghcr.io/${{ github.repository_owner }}/fleet:${{ github.sha }})
cosign sign --yes ghcr.io/${{ github.repository_owner }}/fleet@$DIGESThelm upgrade --install platform ./platform-chart \
-f values-production.yaml \
--namespace flux-system --create-namespaceResourceSet + Cron — Controlled deployment windows
Trong môi trường highly regulated (fintech, banking, healthcare), deployment phải tuân CAB windows, tránh peak business, và có audit trail đầy đủ.
Flux Operator: ResourceSetInputProvider chỉ fetch commit SHA mới trong cửa sổ được phép. Ngoài window, GitRepository vẫn pin SHA cũ — cluster không đổi dù code đã merge (khác suspend/resume thủ công).
Core mechanism: ResourceSet sinh GitRepository + Kustomization động; InputProvider reconcile theo schedule — ref.commit = inputs.sha. Ngoài window InputProvider không lấy SHA mới.
Nhiều mục trong schedule[] là OR: primary window + emergency window trong cùng một InputProvider.
Hành vi ngoài window: InputProvider không fetch SHA mới; GitRepository giữ pin — cluster bất biến.
| Industry | Window | Cron | Duration |
|---|---|---|---|
| Fintech / Banking | Ngoài giờ giao dịch | 0 22 * * 1-5 | 4h (22:00–02:00) |
| E-commerce | Low-traffic hours | 0 3 * * * | 3h (03:00–06:00) |
| Healthcare | Weekly maintenance | 0 2 * * 0 | 6h (Sun 02:00) |
| SRE / Platform | Business hours only | 0 9 * * 1-5 | 8h (09:00–17:00) |
| Multi-region | Follow-the-sun | 0 8 * * 1-5 | 8h per TZ |
apiVersion: fluxcd.controlplane.io/v1
kind: ResourceSetInputProvider
metadata:
name: payments-app-main
namespace: team-payments
labels:
app.kubernetes.io/name: payments-app
annotations:
fluxcd.controlplane.io/reconcileEvery: "10m"
fluxcd.controlplane.io/reconcileTimeout: "1m"
spec:
schedule:
- cron: "0 9 * * 1-5"
timeZone: "Asia/Ho_Chi_Minh"
window: 8h
- cron: "0 10 * * 6"
timeZone: "Asia/Ho_Chi_Minh"
window: 4h
type: GitHubBranch
url: https://github.com/your-org/payments-app
secretRef:
name: gh-app-auth
filter:
includeBranch: "^main$"
defaultValues:
env: productionapiVersion: fluxcd.controlplane.io/v1
kind: ResourceSetInputProvider
metadata:
name: payments-app-release
namespace: team-payments
labels:
app.kubernetes.io/name: payments-app
spec:
schedule:
- cron: "0 22 * * 2,4"
timeZone: "Asia/Ho_Chi_Minh"
window: 4h
type: GitHubTag
url: https://github.com/your-org/payments-app
secretRef:
name: gh-app-auth
filter:
semver: ">=1.0.0"
limit: 1apiVersion: fluxcd.controlplane.io/v1
kind: ResourceSetInputProvider
metadata:
name: payments-chart-release
namespace: team-payments
labels:
delivery-mode: oci-time-based
spec:
schedule:
- cron: "0 22 * * 2,4"
timeZone: "Asia/Ho_Chi_Minh"
window: 4h
type: OCITag
url: oci://ghcr.io/your-org/payments-chart
secretRef:
name: oci-auth
filter:
semver: ">=1.0.0"
limit: 1apiVersion: fluxcd.controlplane.io/v1
kind: ResourceSet
metadata:
name: payments-app
namespace: team-payments
spec:
inputsFrom:
- kind: ResourceSetInputProvider
selector:
matchLabels:
app.kubernetes.io/name: payments-app
resources:
- apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: payments-app
namespace: "<< inputs.provider.namespace >>"
spec:
interval: 12h
url: https://github.com/your-org/payments-app
ref:
commit: "<< inputs.sha >>"
secretRef:
name: gh-app-auth
sparseCheckout:
- deploy
- apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: payments-app
namespace: "<< inputs.provider.namespace >>"
spec:
interval: 30m
retryInterval: 5m
serviceAccountName: flux-payments
prune: true
wait: true
timeout: 5m
sourceRef:
kind: GitRepository
name: payments-app
path: deploy/<< inputs.env >>
commonMetadata:
annotations:
deployment.company.com/sha: "<< inputs.sha >>"
deployment.company.com/branch: "<< inputs.branch >>"
deployment.company.com/deployed-at: "<< inputs.provider.lastUpdated >>"