GitOps trên AWS EKS
Portfolio kiến trúc GitOps / platform — hero, overview, architecture, VPC ingress diagram.
Enterprise GitOps
on AWS EKS
Production-Grade Multi-Cluster Kubernetes with Advanced Auto-Scaling, GitOps Automation & Enterprise Observability
Built by Duy · gitops.duyne.me
📋 Project Overview
A complete, production-ready GitOps implementation demonstrating modern cloud-native practices on AWS. This project showcases enterprise-grade infrastructure automation, multi-cluster management, and comprehensive CI/CD pipelines.
Advanced networking with network policies and BGP routing for zero-trust pod-to-pod communication.
Fast node provisioning in seconds. Event-driven pod autoscaling with custom metrics. HPA: 3–11 replicas.
Enterprise observability with Alloy agents, Prometheus metrics, Loki logs, and real-time dashboards.
Policy-as-code for Kubernetes. Security baseline enforcement written in YAML, not Rego.
Zero secrets in Git. AWS Secrets Manager synced to Kubernetes via ESO with IRSA authentication.
ReadWriteMany shared storage across multiple pods with init container pattern and HPA compatibility.
🏗️ Architecture
- 3 managed nodes + Karpenter SPOT (1 per AZ)
- 3 NAT gateways (AZ-a, AZ-b, AZ-c — full HA)
- Manual sync — approval required
- HTTPS/TLS via ACM certificates
- DNS: *.gitops.duyne.me via Route 53
🌐 VPC & Networking Architecture
Internet → IGW → public subnets (ALB, NAT) → private worker subnets; VPC interface endpoints cho AWS APIs khi cluster private.
Private cluster = không có route internet từ worker node. Mỗi AWS API call phải qua VPC Endpoint. Thiếu một endpoint → service crash theo cách khó debug.
| Security Group | Inbound | Outbound | Attached to |
|---|---|---|---|
| alb-sg | 443 from 0.0.0.0/0 (internet) 80 redirect → 443 | 8080 → node-sg only | ALB (managed by LBC) |
| node-sg | 8080 from alb-sg All from node-sg (pod-to-pod) 443 from control-plane-sg | All (NAT GW / VPC EP) | EC2 worker nodes |
| efs-sg | 2049 (NFS) from node-sg CIDR only | Deny all | EFS Mount Targets |
| vpce-sg | 443 from node-sg CIDR (10.0.11.0/24, 10.0.12.0/24, 10.0.13.0/24) | Deny all | Interface VPC Endpoints |
| eks-control-plane-sg | 443 from node-sg (kubelet → API server) | All to node-sg | EKS managed (auto-created) |
🎯 Interview Deep Dive — Networking & Design
- Attach vào VPC — cho phép traffic hai chiều giữa internet và resource có public IP.
- EC2 trong public subnet + Elastic IP + route 0.0.0.0/0 → IGW = reachable từ internet.
- Bản thân IGW không làm NAT — nó chỉ là gateway; OS của EC2 giữ private IP.
- Dùng cho: ALB/NLB public, bastion host, public-facing EC2.
- Đặt trong public subnet — cho phép private subnet đi ra internet một chiều (outbound only).
- Private EC2/pod không có public IP → route 0.0.0.0/0 → NAT GW → IGW → internet.
- NAT GW làm SNAT: thay source IP private → Elastic IP của NAT GW.
- Internet không thể initiate connection vào private resource qua NAT GW.
- Dùng cho: EKS pods pull image, Lambda update, RDS patch — không expose ra ngoài.
- Enable RDS logs: log_destination=csvlog, log_statement=all, log_min_duration_statement=1000 (slow query).
- RDS export sang CloudWatch Logs (parameter group: shared_preload_libraries=pgaudit nếu cần audit).
- Option A (managed): CloudWatch Logs → Kinesis Firehose → Elasticsearch Service (có thể thêm Lambda transform).
- Option B (self-hosted): Logstash/Filebeat trong VPC → subscribe CloudWatch Logs stream → push ES.
- RabbitMQ expose metrics qua Management Plugin HTTP API (:15672) và Prometheus endpoint (:15692).
- Logs: /var/log/rabbitmq/*.log — Filebeat sidecar (K8s) hoặc agent trên EC2.
- Filebeat: input.type: log → output.elasticsearch.hosts hoặc qua Logstash pipeline.
- Amazon MQ (managed RabbitMQ): logs auto → CloudWatch Logs → cùng pipeline như RDS.
# RDS: slow query + audit logs
resource "aws_db_parameter_group" "pg" {
parameter {
name = "log_min_duration_statement"
value = "1000" # ms
}
parameter {
name = "shared_preload_libraries"
value = "pgaudit"
}
}
resource "aws_db_instance" "this" {
enabled_cloudwatch_logs_exports = ["postgresql", "upgrade"] # auto push to CW Logs
}
# CW Logs → Kinesis Firehose → ES
resource "aws_cloudwatch_log_subscription_filter" "rds" {
log_group_name = "/aws/rds/instance/prod-db/postgresql"
filter_pattern = ""
destination_arn = aws_kinesis_firehose_delivery_stream.es.arn
}
# Filebeat (K8s sidecar) — ví dụ comment trong filebeat.yml
# output.elasticsearch.hosts: ["https://es.prod.internal:9200"]
# index: "rabbitmq-logs-%{+yyyy.MM.dd}"Flow chuẩn: Internet → Route 53 → ALB (public subnet) → AWS Load Balancer Controller (Ingress) → Kubernetes Service (ClusterIP) → Pod. ALB terminate TLS, check auth, rồi mới forward vào cluster.
| Kiểu tấn công | Layer | Giải pháp | Detect |
|---|---|---|---|
| DDoS / Flood | ALB layer | AWS Shield Standard (free) chặn L3/L4. WAF rate-based rule >1000 req/5min/IP → block. ALB connection limit per target. | CloudWatch RequestCount spike → alarm → SNS |
| SQLi / XSS | Application layer qua ALB | AWS WAF managed: AWSManagedRulesCommonRuleSet, AWSManagedRulesSQLiRuleSet. Attach Web ACL vào ALB. | WAF sampled requests → CloudWatch → alert |
| Bypass Ingress → Pod trực tiếp | VPC/SG layer | Security Group pod chỉ allow từ ALB SG. NetworkPolicy (Calico): deny ingress trừ từ namespace ingress/controller. | VPC Flow Logs → CloudWatch Insights |
| Path traversal / API abuse | Ingress routing | Ingress rule strict path. Kyverno enforce non-root. RBAC tối thiểu cho ServiceAccount. | App logs → metric filter |
| Credential stuffing | ALB → App | WAF Bot Control. ALB Cognito/OIDC auth. Rate limit per IP trên /login. | WAF logs → Athena |
| TLS downgrade / expired cert | ALB TLS policy | ssl-policy: ELBSecurityPolicy-TLS13-1-2-2021-06. ACM auto-renew. HSTS qua response rule. | ACM expiry alert 45 ngày trước |
| Scenario | Dùng gì | Tránh |
|---|---|---|
| Private subnet cần pull image từ ECR | VPC Endpoint (ecr.dkr) + NAT fallback | Kéo image public qua NAT không cần thiết |
| 3+ VPC cần connectivity | Transit Gateway hub-and-spoke | Mesh VPC peering N² |
| Expose internal API cho account khác, CIDR overlap | PrivateLink (NLB + Endpoint Service) | Peering (cần CIDR disjoint) |
| EKS pod → S3 không qua internet | S3 Gateway Endpoint (free) + route | NAT egress cho S3 |
| Multi-account + on-prem share Direct Connect | TGW + Direct Connect Gateway | DX riêng từng account |
🏛️ AWS Landing Zone & Multi-Account Architecture
- DEV + PROD cùng account → IAM policy lỗi → prod data bị xóa.
- Service quota shared: một team dùng hết EC2 limit, team khác không deploy được.
- Blast radius: một IAM role compromise → toàn bộ infra bị ảnh hưởng.
- Cost allocation mơ hồ: không biết team nào tốn bao nhiêu.
- Compliance: PCI/HIPAA workload cùng account với dev sandbox.
- Blast radius isolation: account boundary là security boundary mạnh nhất AWS.
- Service quota per-account: Prod không bị ảnh hưởng bởi dev experiments.
- Cost clarity: cost per account → tag theo team/product.
- Least privilege: Developer không có access vào prod account.
- SCP: Prod account bị deny mọi thứ ngoài whitelist, kể cả root user.
🔄 GitOps Workflow
- flux bootstrap CLI tạo manifest → push Git; khó IaC hóa hết.
- Khó tune requests/limits cho từng controller.
- Flux Operator + FluxInstance CRD — declarative, Terraform/kubectl friendly.
- Upgrade bằng cách đổi spec.distribution.version; operator reconcile controllers.
- Kustomize patches cho concurrency, NetworkPolicy nội bộ flux-system.
🔐 GitHub Actions ↔ AWS OIDC
- IAM user long-lived keys trong GitHub Secrets — lộ là mất quyền vĩnh viễn.
- Rotate thủ công, khó least-privilege theo từng workflow.
- GitHub mint JWT (TTL ngắn), STS AssumeRoleWithWebIdentity → temp creds.
- Trust policy giới hạn repo/branch/environment; CloudTrail audit đầy đủ.
Chi tiết trust policy, workflow YAML và sơ đồ đầy đủ: GitHub Actions OIDC → AWS.
⚡ Advanced Auto-Scaling
Chiến lược multi-layer: Karpenter (node, 30–60s), KEDA (event-driven pod), HPA (CPU/mem 3–11 replicas). Cluster Autoscaler classic thường 3–5 phút cho node mới.
| Component | Old | New | Reason |
|---|---|---|---|
| AWS EKS | 1.31 | 1.34 | Standard support, containerd 2.x, DRA / storage improvements |
| Karpenter | v1.0.5 | v1.4.2 | Stable v1 API; AL2023 AMI |
| KEDA | 2.16.1 | 2.18.x | Supported release line |
| Application | Replicas | Min/Max | CPU | Memory |
|---|---|---|---|---|
| go-app-prod | 3 | 3 / 10 | 0% | 34% |
| demo-app-prod | 3 | 3 / 9 | 1% | 62% |
| api-app-prod | 3 | 3 / 11 | 1% | 34% |
🔒 Security & Compliance
Tầng account (SCP, CloudTrail) → VPC (SG, NACL, flow logs) → cluster (Kyverno, NetworkPolicy, IRSA) → workload (non-root, read-only rootfs). Wiki HTML có thêm sơ đồ defense grid; semantics được giữ ở checklist và stack cards.
🛠️ Technology Stack
- AWS EKS (Kubernetes 1.34)
- VPC with public/private subnets
- Application Load Balancers
- AWS Secrets Manager
- KMS for encryption
- EFS for shared storage (RWX)
- 9 VPC Endpoints
- FluxCD Operator (auto-reconcile, health checks)
- GitHub Actions Reusable Workflows
- Kustomize for config mgmt
- Multi-platform Docker builds
- Automated testing pipeline
- FluxInstance + ResourceSet pattern
- Karpenter v1.4.2 — node scaling
- KEDA 2.18.x — event-driven
- HPA — 3–11 replicas
- Fast provisioning (seconds)
- Cost-optimized SPOT nodes
- Multi-AZ distribution
- Grafana Cloud (SaaS)
- Grafana Alloy agents
- Prometheus metrics
- Loki log aggregation
- Kube State Metrics
- Node Exporter
- Kyverno — policy engine
- External Secrets Operator
- AWS Secrets Manager
- Calico network policies
- IRSA for AWS access
- Non-root containers
- Go REST API
- Node.js Express apps
- React frontends
- REST APIs
- Health check endpoints
- Spring Security (Basic Auth)
🏗️ Infrastructure as Code
| Terraform Module | What It Creates |
|---|---|
| VPC Module | Public/private subnets, NAT gateways (3, 1 per AZ), 9 VPC endpoints, route tables, IGW |
| EKS Module | Managed Kubernetes cluster, managed node groups (2× t3.medium), OIDC provider for IRSA |
| Karpenter Module | IAM roles, EC2NodeClass, NodePool, aws-auth ConfigMap entry |
| ALB Controller Module | IAM role with IRSA, Helm release, IngressClass |
| External Secrets Module | IAM role with IRSA, Helm release, ClusterSecretStore |
Mọi Terraform module follow pattern tagging nhất quán: tag derive từ basename(abspath(path.module)) — Cost breakdown theo team/product; terraform-module tự cập nhật khi rename folder.
# modules/_common/labels.tf — dùng chung mọi module
locals {
default_tags = {
(basename(abspath(path.module))) = var.name
"terraform-module" = basename(abspath(path.module))
"product" = var.product
"team" = var.team
"environment" = var.environment
"managed-by" = "terraform"
}
merged_tags = merge(local.default_tags, var.tags)
}
variable "name" { type = string }
variable "product" { type = string }
variable "environment" { type = string }
variable "tags" { type = map(string); default = {} }
variable "team" { type = string; default = "" }
variable "squad" {
type = string
default = ""
description = "Deprecated: use team instead"
}
locals {
resolved_team = coalesce(var.team, var.squad, "unknown")
}# modules/eks/main.tf — consuming the pattern
module "eks" {
source = "../modules/eks"
name = "prod-cluster"
product = "platform"
team = "infra"
environment = "prod"
tags = {
"cost-center" = "engineering"
"criticality" = "high"
}
}
resource "aws_eks_cluster" "this" {
name = var.name
tags = local.merged_tags
}🔑 IRSA vs EKS Pod Identity
Pod trong EKS cần gọi AWS APIs (S3, Secrets Manager, DynamoDB…). Nếu không có cơ chế riêng, pod phải dùng node IAM role — mọi pod trên node chia sẻ quyền, vi phạm least privilege. IRSA và EKS Pod Identity đều giải quyết — theo hai cơ chế khác nhau.
Wiki HTML có sơ đồ so sánh IRSA vs EKS Pod Identity (flow lớn). Trang React giữ bảng quyết định + Terraform song song — đủ semantic để phỏng vấn.
| Tiêu chí | IRSA | Pod Identity |
|---|---|---|
| Setup phức tạp | Cao — OIDC + trust hardcode | Thấp — một lệnh association |
| IAM Trust Policy | AssumeRoleWithWebIdentity + OIDC URL + sub | AssumeRole + TagSession + Service pods.eks.amazonaws.com |
| Multi-cluster | N cluster = N OIDC = N trust | Một role dùng lại nhiều cluster |
| Cross-account | Native qua trust policy | Cần thêm bước, phức tạp hơn |
| Cluster rotation | OIDC URL đổi → update trust | Association re-bind, ít đụng IAM hơn |
| Fargate | Yes | Chưa (tính đến 2024) |
| Addon | Built-in webhook | eks-pod-identity-agent |
| Project dùng | ESO, Loki, cross-account | go-api, VMAgent, workload mới |
# IRSA — OIDC + trust theo sub
resource "aws_iam_openid_connect_provider" "eks" {
client_id_list = ["sts.amazonaws.com"]
thumbprint_list = [data.tls_certificate.eks.certificates[0].sha1_fingerprint]
url = aws_eks_cluster.prod.identity[0].oidc[0].issuer
}
data "aws_iam_policy_document" "irsa_assume" {
statement {
actions = ["sts:AssumeRoleWithWebIdentity"]
principals {
type = "Federated"
identifiers = [aws_iam_openid_connect_provider.eks.arn]
}
condition {
test = "StringEquals"
variable = "${aws_iam_openid_connect_provider.eks.url}:sub"
values = ["system:serviceaccount:go-api:go-api-sa"]
}
}
}# Pod Identity — trust generic
data "aws_iam_policy_document" "pod_identity_assume" {
statement {
actions = ["sts:AssumeRole", "sts:TagSession"]
principals {
type = "Service"
identifiers = ["pods.eks.amazonaws.com"]
}
}
}
resource "aws_eks_addon" "pod_identity" {
cluster_name = aws_eks_cluster.prod.name
addon_name = "eks-pod-identity-agent"
}
resource "aws_eks_pod_identity_association" "go_api" {
cluster_name = aws_eks_cluster.prod.name
namespace = "go-api"
service_account = "go-api-sa"
role_arn = aws_iam_role.go_api.arn
}- External Secrets Operator — Secrets Manager
- Loki S3 backend, Tempo traces bucket
- Cross-account (staging → prod S3)
- Fargate-based workloads
- go-api, api-app — S3 read, SSM
- VMAgent — CloudWatch, S3 queue
- Karpenter, ALB Ingress Controller
- Workload mới — default
📊 Grafana Cloud → Self-Hosted Operator
- Free tier: 10k series, 50GB logs/traces
- Alloy push ra ngoài VPC qua HTTPS
- Retention/dashboard version control hạn chế
- Scale → giá tăng nhanh; dữ liệu ra ngoài VPC (compliance)
- Grafana, Prometheus, Loki, Tempo trong cluster — data không rời VPC
- Dashboards/datasources/alerts = CRD → GitOps (Flux)
- Retention kiểm soát (Loki compactor, Prometheus TSDB)
- Cost chủ yếu EBS/S3; tradeoff: vận hành HA/upgrade/backup
Wiki gốc có sơ đồ Grafana Operator ~920×480 (Git → Flux → Operator → Grafana + Prometheus/Loki/Tempo/Alloy). Ở đây giữ prose + bảng + Helm snippet; xem HTML wiki nếu cần pixel-level diagram.
| Cloud | Self-hosted | |
|---|---|---|
| Cost @ ~50k series | ~$99/mo | ~$20/mo (EBS+S3) |
| Dashboard GitOps | Click-ops UI | GrafanaDashboard CRD |
| Data locality | Ra ngoài VPC | In-VPC |
| Setup time | ~5 phút | ~2h + ops |
| Alerts | UI | GrafanaAlertRule CRD |
- Deploy Grafana Operator + Grafana instance (HelmRelease/Flux)
- Export dashboard từ Cloud → GrafanaDashboard CRD
- kube-prometheus-stack (tắt Grafana bundled)
- Loki + Alloy → in-cluster Loki thay Cloud
- Tempo + OTLP trong app
- GrafanaDatasource CRD → Prometheus/Loki/Tempo
- GrafanaAlertRule CRD → Git
- Cutover: dừng push Alloy ra Cloud
# Grafana Operator + instance (rút gọn)
helmRelease: grafana-operator
chart: grafana-operator/grafana-operator
---
apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
name: prod-grafana
labels: { dashboards: prod }
spec:
deployment: { spec: { replicas: 2 } }
config:
server: { root_url: https://grafana.gitops.duyne.me }
---
# kube-prometheus-stack: grafana.enabled: false
# loki: storage S3 IRSA cho chunks🚀 Vector — Log pipeline
Project dùng Grafana Alloy (metrics + logs + traces). Vector (Rust) là alternative throughput cao, footprint thấp; cả hai là unified agent — thay Promtail + Fluent Bit + OTel collector rời.
Wiki có sơ đồ pipeline Vector ~960×400 (sources → VRL transforms → sinks). Dưới đây là bảng so sánh đầy đủ.
| Tiêu chí | Vector | Alloy | Fluent Bit | Logstash |
|---|---|---|---|---|
| Ngôn ngữ | Rust | Go | C — rất nhẹ | JVM — nặng |
| Memory (idle) | ~10–30 MB | ~50–80 MB | ~1–5 MB | ~300–500 MB |
| Throughput | ~10M ev/s (benchmark) | ~3–5M ev/s | ~1–2M ev/s | ~500K ev/s |
| Unified agent | metrics+logs+traces | OTel-native + Loki/Tempo | Logs | Logs |
| Transforms | VRL type-safe | River + otelcol | Lua limited | Grok/plugins |
| K8s metadata | kubernetes_logs source | Built-in discovery | K8s filter | Qua Filebeat |
| Grafana stack | Neutral | First-class | OK | ELK oriented |
| Dùng khi | High throughput, VRL phức tạp, fan-out multi sink | Đã dùng Grafana stack, River quen thuộc | Edge / chỉ forward log đơn giản | Legacy Grok / ELK |
📈 VictoriaMetrics — Cluster → Operator
- Helm per cluster → drift staging/prod
- vminsert/vmstorage/vmselect lệch config theo thời gian
- Scrape/alert rules trong ConfigMap — khó review, không GitOps chặt
- Ops overhead ~8h/tháng
- VMCluster CRD — lifecycle vminsert/storage/select
- VMAgent + VMServiceScrape/VMPodScrape — sharding, remote_write
- VMRule CRD — validate + GitOps
- Upgrade: đổi image tag → operator rolling
- ResourceSetInputProvider → parity đa môi trường
- Ops ~45 phút/tháng (ước lượng wiki)
Wiki có architecture SVG ~960×560 (GitOps → vm-operator → VMCluster/VMAgent/VMAlert/Grafana). Phần dưới: lý do migrate + YAML mẫu.
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMCluster
metadata:
name: vm-prod
namespace: monitoring
spec:
retentionPeriod: "3"
replicationFactor: 2
vminsert:
replicaCount: 2
image: { tag: v1.101.0-cluster }
vmstorage:
replicaCount: 3
storage:
volumeClaimTemplate:
spec:
storageClassName: gp3
resources: { requests: { storage: 100Gi } }
vmselect:
replicaCount: 2
extraArgs:
dedup.minScrapeInterval: 1mFlux ResourceSet + ResourceSetInputProvider: một template VMCluster/VMAgent cho mọi env; chỉ defaultValues (replicas, storage, scrape_interval) thay đổi — tránh drift cấu hình.
🛡️ Kyverno — Admission control
Kyverno ("to govern") là policy engine trong cluster — dynamic admission controller giữa kubectl/CI và etcd.
Request đi qua Authn/RBAC → Mutating webhook (Kyverno) → Validating webhook (Kyverno) → etcd.
Sai rule → Enforce block hoặc Audit log.
- non-root
- có limits
- không :latest
- đúng namespace/env
Tự patch: seccomp, label env, default limits.
- inject seccomp RuntimeDefault
- imagePullPolicy Always
Namespace mới → NetworkPolicy/Quota/RoleBinding.
- multi-tenant guardrails
Cosign/Sigstore — chỉ image đã ký.
- supply chain
- SOC2/PCI provenance
| Audit | Enforce | |
|---|---|---|
| Khi vi phạm | Cho qua + PolicyReport | 403 block |
| Cluster impact | Không dừng workload | Deploy fail |
| Dùng khi | Giai đoạn đầu / brownfield | Sau khi policy đã test |
| Project | Đang Audit — violations log, chưa block toàn phần | — roadmap |
| Feature | Gatekeeper | Kyverno |
|---|---|---|
| Mutation | Beta (Assign) | GA, YAML patch |
| Generation | Không native | Có (NetworkPolicy...) |
| Image verify | Extension | Built-in Cosign |
| Policy language | Rego | YAML native |
| PolicyReport | Không | CRD self-service |
| Exception TTL | Custom | PolicyException |
| Multi-platform | OPA everywhere | K8s only |
| Policy | Type | Category | Mục đích |
|---|---|---|---|
| detect-mixed-environments | Validate | Isolation | Cảnh báo trộn dev/prod |
| enforce-environment-separation | Validate | Isolation | Chặn cross-environment |
| kyverno-block-dev-in-prod | Validate | Isolation | Dev workload không vào prod ns |
| kyverno-block-prod-in-dev | Validate | Isolation | Prod workload không vào dev ns |
| require-resource-quotas | Validate | Cost | Quota cho dev namespaces |
| warn-anti-patterns | Validate | Best practice | :latest, thiếu limits, auto-sync prod |
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: kyverno-block-dev-in-prod
spec:
validationFailureAction: Audit
rules:
- name: block-dev-label-in-prod
match:
resources:
namespaceSelector:
matchLabels: { env: prod }
validate:
deny:
conditions:
- key: "{{ request.object.metadata.labels.env || '' }}"
operator: Equals
value: dev📦 Repository structure
Terraform modules đa môi trường: VPC, EKS, Karpenter, ALB Controller, ESO, observability addons, Kyverno. Remote state S3 + DynamoDB.
Kustomize base + DEV/PROD overlays, Flux Kustomization, FluxInstance, ResourceSet (monitoring, apps).
Go REST API: router, middleware, zerolog, tests, multi-arch Docker, /metrics Prometheus.
Express demo: multi-stage Docker, non-root, HEALTHCHECK, patterns GitOps.