OPA Gatekeeper

OPA는 K8S 뿐만 아니라 다양한 영역에서 운영환경에 대한 정책을 적용하고 통합하기 위해 사용되는 Open Source 기반의 플랫폼입니다. 현재 CNCF 의 Graduated Project 상태로 정책의 코드화된 관리를 목적으로 다양하게 사용되고 있습니다. AWS 에서는 EKS 뿐만 아니라 AWS Config 서비스에서도 OPA 기반의 Config 규칙을 사용하실 수 있습니다.

OPA 는 Rego 라는 별도의 Language 를 통해 정책을 생성/관리합니다. Rego Language는              OPA Playground 에서 작성한 OPA 정책을 확인할 수 있습니다.

OPA Gatekeeper 는 OPA 에서 생성된 정책을 K8S 환경에서 적용하기 위해 사용하는 K8S 를 위한 Adminssion Webhook 입니다. OPA Gatekeep 의 기본적인 Flow 는 아래와 같습니다.

ENV

REGION_CODE=$(aws configure get default.region --output text)
AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text)
Shell
복사

OPA Gatekeeper Install

kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/v3.17.1/deploy/gatekeeper.yaml
Shell
복사

kubectl get pods -n gatekeeper-system
Shell
복사

# audit-controller log 확인
kubectl logs -l control-plane=audit-controller -n gatekeeper-system

# gatekeeper-system log 확인
kubectl logs -l control-plane=controller-manager -n gatekeeper-system
Shell
복사

실습

Container Image 제한하기

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: enforceimagelist
spec:
  crd:
    spec:
      names:
        kind: enforceimagelist
      validation:
        openAPIV3Schema:
          properties:
            images:
              type: array
              items: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package enforceimagelist

        allowlisted_images = {images |
            images = input.parameters.images[_]
        }

        images_allowlisted(str, patterns) {
            image_matches(str, patterns[_])
        }

        image_matches(str, pattern) {
            contains(str, pattern)
        }

        violation[{"msg": msg}] {
          input.review.object
          image := input.review.object.spec.containers[_].image
          name := input.review.object.metadata.name
          not images_allowlisted(image, allowlisted_images)
          msg := sprintf("pod %q has invalid image %q. Please, contact Security Team. Follow the allowlisted images %v", [name, image, allowlisted_images])
        }
YAML
복사

# ConstraintTemplate
kubectl apply -f constraint-template-image.yaml 
Shell
복사

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: enforceimagelist
metadata:
  name: k8senforceallowlistedimages
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
  parameters:
    images:
      - 362708816803.dkr.ecr.$AWS_REGION.amazonaws.com/eks-security-shared
      - amazon/aws-node-termination-handler
      - amazon/aws-alb-ingress-controller
      - amazon/aws-efs-csi-driver
      - amazon/cloudwatch-agent
      - docker.io/amazon/aws-alb-ingress-controller
      - grafana/grafana
      - prom/alertmanager
      - prom/prometheus
      - openpolicyagent/gatekeeper
      - amazon/aws-cli
      - busybox
      - nginx
      - falco
YAML
복사

# Constraint
kubectl apply -f constraint-image.yaml
Shell
복사

kubectl get constrainttemplate
kubectl get constraint
Shell
복사

# constraintTemplate 상세 내용 확인
kubectl get constrainttemplate -o yaml enforceimagelist

# constraint 의 상세 내용을 확인
kubectl get constraint -o yaml k8senforceallowlistedimages
Shell
복사

OPA 정책에 위반된 Pod를 배포 (Error)

apiVersion: v1
kind: Pod
metadata:
  name: invalid-image
  labels:
    app: invalid-image
  namespace: default
spec:
  containers:
    - image: docker:latest
      command:
        - sleep
        - "3600"
      imagePullPolicy: IfNotPresent
      name: invalid-docker
  restartPolicy: Always
YAML
복사

kubectl apply -f pod-with-invalid-image.yaml
Shell
복사

# Error 발생
Error from server (Forbidden): error when creating "/home/ec2-user/environment/opa/pod-with-invalid-image.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [k8senforceallowlistedimages] pod "invalid-image" has invalid image "docker:latest". Please, contact Security Team. Follow the allowlisted images {"300861432382.dkr.ecr.ap-northeast-2.amazonaws.com/eks-security-shared", "amazon/aws-alb-ingress-controller", "amazon/aws-cli", "amazon/aws-efs-csi-driver", "amazon/aws-node-termination-handler", "amazon/cloudwatch-agent", "busybox", "docker.io/amazon/aws-alb-ingress-controller", "falco", "grafana/grafana", "nginx", "openpolicyagent/gatekeeper", "prom/alertmanager", "prom/prometheus"}
Shell
복사

OPA 정책에 위반 되지 않은 Pod를 배포 (Success)

apiVersion: v1
kind: Pod
metadata:
  name: valid-image
  labels:
    app: valid-image
  namespace: default
spec:
  containers:
    - image: busybox
      command:
        - sleep
        - "3600"
      imagePullPolicy: IfNotPresent
      name: valid-busybox
  restartPolicy: Always
YAML
복사

kubectl apply -f pod-with-valid-image.yaml
Shell
복사

•

Container Image 사용제한 정책 삭제

kubectl delete -f pod-with-valid-image.yaml
kubectl delete -f constraint-image.yaml
kubectl delete -f constraint-template-image.yaml 
Shell
복사

Privilege Container 사용 제한하기

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: enforceprivilegecontainer
spec:
  crd:
    spec:
      names:
        kind: enforceprivilegecontainer
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package enforceprivilegecontainer

        violation[{"msg": msg, "details": {}}] {
            c := input_containers[_]
            c.securityContext.privileged
            msg := sprintf("Privileged container is not allowed: %v, securityContext: %v", [c.name, c.securityContext])
        }

        input_containers[c] {
            c := input.review.object.spec.containers[_]
        }

        input_containers[c] {
            c := input.review.object.spec.initContainers[_]
        }
YAML
복사

# ConstraintTemplate
kubectl apply -f constraint-template-privileged.yaml
Shell
복사

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: enforceprivilegecontainer
metadata:
  name: privileged-container-security
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
YAML
복사

# Constraint
kubectl apply -f constraint-privileged.yaml
Shell
복사

# constraintTemplate 상세 내용 확인
kubectl get constrainttemplate -o yaml enforceprivilegecontainer

# constraint 의 상세 내용을 확인
kubectl get constraint -o yaml privileged-container-security
Shell
복사

허용되지 않은 Privileged를 사용한 Pod 배포 (Error)

apiVersion: v1
kind: Pod
metadata:
  name: privileged-container
  labels:
    role: privileged-container
  namespace: default
spec:
  containers:
    - image: busybox
      command:
        - sleep
        - "3600"
      imagePullPolicy: IfNotPresent
      name: privileged-container
      securityContext:
        privileged: true # true or false
  restartPolicy: Always
YAML
복사

kubectl apply -f privileged-container.yaml
Shell
복사

"privileged:" 값이 "false" 로 처리되어 있는 Pod를 배포 (Success)

apiVersion: v1
kind: Pod
metadata:
  name: privileged-container
  labels:
    role: privileged-container
  namespace: default
spec:
  containers:
    - image: busybox
      command:
        - sleep
        - "3600"
      imagePullPolicy: IfNotPresent
      name: privileged-container
      securityContext:
        privileged: false # true or false
  restartPolicy: Always
YAML
복사

kubectl apply -f privileged-container.yaml
Shell
복사

•

Container Image 사용제한 정책 삭제

kubectl delete -f privileged-container.yaml
kubectl delete -f constraint-privileged.yaml
kubectl delete -f constraint-template-privileged.yaml
Shell
복사

Repository 사용 제한하기

nginx image 를 Pull 후 "eks-security-shared" Repository에 업로드할 수 있도록 Tag를 부여

aws ecr get-login-password --region $REGION_CODE | \
docker login --username AWS --password-stdin \
$AWS_ACCOUNT_ID.dkr.ecr.$REGION_CODE.amazonaws.com
Shell
복사

docker pull public.ecr.aws/docker/library/nginx
docker tag public.ecr.aws/docker/library/nginx $AWS_ACCOUNT_ID.dkr.ecr.$REGION_CODE.amazonaws.com/eks-security-shared
Shell
복사

docker push $AWS_ACCOUNT_ID.dkr.ecr.$REGION_CODE.amazonaws.com/eks-security-shared
Shell
복사

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: allowedrepos
  annotations:
    description: >-
      Requires container images to begin with a string from the specified list.
spec:
  crd:
    spec:
      names:
        kind: AllowedRepos
      validation:
        openAPIV3Schema:
          type: object
          properties:
            repos:
              description: The list of prefixes a container image is allowed to have.
              type: array
              items:
                type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package allowedrepos

        violation[{"msg": msg}] {
          container := input.review.object.spec.containers[_]
          satisfied := [good | repo = input.parameters.repos[_] ; good = startswith(container.image, repo)]
          not any(satisfied)
          msg := sprintf("container <%v> has an invalid image repo <%v>, allowed repos are %v", [container.name, container.image, input.parameters.repos])
        }

        violation[{"msg": msg}] {
          container := input.review.object.spec.initContainers[_]
          satisfied := [good | repo = input.parameters.repos[_] ; good = startswith(container.image, repo)]
          not any(satisfied)
          msg := sprintf("initContainer <%v> has an invalid image repo <%v>, allowed repos are %v", [container.name, container.image, input.parameters.repos])
        }

        violation[{"msg": msg}] {
          container := input.review.object.spec.ephemeralContainers[_]
          satisfied := [good | repo = input.parameters.repos[_] ; good = startswith(container.image, repo)]
          not any(satisfied)
          msg := sprintf("ephemeralContainer <%v> has an invalid image repo <%v>, allowed repos are %v", [container.name, container.image, input.parameters.repos])
        }
YAML
복사

# ConstraintTemplate
kubectl apply -f constraint-template-repository.yaml
Shell
복사

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: AllowedRepos
metadata:
  name: allowed-repo
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    namespaces:
      - "default"
  parameters:
    repos:
      - "362708816803.dkr.ecr.ap-northeast-2.amazonaws.com/"
YAML
복사

# constraint
kubectl apply -f constraint-repository.yaml
Shell
복사

kubectl get constrainttemplate
kubectl get constraint
Shell
복사

등록되지 않은 Repository 를 이용한 Pod 배포 (Error)

apiVersion: v1
kind: Pod
metadata:
  name: nginx-disallowed
spec:
  containers:
    - name: nginx
      image: nginx
      resources:
        limits:
          cpu: "100m"
          memory: "30Mi"
YAML
복사

kubectl apply -f disallowed-repository.yaml
Shell
복사

등록된 ECR Repository 를 사용하도록 하는 Pod를 배포 (Success)

apiVersion: v1
kind: Pod
metadata:
  name: nginx-allowed
spec:
  containers:
    - name: shared-nginx
      image: 362708816803.dkr.ecr.ap-northeast-2.amazonaws.com/eks-security-shared
      command:
        - sleep
        - "3600"
      resources:
        limits:
          cpu: "100m"
          memory: "30Mi"
YAML
복사

kubectl apply -f allowed-repository.yaml
Shell
복사

•

Repository 사용제한 정책 삭제

kubectl delete -f allowed-repository.yaml
kubectl delete -f constraint-repository.yaml
kubectl delete -f constraint-template-repository.yaml
Shell
복사

특정 namespace(dev)의 deployment에 nodeSelector 강제 할당

apiVersion: mutations.gatekeeper.sh/v1
kind: Assign
metadata:
  name: mutator-dev-ns
  namespace: gatekeeper-system
spec:
  applyTo:
  - groups: ["apps"]
    kinds: ["Deployment"]
    versions: ["v1"]
  match:
    namespaces:
    - "dev"
  location: "spec.template.spec.nodeSelector"
  parameters:
    assign:
      value:
        eks.amazonaws.com/nodegroup: skills-eks-nodegroup
YAML
복사

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample
  namespace: dev
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample
  template:
    metadata:
      labels:
        app: sample
    spec:
      containers:
      - name: sample
        image: nginx
        resources:
          requests:
            memory: "128Mi"
            cpu: "128m"
          limits:
            memory: "128Mi"
            cpu: "128m"
        ports:
        - containerPort: 8080
YAML
복사

kube-system은 모든 정책 제외

apiVersion: config.gatekeeper.sh/v1alpha1
kind: Config
metadata:
  name: kube-system-ns-exclude-config
  namespace: gatekeeper-system
spec:
  match:
    - excludedNamespaces: ["kube-system"]
      processes: ["*"]
YAML
복사

dev ns의 latest tag image 생성 금지

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8sdisallowlatesttag
spec:
  crd:
    spec:
      names:
        kind: K8sDisallowLatestTag
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8sdisallowlatesttag

        violation[{"msg": msg}] {
          input.review.kind.kind == "Deployment"
          container := input.review.object.spec.template.spec.containers[_]
          endswith(container.image, ":latest")
          msg := sprintf("image '%v' uses the ':latest' tag, which is not allowed", [container.image])
        }
YAML
복사

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sDisallowLatestTag
metadata:
  name: disallow-latest-tag-in-dev
spec:
  match:
    kinds:
      - apiGroups: ["apps"]
        kinds: ["Deployment"]
    namespaces:
      - dev
YAML
복사

dev namespace의 모든 pod에 label 자동 삽입

apiVersion: mutations.gatekeeper.sh/v1alpha1
kind: AssignMetadata
metadata:
  name: add-gk-label-to-dev-pods
  namespace: gatekeeper-system
spec:
  match:
    scope: Namespaced
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    namespaces: ["dev"]
  location: "metadata.labels.controlled-by" # <--- controlled-by: gatekeeper
  parameters:
    assign:
      value: "gatekeeper"
YAML
복사