eksctl で何ができるのか

eksctl が何をやってくれるのが、何ができるのかを確認します。

eksctl create cluster

いきなり eksctl create cluster を実行するだけでクラスタが作れるっぽいのでひとまず試してみる。

$ eksctl create cluster
[ℹ]  eksctl version 0.12.0
[ℹ]  using region ap-northeast-1
[ℹ]  setting availability zones to [ap-northeast-1c ap-northeast-1d ap-northeast-1b]
[ℹ]  subnets for ap-northeast-1c - public:192.168.0.0/19 private:192.168.96.0/19
[ℹ]  subnets for ap-northeast-1d - public:192.168.32.0/19 private:192.168.128.0/19
[ℹ]  subnets for ap-northeast-1b - public:192.168.64.0/19 private:192.168.160.0/19
[ℹ]  nodegroup "ng-bb178f10" will use "ami-07296175bc6b826a5" [AmazonLinux2/1.14]
[ℹ]  using Kubernetes version 1.14
[ℹ]  creating EKS cluster "beautiful-badger-1578666058" in "ap-northeast-1" region with un-managed nodes
[ℹ]  will create 2 separate CloudFormation stacks for cluster itself and the initial nodegroup
[ℹ]  if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=ap-northeast-1 --cluster=beautiful-badger-1578666058'
[ℹ]  CloudWatch logging will not be enabled for cluster "beautiful-badger-1578666058" in "ap-northeast-1"
[ℹ]  you can enable it with 'eksctl utils update-cluster-logging --region=ap-northeast-1 --cluster=beautiful-badger-1578666058'
[ℹ]  Kubernetes API endpoint access will use default of {publicAccess=true, privateAccess=false} for cluster "beautiful-badger-1578666058" in "ap-northeast-1"
[ℹ]  2 sequential tasks: { create cluster control plane "beautiful-badger-1578666058", create nodegroup "ng-bb178f10" }
[ℹ]  building cluster stack "eksctl-beautiful-badger-1578666058-cluster"
[ℹ]  deploying stack "eksctl-beautiful-badger-1578666058-cluster"

VPC から一式全部作る CloudFormation の stack (ここでは eksctl-beautiful-badger-1578666058-cluster という名前) が作成されて deploy が開始されました。AWS Console で CloudFormation の stack を確認すれば状況がわかります。ありゃ？エラーが…

[✖]  unexpected status "ROLLBACK_IN_PROGRESS" while waiting for CloudFormation stack "eksctl-beautiful-badger-1578666058-nodegroup-ng-bb178f10"
[ℹ]  fetching stack events in attempt to troubleshoot the root cause of the failure
[!]  AWS::EC2::SecurityGroupEgress/EgressInterClusterAPI: DELETE_IN_PROGRESS
[!]  AWS::EC2::SecurityGroupIngress/IngressInterClusterAPI: DELETE_IN_PROGRESS
[!]  AWS::EC2::SecurityGroupIngress/IngressInterClusterCP: DELETE_IN_PROGRESS
[!]  AWS::EC2::SecurityGroupEgress/EgressInterCluster: DELETE_IN_PROGRESS
[!]  AWS::AutoScaling::AutoScalingGroup/NodeGroup: DELETE_IN_PROGRESS
[!]  AWS::EC2::SecurityGroupIngress/IngressInterCluster: DELETE_IN_PROGRESS
[✖]  AWS::AutoScaling::AutoScalingGroup/NodeGroup: CREATE_FAILED – "AWS was not able to validate the provided access credentials (Service: AmazonAutoScaling; Status Code: 400; Error Code: ValidationError; Request ID: a31c5a54-33b6-11ea-9896-5308bff4139a)"
[ℹ]  1 error(s) occurred and cluster hasn't been created properly, you may wish to check CloudFormation console
[ℹ]  to cleanup resources, run 'eksctl delete cluster --region=ap-northeast-1 --name=beautiful-badger-1578666058'
[✖]  waiting for CloudFormation stack "eksctl-beautiful-badger-1578666058-nodegroup-ng-bb178f10": ResourceNotReady: failed waiting for successful resource state
Error: failed to create cluster "beautiful-badger-1578666058"

NodeGroup の作成でこけたらしいことは分かったが… 😣

AWS was not able to validate the provided access credentials (Service: AmazonAutoScaling; Status Code: 400; Error Code: ValidationError; Request ID: a31c5a54-33b6-11ea-9896-5308bff4139a)

再チャレンジしたら今度は別のところでコケた…

[✖]  AWS::EC2::Subnet/SubnetPublicAPNORTHEAST1A: CREATE_FAILED – "Value (ap-northeast-1a) for parameter availabilityZone is invalid. Subnets can currently only be created in the following availability zones: ap-northeast-1d, ap-northeast-1c, ap-northeast-1b. (Service: AmazonEC2; Status Code: 400; Error Code: InvalidParameterValue; Request ID: 28f3bfed-2939-4efd-ad9c-7f6152d6b041)"
[✖]  AWS::EC2::Subnet/SubnetPrivateAPNORTHEAST1A: CREATE_FAILED – "Value (ap-northeast-1a) for parameter availabilityZone is invalid. Subnets can currently only be created in the following availability zones: ap-northeast-1b, ap-northeast-1c, ap-northeast-1d. (Service: AmazonEC2; Status Code: 400; Error Code: InvalidParameterValue; Request ID: 0e402395-2b61-4b7f-ac5b-6e9f01625f0c)"

この後、数回試したが、全部この AZ の問題でコケた…

それでもあと1回、あと1回と思って試してたら成功した。(eksctl create cluster –zones=ap-northeast-1b,ap-northeast-1c,ap-northeast-1d と –zones を指定すれば良いのかな) しかし、AZ の問題はわかるんだけど、初回のエラーは必ず再発するのかと思ってたので意外

$ eksctl create cluster                                                           
[ℹ]  eksctl version 0.12.0
[ℹ]  using region ap-northeast-1
[ℹ]  setting availability zones to [ap-northeast-1c ap-northeast-1d ap-northeast-1b]
[ℹ]  subnets for ap-northeast-1c - public:192.168.0.0/19 private:192.168.96.0/19
[ℹ]  subnets for ap-northeast-1d - public:192.168.32.0/19 private:192.168.128.0/19
[ℹ]  subnets for ap-northeast-1b - public:192.168.64.0/19 private:192.168.160.0/19
[ℹ]  nodegroup "ng-c384e850" will use "ami-07296175bc6b826a5" [AmazonLinux2/1.14]
[ℹ]  using Kubernetes version 1.14
[ℹ]  creating EKS cluster "wonderful-painting-1578669435" in "ap-northeast-1" region with un-managed nodes
[ℹ]  will create 2 separate CloudFormation stacks for cluster itself and the initial nodegroup
[ℹ]  if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=ap-northeast-1 --cluster=wonderful-painting-1578669435'
[ℹ]  CloudWatch logging will not be enabled for cluster "wonderful-painting-1578669435" in "ap-northeast-1"
[ℹ]  you can enable it with 'eksctl utils update-cluster-logging --region=ap-northeast-1 --cluster=wonderful-painting-1578669435'
[ℹ]  Kubernetes API endpoint access will use default of {publicAccess=true, privateAccess=false} for cluster "wonderful-painting-1578669435" in "ap-northeast-1"
[ℹ]  2 sequential tasks: { create cluster control plane "wonderful-painting-1578669435", create nodegroup "ng-c384e850" }
[ℹ]  building cluster stack "eksctl-wonderful-painting-1578669435-cluster"
[ℹ]  deploying stack "eksctl-wonderful-painting-1578669435-cluster"
[ℹ]  building nodegroup stack "eksctl-wonderful-painting-1578669435-nodegroup-ng-c384e850"
[ℹ]  --nodes-min=2 was set automatically for nodegroup ng-c384e850
[ℹ]  --nodes-max=2 was set automatically for nodegroup ng-c384e850
[ℹ]  deploying stack "eksctl-wonderful-painting-1578669435-nodegroup-ng-c384e850"
[✔]  all EKS cluster resources for "wonderful-painting-1578669435" have been created
[✔]  saved kubeconfig as "/Users/teraoka/.kube/config"
[ℹ]  adding identity "arn:aws:iam::949160801735:role/eksctl-wonderful-painting-1578669-NodeInstanceRole-18QXALFB6Y0W2" to auth ConfigMap
[ℹ]  nodegroup "ng-c384e850" has 0 node(s)
[ℹ]  waiting for at least 2 node(s) to become ready in "ng-c384e850"
[ℹ]  nodegroup "ng-c384e850" has 2 node(s)
[ℹ]  node "ip-192-168-12-245.ap-northeast-1.compute.internal" is ready
[ℹ]  node "ip-192-168-94-158.ap-northeast-1.compute.internal" is ready
[ℹ]  kubectl command should work with "/Users/teraoka/.kube/config", try 'kubectl get nodes'
[✔]  EKS cluster "wonderful-painting-1578669435" in "ap-northeast-1" region is ready

~/.kube/config に保存したよって出てるから kubectl コマンドを試してみる。

$ kubectl get nodes
NAME                                                STATUS   ROLES    AGE     VERSION
ip-192-168-12-245.ap-northeast-1.compute.internal   Ready       3m25s   v1.14.8-eks-b8860f
ip-192-168-94-158.ap-northeast-1.compute.internal   Ready       3m24s   v1.14.8-eks-b8860f

$ kubectl get svc --all-namespaces
NAMESPACE     NAME         TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)         AGE
default       kubernetes   ClusterIP   10.100.0.1            443/TCP         16m
kube-system   kube-dns     ClusterIP   10.100.0.10           53/UDP,53/TCP   16m

$ kubectl get ns 
NAME              STATUS   AGE
default           Active   17m
kube-node-lease   Active   17m
kube-public       Active   17m
kube-system       Active   17m

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                       READY   STATUS    RESTARTS   AGE
kube-system   aws-node-b74p7             1/1     Running   0          11m
kube-system   aws-node-xx89x             1/1     Running   0          11m
kube-system   coredns-58986cd576-nc45g   1/1     Running   0          17m
kube-system   coredns-58986cd576-wcxcj   1/1     Running   0          17m
kube-system   kube-proxy-cfqcd           1/1     Running   0          11m
kube-system   kube-proxy-kzlgz           1/1     Running   0          11m

動いてる 😍

さて、どんなリソースが作られているのか。

VPC: VPC から新しく作られる
Subnet: Public と Private という2種類の Subnet がそれぞれ3つの Availability zone に作成される
Internet Gateway: 新しい VPC を作って Public subnet を作成するので当然 Internet Gateway も必要になる
Role: ServiceRole (Controle Plane 用)、NodeInstanceRole (EC2 Workder Node 用)、FargatePodExecutionRole の3つが作成される。ServiceRole は EKS のドキュメントにある AmazonEKSClusterPolicy、AmazonEKSServicePolicy と eksctl が作る PutMetrics 用 Policy と NLB 管理用の Policy が紐付けられている。NodeInstanceRole は EKS のドキュメントにある AmazonEKSWorkerNodePolicy、AmazonEC2ContainerRegistryReadOnly、AmazonEKS_CNI_Policy の3つが、FargatePodExecutionRole には AmazonEKSFargatePodExecutionRolePolicy が紐付けられている
NAT Gateway: Private subnet があるので NAT Gateway も作成される。デフォルトでは1つの Availability zone にしか作られないが、–vpc-nat-mode で HighlyAvailable, Single, Disable から選択して指定することが可能。デフォルトは Single
SecurityGroup: クラスタ内の node 間での通信用に作成される
ControlePlane: もちろん Kubernetes の Controle Plane が作られる
AutoScalingGroup: Worker node 用の AutoScalingGroup が作られる
LaunchTemplate: Worker node 用の AutoScalingGroup で使われる LaunchTemplate で AMI Image や InstanceType、InstanceProflie、Userdata が設定されている

Worker node として EC2 インスタンスが作成されたらログインしてみたいはず。でも eksctl create cluster に –ssh-access (ファイルを指定するなら –ssh-public-key も) をつけておかないと Public Key が設定されないため worker node へのアクセスには EC2 Instance Connect で接続する必要がある。aws-instance-connect-cli を使えば mssh <instance-id> だけで接続できる。しかし Instance Connect は Instance Metadata Service v1 (IDMSv1) しかサポートしていないらしい。接続先インスタンスは Global IP address を持っている必要があり、SSH (22/tcp) が SecurityGroup で許可されている必要がある。Web Console から AWS のアドレスからの接続を許可しておく必要がある。また、接続する人は IAM Policy で ec2-instance-connect:SendSSHPublicKey 権限を持っている必要がある。公開鍵を送りつけて一時的にログインできるようにしてくれるんですね。

ところで、worker node にログインしてみたら Pod ごとに2つのコンテナが起動していました。複数コンテナが指定された Pod じゃないのになんでだろ？って思ったらどれも “/pause” が実行されてるものと、名前に合ったプログラムが実行されているコンテナの2つでした。

CONTAINER ID        IMAGE                                                                   COMMAND                  CREATED             STATUS              PORTS               NAMES
df061216904c        602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/eks/coredns           "/coredns -conf /etc…"   2 hours ago         Up 2 hours                              k8s_coredns_coredns-58986cd576-6kl57_kube-system_00aec73c-3449-11ea-a7e3-069c17bfdc00_0
379bc8e2844a        602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/eks/pause-amd64:3.1   "/pause"                 2 hours ago         Up 2 hours                              k8s_POD_coredns-58986cd576-6kl57_kube-system_00aec73c-3449-11ea-a7e3-069c17bfdc00_0
09ee64a82565        602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/amazon-k8s-cni        "/bin/sh -c /app/ins…"   2 hours ago         Up 2 hours                              k8s_aws-node_aws-node-7l99l_kube-system_6f2001d2-344e-11ea-a7e3-069c17bfdc00_0
79eef819a1f4        602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/eks/kube-proxy        "kube-proxy --v=2 --…"   2 hours ago         Up 2 hours                              k8s_kube-proxy_kube-proxy-d599c_kube-system_6f202763-344e-11ea-a7e3-069c17bfdc00_0
40ae949a2e0c        602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/eks/pause-amd64:3.1   "/pause"                 2 hours ago         Up 2 hours                              k8s_POD_aws-node-7l99l_kube-system_6f2001d2-344e-11ea-a7e3-069c17bfdc00_0
eeed9d5bf85f        602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/eks/pause-amd64:3.1   "/pause"                 2 hours ago         Up 2 hours                              k8s_POD_kube-proxy-d599c_kube-system_6f202763-344e-11ea-a7e3-069c17bfdc00_0

The Almighty Pause Container ってのを見つけた。これが Pod という複数コンテナをまとめるキモだったんですね。

Worker node は起動時に Kubernetes のクラスタに参加する必要があるが、これは AutoScalingGroup の LaunchTemplate に userdata 設定があります。テキストが gzip されて base64 エンコードされていました、デコードすると次の内容でした。

userdata

#cloud-config
packages: null
runcmd:
- - /var/lib/cloud/scripts/per-instance/bootstrap.al2.sh
write_files:
- content: |
    # eksctl-specific systemd drop-in unit for kubelet, for Amazon Linux 2 (AL2)

    [Service]
    # Local metadata parameters: REGION, AWS_DEFAULT_REGION
    EnvironmentFile=/etc/eksctl/metadata.env
    # Global and static parameters: CLUSTER_DNS, NODE_LABELS, NODE_TAINTS
    EnvironmentFile=/etc/eksctl/kubelet.env
    # Local non-static parameters: NODE_IP, INSTANCE_ID
    EnvironmentFile=/etc/eksctl/kubelet.local.env

    ExecStart=
    ExecStart=/usr/bin/kubelet \
      --node-ip=${NODE_IP} \
      --node-labels=${NODE_LABELS},alpha.eksctl.io/instance-id=${INSTANCE_ID} \
      --max-pods=${MAX_PODS} \
      --register-node=true --register-with-taints=${NODE_TAINTS} \
      --allow-privileged=true \
      --cloud-provider=aws \
      --container-runtime=docker \
      --network-plugin=cni \
      --cni-bin-dir=/opt/cni/bin \
      --cni-conf-dir=/etc/cni/net.d \
      --pod-infra-container-image=${AWS_EKS_ECR_ACCOUNT}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/eks/pause-amd64:3.1 \
      --kubeconfig=/etc/eksctl/kubeconfig.yaml \
      --config=/etc/eksctl/kubelet.yaml
  owner: root:root
  path: /etc/systemd/system/kubelet.service.d/10-eksclt.al2.conf
  permissions: "0644"
- content: |-
    NODE_LABELS=alpha.eksctl.io/cluster-name=hilarious-hideout-1578729265,alpha.eksctl.io/nodegroup-name=ng-b50c30b5
    NODE_TAINTS=
  owner: root:root
  path: /etc/eksctl/kubelet.env
  permissions: "0644"
- content: |
    address: 0.0.0.0
    apiVersion: kubelet.config.k8s.io/v1beta1
    authentication:
      anonymous:
        enabled: false
      webhook:
        cacheTTL: 2m0s
        enabled: true
      x509:
        clientCAFile: /etc/eksctl/ca.crt
    authorization:
      mode: Webhook
      webhook:
        cacheAuthorizedTTL: 5m0s
        cacheUnauthorizedTTL: 30s
    cgroupDriver: cgroupfs
    clusterDNS:
    - 10.100.0.10
    clusterDomain: cluster.local
    featureGates:
      RotateKubeletServerCertificate: true
    kind: KubeletConfiguration
    serverTLSBootstrap: true
  owner: root:root
  path: /etc/eksctl/kubelet.yaml
  permissions: "0644"
- content: |
    -----BEGIN CERTIFICATE-----
    (省略)
    -----END CERTIFICATE-----
  owner: root:root
  path: /etc/eksctl/ca.crt
  permissions: "0644"
- content: |
    apiVersion: v1
    clusters:
    - cluster:
        certificate-authority: /etc/eksctl/ca.crt
        server: https://7950C8F5B4E86FAF920EB443279E8896.sk1.ap-northeast-1.eks.amazonaws.com
      name: hilarious-hideout-1578729265.ap-northeast-1.eksctl.io
    contexts:
    - context:
        cluster: hilarious-hideout-1578729265.ap-northeast-1.eksctl.io
        user: kubelet@hilarious-hideout-1578729265.ap-northeast-1.eksctl.io
      name: kubelet@hilarious-hideout-1578729265.ap-northeast-1.eksctl.io
    current-context: kubelet@hilarious-hideout-1578729265.ap-northeast-1.eksctl.io
    kind: Config
    preferences: {}
    users:
    - name: kubelet@hilarious-hideout-1578729265.ap-northeast-1.eksctl.io
      user:
        exec:
          apiVersion: client.authentication.k8s.io/v1alpha1
          args:
          - token
          - -i
          - hilarious-hideout-1578729265
          command: aws-iam-authenticator
          env: null
  owner: root:root
  path: /etc/eksctl/kubeconfig.yaml
  permissions: "0644"
- content: |
    m5.24xlarge 737
    g4dn.4xlarge 29
    i3.xlarge 58
    ... (省略) ...
    t2.nano 4
    c3.xlarge 58
    c5.large 29
  owner: root:root
  path: /etc/eksctl/max_pods.map
  permissions: "0644"
- content: |-
    AWS_DEFAULT_REGION=ap-northeast-1
    AWS_EKS_CLUSTER_NAME=hilarious-hideout-1578729265
    AWS_EKS_ENDPOINT=https://7950C8F5B4E86FAF920EB443279E8896.sk1.ap-northeast-1.eks.amazonaws.com
    AWS_EKS_ECR_ACCOUNT=602401143452
  owner: root:root
  path: /etc/eksctl/metadata.env
  permissions: "0644"
- content: |
    #!/bin/bash

    set -o errexit
    set -o pipefail
    set -o nounset

    function get_max_pods() {
      while read instance_type pods; do
        if  [[ "${instance_type}" == "${1}" ]] && [[ "${pods}" =~ ^[0-9]+$ ]] ; then
          echo ${pods}
          return
        fi
      done < /etc/eksctl/max_pods.map
    }

    NODE_IP="$(curl --silent http://169.254.169.254/latest/meta-data/local-ipv4)"
    INSTANCE_ID="$(curl --silent http://169.254.169.254/latest/meta-data/instance-id)"
    INSTANCE_TYPE="$(curl --silent http://169.254.169.254/latest/meta-data/instance-type)"

    source /etc/eksctl/kubelet.env # this can override MAX_PODS

    cat > /etc/eksctl/kubelet.local.env <<EOF
    NODE_IP=${NODE_IP}
    INSTANCE_ID=${INSTANCE_ID}
    INSTANCE_TYPE=${INSTANCE_TYPE}
    MAX_PODS=${MAX_PODS:-$(get_max_pods "${INSTANCE_TYPE}")}
    EOF

    systemctl daemon-reload
    systemctl enable kubelet
    systemctl start kubelet
  owner: root:root
  path: /var/lib/cloud/scripts/per-instance/bootstrap.al2.sh
  permissions: "0755"

でも今は Managed Node Group ってのがあるのでこれを使うのが良いのかな。eksctl create cluster に –managed をつけるだけで良さそう。また、–fargate をつければ Fargate で実行する設定が入るっぽい。この辺りを確認して terraform で EKS 環境を構築できるようにしてみることにする。