missing kube-apiserver and other pods

[ankur6ue]

I'm trying to create a kubernetes cluster on AWS using:

kops create cluster --zones=us-east-1a --name=dev.k8s.local --state=s3://<bucket_name> --discovery-store=s3://<bucket_name>/${NAME}/discovery --node-size t2.small --master-size t2.small --node-count 1

However the cluster never gets in a healthy state.

Output of kops validate

output of kubectl get pods -n kube-system

Please help.. using kops version 1.31.0

[hakman]

@ankur6ue t2.small is too small for control plane. Use t2.medium instead or some other instance type with 4GB or memory..

[ankur6ue]

Thanks for the reply. I used t2.medium for the control plane, but get the same as before:

[ankur6ue]

any ideas?

[hakman]

try adding --dns=none when creating the cluster.

[ankur6ue]

Exactly the same as before
kops create cluster --zones=us-east-1a --cloud=aws --name=dev.k8s.local --state=s3://ankur-kops-k8s-state-store --discovery-store=s3://ankur-kops-k8s-oidc-store/${NAME}/discovery --node-size t2.medium --master-size t2.small --node-count 1 --dns=none

I verified the AWS resources (EC2 instances, security groups etc) are created as expected. Also ran journalctl -u kops-configuration.service. on the control plane node, and nodeup seems to be succeeding.

Any other ideas? I'm following the instructions on the deploying to AWS page to the letter.. this used to work when I last tried 2 years ago. Using latest version of kops, kubernetes etc.

[ankur6ue]

This is the output of kubectl get pods -n kube-system

[ankur6ue]

In the above, I was still using a small size node as the master node. I corrected that error, but the results are the same:

[rifelpet]

Can you run kubect describe pod on each pod that is stuck in pending? the Events at the bottom of the output should reveal why they aren't being scheduled.

[ankur6ue]

any ideas/suggestions on the above?

[ankur6ue]

I ran describe pod on the pending pods, the outputs are:
coredns-autoscaler pod:

coredns pod:

ebs-csi-controller pod

so these pending pods are waiting for cloud-controller-manager pod.. and that's in a crashloop, with the following describe pod output:

[ankur6ue]

The logs on the cloud-controller-manager pod are:

so appears to be some permissions issue?

[ankur6ue]

I'm running kops as IAM user kops, which is part of IAM group kops, as suggested in the instructions. The IAM permissions for the group and the user appear to be correct

[ankur6ue]

Also checked the OIDC bucket and there is a jwks file in discovery//openid/v1. I'm not an expert in OIDC, so not sure what more I should look for.. please help!

[rifelpet]

Is the OIDC bucket publicly accessible? You should be able to curl the JWKS object's HTTPS URL without authentication.

For example: curl https://$bucket.s3.$region.amazonaws.com/$prefix/openid/v1/jwks

[ankur6ue]

Thanks for the suggestion, tried it and it is publicly accessible. Returns:

{
"keys": [
{
"use": "sig",
"kty": "RSA",
"kid":

etc
any other ideas?