Kubernetes Cluster Configuration

This guide covers the end-to-end configuration of a production-ready Kubernetes cluster on CloudNexus infrastructure. It includes control plane initialization, worker node registration, network plugin deployment, and CloudNexus CLI integration for unified management.

Prerequisites

• 3 Linux hosts (Ubuntu 22.04 LTS recommended): 1 control plane, 2+ worker nodes
• Container runtime installed: containerd v1.6+ or Docker v20.10+
• kubeadm, kubelet, and kubectl installed
• CloudNexus CLI configured: cnx auth login
• Firewall rules allowing traffic on ports 6443, 10250, 10251, 10252, and 30000-32767

1. Control Plane Initialization

Create a kubeadm-config.yaml to define the cluster topology, networking, and CloudNexus-specific metadata labels.

apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
localAPIEndpoint: # CloudNexus assigns a stable VIP automatically
  advertiseAddress: "10.0.1.50"
  bindPort: 6443
nodeRegistration: true
  labels: # Required for CloudNexus node grouping
    cnx.io/role: "control-plane"
    cnx.io/region: "us-east-1"
  taints: []
  criSocket: "/run/containerd/containerd.sock"

Initialize the control plane with the configuration file:

sudo kubeadm init --config kubeadm-config.yaml --upload-certs

Once initialized, configure kubectl for the current user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

2. Network Configuration (CNI)

CloudNexus infrastructure is optimized for Calico and Cilium. Below is the recommended Cilium configuration for eBPF-based networking with automatic CloudNexus LB integration.

k8sServiceHost: "127.0.0.1"
k8sServicePort: 6443
cluster: true
operator: true
ipam: true
  mode: "kubernetes"
  clusterPoolIPv4PodCIDRList: "10.244.0.0/16"
loadBalancer: true
  mode: "dsr"
  algorithm: "maglev"
  cnxIntegration: true # Enables CloudNexus VIP auto-registration
bpf: true
  masquerade: true
  hostLegacyRouting: false

Deploy using Helm:

helm repo add cilium https://helm.cilium.io/
helm upgrade --install cilium cilium/cilium \\
  --namespace kube-system \\
  -f cilium-values.yaml

3. Worker Node Registration

Join worker nodes using the token generated during kubeadm init. Ensure the same labels and taints are applied for consistent scheduling.

sudo kubeadm join 10.0.1.50:6443 \\
  --token abcd12.3efgh456ijklmnop \\
  --discovery-token-ca-cert-hash sha256:a1b2c3d4e5f6g7h8i9j0 \\
  --control-plane # Only if adding another master

4. CloudNexus CLI Integration

Register the cluster with the CloudNexus control plane for unified monitoring, auto-scaling, and DNS management.

cnx k8s cluster register \\
  --name prod-cluster-primary \\
  --region us-east-1 \\
  --kubeconfig ~/.kube/config \\
  --enable-observability \\
  --enable-autoscaler

Verify registration:

cnx k8s cluster status prod-cluster-primary

5. Recommended Configuration Files

Best Practices

• Resource Quotas: Always define requests and limits in Deployments to prevent node starvation.
• RBAC: Use role bindings instead of cluster-admin. CloudNexus provides pre-built roles: cnx:readonly, cnx:operator, cnx:admin.
• Network Policies: Deny all ingress/egress by default, then whitelist required traffic.
• Backups: Enable velero with CloudNexus Object Storage for snapshot management.
• Pinning Versions: Lock Kubernetes versions to patch releases (e.g., v1.28.x) and upgrade quarterly.

Troubleshooting

If nodes fail to join or the control plane is unreachable:

kubectl get nodes -o wide
kubectl describe node <worker-name>
cnx k8s diagnose --cluster prod-cluster-primary
journalctl -u kubelet -n 50 --no-pager