Recently, I tried to configure calico networking on a self-managed Kubernetes cluster on Azure. It did not work out of the box and many instructions on the internet did not work for me. In the following, I want to share my setup. To set up the network and VMs, I followed this tutorial. After installing the default configuration of calico, inter-node communication between pods did not work.
My working approach uses User-Defined-Routes (UDR) on Azure to route traffic from the different pod-subnets of each node. Be sure to only have one IP address assigned to each network interface! I installed the Azure CNI plugin before, which assigns pod IPs from the secondary IPs of the vnet that are assigned to the network interface. This caused problems in the IP detection in Calico, but it can be easily fixed by deleting the secondary IPs.
When bootstrapping Kubernetes, you should also set the pod subnet (CIDR) to avoid address overlap with the virtual network.
The default for kubeadm is
192.168.0.0/16 which indeed overlapped with my Azure network.
This is my kubeadm config:
To bootstrap the cluster, execute:
./kubeadm init --upload-certs --cri-socket "/run/containerd/containerd.sock" --config conf.yaml
Then, get the calico setup from here.
In the yaml set the pod CIDR:
To configure the inter-node pod communication, I followed these steps here.
Before applying I recommend to set up VXLAN instead of ipip mode (see below). By default, calico uses ipip tunneling for pod communication. A tunnel is a virtual network interface to connect subnets. This did not work in my Azure network. Instead, I configured VXLAN.
Traffic to the pod network (outside the node) should then not be routed to
calioctl and verify that there is only one ippool:
calicoctl get ippools
If not, delete the other pool:
calicoctl delete pool default-ipv4-ippool
You need to reboot, to force a reassignment of the pod IPs to the new network.
Then doublecheck that the the CIDR did not change:
Otherwise, you need to update the Azure routing table!
If you want to dig deeper into the different networking options in calico for Azure, I recommend this video: