Azure CNI vs Kubenet, What are the differences between them and which one to use?

-

if you have ever deployed an AKS cluster, either through the Azure Portal or CLI, there is parameter that needs to be configured related to the networking part of AKS and that is to choose the networking model between Azure CNI and Kubenet. (by default it's Kubenet as of writing this post)

So what's this all about? in a nutshell this parameter is related to how Nodes and Pods gets their IP addresses assigned. It\s important to understand their difference before creating the cluster since you can\t change the networking model of your cluster once it\s deployed, and you defiantly don't want to re-create and re-configure your cluster again just because of this option.

Overview of Kubenet and a few things to keep in mind

  • This is the default options if you don't explicitly change it. With this option the cluster nodes gets their IP addresses from the VNET which your AKS is deployed to. This is not a big deal since the same thing happens if you go with the second option, AKA, CNI. BUT, in this mode the PODS gets their ip addresses from a logically different address space than your VNET, and Pods can access resources in the VNET via NAT which takes place through the Node where the Pods is running.
  •  It is also worth knowing that in this mode "Pods can't communicate directly with each other. Instead User Defined Routing (UDR) and IP forwarding is used for connectivity between pods across nodes." (taken from MS official docs)
  • Also, in this mode you can only have ONE AKS cluster in each Subnet.
  • There will be an additional latency between Pods communication due to extra hop. but the difference is sub-millisecond.
  • The default number of Pods per node is 110.
Kubenet overview, taken from Microsoft.com


Overview of Azure CNI Option and a few things to keep in mind

On the contrary, if you choose Azure CNI over Kubenet, your Pods gets their IP addresses directly from the Subnet, and can access all the resources in the subnet without the needs of NATing via the Cluster Node. in another word, in this mod, each Pod has full connectivity to the Vnet out of the box. 

Azure CNI Network Model, taken from Microsoft.com

Azure CNI is a more advanced option, and requires additional preparation by ensuring that your VNet and Subnet is large enough to accommodate the number of Nodes, and the number of Pods in each node.

I prefer not to write much about how you should plan your Vnet and Subnet sizing before creating your cluster, because that itself can be a great topic for another article, but rather I want to emphasize on two things:

  • Keep in mind that the maximum number of Pods per node is 30 by default. You can increase this number when creating the cluster, but you can NOT change the number once your cluster gets created.
  • Try to be safe that sorry by making sure that your Subnet is large enough not for your current workload but for months and possibly year to come. I myself once deployed an AKS cluster to a 


Which one to choose?

Now comes the question you might be pondering upon, and that is which one to choose? In short, I ALWAYS recommend Azure CNI for any production workload; despite the fact that it comes with additional complexity to begin with. Kubenet is a very basic network plugin that doesn't scale much nor it supports any advance network routing and traffic management. if you happen to run single node cluster then using Azure CNI would be an overkill, and you are better-off to go with Kubenet.

Comments

Post a Comment

Popular posts from this blog

AWS EKS vs Azure AKS - my thoughts and reflection after using both in Production

-
Image
I am lucky enough to work both with Azure AKS, and AWS EKS(EC2), and I decided to dedicate a post on my blog about the two. I have also to admit that this is the most opinionated post I have written so far. I tried to put more emphasises on their differences than to write about their similarities. Here are the three areas I comparing them:  AKS vs EKS from Cluster Management Standpoint,  AKS vs EKS from Networking Standpoint,  AKS vs EKS from Scalability Standpoint AKS vs EKS from Cluster Management Standpoint  Both AKS and EKS mange the master node (AKA Data plane) for you completely, and there isn't much difference I notice in that area between the two. However, Azure does not charge you anything for the Master node it manage and you only pay for the worker nodes, however, EKS charge you a fix price monthly for that Master node (around USD 70 dollar depending on the region) One feature I like in EKS is called Managed Node Groups . This feature automate and off...
Read more

Architecting Kubernetes for High Availability, Fault Tolerance and Business Continuity

-
Image
Kubernetes can take care of many things, and can solve many problems except the ones it doesn't know about such as region failure and  human errors. In this post I want to compare and contrast the differences between Single Cluster setup spread across multi Availability Zones that is very common vs Multi Cluster Setup Spread across different Region . Hopefully by then end of this post you have some clue about when to use which setup no matter which cloud provider you are using; be it AWS, Azure, or GCP. Single Cluster Setup: In this Setup the Kubernetes nodes and their storages are distributed across multiple Availability Zones (AZ). This model ensures the nodes are physically separated from each other and the outage in one of the AZ will not cause the entire cluster to go out of service. At the same time the communication between each node is via private connection and does not route over internet no matter which cloud provider you use. I took the following image wh...
Read more