Setting up A\B split between services in Kubernetes using Nginx Ingress Canary feature

 I have been wanting to setup an A\B split with specific weight ratio between two services in my cluster, but had no clue how to achieve that. I am aware of the fact that if I were to use almost any Service mesh such as Istio, Linkerd, Consul, and etc... then this would be something that is supported out of the box, but how about when I don't have Service mesh?

Well, I did a little bit of research and at the first sight I didn't find much clue on how this can be achieved. Take a look at below image to understand what I am trying to achieve here:

As you can see, I have my Ingress Controller with two services, what I am trying to achieve is that when my nginx controller get the request is routes it 70% of the time to Service A, and 30% of the time to Service B.

After digging a little further I found the answer lies in the nginx official document here. it's a annotation called Canary that enables routing between services using either of the following conditions:

• Header: you can use this option when you want to route the request to specific service based on the some values in the header request. 

• Cookie: Use this option if you want to route the request based on some property in the cookie.

• Weight: This is the one I was looking for. It route the request based on the weight, A weight of 0 implies that no requests will be sent to the service in the Canary ingress by this canary rule. A weight of 100 means implies all requests will be sent to the alternative service specified in the Ingress.

To verify this, I created two Services, one return the word "Green" and the other returns "RED". 

Here is the Deployment and Services I created for "RED" Service:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: aks-helloworld-one  
spec:
  replicas: 1
  selector:
    matchLabels:
      app: aks-helloworld-one
  template:
    metadata:
      labels:
        app: aks-helloworld-one
    spec:
      containers:
      - name: aks-helloworld-one
        image: mcr.microsoft.com/azuredocs/aks-helloworld:v1
        ports:
        - containerPort: 80
        env:
        - name: TITLE
          value: "RED"
---
apiVersion: v1
kind: Service
metadata:
  name: aks-helloworld-one  
spec:
  type: ClusterIP
  ports:
  - port: 80
  selector:
    app: aks-helloworld-one

The "Green" services is: 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: aks-helloworld-two  
spec:
  replicas: 1
  selector:
    matchLabels:
      app: aks-helloworld-two
  template:
    metadata:
      labels:
        app: aks-helloworld-two
    spec:
      containers:
      - name: aks-helloworld-two
        image: mcr.microsoft.com/azuredocs/aks-helloworld:v1
        ports:
        - containerPort: 80
        env:
        - name: TITLE
          value: "GREEN"
---
apiVersion: v1
kind: Service
metadata:
  name: aks-helloworld-two  
spec:
  type: ClusterIP
  ports:
  - port: 80
  selector:
    app: aks-helloworld-two

Setting up the ingress part was the tricky part for me and it took me a while to figure out how exactly it works as it's a little bit different than the regular ingress controller. For the Canary to work you need to create two ingress file with same host name, the only difference between them would be that one of the file will carry the nginx canary annotation which are: 

nginx.ingress.kubernetes.io/canary: "true"

nginx.ingress.kubernetes.io/canary-weight: "20"

Here is the first ingress file I created: 

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: hello-world-ingress
  namespace: ingress-basic
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  rules:
  - host: myapp.westeurope.cloudapp.azure.com
    http:
      paths:
      - backend:
          serviceName: aks-helloworld-one
          servicePort: 80
        path: /(.*)

And the second file with extra annotation is:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: hello-world-ingress2
  namespace: ingress-basic
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/canary: "true" 
    nginx.ingress.kubernetes.io/canary-weight: "50" 
spec:
  rules:
  - host: myapp.westeurope.cloudapp.azure.com
    http:
      paths:
      - backend:
          serviceName: aks-helloworld-two
          servicePort: 80
        path: /(.*)

As you can see in the second ingress definition I have put the weight to 50% which means the second service will be hit for every other request, you can adjust this number according to your needs. Here are the screen-shot of the outcome: