Consul Service-mesh on Kubernetes with Nginx Ingress: Field notes

Recently I had to deploy Consul on a Kubernetes cluster to implement mTLS for existing services.
The need was to ensure that all traffic inside the cluster for the users application to be encrypted, from the ingress to the application.

This article is not a step-by-step guide (although i will cover the step I took), nor a in-depth description of the final setup, but a collection of configurations and yamls I used, as well as issues I encountered on the way.
The primary goal of this article is to help you if you are in trouble, and are trying to configure the Consul service-mesh; with integration of the nginx-ingress.

I followed the guide on the official documentation,on how to install the Consul Service-Mesh on Kubernetes, using helm.
The official documentation if well written, and should help you on most of the issues you might encounter.

Please note that at the time of writing, the official helm chart for consul is 0.49.0 and the deployed version of consul is the 1.13.9, and theses version where used in the deployment.
The Kubernetes version for the deployment was 1.23.9 and the Nginx Ingress-controller was 1.3.0, and was deployed using node-port (bur the node-port part has, as far as I know, no real implications).

Before trying to deploy consul in a Kubernetes Cluster, be aware of the following points:

• If you add in your service mesh, two or more services with the same name, but in different namespace, consul will route incoming traffic on all the service bearing that name
• As for now, consul do not support multiple port on a service.
• If you have a pod, that only need to consume other service in the service mesh, you still need to create a service for it (create an headless service), as the envoy proxy that is created by consul need to bind to a service, and the envoy proxy is used for outbound connections

I proceeded in three main steps:
1 — Install the consul Helm Chart.
2 — Add the nginx ingress-controller in the Service-mesh.
3 — Add all the services in the service mesh.

After installing the Consul Helm chart following the official instructions (no special values where passed other that the name of the consul cluster): https://www.consul.io/docs/k8s/installation/install#install-consul

The commands to add the service mesh in the cluster using helm are :

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update
helm install consul hashicorp/consul -n consul --create-namespace -f values.yaml

And the values I provided to in the values.yaml are shown below (basic values, only the name was modified depending on the environment):

connectInject:
	enabled: true
controller:
	enabled: true
global:
	name: consul

On a side note before continuing: I had issues with the consul cluster in an instable state, where no leader could be elected, because pods could not communicate between each-others rapidly enough, you can try to set bigger pod ressources if this is the case, and that you are sure the network latency do not come from your cluster.

After installing the consul, an made sure that all pod are running correctly (using a good old kubectl get pods -n consul or kgp for the connoisseurs), the nginx ingress-controller in the service-mesh.
If you do not do that, only the traffic between your service inside the service-mesh will be encrypted.
Implementing your ingress-controller in the service-mesh, is a good zero-trust practice.

The main point of attention on the ingress controller is that you need to exclude the inbound port, otherwise the setup won’t work, as the envoy-proxy of the consul service-mesh will receive request that are not from the service-mesh, and will throw an error.

So, I placed the following annotation on the nginx deployment:

• consul.hashicorp.com/connect-service: "ingress-nginx-controller" Name of the service on which the proxy will listen, will depend on the name of you helm deployment, you need to precise it, as the nginx-ingress helm deployment will create two service: one for the ingress and another for the admission-controller.
• consul.hashicorp.com/transparent-proxy-exclude-inbound-ports: "80,443,8443" port of the pod 80.443 are the http/s port and 8443 is the admission wehbhook, this annotation ensure that the incoming requests on the ingress are not managed by the sidecar-proxy.

Be sure to exclude the admission webhook port, wich is 8443 by default, otherwise, you won’t be able to create nginx custom ressources, like ingress, because an incorrect certificate error will be thrown, as the certificate that will be presented by the admission-controller service will be the mTLS one, and not the certificate of the admission-controller service.

Please note that, if you have a different setup (different port, name, or others specificity in your setup, you may need to add other annotations on the nginx deployment), refer to the official documentations for the annotations.
I recommend this Github Repo, if you have an nginx-ingress and want to configure consul, it is cited in the official documentation and is well maintained.

Finally you can add your applications on the service mesh !
To do this, add the following annotations on the pod (edit your deployment definitions):

• consul.hashicorp.com/connect-inject: "true" This one will activate the injection of the sidecar proxy of consul on your microservice.
• consul.hashicorp.com/connect-service-port: "80" This one will specify the port on which the sidecar proxy should listen, this is not mandatory, but if your pod has multiple port, this ensure that the sidecar proxy will listen on the right port

You also may need to add the annotation consul.hashicorp.com/connect-service-name on the pod, if you have multiple service on the pod.

Also, you can simply configure the consul helm deployment to activate the sidecar proxy on all pod in a given namespace, or cluster-wide (and exclude namespaces explicitly, also note that consul will not add the sidecar-proxy on the kube-system namespace), by doing so, you are not obligated to add the consul.hashicorp.com/connect-inject: "true"

Your services will now communicate using the service mesh, if you want to see graphically the state of your service mesh, you can port-forward the consul-ui that was deployed with your consul.

For more security, consul implemented the concept of intentions, that are basically Network Policy for your service mesh.