Ingresses Controllers on k3s

Ingresses Controllers on k3s

An ingress controller is a special loadbalancer on kubernetes. It accepts traffic from outside kubernetes (k8s) and routes it to resources inside k8s.

By default kubernetes installations don’t come with an ingress controller. K3s however, does come with Traefik by default.

Traefik is an open sourced edge router, it automatically discovers the configuration (based on labels and ingress definitions) and routes as requested. However, after trying out the default Traefik installation, I ran into a series of noob problems that soured my experince:

  • I struggled to reach the default dashboard as I was using the URL xyz.com/dashboard instead of xyz.com/dashboard/. The trailing / was needed to trigger the router. I’m sure there’s some way to disable or change this, but as someone new to Traefik, it was annoying.
  • I wanted to expose a self-hosted homepage called Flame. The route was discovered as expected, the setup looks great but I kept hitting a blank page. After some curl fu, it seems like there were some header shenanigans going on that prevented JS from loading. I was able to find a few issues online hitting the same issue, however most solutions seem to use Nginx behind Traefik or using some Middleware service to fix the headers before routing. Again, extremely overwhelming for a noob to Traefik.

Having used Nginx and HAProxy extensively in the past, I didn’t see a real reason to stick with Traefik and its potential unknowns any longer. So I got rid of my existing cluster with the default Traefik (an easy action thanks to pulumi) and setup a new cluster with the flag --disable=traefik during the k3s server setup process. This would setup k3s without a default ingress controller.

Install Nginx as ingress controller

Once the new k3s cluster is setup, check again that there indeed is no ingress controller. Once checked, we’d want to go to the Nginx bare metal setup docs found here.

There’s a simple in the above URL, fetch it and save it on a local file called ingress-controller.yaml. kubectl apply -f said file to specify all the required components. To make it a bit more transparent, I fetched the file and saved it in my local git repo to be a bit more explicit as to what is deployed on the cluster.

We still need a entrypoint, so lets depoy one:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx-controller-loadbalancer
  namespace: ingress-nginx
spec:
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: 80
    - name: https
      port: 443
      protocol: TCP
      targetPort: 443
  type: LoadBalancer

Testing nginx ingress controller

Lets create a new namespace called nginx-test with kubectl create namespace nginx-test. We’ll deploy all the required resources for the test in this namespace.

Now we will attempt to deploy a Deployment called test-nginx-dep, this will deploy a Pod called nginx-backend and a Service called test-nginx-svc. We’ll also need to deploy a Ingressresource called test-nginx-ingress, this will specify the host to listen on and the path as well as the backend to route the request to. The IngressController will then route the request specified by the Ingress resource.

Save the below YAML as a file called example.yaml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-nginx-dep
  namespace: nginx-test
spec:
  selector:
    matchLabels:
      name: test-nginx-backend
  template:
    metadata:
      labels:
        name: test-nginx-backend
    spec:
      containers:
        - name: nginx-backend
          image: docker.io/nginx:alpine
          imagePullPolicy: Always
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: test-nginx-svc
  namespace: nginx-test
spec:
  ports:
    - name: http
      port: 81
      protocol: TCP
      targetPort: 80
  selector:
    name: test-nginx-backend
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-nginx-ingress
  namespace: nginx-test
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
  ingressClassName: "nginx"
  rules:
  - host: xyz.com
    http:
      paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: test-nginx-svc
              port:
                number: 81

Note: Ensure that the URL specified in host actually routes to your k3s cluster. I achieved this by setting a DNS rule on my router to resolve xyz.com to my cluster. The same can be achieved by modifying /etc/hosts on a linux machine.

Apply the file with: kubectl apply -n nginx-test -f example.yaml. This should create all the resouces required. You can now navigate to xyz.com on your machine to see the Nginx default landing page - indicating that the nginx IngressController picked up the Ingress route and passes traffic as expected.

Conclusion

We switched the k3s cluster from the default Traefik to a Nginx ingress controller and we will now use Nginx for exposing future services as well as configuring SSL certificates.