The experience just showed me how much I dislike kubernetes again - its way too over-engineered, atleast for a homelab. While I did learn a lot from the experience, I prefer knowing the moving bits and pieces! However the experience also showed me what I was missing with the plain docker compose approach I had before! I appreciated having the ability to auto-create certificates, having git as my source of truth and not thinking about routing or monitoring!

Thinking about the requirements led to the following:

• Github as the source of truth, with a services and a core folder
• The core folder outside the scope of gitops, its the ingress and deployment manager (to avoid the bootstrap problem)
• The services contain all the configs that need deploying
• Github bot/action to track releases and auto-create PRs that can be automatically deployed on approval

With Claude code all the hype this year, I decided to see how nice it was for design exploration! I came up with a list of requirements and let claude interactively suggest options for me, once we narrowed down a list of options - I asked it to generate all the template configs so that I could visually see what was possible and not. Additionally I could setup small test examples and prod at them, refining my requirement list each time. This sort of iterative exploration would have taken days in the past, I didn’t have to deal with outdated READMEs or bespoke bugs!

The stack I ended up deploying (and liking!):

• Traefik to handle ingress and certificates. I avoided traefik in the past because the middleware concept wasn’t intuitive for me, but using examples from claude - I could finally grok it! My old default was HAProxy and at work we use nginx + proxy mangager, both now seem more cumbersome compared to traefik.
• Komodo for deployments. Komodo handles gitOps for docker compose files, it can track a folder in a github repository and deploy the changes. However, unlike Kubernetes, I can still SSH into the machine and make changes manually if I wish (the next run would overwrite it unless I upstream the changes). This lets me easily interact with my homelab and extend it with host binaries (during dev), without having to write a helm chart to even start. Komodo also has the concept of a core and a periphery, so I can theoretically have it running on multiple hosts if I want to expand my homelab. The config for Komodo is a simple TOML file, so the deployment orchestrator settings is also tracked in git.
• Renovate bot for PRs, deployed as a github action with scoped access and permissions. This allows me to automate away updates as komodo will pick them up after merging.
• PockedID for Passkey based OIDC. This is something I wanted to try out this year, Traefik integrates well with OIDC and so do most of the services I plan on using. I wanted to centralize my user management and use passkeys by default everywhere. I’m not sold on the idea of passkeys, but I must admit that they work better for me since moving to iPhone from a Samsung.

Exploring, iterating and debugging on the entire workflow wouldn’t have been possible without Claude! The last years, this exercise used to take 2-3d of toil, that’s down to an afternoon now. Claude isn’t bulletproof yet, it makes mistakes that are often hard to catch or would only show up later in time - but we’ve really come a long way with AI tooling!

Apply allows us to use osascript to create a notification. This can be a silent notification (display notification) or an “in your face” alert (display alert). I chose the in your face alert because I don’t want to miss it, lets see if i regret it during my first presentation when the alert goes off.

The one liner for the alert,

1

osascript -e 'display alert "20-20-20 Rule" message "Take a 20s break!"'

However, I want to also receive an audio cue. You can use afplay on mac to call a system sound. Putting the two together, gives you this script:

1
2
3
4
5

#!/bin/zsh
# script called eye-strain-reminder.sh
osascript -e 'display alert "20-20-20 Rule" message "Take a 20s break!"'; afplay /System/Library/Sounds/Funk.aiff
sleep 20
osascript -e 'display alert "20-20-20 Rule" message "20s done!"'; afplay /System/Library/Sounds/Funk.aiff

Now we have to wrap it inside a launch agent, this has to be saved in ~/Library/LaunchAgents/eye-strain-reminder.plist with the following content:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>eye-strain-reminder</string>

    <key>ProgramArguments</key>
    <array>
        <string>sh</string>
        <string><ABSOLUTE_PATH_TO>/eye-strain-reminder.sh</string>
    </array>

    <key>StartInterval</key>
    <integer>1200</integer>
</dict>
</plist>

Don’t forget to update the path to the absolute path of your script!

Now you need to load and start the agent:

1
2

launchctl load ~/Library/LaunchAgents/eye-strain-reminder.plist
launchctl start eye-strain-reminder.plist

You can always stop and unload the above plist file if you want to get rid of the job.

By default kubernetes installations don’t come with an ingress controller. K3s however, does come with Traefik by default.

Traefik is an open sourced edge router, it automatically discovers the configuration (based on labels and ingress definitions) and routes as requested. However, after trying out the default Traefik installation, I ran into a series of noob problems that soured my experince:

• I struggled to reach the default dashboard as I was using the URL xyz.com/dashboard instead of xyz.com/dashboard/. The trailing / was needed to trigger the router. I’m sure there’s some way to disable or change this, but as someone new to Traefik, it was annoying.
• I wanted to expose a self-hosted homepage called Flame. The route was discovered as expected, the setup looks great but I kept hitting a blank page. After some curl fu, it seems like there were some header shenanigans going on that prevented JS from loading. I was able to find a few issues online hitting the same issue, however most solutions seem to use Nginx behind Traefik or using some Middleware service to fix the headers before routing. Again, extremely overwhelming for a noob to Traefik.

Having used Nginx and HAProxy extensively in the past, I didn’t see a real reason to stick with Traefik and its potential unknowns any longer. So I got rid of my existing cluster with the default Traefik (an easy action thanks to pulumi) and setup a new cluster with the flag --disable=traefik during the k3s server setup process. This would setup k3s without a default ingress controller.

Install Nginx as ingress controller

Once the new k3s cluster is setup, check again that there indeed is no ingress controller. Once checked, we’d want to go to the Nginx bare metal setup docs found here.

There’s a simple in the above URL, fetch it and save it on a local file called ingress-controller.yaml. kubectl apply -f said file to specify all the required components. To make it a bit more transparent, I fetched the file and saved it in my local git repo to be a bit more explicit as to what is deployed on the cluster.

We still need a entrypoint, so lets depoy one:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx-controller-loadbalancer
  namespace: ingress-nginx
spec:
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: 80
    - name: https
      port: 443
      protocol: TCP
      targetPort: 443
  type: LoadBalancer

Testing nginx ingress controller

Lets create a new namespace called nginx-test with kubectl create namespace nginx-test. We’ll deploy all the required resources for the test in this namespace.

Now we will attempt to deploy a Deployment called test-nginx-dep, this will deploy a Pod called nginx-backend and a Service called test-nginx-svc. We’ll also need to deploy a Ingressresource called test-nginx-ingress, this will specify the host to listen on and the path as well as the backend to route the request to. The IngressController will then route the request specified by the Ingress resource.

Save the below YAML as a file called example.yaml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-nginx-dep
  namespace: nginx-test
spec:
  selector:
    matchLabels:
      name: test-nginx-backend
  template:
    metadata:
      labels:
        name: test-nginx-backend
    spec:
      containers:
        - name: nginx-backend
          image: docker.io/nginx:alpine
          imagePullPolicy: Always
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: test-nginx-svc
  namespace: nginx-test
spec:
  ports:
    - name: http
      port: 81
      protocol: TCP
      targetPort: 80
  selector:
    name: test-nginx-backend
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-nginx-ingress
  namespace: nginx-test
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
  ingressClassName: "nginx"
  rules:
  - host: xyz.com
    http:
      paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: test-nginx-svc
              port:
                number: 81

Note: Ensure that the URL specified in host actually routes to your k3s cluster. I achieved this by setting a DNS rule on my router to resolve xyz.com to my cluster. The same can be achieved by modifying /etc/hosts on a linux machine.

Apply the file with: kubectl apply -n nginx-test -f example.yaml. This should create all the resouces required. You can now navigate to xyz.com on your machine to see the Nginx default landing page - indicating that the nginx IngressController picked up the Ingress route and passes traffic as expected.

Conclusion

We switched the k3s cluster from the default Traefik to a Nginx ingress controller and we will now use Nginx for exposing future services as well as configuring SSL certificates.

For my usecase, I have a Promxox node (Minisforum 773 Lite, 64GB RAM, 1TB SSD) and would want to use it to host my VMs. Initially I want to run 3 VMs to host a kubernetes cluster that I can use to better learn the technology.

I’m assuming that Pulumi is already installed and configured. Let’s now create a new stack with pulumi new python -y. This will create a few new files for us: Pulumi.yaml(Project metadata), __main__.py(Program entrypoint), requirements.txt(dependencies), venv (virtual env for the project).

Proxmox has an API that allows us to provision resources, however we would need to know the calls to make. To simplify the process, we can use a pulumi proxmox provider, which can be installed as a basic pip package with venv/bin/pip install pulumi-proxmoxve (or install via the requirements.yaml). This will allow us to create a proxmox provider and to define the VM in an easier manner and the package takes care of the required API calls to provision the VM.

In order to make configuration of VMs a bit easier, lets also define a folder called vms that contains YAML files containing all the values that we’d want to configure. We can then read the files in said folder, parse the values and then configure the pulumi proxmox provider to create the VMs for us. 

To start with, lets configure .gitignore to ignore the .env file that will be containing our secrets. Then create a .env file with the following contents:

1
2
3
4
5

PROXMOX_ENDPOINT="https://<PROXMOX-IP>:8006"
PROXMOX_INSECURE="true"
PROXMOX_USERNAME="username@pam"
PROXMOX_PASSWORD="PASSWORD"
PROXMOX_USER_ACCOUNT_PASSWORD="DEBIAN_BASE_PASSWORD"

The pulumi python logic for `_main_.py` is the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129

import pulumi
import pulumi_proxmoxve as proxmox
import os,yaml
from dotenv import load_dotenv
import ipaddress
load_dotenv()
provider = proxmox.Provider('proxmoxve',
                            endpoint=os.getenv("PROXMOX_ENDPOINT"),
                            insecure=os.getenv("PROXMOX_INSECURE"),
                            username=os.getenv("PROXMOX_USERNAME"),
                            password=os.getenv("PROXMOX_PASSWORD"),
                            )

folder_path = "vms/"

def load_yaml_files_from_folder(folder_path):
    yaml_files = [file for file in os.listdir(folder_path) if file.endswith(".yaml")]
    loaded_data = []

    for yaml_file in yaml_files:
        file_path = os.path.join(folder_path, yaml_file)
        with open(file_path, 'r') as file:
            yaml_data = yaml.safe_load(file)
            loaded_data.append(yaml_data)

    return loaded_data

parsed_data = load_yaml_files_from_folder(folder_path)

for vm in parsed_data:
    disks = []
    nets = []
    ip_configs = []
    ssh_keys = []

    for v in vm:
        for vmcount in range(v['count']):
            base_resource_name=v['resource_name']
            name_counter = vmcount + 1
            base_vm_id=v['vm_id'],
            for disk_entry in v['disks']:
                for d in disk_entry:
                    disks.append(
                        proxmox.vm.VirtualMachineDiskArgs(
                            interface=disk_entry[d]['interface'],
                            datastore_id=disk_entry[d]['datastore_id'],
                            size=disk_entry[d]['size'],
                            file_format=disk_entry[d]['file_format'],
                            cache=disk_entry[d]['cache']
                        )
                    )

            for ip_config_entry in v['cloud_init']['ip_configs']:
                ipv4 = ip_config_entry.get('ipv4')

                if ipv4:
                    new_address = ''
                    ip, subnet = ipv4.get('address', '').split('/')
                    new_ip = str(ipaddress.ip_address(ip) + vmcount)
                    new_address = f"{new_ip}/{subnet}"

                    ip_configs = []
                    ip_configs.append(
                        proxmox.vm.VirtualMachineInitializationIpConfigArgs(
                            ipv4=proxmox.vm.VirtualMachineInitializationIpConfigIpv4Args(
                                address=new_address,
                                gateway=ipv4.get('gateway', '')
                            )
                        )
                    )

            for ssk_keys_entry in v['cloud_init']['user_account']['keys']:
                ssh_keys.append(ssk_keys_entry)

            for net_entry in v['network_devices']:
                for n in net_entry:
                    nets.append(
                        proxmox.vm.VirtualMachineNetworkDeviceArgs(
                            bridge=net_entry[n]['bridge'],
                            model=net_entry[n]['model']
                        )
                    )

            virtual_machine = proxmox.vm.VirtualMachine(
                vm_id=base_vm_id[0] + vmcount,
                resource_name=f"{base_resource_name}-{name_counter}",
                node_name=v['node_name'],
                agent=proxmox.vm.VirtualMachineAgentArgs(
                    enabled=v['agent']['enabled'],
                    # trim=v['agent']['trim'],
                    type=v['agent']['type']
                ),
                bios=v['bios'],
                cpu=proxmox.vm.VirtualMachineCpuArgs(
                    cores=v['cpu']['cores'],
                    sockets=v['cpu']['sockets']
                ),
                clone=proxmox.vm.VirtualMachineCloneArgs(
                    node_name=v['clone']['node_name'],
                    vm_id=v['clone']['vm_id'],
                    full=v['clone']['full'],
                ),
                disks=disks,
                memory=proxmox.vm.VirtualMachineMemoryArgs(
                    dedicated=v['memory']['dedicated']
                ),
                name=f"{base_resource_name}-{name_counter}",
                network_devices=nets,
                initialization=proxmox.vm.VirtualMachineInitializationArgs(
                    type=v['cloud_init']['type'],
                    datastore_id=v['cloud_init']['datastore_id'],
                    interface=v['cloud_init']['interface'],
                    dns=proxmox.vm.VirtualMachineInitializationDnsArgs(
                        domain=v['cloud_init']['dns']['domain'],
                        server=v['cloud_init']['dns']['server']
                    ),
                    ip_configs=ip_configs,
                    user_account=proxmox.vm.VirtualMachineInitializationUserAccountArgs(
                        username=v['cloud_init']['user_account']['username'],
                        password=os.getenv("PROXMOX_USER_ACCOUNT_PASSWORD"),
                        keys=ssh_keys
                    ),
                ),
                on_boot=v['on_boot'],
                reboot=v['on_boot'],
                opts=pulumi.ResourceOptions(provider=provider,ignore_changes=v['ignore_changes']),
            )

            pulumi.export(v['name'], virtual_machine.id)

The YAML for the proxmox VMs looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

---
- name: "k3s-nodes"
  count: 3
  node_name: "proxmox-vmfarm"
  resource_name: "k3s-node"
  vm_id: 1000
  agent:
    enabled: true
    type: "virtio"
  bios: "seabios"
  ignore_changes:
    - "disks"
    - "cdrom"
  cpu:
    cores: 4
    sockets: 1
  cloud_init:
    type: "nocloud"
    interface: "ide0"
    datastore_id: "local-lvm"
    dns:
      domain: ""
      server: "1.1.1.1 8.8.8.8"
    ip_configs:
      - ipv4:
          address: "192.168.90.50/32"
          gateway: "192.168.90.1"
    user_account:
      username: "root"
      keys:
       - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAiL82k0HKfSFp3JgdpShVvx9mRDk5fyYQyto00tBdeC

  clone:
    node_name: "proxmox-vmfarm"
    vm_id: 100
    full: true
  disks:
    - disk1:
        interface: "scsi0"
        datastore_id: "local-lvm"
        size: 180
        file_format: "raw"
        cache: "none"
  memory:
    dedicated: 16384
  network_devices:
    - net1:
        bridge: "vmbr0"
        model: "virtio"
  on_boot: true

The values being configured are quite self-explanatory, if not there is a detailed documentation in the proxmox provider page.

Once done, all you have to do is run pulumi up in the folder and the pulumi interactive shell will guide you through the rest!

Once the pulumi job is done, you should be able to see brand new VMs created in your proxmox host. If you want to change a value, you can simply modify the YAML and run pulumi up again. To destroy the VMs, you can run pulumi down or set the count to 0 to not affect the rest of the pulumi stack.

I looked into kubernetes the hard way, it was a great deep dive into the components - but I was looking for something a bit more lightweight. K3s seemed perfect for my usecase, it was: Lightweight, Easy to install (via ansible or k3sup) and had low ram requirements. Time to setup the VMs and get to the races!

Setting up VMs on my NAS

My home NAS is an 4 core CPU (i3-10100 CPU ) with 16GB of RAM (Crucial 2666Mhz) and 16TB of space running on a B460M motherboard with Unraid as my software of choice. My considerations at the time of purchase was that it should be low power as well as support Intel Quick Sync for Plex - So the setup isn’t optimized for heavy workloads. That being said, its already running 5-6 docker containers and 1 VMs without any issues - All at a relatively low power draw.

I uploaded the latest Debian 12 ISO on to the NAS images and went to the VM page to create a new VM. Unfortunately Unraid allows you to create just 1VM at a time (Some posts online indicate you can copy over the disk image to clone it) - So I just did the creation process 3 times manually. In the future, I should probably have a pre-provisioned image with user account and basic tools installed and use that as the base.

The 3 VMs worked great, I was able to SSH into all of them and ensured that they have a static IP. Now I had to decide how to install k3s on these VMs.

Setting up k3s

I tested out two approaches(k3sup and ansible), eventually settling on the Ansible based approach since I’m most comfortable with it. I’ll describe the k3sup approach and then the ansible approach later.

k3sup

k3sup is an extremely easy to use and one-command approach to setting up k3s. You can target a remote host by just specifying the IP, k3sup will execute the commands via SSH. Install k3sup via their github repo instructions before proceeding.

Kubernetes has a server-agent model, so its prudent to setup the server first and then join the agents to it. In order to setup the server, you can just run the command k3sup install --ip $IP --user $USER. This will already result in a kubeconfig file that you need for accessing the cluster in the current location from which you ran the k3sup command.

In order to setup the agents, simply run k3sup join --ip $AGENT_IP --server-ip $SERVER_IP --user $USER and k3sup will automagically install the required software and setup the cluster.

You can run kubectl --kubeconfig=$(pwd)/kubeconfig get nodes to check if the nodes are all present as expected.

k3sup was extremely easy to use, but the setup process felt a bit ephemeral to me. I’m sure you can also track which version of k3s the nodes run, but I still felt a bit uneasy that it wasn’t defined in a static manner. Most of the problems can also be fixed by a simple script or by allowing ansible to call k3sup instead - meaning easier integration with the rest of my homelab setup.

Ansible

Most of my homelab is already controlled by ansible, so it felt natural to try and rely on ansible atleast for the provisioning aspects of the homelab. The team I work with at the Ethereum Foundation maintains this massive list of ansible roles for use in most of our day to day activities. We do have some baremetal k3s clusters that have been provisioned with the bootstrap and k3s roles. I know that the setup has worked well for almost a year and the config and versions are easy to track/read - So I wanted to reuse them for my homelab as well. 

I modified my inventory.ini to include the new VM hosts that I provisioned:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

# VMs on home NAS, used for k3s  
k8s-node-1 ansible_host=...  
k8s-node-2 ansible_host=...  
k8s-node-3 ansible_host=...

[k3s_cluster]  
k8s-node-1  
k8s-node-2  
k8s-node-3  
  
[k3s_server]  
k8s-node-1  
  
[k3s_agent]  
k8s-node-2  
k8s-node-3

The above inventory.ini now defines the 3 VMs I created, the ansible_host variable defines the IP address they are reachable with, The k3s_cluster defines the entire cluster, the others define the server and the agents. 

I then created a file in group_vars/k3s_cluster.yaml that defines the k3s version to use as well as the server IP:

1
2

k3s_version: v1.27.4+k3s1
k3s_server_ip: ...

The server and agent have a very similar group_vars/k3s_server.yaml or group_vars/k3s_agent.yaml file:

1
2
3

k3s_node_type: server # or agent
k3s_server_extra_args: >- # or `k3s_agent_extra_args`
  --node-external-ip={{ ansible_host }}

It is also possible with the ansible role to setup a wireguard mesh in between the hosts (Bit overkill for a homelab, but useful if you have many hosts).

Then you’d have to install the ethpandaops ansible collection found here.

The playbook to run would look as below:

1
2
3
4
5
6

- hosts: k3s_cluster
  become: true
  serial: "{{ batch_count | default('100%') }}"
  roles:
    - role: ethpandaops.general.k3s
      tags: k3s

Now it all comes together with ansible-playbook -i inventories/servers/inventory.ini playbooks/setup_k3s.yml. This will setup the cluster in the configuration defined and allow you to update it with ease if you ever need to. You can SSH into one of the nodes and grab the kubeconfig, place it in your local .kube/ folder and from then on kubectl get nodes should work without any hitches(make sure you have the right context set)!

Cleanups can be managed with a simple variable being set, -e k3s_cleanup=true!

Conclusion

We looked into the why as well as a few approach to setup k3s, we now should have a running cluster that’s ready for use! We definitely could have done certain things in a more automated manner, but I was able to get the cluster up in under an hour - so its probably enough for now. Stay tuned for more about how to actually deploy things to the cluster and to learn if I decide to give up on it altogether :D

Information that is usually at hand for debugging: Slot number. So we can just
use the endpoint /eth/v2/beacon/blocks/{block_id}, where block_id can be either
“finalized”,”genesis”,”slot” or a “hex encoded root”.

Since we would want to do this often, let’s abstract it into a cli function. I
will assume from here on that your system uses zsh and has jq installed, it
should however work similarly for bash.

1. Open your ~/.zshrc file
2. Go to the bottom and add a comment # fetch the entire block given a CL slot
   number or something similar
3. Below the comment, enter: fetch_block_at_slot(){curl -sS
   “$1”/eth/v1/beacon/blocks/$2 | jq }
4. Reload the config with source ~/.zshrc

This will add the function fetch_block_at_slot and it accepts 2 arguments: The
first argument being the endpoint to which the data is feched from. The second
is the slot number.

An example of the usage is:

1
2
3
4
5
6
7
8
9
10
11

fetch_block_at_slot localhost:4000 1100
> {
  "data": {
    "message": {
      "slot": "1100",
      "proposer_index": "12798",
      "parent_root": "0xf130e3162dc304d05d397c8d79f81b38e7d7a1b92a7ca87b65f5c2f81f03a10a",
      "state_root": "0xea580fc86377c3db5301acd224204603394f607e98c9a63813d6e2ef68cd605c",
      "body": {
....
}

Supposing we need to fetch the execution payload very often, well, we can just
add a new function to handle that as well! Similar to before, add a function
called: fetch_execution_payload(){curl -sS "$1"/eth/v1/beacon/blocks/$2 | jq -r '.data.message.body.execution_payload' }.

An example of this function is:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

fetch_execution_payload localhost:4000 1100
> {
  "parent_hash": "0x0000000000000000000000000000000000000000000000000000000000000000",
  "coinbase": "0x0000000000000000000000000000000000000000",
  "state_root": "0x0000000000000000000000000000000000000000000000000000000000000000",
  "receipt_root": "0x0000000000000000000000000000000000000000000000000000000000000000",
  "logs_bloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
  "random": "0x0000000000000000000000000000000000000000000000000000000000000000",
  "block_number": "0",
  "gas_limit": "0",
  "gas_used": "0",
  "timestamp": "0",
  "extra_data": "0x",
  "base_fee_per_gas": "0",
  "block_hash": "0x0000000000000000000000000000000000000000000000000000000000000000",
  "transactions": []
}

TTD stands for total terminal difficulty, it defines when the Ethereum PoW is
ready for the merge. Difficulty goes up with each block till TTD is hit, if
things go correctly, we then switch to PoS mode. Total Difficulty can be
obtained from the block information, similar to the earlier examples.

1. Open your ~/.zshrc file
2. Go to the bottom and add a comment # calculate percentage of TTD doneor
   something similar
3. Below the comment, enter: fetch_ttd_status(){echo “Total Difficulty
   completed till TTD: $(($(curl -s --request POST "$1" --header 'Content-Type: application/json' --data-raw '{ "jsonrpc":"2.0", "method":"eth_getBlockByNumber", "params":[ "latest", false ], "id":1 }' | jq -r '.result.totalDifficulty')))/"$2" * 100" | bc) % }
4. Reload the config with source ~/.zshrc
5. You need to pass the RPC URL as well as the Total Terminal Difficulty while
   calling this function

An example of this function is:

1
2

fetch_ttd_status https://rpc.devnet3.themerge.dev 5000000000
> Total Difficulty completed till TTD:  8.1700 % 

Some time passes and you realize you need something from that file, but for
whatever reason the file doesn’t exist anymore. Yikes. Well, no need to worry.
Git will have stored the file as a blob to its object store, you just need to
figure out which one it is.

Let’s recreate the scenario really quick:

1
2
3
4
5
6
7
8
9

$ git init
> Initialized empty Git repository in /tmp/.git/

$ echo "text" > test

$ git add test.txt

$ git rm -f test
> rm 'test'

Great, now we want to try and get back the test file. There are two ways to do
this:

1. You can list the dangling objects by using git fsck –lost-found giving us
   the result dangling blob 8e27be7d6154a1f68ea9160ef0e18691d20560dc. We can
   check the file content by using git show
   8e27be7d6154a1f68ea9160ef0e18691d20560d and restore the file.
2. You can manually check the folders and list the objects to check which one
   is your file, finally restoring it.

1
2
3
4
5
6
7

$ ls .git/objects/
> 8e  info  pack

$ ls .git/objects/8e
27be7d6154a1f68ea9160ef0e18691d20560dc

$ git show 8e27be7d6154a1f68ea9160ef0e18691d20560d

altair-devnet-3 is the fourth devnet spun up with the express intent to test client compatibility and catch any
issues before a larger public testnet. The specs for the devnet can be found here.
The devnet had a rocky start due to improper configs by some participant, the issue was found quite quickly and the devnet
started to finalize. The altair hardfork was successfully completed and the devnet continued as expected. The attestation
rates were relatively good and sync aggregate participation rates were hovering around the 95% mark. The devnet had worked
as expected and things seemed good. The rest of the blogpost purely refers to steps used in debugging and not the actual
fixe itself.

Metrics and logs seemed to indicate something was wrong though. First step was to check the logs - the lighthouse nodes
seem to have lost all their peers and were at roughly 3 peers. The next step was to check the metrics to see what the timeline
looked like. I logged into Grafana and created a quick dashboard that logged peers per client. The metrics clearly showed
that the lighthouse client was leaking peers slowly. That tells us that there was no issue with the hardfork leading to a
sharp drop in peers. It additionally tells us that the peer score was slowly being reduced until the lighthouse node banned
the peer.

Armed with this information, I restarted the nodes with debug logging enabled to get more information about peer scoring.
Since debug logs can be extremely overwhelming, some grep-fu was required. Since we don’t have an exact log line we are looking
for, general grepping of terms such as voted, invalid and downvoted would make sense.

I ran docker logs beacon 2>&1 | grep "invalid" and most results were related to PastSlots indicating a high latency to
other peers and receiving slots too late. But there was one log message in particular that was interesting:

1

DEBG Invalid sync committee message from network, type: "sync_contribution", peer_id: 16Uiu2HAmP4jDeYxvVkVizndoYksJYZvpCkgSrWxeHEL5qj4tsTtk, reason: InvalidSignature

Now we had an invalid signature (bad thing) and a peerID to hunt down. The lighthouse API endpoint allows us to get all the
peers of that node along with metadata about the client and the IP address. The standard endpoint is /eth/v1/beacon/peers,
but lighthouse also provides us with /lighthouse/peers that contains more metadata. Since the response would be a wall of json,
we use jq to make it look pretty and grep to fetch the corresponding peer. Grep allows us to get the lines above and below
the matched line, allowing us to get the entire json body.

Running curl localhost:4000/lighthouse/peers | jq | grep -B 1 -A 52 "16Uiu2HAmP4jDeYxvVkVizndoYksJYZvpCkgSrWxeHEL5qj4tsTtk"
gives us the information about the peer. The response looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

{
  "peer_id": "16Uiu2HAmP4jDeYxvVkVizndoYksJYZvpCkgSrWxeHEL5qj4tsTtk",
  "peer_info": {
    "_status": "Healthy",
    "score": {
      "Real": {
        "lighthouse_score": -58.15072332402626,
        "gossipsub_score": 8.653623324336476,
        "ignore_negative_gossipsub_score": false,
        "score": -58.14044714632861
      }
    },
    "client": {
      "kind": "Nimbus",
      "version": "unknown",
      "os_version": "unknown",
      "protocol_version": "ipfs/0.1.0",
      "agent_string": "nimbus"
    },
    "connection_status": {
      "status": "banned",
      "connections_in": 0,
      "connections_out": 0,
      "last_seen": 23148
    },...

That tells us the peer in question that produced the invalid signature is a nimbus client and it was banned by lighthouse due
to its poor score.

Since the devnet is a multiclient testnet, we should be able to see what the other clients think of this peer. So I SSH-ed into
an instance containing a prysm instance and checked the logs for this peer. I switched prysm to debug logging as well and
ran docker logs beacon 2>&1 | grep "16Uiu2HAmP4jDeYxvVkVizndoYksJYZvpCkgSrWxeHEL5qj4tsTtk". The logs however indicate no
issues. Prysm simply listed one log message when it connected to the peer and there were no other entries about the peer.

The information was passed on to the respective client teams along with access to the logs. They are now investigating the issue further.
Since this is not a critical issue and hasn’t stopped the chain from finalizing, it would be treated as something that needs a patch. I
will try my best to update this blogpost in the future with the final outcome.

Data de-duplication refers to a method to eliminate storing duplicate copies of repeating data. This is extremely advantageous
if we want to save space on our backup machine for our regular backups. Since the offsite location might not be fully under
our control we would also want to encrypt our backups as well as compress it to save space.

So essentially we are looking for a backup tool that:
- is space efficient
- supports encryption
- supports compression
- run from the cli/scriptable
- performs de-duplication
- available on multiple platforms

Enter BorgBackup(short Borg), a client-server based backup tool that fits all of our criteria! Let’s set it up and see how it works.

Installation

For the purpose of this post, I will assume the host and target machine is a linux based system (Ubuntu/Debian ideally).

Borg is part of the official ubuntu/debian repos, so you can install it with sudo apt install borgbackup

Configuration

Borgbackup can be run to backup locally or to a mounted volume or to backup remotely. This config will focus on remote backups.

• Initialize the backup repository on your backup server with borg init --encryption=repokey </path/to/repo>. Borg will setup an
  empty backup repository that is ready for future backups. Please note the path, as that is important information. Enter
  an encryption password when prompted, this is your encryption key and is needed for each future backup or restore.
• Now shift to your host machine and get your SSH public key, typically cat ~/.ssh/id_rsa.pub and add this key to the
  backup machine’s file named ~/.ssh/authorized_keys. This would allow your host machine to SSH into your target machine
  without an SSH password.
• Test the SSH connection between the two with ssh username@backup-ip-address. Additionally, you can create a dedicated
  backup user on the backup machine for a tighter control.
• We can create a test backup to check if everything works as expected with this command: borg create -v --stats username@backup-ip-address:</path/to/repo>::BACKUP_NAME </path/to/something/to/backup>.
• Now run borg list username@backup-ip-address:</path/to/repo>, it should list the backup you just created.
• We can start scripting this process now, create a file called borgbackup.sh:
  Note: I like to have my backups with unique names, so I append the backup name with the current time.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

#!/bin/bash
# Backup a folder to a remote address using borg.
set -eu

echo "Successfully set the time"
TIME=$(date +%s)

# export the borg passphrase as an environment variable to avoid having to enter it each time
export BORG_PASSPHRASE='blablabla'
echo "Backing up local to remote server"

# Backing up xyz content
borg create --progress --stats username@backup-ip-address:</path/to/repo>::BACKUP_NAME-$TIME </path/to/something/to/backup>

# Prune old backups
borg prune username@backup-ip-address:</path/to/repo> --keep-weekly=1 --keep-monthly=2

• Make the script executable with chmod +x borgbackup.sh and run the script to test it. Run borg list username@backup-ip-address:</path/to/repo>
  to confirm.
• Let’s make the script run periodically now, we will do this using crontab. Run crontab -e and configure the
  frequency and script, e.g: 0 3 * * * /path/to/borgbackup.sh to run the script at 3AM every morning. Check a crontab guide
  to figure out how to modify the time.
• Run crontab -l to confirm the crontab has been setup.

Conclusion

Have fun with your nightly backups! I will try to write up a guide for setting up a wireguard system, so that you can roll
your own private VPN and move your backup server far away, yet private.

In this blogpost, we will look at deploying Loki, deploying Promtail and collecting docker logs with the log driver.

Loki

We will be deploying Loki as a docker container.

• Create a directory to store the config
• Copy the loki configuration inside this folder called loki-config.yml with the below contents:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83

# Enables authentication through the X-Scope-OrgID header, which must be present
# if true. If false, the OrgID will always be set to "fake".
auth_enabled: false

# Configures the server of the launched module(s).
server:
  http_listen_port: 3100
  grpc_listen_port: 9096

# Configures the ingester and how the ingester will register itself to a
# key value store.
ingester:
  wal:
    enabled: true
    dir: /tmp/wal
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 1h       # Any chunk not receiving new logs in this time will be flushed
  max_chunk_age: 1h           # All chunks will be flushed when they hit this age, default is 1h
  chunk_target_size: 1048576  # Loki will attempt to build chunks up to 1.5MB, flushing first if chunk_idle_period or max_chunk_age is reached first
  chunk_retain_period: 30s    # Must be greater than index read cache TTL if using an index cache (Default index read cache TTL is 5m)
  max_transfer_retries: 0     # Chunk transfers disabled

# Configures the chunk index schema
schema_config:
  configs:
    - from: 2021-06-23
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

# Configures where Loki will store data.
storage_config:
  boltdb_shipper:
    active_index_directory: /tmp/loki/boltdb-shipper-active
    cache_location: /tmp/loki/boltdb-shipper-cache
    cache_ttl: 24h         # Can be increased for faster performance over longer query periods, uses more disk space
    shared_store: filesystem
  filesystem:
    directory: "/mnt/config/chunks"

# Configures the compactor component which compacts index shards for performance.
compactor:
  working_directory: /tmp/loki/boltdb-shipper-compactor
  shared_store: filesystem

# Configures limits per-tenant or globally
# ingestation when promtail is run for the first time can be a lot of data, hence the high burst limit
limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  ingestion_rate_mb: 50
  ingestion_burst_size_mb: 150

# Configures how Loki will store data in the specific store.
chunk_store_config:
  max_look_back_period: 0s

# Configures the table manager for retention
table_manager:
  retention_deletes_enabled: false
  retention_period: 0s

# The ruler_config configures the Loki ruler.
ruler:
  storage:
    type: local
    local:
      directory: /tmp/loki/rules
  rule_path: /tmp/loki/rules-temp
  alertmanager_url: http://localhost:9093
  ring:
    kvstore:
      store: inmemory
  enable_api: true

• Run the docker container with the below command:

1

docker run -d --name loki --restart unless-stopped -p 3100:3100 -v {{ loki_dir }}:/mnt/config grafana/loki:latest --config.file=/mnt/config/loki-config.yaml

• Run curl localhost:3100/ready to check if the container is up as expected

Now we have a running version of Loki, we just need to get logs to it.

Promtail

Promtail is a collector agent that scrapes the logs on a
local machine, parses them and sends the logs to a Loki
instance for longer term storage.

We will also run promtail as a docker container. We use a volume mapping in read-only mode to allow Promtail to read the
system logs of the machine. These logs are then scraped as specified in the scrape_configs, similar to Prometheus.
Any extra labels can be added at this stage.

• Create a directory to store the config
• Copy the promtail configuration inside this folder called promtail-config.yml with the below contents:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

# speficies the promtail server ports
server:
  http_listen_port:9080
  grpc_listen_port:0

# specifies the temp positions directory
positions:
  filename: /tmp/positions.yaml

# specifies the loki server URL to which logs are sent
clients:
  - url: http://localhost:3100/loki/api/v1/push

# specifies the scrape locations for the logs as well as any postprocessing
scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          __path__: /var/log/*log

• Run the docker container with the below command:

1

docker run -d --name promtail --restart unless-stopped  -v {{ promtail_dir }}:/mnt/config -v /var/log:/var/log:ro grafana/promtail:2.2.1 --config.file=/mnt/config/promtail-config.yaml

Docker logs

While it is easy for promtail to forward docker logs and tag them, it is relatively hard for it to label the logs with the
associated container_name or other labels. This could be done before running the docker container itself, but for sake
of simplicity as well as not needing to always add labels, we can set it as the default docker log method.

We achieve this by installing the loki docker plugin and then by setting the default docker log method to it. That way
all newly created containers will use the new log plugin.

• Install the docker loki plugin with:
  docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions
• Either modify or create a file at /etc/docker/daemon.json with the following content. Add any extra log-opts if needed:

1
2
3
4
5
6
7
8
9

{
  "debug" : true,
  "log-driver": "loki",
  "log-opts": {
    "loki-url": "http://localhost:{{ loki_http_listen_port }}/loki/api/v1/push",
    "loki-batch-size": "400",
    "loki-external-labels": "job=dockerlogs,container_name={{.Name}}"
  }
}

• Restart the docker engine to apply the changed with: sudo systemctl restart docker
• Now we can try running a new container and send logs directly to loki with docker run -it --name test alpine echo hi

As long as Loki has been added as a data source on Grafana, we should be able to see the logs on the Explore tab on Grafana.

Let’s assume for a second you have a lot of testnet ether lying around and you
want to test out your staking setup. Sounds great!…until you have to make the
dozens of deposits. Thankfully we can script this and perform it in stages!

Pre-requisites:
We will use the following tools in this blogpost:

• eth2-val-tools: https://github.com/protolambda/eth2-val-tools
• ethereal: https://github.com/wealdtech/ethereal

Deposit steps:

1. We start by obtaining the Goerli ether, either from faucets or from someone
   who has a lot. For the sake of this blog, let’s assume the funds are held on
   metamask.

2. Create a new folder and a file inside that folder called secrets.env. The
   secrets.env file will contain our secrets to avoid having to store them in our
   script directly. You should add this file to your .gitignore to avoid committing
   it by mistake.

3. Fill secrets.env with the following variable names:

1
2
3
4
5
6
7
8
9
10
11
12
13

ACC_START_INDEX=
ACC_END_INDEX=
DEPOSIT_AMOUNT=
FORK_VERSION=
VALIDATORS_MNEMONIC=
WITHDRAWALS_MNEMONIC=
DEPOSIT_DATAS_FILE_LOCATION=

DEPOSIT_CONTRACT_ADDRESS=
ETH1_FROM_ADDR=
ETH1_FROM_PRIV=
FORCE_DEPOSIT=false
ETH1_NETWORK=goerli

Find the deposit contract address, deposit amount and fork version
for your required scenario. For example, we have these values for the Prater
testnet:

1
2
3
4

DEPOSIT_AMOUNT=32000000000
FORK_VERSION="0x00001020"

DEPOSIT_CONTRACT_ADDRESS="0xff50ed3d0ec03aC01D4C79aAd74928BFF48a7b2b"

5. Our Mnemonic is the source of the private keys of the validators. Loosing
   this mnemonic means the loss of all funds, so please store the mnemonic safely.
   We can generate the validator and withdrawal mnemonic using the tool
   eth2-val-tools. In your terminal, type out:

1

eth2-val-tools mnemonic

The output should contain a list of 24 words. Copy those words into the
secrets.env file under the variable VALIDATORS_MNEMONIC. Don’t forget to wrap
the 24 words inside “”!

Repeat the process for WITHDRAWALS_MNEMONIC.

Please backup these mnemonics safely! Ideally inside a password manager as a
secure note.

6. Decide on how many deposits you would like to perform. Generally this would
   be (amount of testnet ether you possess)/32. Set the ACC_START_INDEX and
   ACC_END_INDEX in secrets.env accordingly.

7. Now we can finally add our private key and address from which the deposits
   will be made. This will be the in metamask, go to the account with the goerli
   ether > click on the three dots > click on Account details > click on export
   private key. We will need the private key in the script to access the funds and
   perform the deposits, so save this somewhere safely. Fill out ETH1_FROM_ADDR and
   ETH1_FROM_PRIV in secrets.env accordingly. Don’t forget to wrap both inside “”
   and they should start with 0x.

8. We are finally ready to start generating our deposit data! Instead of
   directly performing the deposits, we will split our scripts into two halves- One
   to generate the deposit data and one to execute the deposits.

We will store the deposit data temporarily to allow us to manually inspect it,
enter the full path of where the file should be stored as the variable
DEPOSIT_DATAS_FILE_LOCATION in secrets.env.

Create a file called build_deposits.sh with the following contents:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

#!/bin/bash

echo "USE AT YOUR OWN RISK"
read -p "Are you sure? " -n 1 -r
echo
if [[ ! $REPLY =~ ^[Yy]$ ]]
then
    [[ "$0" = "$BASH_SOURCE" ]] && exit 1 || return 1
fi

source secrets.env

eth2-val-tools deposit-data \
  --source-min=$ACC_START_INDEX \
  --source-max=$ACC_END_INDEX \
  --amount=$DEPOSIT_AMOUNT \
  --fork-version=$FORK_VERSION \
  --withdrawals-mnemonic="$WITHDRAWALS_MNEMONIC" \
  --validators-mnemonic="$VALIDATORS_MNEMONIC" > $DEPOSIT_DATAS_FILE_LOCATION.txt

The script will read the variables inside secrets.env and build the deposit data
for you in a text file stored at $DEPOSIT_DATAS_FILE_LOCATION.

Run the script using ./build_deposits.sh.

9. The deposit data will be a bunch of json with the following format:

1
2
3
4
5
6
7
8
9

{
  "account": "m/12381/3600/1/0/0",
  "deposit_data_root": "",
  "pubkey": "",
  "signature": "",
  "value": 32000000000,
  "version": 1,
  "withdrawal_credentials": ""
}

Manually check if it looks okay to begin with. The account format
would be m/12381/3600/key_number/0/0. The key_number should co-relate to the
ACC_START_INDEX and ACC_END_INDEX entered earlier.

10. Manually generate the public keys using eth2-val-tools to cross check if the
    public keys generated are correct.

1

eth2-val-tools pubkeys --validators-mnemonic=$VALIDATORS_MNEMONIC --source-min=$ACC_START_INDEX --source-max=$ACC_END_INDEX

You will get a list of public keys in order of the min and max entered. I
normally check the first and last public key as a sanity check.

11. Perform the deposits! Please be really sure before executing this step!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

#!/bin/bash
echo "USE AT YOUR OWN RISK"
read -p "Are you sure? " -n 1 -r
echo
if [[ ! $REPLY =~ ^[Yy]$ ]]
then
    [[ "$0" = "$BASH_SOURCE" ]] && exit 1 || return 1
fi

source secrets.env

if [[ -z "${DEPOSIT_CONTRACT_ADDRESS}" ]]; then
  echo "need DEPOSIT_CONTRACT_ADDRESS environment var"
  exit 1 || return 1
fi


if [[ -z "${ETH1_FROM_ADDR}" ]]; then
  echo "need ETH1_FROM_ADDR environment var"
  exit 1 || return 1
fi
if [[ -z "${ETH1_FROM_PRIV}" ]]; then
  echo "need ETH1_FROM_PRIV environment var"
  exit 1 || return 1
fi

# Iterate through lines, each is a json of the deposit data and some metadata
while read x; do
   # TODO: check validity of deposit before sending it
   account_name="$(echo "$x" | jq '.account')"
   pubkey="$(echo "$x" | jq '.pubkey')"
   echo "Sending deposit for validator $account_name $pubkey"
   ethereal beacon deposit \
      --allow-unknown-contract=$FORCE_DEPOSIT \
      --address="$DEPOSIT_CONTRACT_ADDRESS" \
      --network=$ETH1_NETWORK \
      --data="$x" \
      --value="$DEPOSIT_ACTUAL_VALUE" \
      --from="$ETH1_FROM_ADDR" \
      --privatekey="$ETH1_FROM_PRIV"
   echo "Sent deposit for validator $account_name $pubkey"
   sleep 2
done < "$DEPOSIT_DATAS_FILE_LOCATION.txt"

This script will read the values in secrets.env and then go through the
temporary deposits file specified in $DEPOSIT_DATAS_FILE_LOCATION. It will
process the json line by line and perform the deposit using the private key.

Since we want to perform a large amount of deposits, we ideally want to avoid
dos-ing the network all at once. To avoid that, there is a sleep command between
each deposit to allow some time between deposits.

Note: It is possible that one or more deposits might fail if done too quickly or
due to the network state. To avoid this, its best to split up deposits into
smaller batches of 500/1000 or so - this would also make debugging nonce issues
easier. I have tested doing 2000 deposits in one go on Goerli without an issue
with the above mentioned parameters.

Finally,Run the script using exec_deposits.sh .

12. Give the transactions some time to be included in the block and indexed.
    Then head over to a explorer such as Otterscan
    [https://github.com/wmitsuda/otterscan] or Etherscan
    [https://goerli.etherscan.io/] and enter either your own address or the deposit
    contract address. You should now be able to see a list of all your deposits,
    also the same as the output from exec_deposits.sh.

13. Collect one or more public keys, either from the sanity check stage of 10.
    or from the deposit data and head over to Beaconchain
    [https://prater.beaconcha.in/], enter your Validator public key here to check
    its status. You would usually have to wait for the deposit to be processed and
    for the validator to be activated. Beaconchain will also show you if your
    deposit was valid or if there was some error.

14. Setup your validator ahead of time! Do not wait for the validator to be
    activated to begin the setup process, there is no penalty for starting a
    validator before it is activated. To help with client diversity, please setup a
    non-majority [https://twitter.com/superphiz/status/1434857668783575040] client!

15. Profit ???

Huge shoutout to Protolambda [https://protolambda.com/] and the tool developers
for creating both the scripts and the tools!