Setting up VMs on Proxmox with Pulumi

Pulumi is an open sourced IaC SDK that allows you to define and manage infrastructure through the use of various programming languages. This helps reduce the need to learn another HCL-ish language and gives you the extensibility and toolkit of whatever (popular) programming language you are comfortable with.

For my usecase, I have a Promxox node (Minisforum 773 Lite, 64GB RAM, 1TB SSD) and would want to use it to host my VMs. Initially I want to run 3 VMs to host a kubernetes cluster that I can use to better learn the technology.

I’m assuming that Pulumi is already installed and configured. Let’s now create a new stack with pulumi new python -y. This will create a few new files for us: Pulumi.yaml(Project metadata), __main__.py(Program entrypoint), requirements.txt(dependencies), venv (virtual env for the project).

Proxmox has an API that allows us to provision resources, however we would need to know the calls to make. To simplify the process, we can use a pulumi proxmox provider, which can be installed as a basic pip package with venv/bin/pip install pulumi-proxmoxve (or install via the requirements.yaml). This will allow us to create a proxmox provider and to define the VM in an easier manner and the package takes care of the required API calls to provision the VM.

In order to make configuration of VMs a bit easier, lets also define a folder called vms that contains YAML files containing all the values that we’d want to configure. We can then read the files in said folder, parse the values and then configure the pulumi proxmox provider to create the VMs for us. 

To start with, lets configure .gitignore to ignore the .env file that will be containing our secrets. Then create a .env file with the following contents:

1
2
3
4
5
PROXMOX_ENDPOINT="https://<PROXMOX-IP>:8006"
PROXMOX_INSECURE="true"
PROXMOX_USERNAME="username@pam"
PROXMOX_PASSWORD="PASSWORD"
PROXMOX_USER_ACCOUNT_PASSWORD="DEBIAN_BASE_PASSWORD"

The pulumi python logic for `__main__.py` is the following: 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
import pulumi
import pulumi_proxmoxve as proxmox
import os,yaml
from dotenv import load_dotenv
import ipaddress
load_dotenv()
provider = proxmox.Provider('proxmoxve',
                            endpoint=os.getenv("PROXMOX_ENDPOINT"),
                            insecure=os.getenv("PROXMOX_INSECURE"),
                            username=os.getenv("PROXMOX_USERNAME"),
                            password=os.getenv("PROXMOX_PASSWORD"),
                            )

folder_path = "vms/"

def load_yaml_files_from_folder(folder_path):
    yaml_files = [file for file in os.listdir(folder_path) if file.endswith(".yaml")]
    loaded_data = []

    for yaml_file in yaml_files:
        file_path = os.path.join(folder_path, yaml_file)
        with open(file_path, 'r') as file:
            yaml_data = yaml.safe_load(file)
            loaded_data.append(yaml_data)

    return loaded_data

parsed_data = load_yaml_files_from_folder(folder_path)

for vm in parsed_data:
    disks = []
    nets = []
    ip_configs = []
    ssh_keys = []

    for v in vm:
        for vmcount in range(v['count']):
            base_resource_name=v['resource_name']
            name_counter = vmcount + 1
            base_vm_id=v['vm_id'],
            for disk_entry in v['disks']:
                for d in disk_entry:
                    disks.append(
                        proxmox.vm.VirtualMachineDiskArgs(
                            interface=disk_entry[d]['interface'],
                            datastore_id=disk_entry[d]['datastore_id'],
                            size=disk_entry[d]['size'],
                            file_format=disk_entry[d]['file_format'],
                            cache=disk_entry[d]['cache']
                        )
                    )

            for ip_config_entry in v['cloud_init']['ip_configs']:
                ipv4 = ip_config_entry.get('ipv4')

                if ipv4:
                    new_address = ''
                    ip, subnet = ipv4.get('address', '').split('/')
                    new_ip = str(ipaddress.ip_address(ip) + vmcount)
                    new_address = f"{new_ip}/{subnet}"

                    ip_configs = []
                    ip_configs.append(
                        proxmox.vm.VirtualMachineInitializationIpConfigArgs(
                            ipv4=proxmox.vm.VirtualMachineInitializationIpConfigIpv4Args(
                                address=new_address,
                                gateway=ipv4.get('gateway', '')
                            )
                        )
                    )

            for ssk_keys_entry in v['cloud_init']['user_account']['keys']:
                ssh_keys.append(ssk_keys_entry)

            for net_entry in v['network_devices']:
                for n in net_entry:
                    nets.append(
                        proxmox.vm.VirtualMachineNetworkDeviceArgs(
                            bridge=net_entry[n]['bridge'],
                            model=net_entry[n]['model']
                        )
                    )

            virtual_machine = proxmox.vm.VirtualMachine(
                vm_id=base_vm_id[0] + vmcount,
                resource_name=f"{base_resource_name}-{name_counter}",
                node_name=v['node_name'],
                agent=proxmox.vm.VirtualMachineAgentArgs(
                    enabled=v['agent']['enabled'],
                    # trim=v['agent']['trim'],
                    type=v['agent']['type']
                ),
                bios=v['bios'],
                cpu=proxmox.vm.VirtualMachineCpuArgs(
                    cores=v['cpu']['cores'],
                    sockets=v['cpu']['sockets']
                ),
                clone=proxmox.vm.VirtualMachineCloneArgs(
                    node_name=v['clone']['node_name'],
                    vm_id=v['clone']['vm_id'],
                    full=v['clone']['full'],
                ),
                disks=disks,
                memory=proxmox.vm.VirtualMachineMemoryArgs(
                    dedicated=v['memory']['dedicated']
                ),
                name=f"{base_resource_name}-{name_counter}",
                network_devices=nets,
                initialization=proxmox.vm.VirtualMachineInitializationArgs(
                    type=v['cloud_init']['type'],
                    datastore_id=v['cloud_init']['datastore_id'],
                    interface=v['cloud_init']['interface'],
                    dns=proxmox.vm.VirtualMachineInitializationDnsArgs(
                        domain=v['cloud_init']['dns']['domain'],
                        server=v['cloud_init']['dns']['server']
                    ),
                    ip_configs=ip_configs,
                    user_account=proxmox.vm.VirtualMachineInitializationUserAccountArgs(
                        username=v['cloud_init']['user_account']['username'],
                        password=os.getenv("PROXMOX_USER_ACCOUNT_PASSWORD"),
                        keys=ssh_keys
                    ),
                ),
                on_boot=v['on_boot'],
                reboot=v['on_boot'],
                opts=pulumi.ResourceOptions(provider=provider,ignore_changes=v['ignore_changes']),
            )

            pulumi.export(v['name'], virtual_machine.id)

The YAML for the proxmox VMs looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
---
- name: "k3s-nodes"
  count: 3
  node_name: "proxmox-vmfarm"
  resource_name: "k3s-node"
  vm_id: 1000
  agent:
    enabled: true
    type: "virtio"
  bios: "seabios"
  ignore_changes:
    - "disks"
    - "cdrom"
  cpu:
    cores: 4
    sockets: 1
  cloud_init:
    type: "nocloud"
    interface: "ide0"
    datastore_id: "local-lvm"
    dns:
      domain: ""
      server: "1.1.1.1 8.8.8.8"
    ip_configs:
      - ipv4:
          address: "192.168.90.50/32"
          gateway: "192.168.90.1"
    user_account:
      username: "root"
      keys:
       - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAiL82k0HKfSFp3JgdpShVvx9mRDk5fyYQyto00tBdeC

  clone:
    node_name: "proxmox-vmfarm"
    vm_id: 100
    full: true
  disks:
    - disk1:
        interface: "scsi0"
        datastore_id: "local-lvm"
        size: 180
        file_format: "raw"
        cache: "none"
  memory:
    dedicated: 16384
  network_devices:
    - net1:
        bridge: "vmbr0"
        model: "virtio"
  on_boot: true

The values being configured are quite self-explanatory, if not there is a detailed documentation in the proxmox provider page.

Once done, all you have to do is run pulumi up in the folder and the pulumi interactive shell will guide you through the rest!

Once the pulumi job is done, you should be able to see brand new VMs created in your proxmox host. If you want to change a value, you can simply modify the YAML and run pulumi up again. To destroy the VMs, you can run pulumi down or set the count to 0 to not affect the rest of the pulumi stack.