Skip to main content

MicroVMs

Understanding common MicroVM boot/run errors

Data and logs for each MicroVM will be stored under /var/lib/flintlock/vm, with the full path for a MicroVM being /var/lib/flintlock/vm/NAMESPACE/NAME/UUID.

Under that directory are the following files:

  • firecracker.cfg: the configuration set by flintlock for firecracker
  • firecracker.log: logs from the firecracker process
  • firecracker.metrics: metrics from the firecracker process
  • firecracker.pid: the pid of the running firecracker process
  • firecracker.stderr: MicroVM boot errors
  • firecracker.stdout: MicroVM boot logs
  • metadata.json: MicroVM boot userdata

MicroVM network configuration failure/timeout​

When MicroVMs boot, they will be assigned an address from wherever you have a dhcp server. This may be your home router or a dedicated pool you have configured for a VLAN.

If you see the following logs, then it means this has failed.

A start job is running for Wait for Network to be Configured (38s / no limit)
... etc for a couple of mins. but really if it is going to work it will do
... so within 10 seconds
Failed Wait for Network to be Configured.

Steps:

  • Check that the host can ping the router. If not, and you have access to the router, disconnect the device (turn it off) and remove the previous record from the router's list. Reconnect the device and try pinging again.
  • Ensure that the correct interface has been configured in /var/lib/flintlockd/config.yaml. parent-iface should be set to a wired interface (eg eth0). If you are using flintlock in bridge mode, bridge-name should be a bridge which is mastered to a parent interface with internet access.
  • On the MicroVM's host device run tcpdump -i <parent interface name> -vv while creating a MicroVM. Look for dhcp Discover, Offer, Request and ACK messages. If you see none, it means the MicroVM cannot reach your dhcp server at all. If you see the first 2 but not the last, it means the dhcp server has offered an IP by the client for some reason does not receive or accept that. From here you are debugging based on your particular device and networking setup.

Cannot SSH into MicroVM​

  • Check that you set a public SSH key in the spec user-data or the cluster manifest.
  • Check the firecracker.stdout boot logs of the MicroVM to make sure you have the correct IP, or that one has been set at all (see the above section).

MicroVM has not started​

If you create a MicrmVM but cannot see a firecracker process or the boot logs at /var/lib/flintlock/vm/NS/NAME/UID/firecracker.stdout are empty, try these steps:

  • Look at journalctl -fu flintlockd.service and for the line:
    "finished executing plan" controller=microvm execution_id=UID execution_time=4m3.498577899s num_steps=6 plan_name=microvm_create_update
    If this is not present, find what the last action for that UID was.
    note

    On the first create with a new kernel or OS image, it can take a while for containerd to pull it down.

  • Check the MicroVM error logs at /var/lib/flintlock/vm/NS/NAME/UID/firecracker.stderr
  • Check the firecracker logs at /var/lib/flintlock/vm/NS/NAME/UID/firecracker.log

If you are using ARM images, ensure that you have updated the kernel.filename in the manifest to be boot/image. For x86 this should be boot/vmlinux.