Bypassing VM detection

Some software refuses to run in a virtual machine. Some runs, but changes what it does once it notices. Malware that spots a VM goes quiet, so a sandbox watching it learns nothing about what it really does. Gaming anti-cheat refuses to start. A bit of DRM drops to a worse path. In every case the software is asking one question - am I in a VM - and I wanted to know how hard it is to make the answer come back no.

Short answer: hard for about a week, then not hard at all. The guest ended at one detection out of 85 on an open scanner, and the scanner stops calling it a VM:

VM likeliness: 20%
VM confirmation: false
VM detections: 1/85
===== CONCLUSION: Running on bare metal =====

The hiding cost no performance I could measure. And the thing that took the most time was not a detection. It was a clock.

What gives a VM away#

A virtual machine answers a lot of small questions differently from real hardware. The firmware vendor strings, the disk model and serial, the PCI IDs of devices that only exist in emulators. Most of that is cosmetic. QEMU can be told to present a real board's values instead of its own, and once it does, those checks go quiet.

The interesting failures were the ones no config file touches. Three of them were KVM itself diverging from what the CPU architecture specifies - the exact exception a privileged instruction raises, the state a control bit is left in. A guest reads those straight off the silicon. You cannot paper over them from the outside. The kernel has to be right.

One check cost me two days on a false lead. Its debug line read: KVM caught patching instructions on the fly. KVM really does that in one narrow case, so I turned the behaviour off, and nothing changed - the message was pointing at the wrong thing. The real fault was small and elsewhere. A privileged instruction, run from user space, was raising a general protection fault where a real CPU raises invalid-opcode, and that one wrong exception is all the check looks at. Two small patches, and it stopped firing.

The next detection printed nothing at all, and the empty output was the clue - the check had bailed out on its one silent path. The cause was a bit KVM never clears. The hardware needs a virtualisation-enable bit set to run a guest, so KVM sets it and leaves it on the whole time. On real silicon that bit is off unless you turn it on. So a guest that never touched virtualisation still looks like it did, and an instruction that should raise invalid-opcode raises a protection fault instead. The guest reads the difference. KVM already carried a FIXME for exactly this, and I turned it into a patch.

The clock that cost 85x#

With the detections down I turned to performance, and the first number the benchmark printed stopped me:

clock qpc_ns=1298.5  rdtsc_ns=6.5

QueryPerformanceCounter, the call a game makes several times a frame to know what time it is, was taking 1298 nanoseconds. On bare metal it is tens. The raw TSC beside it read 6.5, so the clock hardware was fine - Windows had simply decided not to use it.

Windows calibrates the TSC once, at boot. If that calibration goes badly it falls back to a clock that traps out to the hypervisor on every read, and it stays fallen back for the rest of that boot. 1298 nanoseconds is the cost of that trap, paid on every frame, for the life of the session.

I went the wrong way with it first. I dropped the guest from 32 vCPUs to 16, QPC snapped back to 15 nanoseconds, and I wrote down never exceed 16 vCPUs. I never tried a size in between.

It is not a vCPU ceiling. It is a scheduling race.

Windows boots
     |
     v
calibrate the TSC      <- one shot, never retried
     |
     +-- vCPU threads scheduled cleanly    -> TSC kept        -> QPC   15 ns
     |     host still has spare cores
     |
     +-- vCPU threads preempted mid-run    -> TSC abandoned   -> QPC 1300 ns
           host squeezed to nothing             ...for the rest of that boot

What decides it is whether the host has spare cores to run the vCPU threads cleanly while that one-shot calibration happens. Four cold boots at each size on a 16-core host:

Headroom, not a ceiling. Hand the guest 24 of 32 threads and it wins the race every time, which matters if the machine does real work and not only games.

The cost is worse than the clock reading alone. On the boots that lose the race, long scheduling stalls go from near zero into the hundreds, and that is felt as microstutter no average frame time will show.

The scheduler trick that hung my host#

The obvious fix, once you know it is a scheduling race, is to stop the vCPU threads being preempted at all. libvirt lets you pin them to realtime priority. Do not do it to all of them. On a guest sized to the whole machine, realtime priority on every vCPU means nothing else on the host runs, including the emulator thread the guest itself depends on. It booted once, hung twice, and left me killing QEMU by hand.

Hiding turned out to be free#

Everyone says hiding the hypervisor costs performance, because Windows stops using its paravirtual shortcuts. I built the opposite as a reference - same guest, everything visible, every enlightenment on - and there was no gap. The spread between two runs of the same config was wider than the difference between hidden and visible. The one place the shortcuts should have helped is the clock, and once the guest is sized right Windows uses the real TSC on its own. So there is no trade-off to make, and no reason to run a half-hidden guest.

The last two checks#

Two held out longest. One I beat with a workaround, one I did not beat at all.

The stubborn one times how long a CPU-identification instruction takes, because in a VM it traps to the hypervisor and on hardware it does not. There is no faking that from inside - the trap is real and it costs what it costs, about 400 nanoseconds of world switch. The only way to stop paying it is to stop trapping, so the workaround stops intercepting the instruction on a pinned vCPU and reprograms the CPU-name registers by hand so the guest still reads the right processor. It has to be switched on after boot, because a Windows still enumerating the CPU hangs if the instruction changes shape under it. Ugly, but it clears the check.

There is a second detector inside that same check, and for a while I was sure it was a hard wall. I measured it, got a ratio that looked like a hardware floor, and left it at that. The probe was wrong - it timed the wrong call on the software side. Once I fixed it the ratio came back fine, and the detector had not been firing to begin with.

The one I did not beat wants a display that can report a gamma ramp, which a headless guest does not have. The only real fix is passing a physical GPU through, and I did not have a spare card to give up. That is the single detection still standing.

What I got wrong#

Two more.

My first performance comparison showed hiding costing three percent, consistent across every run, and I believed it - until I noticed the benchmark was running three tests at once that fought over the same cores. Both configs were measured the same wrong way, so the gap looked real and was not. Run sequentially, it vanished.

And while automating the host-side setup I wrote a libvirt hook whose fallback called back into libvirt. libvirt was blocked waiting for the hook to return, the hook was blocked waiting for libvirt to answer, and the visible symptom was my VM manager showing no virtual machines at all - a deadlock I had built by having the hook call the thing already waiting on it.

Where it ended up#

All of it became three Arch packages and a setup command that reads the host, sizes the guest, and writes the whole configuration in one pass. It is on the AUR now.

I do not run anything that needs this. No game with kernel anti-cheat, no malware to fool into behaving. The detections were the reason I started, and the clock was the part I did not expect. Most of the work turned out to be old bugs in KVM and one calibration Windows only gets one shot at. Making the VM quiet was the easy half.

Full writeup, copy-paste grade: VM Detection Hardening.

Source: git.archworks.co/sandwich/vfio-native.