Measuring CPU impact on Qubes operation--the first step to a Qubes performance benchmark?

General Discussion
#1

While looking through various Qubes forums, I noticed that prospective Qubes users tend to worry about whether their systems are powerful enough. I’ve also wondered what the impact of CPU power on Qubes OS’ operating speed is. There aren’t any good resources on this. Since I don’t have access to a bunch of computers to experiment on myself, I came up with a standardized test we could use to create a handy reference table.

 

The VM start-up test

  1. Take an up-to-date version of the current release (R4.0, at the time of writing). It doesn’t matter if you’ve installed point releases of the current release (e.g. R4.0.3).
  • Entering sudo qubes-dom0-update into dom0 should return: No new updates available.
  1. Install a pristine, untouched version of the debian-x-minimal template, ‘x’ being the current supported release, which is debian-10-minimal at the time of writing. If you already have the template and would like to keep it, you can clone it and delete the original package using sudo dnf remove qubes-template-debian-[x]-minimal.
  • The template can be installed by typing sudo qubes-dom0-update qubes-template-debian-[x]-minimal into dom0, replacing [x] with the current release.
  1. For the sake of consistency, do not update this template or install anything onto it. This way everyone is attempting to start the exact same VM (AFAIK templates of releases are unchanged throughout their lifecycles).
  • I don’t feel this is ultra-important since I imagine most users who already have that template installed already have offshoots or haven’t installed much that would impact start-up time, but it’s nonetheless nice to have for the sake of good science.
  1. Install the latest VM kernel (enter sudo qubes-dom0-update kernel-latest-qubes-vm into dom0), then head to the template’s settings, select the correct version number, and save settings.
  • Kernels are trickier to keep consistent. Since point releases come with updated kernels, we can’t simply say, “Set the kernel to what your release shipped with”. It might be best to just use whatever the newest supported version is. I’m not sure how decent this solution is, since kernel version seems most likely to have a major impact on start-up time. Anyone have any ideas?

  1. Shut down all other VMs using qvm-shutdown --all (make sure all your work has been saved).

  2. Run time qvm-start debian-10-minimal and report the real time returned, converted into seconds if necessary.

  3. Submit your CPU model and your VM boot time in seconds

  • @deeplow is there a Discourse form for this, similar to a poll? Something simple that will create a database/spreadsheet with two fields

 

The test above doesn’t have much to do with RAM and parallel processing, but this would answer a central question for people looking to get into Qubes, since you can easily expand your RAM while switching your CPU typically implies getting a new system. A minimal template was chosen as the test vehicle because it varies the least, is relatively quick to download, and those who already have them on their systems tend to be more experienced with Qubes and won’t find it bothersome to clone and delete it. Debian was chosen arbitrarily.

I’m sure there are ways to take this test and make it parallel and with workloads so that the impacts of RAM and CPU cores can be more accessible, but that’s beyond my current abilities. This crude test should provide enough information and gives the most bang for the proverbial buck.

I hope someday there’ll be a Qubes benchmark that can go into the Hardware Compatibility List. Maybe something like the test above can be incorporated into qubes-hcl-report and return a benchmark for just the CPU.

1 Like
Survey: CPU and VM boot time
#2

Just wanted to add that this is especially relevant to me since I start and shut down VMs all the time. For people like me (typically those who have highly segregated systems and extensively using dispVMs), a system that minimizes VM startup and shutdown times would save a lot of time.

#3

There are way to do forms in discourse, but it requires a plugin. We keep this installation as vanilla as possible so installing such a plugin will probably not happen.

But for such simple table there is a better alternative: you can make a table and I then turn your post onto a wikipost so others can edit it to add their values to the table. We did a similar thing here.

1 Like
#4

This is a good workaround. Do you think we should limit contributions to established (level 1, maybe 2) users so we can sidestep issues of data cleanliness and issues that stem from having an openly editable wikipost?

#5

I did a similar test with x220 and x230, varying CPU, RAM and disk. You can
find it on the mailing list archive.
https://www.mail-archive.com/qubes-users@googlegroups.com/msg30816.html

The problem is that this tells users nothing about how their use of
Qubes will go.
You can tell people that an x220 i5 with 12GB RAM is fine for Qubes, and it
is, in some use cases.
This sort of benchmark will mislead people unless you make it very clear
what limitations are, and what conclusions can be drawn.
Most benchmark suites use a standard load to baseline, and it would be
good to set that out.
Example: 5 open qubes, running standardised tasks

  • browsing, using YouTube, Writing OpenOffice document, displaying PDF,
    downloading file.
    Then the information would be of more relevance in real world cases.

For what it’s worth, I think most people greatly overestimate Qubes
requirements, and underestimate the difficulty in using Qubes on their
brand new hardware.
I often read of people who try to install Qubes without checking the HCL,
or reading any background information. I fear that this benchmark
would also be ignored.

2 Likes
#6

I like your post. In systems with HDDs, the HDD tends to be the bottleneck, but the large amount of time needed to open a network stack + browser (133 seconds!) is eye-opening. There might be something about Qubes that makes it horribly unsuited for running on HDDs.

Fortunately it seems like almost all of today’s laptops come with SSD/eMMC storage, which provides a nice segue into the next point:

In my experience (two new laptops just for Qubes) the challenges are not as bad as they seem. Aside from issues with the Nvidia GPU during installation in one (remedied by modifying xen.cfg in Anaconda) and an unresponsive touchpad + slideshow-like screen refresh in the other (remedied by using the keyboard to download kernel-latest), it was smooth sailing. With newer releases like R4.0.4 including updated kernels, this will be even less of an issue. I can’t speak for AMD PCs though. According to some threads it looks like the latest require a lot of work to get the OS running on cutting-edge AMD PCs.

However, coming from lightweight systems like Ubuntu and Chrome OS, Qubes feels slow even on high-spec modern hardware, which is why even though Qubes requirements are relatively low, higher specifications are key to a smooth experience and should therefore be recommended. In the past I did say that Qubes shouldn’t be too easy to use lest people forget why they use Qubes and slip back into their insecure ways, but nobody wants to wait for more than 10 seconds for a browser to start up.

A test like this is something I had in mind when thinking about a Qubes benchmark. My crude test was just for determining the relationship between CPU and VM start times, which is especially important if your real-world workflow involves starting and stopping lots of VMs often (e.g. heavy segregation and dispVM usage).

This number will also be useful for those looking to get into Qubes. I can’t speak for everyone, but I consulted the HCL a lot when planning my entry into Qubes, and I’m sure information on how processing power/CPU affects something as fundamental as VM start-up time would have been a factor in my decision-making. I’d argue that a benchmark (with all the limitations spelled out) would actually make Qubes less intimidating for the complete newbie.

#7

From Major UX Pain Points.

#8

I can’t believe it’s this simple, but there’s a function called time in bash that allows you to measure the running time of a program, and qvm-start releases the prompt (or whatever the correct terminology is) when debian-10-minimal is displayed, so running time qvm-start debian-10-minimal would give us exactly what we need (just make sure to report only the real time).

In my case, an i7-1065G7 running R4.1 (can’t use R4.0 until R4.0.4 comes out) takes 7.89 seconds.

#9

How is starting time of a VM relevant for the feeling of smooth experience? As I mentioned above, smoothness is a problem with (lack of) GPU acceleration, not loading time. I don’t see what your survey can achieve.