Alt RPM Distro (CentOS?) in dom0

General Discussion
#1

I wanted to continue discussion from the Debian in dom0 and Alt RPM Distro (CentOS, RHEL, Suse, Oracle Linux) in dom0 tickets. To keep discussion on track, this thread is to discuss RPM based distros only.

Recap

The short Fedora release cycle is a drag on development, but sticking to a Red Hat adjacent distro has many benefits:

  • A lot less work than switching to Debian or Nix.
  • Efficiency gains from using the same distro in dom0 and VMs (via UBI or Fedora CoreOS/IoT).
  • A lot of money is poured into RHEL compatibility.
  • We can use Gitian to make the existing system reproducible.

CentOS

CentOS Stream surfaced as a top choice for a few reasons:

  • CentOS signs both packages and repo metadata.
  • The CentOS community is repositioning itself as the integration point for multiple RHEL derivatives.
    • Stream roughly correlates to latest minor release of RHEL.
    • CI pipeline Fedora -> Stream -> RHEL.
    • Not RHEL-beta: Red Hat, FB, AWS, Oracle, VMWare, SuSE, etc will still deliver some patches to their own customers before they are cherry picked for CentOS.
  • New major version every 3 years, support for 5 years.

The biggest sticking point was that RHEL/CentOS has fewer packages than Fedora. The push to CentOS Stream is about streamlining the pipeline from Fedora to RHEL, so it should get easier to push Fedora packages we rely on downstream. I don’t know much about their community, but AFAICT CentOS Special Interest Groups are independent from Red Hat. The Virtualization SIG was highlighted in various blogposts and is probably the best place to start.

However, how much of that could be solved feature gating development? I was able to find many of the packages from @fepitre’s coprs (epel-8-qubes, epel-8-python38) in EPEL and AppStreams (see 8.3.2011 or search via pkgs.org). Some vendors (like Salt) want you to use their repos directly. I was able to find even more (albiet older) packages on CentOS’s Pagure/Git hosting.

Silverblue

Once the GUI layer gets moved out of dom0, the dependency on OS packaging becomes much smaller. At that point Fedora Silverblue is likely the next best choice. I won’t recap explain all of Silverblue here, but it is a container focused immutable OS that uses Git to manage the OS layer. But we can still install and use DNF, so the transition would be easier than a purely functional distro like Nix.

#2

My reading of Streanm differs from yours.
Centos Stream does not “roughly correlate[s] to latest
minor release of RHEL” - it’s intended to be always slightly ahead
of the latest release.
Centos Stream will be a development project for RHEL.
I don’t think that dom0 should be based on anything but a rock steady,
slow moving distro.

#3

+1 second that.
Always thought dom0 should have been Centos, since dom0 is really mostly a server, and does not run apps.
Centos stream will be upstream from RHEL, nowhere near as far upstream as Fedora, so a better candidate for dom0 IMHO. Qubes should stick with RH as that will have best chance for enterprise acceptance. Centos replacements like Rocky will probably come to the same end, as RH can make it arbitrarily hard (expensive) to follow.

#4

So I fell down a rabbit hole a while back researching a related question: how large of a delay is there between security fixes being released in RHEL|mainline and uptake by Fedora or CentOS? Answer: it’s a trick question. Fixes originate from any number of places and circulate between Linux distros in a non-linear fashion before finally hitting your machine.

Since clones were already using CentOS as a synchronization point for patches that were (fingers crossed) close enough to RHEL they decided to just formalize the arraignment and switch CentOS’s nominal designation from being “downstream” to being “upstream”.

Back to your assertion: we already are already about as far upstream from RHEL as you can get. So we wouldn’t be losing stability. If you buy Oracle’s marketing bullshit of 100% binary compatibility (like I did) switching over requires fiddling with your DNF settings. But it’s a lie, they have to reverse engineer code drops from RHEL just like everyone else (old school CentOS included).

1 Like
#5

I very much like what Marek wrote earlier this year:

[…] at the point when we’ll be minimizing dom0, more likely option
would be something capable of building light system images, for
example Yocto."

#6

" Back to your assertion" - don’t know which assertion you refer to.
Since Centos Stream is a rolling distribution, (rolling-minor, ha)
it cant be a serious candidate for dom0, regardless of its weight.

#7

I doubt @marmarek read Yocto’s security page:

Yocto Project does not have a Security team … there is some research and proof of concept work occurring with some tools but its struggling due to lack of people/resources.

And since this thread is about RPM distros … what do you think about SilverBlue/CoreOS or SuSE MicroOS? SilverBlue and CoreOS leverage git to provide immutability, de-duplication, and rollbacks but leaves us the option to use DNF, which would make the transition easier. I know those aren’t the only alternatives, but be wary of marketing fluff:

…

This one:

I’m skeptical that it will be any less stable than Fedora or what we can get for free from volunteers reverse engineering code drops from RHEL, like CentOS used to do. It also means security fixes take 24-72+ hours to hit the update servers. Seriously, go read the CentOS Blog about the new integrated release engineering process:

Whatever work that is under NDA or embargo, that will eventually get reverse engineered by Facebook (or whomever) and pushed onto CentOS Stream.

Do you have any input on letting users choose between CentOS Stream and a RHEL clone like Oracle or Rocky Linux?

Fedora Silverblue, CentOS Stream, or a RHEL clone all seem like acceptable options to me. Silverblue has the potential to reduce breakage, but I think that would have to wait until the GUI is out of dom0. The primary objection to RHEL was that some packages don’t work and it is stuck on old software. But with a major release being cut every three years from Fedora via CI, it seems like a viable option.

Or at least @DemiMarie expressed a preference for CentOS:

I think that CentOS would be a good choice for dom0, as it can have a recent kernel, yet is extremely stable and has signed packages and signed metadata.

And @fepitre is the guy who handles CentOS stuff and mentioned CentOS Stream:

dom0 is in a rather weird position. It needs to be stable, so that changes don’t catch the Qubes core team by surprise, but it needs the latest hardware support and installer. I doubt there are any existing distros that are good fits for us. A derivative of CentOS 8 is the best option I know of, but a fully custom distro would be ideal.

I agree. I’m seeing that CentOS 8 Stream has moved forward a lot. I could redo some tests but well, that would deserves to be a specific task and @marmarek would need to arbitrate this.

Please stick to constructive feedback.

#8

The Problem with dnf Always come back

I think its better leave Fedora yo a more Security distro or Os for dom0

#10

I doubt @marmarek read Yocto’s security
page

Hm, that kind of misses the point. Let me try to be more verbose:

What does dom0 do currently?

  • hosts the GUI (XFCE)
  • hosts the audio subsystem (Pulse)
  • Kernel
  • XEN
  • other hardware is already isolated in sys-net and sys-usb

I think what makes most posters so passionate about which distro runs in
dom0 is their wish to have a different or more up-to-date GUI
environment. That goes away with the GUI VM.

What’s left? What does dom0 do in future?

  • Kernel
  • XEN
  • Qubes specific binaries

This is what I understand when I read “minimizing dom0”.

–> There is no longer a need for a distro. <–

Maybe I am wrong, but that is what I am expecting to happen and it looks
more secure to me than trusting any distro outside the Qubes project.

1 Like
#11

Experience shipping production systems matters. Alpine was shipping ‘null’ as the default password for years.

A “distro” provides us with a critical mass of support, both in terms of eyeballs on the code and support from hardware vendors. Community distros haven’t poured millions of dollars into testing and delivery infrastructure.

And aside from being shiny, what does Yocto or Alpine do that CoreOS doesn’t?

#12

And aside from being shiny, what does Yocto or Alpine do that CoreOS doesn’t?

Again, I am not an expert. But the point as I understand it is the
reverse of your question. The goal would be a dom0 that is extremely
minimal, which means: reduced attach surface.

Currently we need a distro in dom0 because the GUI lives in dom0. Once
that is no longer the case, there should be no reason left for any user
to interact with dom0 in any way. There shouldn’t be anything anybody
would want to install, configure or maintain in dom0. It should just be
the Kernel, XEN and Qubes daemons.

#13

That’s what CoreOS is all about: reducing the base system to the maximum extent possible. Red Hat acquihired the leading container oriented distro and brought a bunch of process enhancements (like better release QA and back-porting) to make it more suitable for use in production.

The idea that Qubes doesn’t need a base distro is silly: Linux is just a kernel and it doesn’t make any internal stability guarantees. A distro is a coalescence of disparate upstream sources, which is a lot of work. Anyone can roll-their-own Linux distro badly, but its much easier to draft off of Debian or Red Hat.

Yocto and Alpine advertising themselves as minimal is less honest than the sticker price on a new car. 2.5 megabytes won’t get you running on Xen or boot a laptop. After you shove in everything required to actually do anything Yocto and Alpine are the same size as everyone else. The main difference is that they don’t have nearly as many resources to make sure it all works correctly.

Even if dom0 doesn’t require a GUI, we need a GUI somewhere and it’s more efficient if we use the same base system in dom0 as we do in the virtual machines. At a minimum, you get space savings by de-duplicating what’s on disk. But compatibility is the biggest issue: we literally can’t use CentOS right now because the userland is not in sync with Fedora and no one else has bothered to backport everything. Which leaves us stuck with a lot of extra porting work.

Ideally, we will someday get to use SeL4 and run everything in WASI isolates. But Qubes is not a research project and it needs to run on real hardware.

#14

it’s more efficient if we use the same base system in dom0 as we do
in the virtual machines.

How and why?

At a minimum, you get space savings by de-duplicating what’s on
disk.

Shared files between dom0 and a domU? I don’t think so.

But compatibility is the biggest issue: we literally can’t use
CentOS right now because the userland is not in sync with Fedora and
no one else has bothered to backport everything. Which leaves us
stuck with a lot of extra porting work.

I see. You want to use CentOS as domU and are frustrated by the amount
of work that needs to be done to get there. I can empathize with that.

That doesn’t make it a good or better choice than what we already have.
In addition having a rolling distro by definition would be the opposite
of what we need in dom0. There should only be very very tightly managed,
infrequent and necessary (security & bug fix) updates to dom0 after release.

I lived on a rolling distro before (SuSe Tumbleweed). It was fun but not
stable in any sense of the word.

/Sven

#15

For the same reason that it’s more efficient to target a single execution environment in any other setting, be it browsers or specific versions of specific operating systems.

Like the Linux kernel or userland. You know, whatever is in the 130 megabytes of the Alpine Xen image ; )

No, I am not. I am talking about using the exact same version of hundreds of different pieces of software that makeup a distro. For example, there are at least 75 packages (see: [1, 2]) that Qubes uses in Fedora which are not available by default in RHEL.

What we have is Fedora, which is only stable for 1 year and porting between major releases is a resource drain on the project.

I will repeat this point one last time: you don’t have to use CentOS Stream. The project could use Rocky Linux or any other RHEL clone.

CentOS Stream is downstream of Fedora and represents a minor point release for RHEL. So no major breaking changes, just new features and bugfixes. I’ve had Fedora stop booting on some laptops because it will switch between kernels without a major version bump. So I would imagine that CentOS Stream would be more stable than Fedora.

@Sven I’m not finding any of your responses productive. You advocated for a minimalist distro, then switched to expressing dislike for CentOS Stream. I already addressed most of the concerns you raised, either in this thread or in the Github tickets. And you don’t discuss alternatives that are based on an RPM or RHEL adjacent distro.

I get the impression you do not like Red Hat, which is fine. But this thread is focusing on RPM based distros. Not because I am a Red Hat fanboy, but because it means we don’t have to replace all of our infrastructure at once. Some non-RHEL adjacent distro may be better for dom0. That’s fine, but please discuss that distro in another thread.

That being said, I’m not wasting any more time entertaining low effort comments. I will flag anymore off-topic comments as such.

Thank you!

#16

It’s not “off-topic”, “low effort”, or “not constructive” to question
the assumptions underlying this thread or your assertion that:
CentOS Stream surfaced as a top choice for a few reasons

In fact, it seems that following those discussions is far more
productive than a narrow focus on “what RPM distro”, and productive in
the long run.
Identifying what needs to be in dom0, researching possible candidates,
and the work that would need to be done, that’s useful. This isn’t.

#17

@Sven I’m not finding any of your responses productive.

I am sorry to hear that.

It was my intention to share my understanding, which is that the choice
of distro in dom0 has very little impact beyond hardware support now and
will matter even less in the future.

We have Fedora now for historical reasons. The project lead pointed out
that once dom0 is minimized, we might get rid of Fedora in dom0 and
switch to something minimal.

In this context I recognized that we perceived things very differently
and it might be interesting to have a conversation to uncover incorrect
assumptions.

I get the impression you do not like Red Hat, which is fine.

My preferences boil down to a specific look & feel and stability (no
updates except security patches). I couldn’t care less about the distro
or the packaging format in domU.

In dom0 I want no changes or user facing features at all unless they
significantly improve security.

this thread is focusing on RPM based distro not because I am a
Red Hat fanboy but because it means we don’t have to replace all of
our infrastructure at once. Some non-RHEL adjacent distro may be
better for dom0. That’s fine, but please discuss that distro in
another thread.

I understood that from the start and tried to reason with you why
arguing for a distro change in dom0 might not be the best use of your
attention.

#18

I largely started this thread and the Alt RPM Distro in dom0 ticket as a brain dump after doing a bunch of research. I was willing to rehash discussion from Github because I thought it would be helpful to have the discussion all in one place. But now it feels as if we are retreading ground already covered in this thread.

The vast majority of stuff in a Linux kernel is for hardware support. What is the testing infrastructure like for most distros? I would be surprised if anyone outside of Debian and CentOS have a good hardware testing infrastructure.

Define minimal: you can play at the margins, like replacing some GNU userland stuff with Busybox or a smaller standard C library. But those alternatives compromise on features and aren’t going to be as well maintained. Since the application VMs will need the standard GNU userland and C library, we might as well just use the same distro in dom0 and dedupe the base image. At that point, it feels a lot like arguing about sports teams: it’s all the same code, just with different levels of integration and infrastructure.

What couldn’t we accomplish using a Fedora Spin, CentOS Variant, or even a Debian Derivative? SilverBlue/CoreOS makes this very easy. Then we can push Qubes specific software into the distribution’s continuous integration system, with all the benefits that provides :thinking:.

#19

Wanted to highlight two developments:

  1. Red Hat has expanded their free tier to 16 RHEL licenses for use in production; users unhappy with CentOS Stream could switch to RHEL for both dom0 and 15 appVMs without paying anything.
  2. Red Hat has a free for FOSS infrastructure program; Qubes developers shouldn’t need to worry about bumping up against the 16 cap of the aforementioned program.

I also realized that dom0 gets updated between ~1 and ~3.5 years (~1.9 years on average). If Red Hat sticks to their revamped 3 year release cycle, we wouldn’t be stuck on old kernels much longer than what we are used to. The CentOS Hyperscale SIG provides accelerated backports (slides) of specific items (like systemd) and is working on backporting LTS kernels as well (which are released ~yearly). So it should be feasible to keep dom0 on a fresher kernel in future.

Alt Distro in dom0
Alt Distro in dom0
#20

I think this could be something interesting to post on the qubes-devel mailing list so it more easily reaches the developers (only a few see the forum).

#21

If you feel like it’s gotten to the point of being useful, feel free to ping them :slight_smile:. I’ve also started up this meta-thread.

1 Like