There is no way to validate Qubes Master Signing Key

From a recent Hacker News discussion:

If Qubes ISP is malicious or compromised they can intercept http connections towards Qubes’ servers. This allows them to trivially obtain a SSL cert for keys.qubes-os.org, etc.

Armed with a valid SSL cert, they MITM traffic. When a target downloads qubes they give them a tampered version, when a target downloads the master key, they give them a lookalike key.

The user can happily verify the tampered qubes with the lookalike key.

PGP proves nothing if you cannot verify that you have an authentic copy of the key.

Qubes gives some handwavy suggestions at how to check the key which do not work.

The first item “Use the PGP Web of Trust.” cannot be done because the key isn’t signed by anyone . The other suggestions are inapplicable or won’t achieve anything if the target’s network is being tampered with.

This is not some newfangled problem. PGP has the ability to sign keys for precisely this reason. The main Qubes developers should have personal PGP keys which they use to sign the master key. Their personal keys should be certified by other FOSS developers that they’ve met (maybe the master key too). Then people who are interested in obtaining high confidence in the key can inspect the chain from their own key to the qubes masterkey.

Obviously not everyone will perform careful validation, but if some do then substituting the key becomes riskier. Unfortunately not only is the key not verifiable at the moment, but the current situation is looks pretty similar to what an ongoing key replacement attack would look like.

I am not knowledgeable enough to answer this, so I am curious what the reasoning behind the Qubes signing procedure is.

Relevant documentation I’ve consulted:


Well, without knowing why the commenter thinks that the suggestions
“do not work”, it’s difficult to answer them.

The suggestions for getting the key are:

  1. Get from keys.qubes-os.org
  2. Get it from a keyserver
  3. Get it from the qubes-security pack (currently on GitHub
  4. Get it from the mailing list archives.

Once a user has the Key, it is recommended that they verify the
fingerprint: different approaches are given.
Those that address the commenter’s concern about keys.qubes-os.org are:
checking various keyservers, various websites, (including github)
fingerprints from PDFs, mail archives, github.
So even if the Qubes servers are compromised or traffic is
intercepted, it’s trivial to demonstrate that any downloaded key is not
kosher. That’s why the documentation goes in to detail about this.

The comment is made that “The other suggestions are inapplicable or
won’t achieve anything if the target’s network is being tampered with”,
which is precisely why the advice is to use Tor, VPNs, different ISPs,
and to repeat from different devices. The commenter seems to have missed
that.

How any of this makes the key “not verifiable” is beyond me.

1 Like

I missed the bogus claim that the key isn’t signed by anyone. On the
contrary, it’s trivial to find copies of the key signed by various folk,
so using the Web of Trust is possible.
The fact that the commenter thinks this isn’t so speaks volumes.

1 Like

Also keep in mind that the Qubes team handed out t-shirts with the fingerprint at least once during Chaos Communication Congress :wink: . While this of course is clever marketing, for me it also shows that folks actually thought about how to securely spread the key fingerprint.
I am quite certain that the Qubes assembly at CCC was benign and the folks there were actually affiliated with the project and handed out legitimate fingerprints, although I of course don’t know them personally.

Since I doubt someone will break into my appartment to replace the t-shirt in my wardrobe, I am confident that I know the actual fingerprint.

The more often such things happen (handing out fingerprints as merchandise, actually meeting people and verifying keys and signatures) the more thorough the Web of Trust gets.

Yes, the Master Key is, as far as I can see, not signed by any of the developer keys. Instead, they are signed by it (see, e.g., one of marmarek’s keys: http://keys.gnupg.net/pks/lookup?op=vindex&fingerprint=on&search=0x063938BA42CFA724).
To me this makes sense, as the master key is considered “the root of everything Qubes-related”.

3 Likes

@unman covered everything.

One addition: I imagine the reason is similar for other members of the Qubes team, but speaking for myself, the reason I haven’t directly signed the Qubes Master Signing Key (QMSK) is the same reason I haven’t directly signed any keys, namely because my master key is in a secure vault VM that I never import anything into. I don’t even expose my master key to my Split GPG backend. I only export my subkeys there (see here for details). This means that I can’t directly sign the QMSK with my own master key. Instead, I simply post a clearsigned text file that contains the QMSK fingerprtint.

(This has all come up before, by the way.)

This brings a whole new meaning to “evil maid attack.” :thinking:

1 Like

In case anyone still doubts this without even bothering to check, here are some easy keyserver links showing a bunch of signatures on the QMSK:

Edit: It just occurred to me that perhaps the reason the commenter thinks that the QMSK has no signatures is due to a recent change in GnuPG itself. As you may recall, back in mid-2019 there was a certificate flooding attack. In response, the GnuPG developers released a new version that ignores all key signatures received from keyservers by default. This resulted in unexpected behavior for some users, who helped us to update our verifying signatures documentation to add --keyserver-options no-self-sigs-only,no-import-clean. If the commenter was not using these keyserver options when receiving the QMSK (or any key, for that matter) from a keyserver, it would appear that the QMSK (or any key) has no signatures at all. However, with these keyserver options, gpg receives the QMSK along with 70+ key signatures on it.

Found a photo of the T-shirt, by the way:

The thread I linked earlier also has a bunch of links to the fingerprint in different places around the web.

1 Like

Ahh, the t-shirt and its resonably funny spelling error in all its glory ;). It could almost be considered a security feature. The evil maid would need to make the same spelling error, otherwise the counterfeit t-shirt could easily be identified. :wink:

2 Likes

By the way, here’s the Qubes Master Signing Key fingerprint:

pub   4096R/36879494 2010-04-01
      Key fingerprint = 427F 11FD 0FAA 4B08 0123  F01C DDFA 1A3E 3687 9494
uid   Qubes Master Signing Key
1 Like

Not sure what exactly the problem is, but:

Signatures should go from the devs keys to the master key, not the other way around. Any other method of verification makes zero sense, it is bizarre the Qubes folks would think otherwise.

Beyond being bizarre, it makes it look compromised.

(I’m not suggesting that it is-- it’s just an extremely bad failure mode when you undermine your users ability to detect key substitution attacks by always looking like a key substitution is in progress).

Then again, it’s not just them. OpenSSL did this previously too-- replaced their key and the only source for the new key was the same https site as the software, and the key was signed by nothing and it stayed this way for a month.

During which time most Linux distributions shipped the new update, which presumably means that they’re not doing due diligence either. OpenSSL promptly fixed it after I reported that I couldn’t validate their key.

Fedora itself now has a nearly non-verifyable key, unlike qubes they don’t even have a master key that signs their release keys or sign new release keys with old ones. Presumably Qubes picked up their bad practices from Fedora.

It used to be that various developers at redhat would sign the fedora keys and post the signatures to the keyservers, but the DOS attacks on keyservers mean they can’t do this anymore.

From here: https://news.ycombinator.com/item?id=25054699.

Apparently, a few people have their doubts concerning the current situation with the Master Key. Perhaps docs should have more information about it.

Not sure what exactly the problem is, but:

Signatures should go from the devs keys to the master key, not the other way around. Any other method of verification makes zero sense, it is bizarre the Qubes folks would think otherwise.

Beyond being bizarre, it makes it look compromised.

(I’m not suggesting that it is-- it’s just an extremely bad failure mode when you undermine your users ability to detect key substitution attacks by always looking like a key substitution is in progress).

Then again, it’s not just them. OpenSSL did this previously too-- replaced their key and the only source for the new key was the same https site as the software, and the key was signed by nothing and it stayed this way for a month.

During which time most Linux distributions shipped the new update, which presumably means that they’re not doing due diligence either. OpenSSL promptly fixed it after I reported that I couldn’t validate their key.

Fedora itself now has a nearly non-verifyable key, unlike qubes they don’t even have a master key that signs their release keys or sign new release keys with old ones. Presumably Qubes picked up their bad practices from Fedora.

It used to be that various developers at redhat would sign the fedora keys and post the signatures to the keyservers, but the DOS attacks on keyservers mean they can’t do this anymore.

From here: https://news.ycombinator.com/item?id=25054699.

Not sure what exactly the problem is, but:

The problem is that they have a mental model of how keys should be used,
and cannot think beyond it.

Signatures should go from the devs keys to the master key, not the other way around. Any other method of verification makes zero sense, it is bizarre the Qubes folks would think otherwise.
Why “should” they? Why does it make zero sense?

Beyond being bizarre, it makes it look compromised.

Why does it make it look compromised? Because a key that isn’t signed by
the devs looks compromised.
This is just bizarre thinking.

Of course, there may be good arguments here - it’s just that I cant
see any.
And since the original comment that was quoted was just wrong in many
respects, I wouldn’t spend too much time one this.

1 Like

They have their doubts because, frankly, they don’t understand what they’re talking about.

In order to have a productive debate about this, they (or someone on their behalf) needs to advance specific arguments, not just vague insinuations and passive-aggressive ad hominem quips.

So far, all of the specific arguments they’ve advanced have been cleanly rebutted. For example, one argument was that the QMSK cannot be authenticated because no one has signed it at all (i.e., there are no signatures on the key at all). This has been proven false in two ways:

  1. There are actually 70+ signatures on the key (see above).
  2. Key signatures are not the only way to authenticate a key. If you can learn the genuine fingerprint (through any means), you can always use that to authenticate the key, since the fingerprint is cryptographically unique.

Again, this is just one example of an argument that they actually made and that we’ve already debunked.

Remember, any anonymous entity can spread misinformation and sow FUD by making vague, baseless accusations while sounding like they know what they’re talking about. This is not uncommon on the internet.

If there is any actual problem with the way we handle cryptographic signatures (or worthwhile improvement to be made), we’ll be happy to hear it and act on it, as we have for many years. Our track record proves that.

I just recently updated the page (in response to this very thread) to be more hand-holdy for people who don’t know much about this topic. Here’s a direct link to the most relevant section:

As always, we welcome feedback and contributions to help improve the documentation.

2 Likes

For the technically challenged user, is it possible to order a CD/DVD disk with all the needed keys/fingerprints and etc. from Qubes? Perhaps an app can be included that auto-checks uncertain files with these trusted keys? For a small fee and includes postal charges of course.

It’s amazing how careful Qubes OS team is and how open you are to the users’ suggestions and questions, both here and in the GitHub issues. Thank you so much!

2 Likes

This has come up many times before. There are two main problems:

  1. We can’t manufacture, warehouse, and ship physical inventory around the globe. We simply don’t have the capacity for it. Our forte is software, not selling physical merchandise. We could outsource it, but that exacerbates the second problem.

  2. It’s very difficult to ensure the physical security of such objects, especially if we outsource the manufacturing, warehousing, and shipping to a third party. Without some reasonable assurance that the physical object hasn’t been tampered with (or simply replaced) before it reaches you, it is practically useless as a means of key authentication.

Can’t we check the signature of a DVD? Or it’s hash? You distrust the infrastructure according to the FAQ.

There is some talk about the idea of making a tool like the “tails verification extension” to help not-so-technical users verify the downloaded files.

1 Like

Sure you can, but then it just pushes the problem back one step:

Need to verify ISO -> Get DVD -> Need to verify DVD

Yes, and…?

You probably did not understand what I mean. Can we check the signature of the whole information on the DVD, just like we check an .iso file? Or check the hash of this information. Shouldn’t it coincide with the hash of the .iso? (Or at least I expect it to be predictable).

I understood you perfectly well. My answer stands, same as above.

I’m puzzled about why you think I’m not understanding you. I believe I’m giving very clear and direct answers to your questions, yet there’s clearly some kind of gap in communication here. Perhaps it will help if I recap the discussion thus far, beginning with the context in which the DVD was brought up in the first place:

In other words, the DVD is supposed to serve as a secure root of trust for users. Rather than having to use the web of trust or learn the genuine QMSK fingerprint on their own, download keys and signature files, then using GnuPG or a similar tool to verify signatures, users would instead be able to get a DVD that contains everything they need to authenticate Qubes files, including ISOs.

That’s the theory. However, as I explained above, it is unlikely to work in practice. An attacker could replace the DVD while it is in transit or replace files at the facility where the DVDs are burned, or countless other things. A professional-looking Qubes logo printed on a DVD provides zero security. Anyone can do that.

Therefore, we would not be able to automatically trust that some DVD with a Qubes logo printed on it that came from some other part of the world is genuine. So, we’ll have to authenticate the DVD when we receive it. But notice now that we have simply pushed the original problem back by one step. Our original problem is that we had to authenticate the Qubes ISO. The proposed solution was to ship a DVD containing everything needed to authenticate the ISO. (Alternatively, it could simply be the desired Qubes ISO burned onto a DVD.) But now we have to authenticate the DVD. We have not solved the original problem. We have simply relocated it.

Now, it is certainly possible to authenticate the DVD (using essentially the same means as we would use to authenticate the ISO), but it is a pointless extra step. In that case, we might as well cut out the middle man and directly authenticate the ISO, which was our goal in the first place.

In the abstract, downloading an ISO over the internet and receiving a DVD in the mail are both means of receiving data through an untrusted infrastructure. Therefore, it should not be surprising that the same general security properties apply to both.