Nvidia driver installation

User Support Testing 4.1
#1

Hello,

I am using Qubes 4.1 with kernel 5.8.16, but I am not sure if this issue belongs to Testing 4.1.
Currently I am trying to install nvidia drivers. I’ll summarize here a few issues I’ve encountered so as not to create too many posts.

First, I tried to apply this docu (https://github.com/Qubes-Community/Contents/blob/master/docs/configuration/install-nvidia-driver.md).

In a fedora-32 standalone VM I first did

sudo dnf install https://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-(rpm -E %fedora).noarch.rpm sudo dnf install https://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-(rpm -E %fedora).noarch.rpm

and get stuck with the error:

[user@dev-testing ~]$ yumdownloader --resolve xorg-x11-drv-nvidia
Last metadata expiration check: 0:44:43 ago on Wed Dec 30 12:07:54 2020.
No package xorg-x11-drv-nvidia available.
Exiting due to strict setting.

To get on with it already, I then looked at the next steps: In most forums there is something about editing the grub/efi/xen config file.
However, I can’t find such anywhere.

Neither in

[root@dom0 boot]# ls -R grub2/
grub2/:
grubenv  themes

grub2/themes:
qubes

grub2/themes/qubes:
progress_bar2_c.png    qubes.png            unifont-regular-14.pf2
progress_bar_c.png     theme.txt            unifont-regular-16.pf2
progress_bar_hl_c.png  unifont-bold-16.pf2  unifont-regular-32.pf2

or

[root@dom0 boot]# ls -R efi
efi:
EFI

efi/EFI:
BOOT  fedora  qubes

efi/EFI/BOOT:

efi/EFI/fedora:

efi/EFI/qubes:
fonts  grub.cfg  grubx64.efi  xen-4.14.0.efi  xen.efi

efi/EFI/qubes/fonts:

nor

[root@dom0 etc]# ls | grep grub
grub2.cfg
grub2-efi.cfg
grub.d

I find corresponding config files. How do I continue?

Edit: My system spec are listed in Qubes 4.1 installation failure.

Edit: I followed the steps in

and downloaded the related rpmfusion files (i downloaded and installed
https://download1.rpmfusion.org/free/fedora/releases/32/Everything/x86_64/os/repoview/rpmfusion-free-release.html and https://download1.rpmfusion.org/nonfree/fedora/releases/32/Everything/x86_64/os/repoview/rpmfusion-nonfree-release.html since my dom0 has fedora 32.

Then I tried

[root@dom0 Downloads]# sudo qubes-dom0-update akmod-nvidia
Using debian-clone as UpdateVM to download updates for Dom0; this may take some time…
Cannot retrieve repository metadata (repomd.xml) for repository: rpmfusion-free. Please verify its path and try again

and there is the next obstacle.

Deletion policy
#2

This was my approach based on issue #2526 and works on kernel-5.9.14-1:

  1. Install the necessary tools:
    dom0: qubes-dom0-update gcc kmod grub2-tools perl-bignum make
  2. Install kernel-devel package from RPM file;
    dom0: dnf install kernel-latest-devel-5.9.14-1.qubes.x86_64.rpm
  3. Download the latest Nvidia driver from www.nvidia.com/en-us/geforce/drivers/
    In my case “NVIDIA-Linux-x86_64-455.45.01.run”
  4. Copy the downloaded driver to dom0:
    dom0: qvm-run -p ${VM} 'cat /home/user/Downloads/NVIDIA-Linux-x86_64-455.45.01.run' > NVIDIA-Linux-x86_64-455.45.01.run
    dom0: chmod +x NVIDIA-Linux-x86_64-455.45.01.run
  5. Extract driver sources
    dom0: ./NVIDIA-Linux-x86_64-455.45.01.run --ui=none --no-x-check --keep --extract-only
  6. Build nvidia.ko kernel driver
    dom0: cd NVIDIA-*/kernel/; make module IGNORE_XEN_PRESENCE=y CC="gcc -DNV_VMAP_4_PRESENT -DNV_SIGNAL_STRUCT_RLIM"
  7. Copy compiled driver to /lib/modules/$(uname -r)/extra
    dom0: sudo cp nvidia.ko /lib/modules/$(uname -r)/extra/
  8. Load the driver and check
    dom0: sudo depmod -a; modinfo nvidia
  9. Edit grub2 entry add rd.driver.blacklist=nouveau to the end of GRUB_CMDLINE_LINUX
    dom0: sudo vim /etc/sysconfig/grub
  10. Update grub.cfg for UEFI
    dom0: grub2-mkconfig -o /boot/efi/EFI/qubes/grub.cfg
  11. Disable nouveau driver; add to blacklist
    dom0: echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
  12. Reboot the system and enjoy the Nvidia driver

I’ve uploaded my build kernel rpm files to my git repository: GitHub Repo

3 Likes
#3

Hi, thanks for the detailed response!
It’s working fine until step 6:

[user@dom0 Downloads]$ cd NVIDIA-*/kernel/; make module IGNORE_XEN_PRESENCE=y CC="gcc -DNV_VMAP_4_PRESENT -DNV_SIGNAL_STRUCT_RLIM"
make[1]: *** /lib/modules/5.9.14-1.qubes.x86_64/build: No such file or directory.  Stop.
make: *** [Makefile:80: modules] Error 2

Before I run this command I rebooted and verified

    [user@dom0 Downloads]$ uname -r
    5.9.14-1.qubes.x86_64

Then I checked the folder

[root@dom0 5.9.14-1.qubes.x86_64]# ll /lib/modules/5.9.14-1.qubes.x86_64/
total 4612
lrwxrwxrwx  1 root root      38 Dec 15 01:00 build -> /usr/src/kernels/5.9.14-1.qubes.x86_64
drwxr-xr-x  3 root root    4096 Jan 11 22:06 extra
drwxr-xr-x 13 root root    4096 Jan 11 22:07 kernel
-rw-r--r--  1 root root 1150198 Jan 11 22:07 modules.alias
-rw-r--r--  1 root root 1125299 Jan 11 22:07 modules.alias.bin
-rw-r--r--  1 root root    1550 Dec 15 01:00 modules.block
-rw-r--r--  1 root root    9473 Dec 15 01:00 modules.builtin
-rw-r--r--  1 root root   25490 Jan 11 22:07 modules.builtin.alias.bin
-rw-r--r--  1 root root   12074 Jan 11 22:07 modules.builtin.bin
-rw-r--r--  1 root root   74116 Dec 15 01:00 modules.builtin.modinfo
-rw-r--r--  1 root root  415504 Jan 11 22:07 modules.dep
-rw-r--r--  1 root root  583209 Jan 11 22:07 modules.dep.bin
-rw-r--r--  1 root root     375 Jan 11 22:07 modules.devname
-rw-r--r--  1 root root     176 Dec 15 01:00 modules.drm
-rw-r--r--  1 root root      34 Dec 15 01:00 modules.modesetting
-rw-r--r--  1 root root    2215 Dec 15 01:00 modules.networking
-rw-r--r--  1 root root  149542 Dec 15 01:00 modules.order
-rw-r--r--  1 root root     738 Jan 11 22:07 modules.softdep
-rw-r--r--  1 root root  501178 Jan 11 22:07 modules.symbols
-rw-r--r--  1 root root  609192 Jan 11 22:07 modules.symbols.bin
lrwxrwxrwx  1 root root       5 Dec 15 01:00 source -> build
drwxr-xr-x  2 root root    4096 Dec 15 01:00 updates
drwxr-xr-x  3 root root    4096 Jan 11 22:07 vdso
drwxr-xr-x  2 root root    4096 Dec 15 01:00 weak-updates
[root@dom0 5.9.14-1.qubes.x86_64]#

and noticed that the folder with the symlink is empty.

[root@dom0 kernels]# ll /usr/src/kernels/
total 0

This seems kinda related to known issues. Do you also know how to fix this?
Do I have to do more than dnf install kernel.rpm to upgrade the kernel?

#4

Install kernel-latest-devel-5.9.14-1.qubes.x86_64.rpm should fix the problem. This package include the files:

rpm -ql kernel-latest-devel-5.9.14-1.qubes.x86_64.rpm
[...]
/usr/src/kernels/5.9.14-1.qubes.x86_64/usr/include/Makefile
/usr/src/kernels/5.9.14-1.qubes.x86_64/virt
/usr/src/kernels/5.9.14-1.qubes.x86_64/virt/Makefile
/usr/src/kernels/5.9.14-1.qubes.x86_64/virt/kvm
/usr/src/kernels/5.9.14-1.qubes.x86_64/virt/kvm/Kconfig
/usr/src/kernels/5.9.14-1.qubes.x86_64/virt/lib
/usr/src/kernels/5.9.14-1.qubes.x86_64/virt/lib/Kconfig
/usr/src/kernels/5.9.14-1.qubes.x86_64/virt/lib/Makefile
#5

Same error as before. The folder is still empty

[root@dom0 kernels]# ll /usr/src/kernels/
total 0

Even though the log of
[user@dom0 kernel]$ sudo rpm -ql kernel-latest-devel-5.9.14-1.qubes.x86_64.rpm
[…]
/usr/src/kernels/5.9.14-1.qubes.x86_64/usr/Makefile
/usr/src/kernels/5.9.14-1.qubes.x86_64/usr/include
/usr/src/kernels/5.9.14-1.qubes.x86_64/usr/include/Makefile
/usr/src/kernels/5.9.14-1.qubes.x86_64/virt
/usr/src/kernels/5.9.14-1.qubes.x86_64/virt/Makefile
/usr/src/kernels/5.9.14-1.qubes.x86_64/virt/kvm
/usr/src/kernels/5.9.14-1.qubes.x86_64/virt/kvm/Kconfig
/usr/src/kernels/5.9.14-1.qubes.x86_64/virt/lib
/usr/src/kernels/5.9.14-1.qubes.x86_64/virt/lib/Kconfig
/usr/src/kernels/5.9.14-1.qubes.x86_64/virt/lib/Makefile

Seems to copy in this folder.

[user@dom0 Downloads]$ cd NVIDIA-*/kernel/; make module IGNORE_XEN_PRESENCE=y CC="gcc -DNV_VMAP_4_PRESENT -DNV_SIGNAL_STRUCT_RLIM"
make[1]: *** /lib/modules/5.9.14-1.qubes.x86_64/build: No such file or directory.  Stop.
make: *** [Makefile:80: modules] Error 2
[user@dom0 kernel]$ sudo cd NVIDIA-*/kernel/; make module IGNORE_XEN_PRESENCE=y CC="gcc -DNV_VMAP_4_PRESENT -DNV_SIGNAL_STRUCT_RLIM"
/usr/bin/cd: line 2: cd: NVIDIA-*/kernel/: No such file or directory
make[1]: *** /lib/modules/5.9.14-1.qubes.x86_64/build: No such file or directory.  Stop.
make: *** [Makefile:80: modules] Error 2
#6

Only to be sure; the rpm command that I former posted is just a command to list the content of the rpm package. If you like to install the package you have to use dnf instead :).

#7

Ahhh… and I thought it was an extension somehow. I’m just getting into fedora - was using ubuntu until now.

However

$ sudo dnf install kernel-latest-devel-5.9.14-1.qubes.x86_64.rpm 
Qubes OS Repository for Dom0                                              2.9 MB/s | 3.0 kB     00:00    
Error: 
 Problem: conflicting requests
  - nothing provides elfutils-libelf-devel needed by kernel-latest-devel-1000:5.9.14-1.qubes.x86_64
(try to add '--skip-broken' to skip uninstallable packages)

Shall I use the --skip-broken flag?

Edit: Or download (RPM resource elfutils-libelf-devel) and install it?

#10

Now I am until step 11. but at the echo command i got an error permission denied. weahter I run it with sudo or not. That is bad at the last step:( hope to fix that soon;)

#11

May you stop deleting your previous problems over and over? This post should help others as well.

Just add your new results with an ‘edit:’.

2 Likes
#12

I resolved the issue by simply
sudo qubes-dom0-update elfutils-libelf-devel
I previously tried
sudo dnf install elfutils-libelf-devel
which failed (obviously)

Edit: Step 6 failed aigan with permission denied error
dom0: cd NVIDIA-*/kernel/; make module IGNORE_XEN_PRESENCE=y CC="gcc -DNV_VMAP_4_PRESENT -DNV_SIGNAL_STRUCT_RLIM"
but sudo works fine:
dom0:
cd NVIDIA-*/kernel/
sudo make module IGNORE_XEN_PRESENCE=y CC="gcc -DNV_VMAP_4_PRESENT -DNV_SIGNAL_STRUCT_RLIM"

edit step 7 changes then to
dom0: sudo cp /NVIDIA-*/kernel/nvidia.ko /lib/modules/$(uname -r)/extra/

edit step 11:
I also got the permission denied error on
dom0: sudo echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
however:
dom0: sudo vim /etc/modprobe.d/blacklist.conf works fine.

#13

On the last step there occurs an error while booting:

Usually I’m getting asked with which kernel I want to boot. This screen was omitted and I was directly prompted the disk password. After that the screen stuck at bootup (see image)

IMG_20210120_131437

I had to ctrl+alt+f2 and reset the additional grub parameter to get it working aigan. Do you know what’s happening here?

1 Like
#14

sorry for deleting the post. I will write a summary later. i just taught too much post aren’t good, too. Will change it :wink: I can disable the nouveau, too but cannot boot because of the same error as u, and cannot select the the kernel

and if u press ctrl alt f2 to reset the additonal grub parameter then the nvidia driver isn*t enable or?

sys-whonix failed to boot at ur screen, but I have disable the qube a long time before installing the driver the it don’t boot automatically on start up and be stocked at the sam eror. So sys-whonix shouldnt be the problem. Hm that will be really great to fix it and to use my nvida cards for video/audio …But I think it is just a small step and we can fix it soon.
At step6 I was stucked because of the wrong syntax after "gcc-DNV . Here is just need to delete a space and didnt look correctly at the description. And at the very first I didnt notice the under step 12. he has uploaded the rpm. files to install the kernels. But with kernel 5.9… I have some problems, too and my network is just working over my wirelesscard. The wired connection always says that it is disconected. If I change to a lower kernel it will be working again. Maybe anybody nows how to do that?
Thank you for ur help and sorry again for deleting the posts, but now I summarized all the trial and erors…
another problem at step 11 is that via vim and the path to the .conf file it open a empty file. is that correct and I just need to add the lines there or is there a error before and this file shouldnt be empty?
stay healthy

1 Like
#15

nobody a solution how to get this working? it is really disappointing to be stucked at the last step:D @Rnd3sB3g13rng did u find anything else what can solve this problem?

So I will try that again… I am now at the last step again where I need to edit the grub.cfg and disable noveau driver. If I installed Qubes in legacy mode and don*t have an efi bios then I need to update the etc/default/grub.cfg instead of the efi or doesn’t that matter?

and another problem I noticed is that with kernel 5.9 or 5.10x I only can connect via wifi to the network and sys-net recognizes my network card, but it has not set up the wired connection. maybe I have to set up it manual?

#16

unfortunately I am always getting the same or an similiar error. loading plymouth service …starting hold until boot process… ctrl alt f1 dont bring anything new. I tried it with lightdm and sddm because I hope that the display manager is the problem ,but no… I don’t know what we can do. @dhn u cannot say anythings to this problem? Tried it with ur Kernel 5.9 ,the new Kernel 5.10x but never boots…
have a nice weekend…

Just an overview. I am running 2 x 980m nivdia cards and and intel core i7 -I need the driver running in dom0 or is there a other solution with gpu passthrough for me to the vm? It will be very nice to use the nvidia high definition audio controller and the video cards in a vm. I know that I will weak security but it should be better as dual booting, and it will increase my performance for my working vm. Is there a doc about gpu passthrough ? Please tell me how I can try that.

Now I am trying to run through the description again but always getting 2 make errors, that complier dont support fentry as an error. And I dont know how to fix that again that I can compile the nvida.ko driver. These errors are at ur build 5.9 kernel and the new kernel 5.10x, too

back to kernel 5.9.14 only make is working until the last step the error at the boot screen…
so I am trying to get the newest nvidia driver working and working but always getting stucked wit the boot error. I noticed that maybe it is a problem the the noveau driver ist not blacklisted right ,if we do it with vim ? Because the blacklist.conf is not here and I have to create that file first. What is the correct syntax for blacklist nouveau in that file or what is the best soltuion I can try to blacklist nouveau definitely. Perhaps then it will be working.
@GWeck do u know how to blacklist nouveau?

I did the update to the new kernel today 5.10.13 and there it is exactly the same error and stucked at the bootscreen without running the nvidida drivers. The lightdm display manager could not load however. With sddm there is a black screen only with a blinking _ .So I think it may be a problem with the display managers. But I don’'t know it. Strange that the newest kernel has the same error without nvidia drivers as kernel 5.9.14 trying to run nvidia drivers and add the line to the grub…
The loockscreen to enter the password has some interface errors, too . Ur able to enter ur luks passphrase but it looks completely different and if u enter the pass then it will boot everything fine until the last step booting and showing up display manager failed…so it should not be a problem of the nvida driver only, the kernel could be the problem as well, but without nvidia kernel 5.9.14-10 is runing, just the problem with the wired network I cannot fix until now and have to use wireless lan… maybe anybody knows now what could be the problem.
I am very thankful for ur help, stay healthy and have a nice weekend!

#17

Hey, 1 month later it looks a bit better :wink:
I added rd.driver.blacklist=nouveau in the first GRUB_cmdline_linux ,created the sudo vim /etc/modprobe.d/blacklist.conf with blacklist nouveau written in it and installed the sddm display manager,completely switched to KDE and now I don’t get Starting Hold until boot process finishes up and can boot in the system and it seems that it is working with all my kernels 5.4.98 and higher. To compile the driver I used the kernel 5.9.1-4 he posted here. But I don’t know blacklist=nouveau in the grub line is right and don’t know the command how to check if I am using nvidia driver now. And I have to look what advantages it will bring, and have to test if in my fedora template there is a better video resolution and playing videos in fullscreen is possible now or something like that, that didn’t work before. Will update after testing more but now the steps 12 steps posted on the top @dhn are working.

I even noticed that with kernel 5.10x there was a similiar problem and it didn’t show the luks passwort gui and then hangs on boot. This was fixed by the team of qubes the last weeks and is working now, too. Maybe there was the fix with the plymouth service /display manager and now it is working with lightdm, too. :slight_smile:
BTW: Qubes 4.1 is much more compatible with templates, Hardware and else like my other Qubes 3.2 and 4.0 before and I am running it as a daily work system, because of this. You are amazing ! As soon I’ll have some time the next days I will make and summarized a structured post about installing KDE and and else on 4.1 and how I prefer to customize it for daily and usual ,none development uses.
Update: with lightdm it is working, too. This bug definitely was fixed by the amazing teams of qubes in any kernel update the last weeks. :slight_smile:

Update: I don’t think that with this method dom0 uses nvidia instead of nouveau, because I run glxinfo | egrep “OpenGL vendor|OpenGl renderer” and it says: OpenGL vendor string: nouveau . So it is not using my nvdiida and nouveau is not disabled or? But how to disable it to 100 % ? :wink:
Much blesses
Next I tried to make an overview again how it is working and included the corrected steps and tried to write it more detail for new user :slight_smile:
1. Install the necessary tools:
in dom0: sudo qubes-dom0-update gcc kmod grub2-tools perl-bignum make
2. Install kernel-devel package from RPM files; (Kerneldevel59, kernellatest59, Kernelqubesvm59)
Download from here: qubes-thinkpad-x1-extreme-gen3/rpm at main · dhn/qubes-thinkpad-x1-extreme-gen3 · GitHub and copy files to dom0 and install them.
in dom0: for example sudo dnf install kernel-latest-devel-5.9.14-1.qubes.x86_64.rpm
3. Download the latest Nvidia driver from www.nvidia.com/en-us/geforce/drivers/
In my case “ NVIDIA-Linux-x86_64-460.67.run ” but I suggest you to rename it to nvidia.run for easier use in the command lines then you dont need to write the numbers everytime :wink:
4. Copy the downloaded driver to dom0:
in dom0: qvm-run --pass-io APPVM ‘cat FOLDERAPPVM/FILENAME.file’ > /home/USERNAME/Downloads/FILENAME.file (of course u have to replace APPVM with yours for example personal FolderAPPVM with the path of the file like Downloads and the USERNAME with your one:) But with this command you always can copy files from a vm to dom0
in dom0 chmod +x NVIDIA-Linux-x86_64-455.45.01.run (or nivida.run or whatever u renamed it:)
5. Extract driver sources
in dom0: ./NVIDIA-Linux-x86_64-455.45.01.run --ui=none --no-x-check --keep --extract-only
6. Build nvidia.ko kernel driver
in dom0: Enter the folder NVIDIA-*/kernel/ with cd command or in file manager and then run
sudo make module IGNORE_XEN_PRESENCE=y CC=“gcc -DNV_VMAP_4_PRESENT -DNV_SIGNAL_STRUCT_RLIM”
7. Copy compiled driver to /lib/modules/$(uname -r)/extra
in dom0: sudo cp nvidia.ko /lib/modules/$(uname -r)/extra/
8. Load the driver and check
in dom0 sudo depmod -a; modinfo nvidia
9. Edit grub2 entry add rd.driver.blacklist=nouveau to the end of GRUB_CMDLINE_LINUX (the first grub_cmdline_linux)
in dom0: sudo vim /etc/sysconfig/grub
10. Update grub.cfg for UEFI
in dom0: grub2-mkconfig -o /boot/efi/EFI/qubes/grub.cfg
11. Disable nouveau driver; add to blacklist ; frist run sudo -i for root before then this step is working fine,too:)
in dom0: echo “blacklist nouveau” >> /etc/modprobe.d/blacklist.conf"
12. Reboot the system and enjoy the Nvidia driver

Today I run a qubes-dom0-update and in the stable repo is the kernel 5.11x now. This unfortunately is not working anymore and has the same error as everytime hanging on boot and with the wrong luks gui. Of course it destroyed the setup of my kernel 5.9.14 with nvidia,too and I hope doing all these steps again will fix it. So next step if it is working again to mark /prevent this kernel from updating /any changes. Can anybody tell me how to do this? I saw a documenation /command line about it a few days ago, but did not know the topic / page anymore. Thank you very much :slight_smile:

Update: If u do the steps 7-12 again, it is working with kernel 5.9.14 like before.

So be careful updating ur system if u have a working solution!

2 Likes