When I run nvidia-smi, I get the following message:
Failed to initialize NVML: Driver/library version mismatch
An hour ago I received the same message and uninstalled my CUDA library and I was able to run nvidia-smi, getting the following result:
After this I downloaded cuda-repo-ubuntu1604-8-0-local-ga2_8.0.61-1_amd64.deb from the official NVIDIA page and then simply:
sudo dpkg -i cuda-repo-ubuntu1604-8-0-local-ga2_8.0.61-1_amd64.deb
sudo apt-get update
sudo apt-get install cuda
export PATH=/usr/local/cuda-8.0/bin${PATH:+:${PATH}}
Now I have CUDA installed, but I get the mentioned mismatch error.
Some potentially useful information:
Running cat /proc/driver/nvidia/version I get:
NVRM version: NVIDIA UNIX x86_64 Kernel Module 378.13 Tue Feb 7 20:10:06 PST 2017
GCC version: gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4)
I'm running Ubuntu 16.04.2 LTS (Xenial Xerus).
The kernel release is 4.4.0-66-generic.
19 Answers
Surprise surprise, rebooting solved the issue (I thought I had already tried that).
The solution Robert Crovella mentioned in the comments may also be useful to someone else, since it's pretty similar to what I did to solve the issue the first time I had it.
As etal said, rebooting can solve this problem, but I think a procedure without rebooting will help.
For Chinese, check my blog -> 中文版
The error message
NVML: Driver/library version mismatch
tell us the Nvidia driver kernel module (kmod) have a wrong version, so we should unload this driver, and then load the correct version of kmod
How can we do that?
First, we should know which drivers are loaded.
lsmod | grep nvidia
You may get
nvidia_uvm 634880 8
nvidia_drm 53248 0
nvidia_modeset 790528 1 nvidia_drm
nvidia 12312576 86 nvidia_modeset,nvidia_uvm
Our final goal is to unload nvidia mod, so we should unload the module depend on nvidia:
sudo rmmod nvidia_drm
sudo rmmod nvidia_modeset
sudo rmmod nvidia_uvm
Then, unload nvidia
sudo rmmod nvidia