What is Wayland on Linux?

Wayland is a protocol for a compositor to talk to its clients as well as a C library implementation of that protocol. The compositor can be a standalone display server running on Linux kernel modesetting and evdev input devices, an X application, or a wayland client itself.

Should I use Wayland or X11?

Wayland is much newer than X11. If Wayland can run on your system without extensive setup, then you should stick with it. However some drivers have much less support for Wayland atm. If you’re using an NVIDIA gpu under NVIDIA’s notorious proprietary driver, then you should use X11.