How to debug Wayland problems

From FedoraProject

Jump to: navigation, search

Wayland is intended as a simpler replacement for X11. It changes the design of a Linux desktop architecture considerably. Unlike X11, there is no dedicated standalone server in Wayland. What was previously done between the app, its toolkit, the Xserver and the window manager is now shared between the app, its toolkit and the Wayland compositor which manages the compositing, input, windows management, etc. The apps and toolkits are now in charge of their own rendering and decorations (client side decorations), so any issues usually sit between the toolkit (e.g. GTK+) and the Wayland compositor (e.g. mutter).

You can read more about Wayland on the GNOME Wayland initiative wiki page. You can read more about the current state of Wayland features on Wayland features page.

Contents

Identifying Wayland problems

Are you running a Wayland session?

In GNOME, there's a gear button at the login screen which can be used to either log into a Wayland session (simply called GNOME, it's the default option), or a legacy X11 session (called GNOME on Xorg). If you have a password-less user account, you won't see the gear icon, it is displayed only when the password prompt appears. Use the gear button to determine type of session you're logging into. If you want to start your session in a different way, read the advanced techniques for trying Wayland.

Gdm-pick-wayland.png

In KDE, there is support for running a nested Wayland session inside your X11 session. You'll need to install kwin-wayland package and then follow up with this howto. There doesn't seem to be out-of-the-box support for running a full Wayland session at the moment.

Other desktop environments are not currently capable of running a Wayland session.

Identifying the session type in runtime

If you want to figure out which type of session you're running right now, without logging out and in again, you can use several ways to figure it out:

  • Wayland session should have WAYLAND_DISPLAY variable set, X11 session should not have it:
    $ echo $WAYLAND_DISPLAY 
    wayland-0
  • loginctl can give you this information. First run loginctl and find your session number (if should be an integer number, with your username and seat assigned). Then look at the session type (x11 or wayland):
    $ loginctl show-session <YOUR_NUMBER> -p Type
    Type=x11

If you're running an X11 session, not a Wayland session, your problems are not related to Wayland. It's a bug either in that particular application, or X11 itself, see How to debug Xorg problems.

Does your application run on Wayland natively, or uses XWayland (X11 compatibility layer)?

It is important to know whether the problematic application is a native Wayland application, or runs through XWayland, which allows legacy applications to still run on top of Xorg server, but display in a Wayland session.

There are several ways how to identify whether an application is using Wayland or XWayland:

  • Select the window using xwininfo or xprop. Run:
    $ xwininfo

    Your mouse pointer should change to a cross under X11, it doesn't seem to change under Wayland. Now click anywhere inside the app window you want to test. If the xwininfo command finishes (it should print window properties into the terminal), the app under test is running under XWayland. If nothing happens (the xwininfo command is still waiting until you select a window), the app under test is running under Wayland (you can close the command with Ctrl+C).
    You can also use xprop command using the same instructions.

  • XWayland applications are listed in xlsclients output. Run:
    $ xlsclients

    However, this list of not always entirely reliable, some apps might be missing.

  • You can try to run the app while unsetting DISPLAY environment variable:
    $ DISPLAY='' command

    If the application runs OK, it should be using Wayland natively.

  • You can run the app with WAYLAND_DEBUG=1 environment variable:
    $ WAYLAND_DEBUG=1 command

    If you see loads of output (when compared to a standard run), the app is using Wayland natively.

  • Under GNOME, you can determine this using integrated Looking Glass tool. Hit Alt+F2, run:
    lg

    click on Windows in the upper right corner of the tool and select desired window by clicking on its name. If you see MetaWindowWayland in the first line, this app is running under Wayland. If you see MetaWindowX11 in the first line, this app is running under X11.

If you have identified the problem to be in a XWayland application, try to reproduce the issue in a standard X11 session. If it happens as well, it is not related to Wayland, it's a bug either in that particular application, or Xorg server, see How to debug Xorg problems. If the problem happens only under XWayland but not in an X11 session, it should still be reported against Xorg server (xorg-x11-server package), because XWayland is included in it (as xorg-x11-server-Xwayland subpackage).

Identifying problem component

Wayland itself is a protocol and the problem is rarely in the protocol itself. Rather, the problem is likely to be in the app or its toolkit, or in the compositor.

The most notable Wayland-ready toolkits are:

  • GTK+ 3 - default apps in GNOME environment use almost exclusively this toolkit. Please note that apps using older GTK+ 2 are not Wayland-ready.
  • QT 5 - many apps in KDE environment use this toolkit. Please note that apps using older QT 4 are not Wayland-ready.

The most notable Wayland compositors are:

  • Weston - the reference implementation of a Wayland compositor, maintained directly by the Wayland project
  • Mutter - compositor in GNOME. If you run GNOME, it is using this compositor.
  • Kwin - compositor in KDE. If you run KDE, it is using this compositor.

Testing under different compositors

If you experience a problem with a Wayland app, it is very useful to know whether the problem is present under just a single compositor (in that case it's likely a compositor bug) or under multiple compositors (in that case it's likely an app/toolkit bug).

Please run your session with the reference Weston compositor and try to reproduce the issue. You can either run Weston as a nested window, or as a full session. First, install weston package (you can read many useful information in its man page):

$ sudo dnf install weston

Then create a config file which will specify that you want to have XWayland support enabled in your weston sessions. Create ~/.config/weston.ini with this contents:

[core]
modules=xwayland.so

Now you can start weston either as nested window or as a full session.

  • To start a nested Weston window, run this from a terminal:
    $ weston

    A Weston window should open and you should see and a terminal icon in its top left corner. Use that icon to launch a terminal and from that you can run apps and other commands using Weston. Exit the compositor by simply closing the window or killing the weston process.

  • To start a full Weston session (not nested inside another X11 or Wayland session), switch to a free VT (Ctrl+Alt+Fx) and run:
    $ weston-launch

    You can exit the session by pressing Control+Alt+Backspace shortcut.

If you can reproduce the issue with Weston, file an issue against the app or its toolkit (gtk+, qt, etc). Otherwise file the issue against the compositor your environment uses (mutter, kwin, etc). If the problem occurs only with XWayland apps but not native Wayland apps, report a bug against Xorg server.

Reporting the issue

Using up-to-date software

Before reporting the bug, please make sure you use the latest available software. You need to run on Fedora 23 or later, older Fedora versions are not going to get updated with latest Wayland fixes. Make sure there are no system updates waiting:

$ sudo dnf update

If there are (and the available updates look plausibly related to the components you're seeing issues with), please update the system and verify whether the issue is still present or has been fixed.

Looking for similar reports

In order to avoid duplicate reports and also wasting your time debugging something someone has maybe already debugged, please search through the existing reports first. The most visible issues or concerns are listed in Known issues, frequent complaints, fundamental changes. If you don't see it there, you need to search deeper. You can find Wayland related issues most likely in here:

Filing a bug

After you've identified against which component to (most probably) report the issue and found no existing report of it, there are several places where to report it:

  • Red Hat Bugzilla - recommended for issues related to wayland itself, weston compositor, non-GNOME apps, KDE project, QT toolkit
  • GNOME Bugzilla - recommended for issues related to mutter compositor, GTK+ toolkit, applications under the GNOME project (most of default apps in Fedora Workstation)

When reporting the issue, please make your report block our tracker, so that we have a good overall picture of what is broken across many different components. In your bug report, set Blocks: WaylandRelated or Blocks: KDEWaylandRelated (you might need to toggle showing advanced fields to see the Blocks: field). That will make it block one of these trackers, depending where you reported the bug:

Information to include in your bug report

  1. System journal. Since there is no unique server like the X11 server, most of the important information will come from the the Wayland compositor and the apps. All of that should be in system journal nowadays. You can save a full journal since last boot like this:
    $ journalctl -ab > journal.log

    You can also edit the file and according to the timestamps remove everything long prior to when the issue occurred, in order to make the log more succinct.

    • If your system crashed or became unresponsive so that you had to reboot it, you can see the journal from the previous boot using journalctl -a -b -1 instead.
  2. Wayland debug output. If you can reproduce the issue, please run the problematic app like this:
    $ WAYLAND_DEBUG=1 command |& tee debug.out

    You should see loads of output being printed out. It will involve all communication between the app and the compositor.

  3. Information whether the same problem occurs when you run the app using XWayland instead of Wayland. For GTK+ 3 apps, you can force a native Wayland app to run using XWayland like this:
    $ GDK_BACKEND=x11 command

    Vice versa, you can also force a XWayland app to run using Wayland (in case it has just experimental support):

    $ GDK_BACKEND=wayland command

    QT 5 apps run with XWayland by default. You can force Wayland backend:

    $ QT_QPA_PLATFORM=wayland-egl command

    All of this applies to just GTK+ 3 and QT 5 apps.

  4. Hardware description is useful for some hardware-related bugs:
    $ lspci -nn > lspci.out
  5. Package versions. You can collect the list and versions of all your packages installed using:
    $ rpm -qa | sort > packages.out
  6. The usual information that every bug report should have.

Debugging gnome-shell

If gnome-shell gets stuck and unresponsive, it's very helpful to obtain a backtrace from its process and attach it to the report. If this happens, switch to a different VT if possible (Ctrl+Alt+F3 through F7), or log in using ssh. First install debug symbols:

$ sudo dnf debuginfo-install `rpm -q gnome-shell`

Then attach gdb debugger to your gnome-shell process:

$ gdb -p `pgrep -U $(id -un) -x gnome-shell`
...
(gdb) set logging on
(gdb) thread apply all backtrace full
... press Enter until the whole backtrace is displayed ...
(gdb) quit

You should have the backtrace saved in gdb.txt file.

Debugging mutter

You can debug mutter (used in gnome-shell) by setting its environment variables. These need to be set prior to run gnome-shell, so if you want to log into GNOME from GDM, you need to create a wrapper script called from a desktop file in /usr/share/wayland-sessions.

FIXME: Putting the wrapper script and desktop file here would be helpful.


Known issues, frequent complaints, fundamental changes

Here we will list high-profile issues which are known to be broken, not yet implemented, or intentionally behaving differently from regular X11 apps. Also please look at Wayland features which lists all current missing or in-progress features and their details.

To see all known issues, look at Bugzilla reports as mentioned in Looking for similar reports.

Graphical applications can't be run as root from terminal

It is not possible to start graphical apps under the root account from terminal when using su or sudo. Apps which use polkit to request administrator permissions for just certain operations and only when needed are not affected (they are not started as root right away). The discussion is ongoing about the best approach to take, see bug 1274451 and "On running gui applications as root" thread in fedora-devel mailing list.

Many well-known X11 utilities don't work

Power users are familiar with a large range of X11-related utilities, like xkill, xrandr, xdotool, xsel. These tools won't work under Wayland session, or will only work with XWayland applications but not Wayland applications. Some tools might have a replacement which allows to perform similar tasks.

FIXME: add some Wayland-ready replacements for popular X11 tools

Games and other apps can't change monitor resolution

It is no longer possible for an app to change monitor resolution. Usually this was done by games to increase performance. Wayland-based games will use a different approach - scaling its output. But for X11 games (running through XWayland) this solution is not available. This results in a number of different types of behavior, based on how the game is written - the game might be fixed in the desktop resolution, or rendered as a small centered image with black bars around it, or crash on startup, or something different. See bug 1289714.

For some games, a possible workaround is to manually set custom monitor resolution before running the game, if you really need it. It will not help always, though.

Certain OpenGL apps do not update live in GNOME overview

For certain apps using OpenGL you won't see their live preview in the overview mode (when you press Activities button or Win key). You'll see a default, static window frame instead. This involves apps like Totem (video player) or Maps. You can read more about this in bug 756715.

Screen capture is not available with usual apps

One of the features of Wayland is its security design, which helps to guard the user against malicious apps. Apps can no longer see everything on the screen and spy on you. But that also means you cannot run a common application (like shutter or gtk-recordmydesktop) and use it to make a screenshot or a screencast of your desktop - it will see only its own window, but nothing else (or it might crash right away). System (trusted) apps need to be used to perform these actions.

In GNOME, you can use Screenshot tool (available in overview or as Printscreen hotkey or as gnome-screenshot command) to capture a screenshot of the full desktop or a particular window. You can press Ctrl+Alt+Shift+R keyboard shortcut to start video recording of the whole desktop (stop it by pressing the same shortcut again, there's an indicator in the upper right corner, or it stops automatically after 30 seconds by default) and find the screencast in ~/Videos. For screencast, you can also use EasyScreenCast gnome-shell extension.

Desktop icons not showing in GNOME

If you configured GNOME to show icons on your desktop, this doesn't currently work under Wayland. See bug 1280195.

Mouse pointer is lagging/stuttering under load

If your computer is under load, your mouse pointer movement might stop being fluent, but start lagging (get stuck in a place for a short time, then jump to a different place instantly). This is probably more noticeable on slow systems/systems with fewer CPU cores. See bug 745032.

Keyboard events are sometimes quickly repeated

There is a rare issue when you press a key to type a letter and you'll see multiple copies of the letter typed in. See bug 757942.

Not all keys can be sent to a remote desktop or a virtual machine

Some applications forward all input, including system-specific keys/shortcuts like Win or Alt+Tab , to a remote system. This is mostly remote desktop viewers like vncviewer or virtual machine managers like virt-manager or boxes. Under Wayland, some of these shortcuts can't be intercepted, and therefore are used in the host system, not the remote/guest system. See bug 1285770.