Tag: Linux

KRDP Fails to Start

Scott has been trying to set up KRDP recently, and continued to get a lot of strange errors attempting to start the server. Through the GUI, it would fall over. From the command line, it output a lot of text. But they all seemed to indicate something couldn’t load. The log file had shared libraries (although ldd said all dependencies were met). The command line said things were found but could not run.

Had him run netstat to see if something else was bound to the port … and it was, but instead of printing the pid and binary name, it said off … which was a new one to me. Fortunately, lsof didshow us what was listening on the port. Stopped xrdp and, voila, krdp starts and runs.

[lisa@fedora01 ~/]# netstat -nap | grep 3389
tcp 0 0 0.0.0.0:3389 0.0.0.0:* LISTEN off...

[lisa@fedora01 ~/]# lsof -i TCP:3389
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
xrdp 1855 root 13u IPv4 39470 0t0 TCP *:ms-wbt-server (LISTEN)

In retrospect, it does tell you what the problem is. ‘Unable to listen for connections on QHostAddress(“”) 0’ means “unable to bind to ip:port

Jan 29 06:49:14 fedora01 systemd[10239]: Started plasma-krdp_server.service - KRDP Server.
Jan 29 06:49:16 fedora01 krdpserver[11054]: libEGL warning: egl: failed to create dri2 screen
Jan 29 06:49:16 fedora01 krdpserver[11054]: libEGL warning: egl: failed to create dri2 screen
Jan 29 06:49:17 fedora01 krdpserver[11054]: org.kde.krdp: Unable to listen for connections on QHostAddress("") 0
Jan 29 06:49:17 fedora01 krdpserver[11054]: qt.dbus.integration: QDBusConnection: error: could not send message to service "org.freedesktop.portal.Desktop" path "/org/freedesktop/portal/desktop" interface "org.freedesktop.portal.Re moteDesktop" member "NotifyKeyboardKeycode": Marshalling failed: Invalid object path passed in arguments
Jan 29 06:49:17 fedora01 krdpserver[11054]: qt.dbus.integration: QDBusConnection: error: could not send message to service "org.freedesktop.portal.Desktop" path "/org/freedesktop/portal/desktop" interface "org.freedesktop.portal.Re moteDesktop" member "NotifyKeyboardKeycode": Marshalling failed: Invalid object path passed in arguments
Jan 29 06:49:17 fedora01 krdpserver[11054]: qt.dbus.integration: QDBusConnection: error: could not send message to service "org.freedesktop.portal.Desktop" path "/org/freedesktop/portal/desktop" interface "org.freedesktop.portal.Re moteDesktop" member "NotifyKeyboardKeycode": Marshalling failed: Invalid object path passed in arguments
Jan 29 06:49:17 fedora01 krdpserver[11054]: qt.dbus.integration: QDBusConnection: error: could not send message to service "org.freedesktop.portal.Desktop" path "/org/freedesktop/portal/desktop" interface "org.freedesktop.portal.Re moteDesktop" member "NotifyKeyboardKeycode": Marshalling failed: Invalid object path passed in arguments
Jan 29 06:49:17 fedora01 krdpserver[11054]: qt.dbus.integration: QDBusConnection: error: could not send message to service "org.freedesktop.portal.Desktop" path "/org/freedesktop/portal/desktop" interface "org.freedesktop.portal.Re moteDesktop" member "NotifyKeyboardKeycode": Marshalling failed: Invalid object path passed in arguments
Jan 29 06:49:17 fedora01 krdpserver[11054]: qt.dbus.integration: QDBusConnection: error: could not send message to service "org.freedesktop.portal.Desktop" path "/org/freedesktop/portal/desktop" interface "org.freedesktop.portal.Re moteDesktop" member "NotifyKeyboardKeycode": Marshalling failed: Invalid object path passed in arguments
Jan 29 06:49:17 fedora01 krdpserver[11054]: qt.dbus.integration: QDBusConnection: error: could not send message to service "org.freedesktop.portal.Desktop" path "/org/freedesktop/portal/desktop" interface "org.freedesktop.portal.Re moteDesktop" member "NotifyKeyboardKeycode": Marshalling failed: Invalid object path passed in arguments
Jan 29 06:49:17 fedora01 krdpserver[11054]: qt.dbus.integration: QDBusConnection: error: could not send message to service "org.freedesktop.portal.Desktop" path "/org/freedesktop/portal/desktop" interface "org.freedesktop.portal.Re moteDesktop" member "NotifyKeyboardKeycode": Marshalling failed: Invalid object path passed in arguments
Jan 29 06:49:17 fedora01 krdpserver[11054]: qt.dbus.integration: QDBusConnection: error: could not send message to service "org.freedesktop.portal.Desktop" path "" interface "org.freedesktop.portal.Session" member "Close": Object p ath cannot be empty
Jan 29 06:49:17 fedora01 systemd[10239]: plasma-krdp_server.service: Main process exited, code=exited, status=255/EXCEPTION
Jan 29 06:49:17 fedora01 systemd[10239]: plasma-krdp_server.service: Failed with result 'exit-code'.

adsaf

Fedora 41, KVM, QEMU, and the Really (REALLY!) Bad Performance

Ever since we upgraded to Fedora 41, we have been having horrible problems with our Exchange server. It will drop off the network for half an hour at a time. I cannot even ping the VM from the physical server. Some network captures show there’s no response to the ARP request.

Evidently, the VM configuration contains a machine type that doesn’t automatically update. We are using PC-Q35 as the chipset … and 4.1 was the version when we built our VMs. This version, however has been deprecated. Which you can see by asking virsh what capabilities it has:


2025-01-02 23:17:26 [lisa@linux01 /var/log/libvirt/qemu/]# virsh capabilities | grep pc-q35
      <machine maxCpus='288' deprecated='yes'>pc-q35-5.2</machine>
      <machine maxCpus='288' deprecated='yes'>pc-q35-4.2</machine>
      <machine maxCpus='255' deprecated='yes'>pc-q35-2.7</machine>
      <machine maxCpus='4096'>pc-q35-9.1</machine>
      <machine canonical='pc-q35-9.1' maxCpus='4096'>q35</machine>
      <machine maxCpus='288'>pc-q35-7.1</machine>
      <machine maxCpus='1024'>pc-q35-8.1</machine>
      <machine maxCpus='288' deprecated='yes'>pc-q35-6.1</machine>
      <machine maxCpus='255' deprecated='yes'>pc-q35-2.4</machine>
      <machine maxCpus='288' deprecated='yes'>pc-q35-2.10</machine>
      <machine maxCpus='288' deprecated='yes'>pc-q35-5.1</machine>
      <machine maxCpus='288' deprecated='yes'>pc-q35-2.9</machine>
      <machine maxCpus='288' deprecated='yes'>pc-q35-3.1</machine>
      <machine maxCpus='288' deprecated='yes'>pc-q35-4.1</machine>
      <machine maxCpus='255' deprecated='yes'>pc-q35-2.6</machine>
      <machine maxCpus='4096'>pc-q35-9.0</machine>
      <machine maxCpus='288' deprecated='yes'>pc-q35-2.12</machine>
      <machine maxCpus='288'>pc-q35-7.0</machine>
      <machine maxCpus='288'>pc-q35-8.0</machine>
      <machine maxCpus='288' deprecated='yes'>pc-q35-6.0</machine>
      <machine maxCpus='288' deprecated='yes'>pc-q35-4.0.1</machine>
      <machine maxCpus='288' deprecated='yes'>pc-q35-5.0</machine>
      <machine maxCpus='288' deprecated='yes'>pc-q35-2.8</machine>
      <machine maxCpus='288' deprecated='yes'>pc-q35-3.0</machine>
      <machine maxCpus='288'>pc-q35-7.2</machine>
      <machine maxCpus='288' deprecated='yes'>pc-q35-4.0</machine>
      <machine maxCpus='1024'>pc-q35-8.2</machine>
      <machine maxCpus='288'>pc-q35-6.2</machine>
      <machine maxCpus='255' deprecated='yes'>pc-q35-2.5</machine>
      <machine maxCpus='288' deprecated='yes'>pc-q35-2.11</machine>
      <machine maxCpus='288' deprecated='yes'>pc-q35-5.2</machine>
      <machine maxCpus='288' deprecated='yes'>pc-q35-4.2</machine>
      <machine maxCpus='255' deprecated='yes'>pc-q35-2.7</machine>
      <machine maxCpus='4096'>pc-q35-9.1</machine>
      <machine canonical='pc-q35-9.1' maxCpus='4096'>q35</machine>
      <machine maxCpus='288'>pc-q35-7.1</machine>
      <machine maxCpus='1024'>pc-q35-8.1</machine>
      <machine maxCpus='288' deprecated='yes'>pc-q35-6.1</machine>
      <machine maxCpus='255' deprecated='yes'>pc-q35-2.4</machine>
      <machine maxCpus='288' deprecated='yes'>pc-q35-2.10</machine>
      <machine maxCpus='288' deprecated='yes'>pc-q35-5.1</machine>
      <machine maxCpus='288' deprecated='yes'>pc-q35-2.9</machine>
      <machine maxCpus='288' deprecated='yes'>pc-q35-3.1</machine>
      <machine maxCpus='288' deprecated='yes'>pc-q35-4.1</machine>
      <machine maxCpus='255' deprecated='yes'>pc-q35-2.6</machine>
      <machine maxCpus='4096'>pc-q35-9.0</machine>
      <machine maxCpus='288' deprecated='yes'>pc-q35-2.12</machine>
      <machine maxCpus='288'>pc-q35-7.0</machine>
      <machine maxCpus='288'>pc-q35-8.0</machine>
      <machine maxCpus='288' deprecated='yes'>pc-q35-6.0</machine>
      <machine maxCpus='288' deprecated='yes'>pc-q35-4.0.1</machine>
      <machine maxCpus='288' deprecated='yes'>pc-q35-5.0</machine>
      <machine maxCpus='288' deprecated='yes'>pc-q35-2.8</machine>
      <machine maxCpus='288' deprecated='yes'>pc-q35-3.0</machine>
      <machine maxCpus='288'>pc-q35-7.2</machine>
      <machine maxCpus='288' deprecated='yes'>pc-q35-4.0</machine>
      <machine maxCpus='1024'>pc-q35-8.2</machine>
      <machine maxCpus='288'>pc-q35-6.2</machine>
      <machine maxCpus='255' deprecated='yes'>pc-q35-2.5</machine>
      <machine maxCpus='288' deprecated='yes'>pc-q35-2.11</machine>

Or filtering out the deprecated ones …

2025-01-02 23:16:50 [lisa@linux01 /var/log/libvirt/qemu/]# virsh capabilities | grep pc-q35 | grep -v "deprecated='yes'"
      <machine maxCpus='4096'>pc-q35-9.1</machine>
      <machine canonical='pc-q35-9.1' maxCpus='4096'>q35</machine>
      <machine maxCpus='288'>pc-q35-7.1</machine>
      <machine maxCpus='1024'>pc-q35-8.1</machine>
      <machine maxCpus='4096'>pc-q35-9.0</machine>
      <machine maxCpus='288'>pc-q35-7.0</machine>
      <machine maxCpus='288'>pc-q35-8.0</machine>
      <machine maxCpus='288'>pc-q35-7.2</machine>
      <machine maxCpus='1024'>pc-q35-8.2</machine>
      <machine maxCpus='288'>pc-q35-6.2</machine>
      <machine maxCpus='4096'>pc-q35-9.1</machine>
      <machine canonical='pc-q35-9.1' maxCpus='4096'>q35</machine>
      <machine maxCpus='288'>pc-q35-7.1</machine>
      <machine maxCpus='1024'>pc-q35-8.1</machine>
      <machine maxCpus='4096'>pc-q35-9.0</machine>
      <machine maxCpus='288'>pc-q35-7.0</machine>
      <machine maxCpus='288'>pc-q35-8.0</machine>
      <machine maxCpus='288'>pc-q35-7.2</machine>
      <machine maxCpus='1024'>pc-q35-8.2</machine>
      <machine maxCpus='288'>pc-q35-6.2</machine>

So I shut down my Exchange server again (again, again), used “virsh edit “exchange01”, changed

  <os>
    <type arch='x86_64' machine='pc-q35-4.1'>hvm</type>
    <boot dev='hd'/>
  </os>

to

  <os>
    <type arch='x86_64' machine='pc-q35-7.1'>hvm</type>
  </os>

And started my VM. It took about an hour to boot. It absolutely hogged the disk physical server’s resources. It was the top listing in iotop -o

But then … all of the VMs dropped off of iotop. My attempt to log into the server via the console was logged in and waiting for me. My web mail, which had failed to load all day, was in my e-mail. And messages that had been queued for delivery had all come through.

The load on our physical server dropped from 30 to 1. Everything became responsive. And Exchange has been online for a good thirty minutes now.

Fedora 41 – Using DNF to List Installed Packages

We upgraded all of our internal servers to Fedora 41 after a power outage yesterday — had a number of issues to resolve (the liblockdev legacy config reverted so OpenHAB no longer could use USB serial devices, the physical server was swapping 11GB of data even though it had 81GB of memory free, and our Gerbera installation requires some libspdlog.so.1.12 which was updated to version 1.14 with the Fedora upgrade.

The last issue was more challenging to figure out because evidently DNF is now DNF5 and instead of throwing an error like “hey, new version dude! Use the new syntax” when you use an old command to list what is installed … it just says “No matching packages to list”. Like there are no packages installed? Since I’m using bash, openssh, etc … that’s not true.

Luckily, the new syntax works just fine. dnf repoquery –installed

Also:

dnf5 repoquery –available
dnf5 repoquery –userinstalled

SSSD LDAP Schema

I lost access to all of my Linux servers at work. And, unlike the normal report where nothing changed but xyz is now failing, I knew exactly what happened. A new access request had been approved about ten minutes previously. Looking at my ID, for some reason adding a new group membership changed account gid number to that new group. Except … that shouldn’t have actually dropped my access. If I needed the group to be my primary ID, I should have been able to use newgrp to switch contexts. Instead, I got prompted for a group password (which, yes, is a thing. No, no one uses it).

The hosts were set up to authenticate to AD using LDAP, and very successfully let me log in (or not, if I mistyped my password). They, however, would only see me as a member of my primary group. Well, today, I finally got a back door with sufficient access to poke around.

Turns out I was right — something was improperly configured so groups were not being read from the directory but rather implied from the gid value. I added the configuration parameter ldap_schema to instruct the server to use member instead of memberUid for memberships. I used rfc2307bis as that’s the value I was familiar with. I expect “AD” could be used as well, but figured we were well beyond AD 2008r2 and didn’t really want to dig farther into the nuanced differences between the two settings.

From https://linux.die.net/man/5/sssd-ldap

ldap_schema (string)

Specifies the Schema Type in use on the target LDAP server. Depending on the selected schema, the default attribute names retrieved from the servers may vary. The way that some attributes are handled may also differ.

Four schema types are currently supported:

  • rfc2307
  • rfc2307bis
  • IPA
  • AD

The main difference between these schema types is how group memberships are recorded in the server. With rfc2307, group members are listed by name in the memberUid attribute. With rfc2307bis and IPA, group members are listed by DN and stored in the member attribute. The AD schema type sets the attributes to correspond with Active Directory 2008r2 values.

 

Quick sed For Sanitizing Config Files

When sending configuration files to other people for reference, I like to redact any credential-type information … endpoints that allow you to post data without creds, auth configs, etc. Sometimes I replace the string with REDACTED and sometimes I just drop the line completely.

Make a copy of the config files elsewhere, then run sed


# Retain parameter but replace value with REDACTED
sed -i 's|http_post_url: "https://.*"|post_url: "REDACTED"|' *.yaml

# Remove line from config
sed -i '/authorization: Basic/d' *.yaml

Fedora 40: NFTables not logging

We upgraded Anya’s laptop to Fedora 40, and Skype has evidently moved from an installable RPM to a snap package. Which didn’t work with the firewall rules we built earlier in the year (video and audio calls would not connect); and, worse, nothing logs out. Looks like the netfilter kernel logging isn’t enabled

Enabled the logging:

echo 1 | sudo tee /proc/sys/net/netfilter/nf_log_all_netns

And, voila, we’ve got log records from nftables. And now Skype works … so I don’t know what to add. Sigh!

Linux Command: Column

Column is an interesting command – it will turn delimited text into, well, columns. Simply tell it you want a table (-t) and indicate what separator to use (-s). Optionally, you can add table column headers

[lisa@linux01 ~/]# cat /etc/group | column -t -s :
root              x  0      root,lisa
bin               x  1
daemon            x  2
...
passim            x  987

Alternately, you can use -J to get JSON-formatted output. Here you need the –table-columns as a comma delimited list of column names:

[lisa@linux01 ~/]# cat /etc/group | column -J -s : --table-columns "group,password,gid,members"
{
   "table": [
      {
         "group": "root",
         "password": "x",
         "gid": "0",
         "members": "root,lisa"
      },{
         "group": "bin",
         "password": "x",
         "gid": "1",
         "members": null
      },{
         "group": "daemon",
         "password": "x",
         "gid": "2",
         "members": null
      },{
         "group": "passim",
         "password": "x",
         "gid": "987",
         "members": null
      }
   ]
}

Which can then be parsed with jq

[lisa@linux01 ~/]# cat /etc/group | column -J -s : --table-columns "group,password,gid,members" | jq '[.table[] | {group: .group, members: .members}]'
[
  {
    "group": "root",
    "members": "root,lisa"
  },
  {
    "group": "bin",
    "members": null
  },
  {
    "group": "daemon",
    "members": null
  },
  {
    "group": "passim",
    "members": null
  }
]

OpenZFS On RedHat 8 (From Package)

This process presumes you have generated a signing key (/root/signing/MOK.priv and /root/signing/MOK.der) that has been registered for signing modules.

################################################################################
## Install from Repo and Sign Modules
################################################################################
yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
yum install kernel-devel

# Install kmod version of ZS
yum install https://zfsonlinux.org/epel/zfs-release-2-3$(rpm --eval "%{dist}").noarch.rpm
dnf config-manager --disable zfs
dnf config-manager --enable zfs-kmod
yum install zfs

# And autoload
echo zfs >/etc/modules-load.d/zfs.conf

# Use rpm -ql to list out the kernel modules that this version of ZFS uses -- 2.1.x has quite a few of them, and they each need to be signed
# Sign zfs.ko and spl.ko in current kernel
/usr/src/kernels/$(uname -r)/scripts/sign-file sha256 /root/signing/MOK.priv /root/signing/MOK.der /lib/modules/$(uname -r)/weak-updates/zfs/zfs/zfs.ko
/usr/src/kernels/$(uname -r)/scripts/sign-file sha256 /root/signing/MOK.priv /root/signing/MOK.der /lib/modules/$(uname -r)/weak-updates/zfs/spl/spl.ko
# And sign the bunch of other ko files in the n-1 kernel rev (these are symlinked from the current kernel)
/usr/src/kernels/$(uname -r)/scripts/sign-file sha256 /root/signing/MOK.priv /root/signing/MOK.der /lib/modules/4.18.0-513.18.1.el8_9.x86_64/extra/zfs/avl/zavl.ko
/usr/src/kernels/$(uname -r)/scripts/sign-file sha256 /root/signing/MOK.priv /root/signing/MOK.der /lib/modules/4.18.0-513.18.1.el8_9.x86_64/extra/zfs/icp/icp.ko
/usr/src/kernels/$(uname -r)/scripts/sign-file sha256 /root/signing/MOK.priv /root/signing/MOK.der /lib/modules/4.18.0-513.18.1.el8_9.x86_64/extra/zfs/lua/zlua.ko
/usr/src/kernels/$(uname -r)/scripts/sign-file sha256 /root/signing/MOK.priv /root/signing/MOK.der /lib/modules/4.18.0-513.18.1.el8_9.x86_64/extra/zfs/nvpair/znvpair.ko
/usr/src/kernels/$(uname -r)/scripts/sign-file sha256 /root/signing/MOK.priv /root/signing/MOK.der /lib/modules/4.18.0-513.18.1.el8_9.x86_64/extra/zfs/unicode/zunicode.ko
/usr/src/kernels/$(uname -r)/scripts/sign-file sha256 /root/signing/MOK.priv /root/signing/MOK.der /lib/modules/4.18.0-513.18.1.el8_9.x86_64/extra/zfs/common/zcommon.ko
/usr/src/kernels/$(uname -r)/scripts/sign-file sha256 /root/signing/MOK.priv /root/signing/MOK.der /lib/modules/4.18.0-513.18.1.el8_9.x86_64/extra/zfs/zstd/zzstd.ko

# Verify they are signed now
modinfo -F signer /usr/lib/modules/$(uname -r)/weak-updates/zfs/zfs/zfs.ko
modinfo -F signer /usr/lib/modules/$(uname -r)/weak-updates/zfs/spl/spl.ko

modinfo -F signer /lib/modules/4.18.0-513.18.1.el8_9.x86_64/extra/zfs/avl/zavl.ko
modinfo -F signer /lib/modules/4.18.0-513.18.1.el8_9.x86_64/extra/zfs/icp/icp.ko
modinfo -F signer /lib/modules/4.18.0-513.18.1.el8_9.x86_64/extra/zfs/lua/zlua.ko
modinfo -F signer /lib/modules/4.18.0-513.18.1.el8_9.x86_64/extra/zfs/nvpair/znvpair.ko
modinfo -F signer /lib/modules/4.18.0-513.18.1.el8_9.x86_64/extra/zfs/unicode/zunicode.ko
modinfo -F signer /lib/modules/4.18.0-513.18.1.el8_9.x86_64/extra/zfs/zcommon/zcommon.ko
modinfo -F signer /lib/modules/4.18.0-513.18.1.el8_9.x86_64/extra/zfs/zstd/zzstd.ko

# Reboot
init 6

# And we've got ZFS, so create the pool
zpool create pgpool sdc
zfs create zpool/zdata
zfs set compression=lz4 zpool/zdata

zfs get compressratio zpool/zdata
zfs set mountpoint=/zpool/zdata zpool/zdata

What happens if you only sign zfs.ko? All sorts of errors that look like there’s some sort of other problem — zfs will not load. It will tell you the required key is not available

May 22 23:42:44 sandboxserver systemd-modules-load[492]: Failed to insert 'zfs': Required key not available

Using insmod to try to manually load it will tell you there are dozens of unknown symbols:

May 22 23:23:23 sandboxserver kernel: zfs: Unknown symbol ddi_strtoll (err 0)
May 22 23:23:23 sandboxserver kernel: zfs: Unknown symbol spl_vmem_alloc (err 0)
May 22 23:23:23 sandboxserver kernel: zfs: Unknown symbol taskq_empty_ent (err 0)
May 22 23:23:23 sandboxserver kernel: zfs: Unknown symbol zone_get_hostid (err 0)
May 22 23:23:23 sandboxserver kernel: zfs: Unknown symbol tsd_set (err 0)

But the real problem is that there are unsigned modules so … there are unknown symbols. But not because something is incompatible. Just because the module providing that symbol will not load.

OpenZFS on RedHat 8 – Build from Source

This process presumes you have generated a signing key (/root/signing/MOK.priv and /root/signing/MOK.der) that has been registered for signing modules.

# Install prerequisites
dnf install --skip-broken epel-release gcc make autoconf automake libtool rpm-build libtirpc-devel libblkid-devel libuuid-devel libudev-devel openssl-devel zlib-devel libaio-devel libattr-devel elfutils-libelf-devel kernel-devel-$(uname -r) python3 python3-devel python3-setuptools python3-cffi libffi-devel git ncompress libcurl-devel

dnf install --skip-broken --enablerepo=epel --enablerepo=powertools python3-packaging dkms

# Clone OpenZFS repo
git clone https://github.com/openzfs/zfs
cd zfs
# generally stay in the main branch, but if you want to use the latest then check out the staging branch
# git checkout zfs-2.2.5-staging
./autogen.sh
./configure
make 
make install

# Sign the kernel modules
/usr/src/kernels/$(uname -r)/scripts/sign-file sha256 /root/signing/MOK.priv /root/signing/MOK.der /lib/modules/$(uname -r)/extra/zfs.ko
/usr/src/kernels/$(uname -r)/scripts/sign-file sha256 /root/signing/MOK.priv /root/signing/MOK.der /lib/modules/$(uname -r)/extra/spl.ko

# And verify the modules are signed
modinfo -F signer /usr/lib/modules/$(uname -r)/extra/zfs.ko
modinfo -F signer /usr/lib/modules/$(uname -r)/extra/spl.ko