Category: System Administration

The end of password changes?

I knew Microsoft was publishing recommendations against forced password expiry, but it was still surprising to see this banner in my Azure admin portal. It would be nice if their message was clearer on the nuances here — especially that enabling MFA (preferably not SMS-based MFA that is just asking for someone important’s number to get hijacked) is an important component of this recommendation.

In one of my first jobs, I was a sys admin for call-center systems. As such, I interacted with a lot of the call center management and staff … and, when you know someone in IT, you ping them when the proper support route isn’t as responsive as you’d like. Which is to say I did a good bit of end-user support as well. The number of people whose password was written on a post-it note under the keyboard astonished me. This particular call center didn’t have floating seating, but two or three people would share a cube because they worked different schedules. If I’ve got to come up with a new thing I need to remember every 90 days … well, that’s how you end up with Winter19, Spring20, Summer20, Autumn20 or Maggie12, Maggie13, Maggie14 passwords. That then get posted under the keyboard so I can remember that I’m up to “14” now. Couple that with the overhead of supporting password resets for those who didn’t write it down and happened to forget the password. I’d been a proponent of long password expiry coupled with increased complexity requirements. Maybe !Maggie-19? is good for all of 2019. It’s nice to see a major IT vendor starting to realize the real-world impact of IT policies.

Screen Mouse Scrolling Sends Arrows

I’m finding all sorts of things we researched and sorted on the old server without documenting 🙁 Today’s discovery is that screen uses alternate screens where the mouse wheel sends arrow keys. Which means you end up going through your history instead of scrolling up a few lines. Fix? Either create a .screenrc in the user’s account or globally change /etc/screenrc (which is what we do):

 

[root@server-replacement ~]# diff /etc/screenrc /root/oldserver-screenrc
216a217,218
>
> termcapinfo xterm* ti@:te@

Zoneminder, time zones, and php-fpm

I’m in the process of installing Zoneminder on our new server. It was a fairly straightforward process — stop Zoneminder on the old server, dump the SQL database, fix the DEFINER values since I’m using a central database server instead of a server on localhost, install Zoneminder, copy the config file, set up the database user, pull in the SQL file, and start it all up.

Visiting the website, I get “ZoneMinder is not installed properly: php’s date.timezone is not set to a valid timezone”. I’d forgotten to set the timezone in php.ini. Added ‘date.timezone = “America/New_York”‘, restarted httpd and Zoneminder. And got the same error.

<?php
error_reporting(E_ALL);
var_dump(ini_get('date.timezone'),date_default_timezone_get());
?>

It’s not set. This isn’t a funky Zoneminder thing — this is a PHP problem. I realized that PHP now runs as its own service. Restarting httpd is insufficient. Restarted php-ftm and the time zone I’d set in php.ini showed up. This is a case where a reboot would have sorted it … but good to remember that, when changing PHP settings, the php service needs to be restarted.

Samba and SELinux

I had a horrendous time trying to get the Samba share on our new server working. It worked insomuchas I could map a drive to the share … but I couldn’t actually see any files. Increasing the log level (smb.conf)

log level = 10 passdb:5 auth:5

showed that, yeah, I was getting a lot of access denied errors.

[2019/12/14 23:04:53.249959, 10, pid=17854, effective(0, 0), real(0, 0)] ../../source3/smbd/open.c:5438(create_file_unixpath)
create_file_unixpath: NT_STATUS_ACCESS_DENIED
[2019/12/14 23:04:53.249982, 10, pid=17854, effective(0, 0), real(0, 0)] ../../source3/smbd/open.c:5716(create_file_default)
create_file: NT_STATUS_ACCESS_DENIED
[2019/12/14 23:04:53.250012, 3, pid=17854, effective(0, 0), real(0, 0), class=smb2] ../../source3/smbd/smb2_server.c:3254(smbd_smb2_request_error_ex)
smbd_smb2_request_error_ex: smbd_smb2_request_error_ex: idx[1] status[NT_STATUS_ACCESS_DENIED] || at ../../source3/smbd/smb2_create.c:296
[2019/12/14 23:04:53.250038, 10, pid=17854, effective(0, 0), real(0, 0), class=smb2] ../../source3/smbd/smb2_server.c:3142(smbd_smb2_request_done_ex)
smbd_smb2_request_done_ex: idx[1] status[NT_STATUS_ACCESS_DENIED] body[8] dyn[yes:1] at ../../source3/smbd/smb2_server.c:3304

Many, many iterations of samba configs later, I wondered if SELinux was causing a problem. Temporarily disabling SELinux allowed files to be seen in the mapped drive … so that was the problem. I needed to tweak the SELinux settings to allow Samba to actually share files.

semanage fcontext -a -t samba_share_t "/data(/.*)?"

And

setsebool -P samba_export_all_rw=1

Apache — Switching to PHP-FPM

A few system updates ago, PHP fell over completely because of some multi-processing module. The quick fix was to change the multi-processing module and avoid having to figure out what changed and how to use php-fpm. Part of moving my VM’s to the new server, though, is cleaning up anything I’ve patched together as a quick fix. And, supposedly, php-fpm is a lot faster than the old-school Apache handler. Switching was a lot less involved than I had expected.

Install php-fpm:

dnf install php-fpm

Edit 00-mpm.conf

My quick fix was to switch to a non-default multi-processing module. That change is reverted to re-enable the ‘event’ module

vim /etc/httpd/conf.modules.d/00-mpm.conf

Configure Apache PHP Module

Verify the socket name used in /etc/php-fpm.d/ — Fedora is configured from /etc/php-fpm.d/www.conf with a socket at /var/run/php-fpm/www.sock

cp /etc/httpd/conf.modules.d/15-php.conf /etc/httpd/conf.modules.d/15-php.conf.orig
vi /etc/httpd/conf.modules.d/15-php.conf

# Handle files with .php extension using PHP interpreter

# Proxy declaration
<Proxy "unix:/var/run/php-fpm/www.sock|fcgi://php-fpm">
    	ProxySet disablereuse=off
</Proxy>

# Redirect to the proxy
<FilesMatch \.php$>
	SetHandler proxy:fcgi://php-fpm
</FilesMatch>

#
# Allow php to handle Multiviews
#
AddType text/html .php

#
# Add index.php to the list of files that will be served as directory
# indexes.
#
DirectoryIndex index.php

Enable php-fpm to auto-start, start php-fpm, and restart Apache

systemctl enable php-fpm
systemctl start php-fpm
systemctl restart httpd

Voila — phpinfo() confirms that I am using FPM/FastCGI

We’ll see if this actually does anything to improve performance!

External Access to libvirt VMs

Instead of trying to map individual ports over to guest OS’s, I am just routing traffic to the VM bridge from the host.

Testing to ensure it works:

systemctl start firewalld
firewall-cmd –direct –passthrough ipv4 -I FORWARD -i br5 -j ACCEPT
firewall-cmd –direct –passthrough ipv4 -I FORWARD -o br5 -j ACCEPT
firewall-cmd –reload

Permanent setup:

systemctl enable firewalld
firewall-cmd –permanent –direct –passthrough ipv4 -I FORWARD -i br5 -j ACCEPT
firewall-cmd –permanent –direct –passthrough ipv4 -I FORWARD -o br5 -j ACCEPT
firewall-cmd –reload

Then I just added a static route for the network defined on br5 to the VM host.

Migrating from Hyper-V to libvirt

We finally got a new server, and I’m starting to migrate our servers to the new box. We currently have a Windows virtualization platform (Hyper-V) — Windows Data Center edition was supposed to provide unlimited licenses for standard servers running on the host, so it seemed like a great deal. Except “all of the Windows servers” turned out to be, well, one. So we decided to use Fedora on the host. Worst case, that would mean re-installing a few servers. But I wanted to try converting the existing Hyper-V VMs.

Install libvirt and associated packages:

dnf -y install bridge-utils libvirt virt-install qemu-kvm virt-top libguestfs-tools qemu-img virt-manager

Start libvirtd and set it to auto-start on boot:

systemctl start libvirtd
systemctl enable libvirtd

Create an XML file with the definition for a new bridge:

[root@localhost ~]# cat br5.xml

<network>
<name>br5</name>
<forward mode=’nat’>
<nat>
<port start=’1024′ end=’65535’/>
</nat>
</forward>
<bridge name=’br5′ stp=’on’ delay=’0’/>
<ip address=’10.1.2.1′ netmask=’255.255.255.0′>
<dhcp>
<range start=’10.1.2.200′ end=’10.1.2.250’/>
</dhcp>
</ip>
</network>

Build a new bridge from this definition and set it to auto-start on boot:

[root@localhost ~]# virsh net-define br5.xml
Network br10 defined from br5.xml

[root@localhost ~]# virsh net-autostart br5
Network br5 marked as autostarted

Verify the network is running and set to auto-start

[root@localhost ~]# virsh net-list –all
Name State Autostart Persistent
———————————————-
br5 active yes yes

View the IP address associated with the bridge:

[root@localhost ~]# ip addr show dev br5
5: br5: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 52:54:00:33:3f:0c brd ff:ff:ff:ff:ff:ff
inet 10.1.2.1/24 brd 10.1.2.255 scope global br10
valid_lft forever preferred_lft forever

Copy the VHDX from Hyper-V to the Linux host and convert it to a qcow2 image:

qemu-img convert -O qcow2 fedora02.vhdx fedora02.qcow2

If needed, sysprep to clean up system SSH host keys, persistent network MAC configuration, and removing user accounts.
virt-sysprep -a fedora02.qcow2

When finished, use virt-manager to create a host by importing an existing HDD. Provided the drive type remains the same (SATA, in my case), the server boots right up.

Recovering a bricked Netgear router

Netgear provides instructions for using TFTP to write firmware to a basically bricked router (it boots into a recovery mode, indicated by a flashing power light). The instructions are, unfortunately, specific to Windows. To use a Linux computer to recover the router

 

(1) Plug your computer into the router & unplug everything else, as in the instructions. Hard-code an IP address. Then verify that the router shows up in your arp table:

arp -a

If the router does not appear, add it — you’ll need to get the device MAC address from the sticker on the back of the device.

arp -s 192.168.1.1 ??-??-??-??-??-??

(2) If you don’t already have a TFTP client, install one. Once you have a client, follow the instructions to get the router into recovery mode. On the Linux computer, run “tftp 192.168.1.1”

You’ll be in a TFTP console. Type binary and hit enter to set the transfer mode to binary. Then use put /path/to/file.name to upload the firmware file to the device. Wait and proceed with device setup.

 

Shell Script: Path To Script

We occasionally have to re-home our shell scripts, which means updating any static path values used within scripts. It’s quick enough to build a sed script to convert /old/server/path to /new/server/path, but it’s still extra work.

The dirname command works to provide a dynamic path value, provided you use the fully qualified path to run the script … but it fails spectacularly whens someone runs ./scriptFile.sh and you’re trying to use that path in, say, EXTRA_JAVA_OPTS. The “path” is just . — and Java doesn’t have any idea what to do with “-Xbootclasspath/a:./more/path/goes/here.jar”

Voila, realpath gives you the fully qualified file path for /new/server/path/scriptFile.sh, ./scriptFile.sh, or even bash scriptFile.sh … and the dirname of a realpath is the fully qualified path where scriptFile.sh resides:

#!/bin/bash 
DIRNAME=`dirname $(realpath "$0")`
echo ${DIRNAME}

Hopefully next time we’ve got to re-home our batch jobs, it will be a simple scp & sed the old crontab content to use the new paths.