After the power came back on, I had to recover ext and xfs partitions —
# FSCK to clean up bad superblock
lsblk
fsck /dev/sdb
e2fsck -b 32768 /dev/sdb1
mount /dev/sdb1 /var
# xfs_repair for XFS FS on LVM
lvscan # get dev for LVM
mount /dev/fedora/root /mnt/oh
xfs_repair -L /dev/fedora/root
mount /dev/fedora/root /mnt/oh
We had a power outage on Monday that took out the drive that holds our VMs. There are backups, but the backup drive copies had superblock errors and all sorts of issues. To recover our data, I learned all sorts of new things — firstly that you can mount a QCOW file and copy data out. First, you have to connect a network block device to the file. Once it is connected, you can use fdisk to list the partitions on the drive and mount those partitions. In this example, I had a partition called nbd0p1 that I mounted to /mnt/data_recovery
modprobe nbd max_part=2
qemu-nbd --connect=/dev/nbd0 /path/to/server_file.qcow2
fdisk /dev/nbd0 -l
mount /dev/nbd0p1 /mnt/data_recovery
Once you are done, unmount it and disconnect from the network block device.
umount /mnt/data_recovery
qemu-nbd --disconnect /dev/nbd0
rmmod nbd
To view the scripts being executed by an RPM, use rpm -qp --scripts packagename
We needed to console into some Cisco access points — RJ45 to USB to plug into the device console port and the laptop’s USB port? Check! OK … now what? Turns out you can use the screen command as a terminal emulator. The basic syntax is screen <port> <baud rate> — since the documentation said to use 9600 baud and the access point showed up on /dev/ttyUSB0, this means running:
screen /dev/ttyUSB0 9600More completely, screen <port> <baud rate>,<7 or 8 bits per byte>,<enable or disable sending flow control>,<enable or disable rcving flow control>,<keep or clear the eight bit in each byte>
screen /dev/ttyUSB0 9600,cs8,ixon,ixoff,istrip- or -screen /dev/ttyUSB0 9600,cs7,-ixon,-ixoff,-istrip
While using telnet to connect to a host:port is a quick way to verify connectivity … it only works for tcp ports. There’s no udp telnet. But there are a few ways to verify a port is open and available to you …
nmap with -sU will show the port as open or closed
nc will send a packet
I’m using IPv6 on a server — the server wasn’t using NetworkManagement, so I’ve configured it directly in the network script file.
After restarting the network (systemctl restart network), I was able to ping other IPv6 addressed equipment.
We occasionally get alerted that our /var volume is over 80% full … which generally means /var/log has a lot of data, some of which is really useful and some of it not so useful. The application-specific log files already have the shortest retention period that is reasonable (and logs that are rotated out are compressed). Similarly, the system log files rotated through logrotate.conf and logrotate.d/* have been configured with reasonable retention.
Using du -sh /var/log/ showed the /var/log/sa folder took half a gig of space.
This is the daily output from sar (a “daily summary of process accounting” cron’d up with /etc/cron.d/sysstat). This content doesn’t get rotated out with the expected logrotation configuration. It’s got a special configuration at /etc/sysconfig/sysstat — changing the number of days (or, in my case, compressing some of the older files) is a quick way to reduce the amount of space the sar output files consume).
To copy files and keep the original timestamp, use
cp --preserve=timestamps /path/to/source.txt /path/to/destination.txt
“Dependency hell” used to be a big problem — you’d download one package, attempt to install it, and find out you needed three other packages. Download one of them, attempt to install it, and learn about five other packages. Fifty seven packages later, you forgot what you were trying to install in the first place and went home. Or, I suppose, you managed to install that first package and actually use it. The advent of repo-based deployments — where dependencies can be resolved and automatically downloaded — has mostly eliminated dependency hell. But, occasionally, you’ll have a manual install that says “oh, I cannot complete. I need libgdkglext-x11-1.0.so.0 or libminizip.so.1 … and, if there’s a package that’s named libgdkglext-x11 or libminizip … you’re good. There’s not. Fortunately, you can use “dnf provides” to search for a package that provides a specific file — thus learning that you need the gtkglext-libs and minizip-compat packages to resolve your dependencies.
Evidently you cannot just scp files from an old Solaris box when you’re on a RHEL/CentOS system … there’s an incompatibility between them that requires you to (1) install scp1 on the Solaris server {not likely in a prod environment} or (2) use sftp to transfer the files.
Server1: Red Hat Enterprise Linux Server release 7.6 (Maipo)
Server2: Solaris 5.9
lisa@server1:~/$ scp lisa@server2:/data/stuff/file1.txt ./input/
lisa@server2’s password:
scp: warning: Executing scp1.
scp: FATAL: Executing ssh1 in compatibility mode failed (Check that scp1 is in your PATH).