Month: July 2019

Cleaning Up Unused Docker Images

I’ve been using Docker for quite some time, but never had unused container images. This is partially because I installed a new hard drive and started from a blank slate, but also because I haven’t needed to use many different images to build my containers.

I’ve changed jobs recently and wanted to set up a container to mirror our web server. Which meant trying to get a CentOS 6.8 container going. Except there isn’t one from Cent anymore. And I don’t exactly trust random-dude-from-the-Internet’s OS. Download it and poke around without running it, sure … but that’s not a platform on which I can do my development.

And that means I’ve got a few images that I do not need. To view the list of images, use “docker images -a”

 

D:\docker>docker images -a
REPOSITORY TAG IMAGE ID CREATED SIZE
openhab/openhab snapshot 8a4749c86ff3 4 weeks ago 527MB
docker4w/nsenter-dockerd latest 2f1c802f322f 9 months ago 187kB
centos/php-56-centos7 latest 92ed8b3a7cb4 15 months ago 617MB

13652604711/centos6.8-ssh latest 59ab169b5158 2 years ago 289MB

Then use “docker rmi imagename” to remove any unnecessary ones.

D:\docker>docker rmi centos/php-56-centos7
Untagged: centos/php-56-centos7:latest
Untagged: centos/php-56-centos7@sha256:f3c95020fa870fcefa7d1440d07a2b947834b87bdaf000588e84ef4a599c7546
Deleted: sha256:92ed8b3a7cb4d56d3a1c58386d966f22736010a292a81a72dddbc4ffc7cae3fd
Deleted: sha256:bdcb229c59ed69d26750cd0d24362670e1fa2ae9be6ef19aa3e7c5571a4a8503
Deleted: sha256:90eb7fca62f6c0febd9cc21544269029ff231f39f16054ba6b0ca93ec1037d97
Deleted: sha256:cdcf05e149fc6cb2801f7f93dce3acb54465fe6c46a16dd6135aa74d79bedffa
Deleted: sha256:139498a5907a4d17cf07b1400bdbdb4db5e9f1ac4e3985aac2b374eaa712d5fb
Deleted: sha256:5f0780b14e43db37e84162e0045657203ac1e9fb531cc3e879fa464eda013e79
Deleted: sha256:7e117241875497974bb56f09e6340e142a9acaa11af76917afab345acc25b5c1
Deleted: sha256:4b170488c295918f4d7618c2cd0b9b428d55ec952dd6a715593e3af34e538d94
Deleted: sha256:1e889f7360c52d1b20f93335382290445e4f257f08ccef01694837572842e95f
Deleted: sha256:43e653f84b79ba52711b0f726ff5a7fd1162ae9df4be76ca1de8370b8bbf9bb0

D:\docker>docker rmi 13652604711/centos6.8-ssh
Untagged: 13652604711/centos6.8-ssh:latest
Untagged: 13652604711/centos6.8-ssh@sha256:41bbe66ac18f199efac325d0d4bcb5d0390ec501ca82d6d1ce223df8a050be3a
Deleted: sha256:59ab169b5158a172079e2a89442936bc49292ea951f2eb9acb688a0ee34f95e1
Deleted: sha256:12d850520660ec9de87e84735a7067e663db282245502820f09dae5c937a93d2
Deleted: sha256:6b5c6954e3d511934786375730a068d0f013dcc99356a341a8c5d268a3b1cf3d

Followup – Straws as a Marketing Stunt

Well, I wasn’t wrong ๐Ÿ™‚ Plastic straws + free media attention were a great combination for the Trump campaign. 140k straws at 15$ a pack is over 2 million dollars. They cleared 200k on that, which isn’t bad for a week or two of fundraising. And half of the purchasers were new donors — which means a lot of new contact information to solicit future donations and to target “get out the vote” efforts. And it’s pretty easy to figure out what message will entice this demographic.

openHAB – Motion Detection With Zoneminder Via SQL Triggers

We had used ZoneMinder filters to run a script which turned a “motion detected” switch on and off in openHAB. We had turned that off in favor of an openHAB/ZoneMinder binding; but the binding polled ZoneMinder for motion events, and this added significant load to our system. We tried re-enabling the filters we’d used previously, but they didn’t work. There are a lot of caveats around using filters (tl;dr: filtering can be delayed by several minutes, which renders ‘now’ filters ineffective) and more recent versions of ZoneMinder don’t have a number of alarm frames until after the event (which means filtering on alarm frames > 1 only detects motion after the fact). All of this means that the filters which worked pretty well a year or two ago no longer work reliably. Architecturally, the ZoneMinder filter process seemed ill suited for our needs. Actions that are not time sensitive, like file cleanup or roll-up reporting, could be done through a filter. But it’s not a good solution for identifying the FexEx guy in the driveway.

ZoneMinder uses a database to maintain system and alert data — I use MariaDB 10.3.18-1. MySQL introduced TRIGGER back in version 5. A trigger is essentially a bit of SQL automatically executed by the database when operations occur within a table — table activity triggers execution. When ZoneMinder first detects motion, an event is recorded in the database. When motion is no longer detected, the motion event is updated with event info (number of frames, event duration). Since both inserting a motion event and updating the event when motion ends are events within tables, a trigger can execute some SQL code almost immediately without much impact to system load.

The only problem is that SQL code does not, normally, POST data to a URI. Creating a trigger which can execute external binaries requires creating a UDF (user-defined function). I am using lib_mysqludf_sys which creates sys_get, sys_set, sys_exec, and sys_eval functions. The sys_get and sys_set functions are used for setting/getting environment variables. The sys_exec function returns the return code from execution, whereas sys_eval returns the output from execution.

Adding SYS UDF’s To MariaDB:

After cloning the lib_mysqludf_sys repo locally, edit Makefile to set LIBDIR to the appropriate directory for the MariaDB installation (/usr/lib64/mariadb/plugin/ in my case). I also needed to modify the compilation line to:

gcc -fPIC -Wall -I/usr/include/mysql/server -I. -shared lib_mysqludf_sys.c -o $(LIBDIR)/lib_mysqludf_sys.so

** 01 August 2020 update — I had to include an additional folder to build the latest version of this program on Fedora 31.

Run install.sh to install and register the user-defined functions in the MariaDB server. Because the output of command execution is unnecessary, the sys_exec is sufficient. Before registering a trigger, use the CLI SQL to verify sys_exec is working:

MariaDB [zm]> SELECT sys_exec('cat /etc/fedora-release');
+-------------------------------------+
| sys_exec('cat /etc/fedora-release') |
+-------------------------------------+
| 0 |
+-------------------------------------+
1 row in set (0.012 sec)

Creating the SQL Trigger:

To create a trigger for motion events, there needs to be a mapping between the monitorID used in ZoneMinder. You see the monitorID in the URL when you view a feed — “mid” in the GET query string:

Or use a SQL client to obtain a list of monitors from the ZoneMinder database:

MariaDB [zmdb]> select Id, Name from Monitors;
+----+-----------------------------------+
| Id | Name                              |
+----+-----------------------------------+
| 15 | IPCam01 - Area 123                |
| 16 | IPCam02 - Area 234                |
| 17 | IPCam03 - Area 345                |
| 18 | IPCam04 - Area 456                |
| 19 | IPCam05 - Area 567                |
+----+-----------------------------------+

Once you can correlate monitor ID values to OpenHAB items, update the IF/THEN section of the trigger. Update the strOpenHABHost variable to your server URL. There are two useful SQL commands commented out (– ) below. SHOW TRIGGERS does exactly that – it lists triggers that are registered in the database. DROP TRIGGER is used to remove the trigger. If you are using HTTPS to communicate with OpenHAB, you may need to add “–insecure” to the curl command to ignore certificate errors (or use –cacert to to establish a trust chain).

The sys_exec function in this trigger uses curl to post an item stage change to the OpenHAB REST API. Camera items are on when motion is detected.

To create the TriggerMotionOnNewEvent trigger, paste the following into your SQL client:

-- SHOW TRIGGERS
-- DROP TRIGGER zm.TriggerMotionOnNewEvent;
DELIMITER @@

CREATE TRIGGER TriggerMotionOnNewEvent
AFTER INSERT ON `Events`
FOR EACH ROW
BEGIN

DECLARE strCommand CHAR(255);
DECLARE strCameraName CHAR(64);
DECLARE iCameraID INT(10);
DECLARE iResult INT(10);
-- variables for local openHAB REST API hostname and port
DECLARE strOpenHABHost CHAR(64);
SET strOpenHABHost='http://openhabhost.example.com:8080';


-- Translate ZoneMinder IP camera ID with openHAB item name
SET iCameraID = NEW.monitorID;
IF(iCameraID = 10) THEN
SET strCameraName='IPCam05_Alarm';
ELSEIF(iCameraID = 11) THEN
SET strCameraName='IPCam03_Alarm';
ELSEIF(iCameraID = 12) THEN
SET strCameraName='IPCam04_Alarm';
ELSEIF(iCameraID = 13) THEN
SET strCameraName='IPCam01_Alarm';
ELSEIF(iCameraID = 14) THEN
SET strCameraName='IPCam02_Alarm';
END IF;

SET strCommand=CONCAT('/usr/bin/curl ', '-s --connect-timeout 10 -m 10 -X PUT --header "Content-Type: text/plain" --header "Accept: application/json" -d "ON" "',strOpenHABHost,'/rest/items/',strCameraName,'/state"');
SET iResult = sys_exec(strCommand);
END;
@@
DELIMITER ;

There is a second trigger to clear the motion event — set the camera item to off when there is no longer motion detected. ZoneMinder updates event records to record and EndTime for the event. This trigger executes any time an Event item is updated, but there is an IF statement that verifies that the EndTime is not null to avoid clearing the motion event too soon.

To create the ClearMotionOnEventEnd trigger, paste the following into your SQL client (at some point, the Events table EndTime column was renamed to match the DateTime column format — so it is now called EndDateTime … I’ve updated the trigger with the new column name; but, if your motion events do not clear, try using “describe Events” to see what the column name for the event end time is):

-- SHOW TRIGGERS
-- DROP TRIGGER zm.ClearMotionOnEventEnd;
DELIMITER @@

CREATE TRIGGER ClearMotionOnEventEnd
AFTER UPDATE ON `Events`
FOR EACH ROW
BEGIN

DECLARE strCommand CHAR(255);
DECLARE iResult int(10);
DECLARE strCameraName CHAR(25);
DECLARE iCameraID int(5);
-- variables for local openHAB REST API hostname and port
DECLARE strOpenHABHost CHAR(64);
SET strOpenHABHost='http://openhabhost.example.com:8080';

-- Translate ZoneMinder IP camera ID with openHAB item name
SET iCameraID = NEW.monitorID;
IF iCameraID = 10 THEN
SET strCameraName='IPCam05_Alarm';
ELSEIF iCameraID = 11 THEN
SET strCameraName='IPCam03_Alarm';
ELSEIF iCameraID = 12 THEN
SET strCameraName='IPCam04_Alarm';
ELSEIF iCameraID = 13 THEN
SET strCameraName='IPCam01_Alarm';
ELSEIF iCameraID = 14 THEN
SET strCameraName='IPCam02_Alarm';
END IF;

IF NEW.EndDateTime IS NOT NULL THEN
SET strCommand=CONCAT('/usr/bin/curl ', '-s --connect-timeout 10 -m 10 -X PUT --header "Content-Type: text/plain" --header "Accept: application/json" -d "OFF" "',strOpenHABHost,'/rest/items/',strCameraName,'/state"');
SET iResult = sys_exec(strCommand);
END IF;

END;
@@
DELIMITER ;

Now when new motion detection events are inserted into the Events database table, the openHAB item corresponding to the camera will be turned on. When the event record is updated with an end timestamp, the openHAB item corresponding to the camera will be turned off.

Our implementation executes a second external command. Getting notified of motion when we’re home is great — pull up ZoneMinder, see the FedEx truck. But we don’t publish most of our infrastructure to the Internet — watching the video feed from ZoneMinder means VPN’ing into the network. I put together a quick shell script to pull the 25th image from the motion event (we retain a few seconds prior to motion being detected, and the number of frames recorded per second will vary … so there is trial-and-error involved in identifying an early-in-the-event frame that includes the triggering object). The sleep ensures enough time has elapsed for the motion images to be committed to disk.

#!/bin/bash
# parameter 1 is camera ID
# parameter 2 is camera name
# parameter 3 is event ID
sleep 5
strDate=$(date +%F)
strFile='/mnt/data/zoneminder/events/'$1'/'$strDate'/'$3'/00025-capture.jpg'
echo $strFile

echo "Image for event ID $2 on $strDate is attached to this message" | mailx -r "zoneminder@example.com" -s "$2 Motion Event" -a $strFile Us@example.com

TriggerMotionOnNewEvent includes the following two lines to trigger execution of the shell script when motion is detected.

SET strCommand=CONCAT('/path/to/shell/scripts/sendZoneminderEventImage.sh ',iCameraID,' "',strCameraName,'" ',NEW.Id,"&");
SET iResult=sys_exec(strCommand);

In doing so, we have an e-mail on our phones with a JPG from the motion event — I can quickly see the difference between a cat and a cat-burgler prowling around the patio when we’re away from home.

Un-killable Process

Scott had a Dolphin instance veg out on him. Not much for it other than killing the process. Except it didn’t kill. Now SIGTERM (15) I don’t expect to kill a vegged out process, but SIGKILL (9)? I’ve never seen that fail. A little research later, and I’ve discovered “uninterruptible sleep”.ย  Which, at 11PM is starting to sound really good to me. But not something I associate with computer programs. Essentially, processes that are waiting on I/O very briefly pop into this state and pop out of the state when the I/O operation completes. Code needs to have timeouts to prevent the application from getting stuck waiting for I/O. And, evidently, Scott has discovered a scenario in which Dolphin does not have a timeout.

How can you tell that your process is stuck in uninterruptible sleep? Use “ps u” (or “ps aux” forย all processes) and check the “STAT” column.

From “man ps”:

PROCESS STATE CODES
Here are the different values that the s, stat and state output
specifiers (header "STAT" or "S") will display to describe the state of
a process:

D uninterruptible sleep (usually IO)
I Idle kernel thread
R running or runnable (on run queue)
S interruptible sleep (waiting for an event to complete)
T stopped by job control signal
t stopped by debugger during the tracing
W paging (not valid since the 2.6.xx kernel)
X dead (should never be seen)
Z defunct ("zombie") process, terminated but not reaped by its parent

For BSD formats and when the stat keyword is used, additional
characters may be displayed:

< high-priority (not nice to other users)
N low-priority (nice to other users)
L has pages locked into memory (for real-time and custom IO)
s is a session leader
l is multi-threaded (using CLONE_THREAD, like NPTL pthreads do)
+ is in the foreground process group

Rebooting clears the process (or sorting whatever is blocking the I/O operation). But there are processes that “kill -9” won’t terminate.

Straws as a Marketing Stunt

As a campaign/marketing stunt, the Trump campaign’s plastic straws are brilliant. It is a solid component of the “troll the liberals” campaign plank. Garnered a lot of attention (not *good* attention, but that seemingly doesn’t matter). Sure, major news outlet aren’t exactly saying “go shop the Trump reelection store!!!”, but media outlets are still providing free advertising for Trump.

Recreating Grub Bootloader After the Windows Install Wipes It

Setting up the dual-boot Windows/Fedora system was straight-forward on my laptop. I installed Windows, then installed Linux and grub mkconfig found Windows and included it in the menu. Scott already had Fedora, and we needed to repair his Windows installation. Which, of course, blew away grub. Easy enough to get back, provided you’ve got a Live USB installation from which to boot.

Boot the Live media and use fdisk to find the Linux partition (in our installations, /boot is contained within the root partition).

[root@fedora02 ~]# fdisk -l
Disk /dev/sda: 10 GiB, 10737418240 bytes, 20971520 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0xd847fbc2

Device Boot Start End Sectors Size Id Type
/dev/sda1 * 2048 1026047 1024000 500M 83 Linux
/dev/sda2 1026048 20971519 19945472 9.5G 8e Linux LVM

Mount that partition somewhere:

mkdir /mnt/mycomputer
mount /dev/sda2 /mnt/mycomputer

Add bind mounts so /dev and /proc are in there

mount –bind /dev /mnt/mycomputer/dev
mount –bind /proc /mnt/mycomputer/proc

Chroot yourself into the mount point

chroot /mnt/mycomputer

Now you can reinstall grub. You don’t want the partition (e.g. /dev/sda2) but the disk. The following commands install the grub2 bootloader and reboot.

grub2-install /dev/sda
reboot

If you forget to pay attention on boot and thus inadvertently end up in the default operating system (<G>), edit /etc/default/grub and increase “GRUB_TIMEOUT=5”. Build the grub config — this should identify your Windows partition and include it in the menu

grub2-mkconfig -o /boot/grub2/grub.cfg

Reboot again, and you’ll be able to select between Windows and Linux.

Using grub rescue to boot machine and repair MBR

Using grub rescue to boot machine and repair MBR

Use “ls” to find your partition list:
ls
(hd0) (hd0,msdos3) (hd0,msdos2) (hd0,msdos1)

Check the content of each to find your Linux partition:
ls (hd0,msdos3)/
ls (hd0,msdos2)/
ls (hd0,msdos1)/

You want the one with the /boot folder. In our case, this is (hd0,msdos3). The following commands will boot your Linux OS.

set prefix=(hd0,msdos3)/boot/grub2
set root=(hd0,msdos3)
insmod normal
normal

<root password for maintenance>

Use “df” or “mount” to figure out which disk holds the Linux partition. In our case, it is /dev/sdb. You don’t want the partition (e.g. /dev/sdb2) but the disk. The following commands install the grub2 bootloader and build a config file.

grub2-install /dev/sdb
grub2-mkconfig -o /boot/grub2/grub.cfg

Embroidery Thread Storage

Two years ago, I picked up a bunch of ArtBin 9101AB boxes for 9$ each in a pre-Christmas sale. They stack nicely, and I use them to store zippers, fold over elastic, sewing feet, beads … all sorts of craft supplies. I’m trying to use them to store my embroidery thread. I have a lot — a whole rainbow of colors, plus another box with grays, browns, and ‘special’ thread (metallic, glow in the dark). But the little bobbin cards aren’t quite big enough. It’s still an improvement over random skeins of thread, and unused portions can be wrapped around the bobbin card to ensure you know exactly which light blue that bit is. Hopefully I’ll come up with something that keeps these things upright … other than stuffing the box so there’s nowhere for them to move ๐Ÿ˜€

Linux – Finding “Missing” Disk Space

You can have ‘stuff’ in a folder, mount a partition to that folder, and have disk space used for which you cannot account. Using a tool like df, you will see that 10GB of your 10GB disk is used, but using du the individual folders only account for 5GB.

Rather than randomly umounting partitions to see if there’s anything in the mount point folder, you can bind mount root to another location and check the disk utilization on the new mount.

 

[root@fedora123 ~]# mount -o bind / /mnt/fakeout/
[root@fedora123 ~]# du -sh `ls /mnt/fakeout | grep -v mnt`
0 bin
0 boot
280K ca
8.0K cacert.pem
4.0K careq.pem
0 certs
0 crl
0 dev
44M etc
52K home
0 index.txt
0 lib
0 lib64
0 media
0 newcerts
0 openhab
724K opt
4.0K private
0 proc
847M root
0 run
0 sbin
227M srv
0 sys
4.0K tmp
3.0G usr
4.0K var
[root@fedora123 ~]# umount fakeout