Category: System Administration

June 1, 2018

OUD Returning Some DirectoryString Syntax Values As UTF-8 Encoded Bytes

We are still in the process of moving the last few applications from DSEE to OUD 11g so the DSEE 6.3 directory can be decommissioned. Just two to go! But the application, when pointed to the OUD servers, gets “Unable to cast object of type ‘System.Byte[]’ to type ‘System.String'” when retrieving values for a few of our DirectoryString syntax custom schema.

This code snippet works fine with DSEE 6.3.

string strUserGivenName = (String)searchResult.Properties["givenName"][0]; 
string strUserSurame = (String)searchResult.Properties["sn"][0]; 
string strSupervisorFirstName = (String)searchResult.Properties["positionmanagernamefirst"][0]; 
string strSupervisorLastName = (String)searchResult.Properties["positionmanagernamelast"][0];

Direct the connection to the OUD 11g servers, and an error is returned.

The attributes use the same syntax – DirectoryString, OID 1.3.6.1.4.1.1466.115.121.1.15.

00-core.ldif:attributeTypes: ( 2.5.4.41 NAME ‘name’ EQUALITY caseIgnoreMatch SUBSTR caseIgnoreSubstringsMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.15{32768} X-ORIGIN ‘RFC 4519’ )
00-core.ldif:attributeTypes: ( 2.5.4.4 NAME ( ‘sn’ ‘surname’ ) SUP name X-ORIGIN ‘RFC 4519’ )
00-core.ldif:attributeTypes: ( 2.5.4.42 NAME ‘givenName’ SUP name X-ORIGIN ‘RFC 4519’ )

99-user.ldif:attributeTypes: ( positionManagerNameMI-oid NAME ‘positionmanagernamemi’ DESC ‘User Defined Attribute’ SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 SINGLE-VALUE X-ORIGIN ‘user defined’ )
99-user.ldif:attributeTypes: ( positionManagerNameFirst-oid NAME ‘positionmanagernamefirst’ DESC ‘User Defined Attribute’ SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 SINGLE-VALUE X-ORIGIN ‘user defined’ )
99-user.ldif:attributeTypes: ( positionManagerNameLast-oid NAME ‘positionmanagernamelast’ DESC ‘User Defined Attribute’ SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 SINGLE-VALUE X-ORIGIN ‘user defined’ )

I’ve put together a quick check to see if the returned value is an array, and if it is then get a string from the decoded byte array.

string strUserGivenName = (String)searchResult.Properties["givenName"][0]; 
string strUserSurame = (String)searchResult.Properties["sn"][0]; 

string strSupervisorFirstName = "";
string strSupervisorLastName = "";
if (searchResult.Properties["positionmanagernamefirst"][0].GetType().IsArray){
    strSupervisorFirstName = System.Text.Encoding.UTF8.GetString((byte[])searchResult.Properties["positionmanagernamefirst"][0]);
}
else{
    strSupervisorFirstName = searchResult.Properties["positionmanagernamefirst"][0].ToString();
}

if (searchResult.Properties["positionmanagernamelast"][0].GetType().IsArray){
    strSupervisorLastName = System.Text.Encoding.UTF8.GetString((byte[])searchResult.Properties["positionmanagernamelast"][0]);
}
else{
    strSupervisorLastName = searchResult.Properties["positionmanagernamelast"][0].ToString();
}

Voila

The outstanding question is if we need to wrap *all* DirectoryString syntax attributes in this check to be safe or if there’s a reason core schema attributes like givenName and sn are being returned as strings whilst our add-on schema attributes have been encoded.

May 31, 2018

Isolated Guest Network On Merlin 380.69_2 (Asus RT-AC68R)

We finally got rid of Time Warner Cable / Spectrum / whatever they want to call themselves this week’s overpriced Internet that includes five free outages between 1100 and 1500 each day. But the firmware on the new ISP’s router doesn’t have a facility to back up the config. And if we’re going to have static IPs for all of our speakers, printers, servers … we don’t want to have to re-enter all of that data if the router config gets reset. Same with configuring the WiFi networks. And, and and. So instead of using the snazzy new router, we are using our old router on .2, the new router on .1 … and everything actually connects to the old router, uses the DHCP server on the old router. And only uses the new router as its default gateway. Worked fine until we tried to turn on the guest network.

I found someone in Internet-land who has the exact same configuration and wants to permit guests to use the LAN printer. His post included some ebtables rules to allow guest network clients access to his printer IP. Swapped his printer IP for our router IP and … nada.

And then I realized that the router is not the packet destination IP when the guest client attempts to communicate outside our network. The router is the destination MAC address. So you cannot add an ebtables rule to the router’s IP address and expect traffic to flow.

The first thing you need to do is figure out the upstream router’s MAC address. From the Asus, you can query the arp table. If the command says “No match found in # entries”, ping the router and try again.

root@ASUS-RT-AC68R:/tmp/home/root# arp -a 10.5.5.1
? (10.5.5.1) at a3:5e:c4:17:a3:c0 [ether] on br0

The six pairs of hex numbers separated by colons – that’s the MAC address. You have to allow bidirectional communication from the guest network interface (wl0.2 for us) with the upstream router’s MAC address. You also have to allow broadcast traffic so guest devices are able to ARP for the router’s MAC address.

To have a persistent config, enable jffs and add the config lines to something like services-start:

root@ASUS-RT-AC68R:/tmp/home/root# cat /jffs/scripts/services-start
#!/bin/sh
logger "SERVICES-START: script start"
# Prevent Echo dots from sending multicast traffic to speaker network
ebtables -I FORWARD -o wl0.1 --protocol IPv4 --ip-source 10.0.0.36 --ip-destination 239.255.255.250 -j DROP
# Guest network - allow broadcast traffic so devices can ARP for router MAC
ebtables -I FORWARD -d Broadcast -j ACCEPT
# Guest network - allow communication to and from router MAC
ebtables -I FORWARD -s a3:5e:c4:17:a3:c0 -j ACCEPT
ebtables -I FORWARD -d a3:5e:c4:17:a3:c0 -j ACCEPT
# This should be automatically added for guest network, but it goes missing sometimes so I am adding it again
ebtables -A FORWARD -o wl0.2 -j DROP
ebtables -A FORWARD -i wl0.2 -j DROP

Use -L to view your ebtables rules:

root@ASUS-RT-AC68R:/tmp/home/root# ebtables -L
Bridge table: filter

Bridge chain: INPUT, entries: 0, policy: ACCEPT

Bridge chain: FORWARD, entries: 16, policy: ACCEPT
-d a3:5e:c4:17:a3:c0 -j ACCEPT
-s a3:5e:c4:17:a3:c0 -j ACCEPT
-d Broadcast -j ACCEPT
-p IPv4 -o wl0.1 --ip-src 10.0.0.36 --ip-dst 239.255.255.250 -j DROP
-o wl0.2 -j DROP
-i wl0.2 -j DROP

Voila, guests who can access the Internet & DNS on the .1 router, but cannot access anything on the internal network. Of course you can add some specific IPs as allowed destinations too – like the printers in the example that started me down this path.

May 14, 2018

DSEE 6.3 To OUD 11g Transition

There’s no direct path to replicate data from DSEE6.3 to OUD11g. Not unreasonable since DSEE is the Sun product based on the Netscape Directory Server and OUD is the Oracle product based on OpenLDAP – they weren’t exactly designed to allow easy coexistence that would permit customers to switch from one to the other. Problem is, with Oracle’s acquisition of Sun & axing the DSEE product line … customers *need* to interoperate or do a flash cut.

Since our Identity Management (IDM) platform was not able to prep development work and implement their changes along with the directory replacement, a flash cut was right out. I’ve done flash cuts before — essentially ran two completely different directories in parallel with data fed from the Identity Management platform, tested against the new directory using quick modification to the OS hosts file, then reconfiguring the virtual IP on the load balancer to direct the existing VIP to the new service hosts. Quick/easy fail-back is to set the VIP to the old config and sort out whatever is wrong on the new hosts. A lot lower risk than a traditional ‘flash cut’ approach as long as you trust the IDM system to keep data in sync. But lacking an IDM system, flash cut is typically a non-starter anyway.

There is a migration path. Oracle put some development effort into the DSEE product line prior to discontinuing it. DSEE7 was the Sun distributed “next version”. It was not widely deployed prior to the Oracle acquisition. Oracle took over DSEE7 development but called it DSEE11 (to match the OUD version numbering, I guess?). Regardless of the rational, you’ll see the “next version” DSEE product referred to as both DSEE7 and DSEE11.

There’s not a direct replication between Oracle DSEE11 and Oracle OUD11. Oracle created a “replication gateway” that handles, among other things, schema name mapping (only Netscape would use attribute names like nsAccountLockout, and that nomenclature carried through to the Sun product). Oracle did a decent job of testing DSEE11<=>OUD11 Replication Gateway interoperability. I don’t know if they just assumed DSEE6 would work because DSEE11 did or if they assumed the installation base for DSEE6 was negligible (i.e. didn’t bother to test older revisions) but we found massive bugs in the replication gateway working with DSEE6. “You cannot import the data to initialize the OUD11 directory” type of bugs which I was willing to work around by manually editing the export file, but subsequent “updates do not get from point ‘A’ to point ‘B’ bugs too. The answer from Oracle was essentially “upgrade to DSEE11” … which, if i could flash-cut upgrade DSEE6 to DSEE11 (see: IDM platform couldn’t do that), I could just cut it to OUD11 and be done. Any non-trivial change was a non-starter, but Oracle wasn’t going to dump a bunch of development time into fixing replication for a dead product to their shiny new thing.

I worked out a path that used tested and working components — DSEE6 replicated just fine with DSEE11. DSEE11 replicated just fine with the OUD11g replication gateway, and the OUD11g replication gateway replicated fine with OUD11g. Instead of introducing additional expense and time setting up dedicated replication translation servers, I installed multiple components on the new servers. There is a DSEE11 directory on one of the new OUD servers, the replication gateway on another one of the new OUD servers, and (of course) the OUD11g directory that we actually intended to run on the new servers is on those new OUD servers.

This creates additional monitoring overhead – watching replication between three different directories and ensuring all of the services are running – but allows the IDM platform to continue writing changes to the DSEE6.3 directory until they are able to develop and test changes that allow them to use OUD11g directly.

May 7, 2018

Systemd (a.k.a. where did my log files go!?!?!)

A systemd Primer For sysvinit Users

Background:

Starting in Fedora 15 and RHEL 7, systemd replaces sysvinit. This is a touchy subject among Unix folks – some people think it’s a great change, others think Linux has been ruined forever. Our personal opinions of the shift doesn’t matter: vendors are implementing it, WIN Linux servers use it, so we need to know it. Basically, throw “systemd violates the minimalist, modular philosophy at the core of Unix development” on the “but emacs is so awesome, why are we using vim” and “BETA outperforms VHS any day of the week” pile.

Quick terminology – services are now called units. You’ll see that word a lot. A unit is configured in a “unit file”. Additionally, “run levels” (0-6) have been replaced with the concept of “targets” that have friendly names.

What’s the difference?

Sysvinit wasn’t designed to know about your system, it was designed to run scripts on your system. Sysvinit essentially runs scripts, whereas systemd is a service manager. Systemd knows about the system. One place this becomes apparent – if you manually run the run line from a sysvinit script then check the service status, it will show running because the binary has a PID. If you do the same with systemd, it will say the service is down. This is like Windows – if you have a Docker service that runs “”C:\Program Files\Docker\Docker\com.docker.service”” set to run manually, and use start-run to run the exact same string … the service will not show as running.

Systemd manages a lot of different unit types. As application owners, we’ll use ‘service’ units. ‘Mount’ or ‘automount’ type units manage mountpoints. Socket and device unit types manage sockets (which have associated service unit files using the socket) and devices. Because systemd manages sockets, inetd/xinetd have been obsoleted.

Sysvinit scripts could run user-defined commands. If the init script for myapplication has a section called “bob”, you can run “service myapplication bob” and it will do whatever the ‘bob’ part of the script says to do. Systemd has a fixed list of directives – start, stop, restart, reload, status, enable, disable, is-enabled, list-unit-files, list-dependencies, daemon-reload. You cannot just make a new one.

Systemd may also require a system reboot for more than just kernel patches. This is really different, and I expect there will be a learning curve as to what requires a reboot.

Log files have “vanished”. If you are using a default installation, you won’t find /var/log/messages. You can use “journalctl -f” to tail the equivalent of the messages file. The systemd log files are stored in binary format – potentially corruptible, which is another aspect of the change Unix-types don’t care for.

What does systemd give me?

Systemd doesn’t just start/stop a service when run levels change. A unit can be started because it is configured to start on the runlevel (just like sysvinit scripts), if another service requires it, if the service abends, or if dbus triggers it. “If another service requires it” – that’s a dependency chain. Instead of defining an order and hoping everything you need was loaded by the time the init script ran, systemd allows you to include an “After” directive – units started before the current unit or “Before” – units that will not be started until the current unit starts. Additional directives for “Requires” – units which must be activated to activate the current unit and “Wants” – units that will be started in parallel with the current unit but failing to start these units will not fail the current unit.

A directive, “Conflicts”, allows systemd to identify other units that cannot coexist with the current unit. Conflicting units will be stopped to allow the current unit to start. In addition to the base command starting in the unit file (ExecStart), there are pre (ExecStartPre) and post (ExecStartPost) operations that are run before/after the base command. These could be related to the service itself but do not have to be. You could run a mail command line to alert an admin every time the unit starts or stops cleanly.

Another nice feature of systemd is user-level services – using systemctl –user will control unit files located in user-specific directories like /usr/lib/systemd/user/ and ~/.config/system/user/

Using systemd: (Warning: this is going to get odd)

You use systemctl to control units, and you use journalctl to view the binary blobs that have replaced log files. Use the man pages or your favourite search engine if you want details. The general syntax for systemclt is “systemctl operation unit.type” – e.g. “systemctl restart sendmail” would restart sendmail.

Chkconfig has been completely supplanted. Use “systemctl enable unit.type” and “systemctl disable unit.type” to control if a service auto-starts. Instead of using chkconfig –list, you can query the startup state of an individual unit. Use systemctl –is-enabled unit.type

There’s a service shell script that replaces ‘service’ that you used with sysvinit systems. It turns the old “service something-or-other action” into “systemctl action name.service” so it still works.

Here’s the odd part – it is quite easy to define a permitted sudo operation that allows a non-root user to control sysvinit services. Allow “service sendmail” and the user can run “service sendmail start”, “service sendmail stop”, “service sendmail status”, “service sendmail RandomStuffITossedIntoTheFile”. Because the service name and directive are swapped around in systemctl, we would have to enumerate each individual directive that should be permitted. More secure, because RandomStuffITossedIntoTheFile should not make the cut. But we haven’t done this yet. So until we go through and enumerate the reasonable actions (Are there directives beyond start/stop/status that we should be running? Do we have any business enabling and disabling our services?), submit the access request, confirm it’s all functioning as expected, and remove the “sudo service” access … continue using “sudo service something-or-other action”. We will advise you when the systemctl sudo access has been granted so we can start using the “new way” to control services on RHEL7 systems.

Unlike init scripts, changes to systemd unit files are not immediately activated on the system. Running “systemctl daemon-reload” makes systemd aware of the config change.

Using journalctl:

Our Unix team has implemented rsyslogd to output log data to the expected files. This means you can more or less ignore journalctl – tail/grep the log file as usual. I don’t foresee this changing in the near to mid term, but if you use cloud-hosted sandbox servers (i.e. boxes that don’t have the Unix group’s standard config) … journalctl is what happened to all the log files you cannot find.

To view logs specific to an individual unit, use journalctl -u unit.type. Additionally “systemctl unit.type status” will display the last handful of log lines from the unit.

May 4, 2018

Load Balance and Failover Sendmail Mailertable Relays

A coworker asked me today how to get the mailertable relays to load balance instead of fail over. Trick is to think beyond sendmail. The square brackets around hosts tell sendmail not to check for an MX record (you’re generally using an A record, so this saves a tiny little bit of time … not to mention *if* there is an MX record there, it creates a whole heap-o confusion). *But* the MX lookup is right useful when setting up load balanced or failover relay targets.

Single host relay in the mailertable looks like this:
yourdomain.gTLD relay:[somehost.mydomain.gTLD]

If you want to fail over between relays (that is try #1, if it is unavailable try #2, and so on), you can stay within the mailertable and use:
yourdomain.gTLD relay:[somehost.mydomain.gTLD]:[someotherhost.mydomain.gTLD]

Or even try direct delivery and fail back to a smart host:
yourdomain.gTLD relay:%1:smart-host

But none of this evenly distributes traffic across multiple servers. The trick to load balancing within the mailertable is to create equal weight MX records in your domain to be used as the relay.

In ISC Bind, this looks like:
yourdomainmailrouting.mydomain.gTLD IN MX 10 somehost.mydomain.gTLD.
yourdomainmailrouting.mydomain.gTLD IN MX 10 somehost.mydomain.gTLD.

Once you have created the DNS records, simply use the MX record hostname in your mailertable:

yourdomain.gTLD relay:yourdomainmailrouting.mydomain.gTLD

By leaving out the square brackets, sendmail will resolve an MX record for ‘yourdomainmailrouting.mydomian.gTLD’, find the equal weight MX records, and do the normal sendmail thing to use both.

April 17, 2018

Sendmail In CHROOT Jail

Running our sendmail mail relay in a chroot jail, ‘make’ does not update sendmail config files with changes. While I’m certain there’s a way to sort that, it’s a lot easier to go back to the old-school way of updating sendmail.cf and sendmail’s hash files.

Modifying Sendmail Configuration (sendmail.mc) on Servers with CHROOT Jailed Sendmail

SSH to server using your ID
Change to the sendmail service account (e.g. sudo /bin/su – sendmail)
Change directory to the jailed sendmail /etc/mail locatio (e.g. cd /smt00p20/sendmail/etc/mail)
vi sendmail.mc
Make requisite changes and save file
m4 sendmail.mc > sendmail.cf
Under your ID, restart sendmail using “sudo systemctl stop sendmail stop;sudo systemctl start sendmail”
Validate changes

Modifying Sendmail Data Files on Servers with CHROOT Jailed Sendmail

SSH to server using your ID
Change to the sendmail service account (e.g. sudo /bin/su – sendmail)
Change directory to the jailed sendmail /etc/mail locatio (e.g. cd /smt00p20/sendmail/etc/mail)
vi filetoedit
Make requisite changes and save file
makemap hash ./filetoedit.db < ./filetoedit
Under your ID, restart sendmail using “sudo systemctl stop sendmail stop;sudo systemctl start sendmail”
Validate changes

Where filetoedit is the name of the data file. For example, run “makemap hash ./access.db < ./access” to update the changes to the access file into access.db

April 8, 2018

Running Sendmail In A CHROOT Jail

My employer’s OS-support model restricts root access to members of the Unix support team. Applications are normally installed into a package directory and run under a service ID. While this model works well for most applications, sendmail is tightly integrated into the OS and is not readily built into an application directory. We attempted to run sendmail as a non-root user with modified permissions on application directories such as /var/spool/mqueue – this worked, until OS patches were applied and permissions reset. We needed a way to run sendmail as a non-root user and allow the OS support team to patch servers without impacting the sendmail application.

Chroot is a mechanism that uses a supplied directory path as the environment’s root directory. The jailed process, and its children, should not be able to access any part of the file hierarchy outside of the new root. As a security mechanism, the approach has several flaws – abridged version of the story is that it’s not terribly difficult to break out of jail here; and there are far more effective security approaches (e.g. SELinux). However, chroot jails have their own copies of system owned directories (such as /var/spool/mqueue), binaries, and libraries. Using a chroot jail will allow us to maintain a sendmail application in the package directory that is not impacted by OS updates.

This approach works on relaying mail servers (i.e. those that queue mail to /var/spool/mqueue and send it on its merry way). If sendmail is hosting mailboxes, there are additional challenges to designing a chroot configuration that actually drops messages into mailbox files that users can access.

Preliminaries: To copy/paste, view the single article. Create a service account under which sendmail will run. The installation directory should be owned by the service account user.

Set up the chroot jail location in the installation directory. In this example, that directory is /smt00p20.

mkdir /smt00p20/sendmail
mkdir /smt00p20/sendmail/dev
mkdir /smt00p20/opendkim

We need a null and random in the sendmail jail. On a command line, run:

# Create sendmail jail /dev/null
mknod /smt00p20/sendmail/dev/null c 1 3
# Create sendmail jail /dev/random
mknod /smt00p20/sendmail/dev/random c 1 8

We need an rsyslog socket added under each jail. In /etc/rsyslog.conf, add the following:

# additional log sockets for chroot'ed jail
# Idea from http://www.ispcolohost.com/2014/03/14/how-to-get-syslog-records-of-chrooted-ssh-sftp-server-activity/
$AddUnixListenSocket /smt00p20/sendmail/dev/log
$AddUnixListenSocket /smt00p20/opendkim/dev/log

Additionally, these instructions assume both sendmail and sendmail-cf have been installed on the server. If they have not, you can download the RPMs, unpack them, and copy the files to the appropriate relative jail locations.

Chrooting Sendmail

Logged in with the sendmail ID, ensure you have a .bash_profile that loads .bashrc

-bash-4.2$ cat ~/.bash_profile
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi

Edit ~/.bashrc and add the following, where smt00p20 is the appropriate installation directory, to allow copy/paste

export SENDMAILJAIL=/smt00p20/sendmail
export OPENDKIMJAIL=/smt00p20/opendkim

Log out of the service account and back in (or just source in the .bashrc file). Verify SENDMAILJAIL and OPENDKIMJAIL are set.

Copy a whole heap of ‘stuff’ into the jail – this includes some utilities used to troubleshoot issues within the jail which aren’t strictly needed. I’ve also unpacked the strace RPM to the respective directories within the jail.

mkdir $SENDMAILJAIL/bin
mkdir $SENDMAILJAIL/etc
mkdir $SENDMAILJAIL/etc/alternatives
mkdir $SENDMAILJAIL/etc/mail
mkdir $SENDMAILJAIL/etc/smrsh
mkdir $SENDMAILJAIL/lib64
mkdir $SENDMAILJAIL/lib
mkdir $SENDMAILJAIL/lib/tls
mkdir $SENDMAILJAIL/tmp
mkdir $SENDMAILJAIL/usr
mkdir $SENDMAILJAIL/usr/bin
mkdir $SENDMAILJAIL/usr/sbin
mkdir $SENDMAILJAIL/usr/lib
mkdir $SENDMAILJAIL/usr/lib/sasl2
mkdir $SENDMAILJAIL/var
mkdir $SENDMAILJAIL/var/log
mkdir $SENDMAILJAIL/var/log/mail
mkdir $SENDMAILJAIL/var/run
mkdir $SENDMAILJAIL/var/spool
mkdir $SENDMAILJAIL/var/spool/mqueue
mkdir $SENDMAILJAIL/var/spool/clientmqueue
 
cp /etc/aliases $SENDMAILJAIL/etc/
cp /etc/aliases.db $SENDMAILJAIL/etc/
cp /etc/passwd $SENDMAILJAIL/etc/
cp /etc/group $SENDMAILJAIL/etc/
cp /etc/resolv.conf $SENDMAILJAIL/etc/
cp /etc/host.conf $SENDMAILJAIL/etc/
cp /etc/nsswitch.conf $SENDMAILJAIL/etc/
cp /etc/services $SENDMAILJAIL/etc/
cp /etc/hosts $SENDMAILJAIL/etc/
cp /etc/localtime $SENDMAILJAIL/etc/
 

# If cloning an existing server, scp /etc/mail/* from source to /smt00p20/sendmail/etc/mail

# Verify the sendmail.mc has a RUNAS_USER set to the same service account you are using - the account on our servers is named 'sendmail'. Our old servers are not all set up with a runas user, and failing to have one will cause write failures to the jail /var/spool/mqueue

cp -r /etc/mail/ $SENDMAILJAIL/etc/etc/mail/
cp /usr/sbin/sendmail.sendmail $SENDMAILJAIL/usr/sbin/sendmail.sendmail

cd /smt00p20/sendmail/etc/alternatives
ln -s ../../usr/sbin/sendmail.sendmail ./mta

cd /smt00p20/sendmail/usr/sbin
ln -s ../../etc/alternatives/mta ./sendmail
ln -s ./sendmail ./newaliases
ln -s ./sendmail ./newaliases.sendmail

cd /smt00p20/sendmail/usr/bin
ln -s ../sbin/sendmail ./mailq
ln -s ../sbin/sendmail ./mailq.sendmail
ln -s ../sbin/sendmail.sendmail ./hoststat
ln -s ../sbin/sendmail.sendmail ./purgestat
ln -s ../sbin/makemap ./makemap
ln -s ./rmail.sendmail ./rmail
cp /usr/lib64/libssl.so.10 $SENDMAILJAIL/usr/lib64/libssl.so.10
cp /usr/lib64/libcrypto.so.10 $SENDMAILJAIL/usr/lib64/libcrypto.so.10
cp /usr/lib64/libnsl.so.1 $SENDMAILJAIL/usr/lib64/libnsl.so.1
cp /usr/lib64/libwrap.so.0 $SENDMAILJAIL/usr/lib64/libwrap.so.0
cp /usr/lib64/libhesiod.so.0 $SENDMAILJAIL/usr/lib64/libhesiod.so.0
cp /usr/lib64/libcrypt.so.1 $SENDMAILJAIL/usr/lib64/libcrypt.so.1
cp /usr/lib64/libdb-5.3.so $SENDMAILJAIL/usr/lib64/libdb-5.3.so
cp /usr/lib64/libresolv.so.2 $SENDMAILJAIL/usr/lib64/libresolv.so.2
cp /usr/lib64/libsasl2.so.3 $SENDMAILJAIL/usr/lib64/libsasl2.so.3
cp /usr/lib64/libldap-2.4.so.2 $SENDMAILJAIL/usr/lib64/libldap-2.4.so.2
cp /usr/lib64/liblber-2.4.so.2 $SENDMAILJAIL/usr/lib64/liblber-2.4.so.2
cp /usr/lib64/libc.so.6 $SENDMAILJAIL/usr/lib64/libc.so.6
cp /usr/lib64/libgssapi_krb5.so.2 $SENDMAILJAIL/usr/lib64/libgssapi_krb5.so.2
cp /usr/lib64/libkrb5.so.3 $SENDMAILJAIL/usr/lib64/libkrb5.so.3
cp /usr/lib64/libcom_err.so.2 $SENDMAILJAIL/usr/lib64/libcom_err.so.2
cp /usr/lib64/libk5crypto.so.3 $SENDMAILJAIL/usr/lib64/libk5crypto.so.3
cp /usr/lib64/libdl.so.2 $SENDMAILJAIL/usr/lib64/libdl.so.2
cp /usr/lib64/libz.so.1 $SENDMAILJAIL/usr/lib64/libz.so.1
cp /usr/lib64/libidn.so.11 $SENDMAILJAIL/usr/lib64/libidn.so.11
cp /usr/lib64/libfreebl3.so $SENDMAILJAIL/usr/lib64/libfreebl3.so
cp /usr/lib64/libpthread.so.0 $SENDMAILJAIL/usr/lib64/libpthread.so.0
cp /usr/lib64/libssl3.so $SENDMAILJAIL/usr/lib64/libssl3.so
cp /usr/lib64/libsmime3.so $SENDMAILJAIL/usr/lib64/libsmime3.so
cp /usr/lib64/libnss3.so $SENDMAILJAIL/usr/lib64/libnss3.so
cp /usr/lib64/libnssutil3.so $SENDMAILJAIL/usr/lib64/libnssutil3.so
cp /usr/lib64/libplds4.so $SENDMAILJAIL/usr/lib64/libplds4.so
cp /usr/lib64/libplc4.so $SENDMAILJAIL/usr/lib64/libplc4.so
cp /usr/lib64/libnspr4.so $SENDMAILJAIL/usr/lib64/libnspr4.so
cp /usr/lib64/ld-linux-x86-64.so.2 $SENDMAILJAIL/usr/lib64/ld-linux-x86-64.so.2
cp /usr/lib64/libkrb5support.so.0 $SENDMAILJAIL/usr/lib64/libkrb5support.so.0
cp /usr/lib64/libkeyutils.so.1 $SENDMAILJAIL/usr/lib64/libkeyutils.so.1
cp /usr/lib64/librt.so.1 $SENDMAILJAIL/usr/lib64/librt.so.1
cp /usr/lib64/libselinux.so.1 $SENDMAILJAIL/usr/lib64/libselinux.so.1
cp /usr/lib64/libpcre.so.1 $SENDMAILJAIL/usr/lib64/libpcre.so.1
cp /usr/lib64/libnss_dns.so.2 $SENDMAILJAIL/usr/lib64/libnss_dns.so.2
cp /usr/lib64/libnss_files.so.2 $SENDMAILJAIL/usr/lib64/libnss_files.so.2

cd $SENDMAILJAIL/lib64
cp /lib64/libnss_dns-2.17.so $SENDMAILJAIL/lib64/libnss_dns-2.17.so
ln -s ./libnss_dns-2.17.so ./libnss_dns.so.2

cp /lib64/libresolv-2.17.so $SENDMAILJAIL/lib64/libresolv-2.17.so
ln -s ./lib64/libresolv-2.17.so ./libresolv.so.2

cp /lib64/libnss_files-2.17.so $SENDMAILJAIL/lib64/libnss_files-2.17.so
ln -s ./lib64/libnss_files-2.17.so ./libnss_files.so.2

cd $SENDMAILJAIL/lib 
cp /lib64/libnss_dns-2.17.so $SENDMAILJAIL/lib/libnss_dns-2.17.so
ln -s ./lib/libnss_dns-2.17.so ./libnss_dns.so.2

cp /lib64/libresolv-2.17.so $SENDMAILJAIL/lib/libresolv-2.17.so
ln -s ./lib/libresolv-2.17.so ./libresolv.so.2

cp /lib64/libnss_files-2.17.so $SENDMAILJAIL/lib/libnss_files-2.17.so
ln -s ./lib/libnss_files-2.17.so ./libnss_files.so.2

mkdir $SENDMAILJAIL/usr/lib64/sasl2
cp /usr/lib64/sasl2/* $SENDMAILJAIL/usr/lib64/sasl2/

mkdir $SENDMAILJAIL/lib64/sasl2/
cp /lib64/sasl2/* $SENDMAILJAIL/lib64/sasl2/
cp /etc/sasl2/Sendmail.conf $SENDMAILJAIL/usr/lib64/sasl2/

mkdir $SENDMAILJAIL/etc/sasl2
cp /etc/sasl2/Sendmail.conf $SENDMAILJAIL/etc/sasl2/


cp /usr/sbin/makemap $SENDMAILJAIL/usr/sbin/makemap
ln -s ../sbin/makemap ./makemap
cp /usr/bin/rmail.sendmail $SENDMAILJAIL/usr/bin/rmail.sendmail
ln -s ./rmail.sendmail ./rmail

cp /usr/sbin/mailstats $SENDMAILJAIL/usr/sbin/mailstats
cp /usr/sbin/makemap $SENDMAILJAIL/usr/sbin/makemap
cp /usr/sbin/praliases $SENDMAILJAIL/usr/sbin/praliases
cp /usr/sbin/smrsh $SENDMAILJAIL/usr/sbin/smrsh

cp /lib64/ld-linux-x86-64.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libc.so.6 $SENDMAILJAIL/lib64/
cp /lib64/libcom_err.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libcrypt.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libcrypto.so.10 $SENDMAILJAIL/lib64/
cp /lib64/libdb-5.3.so $SENDMAILJAIL/lib64/
cp /lib64/libdl.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libfreebl3.so $SENDMAILJAIL/lib64/
cp /lib64/libgssapi_krb5.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libhesiod.so.0 $SENDMAILJAIL/lib64/
cp /lib64/libidn.so.11 $SENDMAILJAIL/lib64/
cp /lib64/libk5crypto.so.3 $SENDMAILJAIL/lib64/
cp /lib64/libk5crypto.so.3: $SENDMAILJAIL/lib64/
cp /lib64/libkeyutils.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libkrb5.so.3 $SENDMAILJAIL/lib64/
cp /lib64/libkrb5support.so.0 $SENDMAILJAIL/lib64/
cp /lib64/liblber-2.4.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libldap-2.4.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libnsl.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libnspr4.so $SENDMAILJAIL/lib64/
cp /lib64/libnss3.so $SENDMAILJAIL/lib64/
cp /lib64/libnssutil3.so $SENDMAILJAIL/lib64/
cp /lib64/libpcre.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libplc4.so $SENDMAILJAIL/lib64/
cp /lib64/libplds4.so $SENDMAILJAIL/lib64/
cp /lib64/libpthread.so.0 $SENDMAILJAIL/lib64/
cp /lib64/librt.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libsasl2.so.3 $SENDMAILJAIL/lib64/
cp /lib64/libselinux.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libsmime3.so $SENDMAILJAIL/lib64/
cp /lib64/libssl.so.10 $SENDMAILJAIL/lib64/
cp /lib64/libssl3.so $SENDMAILJAIL/lib64/
cp /lib64/libwrap.so.0 $SENDMAILJAIL/lib64/
cp /lib64/libz.so.1 $SENDMAILJAIL/lib64/
cp /usr/lib64/libk5crypto.so.3 $SENDMAILJAIL/usr/lib64/

cp /lib64/libdns.so.100 $SENDMAILJAIL/lib64/
cp /lib64/liblwres.so.90 $SENDMAILJAIL/lib64/
cp /lib64/libbind9.so.90 $SENDMAILJAIL/lib64/
cp /lib64/libisccfg.so.90 $SENDMAILJAIL/lib64/
cp /lib64/libisccc.so.90 $SENDMAILJAIL/lib64/
cp /lib64/libisc.so.95 $SENDMAILJAIL/lib64/
cp /lib64/libgssapi_krb5.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libkrb5.so.3 $SENDMAILJAIL/lib64/
cp /lib64/libk5crypto.so.3 $SENDMAILJAIL/lib64/
cp /lib64/libcom_err.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libcrypto.so.10 $SENDMAILJAIL/lib64/
cp /lib64/libcap.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libpthread.so.0 $SENDMAILJAIL/lib64/
cp /lib64/libGeoIP.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libxml2.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libz.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libm.so.6 $SENDMAILJAIL/lib64/
cp /lib64/libdl.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libidn.so.11 $SENDMAILJAIL/lib64/
cp /lib64/libc.so.6 $SENDMAILJAIL/lib64/
cp /lib64/libkrb5support.so.0 $SENDMAILJAIL/lib64/
cp /lib64/libkeyutils.so.1 $SENDMAILJAIL/lib64/
cp /lib64/ld-linux-x86-64.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libattr.so.1 $SENDMAILJAIL/lib64/
cp /lib64/liblzma.so.5 $SENDMAILJAIL/lib64/
cp /lib64/libselinux.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libpcre.so.1 $SENDMAILJAIL/lib64/
cp /bin/dig $SENDMAILJAIL/bin/

cp /lib64/libtinfo.so.5 $SENDMAILJAIL/lib64/
cp /lib64/libdl.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libc.so.6 $SENDMAILJAIL/lib64/
cp /lib64/ld-linux-x86-64.so.2 $SENDMAILJAIL/lib64/
cp /bin/bash $SENDMAILJAIL/bin/

cp /bin/ls $SENDMAILJAIL/bin/
cp /lib64/libcap.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libacl.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libc.so.6 $SENDMAILJAIL/lib64/
cp /lib64/libpcre.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libdl.so.2 $SENDMAILJAIL/lib64/
cp /lib64/ld-linux-x86-64.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libattr.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libpthread.so.0 $SENDMAILJAIL/lib64/

cp /bin/vi $SENDMAILJAIL/bin/
cp /usr/sbin/pidof $SENDMAILJAIL/usr/sbin/pidof
cp /lib64/libprocps.so.4 $SENDMAILJAIL/lib64/
cp /lib64/libsystemd.so.0 $SENDMAILJAIL/lib64/
cp /lib64/libdl.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libc.so.6 $SENDMAILJAIL/lib64/
cp /lib64/libcap.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libm.so.6 $SENDMAILJAIL/lib64/
cp /lib64/librt.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libselinux.so.1 $SENDMAILJAIL/lib64/
cp /lib64/liblzma.so.5 $SENDMAILJAIL/lib64/
cp /lib64/libgcrypt.so.11 $SENDMAILJAIL/lib64/
cp /lib64/libgpg-error.so.0 $SENDMAILJAIL/lib64/
cp /lib64/libdw.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libgcc_s.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libpthread.so.0 $SENDMAILJAIL/lib64/
cp /lib64/ld-linux-x86-64.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libattr.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libpcre.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libelf.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libz.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libbz2.so.1 $SENDMAILJAIL/lib64/
cp /bin/rm $SENDMAILJAIL/bin/

Under your ID, ensure the proper permissions are set on the chroot jail

sudo chown -R sendmail:mail /smt00p20/sendmail/
sudo chown sendmail /smt00p20/sendmail/var/spool/mqueue
sudo chmod 0700 /smt00p20/sendmail/var/spool/mqueue
sudo chmod -R go-w /smt00p20/sendmail
sudo chmod 0400 /smt00p20/sendmail/etc/mail/*.cf

Now verify it works – still under your ID as you have sudo permission to run chroot.

sudo /sbin/chroot /smt00p20/sendmail /bin/ls
# You should see a directory listing like this, not an error
bin  dev  etc  lib  lib64  tmp  usr  var

Assuming there are no problems, run sendmail:

sudo /sbin/chroot /smt00p20/sendmail /usr/sbin/sendmail -bd -q5m

Test sending mail through the server to verify proper functionality.

Unit Config: Edit the systemd unit file and add the “RootDirectory” directive

sudo vi /etc/systemd/system/multi-user.target.wants/sendmail.service

[Unit]
Description=Sendmail Mail Transport Agent
After=syslog.target network.target
Conflicts=postfix.service exim.service
Wants=sm-client.service

[Service]
RootDirectory=/smt00p20/sendmail
Type=forking
StartLimitInterval=0
# Known issue – pid causes service hang/timeout that bothers Unix guys
# https://bugzilla.redhat.com/show_bug.cgi?id=1253840
#PIDFile=/run/sendmail.pid
Environment=SENDMAIL_OPTS=-q15m
EnvironmentFile=-/smt00p20/sendmail/etc/sysconfig/sendmail
ExecStart=/usr/sbin/sendmail -bd $SENDMAIL_OPTS $SENDMAIL_OPTARG

[Install]
WantedBy=multi-user.target
Also=sm-client.service

Then run “systemctl daemon-reload” to ingest the changes.

You can now use systemctl to start and stop the sendmail service.

Chrooting opendkim

Create the chroot jail and lib64 directory, create the base directories, then add a few core Linux files so you have a bash shell:

mkdir $OPENDKIMJAIL
mkdir $OPENDKIMJAIL/lib64
mkdir $OPENDKIMJAIL/usr/lib64
mkdir $OPENDKIMJAIL/bin
mkdir $OPENDKIMJAIL/etc

cp /lib64/libtinfo.so.5 $OPENDKIMJAIL/lib64/
cp /lib64/libdl.so.2 $OPENDKIMJAIL/lib64/
cp /lib64/libc.so.6 $OPENDKIMJAIL/lib64/
cp /lib64/ld-linux-x86-64.so.2 $OPENDKIMJAIL/lib64/

cp /bin/bash $OPENDKIMJAIL/bin/
cp /lib64/libstdc++.so.6* $OPENDKIMJAIL/lib64
cp /lib64/libm.so.6 $OPENDKIMJAIL/lib64
cp /lib64/libgcc_s.so.1 $OPENDKIMJAIL/lib64
cp /lib64/libnss_files* $OPENDKIMJAIL/lib64/

Unpack the following RPMs:

rpm2cpio opendkim-2.11.0-0.1.el7.x86_64.rpm | cpio -idmv
rpm2cpio libopendkim-2.11.0-0.1.el7.x86_64.rpm | cpio -idmv
rpm2cpio sendmail-milter-8.14.7-5.el7.x86_64.rpm | cpio -idmv
rpm2cpio opendbx-1.4.6-6.el7.x86_64.rpm | cpio -idmv
rpm2cpio libmemcached-1.0.16-5.el7.x86_64.rpm | cpio -idvm
rpm2cpio libbsd-0.6.0-3.el7.elrepo.x86_64.rpm | cpio -idvm

Then move the unpacked files into the corresponding location in the $OPENDKIMJAIL directory.

Copy host configuration ‘stuff’ from /etc

cp /etc/aliases $OPENDKIMJAIL/etc/
cp /etc/aliases.db $OPENDKIMJAIL/etc/
cp /etc/passwd $OPENDKIMJAIL/etc/
cp /etc/group $OPENDKIMJAIL/etc/
cp /etc/resolv.conf $OPENDKIMJAIL/etc/
cp /etc/host.conf $OPENDKIMJAIL/etc/
cp /etc/nsswitch.conf $OPENDKIMJAIL/etc/
cp /etc/services $OPENDKIMJAIL/etc/
cp /etc/hosts $OPENDKIMJAIL/etc/
cp /etc/localtime $OPENDKIMJAIL/etc/

Copy some more files:

cp /lib64/libcom_err.so.2 $OPENDKIMJAIL/lib64/
cp /lib64/libcrypt.so.1 $OPENDKIMJAIL/lib64/
cp /lib64/libcrypto.so.10 $OPENDKIMJAIL/lib64/
cp /lib64/libdb-5.3.so $OPENDKIMJAIL/lib64/
cp /lib64/libfreebl3.so $OPENDKIMJAIL/lib64/
cp /lib64/libgssapi_krb5.so.2 $OPENDKIMJAIL/lib64/
cp /lib64/libk5crypto.so.3 $OPENDKIMJAIL/lib64/
cp /lib64/libkeyutils.so.1 $OPENDKIMJAIL/lib64/
cp /lib64/libkrb5.so.3 $OPENDKIMJAIL/lib64/
cp /lib64/libkrb5support.so.0 $OPENDKIMJAIL/lib64/
cp /lib64/liblber-2.4.so.2 $OPENDKIMJAIL/lib64/
cp /lib64/libldap-2.4.so.2 $OPENDKIMJAIL/lib64/
cp /lib64/libnspr4.so $OPENDKIMJAIL/lib64/
cp /lib64/libnss3.so $OPENDKIMJAIL/lib64/
cp /lib64/libnssutil3.so $OPENDKIMJAIL/lib64/
cp /lib64/libpcre.so.1 $OPENDKIMJAIL/lib64/
cp /lib64/libplc4.so $OPENDKIMJAIL/lib64/
cp /lib64/libplds4.so $OPENDKIMJAIL/lib64/
cp /lib64/libpthread.so.0 $OPENDKIMJAIL/lib64/
cp /lib64/libresolv.so.2 $OPENDKIMJAIL/lib64/
cp /lib64/librt.so.1 $OPENDKIMJAIL/lib64/
cp /lib64/libsasl2.so.3 $OPENDKIMJAIL/lib64/
cp /lib64/libselinux.so.1 $OPENDKIMJAIL/lib64/
cp /lib64/libsmime3.so $OPENDKIMJAIL/lib64/
cp /lib64/libssl.so.10 $OPENDKIMJAIL/lib64/
cp /lib64/libssl3.so $OPENDKIMJAIL/lib64/
cp /lib64/libz.so.1 $OPENDKIMJAIL/lib64/
cp /usr/lib64/libssl.so.10 $OPENDKIMJAIL/usr/lib64/

cp $OPENDKIMJAIL/usr/lib64/libmilter.so.1.0 $OPENDKIMJAIL/usr/lib/
cp $OPENDKIMJAIL/usr/lib64/libmilter.so.1.0.1 $OPENDKIMJAIL/usr/lib/

cp $OPENDKIMJAIL/usr/lib64/libmilter.so.1.0 $OPENDKIMJAIL/lib64/
cp $OPENDKIMJAIL/usr/lib64/libmilter.so.1.0.1 $OPENDKIMJAIL/lib64/

cp $OPENDKIMJAIL/usr/lib64/libmilter.so.1.0 $OPENDKIMJAIL/usr/lib/
cp $OPENDKIMJAIL/usr/lib64/libmilter.so.1.0.1 $OPENDKIMJAIL/usr/lib/

cp $OPENDKIMJAIL/usr/lib64/libmilter.so.1.0 $OPENDKIMJAIL/lib64/
cp $OPENDKIMJAIL/usr/lib64/libmilter.so.1.0.1 $OPENDKIMJAIL/lib64/

Configure OpenDKIM ($DKIMJAIL/etc/opendkim.conf) and populate keys (copy from server being replaced or generate new keys). Then, under your ID, run:

sudo /sbin/chroot /smt00p20/opendkim /usr/sbin/opendkim -u sendmail -v

The systemd unit file, /usr/lib/systemd/system/opendkim.service, needs to contain:

# If you are using OpenDKIM with SQL datasets it might be necessary to start OpenDKIM after the database servers.
# For example, if using both MariaDB and PostgreSQL, change "After=" in the "[Unit]" section to:
# After=network.target nss-lookup.target syslog.target mariadb.service postgresql.service

[Unit]
Description=DomainKeys Identified Mail (DKIM) Milter
Documentation=man:opendkim(8) man:opendkim.conf(5) man:opendkim-genkey(8) man:opendkim-genzone(8) man:opendkim-testadsp(8) man:opendkim-testkey http://www.opendkim.org/docs.html
After=network.target nss-lookup.target syslog.target

[Service]
RootDirectory=/smt00p20/opendkim
Type=forking
PIDFile=/smt00p20/opendkim/var/run/opendkim/opendkim.pid
EnvironmentFile=-/etc/sysconfig/opendkim
ExecStart=/usr/sbin/opendkim -u sendmail -v $OPTIONS
ExecReload=/bin/kill -USR1 $MAINPID
User=sendmail
Group=mail

[Install]
WantedBy=multi-user.target

Upgrading Sendmail – After Unix Applies Patches

This process grabs a new copy of sendmail, associated diagnostic utilities, and their dependencies from the OS installation. If you want to apply patches prior to Unix support doing so, you can stage a sendmail build (everything up to ‘make install’) and copy the files out or, if an updated RPM is in the repo but just not installed, download the RPMs, unpack them, and copy the files in. I would do that in addition to (and after) this process to ensure library updates are reflected in our jailed sendmail installation (i.e. if there’s an update to the crypto libraries, we get those updates).

cp /usr/sbin/sendmail.sendmail $SENDMAILJAIL/usr/sbin/sendmail.sendmail
cp /usr/lib64/libssl.so.10 $SENDMAILJAIL/usr/lib64/libssl.so.10
cp /usr/lib64/libcrypto.so.10 $SENDMAILJAIL/usr/lib64/libcrypto.so.10
cp /usr/lib64/libnsl.so.1 $SENDMAILJAIL/usr/lib64/libnsl.so.1
cp /usr/lib64/libwrap.so.0 $SENDMAILJAIL/usr/lib64/libwrap.so.0
cp /usr/lib64/libhesiod.so.0 $SENDMAILJAIL/usr/lib64/libhesiod.so.0
cp /usr/lib64/libcrypt.so.1 $SENDMAILJAIL/usr/lib64/libcrypt.so.1
cp /usr/lib64/libdb-5.3.so $SENDMAILJAIL/usr/lib64/libdb-5.3.so
cp /usr/lib64/libresolv.so.2 $SENDMAILJAIL/usr/lib64/libresolv.so.2
cp /usr/lib64/libsasl2.so.3 $SENDMAILJAIL/usr/lib64/libsasl2.so.3
cp /usr/lib64/libldap-2.4.so.2 $SENDMAILJAIL/usr/lib64/libldap-2.4.so.2
cp /usr/lib64/liblber-2.4.so.2 $SENDMAILJAIL/usr/lib64/liblber-2.4.so.2
cp /usr/lib64/libc.so.6 $SENDMAILJAIL/usr/lib64/libc.so.6
cp /usr/lib64/libgssapi_krb5.so.2 $SENDMAILJAIL/usr/lib64/libgssapi_krb5.so.2
cp /usr/lib64/libkrb5.so.3 $SENDMAILJAIL/usr/lib64/libkrb5.so.3
cp /usr/lib64/libcom_err.so.2 $SENDMAILJAIL/usr/lib64/libcom_err.so.2
cp /usr/lib64/libk5crypto.so.3 $SENDMAILJAIL/usr/lib64/libk5crypto.so.3
cp /usr/lib64/libdl.so.2 $SENDMAILJAIL/usr/lib64/libdl.so.2
cp /usr/lib64/libz.so.1 $SENDMAILJAIL/usr/lib64/libz.so.1
cp /usr/lib64/libidn.so.11 $SENDMAILJAIL/usr/lib64/libidn.so.11
cp /usr/lib64/libfreebl3.so $SENDMAILJAIL/usr/lib64/libfreebl3.so
cp /usr/lib64/libpthread.so.0 $SENDMAILJAIL/usr/lib64/libpthread.so.0
cp /usr/lib64/libssl3.so $SENDMAILJAIL/usr/lib64/libssl3.so
cp /usr/lib64/libsmime3.so $SENDMAILJAIL/usr/lib64/libsmime3.so
cp /usr/lib64/libnss3.so $SENDMAILJAIL/usr/lib64/libnss3.so
cp /usr/lib64/libnssutil3.so $SENDMAILJAIL/usr/lib64/libnssutil3.so
cp /usr/lib64/libplds4.so $SENDMAILJAIL/usr/lib64/libplds4.so
cp /usr/lib64/libplc4.so $SENDMAILJAIL/usr/lib64/libplc4.so
cp /usr/lib64/libnspr4.so $SENDMAILJAIL/usr/lib64/libnspr4.so
cp /usr/lib64/ld-linux-x86-64.so.2 $SENDMAILJAIL/usr/lib64/ld-linux-x86-64.so.2
cp /usr/lib64/libkrb5support.so.0 $SENDMAILJAIL/usr/lib64/libkrb5support.so.0
cp /usr/lib64/libkeyutils.so.1 $SENDMAILJAIL/usr/lib64/libkeyutils.so.1
cp /usr/lib64/librt.so.1 $SENDMAILJAIL/usr/lib64/librt.so.1
cp /usr/lib64/libselinux.so.1 $SENDMAILJAIL/usr/lib64/libselinux.so.1
cp /usr/lib64/libpcre.so.1 $SENDMAILJAIL/usr/lib64/libpcre.so.1
cp /usr/lib64/libnss_dns.so.2 $SENDMAILJAIL/usr/lib64/libnss_dns.so.2
cp /usr/lib64/libnss_files.so.2 $SENDMAILJAIL/usr/lib64/libnss_files.so.2
cp /lib64/libnss_dns-2.17.so $SENDMAILJAIL/lib64/libnss_dns-2.17.so
cp /lib64/libresolv-2.17.so $SENDMAILJAIL/lib64/libresolv-2.17.so
cp /lib64/libnss_files-2.17.so $SENDMAILJAIL/lib64/libnss_files-2.17.so
cp /lib64/libnss_dns-2.17.so $SENDMAILJAIL/lib/libnss_dns-2.17.so
cp /lib64/libresolv-2.17.so $SENDMAILJAIL/lib/libresolv-2.17.so
cp /lib64/libnss_files-2.17.so $SENDMAILJAIL/lib/libnss_files-2.17.so
cp /usr/lib64/sasl2/* $SENDMAILJAIL/usr/lib64/sasl2/
cp /lib64/sasl2/* $SENDMAILJAIL/lib64/sasl2/
cp /etc/sasl2/Sendmail.conf $SENDMAILJAIL/usr/lib64/sasl2/
cp /etc/sasl2/Sendmail.conf $SENDMAILJAIL/etc/sasl2/
cp /usr/sbin/makemap $SENDMAILJAIL/usr/sbin/makemap
cp /usr/bin/rmail.sendmail $SENDMAILJAIL/usr/bin/rmail.sendmail
cp /usr/sbin/mailstats $SENDMAILJAIL/usr/sbin/mailstats
cp /usr/sbin/makemap $SENDMAILJAIL/usr/sbin/makemap
cp /usr/sbin/praliases $SENDMAILJAIL/usr/sbin/praliases
cp /usr/sbin/smrsh $SENDMAILJAIL/usr/sbin/smrsh

cp /lib64/ld-linux-x86-64.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libc.so.6 $SENDMAILJAIL/lib64/
cp /lib64/libcom_err.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libcrypt.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libcrypto.so.10 $SENDMAILJAIL/lib64/
cp /lib64/libdb-5.3.so $SENDMAILJAIL/lib64/
cp /lib64/libdl.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libfreebl3.so $SENDMAILJAIL/lib64/
cp /lib64/libgssapi_krb5.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libhesiod.so.0 $SENDMAILJAIL/lib64/
cp /lib64/libidn.so.11 $SENDMAILJAIL/lib64/
cp /lib64/libk5crypto.so.3 $SENDMAILJAIL/lib64/
cp /lib64/libk5crypto.so.3: $SENDMAILJAIL/lib64/
cp /lib64/libkeyutils.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libkrb5.so.3 $SENDMAILJAIL/lib64/
cp /lib64/libkrb5support.so.0 $SENDMAILJAIL/lib64/
cp /lib64/liblber-2.4.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libldap-2.4.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libnsl.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libnspr4.so $SENDMAILJAIL/lib64/
cp /lib64/libnss3.so $SENDMAILJAIL/lib64/
cp /lib64/libnssutil3.so $SENDMAILJAIL/lib64/
cp /lib64/libpcre.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libplc4.so $SENDMAILJAIL/lib64/
cp /lib64/libplds4.so $SENDMAILJAIL/lib64/
cp /lib64/libpthread.so.0 $SENDMAILJAIL/lib64/
cp /lib64/librt.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libsasl2.so.3 $SENDMAILJAIL/lib64/
cp /lib64/libselinux.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libsmime3.so $SENDMAILJAIL/lib64/
cp /lib64/libssl.so.10 $SENDMAILJAIL/lib64/
cp /lib64/libssl3.so $SENDMAILJAIL/lib64/
cp /lib64/libwrap.so.0 $SENDMAILJAIL/lib64/
cp /lib64/libz.so.1 $SENDMAILJAIL/lib64/
cp /usr/lib64/libk5crypto.so.3 $SENDMAILJAIL/usr/lib64/

cp /lib64/libdns.so.100 $SENDMAILJAIL/lib64/
cp /lib64/liblwres.so.90 $SENDMAILJAIL/lib64/
cp /lib64/libbind9.so.90 $SENDMAILJAIL/lib64/
cp /lib64/libisccfg.so.90 $SENDMAILJAIL/lib64/
cp /lib64/libisccc.so.90 $SENDMAILJAIL/lib64/
cp /lib64/libisc.so.95 $SENDMAILJAIL/lib64/
cp /lib64/libgssapi_krb5.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libkrb5.so.3 $SENDMAILJAIL/lib64/
cp /lib64/libk5crypto.so.3 $SENDMAILJAIL/lib64/
cp /lib64/libcom_err.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libcrypto.so.10 $SENDMAILJAIL/lib64/
cp /lib64/libcap.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libpthread.so.0 $SENDMAILJAIL/lib64/
cp /lib64/libGeoIP.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libxml2.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libz.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libm.so.6 $SENDMAILJAIL/lib64/
cp /lib64/libdl.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libidn.so.11 $SENDMAILJAIL/lib64/
cp /lib64/libc.so.6 $SENDMAILJAIL/lib64/
cp /lib64/libkrb5support.so.0 $SENDMAILJAIL/lib64/
cp /lib64/libkeyutils.so.1 $SENDMAILJAIL/lib64/
cp /lib64/ld-linux-x86-64.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libattr.so.1 $SENDMAILJAIL/lib64/
cp /lib64/liblzma.so.5 $SENDMAILJAIL/lib64/
cp /lib64/libselinux.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libpcre.so.1 $SENDMAILJAIL/lib64/
cp /bin/dig $SENDMAILJAIL/bin/

cp /lib64/libtinfo.so.5 $SENDMAILJAIL/lib64/
cp /lib64/libdl.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libc.so.6 $SENDMAILJAIL/lib64/
cp /lib64/ld-linux-x86-64.so.2 $SENDMAILJAIL/lib64/
cp /bin/bash $SENDMAILJAIL/bin/

cp /bin/ls $SENDMAILJAIL/bin/
cp /lib64/libcap.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libacl.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libc.so.6 $SENDMAILJAIL/lib64/
cp /lib64/libpcre.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libdl.so.2 $SENDMAILJAIL/lib64/
cp /lib64/ld-linux-x86-64.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libattr.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libpthread.so.0 $SENDMAILJAIL/lib64/

cp /bin/vi $SENDMAILJAIL/bin/
cp /usr/sbin/pidof $SENDMAILJAIL/usr/sbin/pidof
cp /lib64/libprocps.so.4 $SENDMAILJAIL/lib64/
cp /lib64/libsystemd.so.0 $SENDMAILJAIL/lib64/
cp /lib64/libdl.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libc.so.6 $SENDMAILJAIL/lib64/
cp /lib64/libcap.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libm.so.6 $SENDMAILJAIL/lib64/
cp /lib64/librt.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libselinux.so.1 $SENDMAILJAIL/lib64/
cp /lib64/liblzma.so.5 $SENDMAILJAIL/lib64/
cp /lib64/libgcrypt.so.11 $SENDMAILJAIL/lib64/
cp /lib64/libgpg-error.so.0 $SENDMAILJAIL/lib64/
cp /lib64/libdw.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libgcc_s.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libpthread.so.0 $SENDMAILJAIL/lib64/
cp /lib64/ld-linux-x86-64.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libattr.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libpcre.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libelf.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libz.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libbz2.so.1 $SENDMAILJAIL/lib64/

cp /bin/rm $SENDMAILJAIL/bin/

Under your ID, ensure the proper permissions are set on the chroot jail

sudo chown -R sendmail:mail /smt00p20/sendmail/
sudo chown sendmail /smt00p20/sendmail/var/spool/mqueue
sudo chmod 0700 /smt00p20/sendmail/var/spool/mqueue
sudo chmod -R go-w /smt00p20/sendmail
sudo chmod 0400 /smt00p20/sendmail/etc/mail/*.cf

Then start sendmail and verify functionality.

Updating OpenDKIM

cp /lib64/libtinfo.so.5 $OPENDKIMJAIL/lib64/
cp /lib64/libdl.so.2 $OPENDKIMJAIL/lib64/
cp /lib64/libc.so.6 $OPENDKIMJAIL/lib64/
cp /lib64/ld-linux-x86-64.so.2 $OPENDKIMJAIL/lib64/
cp /bin/bash $OPENDKIMJAIL/bin/
cp /lib64/libstdc++.so.6* $OPENDKIMJAIL/lib64
cp /lib64/libm.so.6 $OPENDKIMJAIL/lib64
cp /lib64/libgcc_s.so.1 $OPENDKIMJAIL/lib64
cp /lib64/libnss_files* $OPENDKIMJAIL/lib64/

If there is an update to the opendkim packages, unpack the updated RPM files and move the new files into the corresponding jail locations.

rpm2cpio opendkim-2.11.0-0.1.el7.x86_64.rpm | cpio -idmv
rpm2cpio libopendkim-2.11.0-0.1.el7.x86_64.rpm | cpio -idmv
rpm2cpio sendmail-milter-8.14.7-5.el7.x86_64.rpm | cpio -idmv
rpm2cpio opendbx-1.4.6-6.el7.x86_64.rpm | cpio -idmv
rpm2cpio libmemcached-1.0.16-5.el7.x86_64.rpm | cpio -idvm
rpm2cpio libbsd-0.6.0-3.el7.elrepo.x86_64.rpm | cpio -idvm

March 30, 2018

Creating An OpenHAB 2.3.0 Snapshot Docker Container

We found quick instructions for creating a Docker container for the OpenHAB 2.3.0 snapshot. These instructions evidently presuppose some basic knowledge of building Docker containers, so I thought I’d write the “I don’t know what I am doing” version of the instructions. Beyond the obvious download & install Docker, then make sure it’s functional (service starts).

The linked Dockerfile is not the only thing you need. Go up a level — you need both the Dockerfile and entrypoint.sh files. Create a directory somewhere and grab these two files. Then build the container using

docker build -t oh2imagename .

I used a short, alpha-numeric only name for my image. When I used slashes as in the example, the container would not start. Then make the folders you want to map into OpenHAB2:

mkdir /some/path/to/openhab/addons
mkdir /some/path/to/openhab/conf
mkdir /some/path/to/openhab/userdata

The instructions conflate local users/groups with in-container users/groups. You do not need to create a local user. You do need to indicate the uidNumber and gidNumber for the openhab user and group. Even if you do create the local user and group, then change the /some/path/to/openhab permissions to provide full access to the user … you may well not be able to access the files. That is SELinux, not a file permission issue. The quick/dirty solution is to start the container with the privileged flag:

--privileged=true

Alternately, consult the Universal Archive of All IT Knowledge and figure out how to allow the docker service to write files where you want them. And how to access USB devices if you are trying to use something like a ZWave dongle. We went with the privileged route 🙂 The –name option is just the container name. The –net uses the host network for container communications instead of the bridge network. Saves mapping ports, although you could easily use the bridge network and map out the handful of OpenHab specific ports. The -d runs the container in detached mode. The -e sets some environment flags (used by the user/group creation script that runs upon container startup). The –tty (or -t) attaches a console. Not really used here.

docker run --privileged --name oh2containername --net=host --tty -d -e USER_ID=5555 \
 -e GROUP_ID=5555 oh2imagename

Ideally, your OpenHAB2 instance will be running. Use “docker ps” to list out the running containers. If you don’t see a container with the name supplied above … then something went wrong. You can use “docker history oh2containername” to view a quick history, but “docker logs oh2containername” will probably provide more useful information. We encountered file permission issues (as noted above, due to SELinux) which prevented the initial container setup from running. Once that was sorted, the container showed up in the running container list.

You’re ready to use it — you can access the web console using your computer’s IP address (assuming you set this container up in the host network and not the bridge — if you used the bridge, you can use “docker inspect oh2containername” and look for IPAddress under NetworkSettings) on the default port. You can ssh into the Karaf console with the default user/password on the default port. Or you can shell into the container.

docker exec -it oh2containername /bin/bash

This is a bash shell running on the OH2 container — you’ll find a lot of ‘stuff’ hasn’t been installed, and your normal command aliases won’t be present. But it’s a shell on the server and can be used to start/stop OH2.

March 23, 2018

Enable OUD Changelog Without Replication Partner

Since the Sun Directory Server Enterprise Edition went the way of, well, Sun, we’ve migrated to the Oracle Unified Directory 11g platform for the company’s pure LDAP directory. There is an Oracle Identity Management application that reads the LDAP changelog to ingest user lockout events. In production, our servers replicate with a couple of partners to provide capacity and high availability. In development / sandbox, environments … not so much. But the IDM platform still needs to read the changelog.

Oracle’s documentation tells me to enable replication … which, great, I’ve got to bring up a second, off-port, directory instance and monitor for replication failures just to get a changelog. The site does say “By using this method, you can conceivably set up replication on a standalone server, which will enable you to have access to an ECL on a standalone server.” … conceivably, but it would be nice if they’d mention how. Since all of their documentation for using the dsreplication binary includes a partner server and valid credentials over yonder … that’s a bit of a bust.

But I’ve finally worked out a technique for enabling replication just enough to get the changelog created without having to provide valid credentials on a foreign host with which replication will be established.

./dsconfig -h localhost -p 4444 -D "cn=directory manager" -j ~/pwd.txt -X -n create-replication-server --provider-name 'Multimaster Synchronization' --set replication-port:8989 --set replication-server-id:1 --type generic
 
 
./dsconfig -h localhost -p 4444 -D "cn=directory manager" -j ~/pwd.txt -X -n create-replication-domain --provider-name 'Multimaster Synchronization' --set base-dn:o=windstream --set replication-server:localhost:8989 --set server-id:1 --type generic --domain-name o=windstream

Volia, make changes and we’ve got stuff under cn=changelog.

March 15, 2018

Fedora 26 => 27 & PHP

Since I like to discover major changes by upgrading my server and then realizing something doesn’t work (well “like” might be too strong a word … but I certainly do it) … I randomly upgraded to Fedora 27 without reading any documentation on changes. Aaaand we have PHP! Evidently mod_php has gone away and I’m going to have to figure out how to use FastCGI (php-fpm). Luckily there’s a quick way to switch back to mod_php in the interim:

/etc/httpd/conf.modules.d/00-mpm.conf

# Uncomment this line to use mod_php
LoadModule mpm_prefork_module modules/mod_mpm_prefork.so

# worker MPM: Multi-Processing Module implementing a hybrid
# multi-threaded multi-process web server
# See: http://httpd.apache.org/docs/2.4/mod/worker.html
#
#LoadModule mpm_worker_module modules/mod_mpm_worker.so

# event MPM: A variant of the worker MPM with the goal of consuming
# threads only for connections with active processing
# See: http://httpd.apache.org/docs/2.4/mod/event.html
#
# Commend out this line to use mod_php
#LoadModule mpm_event_module modules/mod_mpm_event.so

/etc/httpd/conf.modules.d/15-php.conf

<IfModule !mod_php5.c>
<IfModule prefork.c>
LoadModule php7_module modules/libphp7.so
</IfModule>
</IfModule>

<IfModule !mod_php5.c>
<IfModule !prefork.c>
# ZTS module is not supported, so FPM is preferred
LoadModule php7_module modules/libphp7-zts.so
</IfModule>
</IfModule>