Tag: httpd

Apache HTTPD Log File Analysis — Hits by IP Address

When we are decommissioning a website (or web server), I always watch the log files to ensure there aren’t a lot of people still accessing it. Sometimes there are and it’s worth tracking them down individually to clue them into the site’s eminent demise. Usually there aren’t, and it’s just a confirmation that our decommissioning efforts won’t be impactful.

This python script looks for IP addresses in the log files and outputs each IP & it’s access count per log file. Not great if you’ll see a bunch of IP addresses in the recorded URI string, but it’s good enough for 99% of our log data. 

import os
import re
from collections import Counter

def parseApacheHTTPDLog(strLogFile):
    regexIPAddress = r'\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}'

    with open(strLogFile) as f:
        objLog = f.read()
        listIPAddresses = re.findall(regexIPAddress,objLog)
        counterAccessByIP = Counter(listIPAddresses)
        for strIP, iAccessCount in counterAccessByIP.items():
            print(f"{strLogFile}\t{str(strIP)}\t{str(iAccessCount)}")

if __name__ == '__main__':
    strLogDirectory = '/var/log/httpd/'
    for strFileName in os.listdir(strLogDirectory):
        if strFileName.__contains__("access_log"):
        #if strFileName.__contains__("hostname.example.com") and strFileName.__contains__("access_log"):
            parseApacheHTTPDLog(f"{strLogDirectory}{strFileName}")

Apache HTTPD and DER Encoded Certificate

We are in the process of updating one of the web servers at work to a newer OS – along with a newer Apache HTTPD and PHP iteration. Ran into a snag just setting up the SSL web site – we couldn’t get HTTPD started with our Venafi certificate.

[Fri Jan 28 14:35:05.092086 2022] [ssl:emerg] [pid 57739:tid 139948816931136] AH02561: Failed to configure certificate hostname.example.com:443:0, check /path/to/certs/production/server.crt

[Fri Jan 28 14:35:05.092103 2022] [ssl:emerg] [pid 57739:tid 139948816931136] SSL Library Error: error:0909006C:PEM routines:get_name:no start line (Expecting: CERTIFICATE) — Bad file contents or format – or even just a forgotten SSLCertificateKeyFile?

[Fri Jan 28 14:35:05.092115 2022] [ssl:emerg] [pid 57739:tid 139948816931136] SSL Library Error: error:140AD009:SSL routines:SSL_CTX_use_certificate_file:PEM lib

The certificate was DER encoded – that’s not what I use, but it was working on the old server.

I think there might be something between httpd-2.4.6-97 and httpd-2.4.37-43 that stopped DER encoded certificates from working. Rather than figure out some way to coerce HTTPD to use this DER file that I don’t really care if I’ve got … I just used a quick command to export the B64 version of the certificate, copied the header/footer/stuff in between, and made a base-64 encoded certificate file.

openssl x509 -inform DER -in server.crt | openssl x509 -text

And, voila, we’ve got a web server.

 

PrivateTmp Strangeness — apachectl v/s systemctl

This is a very strange problem — we had a web server upgraded recently. We use “sudo apachectl start” to bring up the server since the server is maintained by a dedicated Unix support team, and the site worked fine. Until … Sunday morning after the log rotation. Then Box Spout was unable to access the XML data for an Excel file to compress it. Lots of errors:

 

[Mon Sep 14 10:59:39.137728 2020] [:error] [pid 57117] [client 10.1.2.3:49276] PHP Warning: ZipArchive::close(): Zlib error: stream error in /path/to/site/classes/vendor/box/spout/src/Spout/Writer/Common/Helper/ZipHelper.php on line 199, referer: https://hostname.example.com/path/to/code.php
[Mon Sep 14 10:59:39.137785 2020] [:error] [pid 57117] [client 10.1.2.3:49276] PHP Warning: fopen(/tmp/xlsx5f5f934699dcd9.17695276.zip): failed to open stream: No such file or directory in /path/to/site/classes/vendor/box/spout/src/Spout/Writer/Common/Helper/ZipHelper.php on line 213, referer: https://hostname.example.com/path/to/code.php
[Mon Sep 14 10:59:39.137803 2020] [:error] [pid 57117] [client 10.1.2.3:49276] PHP Warning: stream_copy_to_stream() expects parameter 1 to be resource, boolean given in /path/to/site/classes/vendor/box/spout/src/Spout/Writer/Common/Helper/ZipHelper.php on line 214, referer: https://hostname.example.com/path/to/code.php
[Mon Sep 14 10:59:39.137814 2020] [:error] [pid 57117] [client 10.1.2.3:49276] PHP Warning: fclose() expects parameter 1 to be resource, boolean given in /path/to/site/classes/vendor/box/spout/src/Spout/Writer/Common/Helper/ZipHelper.php on line 215, referer: https://hostname.example.com/path/to/code.php
[Mon Sep 14 10:59:39.138360 2020] [:error] [pid 57117] [client 10.1.2.3:49276] PHP Fatal error: Uncaught exception ‘Box\\Spout\\Common\\Exception\\IOException’ with message ‘Cannot perform I/O operation outside of the base folder: /tmp’ in /path/to/site/classes/vendor/box/spout/src/Spout/Common/Helper/FileSystemHelper.php:130\nStack trace:\n#0 /path/to/site/classes/vendor/box/spout/src/Spout/Common/Helper/FileSystemHelper.php(82): Box\\Spout\\Common\\Helper\\FileSystemHelper->throwIfOperationNotInBaseFolder(‘/tmp/xlsx5f5f93…’)\n#1 /path/to/site/classes/vendor/box/spout/src/Spout/Writer/XLSX/Helper/FileSystemHelper.php(369): Box\\Spout\\Common\\Helper\\FileSystemHelper->deleteFile(‘/tmp/xlsx5f5f93…’)\n#2 /path/to/site/classes/vendor/box/spout/src/Spout/Writer/XLSX/Internal/Workbook.php(134): Box\\Spout\\Writer\\XLSX\\Helper\\FileSystemHelper->zipRootFolderAndCopyToStream(Resource id #26)\n#3 /path/to/site/ in /path/to/site/classes/vendor/box/spout/src/Spout/Common/Helper/FileSystemHelper.php on line 130, referer: https://hostname.example.com/path/to/code.php
[Mon Sep 14 10:59:39.139468 2020] [:error] [pid 57117] [client 10.1.2.3:49276] PHP Warning: ZipArchive::close(): Zlib error: stream error in /path/to/site/classes/vendor/box/spout/src/Spout/Writer/Common/Helper/ZipHelper.php on line 199, referer: https://hostname.example.com/path/to/code.php
[Mon Sep 14 10:59:39.139504 2020] [:error] [pid 57117] [client 10.1.2.3:49276] PHP Warning: fopen(/tmp/xlsx5f5f934699dcd9.17695276.zip): failed to open stream: No such file or directory in /path/to/site/classes/vendor/box/spout/src/Spout/Writer/Common/Helper/ZipHelper.php on line 213, referer: https://hostname.example.com/path/to/code.php
[Mon Sep 14 10:59:39.139515 2020] [:error] [pid 57117] [client 10.1.2.3:49276] PHP Warning: stream_copy_to_stream() expects parameter 1 to be resource, boolean given in /path/to/site/classes/vendor/box/spout/src/Spout/Writer/Common/Helper/ZipHelper.php on line 214, referer: https://hostname.example.com/path/to/code.php
[Mon Sep 14 10:59:39.139533 2020] [:error] [pid 57117] [client 10.1.2.3:49276] PHP Warning: fclose() expects parameter 1 to be resource, boolean given in /path/to/site/classes/vendor/box/spout/src/Spout/Writer/Common/Helper/ZipHelper.php on line 215, referer: https://hostname.example.com/path/to/code.php
[Mon Sep 14 10:59:39.139599 2020] [:error] [pid 57117] [client 10.1.2.3:49276] PHP Fatal error: Uncaught exception ‘Box\\Spout\\Common\\Exception\\IOException’ with message ‘Cannot perform I/O operation outside of the base folder: /tmp’ in /path/to/site/classes/vendor/box/spout/src/Spout/Common/Helper/FileSystemHelper.php:130\nStack trace:\n#0 /path/to/site/classes/vendor/box/spout/src/Spout/Common/Helper/FileSystemHelper.php(82): Box\\Spout\\Common\\Helper\\FileSystemHelper->throwIfOperationNotInBaseFolder(‘/tmp/xlsx5f5f93…’)\n#1 /path/to/site/classes/vendor/box/spout/src/Spout/Writer/XLSX/Helper/FileSystemHelper.php(369): Box\\Spout\\Common\\Helper\\FileSystemHelper->deleteFile(‘/tmp/xlsx5f5f93…’)\n#2 /path/to/site/classes/vendor/box/spout/src/Spout/Writer/XLSX/Internal/Workbook.php(134): Box\\Spout\\Writer\\XLSX\\Helper\\FileSystemHelper->zipRootFolderAndCopyToStream(Resource id #26)\n#3 /path/to/site in /path/to/site/classes/vendor/box/spout/src/Spout/Common/Helper/FileSystemHelper.php on line 130, referer: https://hostname.example.com/path/to/code.php

 

The postupdate script is “systemctl reload httpd.service” — so not exactly the same thing we used to launch the service originally. But I’ve never seen differing behavior between apachectl and systemctl started HTTPD instances. Quick/dirty solution is to disable PrivateTmp, but I’m hoping to be able to isolate why exactly the postupdate script appears to break the service’s access to the private tmp space.

30 November 2020 addendum — In discussing the issue with RedHat, they suggested using either

/sbin/killall -HUP httpd

or

/bin/systemctl restart httpd.service > /dev/null 2>/dev/null || true

Doing this has allowed continual access to the Private Tmp space after log rotation. Woohoo! Not sure why the default configuration that came from the Apache httpd package didn’t work (i.e. it’s not like we built some funky weird log rotation script). But success is good enough for me.

Listing Modules In Dynamically Linked Shared Object Libraries

We had to rebuild a server over the weekend — it’s a lot harder to get Apache and PHP set up when you don’t have root access to just install things from the yum repository. And, unlike the servers where I built httpd and php from source … we basically relayed requests to the Unix admin to have packages installed. One of the confusions during the whole process was that we didn’t know what to use as the module name for PHP to load in the httpd.conf file. The line from our old server (LoadModule php5_module /etc/httpd/modules/libphp5.so) produced an error that there was no such thing to load.

When a library fails to load with some error, I know to use ldd … but I didn’t know there was a way to list out the modules in a library. Fortunately, one of my coworkers had already run nm and listed out the modules — nm -D –defined-only sharedLibraryFile | grep module — and we were able to identify that the libphp5.so that we had wasn’t anything like the one on the old server. By listing the modules for each of the shared object libraries installed by the php package, we got the proper module name for httpd.conf

Testing A New Web Server Without DNS Changes

When migrating to a new server, it’s good to validate site functionality before redirecting users to the new host. i.e. I have anya.rushworth.us set up in the httpd config on both server1 and server2. DNS currently points traffic to server1, but I need to test the site on server2.

Approach #1 – With administrative access to the host

Edit your hosts file – open an administrative command prompt

Edit %SYSTEMROOT%\system32\drivers\etc\hosts and add lines with the IP address WHITESPACE and the hostname(s). E.G.
127.0.0.1 lisatest lisatest.rushworth.us lisatest2 lisatest2.rushworth.us
10.1.2.3 otherhost otherhost.rushworth.us
10.2.3.4 anya anya.rushworth.us

Clear your DNS cache (ipconfig /flushdns) and navigate to the URL. You’ll be directed the IP address from your hosts file instead of the DNS registered address.

Approach #2 – No admin access

Install ModHeader in your Chrome browser and click the extension to modify the headers or install ModHeader in your Firefox browser. Click on the extension icon to set a header value.

Add a “Host” header with the value of the virtual host name you need to test

Navigate to the hostname of the new server – https://server2.rushworth.us – but the web server will receive the Host header you configured in ModHeader and serve the web site based on that host header.

 

Apache HTTPD: SSL Virtual Hosts

For quite some time, you couldn’t bind multiple SSL web sites to a single IP:Port combination — this had to do with the mechanics of negotiating an SSL session — the client and server negotiated encryption based on a specific certificate before the server really knew what the client was trying to retrieve. The quick/easy solution was to just add a virtual IP to the box and bind each individual web site to a unique IP address. While this was quite effective in a corporate environment or purely internal network, it was a terrible solution for a set of home-hosted personal web servers — I don’t want to buy four public IP addresses to host four differently named websites. My workaround was to off-port sites no one else would be using (the MQTT WebSockets reverse proxy) and use a reverse proxy to map paths within the family website to the remaining web servers. This page, for instance, is rushworth.us/lisa … which the reverse proxy re-maps to https://lisa.rushworth.us behind the scenes.

With Apache HTTPD 2.2.12 or later built against OpenSSL v0.9.8g or later, you can use Server Name Indication (SNI) to serve multiple SSL websites from a single IP:Port just like you have been able to do with non-SSL sites. Using SNI, the client includes “what they’re looking for” in first message of the SSN negotiation process so the server knows which cert to serve.

In your httpd.conf, indicate that you want to use SNI on an IP:Port combo

# Listen for virtual host requests on all IP addresses
NameVirtualHost *:443

And, optionally, configure one of the named virtual hosts as the default for non-SNI browsers:

SSLStrictSNIVHostCheck off

Now the configuration for your SSL sites can include a ServerName directive. Restart Apache HTTPD, and you’ll be able to access the proper SSL-enabled website without adding virtual IP addresses.

Zoneminder, time zones, and php-fpm

I’m in the process of installing Zoneminder on our new server. It was a fairly straightforward process — stop Zoneminder on the old server, dump the SQL database, fix the DEFINER values since I’m using a central database server instead of a server on localhost, install Zoneminder, copy the config file, set up the database user, pull in the SQL file, and start it all up.

Visiting the website, I get “ZoneMinder is not installed properly: php’s date.timezone is not set to a valid timezone”. I’d forgotten to set the timezone in php.ini. Added ‘date.timezone = “America/New_York”‘, restarted httpd and Zoneminder. And got the same error.

<?php
error_reporting(E_ALL);
var_dump(ini_get('date.timezone'),date_default_timezone_get());
?>

It’s not set. This isn’t a funky Zoneminder thing — this is a PHP problem. I realized that PHP now runs as its own service. Restarting httpd is insufficient. Restarted php-ftm and the time zone I’d set in php.ini showed up. This is a case where a reboot would have sorted it … but good to remember that, when changing PHP settings, the php service needs to be restarted.

The Colloquial Occam’s Razor

Occam’s razor – it is futile to do with more things that which can be done with fewer – is colloquially rendered as “the simplest solution is the most likely”. We had multiple tickets opened today for authentication failures on an Apache web server. Each malfunctioning site uses LDAP authentication and authorization against an Oracle Unified Directory. Nothing in the error logs. The service account from the Apache configuration can log in and query the directory from the box using ldapsearch, so the account is valid and there is nothing in the OUD preventing access from this particular host.

That’s a puzzler, and I was about to take down a lot of web sites to reload the service with its log level set to debug. Not even sure what made me do it, but I went out to the groups and looked at their member lists. Oops. Something had gone wrong with the identity management platform and employee accounts had been cleared from the groups (all of the contractors were still members, which made it even stranger). Added a few people back into groups appropriate for their position, voila they could log into their site again.

No idea how the identity management group restored the memberships, but verifying people who should have been members (who had been members and had done nothing to remove their memberships) were actually members of the group saved a lot of time running through debug logs. Sometimes the simplest answer is the most likely.

Apache HTTP Sandbox With Docker

I set up a quick Apache HTTPD sandbox — primarily to test authentication configurations — in Docker today. It was an amazingly quick process.

Install an image that has an Apache HTTPD server:    docker pull httpd
Create a local file system for Apache config files (c:\docker\httpd\httpd.conf for main config, c:\docker\httpd\conf.d for all of the extras like ssl.conf and php.conf, plus web sites), and c:\docker\httpd\vhtml for the web site content)
Launch the container: docker run -detach –publish 80:80 –publish 443:443 –name ApacheWebServer –restart always -v /c/docker/httpd/httpd.conf:/etc/httpd/conf/httpd.conf:ro -v /c/docker/httpd/conf.d/:/etc/httpd/conf.d/:ro -v /c/docker/httpd/vhtml/:/var/www/vhtml/:ro httpd

Shell into it (docker exec -it ApacheWebServer bash) to look around, or just access http://localhost from the Docker host.