Tag: websockets

Reverse Proxying WebSockets through mod_proxy — HTTP Failback

I’ve been successfully reverse proxying MQTT over WebSockets via Apache HTTPD for quite some time now. The last few weeks, my phone isn’t able to connect. There’s no good rational presented (and manually clicking the “send data” button on my client successfully connects). It was time to upgrade the server anyway. Once I got the latest Linux distro on the server, I couldn’t connect at all to my MQTT server. The error log showed AH00898: Error reading from remote server returned by /mqtt

Evidently, httpd 2.4.47 added functionality to upgrade and tunnel in accordance with RFC 7230. And that doesn’t work at all in my scenario. Haven’t dug in to the why of it yet, but adding ProxyWebsocketFallbackToProxyHttp Off to the reverse proxy config allowed me to successfully communicate with the MQTT server through the reverse proxy.

OwnTracks WebSockets MQTT SSL Error

A few weeks ago, we stopped getting location updates from OwnTracks on our phones. Checking the status, I see an error indicating that the connection failed because my certificate does not have a SAN. Which … true, it does not. I knew some consortium agreed that all certs should have SAN values (and RFCs had been updated to reflect this new direction). Evidently version 2.2.2 of OwnTracks has added SAN verification. I reissued the certificate from my CA and added a SAN. I had to put the cert on both my MQTT websockets reverse proxy and the mosquitto server; but, once both were using the new cert, OwnTracks connected and cleared through the queued updates.

Mosquitto 1.6.8 With Websockets Becomes Unresponsive

When I installed software on our new server, I got the “latest and greatest”. And have found that my mosquitto server hangs after about 20 minutes. No errors. Even strace doesn’t show anything beyond it not doing anything. I am able to use the command line utilities to confirm that the service isn’t there even though it looks like it is working.

Subscribe to the subtree of a topic:

mosquitto_sub -h mqtt.example.com -t testtopic/#

Publish a message to a topic:

mosquitto_pub -h mqtt.example.com -t testtopic/data/lisa -m "Test4"

To test WebSockets, I’ve put together a Python script that subscribes to a topic. Changing the commented lines switches between the WebSocket reverse proxy, the old and new MQTT servers via WebSockets, and the old and new MQTT servers directly in an attempt to isolate what is wrong.

import sys
import paho.mqtt.client as mqtt

def on_connect(mqttc, obj, flags, rc):
    print(f"rc: {str(rc)}")

def on_message(mqttc, obj, msg):
    print(f"{datetime.datetime.now()}: {msg.topic} {str(msg.qos)} {str(msg.payload)}")

def on_publish(mqttc, obj, mid):
    print(f"mid: {str(mid)}")

def on_subscribe(mqttc, obj, mid, granted_qos):
    print(f"Subscribed at {datetime.datetime.now()}: {str(mid)} {str(granted_qos)}")
def on_log(mqttc, obj, level, string):
    print(string)
# Client uses websockets
mqttc = mqtt.Client(transport='websockets', client_id="ljrTestingPythonScript", clean_session=False)
# Client uses MQTT directly
#mqttc = mqtt.Client(client_id="ljrTestingPythonScript", clean_session=False)

mqttc.username_pw_set("whateveruser", "wh@t3v3rP@s5w0rd")
mqttc.on_message = on_message
mqttc.on_connect = on_connect
mqttc.on_publish = on_publish
mqttc.on_subscribe = on_subscribe

#mqttc.connect("oldmosquitto.example.com", 80, 60)
mqttc.connect("newmosquitto.example.com", 80, 60)
#mqttc.connect("newmqtt.example.com", 1883, 60)
#mqttc.connect("newmqtt.example.com", 9001, 60)
#mqttc.connect("oldmqtt.example.com", 1883, 60)
#mqttc.connect("oldmqtt.example.com", 9001, 60)

mqttc.subscribe("owntracks/#", 0)

mqttc.loop_forever()

None of this helped, unfortunately. The reverse proxy couldn’t communicate with the MQTT server because it was unresponsive. Attempting to communicate with the server directly fails too.

I see a few issues in the Mosquitto repository that are similar — and the current discussion indicates that libwebsockets 3.2.0 introduced some incompatibility that they’ve addressed in Mosquitto 1.6.7 … since my problem relates to WebSockets, I wanted to try running the iteration before whatever changed.

git clone --branch v3.1.0 https://libwebsockets.org/repo/libwebsockets
cd libwebsockets
mkdir build
cd build
cmake ..
make
make install

git clone --branch v1.6.2 https://github.com/eclipse/mosquitto.git
cd mosquitto
vi config.mk # Change WITH_WEBSOCKETS:=no to :=yes
make
make install

ln -s /usr/local/lib/libwebsockets.so.14 /lib64/
ln -s /usr/local/lib/libmosquitto.so.1 /usr/lib64/libmosquitto.so.1

/usr/local/sbin/mosquitto -v -c /etc/mosquitto/mosquitto.conf

We’ll see if this is more reliable. It’s been up for 30 minutes … which is longer than the 1.6.8 iteration managed to run.