The yax.im service was unavailable to ~11% of our users since March 28th. This was a result of a bug in a community module that was triggered by our upgrade to prosody 0.11.

The yax.im XMPP service is running on the host xmpp.yaxim.org and uses SRV records to let clients know the actual host name. However, a certain percentage of clients (around 11%) are running on IT infrastructure from the last century and fail to obtain those SRV records. These clients fall back to connecting directly to the yax.im host instead.

To accomodiate the clients, we are using HAProxy and mod_net_proxy to properly forward the connections from the web server to the XMPP server.

After the upgrade to prosody 0.11, mod_net_proxy failed to accept new connections from HAProxy, so that clients not using SRV were caught in an infinite “Connecting…” loop. Unfortunately, this was not detected by our technical monitoring, and only was brought up recently by an administrator’s relative.

The communication between HAProxy and mod_net_proxy has been restored today, and the service should be fully available from now on. We will also improve our monitoring to make sure that similar problems are detected in time in the future.