Connect Meeting RTMP VS/VIPs on Load-Balancers

This article applies to on-premise Adobe Connect servers running behind hardware-based load-balancing devices or SSL accelerators.

A common cause of performance problems in Adobe Connect Meetings stems from the improper configuration of the Virtual Server (VS) Virtual IP Address (VIP) handling Real Time Messaging Protocol (RTMP) traffic in on-premise Connect deployments.

An Adobe Connect Meeting Server is at least two servers in one (possibly more if AEM/Events and UV telephony are incorporated); it is at least always a Tomcat-based HTTP application server and an Adobe Media Server (AMS) using RTMP. The two servers are fully integrated to work together in tandem to support Adobe Connect Meetings.

The most popular load-balancing and SSL acceleration  option in the Adobe Connect on-premise enterprise is the F5 BIG-IP Local Traffic Manager (LTM). This tech-note will illustrate the proper configuration of an RTMP VIP supporting Adobe Connect Meeting on an F5 LTM. The concepts apply to any load-balancing device and SSL accelerator.

The first thing to note is that the general configuration of a Connect server or cluster running behind an SSL accelerator or load-balancing device always requires more then one VIP. There are no exceptions to this rule and any attempts at shortcuts will result in delayed deployments and support cases. Attempts to place all traffic on a single VS/VIP are as common as they are incapacitating. General Connect cluster architecture tech-notes are here:

Adobe Connect Clustering: Configure Secure VIPs and Pools

Configure an Adobe Connect Server Pool/Cluster Test Lab

A simple diagram of an Adobe Connect server behind an F5 LTM follows; see the two VS/VIPs and Fully Qualified Domain Names (FQDNs) for each on the LTM:

Below we add a server to show a basic Connect cluster VIP configuration; see how each Connect Meeting server has its own VS/VIP while one VS/VIP servers both HTTPS application servers.

Note: Neither of these basic diagrams depicts advanced configurations such as the integration of the Adobe Experience Manager (AEM) Events module. This article focuses on the performance of the Adobe Connect Meeting RTMP VIP in its basic context.

There is usually not an option for RTMP in the VIP profile of a hardware-based load-balancing device. A basic TCP profile is the correct choice. Here it is depicted on an F5 BIP-IP LTM:

With detail:

Note that the symptom for an improperly configured VS/VIP is either the inability to launch a Connect Meeting or excessive latency in the Meeting due to RTMP tunneling (RTMPT) encapsulated within HTTP when the RTMP VIP is blocked or inoperable.

The presence of a capitol “T” in the latency indicator of an Adobe Connect Meeting indicates tunneling:

This tech-note describes tunneling induced latency:

Tunneling with RTMP encapsulated in HTTP (RTMPT) should be avoided as it causes latency

Further diagnosis is usually warranted by using the Connect Meeting App logging mode as depicted here:

Enable Logging in the Meeting App

Also here:

Troubleshooting Verbose Meeting App Logging

When the RTMP VS/VIP profile is improperly configured, the Connect Meeting addin verbose log will show it clearly, particularly when it is compared with the server-side debug log.

Example snippet from a Connect Meeting App verbose log (Classic View):

18:51:55    16844    PLAYER_TRACE    SSL connection closed.
18:51:55    16844    PLAYER_TRACE    SSL DoSSLHandshake WaitHandshake not in ssl_active state. (State is 0.) Failing.
18:51:55    16844    PLAYER_TRACE    SSL DoSSLHandshake WaitForSocket not in ssl_active state, failing.
18:51:55    16844    PLAYER_TRACE    SSL Receive socket read error 0x0.
18:51:55    16844    ACTION_TRACE    5/10/2016 14:51:55.101 [DEBUG] breezeLive.main.FCSConnector [attempt 1 of 60] Trying fallback tunneling connection rtmps://onlinemeeting.connectexample.com:443/?rtmp://localhost:8506/meetingas3app/7/1234567/
18:51:55    17179    PLAYER_TRACE    NetConnectionIO::DoConnect rtmps protocol, HTTP(S) tunneling, tunnel open succeeded.

The corresponding snippet in the server debug log as well as the application logs will read: RTMPT and often reconnect=true:

               Line 23456: 2016-01-17  14:25:06              32260    (s)2641173          Asc-Room               IA_CONNECT      [dID:32, ticket:123456789xyz, phase:, uID:, name:]             New client connecting:  { ip=127.0.0.1, protocol=rtmpt, player=MAC 11,9,971,247, savedConnectionSpeed=undefined, reconnect=true }                        –

[11-05 15:08:05] FCSj_Worker:18 (INFO) params: {bytesdown=0, protocol=rtmpt, ticket=123456789xyz, status=C, reconnect=true, nickname=John Doe, action=register-client, role=v, bytesup=0, session-timeout=12}

Correct configuration of the RTMP VS/VIP is extremely important; a Connect Meeting VS/VIP must have a dedicated FQDN.  It must have its own SSL certificate if SSL is accelerated through the load-balancing device and the VS/VIP must not have an HTTP profile; a TCP profile is needed.

For some additional information about troubleshooting Connect architecture with reference to hardware-based load-balancing devises and SSL accelerators, see the following tech-notes:

The Adobe Connect Deployment Guide on the F5 Website needs Updating

Configuring application-level health monitors for Connect on BIG-IP Local Traffic Manager