Blocking Direct RTMP Connections to Adobe Connect Meetings causes Latency and Connection Failures

Tunneling with RTMP encapsulated in HTTP or RTMPT should be avoided as it causes latency that can have a negative impact on user experience in Adobe Connect Meetings. Tunneling can take place when the direct RTMP connection is blocked. In some circumstances,the latency commensurate with tunneling RTMP encapsulated in HTTP, can become so acute that it renders Adobe Connect Meetings unusable for some affected clients. And in rare cases a user may not be able to connect.

While the amount of acceptable latency depends on what one is doing in the room; RTMPT tunneling affects most activities. Some activities, such as screen-sharing are more bandwidth intensive than other activities such as presenting an uploaded PowerPoint from within a meeting room; The high latency commensurate with RTMPT tunneling would affect the former more than the latter. VoIP is often the first Adobe Connect feature to make the effects of high latency felt by end users.

Here is some feedback from a test I did while on-site with a client dealing with tunneling because of their refusal to route RTMPS around a third-party proxy:

External user tunneling during test:
Spike at 3.10/3.02 sec
Latency 403/405 ms up to 3.53/3.52 sec up .064 down 118
When latency peaks to 2.6/2.4 sec I get a mild interruption to the audio V
Video pauses momentarily when the latency spikes

Internal user with direct connection:
2 msec / 1 msec Up 0.08 kbits down 127 kbits
No pauses, delays or spikes

Tunneling should only be considered as a fallback mechanism or safety net to allow connections when RTMP is blocked due to something unplanned or for a few remote clients who must negotiate specific network obstacles. When RTMPT is the default route by network design, you will need to limit your activities within Connect to those feature that use the least bandwidth.

The picture below shows a direct connection over RTMPS on 443 is being blocked somewhere on the client’s network and the fallback mechanism built into Connect of tunneling RTMP encapsulated in HTTPS is engaged. This is usually caused by either proxy servers or firewalls or possibly both – any application-aware routing appliance on one’s network that sees the RTMPS traffic on 443 and recognizes that it is not HTTPS and intercepts it is a potential obstacle; RTMPS traffic is on port 443 and while it is disguised as HTTPS, it still may be blocked. The result is tunneling, indicated by the “T” in the output:

Compare with a connection without tunneling:

The recommended steps for anyone experiencing persistent tunneling, is for their network engineers to trust the source IP addresses of the Adobe Connect Hosted, ACMS, Managed ISP or on-premise Connect/AMS/FMS server VIPs in order to prevent the blocking of RTMPS traffic.

RTMPS is not supported by any third-party proxy server.  Static bypass works well to solve this issue. The problem stems from network policies that require all traffic to go through a proxy. The result is tunneling with commensurate high latency and drops. RTMPS must be allowed to stream around a proxy to avoid the overhead and latency of tunneling encapsulated within HTTPS. Attempts to cache the stream add no value.

Other options will depend on the capabilities of the third-party proxy servers in the affected client infrastructure. Blue Coat ProxySG was one of the popular proxy server options in our niche. Although now they are rare, the principle remains the same. In cases of latency invoked by tunneling RTMPS encapsulated in HTTP on a network that employs a proxy like Blue Coat ProxySG servers, sniff tests done by support representatives have indicated that when an affected client attempts to connect to an Adobe Connect Meeting, those clients would establish both explicit HTTP connections based on PAC file settings in the system registry to the Blue Coat ProxySG pool through a hardware-based load balancing device (HLD) and transparent HTTP and SSL connections through Blue Coat ProxySG via WCCP GRE redirect to several Adobe Connect servers. The problem manifests with RTMPS when the clients attempt to establish an SSL connection directly to the destination host without going through PAC file proxy settings. Since a Blue Coat ProxySG is commonly configured to perform an SSL intercept on both explicit and transparent HTTPS traffic, upon examining the content after decrypting the SSL payload from the clients, the Blue Coat ProxySG will return an exception and close the connection because the request doesn’t contain an HTTP component and cannot be parsed for policy evaluation. As a workaround, other than using static bypass, it is possible to create a proxy service with the destination set to the Adobe Connect server IP range on port 443 and to set the proxy setting to TCP-Tunnel with Early Intercept enabled. This will allow Blue Coat ProxySG to intercept and tunnel the traffic without considering whether it is RTMPS or HTTPS.

There is an API that may be run on the Connect server or cluster (or individual hosted account) that may help clients that employ a pac file:

For on premise, ACMS and managed ISP accounts, the API call to make Connect recognize the pac file will look like this where connect.domainname.com is the actual FQDN of the server or VIP:

https://connect.domainname.com/api/xml?action=acl-field-update&field-id=pac-proxy-flag&value=false&acl-id=7

To view the setting after toggling it, use this API:

https://connect.domainname.com/api/xml?action=acl-field-info&field-id=pac-proxy-flag&acl-id=7

For multi-tenancy hosted customers, the API call will be a little bit different as the acl-id will not be 7 as above but a larger number incremented on the multi-tenancy cluster and which appears in the URL when logged onto Connect Central. In the following multi-tenancy example, replace customerdomainname with your account domain prefix and replace xxxxxxxx with your  unique acl-id.

https://customerdomainname.adobeconnect.com/api/xml?action=acl-field-update&field-id=pac-proxy-flag&value=false&acl-id=xxxxxxxxx

Note: If you are using a PAC file, the value should be set to true.

On-prem: https://connect.domianname.com/api/xml?action=acl-field-update&field-id=pac-proxy-flag&value=true&acl-id=7

Multi-tenancy hosted: https://customerdomainname.adobeconnect.com/api/xml?action=acl-field-update&field-id=pac-proxy-flag&value=true&acl-id=xxxxxxxxx

To view the setting after toggling it, use this API:

https://customerdomainname.adobeconnect.com/api/xml?action=acl-field-info&field-id=pac-proxy-flag&acl-id=xxxxxxxxx

If you are unfamiliar with the API, to run this command, you must already be logged into Connect Central as an Administrator and simply paste the API call into the URL line of the browser and execute it. The feedback in the browser will denote success unless there is a problem noted; here is example output from a hosted multi-tenancy account:

Toggle this proxy flag API trying both the true and the false options in testing with both Classic and Standard view connections.

It is prudent to make sure you are always running the latest Adobe Connect Meeting Application available: Find the standalone and MSI downloads here: https://helpx.adobe.com/adobe-connect/connect-downloads-updates.html

Some Windows 7 clients began tunneling into Meetings when using Adobe Connect Meeting Application version 2019.3.3. To solve this and allow a direct RTMP(S) connection to Connect Meetings for these legacy clients, add the following line to the mms.cfg file in either: /Windows/system32/Macromed/Flash (32-bit Windows) or /Windows/SysWOW64/Macromed/Flash (64-bit Windows).

OpenSSL=false

If the affected Windows 7 client does not have an mms.cfg file, you may create one or download one from here: http://platinum.adobeconnect.com/mms/default/mms.zip

Extract the mms.cfg file and place a copy in: /Windows/system32/Macromed/Flash (32-bit Windows) or /Windows/SysWOW64/Macromed/Flash (64-bit Windows).

There is another option for legacy Windows 7 clients. You may update the client with the following TLS patches: https://support.microsoft.com/en-us/help/3140245/update-to-enable-tls-1-1-and-tls-1-2-as-default-secure-protocols-in-wi 

If participants in a Meeting are tunneling, there is always the option of using the Adobe Connect HTML5 client option. See the following tech-note for details: https://helpx.adobe.com/in/adobe-connect/using/enabling-adobe-connect-html-client.html

You may force the HTML client by appending to the URL of a meeting: html-view=true See the following tech-note: http://blogs.connectusers.com/connectsupport/how-to-force-html-client-in-meeting-adobe-connect/

The option of using Enhanced Audio and Video with Adobe Connect 12 changes the dynamics of the Adobe Connect Meeting Connection by employing WebRTC and is recommended where supported; it can be toggled under Meeting Information:

To avoid latency with WebRTC based Enhanced Audio/Video, follow the instructions in this tech-note: https://blogs.connectusers.com/connectsupport/avoiding-latency-using-webrtc-with-adobe-connect-hosted-meetings-seminars-and-virtual-classrooms/

Note: The difference between a transparent proxy and an explicit proxy is relevant and worthy of a side note. A transparent proxy sits on the network path, either directly or virtually and Web-based traffic goes through it, either because it physically cannot avoid it on the network path out from within an infrastructure or because it is set in line or ports 80 and 443 are custom-routed to it in various ways such as WCCP or PBR.  The routing table directs some traffic to go to the gateway or to the Internet unless it is on 80 or 443 in which case it will go to the proxy, rather than the gateway.

The transparent proxy is invisible to the browser while an explicit proxy on the other hand, is directly programmed, and therefore explicit.  It is set in the browser settings and tells the browser where to send traffic on specific IPs and ports; this IP and port is where you will send all requests to get out.  It is probable that the firewall will not let clients out directly, although it isn’t required.

With a transparent proxy you may tell the proxy that when you see specific destinations, put them on a layer-4 listener that forwards without paying any attention to payload.  You would only do this to sites you trust; in this case the AMS/FMS Adobe Connect servers running RTMPS.

With explicit proxy servers the default listener is an HTTP listener, and this only speaks HTTP.  If you choose to detect the protocol, then when the packet received after the TCP handshake is a client hello rather than a get request, the proxy can see that it is the SSL protocol.  If the setting to detect protocol is enabled, it can hand this session off to the SSL proxy, which does speak SSL.  It can then choose to decrypt, or not decrypt, but it can see the handshake and have influence on it.

On the explicit proxy, you choose an option that doesn’t detect the protocol: url.host=site.com detect_protocol(none)

In this case, the HTTP proxy sees the client hello and does not hand it off to the SSL proxy, but keeps it.  It does not speak SSL, so it handles it at layer 4 and does not pay any attention to the SSL handshake, or even know the protocol.  It just forwards the byte sequence.

Transparent overrides the default SSL (approx layer 6) listener on port 443 by setting up a more specific listener on layer 4.

Explicit is tunneled by saying for this specific destination, don’t hand off to SSL proxy, ignore it, and pass the payload.