Updated July 20, 2021

May 14, 2019 /ACTS /Administration /Clustering /General /Install /Meeting /SSL / Frank DeRienzo, MBA

Adobe Connect Clustering: Configure Secure VIPs and Pools

Introduction

This updated article discusses deploying Adobe Connect servers in clusters/pools on-premise. Adobe Connect users who use servers that are hosted either by Adobe or a managed ISP may ignore this article unless interested in how an Adobe Connect server cluster looks. As soon as multiple Adobe Connect servers are in the deployment equation, they will nearly always be running with secure socket layer (SSL) through a hardware-based SSL accelerator/ load-balancing device (HLD). Since all enterprise-class HLDs are also SSL accelerators (any that are not both are either legacy or low-end by definition), this example-based article offers a configuration option for Adobe Connect servers running Meetings, on-demand content, Training, Curriculum and Virtual Classrooms in clusters/pools with HLDs.

Notes and Caveats

This article is only for on-premise Adobe Connect deployments.

This article is intended as a supplement to published Adobe Connect installation documentation:

Beginning with Adobe Connect 10, if you wish to take advantage of the additional HTML5 client connection option, an Adobe Connect Transmuxing Server (ACTS) or servers are required. For small deployments a single ACTS instance can be co-located on an Adobe Connect origin server, but in most cases a separate server or server pools for redundancy are prudent. ACTS is only needed for the HTML5 client and is covered in more detail in the installation guide and here in these articles:

Adobe Connect integrates with various telephony options. The Adobe Media Gateway (AMG) is an additional server that is required for all telephony integration with the exception of PGi. The telephony adapters that integrate with AMG are part of the Adobe Connect installer. For a diagram of what AMG looks like when integrated with an on-premise cluster along with installation guidance see the following tech-notes:

Beginning with Adobe Connect 9 the Adobe Connect Events module requires a separate server or micro-cluster running Adobe Experience Manager (AEM). Adding the Events module to an Adobe Connect on premise installation is not difficult and can be accomplished during initial installation or subsequently. AEM installation is not covered here in the interest of article length.

Prerequisite knowledge

A basic understanding of Domain Name System (DNS), network infrastructure, routing, bridging, and Network Address Translation (NAT)

Configuring DNS and ports and assigning SSL certificates
Configuring the Adobe Connect server pool
Configuring application-level health monitors on the HLD/SSL accelerator
Adobe Connect server resource monitoring
Clustering Adobe Connect servers across data-centers
Conclusion

Configuring DNS and ports and assigning SSL certificates

The best place to start is with a basic network diagram illustrating the desired end state of an Adobe Connect server cluster/pool running with ACTS behind a high end hardware-based load-balancing device (HLD) running SSL acceleration for both HTTPS and RTMPS:

Figure 1: Adobe Connect server pool running Adobe Connect Meeting and ACTS.

Following the example in Figure 1, the virtual Internet protocol addresses (VIPs) on the HLD and the server pools correspond in the following manner:

Adobe Connect Application HTTPS VIP: connect.adobe.com: 10.10.10.1:443 points to Adobe Connect server pools: 192.168.0.1: 8443 and 192.168.0.2: 8443 using the Round Robin algorithm without session-awareness and without the Nagle’s algorithm (non-sticky or non-session-aware)
Adobe Connect Meeting RTMPS VIP: meeting1.adobe.com: 10.10.10.2:443 points to the first Meeting server, meeting1: 192.168.0.1: 1935
Adobe Connect Meeting RTMPS VIP: meeting2.adobe.com: 10.10.10.3:443 points to the second Meeting server, meeting2: 192.168.0.2: 1935
ACTS HTTPS VIP: acts.adobe.com: 10.10.10.4:443 points to the ACTS server for HTML5 rendering, 192.168.0.3:9002

Except for the lack of session-awareness, the HTTPS (or application) VIP appears quite conventional; it points to a two-server Adobe Connect pool. Under the hood however, while the HLD-based HTTPS VIP and application pool handles fail-over for the tomcat-based Adobe Connect application, it is the Adobe Connect application itself that handles load-balancing for the RTMPS meetings by monitoring meeting health and polling the SQL database to find the least-loaded Adobe Connect Meeting server to host any new meeting rooms.

Resist any temptation to use a single VIP with multiple open ports for both RTMPS and HTTPS. Each VIP also needs its own certificate and unique fully qualified domain name (FQDN); the configuration above requires three unique certificates and three FQDNs. Do not attempt to facilitate traffic using a single FQDN, the impact of your head with the wall will hurt you, delay your deployment and leave the wall unaffected.

The Adobe Connect Meeting RTMPS configuration can be confusing; it may seem odd to have a single VIP point to a single server in a server pool as is the case with RTMPS , but each Adobe Connect Meeting RTMPS VIP for streaming on the HLD must point to a single Adobe Connect Meeting server pool; it is a one-to-one correspondence.

It may also seem odd to run SSL without session-awareness (and on an LTM without Nagle’s algorithm), but in the case of Adobe Connect, the relationship of the HLD is not with a conventional application pool. Connect is both a pool and a cluster and if session-awareness is used, content in a meeting may not appear for clients; session-awareness on the VIP will cause an intermittent lack of content appearing in the client browser.

This example configuration we are following together requires:

One unique SSL certificate and FQDN for the HTTPS VIP: connect.adobe.com
One unique SSL certificate and FQDN for RTMPS VIP: meeting1.adobe.com
One unique SSL certificate and FQDN for RTMPS VIP: meeting2.adobe.com
One unique SSL certificate and FQDN for HTTPS VIP: acts.adobe.com

Note: Use of a wild-card certificate is supported.

The external names for each server are the VIP names: meeting1.adobe.com and meeting2.adobe.com, respectively; the host name is connect.adobe.com and ACTS is acts.adobe.com. The only host name suffix the end users will ever see is: connect.adobe.com. Still, unique FQDNs and certificates are required on the HLD/SSL accelerator: one for each VIP pointing to each Adobe Connect Meeting/RTMPS server, one for both of the Adobe Connect/HTTPS servers and one for HTTPS ACTS.

From the perspective of the HLD/SSL accelerator, in our example deployment, there are actually five servers in four pools: two Connect HTTPS application servers (connect.adobe.com) in one pool, two Adobe Connect Meeting servers (meeting1.adobe.com and meeting2.adobe.com) each in a pool of its own with its own corresponding VIPs and one ACTS server in its own pool (acts.adobe.com).

ACTS need to resolve the external names of the AMS/RTMPS servers. This may require some unique routing including possibly an internal VIP on the load-balancing hardware dedicated to getting the ACTS traffic to the external Meeting names.

For server to server communications between ACTS and Connect, make sure that the Connect server can resolve the machine-name of the ACTS server behind the load-balancing hardware as well as the external name of ACTS at the VIP.

An application-level health monitor on the HLD/SSL accelerator should be associated with the HTTPS VIP, because the Adobe Connect application server pool will handle load balancing and fail-over of the meeting rooms on the collocated Meeting servers (RTMPS) while the HLD handles fail-over of HTTPS for the Adobe Connect application.

Do not use an HTTP profile for RTMP VIPs as it may affect playback of video. Remember that you have two servers running on each box, a Tomcat application server and a Meeting server which is a modified Adobe Media Server (AMS). Do not treat the AMS server as though it were a conventional application server; RTMP is a streaming protocol that requires a TCP profile at the HLD VIP.

See the following tech-note for details on RTMPS VIPs and pools as they should be configured on F5 Local Traffic Manager (LTM): https://blogs.connectusers.com/connectsupport/connect-meeting-rtmp-vsvips-on-load-balancers/

Set any session-timeout variable on the HTTPS VIP on the load-balancing hardware to no less than two minutes to prevent freezing of on-demand videos during playback. On-demand video will often cache at the client and then timeout at the VIP do to inactivity at the VIP since the playback is local after the initial download to local client cache.

This example configuration employs a single IP address on each Adobe Connect server. The single IP address uses multiple ports: 8443 is for the Adobe Connect application and 1935 is for the Adobe Connect Meeting server. All traffic between the HLD/SSL accelerator and the Adobe Connect servers is unencrypted; you must point the HTTPS VIP to the Adobe Connect application pool on port 8443 on each of the Communication servers; port 80 will not work.

Note: Do not try to take shortcuts with reference to the certificates and VIPs; even for a lab environment. The Adobe Connect servers need genuine unique FQDNs and SSL certificates including intermediate certificates; self-signed certificates will not work with Adobe Connect meetings; the meeting rooms simply will not open unless the certificate is trusted. The client session will hang during the hand-off from the conventional HTTPS browser session to the RTMPS meeting session (whether in the client Meeting application or browser) unless the browser automatically accepts the certificate; the hand-off will not offer a popup to accept an untrusted certificate, it will simply fail to connect. To obtain trusted certificates, you must contact a Certificate Authority and supply them with SSL Certificate Signing Requests (CSR) containing organizational information and FQDNs that must correspond with each SSL certificate.

With that said, you may only want to secure HTTPS while leaving RTMP unencrypted; this will protect login data and some content, but will leave chat and notes and streaming data in the clear. There are a number of ways to set this up. If you prefer to route RTMP through the HLD, simply resolve traffic at the RTMP VIPs on the HLD and pass the traffic from 1935 at the VIP to 1935 on each server pool or go around the load-balancing device and terminate the RTMPS traffic directly on each Meeting server. This optional configuration might look like this:

Figure 2: Adobe Connect server pool running Adobe Connect Meeting with RTMP.

Configuring the Adobe Connect server pool

Even though the HLD/SSL accelerator is doing all the encryption, there are still some settings that need to be configured on each Connect server to enable SSL traffic. To configure the Adobe Connect servers to run fully secured through the HLD on a single pool IP address as depicted in this working example, you will need to add the following entries to the custom.ini file in the Adobe Connect root installation directory:

For Connect 9.X servers and later:
ADMIN_PROTOCOL=https://
SSL_ONLY=yes
RTMP_SEQUENCE=rtmps://external-host:443/?rtmp://localhost:8506/
MEETING_TIMEOUT=0
MEETING_CONNECT_BACK_TIME=1

Check the ADMIN_HOST entry in the custom.ini; it should reflect the actual Adobe Connect URL rather than a server machine name. Within an Adobe Connect cluster in particular it can be problematic if multiple ADMIN_HOST names are in multiple custom.ini files:

ADMIN_HOST=connect.adobe.com

After adding these entries, save the custom.ini file.

Note: port 8506 referenced in the RTMP sequence above is for internal server-to-server communication.

On each Adobe Connect server, to enable SSL, uncomment the following two sections in /appserv/conf/server.xml:

<Connector SSLEnabled=”true” URIEncoding=”utf-8″ acceptCount=”250″ connectionTimeout=”20000″ enableLookups=”false” executor=”httpsThreadPool” port=”8443″ protocol=”HTTP/1.1″ proxyPort=”443″ relaxedQueryChars=”{}|<>"” scheme=”https” secure=”true”/>

Beginning with Adobe Connect 9, the installer includes a cluster option. If you begin with a single server installation and expand later to a clustered environment by adding a server or servers, you will need to manually make the following change in the /appserv/conf/server.xml file in order to enable intra-cluster communication over port 8507. It is prudent to double check this in the server.xml file after installing even if the cluster option was selected during installation:

The next step is to properly edit the Adobe Connect server settings to match the settings in the custom.ini file. Select Start > All Programs > Adobe Connect Server > Configure Adobe Connect Server > Server Settings to go to the Adobe Connect server configuration interface on localhost port 8510 in a browser (see Figure 3).

Figure 3: Editing the Adobe Connect server settings.

The server settings depicted in Figure 3 correspond to this working example. The names depicted are FQDNs that resolve on the HLD.

After your custom.ini file and your server settings are configured, stop the Connect services beginning with the Adobe Connect Service, followed by the AMS service on each Windows server in the pool (see Figure 4).

If you are running the Adobe Connect SQL database in a SQL cluster rather than in a mirrored environment, you will want to make sure that Adobe Connect makes multiple database connection attempts during SQL fail-over. If Adobe Connect loses its SQL database, the entire Adobe Connect cluster will go down and it will wait for an administrator to manually reconnect to the database through launching the Connect configuration console on port 8510. Add the following to the custom.ini file to support any delays in clustered SQL fail-over:

DB_URL_CONNECTION_RETRY_COUNT = 15
DB_URL_CONNECTION_RETRY_DELAY= 30

The actual JDBC string is in the config.ini file so you do not need to put it into the custom.ini; double check the config.ini if you are running into any problems with the JDBC reconnection string:

DB_URL=jdbc:macromedia:sqlserver://{DB_HOST}:{DB_PORT};databaseName={DB_NAME};user={DB_USER};password={DB_PASSWORD};ConnectionRetryCount={DB_URL_CONNECTION_RETRY_COUNT};ConnectionRetryDelay={DB_URL_CONNECTION_RETRY_DELAY}

Save the custom.ini and cycle the services. If the HLD/SSL accelerator is properly configured, you will be able to browse the Connect server pool through the HLD/SSL accelerator after restarting the Connect and AMS services.

Figure 4: Stop and start the Adobe Connect and AMS services.

Configuring application-level health monitors on the HLD/SSL accelerator

In order to make sure that the HLD/SSL accelerator performs fail-over in case one of the Adobe Connect application servers should hang or otherwise become unavailable at the application layer, you will want to make certain that the VIP that points to the application server pool is configured with an application-level health monitor. If you simply probe the health of the Adobe Connect servers with a default health monitor at the level of the IP stack, then there are potential cases when the HLD/SSL accelerator might send traffic to a server with a non-responsive application that only seems alive to lower-level probing mechanisms such as the packet Internet groper (PING). Always set the health monitor to probe for an actual string of content on the Adobe Connect server; all high-end HLDs offer application-level health monitoring. It may not always be intuitive how to configure the monitor as each HLD has a different interface and different means of probing an application, but the following guidance will help you get an appropriate monitor in place.

Consider that following our example deployment you have multiple server pools and VIPs. The VIP and pool combination that needs an application-level health monitor for fail-over is the Adobe Connect application HTTPS server VIP and pool:

HTTPS VIP: connect.adobe.com: 10.10.10.1:443 points to Connect servers: 192.168.0.1: 443 and 192.168.0.2: 443

The probe or health monitor should point to a string on each Adobe Connect server in its pool to check the health of each server. If one of the servers in the pool becomes non-responsive, the monitor will mark the server down and the HLD will redirect all traffic to the remaining server.

The Adobe Connect Meeting server VIP/pool combinations do not need a health monitor because the Adobe Connect application server pool handles fail-over for the Adobe Connect meeting rooms:

RTMPS VIP: meeting1.adobe.com: 10.10.10.2:443 points to Adobe Connect Meeting server meeting1 192.168.0.1: 1935
RTMPS VIP: meeting2.adobe.com: 10.10.10.3:443 points to Adobe Connect Meeting server meeting2 192.168.0.2: 1935

Because there is only one server in each pool, there is no place for the HLD to redirect meeting traffic should one of the Adobe Connect meeting servers fail to respond. The only reason to probe the Adobe Connect Meeting server VIP/pool combinations might be to trigger an email message to an administrator to warn that one of the Adobe Connect Meeting servers is problematic and that the application pool has triggered fail-over.

The best string on the servers that you may point your application-level health monitor towards is the testbuilder diagnostic page:

/servlet/testbuilder

The testbuilder page will send back the “status-ok” string.

It is best to point the health monitor to the testbuilder page rather than a simple HTML string, because testbuilder is actually probing the Adobe Connect database to make sure there is a healthy connection. If there is any problem with the Adobe Connect server application, then testbuilder will not report the “status-ok” string.

Each HLD has a different interface to configure these monitors and each one does the check differently, the following example works with F5 BIG-IP LTM against testbuilder:

send “GET /servlet/testbuilder HTTP/1.1\nHost: \nConnection: Close \n\r\n”
“status-ok”

For additional information on the testbulder target page, see the following tech-note:

You may also place an HTML file in the Connect /common directory on each Adobe Connect server and point to that file (always test the access to the html file via a browser to be sure that it does not require a login – the common directory and the help subdirectory are both OK with all prior versions of Adobe Connect). This option should be used along with testbuilder as a separate and supporting health check. The following example shows an HTML file called healthmonitortarget.html containing the string You are being served HTML”

send “GET /common/healthmonitortarget.html”

“You are being served HTML”

Note: With reference to the testbuilder file behavior, if, for example, Connect receives a SQL Exception (DB Down, etc.) it does not change testbuilder’s output string, it tries to reconnect a few times, then, if it fails to reconnect, it fails over, and then ultimately restarts the Connect server. If you have the JDBC restart string (described above) in place in the custom.ini, then, in theory, this is desirable behavior. If Adobe Connect was more aggressive in the changing of the testbuilder output string, then it could trigger superfluous fail-over at the application VIP by marking down a server that may only experience a brief reconnection to the Connect DB. If Connect fails and restarts due to a more severe DB connection problem and is still unable to connect once the server restarts, testbuilder will show the following output:

2
status-critical

If you set up the healthmonitor as described above, then what testbuilder may miss, the html health monitor will pick up and vice versa. The key thing is to test the health-monitors vigorously and inspect the Adobe Connect debug logs for any errors they may generate. Since each HLB acts differently and often SSL profiles and other variables will affect behavior, it is prudent to test the health monitors under all server fail-over conditions.

Adobe Connect server resource monitoring

You may use the Java Monitoring Console to view resource utilization on an Adobe Connect server/cluster. On each Adobe Connect server, edit \appserv\conf\ConnectProSvc.conf and uncomment the Remote JMX Connections section; by default it listens on port 4111:

# Remote JMX connections (uncomment to enable remote JMX access; unauthenticated as-is)
#wrapper.java.additional.200=-Dcom.sun.management.jmxremote
#wrapper.java.additional.201=-Dcom.sun.management.jmxremote.port=4111
#wrapper.java.additional.202=-Dcom.sun.management.jmxremote.authenticate=false
#wrapper.java.additional.203=-Dcom.sun.management.jmxremote.ssl=false

Edit the custom.ini file in the Breeze or Connect root directory by adding the following line:

ENABLE_MONITORING=yes

Save the custom.ini and cycle the services.

You may monitor the Adobe Connect servers from any client that can gain access to the servers directly by their IP addresses; simply run the jconsole \Program Files\Java\jdk##/lib/jconsole and point to the Adobe Connect servers on port 4111.

Clustering Adobe Connect servers across data-centers

There is not a federated option to allow an Adobe Connect origin server cluster to be spread about geographically. Very little latency is tolerable between Adobe Connect servers. With that said, there is the possibility of splitting an origin cluster across two data-centers for additional redundancy if the latency between the two data-centers is consistently 3 milliseconds or less. Intra-cluster communication on ports 8507 and 8506 must be unhampered for a cluster to work properly.

Spreading a cluster across two local data-centers with less than 3ms of latency requires that all servers point to the same database. Basically I am describing a regular cluster except with half of the servers in one building and the other half in another building nearby. When you do this, it is prudent to make sure that the backup meeting rooms that are spawned in support of every primary active meeting room are always created on servers in the separate data-center from the active primary meeting. You can do this in the Connect database in PPS_ENUM_DATA_HOSTS. Configure the hosts in one data center with one pool_id, and give the other servers in the second datacenter another pool_id. Then add this setting to the custom.ini on each origin server:

APP_SERVER_POOLING=true

This setting is necessary to configure fail-over to take advantage of multiple availability zones.

See the following tech-notes for additional information on distributed clustering:

Conclusion

The need for security, redundancy, and scalability is often answered by clustering / pooling application servers and running SSL encryption employing a high-end HLD/SSL accelerator. Let’s look at it in a configuration that goes beyond the simple basic two-server cluster depicted. The maximum size for a single Adobe Connect cluster is ten servers.

Where to go from here

Now that you have successfully configured an Adobe Connect server pool to run behind an HLD/SSL accelerator, you may want to add an Adobe Experience Manager(AEM) micro-cluster to host rich Adobe Connect Events. You may also wish to integrate telephony via Unified Voice driven by our Flash Media Gateway (AMG) servers or with one of the many telephony adaptors. When your team’s staff discover how simple it is to collaborate through Adobe Connect meetings and how Adobe Presenter and Adobe Captivate and Adobe Captivate Prime can convert a plethora of content into fully deployed presentations, courses, and curricula, you may find that further expansion and integration is warranted.

ACTS, Administration, Clustering, General, Install, Meeting, SSL

Frank DeRienzo, MBA

PRINCIPAL TECHNICAL ACCOUNT MANAGER

Adobe Connect Support Blog