July 22, 2019 /ACTS /Administration /Application /Clustering /Database /General /Install /Meeting / Frank DeRienzo, MBA

Configure an Adobe Connect Server Pool/Cluster Test Lab

This updated article discusses deploying Adobe Connect on-premise servers in an example lab pool/cluster configuration behind an application-aware HLD such as F5 BIG-IP LTM. Adobe Connect users who use servers that are hosted either by Adobe or a managed ISP may ignore this article unless interested in the topic. This article offers an example test lab configuration – a general working example of a simple lab cluster running without SSL.

An Adobe Connect test lab cluster deployment is a critical support tool for on-premise deployments. Adobe Connect has rich APIs and sundry integration and customization options ranging from SSO to LMS to branding and compliance, etc. Testing changes and updates to this wide range of options in a lab is most prudent and highly recommended.

Requirements

To complete this tutorial you will need to install the following software and files:

Adobe Connect 10.x with a Cluster License
Microsoft Windows Server 2012 R2 (64-bit), 2016 (64-bit)
Microsoft SQL Server 2012 (64-bit), 2014 (64-bit), 2016 (64-bit).
An F5 BIG-IP LTM or other hardware-based load-balancing device

Note: Here is a snippet from the Adobe Connect EULA that addresses the additional use of Adobe Connect server licenses for non-production test labs; it allows for development or quality-assurance non-production or sandbox installations of one additional server for each server purchased without additional cost:

Prerequisite knowledge

A basic understanding of network infrastructure, DNS name-resolution, routing, and Network Address Translation (NAT).

Notes and Caveats

This article is only for on-premise Adobe Connect lab/test environment deployments.

This article is intended as a supplement to published Adobe Connect installation documentation:

Beginning with Adobe Connect 10, if you wish to take advantage of the additional HTML5 client connection option, an Adobe Connect Transmuxing Server (ACTS) or servers are required. For small deployments a single ACTS instance can be co-located on an Adobe Connect origin server, but in most cases a separate server or server pools for redundancy are prudent. ACTS is only needed for the HTML5 client and is covered in more detail in the installation guide and here in these articles:

Adobe Connect integrates with various telephony options. The Adobe Media Gateway (AMG) is an additional server that is required for all telephony integration with the exception of PGi. The telephony adapters that integrate with AMG are part of the Adobe Connect installer. For a diagram of what AMG looks like when integrated with an on-premise cluster along with installation guidance see the following tech-notes:

Beginning with Adobe Connect 9 the Adobe Connect Events module requires a separate server or micro-cluster running Adobe Experience Manager (AEM). Adding the Events module to an Adobe Connect on premise installation is not difficult and can be accomplished during initial installation or subsequently. AEM cluster installation is not covered here in the interest of article length and simplicity.

Configuring DNS and port assignments
Configuring the Connect server pool
Configuring application-aware health monitors
Conclusion

Configuring DNS and port assignments

The best place to start is with a basic network diagram illustrating the desired end state of a Connect server pool running behind a high end hardware-based load-balancing device (HLD), shown in Figure 1.

Figure 1: Adobe Connect server pool running Connect Meeting

Following the example in Figure 1, the virtual Internet protocol addresses (VIPs) on the HLD and the Adobe Connect server pools correspond in the following manner:

HTTP VIP: connect.adobe.com: 10.10.10.1:80 points to Adobe Connect servers: 192.168.0.1: 80 & 192.168.0.2:80
RTMP (TCP) VIP: meeting1.adobe.com: 10.10.10.2:1935 points to Adobe Connect server meeting1 192.168.0.1:1935
RTMP (TCP) VIP: meeting2.adobe.com: 10.10.10.3:1935 points to Adobe Connect server meeting2 192.168.0.2:1935
HTTP VIP: acts.adobe.com: 10.10.10.4:9002 points to Adobe Connect Transmuxing server acts 192.168.0.3:9002

This configuration can be confusing; it may seem odd to have a single server in a server pool, but each Adobe Connect RTMP Meeting VIP on the HLD must point to a single Adobe Connect Meeting server; it is a one-to-one correspondence. The HTTP application VIP is more conventional; it points toward a two-server Adobe Connect real-server pool. The HTTP application pool handles failover for the RTMP meetings through the SQL database. Resist any temptation to attempt using a single VIP with multiple open ports instead of the sundry list above. Each VIP needs its own unique fully qualified domain name (FQDN); the configuration above requires three unique FQDNs.

One unique FQDN for the HTTP VIP: connect.adobe.com
One unique FQDN for RTMP VIP: meeting1.adobe.com
One unique FQDN for RTMP VIP: meeting2.adobe.com
One unique FQDN for HTTP VIP: acts.adobe.com

Note: If you wish to avoid putting the RTMP VIPs on the HLD and if your network infrastructure allows client routing directly to the Adobe Connect servers, you may channel the RTMP traffic directly to each server by corresponding the FQDN with the server IP addresses and routing RTMP client traffic around the HLD. The configuration depicted assumes that the HLD is client-facing and the Adobe Connect servers are in a separate subnet that is directly inaccessible by clients except through the HLD. If it is possible to route client traffic around the HLD, then you may place the meeting FQDNs resolving directly on each Adobe Connect server in which case you would eliminate the RTMP VIPs on the HLD and only use the HTTP VIP with a Round Robin pool. Once a meeting is opened, the RTMP threads would simply bypass the HLD while the HTTP threads go though the HLD. The same is true for the ACTS server.

In the Connect server configuration wizard, the external names for each server are the same as the VIP names: meeting1.adobe.com and meeting2.adobe.com respectively; the host name is connect.adobe.com. The only host name suffix the end users will see in the browser URL is connect.adobe.com. Still, unique FQDNs are required on the HLD: one for each VIP pointing to each Adobe Connect meeting/RTMP server, one for both of the Application/HTTP servers and one for the ACTs server if configured. From the perspective of the HLD, there are five real-servers in three pools: two Connect application servers (connect.adobe.com) in one pool, two Connect meeting servers (meeting1.adobe.com and meeting2.adobe.com) each in a pool of its own with its own corresponding VIP and one ACTS server, (acts.adobe.com).

An application-level health monitor on the HLD accelerator should be associated with the HTTP VIP because the Connect application server will handle load balancing and failover of the meetings on the Connect Meeting servers (RTMP) while the HLD handles failover of HTTP. This is why it is appropriate to refer to Adobe Connect multi-server deployments as being both a pool and a cluster.

Make certain not to use session-awareness on the HTTP VIP. The Round Robin load-balancing algorithm should be used without any sticky settings. Do not use an HTTP profile for the RTMP VIPs as it will affect playback of video (as well as remote Connect Edge server connectivity). Remember that you have two servers running on each, a Tomcat application server and an AMS server. Do not treat the AMS server as though it were an application server; RTMP is a streaming protocol that requires a TCP profile at the HLD VIP.

Set any session-timeout variable on the HTTP VIP on the load-balancing hardware to no less than two minutes to prevent freezing of on-demand videos during playback. On-demand video will often cache at the client and then timeout at the VIP do to inactivity at the VIP since the playback is local after the initial download to local client cache.

This configuration employs a single IP address on each Adobe Connect server. The single IP address uses two ports: 80 for the Connect server and 1935 for the Connect Meeting server.

Note also the various configurable ports depicted in use on the Adobe Connect servers:

1935: AMS streaming
8507: Content replication server to server
8506: server to server RTMP traffic
2222: AMG (server not depicted)
1433: SQL
389: LDAP
445: shared storage
8080: ACTS server to server
9002: ACTS

Use of server-based software firewalls, even with port exceptions will prevent fail-over between Adobe Connect servers:

Configuring the Connect server pool

The next step is to properly edit the Connect server settings to match the VIP configuration on the HLD. Select Start > Programs > Adobe Connect Server > Configure Connect Server to open the server settings in your browser (see Figure 2).

Figure 2: Editing the Connect server settings.

The server settings depicted in Figure 2 correspond with this working example. Add the corresponding external names to match the FQDN of the Meeting VIPs on the HLD. meeting1.adobe.com and meeting2.adobe .com.

On each Connect 10.x server, add the following to the custom.ini file in the connect directory to enable fail-over:

MEETING_TIMEOUT=1
MEETING_CONNECT_BACK_TIME=1

Note: For the fail-over settings above, there are pending changes in an upcoming release to improve the logic. If you run into any unexpected results with reference to fail-over in your lab, contact Adobe Connect Customer care and provide the details expounded here: Generating Server-side Logs to Troubleshoot On-premise Connect Deployments

It is prudent to make sure that Connect makes multiple database connection attempts during SQL fail-over. If Connect loses its SQL database, the entire Connect cluster will go down and it will wait for an administrator to manually reconnect to the database through launching the Connect configuration console on port 8510. Add the following to the custom.ini file to . If you are running the Connect SQL database in a SQL cluster rather than in a mirrored environment, these entries will mitigate any delays commensurate with clustered SQL fail-over:

DB_URL_CONNECTION_RETRY_COUNT = 10
DB_URL_CONNECTION_RETRY_DELAY= 30

Note: These fail-over settings are adjustable and may vary depending on the type of content storage implemented. Watch for more details on this topic and for further improvements in fail-over logic. If you see any issues with fail-over testing, contact the Adobe Connect Customer Care team and provide the logs prescribed here:

The actual JDBC string is in the config.ini file so you do not need to put it into the custom.ini; double check the config.ini if you are running into any problems with the JDBC reconnection string:

DB_URL=jdbc:macromedia:sqlserver://{DB_HOST}:{DB_PORT};databaseName={DB_NAME};user={DB_USER};password={DB_PASSWORD};ConnectionRetryCount={DB_URL_CONNECTION_RETRY_COUNT};ConnectionRetryDelay={DB_URL_CONNECTION_RETRY_DELAY}

Note that beginning with Adobe Connect 9, the installer itself includes a cluster option. If you begin with a single server installation and expand later to a clustered environment by adding a server or servers, you may need to manually make the following change in the \appserv\conf\server.xml file in order to enable intra-cluster communication over port 8507. It is prudent to double check this in the server.xml file (in the \appserv\conf\directory) after installing even if the cluster option was selected during installation:

namePrefix=”cluster-8507-” maxThreads=”150″
minSpareThreads=”5″/>

Save the custom.ini and cycle the services. If the HLD is properly configured, you will be able to browse the Connect server pool through the HLD after restarting the Connect and AMS services.

Figure 3: Stop and start the Adobe Connect and AMS services.

Configuring application-level health monitors on the HLD

In order to make sure that the HLD performs fail-over in case one of the Adobe Connect application servers should hang or otherwise become unavailable at the application layer, you will want to make certain that the VIP that points to the application server pool is configured with an application-level health monitor. If you simply probe the health of the Adobe Connect servers with a default health monitor at the level of the IP stack, then there are potential cases when the HLD might send traffic to a server with a non-responsive application that only seems alive to lower-level probing mechanisms such as the packet Internet groper (PING). Always set the health monitor to probe for an actual string of content on the Adobe Connect server; all high-end HLDs offer application-level health monitoring. It may not always be intuitive how to configure the monitor as each HLD has a different interface and different means of probing an application, but the following guidance will help you get an appropriate monitor in place.

Consider that following our example deployment you have multiple server pools and VIPs. The VIP and pool combination that needs an application-level health monitor for fail-over is the Adobe Connect application HTTP server VIP and pool:

HTTP VIP: connect.adobe.com: 10.10.10.1:80 points to Connect servers: 192.168.0.1:80 and 192.168.0.2:80

The probe or health monitor should point to a string on each Adobe Connect server in its pool to check the health of each server. If one of the servers in the pool becomes non-responsive, the monitor will mark the server down and the HLD will redirect all traffic to the remaining server.

The Adobe Connect Meeting server VIP/pool combinations do not need a health monitor because the Adobe Connect application server pool handles fail-over for the Adobe Connect meeting rooms:

RTMP VIP: meeting1.adobe.com: 10.10.10.2:443 points to Adobe Connect Meeting server meeting1 192.168.0.1: 1935
RTMP VIP: meeting2.adobe.com: 10.10.10.3:443 points to Adobe Connect Meeting server meeting2 192.168.0.2: 1935

Because there is only one server in each pool, there is no place for the HLD to redirect meeting traffic should one of the Adobe Connect meeting servers fail to respond. The only reason to probe the Adobe Connect Meeting server VIP/pool combinations might be to trigger an email message to an administrator to warn that one of the Adobe Connect Meeting servers is problematic and that the application pool has triggered fail-over.

The best string on the servers that you may point your application-level health monitor towards is the testbuilder diagnostic page:

/servlet/testbuilder

The testbuilder page will send back the “status-ok” string.

It is best to point the health monitor to the testbuilder page rather than a simple HTML string, because testbuilder is actually probing the Adobe Connect database to make sure there is a healthy connection. If there is any problem with the Adobe Connect server application, then testbuilder will not report the “status-ok” string.

Each HLD has a different interface to configure these monitors and each one does the check differently, the following example works with F5 BIG-IP LTM against testbuilder:

send “GET /servlet/testbuilder HTTP/1.1\nHost: \nConnection: Close \n\r\n”
“status-ok”

For additional information on the testbulder target page, see the following tech-note:

You may also place an HTML file in the Connect /common directory on each Adobe Connect server and point to that file (always test the access to the html file via a browser to be sure that it does not require a login – the common directory and the help subdirectory are both OK with all prior versions of Adobe Connect). This option should be used along with testbuilder as a separate and supporting health check. The following example shows an HTML file called healthmonitortarget.html containing the string You are being served HTML”

send “GET /common/healthmonitortarget.html”

“You are being served HTML”

Note: With reference to the testbuilder file behavior, if, for example, Connect receives a SQL Exception (DB Down, etc.) it does not change testbuilder’s output string, it tries to reconnect a few times, then, if it fails to reconnect, it fails over, and then ultimately restarts the Connect server. If you have the JDBC restart string (described above) in place in the custom.ini, then, in theory, this is desirable behavior. If Adobe Connect were more aggressive in the changing of the testbuilder output string, then it could trigger superfluous fail-over at the application VIP by marking down a server that may only experience a brief re-connection to the Connect DB. If Connect fails and restarts due to a more severe DB connection problem and is still unable to connect once the server restarts, testbuilder will show the following output:

2
status-critical

If you set up the healthmonitor as described above, then what testbuilder may miss, the html health monitor will pick up and vice versa. The key thing is to test the health-monitors vigorously and inspect the Adobe Connect debug logs for any errors they may generate. Since each HLB acts differently and often SSL profiles and other variables will affect behavior, it is prudent to test the health monitors under all server fail-over conditions.

To quickly view the health testbuilder output in production click here: http://platinum.adobeconnect.com/servlet/testbuilder

Conclusion

The need for redundancy and scalability is answered by clustering/pooling application servers. In order to test customized integration with third party services such as the various LMS option in the enterprise, a test lab is prudent if not necessary. Adobe Connect upgrades, updaters and patches should also be tested in a lab environment prior to being placed into production.

Where to go from here

Once you have successfully configured an Adobe Connect server pool to run behind an HLD, you may want to add SSL to secure the environment, an Adobe Experience Manager(AEM) micro-cluster to host rich Adobe Connect Events. You may also wish to integrate telephony via Unified Voice driven by our Flash Media Gateway (AMG) servers or with one of the many telephony adapters. When your team’s staff discover how simple it is to collaborate through Adobe Connect meetings and how Adobe Presenter and Adobe Captivate and Adobe Captivate Prime can convert a plethora of content into fully deployed presentations, courses, and curricula, you may find that further expansion and integration is warranted.

ACTS, Administration, Application, Clustering, Database, General, Install, Meeting

Frank DeRienzo, MBA

PRINCIPAL TECHNICAL ACCOUNT MANAGER

Adobe Connect Support Blog