Make Certain that Content is Replicated Across All Servers in a Connect Cluster

Occasionally a specific piece of content may be intermittently available in a cluster. It could be Presenter or Captivate published on-demand content or even content within a Meeting room. Sometimes in these cases, the content published on one server is not replicated to all servers in the cluster. There are a few quick things to check:

First: Note that beginning with Adobe Connect 9, the installer includes a cluster option. If you begin with a single server installation and expand later to a clustered environment by adding a server or servers, you will need to manually make the following change in the /appserv/conf/server.xml file in order to enable communication over port 8507 among clustered servers. It is prudent to double check this in the server.xml file after installing even if the cluster option was selected during installation. Uncomment the following xml code:

<Executor name=”clusterThreadPool”
namePrefix=”cluster-8507-” maxThreads=”150″
minSpareThreads=”5″/>

<!– Define a non-SSL HTTP/1.1 Connector on port 8507 –>
<!– Used for HTTP access for intra-Cluster communications. –>
<!– Equivalent to JRun CLUSTER_PORT –>
<!– Uncomment for clustered deployments
<Connector port=”8507″ protocol=”HTTP/1.1″
executor=”clusterThreadPool”
enableLookups=”false”
acceptCount=”100″
connectionTimeout=”20000″
URIEncoding=”utf-8″/>

Second: Test the 8507 port communications on each server: From a command prompt on each server, type netstat –an|find “8507” and check to be sure that 8507 is active and listening on each.

netstat -an|find “8507”

Use telnet to test connectivity on  8507 between Connect servers. Use telnet to check both IP and machine-name as well.

telnet server-machine-name 8507

Note: The machine name appears to the left of the FQDN under the Connect Servers Setting on port 8510 locally on any server in the cluster; here I have artificially designated them as server1 and server2.

Be sure to check telnet connectivity from and to every server in the cluster:

If the IP works with telnet and the machine-name does not work, it may be necessary to add entries in DNS or add hosts files to each server:

Check the software-based firewall on the server to see if it is potentially blocking replication traffic:

netsh firewall show config

Note: Connect does not support dual stack ipv6 and ipv4 on the same server.

Note: If problems are noticed in the Meeting rooms, check port 8506; it is used for Meeting communication among the servers.

Third: Examine the Connect logs: Look first in the debug.log under the \logs\support directory and search on the string: cluster-  If replication is taking place, you will see this repeating cluster- entry logging the replication activity. Absence of these log entries will indicate that replication is not working:

[10-1 12:00:00,009] cluster-8507-630 (INFO) CLUSTER Sent file: \7\xxx-xxxx\fcs-meeting\public\all\224_XXX_4.fso 9978 bytes 12 ms 6371 kbps to: server1

Check for any error messages in these replication log entries. Search also for the word lucene. If you see a preponderance of lucene lock errors, contact Adobe Enterprise Support: entrsupp@adobe.com and provide a log snippet to expedite diagnosis.

Also check the error.log files for the entry  CLUSTER_CON_BROKEN

2014-10-02 15:28:48 “Server server1 unable to reach server2 on port 8507 to perform cluster operations.” CLUSTER  CLUSTER_CON_BROKEN

Check the Catalina logs for references to port 8507:

[04-26 09:57:49,379] cluster-8507-1 (INFO) The host [CONNECT_PRIME4:8507] is not valid
Note: further occurrences of request parsing errors will be logged at DEBUG level.
java.lang.IllegalArgumentException: The character [_] is never valid in a domain name.
                at org.apache.tomcat.util.http.parser.HttpParser$DomainParseState.next(HttpParser.java:926) ~[tomcat-coyote.jar:9.0.10]
                at org.apache.tomcat.util.http.parser.HttpParser.readHostDomainName(HttpParser.java:822) ~[tomcat-coyote.jar:9.0.10]

An underscore in a domain name will cause this error in Adobe Connect version 10. See: https://stackoverflow.com/questions/53504857/the-character-is-never-valid-in-a-domain-name

Tomcat 9, which we moved to in Connect 10 for security reasons has started enforcing strict checks for host names in accordance with specifications (RFCs 952 and 1123). The error message is a bit misleading because it talks about domain names, but check your host names or machine names (the restriction for domain names is a bit looser). This is a Tomcat security restriction on characters allowed in domain names. It may be necessary to drop special characters from the host name if you see this error. Choose only alphanumeric strings; there appears to be no other workaround since this is a security change.

For more information on searching server-side logs see: http://blogs.connectusers.com/connectsupport/generating-server-side-logs-to-troubleshoot-on-premise-connect-installations/

Fourth: Check the timing of active anti-virus scanning of the content directories \content\7\ on each server; compare the directory sizes on each server to see is there if a significant size delta. Antivirus software can impede replication in manner that is not uniform across servers; active scanning of the content directory during replication may lock the content files. Active scanning after hours or during a window when publishing is unlikely is prudent.

Fifth: Check the updater page. Make sure you are on the latest patches servers-side. For full installers, use LWS:  https://licensing.adobe.com

These steps will solve most replication problems that you encounter. If problems persist, contact our  Enterprise Support Team.