Generating Server-side Logs to Troubleshoot On-premise Adobe Connect Deployments

In order to diagnose unexpected behavior or performance issues within Adobe Connect, it may be necessary for the Adobe Connect Support team to examine server-side logs from an on-premise Adobe Connect deployment. The logs directory is located in the Connect (or legacy Breeze) directory:

Within the logs directory there are sub-directories containing various logs:

The most commonly requested log by the support team, is the debug.log. It can be found in the logs>support folder. With the services running, the current debug.log file will appear without a date at the top of the debug.log file list. The default rollover is 12 hours generating AM and PM logs each day:

In order to make the debug.log file more useful for purposes of diagnosis, you can enable verbose logging by adding entries to the custom.ini file located in the Connect or Breeze version sub-directory. Here you see it located in a directory under the Connect/Breeze root installation/upgrade directory:

Before editing the custom.ini file, be sure to create a backup copy of it. Add the following lines in order to enable verbose logging:

HTTP_TRACE=true
DB_LOG_ALL_QUERIES=true

Note that for versions of Connect 9.2 and prior, the variable used was yes instead of true; if you have upgraded, you may need to manually edit this from yes to true:

HTTP_TRACE=yes
DB_LOG_ALL_QUERIES=yes

Save the custom.ini file (be careful not to accidentally change the file type to .txt) and during a scheduled maintenance window, cycle the Adobe Connect and AMS/FMS services in order to load the changes and begin verbose logging (note this will bring Adobe Connect down while the services cycle):

There are occasions when it may be prudent to provide more than one log for a more complete diagnosis. To provide a full sample of the various Connect logs without sending a massive historical sample of log files, you may simply stop the Adobe Connect and AMS and telephony services (during scheduled downtime as this will bring down Adobe Connect) and rename the entire log directory to log.old. Then upon starting the services back up, a new log folder structure will be populated; now you may recreate the issue being diagnosed and then stop the services again to get the log directory now truncated and focused on the issue of concern just replicated.

This activity generates a new small log directory isolating the issue under scrutiny that you just reproduced in Adobe Connect: Zip/compress this new abbreviated log directory with all its fresh abbreviated sub-directories and provide it to the the Adobe Connect Support team to help expedite more exhaustive server-side log analysis.

This option is particularly helpful when examining a cluster as each server will have a set of logs. When providing cluster logs, always label each compressed log folder to easily identify the server from which it came.

Note that often when diagnosing unexpected behavior in Adobe Connect Meetings, it may also be prudent to enable client-side Connect application/addin verbose logging as well.  The relevant client-side logging tech-notes are here:

Enable logging | Meeting Add-in

Troubleshooting Verbose Meeting Addin Logging

To diagnose issues with a specific Meeting, comparing the client-side verbose application/addin log with the server-side Meeting application logs is often helpful. The server-side logs for a specific Meeting are listed by the Meeting sco in the logs: logs\support\access\_defaultVHost_\meetingas3app\7\1234567(meeting-sco) or (depending on the version) logs\support\apps\_defaultVHost_\meetingas3app\7\1234567(meeting-sco)

To identify the appropriate Meeting sco ID, simply append ?mode=xml to any Meeting URL and scroll down on the page to the xml output.

Other helpful logs may include: access.log, breeze.log, error.log and gc-123456.log. These are all found in the root log folder. Simply choose the appropriate date of the occurrence under scrutiny.

In logs\support\diagnostic, the appropriate admin.log, core.log, edge.log and master.log may be helpful.

In logs\support\access, there are additional access logs that may be helpful.

Note: In some cases, additional logging may be warranted by adding verbose settings to the log4j2.xml in appserv/conf

On lines line 246 and 252, change “info” to level=”debug” –  lower case:

<!-- Standard loggers -->
<Logger name="com.macromedia.fcs" additivity="false" level="info">
<AppenderRef ref="DEBUGLOG" />
</Logger>
<Logger name="com.macromedia" additivity="false" level="debug">
<AppenderRef ref="DEBUGLOG" />
</Logger>
<Logger name="com.presidia" additivity="false" level="info">
<AppenderRef ref="DEBUGLOG" />
</Logger>
<Logger name="com.adobe" additivity="false" level="debug">
<AppenderRef ref="DEBUGLOG" />
</Logger>
<Logger name="com.adobe.connectpro.components" additivity="false" level="info">
<AppenderRef ref="DEBUGLOG" />
</Logger>

By enabling Java network debugging, you can diagnose how or when TLS negotiation fails with the SQL connection between CPS and the database. What may happen is that even if the server supports TLS 1.2, the client may not have all the needed ciphers, so it chooses TLS 1.0 or 1.1 and the server refuses that. the client is Connect and the server is SQL Server. In some cases, after Adobe Connect version 11.4, what may be failing is the negotiation between the Connect and SQL Server. The ciphers on the Java side are pretty clear and fixed, it is a question of whether the ciphers on the SQL server allow for the negotiation to succeed.

The ConnectProSvc.conf file is file to edit for JVM verbose settings; be careful to edit the correct one. There is one the installer uses and one that the Connect server uses:

• Edit \Connect\11.2.0\appserv\conf\ConnectProSvc.conf
• Do not edit: :\Connect\11.2.0\appserv\service-wrapper\64bit\conf\ConnectProSvc.conf

Add this to the wrapper service config, restart the service:

wrapper.java.additional.400=-Djavax.net.debug=all

The ConnectProSvc.log files will have details on where and when the negotiation fails.