Connect Layer Troubleshooting

The following section describes the troubleshooting steps you may carry out on the Tomcat server (Connect Layer) in order to attempt to identify where an issue may be occurring. This article is relevant when you are experiencing one or both of the following issues:

  • Unable to log into the App / Web Portal via an authenticate profile (consistently or intermittently)

  • Product Integrations intermittently not responding

Step 1: Identify whether the WSDL is publicly available

To do this you would need to enter the following URL in a web browser:

https://<basic_auth_username>:<basic_auth_password>@<hostname>/<war_filename>/services/CampusMUniversityService?wsdl

The basic authentication credentials can be located on the Tomcat server at $TOMCAT_HOME/conf/tomcat-users.xml

Alternatively, you may also gather basic auth details via the App Manager: App Manager > App Settings > Application Servers > (Select relevant server e.g. Production / Live).

Step 2: Check the SSL certificate

Identify whether there are any issues with the SSL certificate installation on the Tomcat server by using the website below:

https://www.sslshopper.com/ssl-checker.html

Enter WSDL URL into sslshopper: https://<hostname>/<war_filename>/services/CampusMUniversityService?wsdl

Step 3: Check Connect Layer

On the Connect Layer server, identify whether Tomcat is up and running by using the following command:

ps –ef | grep tomcat

If the output of this command contains the whole path to the tomcat folder then it is running.

Alternatively, depending on if Tomcat has been configured to run as a service, you can use the following command:

service tomcat status

Step 4: Identify whether the WSDL is accessible locally

On the server itself, run the following curl command (or enter into a browser if you have a desktop session to the server):

http://<basic_auth_username>:<basic_auth_password>@<hostname>:<port_8080_depends_on_setup>/<war_file_name>/services/CampusMUniversityService?wsdl

If you are able to see a WSDL response after running the above URL but not able to access the WSDL publicly please check your load balancer / firewall configuration has not changed.

Step 5: Check the Tomcat logs

Check the catalina.out and localhost logs for the following (errors which would indicate a system down):

  • java.io.FileNotFoundException Too many open files

  • Exceeded MaxThreads exception

Step 6: Identify whether the connections have built up on the Connect layer

On the server, run the following command:

netstat -a

If you suspect a specific data source hostname is causing the build-up you can filter traffic using the following command:

netstat -a | grep ‘hostname’

Step 7: Diagnose which services are responding slowly

Run manual curl requests for the pertinent integrations to determine whether the API or database connections are responding slowly.

See Creating a Curl Requests for Testing for more information.

Step 8: Check ALL Tomcat nodes behind the load balancer

If you are experiencing intermittent outages this could be pointing to an issue with one or more of your Connect layer nodes not responding correctly. Please repeat the checks from from Step 3 onwards to diagnose if a particular tomcat node is at fault.

Firewall/Load Balancer

Please inform us beforehand if you are going to make any changes to your networking and firewall rules. If the changes have been misconfigured, it may result in the application becoming unresponsive. For example, there may be an issue with routing between the load balancer and Tomcat server or between the Tomcat server and the backend systems with which campusM communicates and integrates.