You're viewing Apigee and Apigee hybrid documentation.
There is no equivalent
Apigee Edge documentation for this topic.
Symptom
Cassandra heap issues may cause slowness in the Apigee hybrid proxy
execution or even Datastore errors. Sometimes logs are an early
indicator, even before the onset of symptoms.
Error Message
In Cassandra pod logs (Cloud Logging), log entries might be observed that are similar to the following:
WARN [Service Thread] 2023-01-01 01:14:51,121 GCInspector.java:283 - G1 Young Generation GC in 2510ms... ... WARN [Service Thread] 2023-01-01 01:14:51,121 GCInspector.java:283 - G1 Old Generation GC in 3100ms...
2023-01-01 01:14:51,123 FailureDetector.java:278 - Not marking nodes down due to local pause of 45261214670 > 5000000000
java.lang.OutOfMemoryError: Java heap space
Cause: Insufficient Java heap size
Diagnosis
In the overrides.yaml file for the Apigee hybrid
installation, maxHeapSize is not set to a sufficient value.
This could be due to
production settings not applied, or current throughputs requiring a
setting higher than usual.
Resolution
To resolve the issue, adjust maxHeapSize and memory
accordingly, and
apply the changes:
resources:
requests:
cpu: 7
memory: 15Gi
maxHeapSize: 8192M
heapNewSize: 1200M
If the issue persists with the default production configuration, try to increase the values further. Note that you need to ensure that node capacity, disk throughput and network bandwidth are sufficient as well.
resources:
requests:
cpu: 7
memory: 30Gi
maxHeapSize: 16384M
heapNewSize: 2400M
If the 16 GB heap setting is still not sufficient for the traffic volume, continue with horizontal scaling.
Must gather diagnostic information
If the problem persists even after following the above instructions, gather the following diagnostic information and then contact Google Cloud Customer Care:
Apart from the usual data you might be asked to provide, collect the diagnostic data from all the Cassandra pods with the command below:
for p in $(kubectl -n apigee get pods -l app=apigee-cassandra --no-headers -o custom-columns=":metadata.name") ; do \
for com in info describecluster failuredetector version status ring info gossipinfo compactionstats tpstats netstats cfstats proxyhistograms gcstats ; do kubectl \
-n apigee exec ${p} -- bash -c 'nodetool -u $APIGEE_JMX_USER -pw $APIGEE_JMX_PASSWORD '"$com"' 2>&1 '\
| tee /tmp/k_cassandra_nodetool_${com}_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt | head -n 40 ; echo '...' ; done; done
Compress it, and provide it in the Support case:
tar -cvzf /tmp/cassandra_data_$(date +%Y.%m.%d_%H.%M.%S).tar.gz /tmp/k_cassandra_nodetool*