Explanation:
Monitoring Synchronization (fw ctl pstat)
To monitor the synchronization mechanism on ClusterXL or third-party OPSEC certified clustering products, run the following command on a cluster member: The output of this command is a long list of statistics for the VPN-1 Pro Gateway. At the end of the list there is a section called “Synchronization” that applies per Gateway Cluster member. Many of the statistics are counters that can only increase. A typical output is as follows:
The meaning of each line in this printout is explained below.
This line must appear if synchronization is configured. It indicates that new sync is working (as opposed to old sync from version 4.1).
If sync is unable to either send or receive packets, there is a problem. Sync may be temporarily unable to send or receive packets during boot, but this should not happen during normal operation. When performing full sync, sync packet reception may be interrupted.
fw ctl pstat
Version: new
Status: Able to Send/Receive sync packets
Sync packets sent:
total : 3976, retransmitted : 0, retrans reqs : 58, acks : 97
Sync packets received:
total : 4290, were queued : 58, dropped by net : 47
retrans reqs : 0, received 0 acks
retrans reqs for illegal seq : 0
Callback statistics: handled 3 cb, average delay : 1, max delay : 2
Delta Sync memory usage: currently using XX KB mem
Callback statistics: handled 322 cb, average delay : 2, max delay : 8
Number of Pending packets currently held: 1
Packets released due to timeout: 18
Version: new
Status: Able to Send/Receive sync packets
Sync packets sent:
total : 3976, retransmitted : 0, retrans reqs : 58, acks : 97
Monitoring Synchronization (fw ctl pstat) 90
The total number of sync packets sent is shown. Note that the total number of sync packets is non-zero and increasing.
The cluster member sends a retransmission request when a sync packet is received out of order. This number may increase when under load.
Acks are the acknowledgements sent for received sync packets, when an acknowledgement was requested by another cluster member.
The total number of sync packets received is shown. The queued packets figure increases when a sync packet is received that complies with one of the following conditions:
1 The sync packet is received with a sequence number that does not follow the previously processed sync packet.
2 The sync packet is fragmented. This is done to solve MTU restrictions.
This figure never decreases. A non-zero value does not indicate a problem.
The dropped by net number may indicate network congestion. This number may increase slowly under load. If this number increases too fast, a networking error may interfere with the sync protocol. In that case, check the network.
This message refers to the number of received retransmission requests, in contrast to the transmitted retransmission requests in the section above. When this number grows very fast, it may indicate that the load on the machine is becoming too high for sync to handle.
Acks refer to the number of acknowledgements received for the “cb request” sync packets, which are sync packets with requests for acknowledgments.
Retrans reqs for illegal seq displays the number of retransmission requests for packets which are no longer in this member’s possession. This may indicate a sync problem.
Callback statistics relate to received packets that involve Flush and Ack. This statistic only appears for a non-zero value.
Sync packets received:
total : 4290, were queued : 58, dropped by net : 47
retrans reqs : 0, received 0 acks
retrans reqs for illegal seq : 0
Callback statistics: handled 3 cb, average delay : 1, max delay : 2
Starting the Cluster Member
Chapter 6 Monitoring and Troubleshooting Gateway Clusters 91
The callback average delay is how much the packet was delayed in this member until it was released when the member received an ACK from all the other members. The delay happens because packets are held until all other cluster members have acknowledged reception of that sync packet.
This figure is measured in terms of numbers of packets. Normally this number should be small (~1-5). Larger numbers may indicate an overload of sync traffic, which causes connections that require sync acknowledgements to suffer slight latency.
In a heavily loaded system, the cluster member may drop synchronization updates sent from another cluster member.
Delta Sync memory usage only appears for a non-zero value. Delta sync requires requires memory only while full sync is occurring. Full sync happens when the system goes up- after reboot for example. At other times, Delta sync requires no memory because Delta sync updates are applied immediately. For information about Delta sync Number of Pending packets currently held only appears for a non-zero value.
ClusterXL prevents out-of-state packets in non-sticky connections. It does this by holding packets until a SYN-ACK is received from all other active cluster members. If for some reason a SYNACK is not received, VPN-1 Pro on the cluster member will not release the packet, and the connection will not be established.
Packets released due to timeout only appears for a non-zero value. If the Number of Pending Packets is large (more than 100 pending packets), and the number of Packets released due to timeout is small, you should take action to reduce the number of pending packets.
dropped updates as a result of sync overload: 0
Delta Sync memory usage: currently using XX KB mem
Number of Pending packets currently held: 1
Packets released due to timeout: 18
Reference:
R60 ClusterXL.pdf page 90

---