Solved: How do I setup and troubleshoot FortiManager HA cluster？

The below article describes how to configure and troubleshoot a FortiManager High-availability (HA) cluster in Manual and VRRP mode.

Notes on HA modes for FortiManager

Must be the same between the Primary and all other nodes of the cluster:

• FortiManager type of machine
• FortiManager firmware version
• A Layer-2 connection between Primary- FortiManager and Secondary- FortiManager is mandatory to communicate through Cluster Virtual IP via VRRP.
• If Primary- FortiManager and Secondary- FortiManager are in different locations, then connected via MPLS link.
• FortiManager HA is using VRRP (As of 7.2 version) for the floating IP of the cluster members.

Note: Virtual IP should be the same in both Primary and Secondary devices. (VRRP mode)

Sample Diagram with Port and IP Configuration for HA VRRP Mode:

FortiManager HA settings

Below the HA settings of FortiManager HA cluster and their meanings:

• Failover Mode: <Manual> or <VRRP> (VRRP or automatic HA failover mode will be covered later in this document)
• Operation Mode: <Primary> or <Secondary>
• Cluster Settings:

• Peer IP: <Secondary FortiManager IP address>
• Peer SN: <FMGVMXXXXXX> (Secondary FortiManager Serial Number)

The below HA settings must be the same on Primary & Secondary node:

• Cluster ID: Any number (1-255) can be given.
• Group Password: <password> Can give any password.
• File quota: 4096
• Heartbeat Interval: <Interval_Integer>

The time the primary unit waits between sending heartbeat packets, in seconds. The heartbeat interval is also the amount of time that backup unit waits before expecting to receive a heartbeat packet from the primary unit.

The default heartbeat interval is 5 seconds. The heartbeat interval range is 1 to 255 seconds. Users cannot configure the heartbeat interval on the backup units.

• Failover Threshold: <Failover_Interger>

The number of heartbeat intervals that one of the cluster units waits to receive HA heartbeat packets from other cluster units before assuming that the other cluster units have failed.

The default failover threshold is 3. The failover threshold range is 1 to 255. Users cannot configure the failover threshold of the backup units.

These below settings can only be configured when the Failover Mode is VRRP:

• VIP: <Virtual IP address>

This setting can only be configured when the Failover Mode is VRRP. (Make sure this IP is not used in the network)

• VRRP Interface: <port>
• Priority: <1-253>

Set the priority for this device between 1 (lowest) and 253 (highest). The device with a higher priority will operate as the primary unit when possible.

• Unicast

Optionally, toggle this setting ON to use Unicast for the VRRP message.

• Monitored IP

Configure the monitored IP and interface. Users can add additional monitored IPs by selecting the add icon.

Configuration FortiManager HA cluster (Manual mode)

On FortiManager-Primary device:

System Settings > HA > Operation mode select Manual > Primary

Configure the following details:

• Failover Mode: Manual
• Operation Mode: Primary
• Peer IP and Peer SN:
  • Peer IP: x.x.x.17
  • Peer SN: FMGVMTMxxxxxx8

The below HA settings must be the same on Primary and Secondary node:

• Cluster ID: 21
• Group Password: Fortinet@Test
• File quota: 4096
• Heartbeat Interval: 10
• Failover Threshold: 30

On FortiManager-Secondary device:

System Settings > HA > Operation mode select Manual > Secondary

Configure the following details:

• Failover Mode: Manual
• Operation Mode: Secondary
• Peer IP and Peer SN:
  • Peer IP: x.x.x.19
  • Peer SN: FMGVMTMxxxxxx4

The below HA settings must be the same on Primary & Secondary node:

• Cluster ID: 21
• Group Password: Fortinet@Test
• File quota: 4096
• Heartbeat Interval: 10
• Failover Threshold: 30

After configuring Primary & Secondary node of the FortiManager HA cluster green arrows should appear on GUI (Synchronization status).

On FortiManager Primary node:

On FortiManager Secondary node:

FortiManager-HA automatic failover – VRRP mode

VRRP (Virtual Router Redundancy Protocol) is a protocol intended to increase the availability of the default gateway for hosts on the same network.

The goal is to define the default gateway for network hosts as a virtual IP address referencing a group of routers.

Not only a unique IP address will be declared on each machine (Primary and Secondary IP addresses), but also a virtual IP address shared between each FortiManager (VIP) nodes.

The aim of this address is to insure the VRRP availability.

• Source address: The primary IP address of the interface the packet is being sent from.
• Destination IP address: 224.0.0.18 (Multicast IP address).

This is a link local scope multicast address. Routers should not forward a datagram with this destination address regardless of its ttl.

• TTL: 255
• Protocol: The IP protocol number assigned by the IANA for VRRP is 112 (decimal)
• MAC Address in the following format: 00-00-5E-00-01-{VRID}

Into the below packet capture the multicast IP address and its mac address in the following format can be viewed: 00-00-5E-00-01 {VRID}: 01:00:5e:00:12

How does VRRP work?

The FortiManager that gets the highest priority is elected as Primary.

The end user only knows the VIP.

When a node of the FortiManager cluster becomes down, a gratuitous ARP (preload the ARP tables of all other local hosts) request is sent by the FortiManager backup to get the Virtual IP address.

The High-Availability principle then is respected.

At that moment, the active FortiManager node gets the Primary role.

However, when the FortiManager node becomes again available it takes the Secondary role of the cluster even though the ID cluster is higher because the older age value is considered by the VRRP protocol.

Rebooting a FortiManager unit updates the HA roles (Primary/Secondary).

To use automatic failover for FortiManager-HA

Step 1: In FortiManager, go to System Settings > HA.

As of 7.2 version, a new Failover Mode setting is available in the FortiManager HA configuration menu. One can select Manual for manual failover or VRRP to enable automatic failover.

Step 2: Select VRRP as the Failover Mode, and configure the other settings required including the VIP, VRRP Interface, Priority, Unicast, and Monitored IP.

On Primary FortiManager:

On Secondary FortiManager:

Step 3: When the monitored interface for the Primary FortiManager is unreachable or down, HA automatic failover will occur, and the Secondary FortiManager will automatically become the primary.

To configure automatic failover in the FortiManager CLI

Step 1: On the Primary FortiManager, configure the FortiManager settings with VRRP mode selected:

# config system ha
set failover-mode vrrp
set mode primary
config monitored-ips
edit 1
set interface
set ip
next
end
config peer
edit
set ip
set serial-number
next
end
set priority
set vip
set vrrp-interface
end

For example:

# config system ha
set failover-mode vrrp
set mode primary
config monitored-ips
edit 1
set interface "port2"
set ip "192.168.48.63"
next
end
config peer
edit 1
set ip 10.3.106.64
set serial-number "FMG-VM0A1700xxxx"
next
end
set priority 200
set vip "10.3.106.253"
set vrrp-interface "port1"
end

Step 2: On the Secondary FortiManager, configure the FortiManager settings with VRRP mode selected:

# config system ha
set failover-mode vrrp
set mode secondary
config monitored-ips
edit
set interface
set ip
next
end
config peer
edit
set ip
set serial-number
next
end
set priority
set vip
set vrrp-interface
end

For example:

# config system ha
set failover-mode vrrp
set mode secondary
config monitored-ips
edit 1
set interface "port2"
set ip "192.168.48.64"
next
end
config peer
edit 1
set ip 10.3.106.63
set serial-number "FMG-VM0A1600xxxx"
next
end
set priority 1
set vip "10.3.106.253"
set vrrp-interface "port1"
end

Troubleshoot commands

On Primary-FortiManager:

With these 2 commands, troubleshoot the status and the configured values for FortiManager-HA

# diag ha stats

# get system ha

On Secondary-FortiManager:

With these 2 commands, troubleshoot the status and the configured values for FortiManager-HA.

# diag ha stats

# get system ha

The below CLI commands help to troubleshoot FortiManager HA cluster issues:

# diagnose debug application ha -1
# diagnose debug enable

The debug output above can also be downloaded in a txt file:

For example:

The debugging results during the HA cluster build operation can also be seen through CLI.

The below capture shows the negotiation of the FortiManager HA cluster from the primary node.

The same operation can also be performed on the secondary node.

VRRP/Manual synchronization debug results

Below examples of output results of the HA cluster synchronization in VRRP mode on FortiManager.

These debugs are similar for HA configured in ‘manual mode’.

VRRP debugs on Primary node:

VRRP debugs on Secondary node:

Once the HA cluster is built and synchronized, keepalive messages are sent between both nodes on the port chosen for this purpose:

On FortiGate central management, two Serial Numbers are configured (Both SN’s belong to each FortiManager nodes of the cluster).

Both HA manual and VRRP modes use two Serial number.

If a node of the cluster becomes unavailable, the other SN will be automatically used.

Any Serial number will be removed from the configuration of the FortiGate central management in that case.