Skip to Content

How to identify and troubleshoot hard disk / log disk failure with FortiWeb

By: Author Alex Lim

Posted on

Categories Troubleshooting

This article describes how to identify if there is a hard disk / log disk failure and how to troubleshoot it.

Scope

FortiWeb.

Solution

Expected Issues: The following issues may occur if FortiWeb has a disk failure:

  • FortiWeb is not logging/writing any Attack, Event, Traffic logs.
  • FortiWeb cannot boot up And/or stuck in a boot up loop.
  • FortiWeb is not processing the connections and is causing a complete or partial outage.
  • FortiWeb is unable to execute various GUI actions with the error ‘Request aborted. Please check your network connection.’

Follow this guide to verify that the above issues are caused by disk failure and not any other security module, feature, daemon, or bug. there are two scenarios where hard disk issues may be faced:

Scenario 1: FortiWeb is accessible via [GUI,SSH,CLI]

Step 1: Execute the following command:

get system status
International Version: FortiWeb-VM 7.4.3,build0638(GA),240405
Serial-Number: FVVM01TM22000224
license type: remote
Bios version: 04000002
Log hard disk: Not Available
Hostname: lister-esx04
Operation Mode: Reverse Proxy
FIPS-CC mode: disabled
System Uptime: [51 day(s) 23 hour(s) 1 min(s)]
Current HA mode: standalone
Database Status: Not Available

Note 1: If the Database Status is ‘Not Available’ but the Log hard disk status is ‘Available’, it is likely to be a DB issue. Refer to this article for steps on how to fix an issue where the DB is not available.

Note 2: If the status of both the Log Hard disk and also the Database is ‘Not available’, proceed with Step 2 below.

Step 2: Execute the following command:

diagnose system mount list
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/ram0 473 311 161 65% /
none 569 0 569 0% /tmp
none 1897 3 1895 0% /dev/shm
/dev/sdb1 362 265 78 77% /data
/dev/sdb3 91 0 86 0% /home
none 200 200 0 100% /var/log ==============> 100% Use 0% Available

Note: Here, the /var/log has 0% availability and the size of the partition is only 200 MB.

Step 3: Execute the following command:

diagnose hardware check all
************************************************
CPU check Pass
core-number Pass 2
cpu-number Pass 1
frequence Pass 3700
cache-size Pass 20480
model-name Pass Intel(R) Core(TM) i3-4360 CPU @ 3.70GHz

************************************************
************************************************
Memory check Pass
Total-size Pass 8131324
frequence Pass 1600
************************************************
************************************************
logdisk check Fail
size Fail 234
disk-number Fail 1
************************************************
************************************************
NIC check Pass
num Pass 8
Giga nic num Pass 8
10G nic num Pass 0
************************************************

Note: The status of the logdisk check is ‘Fail’, along with the size and disk-number.

Step 4: Execute the following command:

diagnose hardware check logdisk
logdisk check Fail
size Fail 15
disk-number Fail 1
raid-level Fail no raid exists

Note: The status of the logdisk check is ‘Fail’.

Scenario 2: FortiWeb is not accessible via the GUI, SSH, or the CLI and is stuck in the boot up process

Any of the following errors may be seen in the boot up log output:

/dev/sda1: recovering journal
/bin/e2fsck: No such device or address while trying to open /dev/sdb3
---------------------------------------
write error: failed to open new log file for writing: failed to open new file '/var/log/filebeat/filebeat-20240518.ndjson': open /var/log/
FWB login: find: /var/log/debug/tmp/BELGFWB01_230516_1449.nmon: Read-only file system
---------------------------------------
FWB $ [1194099.855910] blk_update_request: I/O error, dev sdb, sector 3241911 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[1194101.266938] blk_update_request: I/O error, dev sdb, sector 15828359 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[1194101.554921] blk_update_request: I/O error, dev sdb, sector 3241911 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[1194101.772904] blk_update_request: I/O error, dev sdb, sector 12164655 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[1194101.933935] blk_update_request: I/O error, dev sdb, sector 4538503 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
---------------------------------------
FWB login: mln sample list not ready.
[tree: error]<redisconn.c:158> Redis server is not ready, will try redis auth again later ...
[tree: error]<redisconn.c:158> Redis server is not ready, will try redis auth again later ...
---------------------------------------

How to Fix a hard disk/ log disk failure

Scenario 1: FortiWeb is accessible via the GUI, SSH, and/or the CLI

Rebuild the DB and format the log disk (doing so during a maintenance window is recommended as formatting the logdisk will trigger a reboot:(

execute db rebuild
execute formatlogdisk

Note 1: Formatting the log disk will perform a reboot and will erase the current logs. The config file will not be impacted. To create a backup of the current logs, either:

  • Use a FortiAnalyzer or similar device.
  • Download the logs locally on the client machine:

Log&Report -> Log Access -> Attack: Right-click the [Log management] icon at the top right of the page and select download.
Log&Report -> Log Access -> Event: Right-click the [Log management] icon at the top right of the page and select download.
Log&Report -> Log Access -> Traffic: Right-click the [Log management] icon at the top right of the page and select download.

Note 2: After the reboot, check the status of ‘Log hard disk’ (as mentioned in step 1 of the verification section). If the status remains as ‘Not available’, proceed with the following ‘clean installation’ step, which should also be performed if the FortiWeb GUI and CLI are not accessible.

Scenario 2: FortiWeb is not accessible via the GUI, SSH, or the CLI, is stuck in the boot up process, or the previous step ‘DB rebuild, Format logdisk’, did not fix the issue

Perform a clean install:

  1. Upload the FortiWeb image to a TFTP server on the client machine.
  2. Connect the management computer to the FortiWeb console port using a RJ-45-to-DB-9 serial cable or a null-modem cable and Initiate the console connection.
  3. Connect port1 of the FortiWeb appliance directly or to the same subnet as a TFTP server ‘Step 1’.
  4. Verify that the TFTP server is currently running, and that the FortiWeb appliance can reach the TFTP server: 
    execute ping 192.0.2.168

Where 192.0.2.168 is the IP address of the TFTP server.
  5. Execute reboot.
  6. As the FortiWeb appliance starts, a series of system startup messages appear. Press any key to display the configuration menu.
  7. Enter ‘F’ to Format the boot device.
  8. Enter ‘G’ to retrieve the image from the TFTP server.

Check the troubleshooting guide for further details on how to restore firmware (‘clean install’):

Note: If the issue persisted after the previous steps, it is most likely a hardware related issue. Collect the following debug command outputs and open a support ticket:

Scenario 1: FortiWeb is accessible via the GUI, SSH, and/or the CLI

get system status
get system performance
diagnose system mount list
diagnose debug crashlog show
diagnose debug coredumplog show
diagnose hardware check all
diag hardware logdisk info
diag hardware harddisk list
diag hardware harddisk errors
fn dmesg

Scenario 2: FortiWeb is not accessible via the GUI, SSH, and/or the CLI

Collect the bootup outputs.