HDD validation basically involves 4 stages for each physical disk (HDD or SSD):
- A SMART short test. This is a test that looks at certain aspects of the electrical and mechanical performance of the HDD. It is not a thorough test of the HDD. The tests take somewhere in the region of 2-5 minutes to complete.
- A SMART conveyance test. This is a test performed on HDDs to check if they have survived transit without any damage. (I don’t know how they differ from the short or long test if someone wants to give me the information Fester will try to add it.)
- A Badblocks test. This is a test where every physical location on the HDD has a write/read test performed on it. The test consists of a block of data that gets written to every physical location on the HDD in sequence. Every physical location on the HDD is then read back also in sequence and each time at each location the value is checked to see if it is correct. This is one pass. The whole process if repeated with a different block of data, this is the second pass. The badblocks test uses 4 patterns by default. This test will take a very long time, usually between 24 hours to a few days depending on the capacity of the drive.
Note 1: This test overwrites any data on the drive - do NOT run this test unless you are sure that the drive is empty of data.
Note 2: SSDs support a finite volume of writes, so be aware that running this test will use a small proportion of these. - A SMART long test. Think of this as an extended version of the short test. It is a much more thorough test of the HDD and will include a surface scan of the drive. This test will take many hours to complete depending on the capacity of the HDD.
¶ SMART Tests: A Short Introduction
SMART stands for Self-Monitoring Analysis and Reporting Technology.
A SMART test is a test a HDD or SDD can perform by itself on itself. These tests, often referred to as “self tests” are carried out by the HDD’s/SDD’s onboard firmware, not a separate piece of software running on the server as we have already seen.
The results of these tests are stored in the drives onboard non-volatile memory so they can be retrieved and utilised by simply interrogating the drive in the correct way.
However, to be able to use the SMART capabilities built into the drives we need a program or an OS that is capable of communicating with the built in SMART functions of the drive.
With such a program or OS present we can simply issue commands to invoke the firmware to initiate a SMART test and/or interrogate a SMART drive to obtain the results of that test (very convenient).
Only one SMART test can be performed per drive. So you cannot run the short test and the long test on the same drive simultaneously. Also the current SMART test must complete before another can be run on the same drive. If a SMART test is running on a drive and you start another then the current test is stopped and abandoned in favour of the newly requested test.
Fortunately, you can run SMART tests in parallel on different drives. So you could have any number of drives all performing the short test at the same time, or the long test or a mix if you wish (i.e. some performing the short test and some performing the long test).
¶ HDD/SSD Validation Via the TrueNAS OS Route
There is more than one way to carry out the HDD/SDD validation tests on the server.
A program specially written for this purpose could be used in a bootable form and run on the server.
However, the easiest way to conduct the HDD/SDD validation tests is to install the TrueNAS OS on the server. It has everything we need. This is not a proper installation of the OS, but just a test installation so we can conduct the SMART and badblocks tests needed.
There are a number of ways the TrueNAS OS can be installed, for example from a CD/DVD, or across a network using PXE. Fester favours a USB stick.
Create the installer USB stick, install TrueNAS, and enable the SSH service, as described on the linked pages. When you first log in to the web GUI, you'll likely see the Initial Wizard; you can just exit out of this at this time.
¶ Conducting The HDD/SDD Validation Tests (Finally!) In TrueNAS Via An SSH Console
TrueNAS comes with certain software tools and capabilities built into it that will make the task of HDD/SDD validation much easier. This is why we needed to install it before conducting the tests.
The SSH console provides no tools for validation purposes, but does provide the means by which we can flexibly interact with the built in tools in TrueNAS to accomplish the validation tests. This is why we needed to set this up before carrying out the tests.
¶ SMART Short Tests Via The SSH Console
Open up the TrueNAS web GUI in your browser and log in.
Go to the “Storage” page (1) and click “Disks” (2).
This should bring up a list of the storage HDDs (i.e. for data, not the OS) that are currently in your system.
Make a list of the names of each drive (shown in a red box in the screen shot) these will be needed soon. Do not include the name of the boot device, which is shown in the list as belonging to boot-pool.
(On Fester’s system this would be da0 – da7, giving a total of 8 HDDs; the screen shot above is from a different system. On this system, the only disk you'd use would be da1.)
Under TrueNAS SCALE, the drive names will most likely be in the form of sda, sdb, etc. Again, make a list of all drives that are not part of boot-pool.
¶ Install the script
To run the drive tests, we're going to use a helpful script created by user dak180 on the TrueNAS forums.
If you're using TrueNAS CORE, ssh to your server as the root user. If you're using SCALE, either SSH to your server as the admin user, or use the built-in shell in the web GUI (System → Shell). Then, at the command prompt, run git clone https://github.com/dak180/disk-burnin-and-testing. This will download the script and its documentation into a new directory. Change into that directory by typing cd disk-burnin-and-testing.
¶ Run the script
WARNING: This test is destructive. Data on any disks tested will be destroyed. DO NOT run this test on disks that are part of a pool, or that otherwise contain data you want to keep.
To run the script under CORE, type ./disk-burnin.sh -tm ‘da1 da2 …’, including the names of each disk you want to test, and then press Enter.
To run the script under SCALE, type sudo ./disk-burnin.sh -tm ‘sda sdb sdc …’, again including the names of each disk you want to test, and then press Enter. You'll likely be prompted for the password for your admin user, so enter it.
Once you run that command, you'll see output like 0: 1 windows (created Sat Nov 16 10:02:50 2024), and you'll then be returned to the command prompt. The tests are running in the background using tmux, which is described below.
¶ Badblocks Test Using tmux
Start an SSH session and log in.
Before starting tmux we need to enable the kernel geometry debug flags, so type in this command at the command prompt.
sysctl kern.geom.debugflags=0x10
(When all the Badblocks tests are done the kernel geometry debug flags must be returned to their normal state. Thankfully no additional command is necessary, just reboot the server as this setting is not persistent and cannot survive the reboot.)
Now type the following command at the command prompt.
tmux
You should see a screen something like this. Notice the green band at the bottom of the screen, this is a tmux session.
I will not be running the Badblocks test on ada0 (the SSD drive) there is no point as already explained and this is a destructive test (the TrueNAS OS is on this drive!).
This leaves the 8 data storage drives to check.
This means I will need 8 sessions opened in tmux (open the number of sessions that suits your requirements).
Let us start by renaming the current session in tmux to something more meaningful than “csh”.
In the tmux window press the “Ctrl” and “b” keys together, release them and then press the “,” key.
The bar at the bottom of the window should turn yellow and you can now delete the “csh” text and rename it (Fester called his “da0” after the drive that will be tested).
When you have typed in the new name press the “Return/Enter” key, the bar should now resort back to its original green colour and the session should be renamed.
At this point we need to create an additional session and rename it for the next drive to be tested.
To create a new session press the “Ctrl” and “b” keys together, release them and then press the “c” key.
You should get something like this where “1:csh*” is the newly created session. Incidentally the asterisk just denotes the currently selected session.
Let us rename this session by pressing the “Ctrl” and “b” keys together, releasing them and then pressing the “,” key. Type in the new name and press the “Return/Enter” key (just as we did before, I called this one “da1” after the next drive to be tested).
Navigation between the different sessions is achieved by pressing the “Ctrl” and “b” keys together, releasing them and then pressing the “n” key. This will take you to the next session along.
Alternatively you can also press the “Ctrl” and “b” keys together, release them and then press the “p” key. This will take you to the previous session.
By using the next and previous navigational keystroke combinations you can navigate through the different sessions, the asterisk signifying which session you are currently viewing.
Using the key combinations already explained let us create the remaining sessions needed and rename each one.
Now we can run the Badblocks tests from within tmux.
Navigate to the first session (i.e. “0:da0”) and type in the following command at the prompt.
badblocks -ws /dev/da0
Fester uses a slightly different command to improve the efficiency of the tests with the WD40EFRX drives. These drives have a sector size of 4096 bytes (even though they report 512 bytes, naughty Western Digital). I also like a more verbose output from these tests so the command includes the –v switch. I include it here for informational purposes only. In addition to improving efficiency, the “-b 4096” below is required with larger disks (i.e., larger than about 4 TB).
badblocks –b 4096 –vws /dev/da0
If the command executes properly you should see something like this.
You will see from the screen the completion progress expressed as a percentage (1) and any errors that have occurred expressed like this “(0/0/0 errors)” (2).
There should be zero errors throughout the test. If you get even one error then you should return the disk for testing.
Now navigate to the next session (in Fester’s case that is “1:da1”) and type this at the command prompt.
badblocks -ws /dev/da1
(Or Fester’s variation if it suits you better, but remember to change the drive name from “da0” to “da1”.)
Repeat this process of changing session and running the Badblocks command for every drive in your system that you want to test. In Fester’s case this means running these commands while changing sessions each time.
badblocks -ws /dev/da2
badblocks -ws /dev/da3
badblocks -ws /dev/da4
badblocks -ws /dev/da5
badblocks -ws /dev/da6
badblocks -ws /dev/da7
¶ Stopping A Badblocks Test
If for any reason you need to stop a Badblocks test then navigate to the applicable session at press the “Ctrl” and “c” keys together, then release them. This should stop the test.
¶ Detaching a tmux Session
Once you've started all your tests running, you can “detach” your tmux session to let the tests run while you do something else with your SSH session (or just close the SSH connection entirely). To do this, press the “Ctrl” and “b” keys together, then press the “d” key. This will return you to your SSH session, without the tmux session (you'll notice that the green bar isn't present on the bottom of the screen).
¶ Resuming A Session In tmux
Badblocks tests can take a long time (when the tests completed Fester was far from where he had started due to Continental Drift and Plate Tectonics).
You do not need to keep the terminal open or the client computer switched on in order to keep the tmux session running.
If you need to pack up for the night then just close the window that the sessions are running in (just don’t shut down the server). Then get your ferrets to shut down your client computer and your pigeons to knock up a suitable night cap (I find a Multiple Orgasm very agreeable before bed).
When you need to re-establish the connection with the tmux session/s simply start an SSH session and log in.
Type the following command in the command prompt.
tmux attach
or simply
tmux a
This should return you to the tmux session(s).
When the tests are complete navigate to an open session, note the results if you need to and then type the following into the command prompt.
exit
This will close that particular session in tmux.
Do this for each session in turn until you have exited all the sessions in tmux.
You will find that on exiting the last open session in tmux you will be returned to the standard SSH console (in Fester’s case PuTTY).
Now reboot the server to reset the kernel geometry debug flags to their standard setting.
That’s the Badblocks tests complete.
In order to complete all the HDD validation tests we must now repeat the SMART long tests. As this has already been documented I won’t repeat it here. Just go back to the relevant section and repeat again.
Once the SMART long tests have completed then it is time to collect the results.
¶ Getting Your Test Results
Getting your test results is always a tense moment.
(I remember such an instance in the doctor’s examination room after an unforgettable trip to Bognor Regis, often referred to as “The Riviera of the South West”. Unfortunately the doctor confirmed Fester had come back with more than just fond memories, but with the liberal application of a strong antibiotic cream Fester was as good as new in a couple of weeks.)
Here is how to get your results.
(Do not start this section until all HDD tests have been completed.)
Open an SSH console and log in.
We are going to issue a command to each HDD/SDD in succession that will interrogate and retrieve the results of the tests stored in each drives memory using SMART commands.
At the command prompt type in the following command using the name of the first drive you want to interrogate (in Fester’s case this is ada0).
smartctl –a /dev/ada0
This should produce the following screen with the test results. The window displaying the information has been maximised (1) so it is easier to read.
At this point Fester copies the information and pastes it into a text editor for ease of use.
If you want to do this then select the text in the SSH console by clicking with the left mouse button where you want to begin, hold it down and then highlight the text you want to include.
When you have done this press the “Ctrl” button and the “v” button together. This keystroke will copy the highlighted text into the clip board.
Open the text editor you wish to use (Fester uses Notepad in Windows) and paste it into the text into the editor.
You now need to repeat this process for the next drive in your system.
At the command prompt type in the following command using the name of the next drive you want to interrogate (in Fester’s case this is da0).
smartctl –a /dev/da0
This will produce the next set of results in the SSH console. Copy and paste as before (if you want to).
Now repeat the process for the next drive and the next until all the drives have been interrogated and their data copied and pasted.
(In this way you will build up a list of each drives test results in a single text file that can be saved for examination later.)
In Festers case this would mean issuing the following commands in the SSH console.
smartctl –a /dev/da1
smartctl –a /dev/da2
smartctl –a /dev/da3
smartctl –a /dev/da4
smartctl –a /dev/da5
smartctl –a /dev/da6
smartctl –a /dev/da7
These commands produce copious amounts of information about the drives. If you want something a little less gregarious then use this command instead (don’t forget to change the drive name each time, and note the capital -A rather than the lowercase -a).
smartctl –A /dev/ada0
This should produce a screen that looks something like this (much more compact).
So you have now gathered your results, but they make about as much sense as a bacon butty at a bar mitzvah.
What now?
¶ Making Sense of SMART Data
When looking at SMART data from a SMART storage device certain entries in the table are not important in terms of data integrity and health. They just give general information (e.g. Model, serial number, etc) and other types of information that could be useful in certain circumstances.
Other entries are very important and should immediately ring alarms bells if certain values are present.
In terms of HDD/SDD hardware validation these are the entries in the SMART data you need to scrutinise.
| ID# |
ATTRIBUTE_NAME |
FLAG |
VALUE |
WORST |
THRESH |
TYPE |
UPDATED |
WHEN_FAILED |
RAW_VALUE |
| 1 |
Raw_Read_Error_Rate |
0x002f |
200 |
200 |
051 |
Prefail |
Always |
- |
0 |
| 5 |
Reallocated_Sector_Ct |
0x0033 |
200 |
200 |
140 |
Prefail |
Always |
- |
0 |
| 7 |
Seek_Error_Rate |
0x002e |
200 |
200 |
000 |
Old_age |
Always |
- |
0 |
| 10 |
Spin_Retry_Count |
0x0032 |
100 |
100 |
000 |
Old_age |
Always |
- |
0 |
| 11 |
Calibration_Retry_Count |
0x0032 |
100 |
100 |
000 |
Old_age |
Always |
- |
0 |
| 196 |
Reallocated_Event_Count |
0x0032 |
200 |
200 |
000 |
Old_age |
Always |
- |
0 |
| 197 |
Current_Pending_Sector |
0x0032 |
200 |
200 |
000 |
Old_age |
Always |
- |
0 |
| 198 |
Offline_Uncorrectable |
0x0030 |
100 |
253 |
000 |
Old_age |
Always |
- |
0 |
| 199 |
UDMA_CRC_Error_Count |
0x0032 |
200 |
200 |
000 |
Old_age |
Always |
- |
0 |
If you get any value other than zero in the “RAW VALUE” for these entries you should be suspicious of this drive and may need to return the device for testing depending on the manufacturer’s warranty.
Another area you should look at is the “SMART Self-test log structure”. Here is an example. It will tell you if the drive passed its tests. Any result other than “Completed without error” is cause for concern.
SMART Self-test log structure revision number 1
| Num |
Test_Description |
Status |
Remaining LifeTime(hours) |
LifeTime(hours) |
LBA_of_first_error |
| # 1 |
Extended offline |
Completed without error |
00% |
503 |
- |
| # 2 |
Conveyance offline |
Completed without error |
00% |
494 |
- |
| # 3 |
Short offline |
Completed without error |
00% |
75 |
- |
(If Fester is misinformed about interpreting SMART data or has omitted something important please let me know and I will try to put it in the guide or you could replace this or any section with your own?)
That’s the HDD/SDD validation completed. Now it is time to reinstall TrueNAS and create a basic server.