I hope to save you a lot of time if you are new to the world of HBA cards and external drive enclosures.
- You might run across the phrases “HBA Adapter” or “RAID Card” used interchangeably in forums and posts – they aren’t necessarily interchangeable terms but are highly related, so the nuances are:
- “HBA” – “Host Bus Adapter” – this is a PCIe card that is installed in the host machine and then connected to an external enclosure of hard drives that will allow you to “see” those drives. For the most part, these adapters are usually ~$100 and behave as “dumb pass throughs” – doing nothing besides making the drives ‘seeable’ from the host machine. As a literal example, if you had an external enclosure with 4 hard drives in it and 1 hard drive in your hose machine and connected it via HBA – if you looked in Windows Explorer, you would see 5 hard drives (C, D, E, F and G).
- “RAID” – “Redundant Array of Independent Disks” – on the high end this will be a PCIe card that is installed in the host machine and EITHER connected to drives inside the host machine or connected out to drives housed in an external enclosure. These cards can be upwards of $300, 500 or $1500 depending on how fancy/fast/high end you want to get. These cards have robust/powerful chips and firmware running on them that implement many or all of the different RAID standards. These cards are manipulating not only how the host system “sees” the drives but also how the drives are managed and data read/written to them. Depending on the RAID setup you use, your host system might see one drive or many drives – it’s all about the data resiliency/performance strategy you are trying to accomplish.
- “LSI” is one of the favorite brands for HBA cards – as far as I can tell the cards everyone loves are all from 2012-2016 so yes it’s “old hardware” but it also works perfectly fine; don’t get nervous and think “I better search for the NEW version of this” – it’s either not necessarily or there isn’t one. I haven’t dug enough to tell which.
- “JBOD” is a generic term that literally means “Just a Bunch of Disks” – it refers to either an enclosure or a setup whereby a large number of hard drives are being housed in an enclosure that isn’t doing anything fancy with them – no RAID, just straight pass through. When you go shopping for mid to large scale drive enclosures (I saw this mostly in rack-mount setups) you need to differentiate if you want the enclosure to be more complex and have a RAID controller on it or if it just wanted a simple “JBOD” type enclosure which had very simply sets of outbound ports on it to connect to a hosting machine.
- Intel i7-11700k
- ASUS TUF Gaming Z590-PLUS (non-wifi)
- LSI 9201-16e Card from eBay
- PC Pitstop 12-bay, SATA External Enclosure
- 12x 14TB SATA drives
The Adventure Begins…
I have very little experience with RAID cards – I’ve done both software RAID setups on Linux (mdadm), used the built-in support on a few consumer motherboards in the past and 1 time set it up in a server – but never really bathed in it’s glory.
I’ve also never had an HBA card in a machine I’ve built before, so I wasn’t entirely sure what to expect here.
Once I got my build together, I installed Ubuntu 20.04 LTS on it and threw the LSI 9201-16e card in it and rebooted.
Problem #1 – Kernel Driver Bug
First things first, upon reboot I didn’t see any change to my boot-up sequence… no supplemental messaging from the card after my computer POST’ed saying something like “Press CTRL-C to enter HBA setup…” – it would just boot straight into Ubuntu.
Like I said – never worked with an HBA card before – so I’m not even sure… how to use it?
In Ubuntu I spent longer than I would like to admit trying to figure out how to… access the card? Use the card?
I wasn’t totally sure – lots of posts mentioned using the “StorCLI” or “MegaCLI” command line utils from Broadcom’s site (that were last published in 2014 or 2016)… I ended up pulling those down and converting the RPM’s into DEBs before installing them to no avail… none of the tools could find the card.
I eventually got smart and decided to start grepping dmesg (the startup log in Linux for your machine)
sudo dmesg | grep mpt2sas <... snipped a lot of output ...> "mpt2sas_cm0: failure at drivers/scsi/mpt3sas/mpt3sas_scsih.c:10791/_scsih_probe()!"
(I originally grepped for
sata and then
sas before noticing the new module that controlled the card was named
Ahhh ok so the driver isn’t loading properly – that’s probably what is going on.
Fortunately there are 2 workarounds – I prefer the 1st one:
- Limit the Queue Depth directly via the kernel arguments.
- Downgrade back to Kernel 5.7.x
As for the solution for #1, you do this by editing your Grub configuration (on Ubuntu 20.04 at least – please adjust instructions for your favorite distro)
sudo nano -w /etc/default/grub
Now you want to edit around line 11 or so and a suitable value for the
NOTE: All the references to this I found online clarify setting a queue depth of
10,000 fixed their errors – I continued to have errors and even thought my arguments weren’t getting picked up until I tried
8,000 and suddenly it started working. So you MAY need to play with this value a few times.
Per the NOTE above, my edited line ended up looking like this:
#GRUB_CMDLINE_LINUX_DEFAULT="quiet splash" GRUB_CMDLINE_LINUX_DEFAULT="mpt3sas.max_queue_depth=8000"
I always duplicate and comment out any line I’m editing in Linux configs so I can always refer to the original contents.
Full disclosure – I didn’t know if I could append the
mpt3sas setting to the
quiet splash list or not, so I just made it the only argument. Please feel free to do your own testing with those other values included – there is no particular reason I needed to remove them.
After saving the file, you need to run the following command to have Grub update itself with the config change and generate new init files to read on boot (thx Daniel & Saul!):
Now go ahead and reboot – and VIOLA, now Linux could see the device and initialization during bootup was working as expected (screenshot provided for ease of glancing):
A quick peek at LSHW and I can confirm the card exists in this realm of reality:
sudo lshw < scrolling down 100s of lines... >
Excellent, now the card must be working and I can see all 12 hard drives in the external enclosure, right?
No luck – it’s also worth noting again – I have no experience with HBAs – so I’m not sure if I’m even expecting the right thing… will I just suddenly see drives popup in Nautilus (Ubuntu’s File Explorer), or more devices under
/dev/sdXXX or is my external storage enclosure even working?
At least at this point I figured the card was working and my war was no longer with Linux but rather with understanding the card, how it is suppose to work and how I get it to do my bidding.
Problem #2 – HBA Firmware
I spent the next hour or so reading through forums, hoping for posts from clueless people like myself asking what is suppose to happen on boot.
DISCOVERY: It appears these LSI 9201 cards can optionally have a BIOS on them that introduces additional behaviors that cause a boot-prompt after POST’ing to show up and allow you to configure them. I also found most people prefer to ERASE the BIOS which allows the computer to boot faster and the card has a default “IT Mode” it operates in, where it just passes through external connections.
Interesting – but helpful!
So it appeared me NOT seeing a messaging during boot-up from the card was fine and signaled that the card didn’t have one of those fancier BIOSs installed.
That did send me down the path of wondering how old the firmware was on this device – it’s probably an 8 or 10 year old piece of hardware and the last firmware was released around 2015 or 16 (P20.00.00.70) – how could it NOT have the latest firmware on it if it was pulled out of production?
I decided to navigate over to the Broadcom official site to find the latest downloads for this card.
This site is not the most intuitive, so to help you out here is a screenshot of the search criteria you should use to find download resources for this Legacy piece of hardware – if you have newer hardware, you might need to adjust this:
(Here is a link to click also)
What I ended up learning is that you need to download 2 pieces of software:
- The utility used to talk to the card / flash the ROM – in my case it was
Installer_P20_for_Linux(link) – version 20.00.00.00
- An archive that contains the actual latest firmware – in my case it was
9201-16e_Package_P20_IT_Firmware_BIOS_for_MSDOS_Windows(link) – version 20.00.00.70.
Go ahead and unzip the 1st package (Installer_P20) – it’ll contain a subdirectory named
sas2flash_linux_i686_x86-64_rel – inside is a file named
sas2flash that you’ll have to make executable:
chmod +x sas2flash
Ok now that the utility is executable, it can actually be used for a lot of things related to interacting with the card besides flashing it – for example – let’s check the status of our card and make sure the utility can find our card by using the
$ sudo ./sas2flash -listall LSI Corporation SAS2 Flash Utility Version 20.00.00.00 (2014.09.18) Copyright (c) 2008-2014 LSI Corporation. All rights reserved Adapter Selected is a LSI SAS: SAS2116_1(B1) Num Ctlr FW Ver NVDATA x86-BIOS PCI Addr 0 SAS2116_1(B1) 13.00.60.00 0d.43.00.10 No Image 00:03:00:00 Finished Processing Commands Successfully. Exiting SAS2Flash.
Oh, holy HELL, v13.00.60.00 of the firmware?! Well now I know it’s not fully upgrade so let’s hope that does the trick.
TIP: I actually found the
sas2flash command reference PDF (link) helpful to glance through.
One more thing I want to check – the
sas2flash tool actually lets you query the link status of the HBA – so I can ask it directly if it see the drives in the external enclosure using the
testlsall command – this is considered an Advanced Mode command so you have to force it with the
-o argument (again, I’m not genius, I just read through the PDF and eventually figured this out)
$ sudo ./sas2flash -o -testlsall LSI Corporation SAS2 Flash Utility Version 20.00.00.00 (2014.09.18) Copyright (c) 2008-2014 LSI Corporation. All rights reserved Advanced Mode Set Adapter Selected is a LSI SAS: SAS2116_1(B1) Executing Operation: Test Link State ALL Phy 0: Link Down Phy 1: Link Down Phy 2: Link Down Phy 3: Link Down Phy 4: Link Down Phy 5: Link Down Phy 6: Link Down Phy 7: Link Down Phy 8: Link Down Phy 9: Link Down Phy 10: Link Down Phy 11: Link Down Phy 12: Link Down Phy 13: Link Down Phy 14: Link Down Phy 15: Link Down Test Link State All PASSED! Finished Processing Commands Successfully. Exiting SAS2Flash.
Well that’s not good… I guess it’s not seeing the drives so at least I won’t continue to worry about finding them or how the HBA works right now because obviously something else is wrong.
Believe it or not this actually felt good to me – I was querying the card, so it’s working and it confirmed for me that it doesn’t see the hard drives in the enclosure so I’ll stop worrying about that for now.
Now that we are on a warpath to update the firmware, go back and unzip the 2nd package you downloaded – the one with the firmware (.bin) in it.
Even though the 2nd download says “for MSDOS_Windows”, once you unzip it, you can find the
.bin file (that represents the firmware) neatly tucked away under the
Under that directory you’ll find the
HBA_9201_16e_IT directory and in there is the file
9201-16e.bin file into the directory containing the
sas2flash binary you just made executable to keep things easy.
TIP: You might need to ‘clear’ the card before flashing the new firmware to it, this can be done with the
sas2flash -o -e 6 command where the
6 indicates a “Clean Flash” (Manual Page 14 / Section 5.4.10) – my thanks to Will5001 for this tip!
I then executed the flashing of the firmware:
$ sudo ./sas2flash -f 9201-16e.bin LSI Corporation SAS2 Flash Utility Version 20.00.00.00 (2014.09.18) Copyright (c) 2008-2014 LSI Corporation. All rights reserved Adapter Selected is a LSI SAS: SAS2116_1(B1) Executing Operation: Flash Firmware Image Firmware Image has a Valid Checksum. Firmware Version 20.00.07.00 Firmware Image compatible with Controller. Valid NVDATA Image found. NVDATA Version 14.01.00.00 Checking for a compatible NVData image... NVDATA Device ID and Chip Revision match verified. NVDATA Versions Compatible. Valid Initialization Image verified. Valid BootLoader Image verified. Beginning Firmware Download... Firmware Download Successful. Verifying Download... Firmware Flash Successful. Resetting Adapter... Adapter Successfully Reset. Finished Processing Commands Successfully. Exiting SAS2Flash.
After that was done I confirmed the firmware upgrade with the
listall command again:
$ sudo ./sas2flash -listall LSI Corporation SAS2 Flash Utility Version 20.00.00.00 (2014.09.18) Copyright (c) 2008-2014 LSI Corporation. All rights reserved Adapter Selected is a LSI SAS: SAS2116_1(B1) Num Ctlr FW Ver NVDATA x86-BIOS PCI Addr 0 SAS2116_1(B1) 20.00.07.00 14.01.00.07 No Image 00:03:00:00 Finished Processing Commands Successfully. Exiting SAS2Flash.
SUCCESS – the firmware is now fully upgrade!
On a whim I decided to try the
testlsall command again to see if suddenly all my problems were done and viola – it can see ALL THE DRIVES!
$ sudo ./sas2flash -o -testlsall LSI Corporation SAS2 Flash Utility Version 20.00.00.00 (2014.09.18) Copyright (c) 2008-2014 LSI Corporation. All rights reserved Advanced Mode Set Adapter Selected is a LSI SAS: SAS2116_1(B1) Executing Operation: Test Link State ALL Phy 0: 6.0 Gbps Phy 1: 6.0 Gbps Phy 2: 6.0 Gbps Phy 3: 6.0 Gbps Phy 4: 6.0 Gbps Phy 5: 6.0 Gbps Phy 6: 6.0 Gbps Phy 7: 6.0 Gbps Phy 8: 6.0 Gbps Phy 9: 6.0 Gbps Phy 10: 6.0 Gbps Phy 11: 6.0 Gbps Phy 12: Link Down Phy 13: Link Down Phy 14: Link Down Phy 15: Link Down Test Link State All PASSED! Finished Processing Commands Successfully. Exiting SAS2Flash.
Right before I decided to get “WORLD’S SMRTEST MAN” tattoo’ed on my chest, I thought to check one more thing, the GParted ‘Disks’ app in Ubuntu, can it really see these drives?
My happiness literally knows no bounds at this point, we DID IT!
This entire journey was probably a 4hrs process – I learned a lot along the way which made it particularly enjoyable and of course we landed the plane successfully at the end.
It appears if your HBA card is working, it really does seamlessly pass through the connectivity of the drives to the host system as if they were local (like having 18 SATA connections on your computer).
I truly hope this writeup helps someone out there that got stuck just like I did trying to understand all of this.