Linux (Ubuntu), ASUS Z590, LSI 9201-16e HBA Card and how the hell you get it all working together

I hope to save you a lot of time if you are new to the world of HBA cards and external drive enclosures.

Quick Reference

  1. You might run across the phrases “HBA Adapter” or “RAID Card” used interchangeably in forums and posts – they aren’t necessarily interchangeable terms but are highly related, so the nuances are:
    1. HBA” – “Host Bus Adapter” – this is a PCIe card that is installed in the host machine and then connected to an external enclosure of hard drives that will allow you to “see” those drives. For the most part, these adapters are usually ~$100 and behave as “dumb pass throughs” – doing nothing besides making the drives ‘seeable’ from the host machine. As a literal example, if you had an external enclosure with 4 hard drives in it and 1 hard drive in your hose machine and connected it via HBA – if you looked in Windows Explorer, you would see 5 hard drives (C, D, E, F and G).
    2. RAID” – “Redundant Array of Independent Disks” – on the high end this will be a PCIe card that is installed in the host machine and EITHER connected to drives inside the host machine or connected out to drives housed in an external enclosure. These cards can be upwards of $300, 500 or $1500 depending on how fancy/fast/high end you want to get. These cards have robust/powerful chips and firmware running on them that implement many or all of the different RAID standards. These cards are manipulating not only how the host system “sees” the drives but also how the drives are managed and data read/written to them. Depending on the RAID setup you use, your host system might see one drive or many drives – it’s all about the data resiliency/performance strategy you are trying to accomplish.
  2. “LSI” is one of the favorite brands for HBA cards – as far as I can tell the cards everyone loves are all from 2012-2016 so yes it’s “old hardware” but it also works perfectly fine; don’t get nervous and think “I better search for the NEW version of this” – it’s either not necessarily or there isn’t one. I haven’t dug enough to tell which.
  3. “JBOD” is a generic term that literally means “Just a Bunch of Disks” – it refers to either an enclosure or a setup whereby a large number of hard drives are being housed in an enclosure that isn’t doing anything fancy with them – no RAID, just straight pass through. When you go shopping for mid to large scale drive enclosures (I saw this mostly in rack-mount setups) you need to differentiate if you want the enclosure to be more complex and have a RAID controller on it or if it just wanted a simple “JBOD” type enclosure which had very simply sets of outbound ports on it to connect to a hosting machine.

My Build

  • Intel i7-11700k
  • ASUS TUF Gaming Z590-PLUS (non-wifi)
  • LSI 9201-16e Card from eBay
  • PC Pitstop 12-bay, SATA External Enclosure
  • 12x 14TB SATA drives

The Adventure Begins…

I have very little experience with RAID cards – I’ve done both software RAID setups on Linux (mdadm), used the built-in support on a few consumer motherboards in the past and 1 time set it up in a server – but never really bathed in it’s glory.

I’ve also never had an HBA card in a machine I’ve built before, so I wasn’t entirely sure what to expect here.

Once I got my build together, I installed Ubuntu 20.04 LTS on it and threw the LSI 9201-16e card in it and rebooted.

Problem #1 – Kernel Driver Bug

First things first, upon reboot I didn’t see any change to my boot-up sequence… no supplemental messaging from the card after my computer POST’ed saying something like “Press CTRL-C to enter HBA setup…” – it would just boot straight into Ubuntu.

Like I said – never worked with an HBA card before – so I’m not even sure… how to use it?

In Ubuntu I spent longer than I would like to admit trying to figure out how to… access the card? Use the card?

I wasn’t totally sure – lots of posts mentioned using the “StorCLI” or “MegaCLI” command line utils from Broadcom’s site (that were last published in 2014 or 2016)… I ended up pulling those down and converting the RPM’s into DEBs before installing them to no avail… none of the tools could find the card.

I eventually got smart and decided to start grepping dmesg (the startup log in Linux for your machine)

sudo dmesg | grep mpt2sas

<... snipped a lot of output ...>

"mpt2sas_cm0: failure at drivers/scsi/mpt3sas/mpt3sas_scsih.c:10791/_scsih_probe()!"

(I originally grepped for sata and then sas before noticing the new module that controlled the card was named mpt2sas)

Ahhh ok so the driver isn’t loading properly – that’s probably what is going on.

After a lot of Googling, I discovered this issue has been in the Linux kernel since the middle of the 5.8.x series and at the time of this writing (May 2021) still is not fixed.

Fortunately there are 2 workarounds – I prefer the 1st one:

  1. Limit the Queue Depth directly via the kernel arguments.
  2. Downgrade back to Kernel 5.7.x

As for the solution for #1, you do this by editing your Grub configuration (on Ubuntu 20.04 at least – please adjust instructions for your favorite distro)

sudo nano -w /etc/default/grub

Now you want to edit around line 11 or so and a suitable value for the mpt3sas.max_queue_depth argument.

NOTE: All the references to this I found online clarify setting a queue depth of 10,000 fixed their errors – I continued to have errors and even thought my arguments weren’t getting picked up until I tried 8,000 and suddenly it started working. So you MAY need to play with this value a few times.

Per the NOTE above, my edited line ended up looking like this:

#GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
GRUB_CMDLINE_LINUX_DEFAULT="mpt3sas.max_queue_depth=8000"

I always duplicate and comment out any line I’m editing in Linux configs so I can always refer to the original contents.

Full disclosure – I didn’t know if I could append the mpt3sas setting to the quiet splash list or not, so I just made it the only argument. Please feel free to do your own testing with those other values included – there is no particular reason I needed to remove them.

After saving the file, you need to run the following command to have Grub update itself with the config change and generate new init files to read on boot (thx Daniel & Saul!):

sudo update-grub

Now go ahead and reboot – and VIOLA, now Linux could see the device and initialization during bootup was working as expected (screenshot provided for ease of glancing):

dmesg output for mpt2sas lsi 9201-16e initialization

A quick peek at LSHW and I can confirm the card exists in this realm of reality:

sudo lshw

< scrolling down 100s of lines... >
sudo lshw output

Excellent, now the card must be working and I can see all 12 hard drives in the external enclosure, right?

No luck – it’s also worth noting again – I have no experience with HBAs – so I’m not sure if I’m even expecting the right thing… will I just suddenly see drives popup in Nautilus (Ubuntu’s File Explorer), or more devices under /dev/sdXXX or is my external storage enclosure even working?

At least at this point I figured the card was working and my war was no longer with Linux but rather with understanding the card, how it is suppose to work and how I get it to do my bidding.

Problem #2 – HBA Firmware

I spent the next hour or so reading through forums, hoping for posts from clueless people like myself asking what is suppose to happen on boot.

DISCOVERY: It appears these LSI 9201 cards can optionally have a BIOS on them that introduces additional behaviors that cause a boot-prompt after POST’ing to show up and allow you to configure them. I also found most people prefer to ERASE the BIOS which allows the computer to boot faster and the card has a default “IT Mode” it operates in, where it just passes through external connections.

Interesting – but helpful!

So it appeared me NOT seeing a messaging during boot-up from the card was fine and signaled that the card didn’t have one of those fancier BIOSs installed.

That did send me down the path of wondering how old the firmware was on this device – it’s probably an 8 or 10 year old piece of hardware and the last firmware was released around 2015 or 16 (P20.00.00.70) – how could it NOT have the latest firmware on it if it was pulled out of production?

I decided to navigate over to the Broadcom official site to find the latest downloads for this card.

This site is not the most intuitive, so to help you out here is a screenshot of the search criteria you should use to find download resources for this Legacy piece of hardware – if you have newer hardware, you might need to adjust this:

Broadcom support website download search criteria

(Here is a link to click also)

What I ended up learning is that you need to download 2 pieces of software:

  1. The utility used to talk to the card / flash the ROM – in my case it was Installer_P20_for_Linux (link) – version 20.00.00.00
  2. An archive that contains the actual latest firmware – in my case it was 9201-16e_Package_P20_IT_Firmware_BIOS_for_MSDOS_Windows (link) – version 20.00.00.70.

Go ahead and unzip the 1st package (Installer_P20) – it’ll contain a subdirectory named sas2flash_linux_i686_x86-64_rel – inside is a file named sas2flash that you’ll have to make executable:

chmod +x sas2flash

Ok now that the utility is executable, it can actually be used for a lot of things related to interacting with the card besides flashing it – for example – let’s check the status of our card and make sure the utility can find our card by using the listall command:

$ sudo ./sas2flash -listall
 LSI Corporation SAS2 Flash Utility
 Version 20.00.00.00 (2014.09.18) 
 Copyright (c) 2008-2014 LSI Corporation. All rights reserved 

 Adapter Selected is a LSI SAS: SAS2116_1(B1)

 Num   Ctlr            FW Ver        NVDATA        x86-BIOS         PCI Addr
 0  SAS2116_1(B1)   13.00.60.00    0d.43.00.10      No Image      00:03:00:00

 Finished Processing Commands Successfully. Exiting SAS2Flash.

Oh, holy HELL, v13.00.60.00 of the firmware?! Well now I know it’s not fully upgrade so let’s hope that does the trick.

TIP: I actually found the sas2flash command reference PDF (link) helpful to glance through.

One more thing I want to check – the sas2flash tool actually lets you query the link status of the HBA – so I can ask it directly if it see the drives in the external enclosure using the testlsall command – this is considered an Advanced Mode command so you have to force it with the -o argument (again, I’m not genius, I just read through the PDF and eventually figured this out)

$ sudo ./sas2flash -o -testlsall
 LSI Corporation SAS2 Flash Utility
 Version 20.00.00.00 (2014.09.18)
 Copyright (c) 2008-2014 LSI Corporation. All rights reserved
     Advanced Mode Set
     Adapter Selected is a LSI SAS: SAS2116_1(B1)
     Executing Operation: Test Link State ALL
     Phy 0: Link Down
     Phy 1: Link Down
     Phy 2: Link Down
     Phy 3: Link Down
     Phy 4: Link Down
     Phy 5: Link Down
     Phy 6: Link Down
     Phy 7: Link Down
     Phy 8: Link Down
     Phy 9: Link Down
     Phy 10: Link Down
     Phy 11: Link Down
     Phy 12: Link Down
     Phy 13: Link Down
     Phy 14: Link Down
     Phy 15: Link Down

Test Link State All PASSED!     
Finished Processing Commands Successfully.
Exiting SAS2Flash.

Well that’s not good… I guess it’s not seeing the drives so at least I won’t continue to worry about finding them or how the HBA works right now because obviously something else is wrong.

MAKING PROGRESS! - Happy Homer | Make a Meme

Believe it or not this actually felt good to me – I was querying the card, so it’s working and it confirmed for me that it doesn’t see the hard drives in the enclosure so I’ll stop worrying about that for now.

Now that we are on a warpath to update the firmware, go back and unzip the 2nd package you downloaded – the one with the firmware (.bin) in it.

Even though the 2nd download says “for MSDOS_Windows”, once you unzip it, you can find the .bin file (that represents the firmware) neatly tucked away under the Firmware directory:

unzipped firmware flash directory contents

Under that directory you’ll find the HBA_9201_16e_IT directory and in there is the file 9201-16e.bin.

Copy the 9201-16e.bin file into the directory containing the sas2flash binary you just made executable to keep things easy.

TIP: You might need to ‘clear’ the card before flashing the new firmware to it, this can be done with the sas2flash -o -e 6 command where the 6 indicates a “Clean Flash” (Manual Page 14 / Section 5.4.10) – my thanks to Will5001 for this tip!

I then executed the flashing of the firmware:

$ sudo ./sas2flash -f 9201-16e.bin 
 LSI Corporation SAS2 Flash Utility
 Version 20.00.00.00 (2014.09.18) 
 Copyright (c) 2008-2014 LSI Corporation. All rights reserved 

 Adapter Selected is a LSI SAS: SAS2116_1(B1)
 Executing Operation: Flash Firmware Image

     Firmware Image has a Valid Checksum.
     Firmware Version 20.00.07.00
     Firmware Image compatible with Controller.
     Valid NVDATA Image found.
     NVDATA Version 14.01.00.00
     Checking for a compatible NVData image...
     NVDATA Device ID and Chip Revision match verified.
     NVDATA Versions Compatible.
     Valid Initialization Image verified.
     Valid BootLoader Image verified.
     Beginning Firmware Download...
     Firmware Download Successful.
     Verifying Download...
     Firmware Flash Successful.
     Resetting Adapter...
     Adapter Successfully Reset. 

Finished Processing Commands Successfully.
Exiting SAS2Flash.

After that was done I confirmed the firmware upgrade with the listall command again:

$ sudo ./sas2flash -listall
 LSI Corporation SAS2 Flash Utility
 Version 20.00.00.00 (2014.09.18) 
 Copyright (c) 2008-2014 LSI Corporation. All rights reserved 

 Adapter Selected is a LSI SAS: SAS2116_1(B1) 

 Num   Ctlr            FW Ver        NVDATA        x86-BIOS         PCI Addr
 0  SAS2116_1(B1)   20.00.07.00    14.01.00.07      No Image      00:03:00:00

Finished Processing Commands Successfully. 
Exiting SAS2Flash.

SUCCESS – the firmware is now fully upgrade!

On a whim I decided to try the testlsall command again to see if suddenly all my problems were done and viola – it can see ALL THE DRIVES!

$ sudo ./sas2flash -o -testlsall
 LSI Corporation SAS2 Flash Utility
 Version 20.00.00.00 (2014.09.18)
 Copyright (c) 2008-2014 LSI Corporation. All rights reserved
     Advanced Mode Set
     Adapter Selected is a LSI SAS: SAS2116_1(B1)
     Executing Operation: Test Link State ALL
     Phy 0: 6.0 Gbps
     Phy 1: 6.0 Gbps
     Phy 2: 6.0 Gbps
     Phy 3: 6.0 Gbps
     Phy 4: 6.0 Gbps
     Phy 5: 6.0 Gbps
     Phy 6: 6.0 Gbps
     Phy 7: 6.0 Gbps
     Phy 8: 6.0 Gbps
     Phy 9: 6.0 Gbps
     Phy 10: 6.0 Gbps
     Phy 11: 6.0 Gbps
     Phy 12: Link Down
     Phy 13: Link Down
     Phy 14: Link Down
     Phy 15: Link Down

Test Link State All PASSED!     
Finished Processing Commands Successfully.
Exiting SAS2Flash.

Right before I decided to get “WORLD’S SMRTEST MAN” tattoo’ed on my chest, I thought to check one more thing, the GParted ‘Disks’ app in Ubuntu, can it really see these drives?

gparted display of HBA disks

My happiness literally knows no bounds at this point, we DID IT!

Conclusion

This entire journey was probably a 4hrs process – I learned a lot along the way which made it particularly enjoyable and of course we landed the plane successfully at the end.

It appears if your HBA card is working, it really does seamlessly pass through the connectivity of the drives to the host system as if they were local (like having 18 SATA connections on your computer).

I truly hope this writeup helps someone out there that got stuck just like I did trying to understand all of this.

23 thoughts on “Linux (Ubuntu), ASUS Z590, LSI 9201-16e HBA Card and how the hell you get it all working together”

  1. Hello,
    just a simple question.
    Aside from RAID/HBA, does Ubuntu work fine on ASUS Z590 and i7-11700k?
    Did you update the Linux kernel or did it work fine with the 5.8 kernel?

    Thank you, Andrea

    Reply
    • Andrea – it works wonderfully smoothly. I’m using the stock kernel in Ubuntu 20.04.2 – I believe when it initially installed it was 5.10 (and upgraded to 5.11) – but yes, everything worked fine out of the box even before the updates were picked up.

      Reply
        • Yep exactly – that was the big reason I wanted to go with the Intel build – so I didn’t need to try and put a graphics card in my Chia miner (in this case). The only other option would have been a G-series AMD cpu but they are impossible to find and the ones you can find are 2 generations old and would have been poor miners for my use.

          Reply
          • I’m running an AMD Ryzen 5 3600 with no GPU in the system just fine, on Debian. Of course during setup you’ll probably want/need a GPU (originally set this system up with an Intel CPU long ago) and you can just slap any old GPU in there for that short period of time.

            Though from your screenshot above it looks like you may be running Ubuntu with a DE so you’ll have issues with X or Wayland (whatever Ubuntu is on now) if there’s no GPU attached.

            Reply
  2. Total Linux noob here, your tutorial is a lifesaver! Thank you so much! Back to that part about me being a Linux noob 🙂 I couldn’t get it to work at first because I didn’t “sudo update grub” after making the changes. Once I did, worked like a charm. May think about adding that to the post for us total noobs. Seriously, thank you so much for this, wasted more hours than I’d like to admit before finding your post.

    Reply
  3. Well done Daniel, thanks for sharing this.Much appreciated – saved a lot of time for me.

    The update command is incorrect, it’s – sudo update-grub

    Reply
  4. Crikey mate!!!!.

    You have made my day!. I did make it on my own to getting the sas controller recognized in lspci but then didn’t have a clue where to go from there, I was in a whirlpool of trying to find drivers and couldnt even find the P20 firmware/installer on the broadcom website that I saw many people talking about on other forums. My card was brand new so only had firmware v9 on it !.

    Following your instructions I literally couldnt believe it when I checked disks and saw my drives just like yourself…… Thankyou for saving me probably 10x the time that you put into setting yours up and writing this blog!.

    Reply
      • Sadly I spoke to soon. I rebooted after filling my HP Storageworks D2600 up with drives and cannot see them anywhere.

        I downloaded/installed SAS2IRCU which does see the enclosure and all drives listed but I have zero idea how to actually access them or do anything with them through ubuntu and the 9201-16e….. I am thinking that I may need a HP smart array card to get in: I will paste the first couple of entries of the SAS2IRCU showing enclosure and first drive listing:

        Initiator at ID #0

        Device is a Enclosure services device Enclosure # : 2
        Slot # : 0
        SAS Address : 5001438-0-204e-58be
        State : Standby (SBY)
        Manufacturer : HP
        Model Number : D2600 SAS AJ832A
        Firmware Revision : 0148
        Serial No : GUID : N/A
        Protocol : SAS
        Device Type : Enclosure services device

        Device is a Hard disk Enclosure # : 2
        Slot # : 13
        SAS Address : 5000cca-0-1aa1-af41
        State : Ready (RDY)
        Size (in MB)/(in sectors) : 2861588/5860533167
        Manufacturer : HITACHI
        Model Number : HUS723030ALS640
        Firmware Revision : A222
        Serial No : YHJVXJXG
        GUID : 5000cca01aa1af40
        Protocol : SAS Drive
        Type : SAS_HDD “

        Reply
          • I have them all mounted now. I bought myself a HP 822 smart array hba which then allowed me to use the HP smart array software to configure all the drives into individual single drive arrays (which all then popped up in disks straight away)

            I’m sure there must be some software that I was missing for the 9201-16 to set what it does with the drives as it was seeing them all there as ‘ready’ on the sas network but no config in between telling ubuntu what they were and what to do with them.

            Reply
  5. I did see the 1 drive popup in disks after following your guide though which is weird…..but after rebooting and filling the enclosure with drives…..nothing. Maybe the enclosure is waiting for array info before it serves multiple drives?. who knows.

    Reply
  6. Huge Thanks Mate for posting this detailed and well written SAS guide. Like many, my card had ancient firmware. Once update to the latest firmware, all 16 drives magically appeared and running perfect.

    Reply
  7. Did you need to do any driver installs?

    My controller shows latest firmware and drives show on boot. But after 10 min or so they disappear.

    Reply
    • No driver installs on my side – just the firmware update and then it started acting correctly.

      Given the behavior your seeing (they work and then after 10mins they disappear) I’m wondering if it’s some power-saver/spin down setting on the hard drives you might be experiencing? Like the drives are going to sleep and straight up going off line.

      Can you try and keep using the drives (Reading/writing to them) just random I/O and see if they stay visible for longer than the 10mins? If so, then I guess it’s probably a power saver mode/bug in the kernel or the hardware device that is causing the issue.

      Reply
    • This could be an issue related to heat/temps. These cards are designed to be in server chassis with fans in the front/back of the box running constantly at 5,000 RPM. I strongly recommend running a fan at max (or as loud as you can handle) and point it directly at the card. Even without a lot of IO (for me no more than 500MB/s) these cards get burning hot to the touch.

      Could also be a hardware issue or something. I’m no expert on these cards

      Reply
  8. i have lsi-9400-16i , i tried to connect a lot of sas expanders in raw but it can see only 4 on one port. anothers port can see only 1 expander in raw, if i connecting next one , even first expander on port is losing disks… doc says i can connect up 1024 drives with expanders, but that is not true. only 140 drives for me..

    Reply

Leave a Comment