Update: I just re-used this guide on a new LSI SAS 9200-16e I got off of eBay and it worked perfectly; when I plugged the card in it couldn’t see any drives, but after wiping it and reflashing the newest bios (instructions below) it suddenly saw all the drives and I was back in action!
I hope to save you a lot of time if you are new to the world of HBA cards and external drive enclosures.
Quick Reference
- You might run across the phrases “HBA Adapter” or “RAID Card” used interchangeably in forums and posts – they aren’t necessarily interchangeable terms but are highly related, so the nuances are:
- “HBA” – “Host Bus Adapter” – this is a PCIe card that is installed in the host machine and then connected to an external enclosure of hard drives that will allow you to “see” those drives. For the most part, these adapters are usually ~$100 and behave as “dumb pass throughs” – doing nothing besides making the drives ‘seeable’ from the host machine. As a literal example, if you had an external enclosure with 4 hard drives in it and 1 hard drive in your hose machine and connected it via HBA – if you looked in Windows Explorer, you would see 5 hard drives (C, D, E, F and G).
- “RAID” – “Redundant Array of Independent Disks” – on the high end this will be a PCIe card that is installed in the host machine and EITHER connected to drives inside the host machine or connected out to drives housed in an external enclosure. These cards can be upwards of $300, 500 or $1500 depending on how fancy/fast/high end you want to get. These cards have robust/powerful chips and firmware running on them that implement many or all of the different RAID standards. These cards are manipulating not only how the host system “sees” the drives but also how the drives are managed and data read/written to them. Depending on the RAID setup you use, your host system might see one drive or many drives – it’s all about the data resiliency/performance strategy you are trying to accomplish.
- “LSI” is one of the favorite brands for HBA cards – as far as I can tell the cards everyone loves are all from 2012-2016 so yes it’s “old hardware” but it also works perfectly fine; don’t get nervous and think “I better search for the NEW version of this” – it’s either not necessarily or there isn’t one. I haven’t dug enough to tell which.
- “JBOD” is a generic term that literally means “Just a Bunch of Disks” – it refers to either an enclosure or a setup whereby a large number of hard drives are being housed in an enclosure that isn’t doing anything fancy with them – no RAID, just straight pass through. When you go shopping for mid to large scale drive enclosures (I saw this mostly in rack-mount setups) you need to differentiate if you want the enclosure to be more complex and have a RAID controller on it or if it just wanted a simple “JBOD” type enclosure which had very simply sets of outbound ports on it to connect to a hosting machine.
My Build
- Intel i7-11700k
- ASUS TUF Gaming Z590-PLUS (non-wifi)
- LSI 9201-16e Card from eBay
- PC Pitstop 12-bay, SATA External Enclosure
- 12x 14TB SATA drives
The Adventure Begins…
I have very little experience with RAID cards – I’ve done both software RAID setups on Linux (mdadm), used the built-in support on a few consumer motherboards in the past and 1 time set it up in a server – but never really bathed in it’s glory.
I’ve also never had an HBA card in a machine I’ve built before, so I wasn’t entirely sure what to expect here.
Once I got my build together, I installed Ubuntu 20.04 LTS on it and threw the LSI 9201-16e card in it and rebooted.
Problem #1 – Kernel Driver Bug
First things first, upon reboot I didn’t see any change to my boot-up sequence… no supplemental messaging from the card after my computer POST’ed saying something like “Press CTRL-C to enter HBA setup…” – it would just boot straight into Ubuntu.
Like I said – never worked with an HBA card before – so I’m not even sure… how to use it?
In Ubuntu I spent longer than I would like to admit trying to figure out how to… access the card? Use the card?
I wasn’t totally sure – lots of posts mentioned using the “StorCLI” or “MegaCLI” command line utils from Broadcom’s site (that were last published in 2014 or 2016)… I ended up pulling those down and converting the RPM’s into DEBs before installing them to no avail… none of the tools could find the card.
I eventually got smart and decided to start grepping dmesg (the startup log in Linux for your machine)
sudo dmesg | grep mpt2sas
<... snipped a lot of output ...>
"mpt2sas_cm0: failure at drivers/scsi/mpt3sas/mpt3sas_scsih.c:10791/_scsih_probe()!"(I originally grepped for sata and then sas before noticing the new module that controlled the card was named mpt2sas)
Ahhh ok so the driver isn’t loading properly – that’s probably what is going on.
After a lot of Googling, I discovered this issue has been in the Linux kernel since the middle of the 5.8.x series and at the time of this writing (May 2021) still is not fixed.
Fortunately there are 2 workarounds – I prefer the 1st one:
- Limit the Queue Depth directly via the kernel arguments.
- Downgrade back to Kernel 5.7.x
As for the solution for #1, you do this by editing your Grub configuration (on Ubuntu 20.04 at least – please adjust instructions for your favorite distro)
sudo nano -w /etc/default/grubNow you want to edit around line 11 or so and a suitable value for the mpt3sas.max_queue_depth argument.
NOTE: All the references to this I found online clarify setting a queue depth of 10,000 fixed their errors – I continued to have errors and even thought my arguments weren’t getting picked up until I tried 8,000 and suddenly it started working. So you MAY need to play with this value a few times.
Per the NOTE above, my edited line ended up looking like this:
#GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
GRUB_CMDLINE_LINUX_DEFAULT="mpt3sas.max_queue_depth=8000"I always duplicate and comment out any line I’m editing in Linux configs so I can always refer to the original contents.
Full disclosure – I didn’t know if I could append the mpt3sas setting to the quiet splash list or not, so I just made it the only argument. Please feel free to do your own testing with those other values included – there is no particular reason I needed to remove them.
After saving the file, you need to run the following command to have Grub update itself with the config change and generate new init files to read on boot (thx Daniel & Saul!):
sudo update-grubNow go ahead and reboot – and VIOLA, now Linux could see the device and initialization during bootup was working as expected (screenshot provided for ease of glancing):
A quick peek at LSHW and I can confirm the card exists in this realm of reality:
sudo lshw
< scrolling down 100s of lines... >
Excellent, now the card must be working and I can see all 12 hard drives in the external enclosure, right?
No luck – it’s also worth noting again – I have no experience with HBAs – so I’m not sure if I’m even expecting the right thing… will I just suddenly see drives popup in Nautilus (Ubuntu’s File Explorer), or more devices under /dev/sdXXX or is my external storage enclosure even working?
At least at this point I figured the card was working and my war was no longer with Linux but rather with understanding the card, how it is suppose to work and how I get it to do my bidding.
Problem #2 – HBA Firmware
I spent the next hour or so reading through forums, hoping for posts from clueless people like myself asking what is suppose to happen on boot.
DISCOVERY: It appears these LSI 9201 cards can optionally have a BIOS on them that introduces additional behaviors that cause a boot-prompt after POST’ing to show up and allow you to configure them. I also found most people prefer to ERASE the BIOS which allows the computer to boot faster and the card has a default “IT Mode” it operates in, where it just passes through external connections.
Interesting – but helpful!
So it appeared me NOT seeing a messaging during boot-up from the card was fine and signaled that the card didn’t have one of those fancier BIOSs installed.
That did send me down the path of wondering how old the firmware was on this device – it’s probably an 8 or 10 year old piece of hardware and the last firmware was released around 2015 or 16 (P20.00.00.70) – how could it NOT have the latest firmware on it if it was pulled out of production?
I decided to navigate over to the Broadcom official site to find the latest downloads for this card.
This site is not the most intuitive, so to help you out here is a screenshot of the search criteria you should use to find download resources for this Legacy piece of hardware – if you have newer hardware, you might need to adjust this:
(Here is a link to click also)
What I ended up learning is that you need to download 2 pieces of software:
- The utility used to talk to the card / flash the ROM – in my case it was
Installer_P20_for_Linux(link) – version 20.00.00.00 - An archive that contains the actual latest firmware – in my case it was
9201-16e_Package_P20_IT_Firmware_BIOS_for_MSDOS_Windows(link) – version 20.00.00.70.
Go ahead and unzip the 1st package (Installer_P20) – it’ll contain a subdirectory named sas2flash_linux_i686_x86-64_rel – inside is a file named sas2flash that you’ll have to make executable:
chmod +x sas2flashOk now that the utility is executable, it can actually be used for a lot of things related to interacting with the card besides flashing it – for example – let’s check the status of our card and make sure the utility can find our card by using the listall command:
$ sudo ./sas2flash -listall
LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved
Adapter Selected is a LSI SAS: SAS2116_1(B1)
Num Ctlr FW Ver NVDATA x86-BIOS PCI Addr
0 SAS2116_1(B1) 13.00.60.00 0d.43.00.10 No Image 00:03:00:00
Finished Processing Commands Successfully. Exiting SAS2Flash.Oh, holy HELL, v13.00.60.00 of the firmware?! Well now I know it’s not fully upgrade so let’s hope that does the trick.
TIP: I actually found the sas2flash command reference PDF (link) helpful to glance through.
One more thing I want to check – the sas2flash tool actually lets you query the link status of the HBA – so I can ask it directly if it see the drives in the external enclosure using the testlsall command – this is considered an Advanced Mode command so you have to force it with the -o argument (again, I’m not genius, I just read through the PDF and eventually figured this out)
$ sudo ./sas2flash -o -testlsall
LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved
Advanced Mode Set
Adapter Selected is a LSI SAS: SAS2116_1(B1)
Executing Operation: Test Link State ALL
Phy 0: Link Down
Phy 1: Link Down
Phy 2: Link Down
Phy 3: Link Down
Phy 4: Link Down
Phy 5: Link Down
Phy 6: Link Down
Phy 7: Link Down
Phy 8: Link Down
Phy 9: Link Down
Phy 10: Link Down
Phy 11: Link Down
Phy 12: Link Down
Phy 13: Link Down
Phy 14: Link Down
Phy 15: Link Down
Test Link State All PASSED!
Finished Processing Commands Successfully.
Exiting SAS2Flash.Well that’s not good… I guess it’s not seeing the drives so at least I won’t continue to worry about finding them or how the HBA works right now because obviously something else is wrong.
Believe it or not this actually felt good to me – I was querying the card, so it’s working and it confirmed for me that it doesn’t see the hard drives in the enclosure so I’ll stop worrying about that for now.
Now that we are on a warpath to update the firmware, go back and unzip the 2nd package you downloaded – the one with the firmware (.bin) in it.
Even though the 2nd download says “for MSDOS_Windows”, once you unzip it, you can find the .bin file (that represents the firmware) neatly tucked away under the Firmware directory:
Under that directory you’ll find the HBA_9201_16e_IT directory and in there is the file 9201-16e.bin.
Copy the 9201-16e.bin file into the directory containing the sas2flash binary you just made executable to keep things easy.
TIP: You might need to ‘clear’ the card before flashing the new firmware to it, this can be done with the sas2flash -o -e 6 command where the 6 indicates a “Clean Flash” (Manual Page 14 / Section 5.4.10) – my thanks to Will5001 for this tip!
I then executed the flashing of the firmware:
$ sudo ./sas2flash -f 9201-16e.bin
LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved
Adapter Selected is a LSI SAS: SAS2116_1(B1)
Executing Operation: Flash Firmware Image
Firmware Image has a Valid Checksum.
Firmware Version 20.00.07.00
Firmware Image compatible with Controller.
Valid NVDATA Image found.
NVDATA Version 14.01.00.00
Checking for a compatible NVData image...
NVDATA Device ID and Chip Revision match verified.
NVDATA Versions Compatible.
Valid Initialization Image verified.
Valid BootLoader Image verified.
Beginning Firmware Download...
Firmware Download Successful.
Verifying Download...
Firmware Flash Successful.
Resetting Adapter...
Adapter Successfully Reset.
Finished Processing Commands Successfully.
Exiting SAS2Flash.After that was done I confirmed the firmware upgrade with the listall command again:
$ sudo ./sas2flash -listall
LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved
Adapter Selected is a LSI SAS: SAS2116_1(B1)
Num Ctlr FW Ver NVDATA x86-BIOS PCI Addr
0 SAS2116_1(B1) 20.00.07.00 14.01.00.07 No Image 00:03:00:00
Finished Processing Commands Successfully.
Exiting SAS2Flash.SUCCESS – the firmware is now fully upgrade!
On a whim I decided to try the testlsall command again to see if suddenly all my problems were done and viola – it can see ALL THE DRIVES!
$ sudo ./sas2flash -o -testlsall
LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved
Advanced Mode Set
Adapter Selected is a LSI SAS: SAS2116_1(B1)
Executing Operation: Test Link State ALL
Phy 0: 6.0 Gbps
Phy 1: 6.0 Gbps
Phy 2: 6.0 Gbps
Phy 3: 6.0 Gbps
Phy 4: 6.0 Gbps
Phy 5: 6.0 Gbps
Phy 6: 6.0 Gbps
Phy 7: 6.0 Gbps
Phy 8: 6.0 Gbps
Phy 9: 6.0 Gbps
Phy 10: 6.0 Gbps
Phy 11: 6.0 Gbps
Phy 12: Link Down
Phy 13: Link Down
Phy 14: Link Down
Phy 15: Link Down
Test Link State All PASSED!
Finished Processing Commands Successfully.
Exiting SAS2Flash.Right before I decided to get “WORLD’S SMRTEST MAN” tattoo’ed on my chest, I thought to check one more thing, the GParted ‘Disks’ app in Ubuntu, can it really see these drives?
My happiness literally knows no bounds at this point, we DID IT!
Conclusion
This entire journey was probably a 4hrs process – I learned a lot along the way which made it particularly enjoyable and of course we landed the plane successfully at the end.
It appears if your HBA card is working, it really does seamlessly pass through the connectivity of the drives to the host system as if they were local (like having 18 SATA connections on your computer).
I truly hope this writeup helps someone out there that got stuck just like I did trying to understand all of this.
Hello,
just a simple question.
Aside from RAID/HBA, does Ubuntu work fine on ASUS Z590 and i7-11700k?
Did you update the Linux kernel or did it work fine with the 5.8 kernel?
Thank you, Andrea
Andrea – it works wonderfully smoothly. I’m using the stock kernel in Ubuntu 20.04.2 – I believe when it initially installed it was 5.10 (and upgraded to 5.11) – but yes, everything worked fine out of the box even before the updates were picked up.
Thank you very much.
You’re using Intel’s integrated GPU (UHD 750), right?
Yep exactly – that was the big reason I wanted to go with the Intel build – so I didn’t need to try and put a graphics card in my Chia miner (in this case). The only other option would have been a G-series AMD cpu but they are impossible to find and the ones you can find are 2 generations old and would have been poor miners for my use.
I’m running an AMD Ryzen 5 3600 with no GPU in the system just fine, on Debian. Of course during setup you’ll probably want/need a GPU (originally set this system up with an Intel CPU long ago) and you can just slap any old GPU in there for that short period of time.
Though from your screenshot above it looks like you may be running Ubuntu with a DE so you’ll have issues with X or Wayland (whatever Ubuntu is on now) if there’s no GPU attached.
Thanks again.
Total Linux noob here, your tutorial is a lifesaver! Thank you so much! Back to that part about me being a Linux noob 🙂 I couldn’t get it to work at first because I didn’t “sudo update grub” after making the changes. Once I did, worked like a charm. May think about adding that to the post for us total noobs. Seriously, thank you so much for this, wasted more hours than I’d like to admit before finding your post.
Daniel you made my day reading this – this is exactly the effect I wanted the writeup to have. Also great suggestion, I’ll go make that edit!
Well done Daniel, thanks for sharing this.Much appreciated – saved a lot of time for me.
The update command is incorrect, it’s – sudo update-grub
(face-palm) great catch Saul, fixed! Thank you!
Crikey mate!!!!.
You have made my day!. I did make it on my own to getting the sas controller recognized in lspci but then didn’t have a clue where to go from there, I was in a whirlpool of trying to find drivers and couldnt even find the P20 firmware/installer on the broadcom website that I saw many people talking about on other forums. My card was brand new so only had firmware v9 on it !.
Following your instructions I literally couldnt believe it when I checked disks and saw my drives just like yourself…… Thankyou for saving me probably 10x the time that you put into setting yours up and writing this blog!.
Phil you just made my entire week! I’m so glad this helped!
Sadly I spoke to soon. I rebooted after filling my HP Storageworks D2600 up with drives and cannot see them anywhere.
I downloaded/installed SAS2IRCU which does see the enclosure and all drives listed but I have zero idea how to actually access them or do anything with them through ubuntu and the 9201-16e….. I am thinking that I may need a HP smart array card to get in: I will paste the first couple of entries of the SAS2IRCU showing enclosure and first drive listing:
Initiator at ID #0
Device is a Enclosure services device Enclosure # : 2
Slot # : 0
SAS Address : 5001438-0-204e-58be
State : Standby (SBY)
Manufacturer : HP
Model Number : D2600 SAS AJ832A
Firmware Revision : 0148
Serial No : GUID : N/A
Protocol : SAS
Device Type : Enclosure services device
Device is a Hard disk Enclosure # : 2
Slot # : 13
SAS Address : 5000cca-0-1aa1-af41
State : Ready (RDY)
Size (in MB)/(in sectors) : 2861588/5860533167
Manufacturer : HITACHI
Model Number : HUS723030ALS640
Firmware Revision : A222
Serial No : YHJVXJXG
GUID : 5000cca01aa1af40
Protocol : SAS Drive
Type : SAS_HDD “
Phil just checking in – did you make any progress here? A bit out of my depth (not familiar with the hardware).
I have them all mounted now. I bought myself a HP 822 smart array hba which then allowed me to use the HP smart array software to configure all the drives into individual single drive arrays (which all then popped up in disks straight away)
I’m sure there must be some software that I was missing for the 9201-16 to set what it does with the drives as it was seeing them all there as ‘ready’ on the sas network but no config in between telling ubuntu what they were and what to do with them.
Nice! Glad to hear it!
I did see the 1 drive popup in disks after following your guide though which is weird…..but after rebooting and filling the enclosure with drives…..nothing. Maybe the enclosure is waiting for array info before it serves multiple drives?. who knows.
Huge Thanks Mate for posting this detailed and well written SAS guide. Like many, my card had ancient firmware. Once update to the latest firmware, all 16 drives magically appeared and running perfect.
Most welcome Keith, I’m glad it helped!
Did you need to do any driver installs?
My controller shows latest firmware and drives show on boot. But after 10 min or so they disappear.
No driver installs on my side – just the firmware update and then it started acting correctly.
Given the behavior your seeing (they work and then after 10mins they disappear) I’m wondering if it’s some power-saver/spin down setting on the hard drives you might be experiencing? Like the drives are going to sleep and straight up going off line.
Can you try and keep using the drives (Reading/writing to them) just random I/O and see if they stay visible for longer than the 10mins? If so, then I guess it’s probably a power saver mode/bug in the kernel or the hardware device that is causing the issue.
This could be an issue related to heat/temps. These cards are designed to be in server chassis with fans in the front/back of the box running constantly at 5,000 RPM. I strongly recommend running a fan at max (or as loud as you can handle) and point it directly at the card. Even without a lot of IO (for me no more than 500MB/s) these cards get burning hot to the touch.
Could also be a hardware issue or something. I’m no expert on these cards
i have lsi-9400-16i , i tried to connect a lot of sas expanders in raw but it can see only 4 on one port. anothers port can see only 1 expander in raw, if i connecting next one , even first expander on port is losing disks… doc says i can connect up 1024 drives with expanders, but that is not true. only 140 drives for me..
Hey, nice article, could you add/share photos of the setup?
No photos to speak of.
Oh man. Where was this post 6 hours ago? This has been really helpful. Were you able to ever get the Avago/LSI Bios config utility to load with Ctrl+C during post? Mine wont load at all and just hangs the server up from posting whenever I try and use. My card is an LSI 9207-8e running in IT mode with P20 firmware on:
Num Board Name FW Ver NVDATA x86-BIOS
—————————————————————–
0 H221 20.00.07.00 14.01.00.06 07.39.02.00
In ubuntu 22.04 LTS i am able to see all the individual drives and can create zfs pools but id really have it configured as hardware RAID5 like my internal Dell PERC card does for the internal drives. I loaded storcli (doesnt see controllers) and the MegaRAID storage manager wasn’t able to actually create RAID with physicals and virtual disks. I assume this has something to do with the IT mode the card was flashed into when i bought it off ebay.
Wait how are you getting board stats + drive status via the card if it won’t let the computer POST?
It will as long as i dont hit the Ctrl-C to load the HBA config utility. I was trying to run that to setup RAID. If i skip that it loads fine and i can see all 12 drives. This firmware must not have that ability.
Ah! Gotcha.
If it helps, the LSI cards in ALL of my machines don’t prompt on boot, they aren’t like the PERCs in the Dells – but they are all flashed to IT mode so that _might_ be why, but I’ve always bought them like that so never got the “Ctrl-C” prompt.
In short, I see the same thing you do.
Cool thanks! I think I may just stay IT mode and a zfs pool for a bit. The guy i bought the card from seems to think you can flash it with the 9217-IR version fw on there to give it ability but I dont want to brick the card if he is wrong.
Dude, I have an LSI card in my utility PC at work to deal with SAS drives in making RAID drives for customers. You are the bomb posting this because my Ubuntu now recognizes the LSI card and drives. Kudos to you man!!!
I’m glad it helped!
Hello! I have a similar problem. Controller LSI 9201-16i, latest firmware and bios. Supermicro 24 disk server. SATA disks work fine, while SAS disks are visible through lsblk, but not visible through fdisk, fdisk: cannot open /dev/sdb: Input/output error. Setting the flag mpt3sas.max_queue_depth=8000 did not help. What could it be? And yes, if you boot into Windows, then all disks work fine without problems.
It sounds like a kernel driver problem given that they work in windows but not in Linux. And also given that the SATA drives work but the SAS don’t. As a super quick test, you could try installing a newer kernel version if it’s available or installing a more cutting edge distro to see if it can suddenly see the drives.
When I started on this adventure, I found that the current LTS version of Ubuntu had an older kernel and wouldn’t support my drives correctly. So I had to update the kernel. You might be in a similar boat
Thanks, but my problem is solved. It consisted in the disks themselves, they were formatted with Protection Type 2.
Protection: prot_en=1, p_type=1, p_i_exponent=0 [type 2 protection]
Solution:
sg_format –format –fmtpinfo=0 /dev/sdX
A detailed description of the solution is here:
https://talesinit.blogspot.com/2015/11/formatted-with-type-2-protection-huh.html
In my case the Adapter did not recognize any disks. The Flash Util by Broadcom showed that everything was fine, but it was not. I solved the Problem with the tools from here (lsirec, lsiutil and a python script sbrtool.py:
https://github.com/marcan/lsirec
and
https://github.com/thomaslovell/LSIUtil
I followed the Advices given in the README.md at lsirec Archive to get my Card at work. Take time for this and follow the steps exactly, then it will work.
In my cas the simple flash of new firmware did not solve my problem. But your Site was neverethless very helpful for me.