I’m looking to replace my sff J5040 Wyze machine. Its still plenty fast enough, but storage has become an issue with its limited USB endpoint availability of ~50 device limit.
I know that just switching it up to a newer Intel system could give me double the endpoints because of the two XHCI chip setup, but I was thinking that if I’m going to replace it, I’d like to not limit myself.
As such, even though Ryzen is far faster than I need, it does now support USB4. Does anyone know if the switch to USB4 would give the system a larger address range and have more than 127 USB devices or is that limitation still in place and I might as well not waste my money?
It’s unfortunate that the conversation has been derailed by people advising me on “better” implementations, so should probably summerise my journey that has got me to this point.
I started home labbing and self-hosting over 25+ years ago, with a large HDD connected to my home PC, HDMI to the TV and ripping DVDs. The disk was probably PATA back then. Yeah, single disk and it probably died at some point making me realise I needed backups in future. I replaced it with a dedicated server I build in an ITX case. A four disk CHENBRO ES34069, back in the good old Athlon days. Each one of those disks was SATA directly connected to the motherboard. And it did the job, except for getting extremely hot and I had a number of disk failures over the years.
Looking back, I can’t guarentee that it was the heat from the system that caused premature failures rather than 2008 era disks just not being as reliable as they are now. But it was hot, and that could not have helped and I had a number of disks over the years that failed in various ways.
I learnt about RAID, and ADM on Linux making arrays and generally not losing any data any more.
My upgrade on that was required as I find that storage requirements outpace what I have, so it ultimately was replaced with a full tower system. A Zalman MS800. At this point I tried to go the whole way. SAS controller cards! Silverstone SAS/SATA hot swap drive bays! RAID disks!
This is when I learnt that I don’t like large RAID arrays, when scrubs took several days and either extending/upgrading/replacing a disk took forever and started using 4 disk arrays in parallel with mergerFS overlaid. Honestly that was the best discovery and its not lost me anything in over a decade.
But SAS became a hassle. I had a controller card fail at one point, so I picked up a replacement which turned out to be a fake, and I’ve had to replace an 8087 cable a couple of times. Its hot, I’ve gone through a few replacement drive bays and they’ve all had small cheap fans on the back of them that over heat, make noise, and don’t do a great job of shifting heat when its part of the same case as the rest of the computer. So I’ve investigated alternatives.
The current solution has evolved to now have 4 “TERRAMASTER D4-300” external storage devices. These are probably the best I’ve come across, they fully expose the disks to the hosts including SMART and even the full correct serial numbers. The speed is absolutely not an issue and I find the benefits of SAS have been completely overstated when dealing with 5/10GB USB connections. I can do a full scrub of a 4 disk RAIDZ over night. These have large fans on the back, and I’ve never had disks this cool when dealing with internal storage, SMART is much happier. You are allowed to have your own opinions, but honestly I’ve been using this setup for a long time and I have no regrets. It is much simpler, and the limitation is very much on the speed of the storage device rather than the type of cable.
The only issue is older Intel gimping the USB enumerator to only allow 50 endpoints, which is 25 devices (less if you have hubs), and even on newer systems they only doubled it.
Let me know why you think my set up is wrong, and I’ll explain how I’ve probably tried your way and I don’t care for it.
Those 2008 disks weren’t great, but airflow advice at that time wasn’t great either.
I can understand how you got where you are, but my concern is that your raidz pools will get more slow as the shared bus fills. You might be able to mitigate this by distributing your pools across different root hubs.
I get the point that external devices will have less heat entrapment, but laying your components in your case to optimize air flow will keep your drives cooler and extend their life. Also, I personally have a lot of risk in my house due to children and pets and can’t imagine leaving my devices out like that.
What size drives and what quantity are you considering?
Just wanna throw in a voice saying your setup sounds completely fine to me. Maybe it’s a bit odd but it also sounds like how I’d do it if I had storage needs that large.
My current storage needs are currently met with a 2.5" SSD connected to a raspberry pi shared with samba over WiFi though so I’m pretty sure every storage nerd in here is gonna tell me my opinion doesn’t count, take it with a grain of salt.
Thank you for the extra context. It’s relieving to know you don’t just have a bunch of USB “backup” drives connected.
To break this down to its simplest elements, you basically have a bunch of small DASes connected to a USB host controller. The rest could be achieved using another interface, such as SATA, SAS, or others. USB has certain compromises that you really don’t want happening to a member of a RAID, which is why you’re getting warnings from people about data loss. SATA/SAS don’t have this issue.
You should not have to replace the cable ever, especially if it does not move. Combined with the counterfeit card, it sounds like you had a bad parts supplier. But yes, parts can sometimes fail, and replacements on SAS are inconvenient. You also (probably) have to find a way to cool the card, which might be an ugly solution.
I eventually went with a proper server DAS (EMC ktn-stl3, IIRC), connected via external SAS cable. It works like a charm, although it is extremely loud and sucks down 250w @ idle. I don’t blame anyone for refusing this as a solution.
I wrote, rewrote, and eventually deleted large sections of this response as I thought through it. It really seems like your main reason for going USB is that specific enclosure. There should really be an equivalent with SAS/SATA connectors, but I can’t find one. DAS enclosures pretty much suck, and cooling is a big part of it.
So, when it all comes down to it, you would need a DAS with good, quiet airflow, and SATA connectors. Presumably this enclosure would also need to be self-powered. It would need either 4 bays to match what you have, or 16 to cover everything you would need. This is a simple idea, and all of the pieces already exist in other products.
But I’ve never seen it all combined. It seems the data hoarder community jumps from internal bays (I’ve seen up to 15 in a reasonable consumer config) straight to rackmount server gear.
Your setup isn’t terrible, but it isn’t what it could/should be. All things being equal, you really should switch the drives over to SATA/SAS. But that depends on finding a good DAS first. If you ever find one, I’d be thrilled to switch to it as well.
What kind of speeds do you get, and how much storage approx?
I’ve gone the route of raiding usb drives before, but 5/10/40gb ports didn’t exist so it was always slow and not worth it.
Sounds like you basically have a DAS that connects over usb, that’s pretty cool if it works well.
Oh also, drives have become a lot better at handling heat or at least are more reliable imo. If you can, try to stick to drives rated to go in a nas. I used to have drives failing all the time, but not so much over the last few years.