An Exploration Into Unifying Cloud Storage With MooseFS

I figured some of you might be interested in some of the experiments I’ve been playing with lately.

I’m trying to solve a problem:  Data redundancy and resiliency in a cloud based storage model.  Also, while I’m at it, storage scalability.

This is surprisingly a tough nut to crack in 2025 when I’m writing this!

My home lab data storage practices are solidly built on dual parities and dozens of drives.

But, in the cloud, I have a different philosophy.  I simply don’t have the budget for expensive disk redundancy and oodles of storage to backup every single thing!

Truth be told, most of my cloud data is easily recreated (i.e. virtual servers) and/or copies of the “important” data are also stored elsewhere (i.e. my home)

Meaning it could theoretically be recreated.

But, that’s a pain.  Especially when you’re like me, living on the fringe of the internet.

It’s taken me months of data transfer time to get the data I wanted into the cloud!

Simply put, I don’t have a parallel to parity based storage in the cloud.  This has become of greater concern as I’ve moved more and more data into my private cloud based storage.

There really aren’t “great solutions” to build something like a NAS or SAN (network based storage) out of disparate storage and technology providers out there.

So, I decided to play around with a distributed file system (specifically, MooseFS), just to see what it could do over the internet.

This article discusses that exploration.

The Basics Of MooseFS & Distributed File Systems

In the event that you’re not familiar with distributed file systems (DFS) or MooseFS, let’s discuss a few of the very basics.

A file system is essentially the structure and format of data used by machines.  You can think of this like files and folders on a computer.

A distributed file system is where you distribute those files and folders across multiple drives.  These can be on the same machine or even across continents.

This image shows individual drives and also virtual machines involved in my initial DFS array built on MooseFS

A very simple way of thinking of DFS is like you are making a “giant” hard drive, but across multiple hard drives on entirely different hardware.  The DFS is basically the “glue” that holds it all together.

It heavily uses networks (either private or the internet) to send data among each of the machines involved in the distribution.

MooseFS is basically a flavor or brand within those distributed file systems.  A few key features?

• Will duplicate data automatically, ensuring data redundancy (e.g. RAID, but not)
• Allows the recreation of data in the event that a hard drive fails
• Will automatically distribute data across all drives to make each drive have an even percentage of the total data
• Allows granular controls over where the data is created and stored, including drive types or locations

The primary use of DFS is in super computing and very large scientific data sets.  Often, the data in these applications is simply “too big” to fit on a single drive.

But, it can also be used for other things.  Like in my case, trying to achieve safety of data so I’m not subject to losing data if hardware fails.

A Primer On Acceptable Risks

Before I get too far, let’s just say that I’m willing to violate all sorts of “best practices” when it comes to my personal data.

The data in question (at least for this discussion) is not “mission critical.”  Typically, I am talking about things like media (i.e. photos, music, some video) or secondary backups of data.

For my mission critical data, I have actual “official” backup practices and follow best practice “321 principles.”

Also, let’s keep in mind here that performance is not my foremost goal.

What I’m trying to solve for here is the ability to survive both individual drive and entire server failures.  I’m not trying to build a super-computer cluster with high speed access to data clusters.

So, I realize that much of what I’m doing in this article blatantly violates the best practices set forth by MooseFS.  This includes and is not limited to:

• MooseFS recommends against virtualizing their servers, but I’m doing exactly that.
• MooseFS recommends using ZFS/XFS and bare metal drive access.  I’m not doing that, but may do that in the future.
• I do not have high speed, 10gbps network between storage sources.  Sometimes I have as little as 100-300mbps!
• Likely many other examples.  Bottom line?  I can sacrifice performance if I can achieve my underlying goals.

So, just keep in mind that the “advice” you see here is not official and is generally not recommended.  Don’t do this at home, kids!

Selecting A Distributed File System

In my research of things to try, I was quite surprised to not find a “purpose built” tool for unifying cloud based storage.

There’s just nothing out there.

So, it looked like I’d need to use a more traditional distributed file system (DFS, henceforth) as the basis of my testing.

Examples of various DFS platforms include CEPH, LizardFS, MinIO,  GlusterFS, Lustre and countless others.

I looked into pretty much all of them.

Some really weren’t what I wanted.  Some were prohibitively expensive.  Many lacked any support.

If you’ve looked at current day DFS, you know there’s a lot of aging/unsupported platforms these days.  (Such as GlusterFS and LizardFS.)  These didn’t seem sensible to base a modern storage solution around.

There’s also “new-fangled” systems (like MinIO) that are entirely based around block storage (i.e. S3 compatible) access, which is not what I need.

So, after a decent review, I settled on MooseFS.

MooseFS is one of the most mature distributed file systems out there that’s both open source (i.e. free) and also actively supported.

For the complex problems something like MooseFS is trying to solve, I found it remarkably easy to both set up and administer.  They’ve clearly done a ton to simplify the solution and make it “just work.”

Documentation wise, it’s fairly well done.  Their documentation is kind of a mess, and in a ton of different places, but everything I’ve needed to know can be found.

I set up a few test instances to proof the concept of whether it “could” work.

Then, I tore all that down and built a “real” instance, with the goal of getting data on it.

The Basics Of My MooseFS Design

For a quick primer of what this looks like, let’s discuss what resources I have at my disposal.

I have a few rather inexpensive (i.e. super low end, old potato) dedicated servers, typically loaded with storage.

I also have one somewhat “beefy” server, also loaded up with some storage.

All my physical servers run Proxmox.  I hate the idea of “bare metal” as it’s often such a waste of good hardware and resources.

In addition, I have several VPS (virtual private servers) and some VPS that have access to block storage.

Ideally, I want all of these resources entirely accessible via MooseFS, just in case I want to use them.

Overall, here’s a few major points from the design:

• MooseFS servers are all virtualized in cloud based Proxmox servers, using LXC
• I pretty much exclusively operate Debian linux, everywhere, so all the MooseFS servers are Debian 12 based
• Storage sources (i.e. Chunk servers) are drives on the Proxmox servers themselves, I have also used VPS instances or sometimes VPS with block storage access
• All VPS and publicly accessible resources run a firewall (UFW) to only allow access via my private cloud VPN network and trusted IP addresses.
• Everything that goes on the public internet is wrapped in Wireguard VPN to protect the actual transit data.  I do try to “mesh” this VPN so ingress and egress to the my private cloud network is as close (geographically/peer-wise) to one of my peering points as possible.
• All sites have >500mpbs connectivity (but sometimes vary) and also have high transport quantity (20TB+)

Essentially, my goal is to be able to utilize storage resources across the internet, regardless of the provider and underlying technologies involved.

I love that with MooseFS, I can have a VPS be both some sort of “technical function” that I actually need, but then I can also harvest excess storage that I typically am not using for chunk storage.

The Specifics Of The MooseFS Design

I figured I wasn’t going to get away “super easy” from my potential “super geeky” reader.

You’re gonna want to know more, right?

Let’s talk about some of the overall design principles that I’ve used.

Master Server/Metaloggers

MooseFS has a concept called the “master server.”  It is the decider, the king.  The penultimate.

This is a SPOF (single point of failure) and if you loose it, you also loose all the data behind MooseFS.  So, you cannot lose it.

I’m using the CE (community edition) of MooseFS, so I don’t have “true” HA (high-availability) access.

Since I see this SPOF as “critical data,” I back up this server nightly, so I can restore it upon drive failure if needed.  (I may even do hourly captures, since it’s that vital!)

But for the CE edition, MooseFS does have what they call a Metalogger concept.

This can function as a backup Master Server, should you experience catastrophic failure.  It’s “just shy” of an actual active/active and highly available solution.

I have one Master Server and two Metalogger servers on entirely separate hardware.  I may add more metaloggers, just to really beef up the design.

Chunk Server Design

So, in the interest of both “server” and “individual drive” survivability, I pursued a specific design.

Each chunk server on my Proxmox servers has the OS (operating system) installed on the same drive that the storage will be shared from.

I also use one Chunk server per drive that I intend to share.

So, if a physical server has three drives, I have three virtual chunk servers installed on that machine.  If it has six drives, there’s six chunk servers.

This ensures that if I loose a drive, I loose both the OS and the stored data.

The point behind something like MooseFS is that the data can be recreated from data on the other servers and drives!

What you don’t want to do is consolidate all your server’s OS on one drive, then share the storage from multiple drives.  If you lose that singular OS drive, you’re screwed and potentially lose all your data drives, too!

Ensuring Highly Available Data

One of the beautiful things about using Proxmox as the basis for this solution is I could easily create failure scenarios.

I could shut down one or more chunk servers and observe what actually happens with the system.  Specifically, I’m looking for “missing chunks,” which implies the data is no longer available in that failure scenario.  (i.e. It’s dead, Jim!)

I’ve heavily used labels, storage classes and rack configurations to ensure that data is replicated onto entirely disparate servers.

This ensures I can loose an entire server (even with multiple drives!) and the data is still recoverable.

To summarize the concepts I’ve used to make sure I can achieve both drive level and entire server level failures:

• All physical servers (or VPS/VPS with block storage) are defined as “racks” in the master server’s configuration files.  (Using both chunk server IP addresses or entire subnets.)
• I defined a label per physical server.  Each “physical device” (i.e. server) gets its own label.  These are typically using A/B/C/D/E, etc.
• I also have an SSD and HDD based class, so I can create tiered based storage.  (I used S and H labels.)
• Given that I liked SSD based intake (ingestion/cache concept), I also created a special label (using I) to identify the ingestion chunk server.

Combining these above concepts, I can ensure that the individual chunks are always stored on entirely separate server hardware.

I learned, after the fact, that there is some “built in” functionality to achieve this.  It’s controlled in the MooseFS master server configuration files.

However, I found it much easier to deal with via storage classes as I could understand both what I’m telling the system to do and what it actually does.

Developing An Appropriate Data Class For MooseFS

From here, it’s all about your MooseFS classes that you apply to each of the directories you’re storing data in.

I decided I needed a few different ways to treat data:

• HDD Class:  Regular hard drive storage, with 2x copies of the data.  Each copy should exist on a physically separate server.  (i.e. rack)
• SSD Class:  Regular “fast” SSD based storage, with 2x copies of the data.  Each copy should exist on a physically separate server.    (i.e. rack)
• HA HDD Class:  Ultra highly available HDD based storage with 3x copies of the data.  Each copy should exist on an physically separate server.    (i.e. rack)
• Backup HDD Class:  A “backup data” class that would archive data using EC (parity) to save on storage.  Each chunk or parity bit is installed on a different chunk server IP address.  Given how small my MooseFS network is, this cannot be redundant to physical server loss.  (But, I don’t care, because it’s a backup of more important data, anyway.)
• All data classes will use the SSD based “ingestion” (or, cache) server as an initial creation point.  Data will be replicated from there.

Here’s are the examples that I used in my actual server:

SSD Class:
mfsscadmin create -C I -K S,S/[RACK] ssdonly

HDD Class:
mfsscadmin create -C I -K H,H/[RACK]:s hddonly

HDD 3x Class:
mfsscadmin create -C I -K H,2H/[RACK]:s hdd3x

2xHDD + EC Storage Class:
mfsscadmin create -C 2I -K H,H/[RACK]:s -A @4+1,H/[IP]:s -d 1 hddec

I will likely play around with this a fair bit more, so my actual design of data classes may change.  If I think of it, I’ll revisit this article with any major updates.

I’d like to get to a point where I can potentially consider using EC (parity) for a majority of my storage.  This saves a huge amount of data, compared to 2x copies of data.

However, this is also not physically redundant at the server level, unless you have a LOT of physical servers.  I’d need at least five physical servers (with sufficient storage) to use 4+1 parity, to ensure survivability against physical server loss.

Not so easy to accomplish at the hobbyist budget level!

Where The Rubber Meets The Road With Cloud Based MooseFS

So, let’s talk figures, performance and data.

As mentioned, I wasn’t expecting this to be highly performant, but I have definitely learned some things.

Overall, this works surprisingly well, even over the public internet, given my relatively low expectations.

When data is copied to the “array,” MooseFS will dutifully work away at replicating and distributing the chunks.

As a network engineer, this image makes me happy.  My cloud lab is heavily segmented across different networks and it uses the “mesh” VPN network I created for it.  Chunk servers talk directly to other chunk servers, regardless of where they are physically located.  (As opposed to a central server model.)

Surprisingly, it doesn’t always rip all the bandwidth that it possibly could.

It seems rather patient with multiple levels of connectivity (i.e. speed) and differing latency between my various chunk servers.  As expected, low latency works at higher speed.  Higher latency works at lower speed.

It even seems to deal with relatively high packet loss scenarios.

Since I go with “budget” hardware providers, sometimes there are problems at the network level.  It’s just part of the deal with a low budget cloud lab.

When this network loss occurs, I’ll get connectivity errors from the master server.  But, it eventually stops complaining once the network conditions clear up.

But, even in these degraded conditions, the chunks balance across all my servers.  Exactly as intended.

I’ve found the system tolerant to restarts, frequent changes, outages and most importantly, my tinkering.

It does “fall down” in some places, too.

• It’s not blazingly fast, at least compared to native storage.  This impacts write speeds much more than read speeds.  Read speeds are actually more performant than native storage since multiple drives can provide data.
• It really doesn’t like when data is being written to a foreign network.  It’s generally best if the “creation” (-C tab for storage classes) uses “local” (to the site) drives.  When the transfer and chunk replication data is forced over the internet, things move like molasses.
• File ingestion happens much faster with larger files (>5mbyte), whereas many small files can (low kbyte) really bring it to the knees.

Overall, though, I’ve found that using MooseFS is not at all unlike using a native file system.  There are some differences, but I can deal with them to achieve data resiliency.

Surviving Problems With Cloud MooseFS

Curiously enough, shortly after I implemented this solution, I experienced a “catastrophic” loss of one of my virtual servers.

Now, fortunately, I recovered gracefully from that problem and was able to fully restore my Proxmox LV’s.  But, it gave me an opportunity to actually see what happened during a total server failure.

Fortunately, prior to this incident, I had done a lot of testing with test data and various failure scenarios.  I knew how the system would respond, but this opportunity gave me a “real world” look.

As expected, with a “total loss” of an entire server, some of the data was potentially lost forever.

Specifically, in my case, any data stored with EC (parity) was at risk.

Since I’m using 4 data parts + 1 parity part (i.e. 4+1), this means data is stored in four chunks and one parity bit.

To reconstruct this file, you need four of the five chunks.

To get EC based technique to server level failure resiliency, I’d have to have five servers of roughly equal storage capacity.

That’s likely an easy thing at corporate or scientific research levels, but not so much at my cheap home lab budget level!

But, also remember, the data I store with EC are almost entirely backups of existing data.  It’s basically expendable and easily recreated.

All of my “important” data is stored with at least two copies on two different physical servers, thus providing full server resiliency.

Things I Still Need To Work On With Cloud MooseFS

I mentioned a few things previously, but overall, this solution does look like it’s going to work for me.

It’s achieving exactly what I want, which is the potential for a very large data “disk” that combines many different providers and storage technologies.

At the time of this capture, I have 3.2 terabytes available.  When I’m finished, it should be about 20 terabytes!

I love that I can simply install a package and mount the disk virtually anywhere I want.  (Well, except for Windows machines, of course.)

It’s kind of like Nextcloud or Owncloud, but for a file system usable by servers and computers.

There’s a few major things I still need to work on for general integration into my cloud lab.  Like everything in my home and cloud labs, everything is subject to change.

A few things off the top of my head:

• Create an NFS to MooseFS gateway, allowing full NFS access to the MooseFS storage
• Create a CIFS to MooseFS gateway, allowing full NFS access to the MooseFS storage.
• Hopefully fix some of the issues I’ve been seeing
• Move more and more data into the system
• See how it runs over time, find new use cases, etc.

Ultimately, when you use a product against the manufacturer’s recommendations like I am, you are kind of your own.

It’s really up to me to develop best practices for this model.  I’m hoping I can identify the “tunables” that are important for a cloud based design.

Then again, maybe the use case is so far outside the typical scenario that it’ll just be a bit buggy.

That’s fine.  As long as it can provide my most essential functions and also protect my cloud media from hardware failure.  Well, that’s what I’m not paying for.  (Sneaky open source joke…)

A Cool Networking & Cloud Based Adventure

I’ve enjoyed getting deep into the weeds with MooseFS and distributed file systems.

As a network engineer, it’s interesting to see this kind of technology operate over networks, VPN technologies and different hardware.

My plan is to try and use it for awhile to see how it works for me.