https://hub.docker.com/r/sciactive/nephele
In the latest version of Nephele, you can now create a WebDAV server that deduplicates files that you add to it.
I created this feature because every night at midnight, my Minecraft world that my friends and I play on gets backed up. Our world has grown to about 5 GB, but every night, the same files get backed up over and over. It's a waste of space to store the same files again and again, but I want the ability to roll back our world to any day in the past.
So with this new feature of Nephele, I can upload the Minecraft backup and only the files that have changed will take up additional space. It's like having infinite incremental backups that never need a full backup after the first time, and can be accessed instantly.
Nephele will only delete a file from the file storage once all copies that share the same file contents have been deleted, so unlike with most incremental backup solutions, you can delete previous backups easily and regain space.
Edit: So, I think my post is causing some confusion. I should make it clear that my use case is specific for me. This is a general purpose deduplicating file server. It will take any files you give it and deduplicate them in its storage. It's not a backup system, and it's not a versioning system. My use case is only one of many you can use a deduplicating file server for.
Yeah, that could work if I could switch to zfs. I’m also using the built in backup feature on Crafty to do backups, and it just makes zip files in a directory. I like it because I can run commands inside the Minecraft server before the backup to tell anyone who’s on the server that a backup is happening, but I’m sure there’s a way to do that from a shell script too. It’s the need for putting in years worth of old backups that makes my use case need something very specific though.
In the future I’m planning on making this work with S3 as the blob storage rather than the file system, so that’s something else that would make this stand out compared to FS based deduplication strategies (but that’s not built yet, so I can’t say that’s a differentiating feature yet). My ultimate goal is to have all my Minecraft backups deduplicated and stored in something like Backblaze, so I’m not taking up any space on my home server.