Optimizing Storage in Media Server Setups: ZFS Expansion and Hardlinking

Sal

— 3 min read

This guide covers two techniques to optimize storage in a media server setup:

  1. Expanding an existing ZFS RAID10 pool with additional drives
  2. Creating hardlinks between download and media directories to eliminate duplicate storage

The Storage Challenge of Media Servers

Media server setups commonly face an inefficient storage pattern:

  • Download directory: Where qBittorrent stores completed downloads
  • Media directory: Where Sonarr/Radarr organize movies and TV shows

By default, when Sonarr/Radarr import media from qBittorrent, they copy files from the download directory to the media directory. This creates duplicate files that consume twice the storage.

For example, a 10GB movie will use 20GB total space: 10GB in downloads and 10GB in the media library. At scale, this becomes a significant waste of storage.

A hardlink is a directory entry that points to the same data as another file. Unlike copies, hardlinks don't duplicate data on disk. If a 10GB file has a hardlink created, it still only uses 10GB of storage, not 20GB.

What is ZFS RAID10?

ZFS RAID10 combines mirroring and striping to provide excellent performance and redundancy. It allows storage expansion by adding pairs of drives while maintaining data protection.

Part 1: Adding Drives to ZFS RAID10

Prerequisites

  • 2 new drives of equal size
  • Proxmox server with existing ZFS RAID10 pool
  • Root access to the Proxmox host

Step 1: Identify the New Drives

# List all drives and their details
lsblk -o NAME,SIZE,MODEL,SERIAL

# Find persistent identifiers
ls -la /dev/disk/by-id/

Step 2: Add the Mirror to Your Existing RAID10

# Replace with actual pool name and drive IDs
zpool add mypool mirror /dev/disk/by-id/drive1-identifier /dev/disk/by-id/drive2-identifier

Step 3: Verify the New Configuration

# Check that the new drives were added successfully
zpool status mypool

Step 4: Optimize the Pool (Optional)

# Enable auto-trim for SSDs
zpool set autotrim=on mypool

# Enable compression if not already enabled
zfs set compression=lz4 mypool

The key to hardlinking is determining where files actually reside. This can be complicated by containerization (LXC, Docker).

For Container-Based Setups (LXC, Docker)

If applications are running in containers, enter the container first:

# For LXC containers in Proxmox
pct enter CONTAINER_ID

# Check mount points inside container
df -h

# Find actual file locations
find /path/to/downloads -type f -size +100M | head -5
find /path/to/media -type f -size +100M | head -5

Implementing the Hardlinking Script

The included hardlinking script performs these functions:

  1. Finds matching files between download and media directories
  2. Verifies content matches between files
  3. Creates backups of media files (optional)
  4. Replaces media files with hardlinks to download files
  5. Processes files in small batches
  6. Provides detailed logging of all operations

The script includes safety features:

  • Dry run mode (enabled by default)
  • Content verification
  • Inode verification
  • Detailed logging

Sonarr/Radarr Configuration

  1. Go to Settings > Media Management
  2. Enable "Use Hardlinks instead of Copy"
  3. Ensure root folders and download paths are on the same filesystem

qBittorrent Configuration

  1. Go to Tools > Options > Downloads
  2. Consider setting up automatic torrent management:
    • Enable "Remove torrents after reaching the share ratio limit"
    • Set a reasonable share ratio limit (e.g., 2.0 for 200%)

Expected Results

After implementing both solutions:

  1. Expanded Storage Capacity: Adding drives to ZFS RAID10 increases total usable space
  2. Eliminated Duplication: Hardlinks ensure files exist only once on disk
  3. Maintained Functionality: qBittorrent can still seed torrents, and Sonarr/Radarr can still manage the library

Important Considerations

  • Hardlinks Only Work Within the Same Filesystem: Files must be on the same ZFS pool
  • Existing Duplicates: The script must be run to convert existing duplicates
  • Container Paths: Pay attention to paths inside vs. outside containers
  • Backup First: Always back up critical data before major storage changes

Conclusion

Combining ZFS expansion and hardlinking can significantly optimize media server storage. This approach increases capacity and eliminates duplication while maintaining full functionality of media server applications.

Adding SSDs to an existing ZFS RAID10 pool and implementing hardlinks between download and media directories can reclaim significant space that would otherwise be wasted on duplicated files.

The included hardlinking script can be adapted for different environments by modifying the paths and settings to match the specific setup.

Read more