Skip to content

Mirroring Repositories with Rsync

Rsync allows you to create and maintain a local mirror of CIQ repositories. This is useful for air-gapped environments, reducing external bandwidth usage, and maintaining consistent content across your infrastructure.

Prerequisites

  • rsync installed on your system
  • Your username and access token from https://portal.ciq.com/token
  • Sufficient disk space for the repositories you want to mirror
  • Network access to rsync.depot.ciq.com (port 873)

Finding the Rsync URI

  1. Log in to https://portal.ciq.com
  2. Navigate to My Products and select a product
  3. Select the repository you want to mirror
  4. Look for the Rsync URI in the repository details

Basic Usage

RSYNC_PASSWORD='YOUR_TOKEN' rsync -avSHP \
  rsync://YOUR_USERNAME@rsync.depot.ciq.com/REPO_PATH \
  /local/mirror/path/

Replace:

  • YOUR_TOKEN: Your CIQ Portal access token
  • YOUR_USERNAME: Your CIQ Portal username
  • REPO_PATH: The repository path from the Rsync URI in the Portal
  • /local/mirror/path/: Your local destination directory
  • -a: Archive mode (preserves permissions, timestamps, etc.)
  • -v: Verbose output
  • -S: Handle sparse files efficiently
  • -H: Preserve hard links
  • -P: Show progress and support partial transfers
  • --del: Remove files from destination that no longer exist in source (for an exact mirror)

Example

Mirror a Rocky Linux repository:

RSYNC_PASSWORD='YOUR_TOKEN' rsync -avSHP --del \
  rsync://YOUR_USERNAME@rsync.depot.ciq.com/lts-9.2:rocky-baseos-9.2.x86_64 \
  /var/www/html/repos/rocky-9.2-baseos/

First Sync

The initial sync can take significant time depending on repository size and network speed. Subsequent syncs are much faster as rsync only transfers changes.

Scheduling Regular Syncs

1. Create a Sync Script

Create /usr/local/bin/sync-ciq-mirror.sh:

#!/bin/bash
# Sync CIQ repository mirror

RSYNC_PASSWORD='YOUR_TOKEN'
export RSYNC_PASSWORD

RSYNC_URI="rsync://YOUR_USERNAME@rsync.depot.ciq.com/REPO_PATH"
LOCAL_PATH="/var/www/html/repos/your-repo"
LOG_FILE="/var/log/rsync-ciq-mirror.log"

mkdir -p "$LOCAL_PATH"

echo "=== Sync started at $(date) ===" >> "$LOG_FILE"

rsync -avSHP --del "$RSYNC_URI" "$LOCAL_PATH" >> "$LOG_FILE" 2>&1

if [ $? -eq 0 ]; then
    echo "=== Sync completed successfully at $(date) ===" >> "$LOG_FILE"
else
    echo "=== Sync failed at $(date) ===" >> "$LOG_FILE"
fi

2. Make It Executable

sudo chmod +x /usr/local/bin/sync-ciq-mirror.sh

3. Schedule with Cron

sudo crontab -e

Add a line for your preferred schedule:

# Daily at 2 AM
0 2 * * * /usr/local/bin/sync-ciq-mirror.sh

# Every 6 hours
0 */6 * * * /usr/local/bin/sync-ciq-mirror.sh

# Weekly on Sunday at 3 AM
0 3 * * 0 /usr/local/bin/sync-ciq-mirror.sh

Serving the Local Mirror

After syncing, configure a web server to serve the repository to your systems:

  1. Configure your web server (Apache or Nginx) to serve the mirror directory
  2. Create repository configuration on client systems pointing to your mirror:
[local-ciq-mirror]
name=Local CIQ Mirror
baseurl=http://your-mirror-server/repos/your-repo/
enabled=1
gpgcheck=1

What if the Repository Doesn't Have the Rsync URI Option?

If a repository doesn't show the Rsync URI option, it is not enabled for rsync mirroring. In this case:

  • Check if the repository supports reposync instead
  • Contact CIQ support at support@ciq.com for alternative mirroring solutions

Troubleshooting

Authentication Failures

Problem: Rsync fails with authentication error.

Solutions:

  • Verify your access token is correct and current
  • Ensure the RSYNC_PASSWORD environment variable is set
  • Verify your organization admin hasn't regenerated the token
  • Test with a simple listing command: 
    RSYNC_PASSWORD='YOUR_TOKEN' rsync rsync://YOUR_USERNAME@rsync.depot.ciq.com/

Network Issues

Problem: Can't connect to rsync.depot.ciq.com.

Solutions:

  • Check firewall rules allow port 873 (rsync)
  • Verify DNS resolution: nslookup rsync.depot.ciq.com
  • Test basic connectivity: ping rsync.depot.ciq.com

Disk Space Issues

Problem: Running out of disk space.

Solutions:

  • Check available space before syncing: df -h /var/www/html/repos
  • Monitor repository sizes: du -sh /var/www/html/repos/*
  • Consider syncing only specific architectures if space is limited

See Also