History of Webstack - Disk Configuration Management

Concept - Why Webstack

Traditional server configuration leaves files scattered across /etc, /usr/lib/systemd and application directories making it difficult to:

• Track what changed and why
• Mirror configuration between servers
• Recover from mistakes
• See the complete picture in one place
• Webstack consolidates all configuration into a single directory tree under git version control.

Structure - What Goes Where

/etc/webstack/
├── systemd/          ← service files (real files, linked FROM /etc/systemd/system/)
├── nginx/            ← entire nginx directory (real files, /etc/nginx linked here)
├── php8/             ← php config (real files, /etc/php8 linked here)
├── firebird/         ← firebird config (real files, linked FROM /opt/firebird5/)
├── radicale/         ← radicale config (real files, linked FROM /etc/radicale)
├── acme/             ← acme.sh domain configs and dhparams.pem
├── cron.daily/       ← cron scripts (real files, linked FROM /etc/cron.daily/)
├── scripts/          ← admin utilities
└── site-config/      ← machine specific site configs
    ├── srv10/
    ├── srv9/
    └── desktop/

Key principle: Real files live in webstack, symlinks point back to where services expect them.

The Symlink Pattern

Direction matters — symlinks go FROM the service location TO webstack, not the other way:

# CORRECT - webstack owns the file
ln -s /etc/webstack/nginx /etc/nginx
ln -s /etc/webstack/systemd/php-fpm.service /etc/systemd/system/php-fpm.service

# WRONG - git would only track the symlink not the content
ln -s /etc/nginx /etc/webstack/nginx

Moving an existing directory:

mv /etc/nginx /etc/webstack/nginx
ln -s /etc/webstack/nginx /etc/nginx
nginx -t && systemctl restart nginx

Setting Up Git

zypper install git

# Configure identity - use server identity not personal details
git config --global user.name "srv10 admin"
git config --global user.email "root@srv10.rdm1.uk"
git config --global init.defaultBranch production

# Initialise
cd /etc/webstack
git init

.gitignore - What To Exclude

# SSL certificates and keys - never in git
nginx/ssl/**/*.pem
nginx/ssl/**/*.cer
nginx/ssl/**/*.cer.ecc
nginx/ssl/**/*.key
nginx/ssl/**/*.csr
# Radicale user passwords
radicale/users
# Site config contains database passwords
site-config/

Add placeholder files so directory structure is preserved:

find /etc/webstack/nginx/ssl -type d -exec touch {}/.gitkeep ;

And allow them despite the ignore rules:

!nginx/ssl/**/.gitkeep

Committing In Logical Sections

Commit each section separately for clean history:

git add systemd/
git commit -m "systemd service files - moved from /usr/lib to allow version control"

git add nginx/
git commit -m "nginx config - full config including ssl and vhosts"

git add php8/
git commit -m "php8 config - clean baseline, opcache isolation settings"

git add firebird/
git commit -m "firebird config - RuntimeDirectory fix for modern systemd"

git add radicale/
git commit -m "radicale CalDAV server - replacing nextcloud calendar/contacts"

The Bare Repo - Mirroring Between Machines

The bare repo acts as a hub — like a local github but private:

# On srv10 - create the hub
git init --bare /srv/git/webstack.git

# On srv10 working copy - point at hub
git remote add origin /srv/git/webstack.git
git push -u origin production

# On srv9 - clone from hub
git clone ssh://root@srv10/srv/git/webstack.git /etc/webstack

Daily Workflow

# Start of session - get latest from other machine
git pull

# Make changes to config files
# Test they work - nginx -t, systemctl restart etc

# End of session - commit and share
git add .
git commit -m "what changed and why"
git push

Renaming files (disabled configs):

# Use git mv to preserve history
git mv nginx/conf.d/site.conf nginx/conf.d/site.conf.off
git commit -m "site disabled - reason why"

Service File Management

Never edit files in /usr/lib/systemd/system/ — they get overwritten by package updates. Copy to webstack and symlink:

cp /usr/lib/systemd/system/php-fpm.service /etc/webstack/systemd/
ln -s /etc/webstack/systemd/php-fpm.service /etc/systemd/system/php-fpm.service
systemctl daemon-reload

Always run systemctl daemon-reload after adding or modifying service files!

PHP-FPM Isolation

Running multiple sites from one PHP-FPM pool risks APCu cache contamination between applications. Isolate heavyweight applications:

# Clone service for separate pool
cp /etc/webstack/systemd/php-fpm.service /etc/webstack/systemd/php-fpm-nextcloud.service
# Edit to point at separate pool config
systemctl daemon-reload
systemctl enable php-fpm-nextcloud
systemctl start php-fpm-nextcloud

Each pool has its own APCu memory space — applications cannot contaminate each other.

Scripts

/etc/webstack/scripts/restart-stack.sh — restart all services in correct order:

#!/bin/bash
echo "Restarting full web stack..."
systemctl restart firebird
systemctl restart php-fpm
systemctl restart php-fpm-nextcloud
systemctl restart nginx
echo "Done - checking status..."
systemctl is-active firebird php-fpm php-fpm-nextcloud nginx

/etc/webstack/scripts/check-crontabs.sh — audit all scheduled tasks:

#!/bin/bash
# Check all user crontabs
for user in $(cut -d: -f1 /etc/passwd); do
    CRON=$(crontab -u $user -l 2>/dev/null)
    if [ ! -z "$CRON" ]; then
        echo "=== $user ==="
        echo "$CRON"
    fi
done
# Check systemd timers
systemctl list-timers --all

Note: crontab -l only shows root's crontab. Applications like Nextcloud add entries under their own service user — always check all users!

Site-Specific Configuration

Files like config_inc.php contain machine-specific database connection strings and passwords. Keep them in webstack but excluded from git, deployed by script:

/etc/webstack/site-config/
├── srv10/
│   ├── medw-config_inc.php
│   └── merg-config_inc.php
└── srv9/
    └── medw-config_inc.php

Deploy script:

#!/bin/bash
HOSTNAME=$(hostname -s)
for f in /etc/webstack/site-config/${HOSTNAME}/*-config_inc.php; do
    site=$(basename $f -config_inc.php)
    cp $f /srv/website/${site}/config/config_inc.php
done

Rsync Between Machines

# Full mirror srv10 → srv9 (delete enabled - true mirror)
rsync -avz --delete srv10:/srv/website/ /srv/website/

# Additive sync srv10 → desktop (no delete - preserves local extras)
rsync -avz srv10:/srv/website/ /srv/website/

# Exclude machine-specific files
rsync -avz --exclude='config_inc.php' srv10:/srv/website/ /srv/website/

Gotchas

 

• systemctl daemon-reload — always needed after adding/modifying service files
• git rm --cached — use to stop tracking files already committed before adding to .gitignore
• .gitignore only ignores untracked files — files already committed stay tracked regardless
• git commit --amend — fix the last commit message before pushing
• Directory permissions need 755 not 644 — files need 6, directories need 7
• Check ALL user crontabs — applications add cron entries under their own user, not root
• PrivateUsers=true in systemd — restricts filesystem access, check ReadWritePaths when moving data directories

 

Developed with Claude AI assistance - Anthropic - April 2026

# Overall health - the key one line answer
smartctl -H /dev/sdX

# Key values for status dashboard
smartctl -A /dev/sdX | grep -E 'Reallocated|Pending|Uncorrectable|Temperature|Power_On_Hours|Start_Stop'

# Current test progress
smartctl -l selftest /dev/sdX

# Full output for cribsheet reference
smartctl -a /dev/sdX

Developed with Claude AI assistance - Anthropic - April 2026