Duplicate page ... intermittant problem with being able to see original
Concept - Why Webstack
Traditional server configuration leaves files scattered across /etc, /usr/lib/systemd and application directories making it difficult to:
- Track what changed and why
- Mirror configuration between servers
- Recover from mistakes
- See the complete picture in one place
- Webstack consolidates all configuration into a single directory tree under git version control.
Structure - What Goes Where
/etc/webstack/
├── systemd/ ← service files (real files, linked FROM /etc/systemd/system/)
├── nginx/ ← entire nginx directory (real files, /etc/nginx linked here)
├── php8/ ← php config (real files, /etc/php8 linked here)
├── firebird/ ← firebird config (real files, linked FROM /opt/firebird5/)
├── radicale/ ← radicale config (real files, linked FROM /etc/radicale)
├── acme/ ← acme.sh domain configs and dhparams.pem
├── cron.daily/ ← cron scripts (real files, linked FROM /etc/cron.daily/)
├── scripts/ ← admin utilities
└── site-config/ ← machine specific site configs
├── srv10/
├── srv9/
└── desktop/
Key principle: Real files live in webstack, symlinks point back to where services expect them.
The Symlink Pattern
Direction matters — symlinks go FROM the service location TO webstack, not the other way:
# CORRECT - webstack owns the file
ln -s /etc/webstack/nginx /etc/nginx
ln -s /etc/webstack/systemd/php-fpm.service /etc/systemd/system/php-fpm.service
# WRONG - git would only track the symlink not the content
ln -s /etc/nginx /etc/webstack/nginx
Moving an existing directory:
mv /etc/nginx /etc/webstack/nginx
ln -s /etc/webstack/nginx /etc/nginx
nginx -t && systemctl restart nginx
Setting Up Git
zypper install git
# Configure identity - use server identity not personal details
git config --global user.name "srv10 admin"
git config --global user.email "root@srv10.rdm1.uk"
git config --global init.defaultBranch production
# Initialise
cd /etc/webstack
git init
.gitignore - What To Exclude
# SSL certificates and keys - never in git
nginx/ssl/**/*.pem
nginx/ssl/**/*.cer
nginx/ssl/**/*.cer.ecc
nginx/ssl/**/*.key
nginx/ssl/**/*.csr
# Radicale user passwords
radicale/users
# Site config contains database passwords
site-config/
Add placeholder files so directory structure is preserved:
find /etc/webstack/nginx/ssl -type d -exec touch {}/.gitkeep ;
And allow them despite the ignore rules:
!nginx/ssl/**/.gitkeep
Committing In Logical Sections
Commit each section separately for clean history:
git add systemd/
git commit -m "systemd service files - moved from /usr/lib to allow version control"
git add nginx/
git commit -m "nginx config - full config including ssl and vhosts"
git add php8/
git commit -m "php8 config - clean baseline, opcache isolation settings"
git add firebird/
git commit -m "firebird config - RuntimeDirectory fix for modern systemd"
git add radicale/
git commit -m "radicale CalDAV server - replacing nextcloud calendar/contacts"
The Bare Repo - Mirroring Between Machines
The bare repo acts as a hub — like a local github but private:
# On srv10 - create the hub
git init --bare /srv/git/webstack.git
# On srv10 working copy - point at hub
git remote add origin /srv/git/webstack.git
git push -u origin production
# On srv9 - clone from hub
git clone ssh://root@srv10/srv/git/webstack.git /etc/webstack
Daily Workflow
# Start of session - get latest from other machine
git pull
# Make changes to config files
# Test they work - nginx -t, systemctl restart etc
# End of session - commit and share
git add .
git commit -m "what changed and why"
git push
Renaming files (disabled configs):
# Use git mv to preserve history
git mv nginx/conf.d/site.conf nginx/conf.d/site.conf.off
git commit -m "site disabled - reason why"
Service File Management
Never edit files in /usr/lib/systemd/system/ — they get overwritten by package updates. Copy to webstack and symlink:
cp /usr/lib/systemd/system/php-fpm.service /etc/webstack/systemd/
ln -s /etc/webstack/systemd/php-fpm.service /etc/systemd/system/php-fpm.service
systemctl daemon-reload
Always run systemctl daemon-reload after adding or modifying service files!
PHP-FPM Isolation
Running multiple sites from one PHP-FPM pool risks APCu cache contamination between applications. Isolate heavyweight applications:
# Clone service for separate pool
cp /etc/webstack/systemd/php-fpm.service /etc/webstack/systemd/php-fpm-nextcloud.service
# Edit to point at separate pool config
systemctl daemon-reload
systemctl enable php-fpm-nextcloud
systemctl start php-fpm-nextcloud
Each pool has its own APCu memory space — applications cannot contaminate each other.
Scripts
/etc/webstack/scripts/restart-stack.sh — restart all services in correct order:
#!/bin/bash
echo "Restarting full web stack..."
systemctl restart firebird
systemctl restart php-fpm
systemctl restart php-fpm-nextcloud
systemctl restart nginx
echo "Done - checking status..."
systemctl is-active firebird php-fpm php-fpm-nextcloud nginx
/etc/webstack/scripts/check-crontabs.sh — audit all scheduled tasks:
#!/bin/bash
# Check all user crontabs
for user in $(cut -d: -f1 /etc/passwd); do
CRON=$(crontab -u $user -l 2>/dev/null)
if [ ! -z "$CRON" ]; then
echo "=== $user ==="
echo "$CRON"
fi
done
# Check systemd timers
systemctl list-timers --all
Note: crontab -l only shows root's crontab. Applications like Nextcloud add entries under their own service user — always check all users!
Site-Specific Configuration
Files like config_inc.php contain machine-specific database connection strings and passwords. Keep them in webstack but excluded from git, deployed by script:
/etc/webstack/site-config/
├── srv10/
│ ├── medw-config_inc.php
│ └── merg-config_inc.php
└── srv9/
└── medw-config_inc.php
Deploy script:
#!/bin/bash
HOSTNAME=$(hostname -s)
for f in /etc/webstack/site-config/${HOSTNAME}/*-config_inc.php; do
site=$(basename $f -config_inc.php)
cp $f /srv/website/${site}/config/config_inc.php
done
Rsync Between Machines
# Full mirror srv10 → srv9 (delete enabled - true mirror)
rsync -avz --delete srv10:/srv/website/ /srv/website/
# Additive sync srv10 → desktop (no delete - preserves local extras)
rsync -avz srv10:/srv/website/ /srv/website/
# Exclude machine-specific files
rsync -avz --exclude='config_inc.php' srv10:/srv/website/ /srv/website/
Gotchas
- systemctl daemon-reload — always needed after adding/modifying service files
- git rm --cached — use to stop tracking files already committed before adding to .gitignore
- .gitignore only ignores untracked files — files already committed stay tracked regardless
- git commit --amend — fix the last commit message before pushing
- Directory permissions need 755 not 644 — files need 6, directories need 7
- Check ALL user crontabs — applications add cron entries under their own user, not root
- PrivateUsers=true in systemd — restricts filesystem access, check ReadWritePaths when moving data directories
Developed with Claude AI assistance - Anthropic - April 2026
