Structure - What Goes Where
|
|
|
|
| /etc/webstack/
|
| ├── systemd/ ← service files (real files, linked FROM /etc/systemd/system/)
|
| ├── nginx/ ← entire nginx directory (real files, /etc/nginx linked here)
|
| ├── php8/ ← php config (real files, /etc/php8 linked here)
|
| ├── firebird/ ← firebird config (real files, linked FROM /opt/firebird5/)
|
| ├── radicale/ ← radicale config (real files, linked FROM /etc/radicale)
|
| ├── acme/ ← acme.sh domain configs and dhparams.pem
|
| ├── cron.daily/ ← cron scripts (real files, linked FROM /etc/cron.daily/)
|
| ├── scripts/ ← admin utilities
|
| └── site-config/ ← machine specific site configs
|
| ├── srv10/
|
| ├── srv9/
|
| └── desktop/
|
|
|
|
|
Key principle: Real files live in webstack, symlinks point back to where services expect them.
|
|
|
The Symlink Pattern
|
|
|
Direction matters — symlinks go FROM the service location TO webstack, not the other way:
|
|
|
|
| # CORRECT - webstack owns the file
|
| ln -s /etc/webstack/nginx /etc/nginx
|
| ln -s /etc/webstack/systemd/php-fpm.service /etc/systemd/system/php-fpm.service
|
|
|
| # WRONG - git would only track the symlink not the content
|
| ln -s /etc/nginx /etc/webstack/nginx
|
|
|
|
|
Moving an existing directory:
|
|
|
|
| mv /etc/nginx /etc/webstack/nginx
|
| ln -s /etc/webstack/nginx /etc/nginx
|
| nginx -t && systemctl restart nginx
|
|
|
|
|
Setting Up Git
|
|
|
|
| zypper install git
|
|
|
| # Configure identity - use server identity not personal details
|
| git config --global user.name "srv10 admin"
|
| git config --global user.email "root@srv10.rdm1.uk"
|
| git config --global init.defaultBranch production
|
|
|
| # Initialise
|
| cd /etc/webstack
|
| git init
|
|
|
|
|
.gitignore - What To Exclude
|
|
|
|
| # SSL certificates and keys - never in git
|
| nginx/ssl/**/*.pem
|
| nginx/ssl/**/*.cer
|
| nginx/ssl/**/*.cer.ecc
|
| nginx/ssl/**/*.key
|
| nginx/ssl/**/*.csr
|
| # Radicale user passwords
|
| radicale/users
|
| # Site config contains database passwords
|
| site-config/
|
|
|
|
|
Add placeholder files so directory structure is preserved:
|
|
|
|
| find /etc/webstack/nginx/ssl -type d -exec touch {}/.gitkeep ;
|
|
|
And allow them despite the ignore rules:
|
|
|
|
| !nginx/ssl/**/.gitkeep
|
|
|
Committing In Logical Sections
|
|
|
Commit each section separately for clean history:
|
|
|
|
| git add systemd/
|
| git commit -m "systemd service files - moved from /usr/lib to allow version control"
|
|
|
| git add nginx/
|
| git commit -m "nginx config - full config including ssl and vhosts"
|
|
|
| git add php8/
|
| git commit -m "php8 config - clean baseline, opcache isolation settings"
|
|
|
| git add firebird/
|
| git commit -m "firebird config - RuntimeDirectory fix for modern systemd"
|
|
|
| git add radicale/
|
| git commit -m "radicale CalDAV server - replacing nextcloud calendar/contacts"
|
|
|
The Bare Repo - Mirroring Between Machines
|
|
|
The bare repo acts as a hub — like a local github but private:
|
|
|
|
| # On srv10 - create the hub
|
| git init --bare /srv/git/webstack.git
|
|
|
| # On srv10 working copy - point at hub
|
| git remote add origin /srv/git/webstack.git
|
| git push -u origin production
|
|
|
| # On srv9 - clone from hub
|
| git clone ssh://root@srv10/srv/git/webstack.git /etc/webstack
|
|
|
Daily Workflow
|
|
|
|
| # Start of session - get latest from other machine
|
| git pull
|
|
|
| # Make changes to config files
|
| # Test they work - nginx -t, systemctl restart etc
|
|
|
| # End of session - commit and share
|
| git add .
|
| git commit -m "what changed and why"
|
| git push
|
|
|
Renaming files (disabled configs):
|
|
|
|
| # Use git mv to preserve history
|
| git mv nginx/conf.d/site.conf nginx/conf.d/site.conf.off
|
| git commit -m "site disabled - reason why"
|
|
|
Service File Management
|
|
|
Never edit files in /usr/lib/systemd/system/ — they get overwritten by package updates. Copy to webstack and symlink:
|
|
|
|
| cp /usr/lib/systemd/system/php-fpm.service /etc/webstack/systemd/
|
| ln -s /etc/webstack/systemd/php-fpm.service /etc/systemd/system/php-fpm.service
|
| systemctl daemon-reload
|
|
|
Always run systemctl daemon-reload after adding or modifying service files!
|
|
|
PHP-FPM Isolation
|
|
|
Running multiple sites from one PHP-FPM pool risks APCu cache contamination between applications. Isolate heavyweight applications:
|
|
|
|
| # Clone service for separate pool
|
| cp /etc/webstack/systemd/php-fpm.service /etc/webstack/systemd/php-fpm-nextcloud.service
|
| # Edit to point at separate pool config
|
| systemctl daemon-reload
|
| systemctl enable php-fpm-nextcloud
|
| systemctl start php-fpm-nextcloud
|
|
|
Each pool has its own APCu memory space — applications cannot contaminate each other.
|
|
|
Scripts
|
|
|
/etc/webstack/scripts/restart-stack.sh — restart all services in correct order:
|
|
|
|
| #!/bin/bash
|
| echo "Restarting full web stack..."
|
| systemctl restart firebird
|
| systemctl restart php-fpm
|
| systemctl restart php-fpm-nextcloud
|
| systemctl restart nginx
|
| echo "Done - checking status..."
|
| systemctl is-active firebird php-fpm php-fpm-nextcloud nginx
|
|
|
|
|
/etc/webstack/scripts/check-crontabs.sh — audit all scheduled tasks:
|
|
|
|
| #!/bin/bash
|
| # Check all user crontabs
|
| for user in $(cut -d: -f1 /etc/passwd); do
|
| CRON=$(crontab -u $user -l 2>/dev/null)
|
| if [ ! -z "$CRON" ]; then
|
| echo "=== $user ==="
|
| echo "$CRON"
|
| fi
|
| done
|
| # Check systemd timers
|
| systemctl list-timers --all
|
|
|
Note: crontab -l only shows root's crontab. Applications like Nextcloud add entries under their own service user — always check all users!
|
|
|
Site-Specific Configuration
|
|
|
Files like config_inc.php contain machine-specific database connection strings and passwords. Keep them in webstack but excluded from git, deployed by script:
|
|
|
|
| /etc/webstack/site-config/
|
| ├── srv10/
|
| │ ├── medw-config_inc.php
|
| │ └── merg-config_inc.php
|
| └── srv9/
|
| └── medw-config_inc.php
|
|
|
Deploy script:
|
|
|
|
| #!/bin/bash
|
| HOSTNAME=$(hostname -s)
|
| for f in /etc/webstack/site-config/${HOSTNAME}/*-config_inc.php; do
|
| site=$(basename $f -config_inc.php)
|
| cp $f /srv/website/${site}/config/config_inc.php
|
| done
|
|
|
Rsync Between Machines
|
|
|
|
| # Full mirror srv10 → srv9 (delete enabled - true mirror)
|
| rsync -avz --delete srv10:/srv/website/ /srv/website/
|
|
|
| # Additive sync srv10 → desktop (no delete - preserves local extras)
|
| rsync -avz srv10:/srv/website/ /srv/website/
|
|
|
| # Exclude machine-specific files
|
| rsync -avz --exclude='config_inc.php' srv10:/srv/website/ /srv/website/
|
|
|
Gotchas
|
|
|
|
|
|
|
| systemctl daemon-reload — always needed after adding/modifying service files
|
| git rm --cached — use to stop tracking files already committed before adding to .gitignore
|
| .gitignore only ignores untracked files — files already committed stay tracked regardless
|
| git commit --amend — fix the last commit message before pushing
|
| Directory permissions need 755 not 644 — files need 6, directories need 7
|
| Check ALL user crontabs — applications add cron entries under their own user, not root
|
| PrivateUsers=true in systemd — restricts filesystem access, check ReadWritePaths when moving data directories
|
|
|
|
|
|
# Full output for cribsheet reference
|
| smartctl -a /dev/sdX |
