History of Webstack - Disk Configuration Management
<h3>Concept - Why Webstack</h3>
<p>Traditional server configuration leaves files scattered across /etc, /usr/lib/systemd and application directories making it difficult to:</p>
<ul>
<li>Track what changed and why</li>
<li>Mirror configuration between servers</li>
<li>Recover from mistakes</li>
<li>See the complete picture in one place</li>
<li>Webstack consolidates all configuration into a single directory tree under git version control.</li>
</ul>
<h3>Structure - What Goes Where</h3>
<pre>
/etc/webstack/
├── systemd/ &amp;amp;larr; service files (real files, linked FROM /etc/systemd/system/)
├── nginx/ &amp;amp;larr; entire nginx directory (real files, /etc/nginx linked here)
├── php8/ &amp;amp;larr; php config (real files, /etc/php8 linked here)
├── firebird/ &amp;amp;larr; firebird config (real files, linked FROM /opt/firebird5/)
├── radicale/ &amp;amp;larr; radicale config (real files, linked FROM /etc/radicale)
├── acme/ &amp;amp;larr; acme.sh domain configs and dhparams.pem
├── cron.daily/ &amp;amp;larr; cron scripts (real files, linked FROM /etc/cron.daily/)
├── scripts/ &amp;amp;larr; admin utilities
└── site-config/ &amp;amp;larr; machine specific site configs
├── srv10/
├── srv9/
└── desktop/
</pre>
<p><strong>Key principle: Real files live in webstack, symlinks point back to where services expect them.</strong></p>
<h3>The Symlink Pattern</h3>
<p>Direction matters — symlinks go FROM the service location TO webstack, not the other way:</p>
<pre>
# CORRECT - webstack owns the file
ln -s /etc/webstack/nginx /etc/nginx
ln -s /etc/webstack/systemd/php-fpm.service /etc/systemd/system/php-fpm.service
# WRONG - git would only track the symlink not the content
ln -s /etc/nginx /etc/webstack/nginx
</pre>
<p>Moving an existing directory:</p>
<pre>
mv /etc/nginx /etc/webstack/nginx
ln -s /etc/webstack/nginx /etc/nginx
nginx -t &amp;amp;amp;&amp;amp;amp; systemctl restart nginx
</pre>
<h3>Setting Up Git</h3>
<pre>
zypper install git
# Configure identity - use server identity not personal details
git config --global user.name &amp;amp;quot;srv10 admin&amp;amp;quot;
git config --global user.email &amp;amp;quot;root@srv10.rdm1.uk&amp;amp;quot;
git config --global init.defaultBranch production
# Initialise
cd /etc/webstack
git init
</pre>
<h3>.gitignore - What To Exclude</h3>
<pre>
# SSL certificates and keys - never in git
nginx/ssl/**/*.pem
nginx/ssl/**/*.cer
nginx/ssl/**/*.cer.ecc
nginx/ssl/**/*.key
nginx/ssl/**/*.csr
# Radicale user passwords
radicale/users
# Site config contains database passwords
site-config/
</pre>
<p>Add placeholder files so directory structure is preserved:</p>
<pre>
find /etc/webstack/nginx/ssl -type d -exec touch {}/.gitkeep ;</pre>
<p>And allow them despite the ignore rules:</p>
<pre>
!nginx/ssl/**/.gitkeep</pre>
<h3>Committing In Logical Sections</h3>
<p>Commit each section separately for clean history:</p>
<pre>
git add systemd/
git commit -m &amp;amp;quot;systemd service files - moved from /usr/lib to allow version control&amp;amp;quot;
git add nginx/
git commit -m &amp;amp;quot;nginx config - full config including ssl and vhosts&amp;amp;quot;
git add php8/
git commit -m &amp;amp;quot;php8 config - clean baseline, opcache isolation settings&amp;amp;quot;
git add firebird/
git commit -m &amp;amp;quot;firebird config - RuntimeDirectory fix for modern systemd&amp;amp;quot;
git add radicale/
git commit -m &amp;amp;quot;radicale CalDAV server - replacing nextcloud calendar/contacts&amp;amp;quot;</pre>
<h3>The Bare Repo - Mirroring Between Machines</h3>
<p>The bare repo acts as a hub — like a local github but private:</p>
<pre>
# On srv10 - create the hub
git init --bare /srv/git/webstack.git
# On srv10 working copy - point at hub
git remote add origin /srv/git/webstack.git
git push -u origin production
# On srv9 - clone from hub
git clone ssh://root@srv10/srv/git/webstack.git /etc/webstack</pre>
<h3>Daily Workflow</h3>
<pre>
# Start of session - get latest from other machine
git pull
# Make changes to config files
# Test they work - nginx -t, systemctl restart etc
# End of session - commit and share
git add .
git commit -m &amp;amp;quot;what changed and why&amp;amp;quot;
git push</pre>
<p>Renaming files (disabled configs):</p>
<pre>
# Use git mv to preserve history
git mv nginx/conf.d/site.conf nginx/conf.d/site.conf.off
git commit -m &amp;amp;quot;site disabled - reason why&amp;amp;quot;</pre>
<h3>Service File Management</h3>
<p>Never edit files in /usr/lib/systemd/system/ — they get overwritten by package updates. Copy to webstack and symlink:</p>
<pre>
cp /usr/lib/systemd/system/php-fpm.service /etc/webstack/systemd/
ln -s /etc/webstack/systemd/php-fpm.service /etc/systemd/system/php-fpm.service
systemctl daemon-reload</pre>
<p><strong>Always run systemctl daemon-reload after adding or modifying service files!</strong></p>
<h3>PHP-FPM Isolation</h3>
<p>Running multiple sites from one PHP-FPM pool risks APCu cache contamination between applications. Isolate heavyweight applications:</p>
<pre>
# Clone service for separate pool
cp /etc/webstack/systemd/php-fpm.service /etc/webstack/systemd/php-fpm-nextcloud.service
# Edit to point at separate pool config
systemctl daemon-reload
systemctl enable php-fpm-nextcloud
systemctl start php-fpm-nextcloud</pre>
<p>Each pool has its own APCu memory space — applications cannot contaminate each other.</p>
<h3>Scripts</h3>
<p><strong>/etc/webstack/scripts/restart-stack.sh</strong> — restart all services in correct order:</p>
<pre>
#!/bin/bash
echo &amp;amp;quot;Restarting full web stack...&amp;amp;quot;
systemctl restart firebird
systemctl restart php-fpm
systemctl restart php-fpm-nextcloud
systemctl restart nginx
echo &amp;amp;quot;Done - checking status...&amp;amp;quot;
systemctl is-active firebird php-fpm php-fpm-nextcloud nginx
</pre>
<p><strong>/etc/webstack/scripts/check-crontabs.sh</strong> — audit all scheduled tasks:</p>
<pre>
#!/bin/bash
# Check all user crontabs
for user in $(cut -d: -f1 /etc/passwd); do
CRON=$(crontab -u $user -l 2&amp;amp;gt;/dev/null)
if [ ! -z &amp;amp;quot;$CRON&amp;amp;quot; ]; then
echo &amp;amp;quot;=== $user ===&amp;amp;quot;
echo &amp;amp;quot;$CRON&amp;amp;quot;
fi
done
# Check systemd timers
systemctl list-timers --all</pre>
<p><strong>Note:</strong> crontab -l only shows root's crontab. Applications like Nextcloud add entries under their own service user — always check all users!</p>
<h3>Site-Specific Configuration</h3>
<p>Files like config_inc.php contain machine-specific database connection strings and passwords. Keep them in webstack but excluded from git, deployed by script:</p>
<pre>
/etc/webstack/site-config/
├── srv10/
│ ├── medw-config_inc.php
│ └── merg-config_inc.php
└── srv9/
└── medw-config_inc.php</pre>
<p>Deploy script:</p>
<pre>
#!/bin/bash
HOSTNAME=$(hostname -s)
for f in /etc/webstack/site-config/${HOSTNAME}/*-config_inc.php; do
site=$(basename $f -config_inc.php)
cp $f /srv/website/${site}/config/config_inc.php
done</pre>
<h3>Rsync Between Machines</h3>
<pre>
# Full mirror srv10 &amp;amp;rarr; srv9 (delete enabled - true mirror)
rsync -avz --delete srv10:/srv/website/ /srv/website/
# Additive sync srv10 &amp;amp;rarr; desktop (no delete - preserves local extras)
rsync -avz srv10:/srv/website/ /srv/website/
# Exclude machine-specific files
rsync -avz --exclude=&amp;amp;#39;config_inc.php&amp;amp;#39; srv10:/srv/website/ /srv/website/</pre>
<h3>Gotchas</h3>
<p> </p>
<ul>
<li><strong>systemctl daemon-reload</strong> — always needed after adding/modifying service files</li>
<li><strong>git rm --cached</strong> — use to stop tracking files already committed before adding to .gitignore</li>
<li><strong>.gitignore only ignores untracked files</strong> — files already committed stay tracked regardless</li>
<li><strong>git commit --amend</strong> — fix the last commit message before pushing</li>
<li><strong>Directory permissions need 755 not 644</strong> — files need 6, directories need 7</li>
<li><strong>Check ALL user crontabs</strong> — applications add cron entries under their own user, not root</li>
<li><strong>PrivateUsers=true in systemd</strong> — restricts filesystem access, check ReadWritePaths when moving data directories</li>
</ul>
<p> </p>
<p><em>Developed with <a href="https://claude.ai">Claude AI</a> assistance - Anthropic - April 2026</em></p>
<p>Traditional server configuration leaves files scattered across /etc, /usr/lib/systemd and application directories making it difficult to:</p>
<ul>
<li>Track what changed and why</li>
<li>Mirror configuration between servers</li>
<li>Recover from mistakes</li>
<li>See the complete picture in one place</li>
<li>Webstack consolidates all configuration into a single directory tree under git version control.</li>
</ul>
<h3>Structure - What Goes Where</h3>
<pre>
/etc/webstack/
├── systemd/ &amp;amp;larr; service files (real files, linked FROM /etc/systemd/system/)
├── nginx/ &amp;amp;larr; entire nginx directory (real files, /etc/nginx linked here)
├── php8/ &amp;amp;larr; php config (real files, /etc/php8 linked here)
├── firebird/ &amp;amp;larr; firebird config (real files, linked FROM /opt/firebird5/)
├── radicale/ &amp;amp;larr; radicale config (real files, linked FROM /etc/radicale)
├── acme/ &amp;amp;larr; acme.sh domain configs and dhparams.pem
├── cron.daily/ &amp;amp;larr; cron scripts (real files, linked FROM /etc/cron.daily/)
├── scripts/ &amp;amp;larr; admin utilities
└── site-config/ &amp;amp;larr; machine specific site configs
├── srv10/
├── srv9/
└── desktop/
</pre>
<p><strong>Key principle: Real files live in webstack, symlinks point back to where services expect them.</strong></p>
<h3>The Symlink Pattern</h3>
<p>Direction matters — symlinks go FROM the service location TO webstack, not the other way:</p>
<pre>
# CORRECT - webstack owns the file
ln -s /etc/webstack/nginx /etc/nginx
ln -s /etc/webstack/systemd/php-fpm.service /etc/systemd/system/php-fpm.service
# WRONG - git would only track the symlink not the content
ln -s /etc/nginx /etc/webstack/nginx
</pre>
<p>Moving an existing directory:</p>
<pre>
mv /etc/nginx /etc/webstack/nginx
ln -s /etc/webstack/nginx /etc/nginx
nginx -t &amp;amp;amp;&amp;amp;amp; systemctl restart nginx
</pre>
<h3>Setting Up Git</h3>
<pre>
zypper install git
# Configure identity - use server identity not personal details
git config --global user.name &amp;amp;quot;srv10 admin&amp;amp;quot;
git config --global user.email &amp;amp;quot;root@srv10.rdm1.uk&amp;amp;quot;
git config --global init.defaultBranch production
# Initialise
cd /etc/webstack
git init
</pre>
<h3>.gitignore - What To Exclude</h3>
<pre>
# SSL certificates and keys - never in git
nginx/ssl/**/*.pem
nginx/ssl/**/*.cer
nginx/ssl/**/*.cer.ecc
nginx/ssl/**/*.key
nginx/ssl/**/*.csr
# Radicale user passwords
radicale/users
# Site config contains database passwords
site-config/
</pre>
<p>Add placeholder files so directory structure is preserved:</p>
<pre>
find /etc/webstack/nginx/ssl -type d -exec touch {}/.gitkeep ;</pre>
<p>And allow them despite the ignore rules:</p>
<pre>
!nginx/ssl/**/.gitkeep</pre>
<h3>Committing In Logical Sections</h3>
<p>Commit each section separately for clean history:</p>
<pre>
git add systemd/
git commit -m &amp;amp;quot;systemd service files - moved from /usr/lib to allow version control&amp;amp;quot;
git add nginx/
git commit -m &amp;amp;quot;nginx config - full config including ssl and vhosts&amp;amp;quot;
git add php8/
git commit -m &amp;amp;quot;php8 config - clean baseline, opcache isolation settings&amp;amp;quot;
git add firebird/
git commit -m &amp;amp;quot;firebird config - RuntimeDirectory fix for modern systemd&amp;amp;quot;
git add radicale/
git commit -m &amp;amp;quot;radicale CalDAV server - replacing nextcloud calendar/contacts&amp;amp;quot;</pre>
<h3>The Bare Repo - Mirroring Between Machines</h3>
<p>The bare repo acts as a hub — like a local github but private:</p>
<pre>
# On srv10 - create the hub
git init --bare /srv/git/webstack.git
# On srv10 working copy - point at hub
git remote add origin /srv/git/webstack.git
git push -u origin production
# On srv9 - clone from hub
git clone ssh://root@srv10/srv/git/webstack.git /etc/webstack</pre>
<h3>Daily Workflow</h3>
<pre>
# Start of session - get latest from other machine
git pull
# Make changes to config files
# Test they work - nginx -t, systemctl restart etc
# End of session - commit and share
git add .
git commit -m &amp;amp;quot;what changed and why&amp;amp;quot;
git push</pre>
<p>Renaming files (disabled configs):</p>
<pre>
# Use git mv to preserve history
git mv nginx/conf.d/site.conf nginx/conf.d/site.conf.off
git commit -m &amp;amp;quot;site disabled - reason why&amp;amp;quot;</pre>
<h3>Service File Management</h3>
<p>Never edit files in /usr/lib/systemd/system/ — they get overwritten by package updates. Copy to webstack and symlink:</p>
<pre>
cp /usr/lib/systemd/system/php-fpm.service /etc/webstack/systemd/
ln -s /etc/webstack/systemd/php-fpm.service /etc/systemd/system/php-fpm.service
systemctl daemon-reload</pre>
<p><strong>Always run systemctl daemon-reload after adding or modifying service files!</strong></p>
<h3>PHP-FPM Isolation</h3>
<p>Running multiple sites from one PHP-FPM pool risks APCu cache contamination between applications. Isolate heavyweight applications:</p>
<pre>
# Clone service for separate pool
cp /etc/webstack/systemd/php-fpm.service /etc/webstack/systemd/php-fpm-nextcloud.service
# Edit to point at separate pool config
systemctl daemon-reload
systemctl enable php-fpm-nextcloud
systemctl start php-fpm-nextcloud</pre>
<p>Each pool has its own APCu memory space — applications cannot contaminate each other.</p>
<h3>Scripts</h3>
<p><strong>/etc/webstack/scripts/restart-stack.sh</strong> — restart all services in correct order:</p>
<pre>
#!/bin/bash
echo &amp;amp;quot;Restarting full web stack...&amp;amp;quot;
systemctl restart firebird
systemctl restart php-fpm
systemctl restart php-fpm-nextcloud
systemctl restart nginx
echo &amp;amp;quot;Done - checking status...&amp;amp;quot;
systemctl is-active firebird php-fpm php-fpm-nextcloud nginx
</pre>
<p><strong>/etc/webstack/scripts/check-crontabs.sh</strong> — audit all scheduled tasks:</p>
<pre>
#!/bin/bash
# Check all user crontabs
for user in $(cut -d: -f1 /etc/passwd); do
CRON=$(crontab -u $user -l 2&amp;amp;gt;/dev/null)
if [ ! -z &amp;amp;quot;$CRON&amp;amp;quot; ]; then
echo &amp;amp;quot;=== $user ===&amp;amp;quot;
echo &amp;amp;quot;$CRON&amp;amp;quot;
fi
done
# Check systemd timers
systemctl list-timers --all</pre>
<p><strong>Note:</strong> crontab -l only shows root's crontab. Applications like Nextcloud add entries under their own service user — always check all users!</p>
<h3>Site-Specific Configuration</h3>
<p>Files like config_inc.php contain machine-specific database connection strings and passwords. Keep them in webstack but excluded from git, deployed by script:</p>
<pre>
/etc/webstack/site-config/
├── srv10/
│ ├── medw-config_inc.php
│ └── merg-config_inc.php
└── srv9/
└── medw-config_inc.php</pre>
<p>Deploy script:</p>
<pre>
#!/bin/bash
HOSTNAME=$(hostname -s)
for f in /etc/webstack/site-config/${HOSTNAME}/*-config_inc.php; do
site=$(basename $f -config_inc.php)
cp $f /srv/website/${site}/config/config_inc.php
done</pre>
<h3>Rsync Between Machines</h3>
<pre>
# Full mirror srv10 &amp;amp;rarr; srv9 (delete enabled - true mirror)
rsync -avz --delete srv10:/srv/website/ /srv/website/
# Additive sync srv10 &amp;amp;rarr; desktop (no delete - preserves local extras)
rsync -avz srv10:/srv/website/ /srv/website/
# Exclude machine-specific files
rsync -avz --exclude=&amp;amp;#39;config_inc.php&amp;amp;#39; srv10:/srv/website/ /srv/website/</pre>
<h3>Gotchas</h3>
<p> </p>
<ul>
<li><strong>systemctl daemon-reload</strong> — always needed after adding/modifying service files</li>
<li><strong>git rm --cached</strong> — use to stop tracking files already committed before adding to .gitignore</li>
<li><strong>.gitignore only ignores untracked files</strong> — files already committed stay tracked regardless</li>
<li><strong>git commit --amend</strong> — fix the last commit message before pushing</li>
<li><strong>Directory permissions need 755 not 644</strong> — files need 6, directories need 7</li>
<li><strong>Check ALL user crontabs</strong> — applications add cron entries under their own user, not root</li>
<li><strong>PrivateUsers=true in systemd</strong> — restricts filesystem access, check ReadWritePaths when moving data directories</li>
</ul>
<p> </p>
<p><em>Developed with <a href="https://claude.ai">Claude AI</a> assistance - Anthropic - April 2026</em></p>