History of Webstack - Server Configuration Management

Comparing versions
Version 1Current version

Concept - Why Webstack

Traditional server configuration leaves files scattered across /etc, /usr/lib/systemd and application directories making it difficult to:

  • Track what changed and why
  • Mirror configuration between servers
  • Recover from mistakes
  • See the complete picture in one place
  • Webstack consolidates all configuration into a single directory tree under git version control.

     

    Structure - What Goes Where

    /etc/webstack/
├── systemd/          ← service files (real files, linked FROM /etc/systemd/system/)
├── nginx/            ← entire nginx directory (real files, /etc/nginx linked here)
├── php8/             ← php config (real files, /etc/php8 linked here)
├── firebird/         ← firebird config (real files, linked FROM /opt/firebird5/)
├── radicale/         ← radicale config (real files, linked FROM /etc/radicale)
├── acme/             ← acme.sh domain configs and dhparams.pem
├── cron.daily/       ← cron scripts (real files, linked FROM /etc/cron.daily/)
├── scripts/          ← admin utilities
└── site-config/      ← machine specific site configs
    ├── srv10/
    ├── srv9/
    └── desktop/
    Key principle: Real files live in webstack, symlinks point back to where services expect them.

    The Symlink Pattern

    Direction matters — symlinks go FROM the service location TO webstack, not the other way: 
    # CORRECT - webstack owns the file
ln -s /etc/webstack/nginx /etc/nginx
ln -s /etc/webstack/systemd/php-fpm.service /etc/systemd/system/php-fpm.service

# WRONG - git would only track the symlink not the content
ln -s /etc/nginx /etc/webstack/nginx
    Moving an existing directory: 
    mv /etc/nginx /etc/webstack/nginx
ln -s /etc/webstack/nginx /etc/nginx
nginx -t && systemctl restart nginx

    Setting Up Git

    zypper install git

# Configure identity - use server identity not personal details
git config --global user.name "srv10 admin"
git config --global user.email "root@srv10.rdm1.uk"
git config --global init.defaultBranch production

# Initialise
cd /etc/webstack
git init

    .gitignore - What To Exclude

    # SSL certificates and keys - never in git
nginx/ssl/**/*.pem
nginx/ssl/**/*.cer
nginx/ssl/**/*.cer.ecc
nginx/ssl/**/*.key
nginx/ssl/**/*.csr
# Radicale user passwords
radicale/users
# Site config contains database passwords
site-config/
    Add placeholder files so directory structure is preserved: 
    find /etc/webstack/nginx/ssl -type d -exec touch {}/.gitkeep ;
    And allow them despite the ignore rules: 
    !nginx/ssl/**/.gitkeep

    Committing In Logical Sections

    Commit each section separately for clean history: 
    git add systemd/
git commit -m "systemd service files - moved from /usr/lib to allow version control"

git add nginx/
git commit -m "nginx config - full config including ssl and vhosts"

git add php8/
git commit -m "php8 config - clean baseline, opcache isolation settings"

git add firebird/
git commit -m "firebird config - RuntimeDirectory fix for modern systemd"

git add radicale/
git commit -m "radicale CalDAV server - replacing nextcloud calendar/contacts"

    The Bare Repo - Mirroring Between Machines

    The bare repo acts as a hub — like a local github but private: 
    # On srv10 - create the hub
git init --bare /srv/git/webstack.git

# On srv10 working copy - point at hub
git remote add origin /srv/git/webstack.git
git push -u origin production

# On srv9 - clone from hub
git clone ssh://root@srv10/srv/git/webstack.git /etc/webstack

    Daily Workflow

    # Start of session - get latest from other machine
git pull

# Make changes to config files
# Test they work - nginx -t, systemctl restart etc

# End of session - commit and share
git add .
git commit -m "what changed and why"
git push
    Renaming files (disabled configs): 
    # Use git mv to preserve history
git mv nginx/conf.d/site.conf nginx/conf.d/site.conf.off
git commit -m "site disabled - reason why"

    Service File Management

    Never edit files in /usr/lib/systemd/system/ — they get overwritten by package updates. Copy to webstack and symlink: 
    cp /usr/lib/systemd/system/php-fpm.service /etc/webstack/systemd/
ln -s /etc/webstack/systemd/php-fpm.service /etc/systemd/system/php-fpm.service
systemctl daemon-reload
    Always run systemctl daemon-reload after adding or modifying service files!

    PHP-FPM Isolation

    Running multiple sites from one PHP-FPM pool risks APCu cache contamination between applications. Isolate heavyweight applications: 
    # Clone service for separate pool
cp /etc/webstack/systemd/php-fpm.service /etc/webstack/systemd/php-fpm-nextcloud.service
# Edit to point at separate pool config
systemctl daemon-reload
systemctl enable php-fpm-nextcloud
systemctl start php-fpm-nextcloud
    Each pool has its own APCu memory space — applications cannot contaminate each other.

    Scripts

    /etc/webstack/scripts/restart-stack.sh — restart all services in correct order: 
    #!/bin/bash
echo "Restarting full web stack..."
systemctl restart firebird
systemctl restart php-fpm
systemctl restart php-fpm-nextcloud
systemctl restart nginx
echo "Done - checking status..."
systemctl is-active firebird php-fpm php-fpm-nextcloud nginx
    /etc/webstack/scripts/check-crontabs.sh — audit all scheduled tasks: 
    #!/bin/bash
# Check all user crontabs
for user in $(cut -d: -f1 /etc/passwd); do
    CRON=$(crontab -u $user -l 2>/dev/null)
    if [ ! -z "$CRON" ]; then
        echo "=== $user ==="
        echo "$CRON"
    fi
done
# Check systemd timers
systemctl list-timers --all
    Note: crontab -l only shows root's crontab. Applications like Nextcloud add entries under their own service user — always check all users!

    Site-Specific Configuration

    Files like config_inc.php contain machine-specific database connection strings and passwords. Keep them in webstack but excluded from git, deployed by script: 
    /etc/webstack/site-config/
├── srv10/
│   ├── medw-config_inc.php
│   └── merg-config_inc.php
└── srv9/
    └── medw-config_inc.php
    Deploy script: 
    #!/bin/bash
HOSTNAME=$(hostname -s)
for f in /etc/webstack/site-config/${HOSTNAME}/*-config_inc.php; do
    site=$(basename $f -config_inc.php)
    cp $f /srv/website/${site}/config/config_inc.php
done

    Rsync Between Machines

    # Full mirror srv10 → srv9 (delete enabled - true mirror)
rsync -avz --delete srv10:/srv/website/ /srv/website/

# Additive sync srv10 → desktop (no delete - preserves local extras)
rsync -avz srv10:/srv/website/ /srv/website/

# Exclude machine-specific files
rsync -avz --exclude='config_inc.php' srv10:/srv/website/ /srv/website/

    Gotchas

  • systemctl daemon-reload — always needed after adding/modifying service files
  • git rm --cached — use to stop tracking files already committed before adding to .gitignore
  • .gitignore only ignores untracked files — files already committed stay tracked regardless
  • git commit --amend — fix the last commit message before pushing
  • Directory permissions need 755 not 644 — files need 6, directories need 7
  • Check ALL user crontabs — applications add cron entries under their own user, not root
  • PrivateUsers=true in systemd — restricts filesystem access, check ReadWritePaths when moving data directories

 

 

Concept - Why Webstack

Traditional server configuration leaves files scattered across /etc, /usr/lib/systemd and application directories making it difficult to:

  • Track what changed and why
  • Mirror configuration between servers
  • Recover from mistakes
  • See the complete picture in one place
  • Webstack consolidates all configuration into a single directory tree under git version control.

Structure - What Goes Where

/etc/webstack/
├── systemd/          > service files (real files, linked FROM /etc/systemd/system/)
├── nginx/            > entire nginx directory (real files, /etc/nginx linked here)
├── php8/             > php config (real files, /etc/php8 linked here)
├── firebird/         > firebird config (real files, linked FROM /opt/firebird5/)
├── radicale/         > radicale config (real files, linked FROM /etc/radicale)
├── acme/             > acme.sh domain configs and dhparams.pem
├── cron.daily/       > cron scripts (real files, linked FROM /etc/cron.daily/)
├── scripts/          > admin utilities
└── site-config/      > machine specific site configs
    ├── srv10/
    ├── srv9/
    └── desktop/

Key principle: Real files live in webstack, symlinks point back to where services expect them.

The Symlink Pattern

Direction matters — symlinks go FROM the service location TO webstack, not the other way:

# CORRECT - webstack owns the file
ln -s /etc/webstack/nginx /etc/nginx
ln -s /etc/webstack/systemd/php-fpm.service /etc/systemd/system/php-fpm.service

# WRONG - git would only track the symlink not the content
ln -s /etc/nginx /etc/webstack/nginx

Moving an existing directory:

mv /etc/nginx /etc/webstack/nginx
ln -s /etc/webstack/nginx /etc/nginx
nginx -t && systemctl restart nginx

Setting Up Git

zypper install git

# Configure identity - use server identity not personal details
git config --global user.name "srv10 admin"
git config --global user.email "root@srv10.rdm1.uk"
git config --global init.defaultBranch production

# Initialise
cd /etc/webstack
git init

.gitignore - What To Exclude

# SSL certificates and keys - never in git
nginx/ssl/**/*.pem
nginx/ssl/**/*.cer
nginx/ssl/**/*.cer.ecc
nginx/ssl/**/*.key
nginx/ssl/**/*.csr
# Radicale user passwords
radicale/users
# Site config contains database passwords
site-config/

Add placeholder files so directory structure is preserved:

find /etc/webstack/nginx/ssl -type d -exec touch {}/.gitkeep ;

And allow them despite the ignore rules:

!nginx/ssl/**/.gitkeep

Committing In Logical Sections

Commit each section separately for clean history:

git add systemd/
git commit -m "systemd service files - moved from /usr/lib to allow version control"

git add nginx/
git commit -m "nginx config - full config including ssl and vhosts"

git add php8/
git commit -m "php8 config - clean baseline, opcache isolation settings"

git add firebird/
git commit -m "firebird config - RuntimeDirectory fix for modern systemd"

git add radicale/
git commit -m "radicale CalDAV server - replacing nextcloud calendar/contacts"

The Bare Repo - Mirroring Between Machines

The bare repo acts as a hub — like a local github but private:

# On srv10 - create the hub
git init --bare /srv/git/webstack.git

# On srv10 working copy - point at hub
git remote add origin /srv/git/webstack.git
git push -u origin production

# On srv9 - clone from hub
git clone ssh://root@srv10/srv/git/webstack.git /etc/webstack

Daily Workflow

# Start of session - get latest from other machine
git pull

# Make changes to config files
# Test they work - nginx -t, systemctl restart etc

# End of session - commit and share
git add .
git commit -m "what changed and why"
git push

Renaming files (disabled configs):

# Use git mv to preserve history
git mv nginx/conf.d/site.conf nginx/conf.d/site.conf.off
git commit -m "site disabled - reason why"

Service File Management

Never edit files in /usr/lib/systemd/system/ — they get overwritten by package updates. Copy to webstack and symlink:

cp /usr/lib/systemd/system/php-fpm.service /etc/webstack/systemd/
ln -s /etc/webstack/systemd/php-fpm.service /etc/systemd/system/php-fpm.service
systemctl daemon-reload

Always run systemctl daemon-reload after adding or modifying service files!

PHP-FPM Isolation

Running multiple sites from one PHP-FPM pool risks APCu cache contamination between applications. Isolate heavyweight applications:

# Clone service for separate pool
cp /etc/webstack/systemd/php-fpm.service /etc/webstack/systemd/php-fpm-nextcloud.service
# Edit to point at separate pool config
systemctl daemon-reload
systemctl enable php-fpm-nextcloud
systemctl start php-fpm-nextcloud

Each pool has its own APCu memory space — applications cannot contaminate each other.

Scripts

/etc/webstack/scripts/restart-stack.sh — restart all services in correct order:

#!/bin/bash
echo "Restarting full web stack..."
systemctl restart firebird
systemctl restart php-fpm
systemctl restart php-fpm-nextcloud
systemctl restart nginx
echo "Done - checking status..."
systemctl is-active firebird php-fpm php-fpm-nextcloud nginx

/etc/webstack/scripts/check-crontabs.sh — audit all scheduled tasks:

#!/bin/bash
# Check all user crontabs
for user in $(cut -d: -f1 /etc/passwd); do
    CRON=$(crontab -u $user -l 2>/dev/null)
    if [ ! -z "$CRON" ]; then
        echo "=== $user ==="
        echo "$CRON"
    fi
done
# Check systemd timers
systemctl list-timers --all

Note: crontab -l only shows root's crontab. Applications like Nextcloud add entries under their own service user — always check all users!

Site-Specific Configuration

Files like config_inc.php contain machine-specific database connection strings and passwords. Keep them in webstack but excluded from git, deployed by script:

/etc/webstack/site-config/
├── srv10/
│   ├── medw-config_inc.php
│   └── merg-config_inc.php
└── srv9/
    └── medw-config_inc.php

Deploy script:

#!/bin/bash
HOSTNAME=$(hostname -s)
for f in /etc/webstack/site-config/${HOSTNAME}/*-config_inc.php; do
    site=$(basename $f -config_inc.php)
    cp $f /srv/website/${site}/config/config_inc.php
done

Rsync Between Machines

# Full mirror srv10 > srv9 (delete enabled - true mirror)
rsync -avz --delete srv10:/srv/website/ /srv/website/

# Additive sync srv10 > desktop (no delete - preserves local extras)
rsync -avz srv10:/srv/website/ /srv/website/

# Exclude machine-specific files
rsync -avz --exclude="config_inc.php" srv10:/srv/website/ /srv/website/

Gotchas

 

  • systemctl daemon-reload — always needed after adding/modifying service files
  • git rm --cached — use to stop tracking files already committed before adding to .gitignore
  • .gitignore only ignores untracked files — files already committed stay tracked regardless
  • git commit --amend — fix the last commit message before pushing
  • Directory permissions need 755 not 644 — files need 6, directories need 7
  • Check ALL user crontabs — applications add cron entries under their own user, not root
  • PrivateUsers=true in systemd — restricts filesystem access, check ReadWritePaths when moving data directories

Developed with Claude AI assistance - Anthropic - April 2026

Page History
Date/CommentUserIPVersion
01 May 2026 (09:06 UTC)
—		Lester Caine192.168.1.2544
Current • Source
Lester Caine192.168.1.2543
View • Compare • Difference • Source
Lester Caine192.168.1.2542
View • Compare • Difference • Source
Lester Caine192.168.1.2541
View • Compare • Difference • Source