customer-presentation/OPERATIONS.md

# Operations

Day-to-day commands for running project-bifrost.

## Managing invites

All commands run on the VPS as the `fenja` user.

```bash
# Add someone
sudo -u fenja node /opt/fenja/bin/invite.js add someone@example.com

# Remove someone (doesn't kill their active session — see below)
sudo -u fenja node /opt/fenja/bin/invite.js remove someone@example.com

# List everyone
sudo -u fenja node /opt/fenja/bin/invite.js list
```

Removing an invite stops a user from requesting *new* codes, but doesn't invalidate their existing session cookie (valid 30 days). To kick them out immediately, also run:

```bash
sudo sqlite3 /opt/fenja/data/fenja.sqlite \
  "DELETE FROM sessions WHERE email = 'someone@example.com';"
```

## Admin web UI (hidden)

There's a small admin page at **`/fenjaops`** — intentionally unlinked from anywhere in the site, and the path is obscure-by-choice so scripted probes of common admin paths (`/admin`, `/wp-admin`, etc.) miss. Access requires:

1. A valid session cookie (standard login), **and**
2. The user's invite row has `is_admin = 1`.

Anything else — logged-out, logged-in-but-not-admin, scripted probe — gets a plain **404**, same as a missing URL. The existence of `/fenjaops` is not leaked. Internal code uses the word "admin" everywhere (files, middleware, CLI) — only the public URL path is obscured.

What the page can and cannot do:

| Action                          | Where                         |
|---------------------------------|-------------------------------|
| View stats / joins / invite list| `/fenjaops` (read)            |
| **Invite a new non-admin user** | `/fenjaops` form (`POST /api/fenjaops/invites`) |
| Grant / revoke admin            | CLI only (`bin/invite.js admin …`) |
| Remove an invite                | CLI only (`bin/invite.js remove …`) |
| Kill a session                  | SQL (`DELETE FROM sessions WHERE email = …`) |

The split is deliberate: a web session compromise can grow the invite list with regular users but cannot escalate anyone (including the attacker) to admin or evict existing users. The POST handler on the server ignores any `is_admin` field in the body and always stores `0` — the only path to `is_admin=1` is through the CLI on the VPS.

Invites created from the form record the acting admin's email in `invited_by` (CLI adds still record `"cli"`), so the log in the invite list shows who added whom.

Grant / revoke admin stays on the CLI:

```bash
# Promote an existing invitee to admin
sudo -u fenja node /opt/fenja/bin/invite.js admin add someone@example.com

# Demote
sudo -u fenja node /opt/fenja/bin/invite.js admin remove someone@example.com

# Who's an admin right now?
sudo -u fenja node /opt/fenja/bin/invite.js admin list
```

The email must already be in `invites` — admin add doesn't create an invite. Invite it first (via the page or `invite.js add`), then promote via CLI.

## Reading Join-CTA clicks

Every press of the final "Join Project Bifrost" CTA is logged to the `bifrost_joins` table. Use `bin/joins.js` to read it:

```bash
# Every click, newest first (id, email, session)
sudo -u fenja node /opt/fenja/bin/joins.js list

# One row per user, with click count + first/last timestamps
sudo -u fenja node /opt/fenja/bin/joins.js summary

# Full click history for a single user
sudo -u fenja node /opt/fenja/bin/joins.js for someone@example.com

# Totals — clicks + unique users
sudo -u fenja node /opt/fenja/bin/joins.js stats
```

One row is written per click (the schema uses auto-increment `id`, not email-as-PK), so re-clicks are preserved. For ad-hoc SQL:

```bash
sudo -u fenja sqlite3 /opt/fenja/data/fenja.sqlite \
  "SELECT email, datetime(clicked_at/1000,'unixepoch') FROM bifrost_joins ORDER BY clicked_at DESC;"
```

## Service control

```bash
sudo systemctl status fenja          # is it running?
sudo systemctl restart fenja         # restart (after config/code changes)
sudo journalctl -u fenja -f          # live log tail
sudo journalctl -u fenja -n 100      # last 100 lines
```

## Deploying code changes

The project lives on a Windows filesystem; deploys run from **WSL** (Ubuntu under Windows 11) because `rsync` and `ssh` come with it and behave identically to Linux. Never deploy from PowerShell directly — Windows path semantics + line endings cause subtle breakage on the VPS.

### One-time WSL setup

```bash
# Inside WSL (Ubuntu):
sudo apt update && sudo apt install -y rsync openssh-client

# Copy your SSH key into WSL's ~/.ssh (if it isn't already there).
# The Windows OpenSSH key lives at C:\Users\<you>\.ssh\ — WSL sees this as:
mkdir -p ~/.ssh
cp /mnt/c/Users/Arlin/.ssh/id_ed25519       ~/.ssh/
cp /mnt/c/Users/Arlin/.ssh/id_ed25519.pub   ~/.ssh/
chmod 700 ~/.ssh
chmod 600 ~/.ssh/id_ed25519

# Sanity check: you should land on the VPS without being prompted for a password.
ssh user@project-bifrost.fenja.ai "hostname"
```

> **File permissions gotcha**: SSH will refuse a key that's group-readable. If it prompts for a password despite a copied key, re-run `chmod 600 ~/.ssh/id_ed25519`.

### Every deploy

> **Destructive-rsync warning**: both rsync steps use `--delete`. Every `--delete` invocation below is paired with `--exclude data --exclude .env --exclude node_modules`; do not omit any of those excludes. The consequence of forgetting them is that `/opt/fenja/data` (SQLite DB + nightly backups) and/or `/opt/fenja/node_modules` gets wiped — the service fails to boot until you recreate them and re-run `npm ci`. Has happened once — April 2026.

```bash
# 1. Open WSL and cd into the repo via its WSL path. The Windows project
#    folder `C:\Users\Arlin\01 DEVELOPMENT\fenja-bifrost` is visible as:
cd "/mnt/c/Users/Arlin/01 DEVELOPMENT/fenja-bifrost"

# 2. Push the tree to a staging dir on the VPS. Notes dirs / secrets /
#    build artefacts are excluded so rsync --delete doesn't nuke things
#    on the server that aren't in your checkout.
rsync -avz --delete \
  --exclude node_modules --exclude data --exclude .env --exclude .git \
  --exclude 'CHANGES*.md' --exclude 'MERGE_NOTES.md' --exclude 'AUTH_SIMPLIFICATION_NOTES.md' \
  ./ user@project-bifrost.fenja.ai:/tmp/fenja-upload/

# 3. SSH in and promote the upload.
#    CRITICAL: the exclude list on the --delete rsync must include
#    `data`, `.env`, AND `node_modules` — all three are intentionally
#    absent from the upload, and without these excludes, --delete
#    wipes the server-side copies. Incident log: April 2026 (data),
#    April 2026 (node_modules, same deploy).
ssh user@project-bifrost.fenja.ai
sudo rsync -a --delete \
  --exclude data --exclude .env --exclude node_modules \
  /tmp/fenja-upload/ /opt/fenja/
sudo chown -R fenja:fenja /opt/fenja
rm -rf /tmp/fenja-upload

# 4. If package.json changed (new dep or version bump):
cd /opt/fenja
sudo -u fenja npm ci --omit=dev

# 5. Take a pre-change DB snapshot if the deploy includes a schema change
#    (new/renamed column, new table — check git diff of src/db.js first):
sudo -u fenja sqlite3 /opt/fenja/data/fenja.sqlite \
  ".backup /opt/fenja/data/pre-change-$(date +%F).sqlite"

# 6. Restart and watch the first ~20 log lines for a clean boot
sudo systemctl restart fenja
sudo journalctl -u fenja -n 20
```

Confirm `[bifrost] listening on 127.0.0.1:3000` appears in the logs. (There is no longer an `[mail] SMTP relay reachable` line — the mail stack was removed in the auth simplification.)

### WSL-specific pitfalls

| Symptom | Cause | Fix |
|---|---|---|
| `rsync: command not found` | WSL image lacks rsync | `sudo apt install rsync` |
| `Permission denied (publickey)` | SSH key missing in WSL, or wrong perms | Copy from `/mnt/c/Users/<you>/.ssh/` and `chmod 600` |
| `CRLF will be replaced by LF` warnings from git / scripts failing on server with `bad interpreter: ^M` | Windows line endings snuck in | In the repo: `git config core.autocrlf input`, then re-save the file. Shell scripts in `bin/` must be LF. |
| Very slow rsync | Running rsync against files on `/mnt/c/` is slower than a native WSL filesystem; acceptable for small deploys, painful for big ones | Fine for this repo (< 50MB). For large trees, clone into `~/repos/` inside WSL instead. |
| `ssh: Could not resolve hostname` | Corporate VPN or DNS quirks | Confirm with `ssh -v`; may need to switch network. |

## Editing env config

`/etc/fenja/env` is intentionally minimal — only `PORT`, `PUBLIC_ORIGIN`, and `NODE_ENV=production`. There are **no secrets** (no pepper, no SMTP, no mail-from): auth is email-only against the invite list, and the mail stack was removed.

```bash
sudo nano /etc/fenja/env
sudo systemctl restart fenja
```

Expected contents:

```
PORT=3000
PUBLIC_ORIGIN=https://project-bifrost.fenja.ai
NODE_ENV=production
```

If you ever add a real secret back (e.g. an analytics token), match the invariant in PROJECT.md: root:fenja, mode 640, never in the repo.

## Backups

Nightly cron at `/etc/cron.d/fenja-backup` snapshots the SQLite file to `/opt/fenja/data/backup-YYYY-MM-DD.sqlite`, keeping 14 days.

Manual snapshot:

```bash
sudo -u fenja sqlite3 /opt/fenja/data/fenja.sqlite \
  ".backup /opt/fenja/data/backup-manual-$(date +%F).sqlite"
```

Pull a backup to your laptop:

```bash
scp user@project-bifrost.fenja.ai:/opt/fenja/data/backup-YYYY-MM-DD.sqlite .
```

## Quick health checks

From **WSL** (bash) on your laptop:

```bash
curl -I https://project-bifrost.fenja.ai/                # expect 200
curl -I https://project-bifrost.fenja.ai/timeline.js     # expect 302 → /
curl -I https://project-bifrost.fenja.ai/api/bifrost-join -X POST   # expect 401 (auth-gated)
```

Or from **PowerShell** — use `curl.exe` (the bare `curl` alias is PowerShell's own `Invoke-WebRequest` with a different flag grammar):

```powershell
curl.exe -I https://project-bifrost.fenja.ai/              # expect 200
curl.exe -I https://project-bifrost.fenja.ai/timeline.js   # expect 302 → /
```

If either fails, check Nginx (`sudo systemctl status nginx`) and Node (`sudo systemctl status fenja`) on the VPS.

## Troubleshooting

| Symptom | First thing to check |
|---|---|
| Invited user can't log in | Confirm the email is actually on the invite list: `node bin/invite.js list`. Email is matched lowercase-trimmed. |
| 502 Bad Gateway | Node crashed — `systemctl status fenja` then `journalctl` |
| 504 Gateway Timeout | Node running but hung — `systemctl restart fenja` |
| Nginx config change broke something | `sudo nginx -t` will tell you exactly what |
| Session cookie shipped without `Secure` | `NODE_ENV=production` missing from `/etc/fenja/env`; add it and restart |

## File locations

```
/opt/fenja/                 code (owned by fenja:fenja)
/opt/fenja/data/            SQLite + nightly backups
/etc/fenja/env              secrets (root:fenja, 640)
/etc/systemd/system/fenja.service
/etc/nginx/sites-available/project-bifrost
/etc/nginx/sites-enabled/project-bifrost  (symlink)
/etc/letsencrypt/live/project-bifrost.fenja.ai/  TLS certs
/etc/cron.d/fenja-backup
```