# xtreambase — Requirements & Architecture Specification (REQ.md)

| Field | Value |
|---|---|
| **Product codename** | Xtreambase |
| **Document status** | v1.8 — Phase 4 COMPLETE + Rule P (DB-first platform settings, admin-configurable) — production-ready |
| **Owner** | Project owner |
| **Last updated** | 2026-08-01 |
| **Related** | `README.md`, original request (see Appendix A) |

> This document is the single source of truth for **what** we build and **how** we architect it.
> It must be updated whenever a phase introduces a new architectural decision.

---

## 1. Product Vision

**xtreambase** is a highly advanced, future-proof platform that lets users build a personal
**archive base of stream data** — video, audio, and playlists.

The core workflow is a focused, immersive workspace ("**Xtreambase**"):

1. The user pastes a **share URL** (or many URLs / a playlist / a channel) into the workspace.
2. The ingestion engine **intercepts, scrapes, and bypasses bot-protections** to extract the media and its metadata.
3. The media is **uploaded to cloud storage** (Vercel Blob for now; S3-compatible storage later).
4. The result is **indexed into the user's personal MongoDB database**, with deduplication and optional overwrite.
5. Advanced modes layer on top: **batch ingestion**, **format/quality controls**, **auto-sync watchers**,
   and **programmatic access via a personal API**.

The platform is a **resilient, proxy-backed ingestion engine**, not a one-off downloader.
It must remain operable when target sites add anti-bot measures, throttle IPs, or change their HTML.

Reference for feature scope (a fraction of our platform): `https://ytdlp.online/playlist-downloader`.

### 1.1 Core value proposition
- **One workspace** for archiving links across many providers (video, music, playlists).
- **Private, personal archive** per user, backed by their own MongoDB database.
- **Resilience by design**: strategy registry, proxy rotation, storage abstraction, async jobs.
- **Automation-ready**: batch inputs, output controls, watchers, and an API for external tools.

### 1.2 Goals
- G1 — Paste a link → get structured media + metadata → uploaded → indexed → visible in the workspace.
- G2 — Survive anti-bot and IP-throttling via a built-in proxy vault and stealth scraping.
- G3 — Guarantee uniqueness of archived media per user unless an explicit overwrite is requested.
- G4 — Zero business-logic changes when pivoting storage (Vercel → S3) or adding scrapers/proxy providers.
- G5 — A productivity-grade UI (Notion/IDE vibe), fully mobile-responsive with bottom-sheet interactions.
- G6 — **Batch:** paste many URLs at once; playlists/channels auto-expand into individual items;
      all items processed concurrently through the job queue.
- G7 — **Format control:** per-request output selection (audio-only, video quality caps, best available)
      chosen in the UI and honored by every scraper strategy.
- G8 — **Watchers:** track a channel/playlist URL for automatic, scheduled ingestion of *new* media
      without manual intervention (architecture ready for cron schedulers).
- G9 — **API-first:** users can generate a personal API key and push links into their queue from
      anywhere on the web (iOS Shortcuts, Zapier, scripts).

### 1.3 Non-goals (for now)
- NG1 — Social features / public sharing of archives.
- NG2 — On-platform media transcoding (raw extraction only at MVP; format selection is extraction-level).
- NG3 — Support for paywalled/DRM content.
- NG4 — A fully distributed worker fleet (we *anticipate* it via Rule F, but do not build it in Phase 1–3).
- NG5 — Outbound webhook *delivery* to third parties (design space only; inbound API is in scope).

---

## 2. Tech Stack Mandate (Non-Negotiable)

| Layer | Choice |
|---|---|
| Framework | **Next.js 16** (App Router, Server Actions) — *latest stable; supersedes the original Next.js 15 mandate per owner instruction* |
| Language | **TypeScript** (strict mode, `strict: true`) |
| Database | **MongoDB** via **Mongoose 9** (schemas with flexible `Mixed` metadata) |
| Auth | **NextAuth.js (Auth.js v5, `next-auth` v5 beta)** — UI auth; API auth via personal keys (Rule J) |
| UI/UX | **Tailwind CSS 4** + **shadcn/ui**; workspace-centric, mobile-responsive |
| Scraping | **youtube-dl-exec** AND **puppeteer-extra-plugin-stealth** (or Playwright) + **DirectFileStrategy** (raw file URLs) |
| Queue | **Agenda.js v6** (MongoDB-backed, native `agendaJobs` collection — no extra Redis) |
| Scheduler | **Agenda `every()` cron** for watcher auto-sync (Rule I); runs inside the worker process |
| Deploy | **Dokploy (Docker)** — two containers: web (`next start`) + worker (`node dist/worker.js`) |

---

## 3. Core Architectural Rules (Non-Negotiable)

### Rule A — The Scraper Strategy Registry (Future-Proof Extensibility)

**Problem.** A single scraper cannot handle every target. Providers differ in anti-bot
aggressiveness, site structure, and media formats.

**Solution.** Implement a **Scraper Registry Pattern**:

- A common interface `IScraperStrategy`:
  - `canHandle(url: URL): boolean` — capability routing (regex/domain/format detection).
  - `extract(url: URL, options: ExtractOptions): Promise<ExtractResult>` — perform the extraction.
- A **registry** that holds all registered strategies, ordered by priority, and routes each URL
  to the first strategy whose `canHandle()` returns `true`.

**Baseline strategies:**

| Strategy | Purpose |
|---|---|
| `DirectFileStrategy` | **Raw file URLs** (CDNs/static hosts): detects by file extension (`.mp4`, `.mp3`, `.wav`, `.m4a`, `.png`, `.jpg`, `.pdf`, `.zip`, `.tar.gz`, …) OR async **HTTP HEAD Content-Type** probe. Bypasses yt-dlp/Puppeteer — streams the file straight into the StorageOrchestrator; subject to the same global `(userId, canonicalUrl)` dedup + multi-Base linking. Highest priority in the registry. |
| `YtDlpStrategy` | Standard media: single videos, audio, playlists via `youtube-dl-exec`. Supports `--proxy`, format selection, metadata (`--dump-json`), and **playlist expansion/inspection** (`--flat-playlist`). |
| `StealthBrowserStrategy` | Heavy anti-bot / same-site impersonation: Puppeteer with `puppeteer-extra-plugin-stealth`, header injection, cookie spoofing, human-like navigation, then intercepts the media stream/URLs. |
| `PageHarvestStrategy` | **Multi-media pages** (Phase 4.5): when a pasted URL's PAGE embeds several videos/audio (article embeds, gallery players, custom hubs) OR loads its real stream URLs from a backend JSON/API, this strategy renders the page in a stealth browser, sniffs **every** network response (media content-types + JSON/API response BODIES, walked recursively for media URLs), plus DOM `<video>`/`<audio>`/`<source>` elements, and returns one `InspectItem` per unique media URL. The registry consults it during `inspect()`/`expand()` when the primary strategy yields nothing or a single item, so the UI's selection modal can fan out one item per media blob. `extract()` streams the best resolved URL as a functional fallback. Gated by `PAGE_HARVEST=off`; hosts where yt-dlp is authoritative are skipped (`HARVEST_SKIP_HOSTS`). Requires the same browser as stealth (auto-discovered, or `PUPPETEER_EXECUTABLE_PATH`). |

**Inspection capability (Selective Playlist Import):** every strategy can optionally implement
`inspect(url, options): Promise<InspectItem[]>` returning **lightweight per-item metadata**
`{ id?, title?, thumbnail?, duration?, url, index }` **without downloading media bytes**. The
registry's `inspect()` uses an async resolver that also runs `canHandleAsync` probes (e.g. HEAD
Content-Type for extensionless direct files) before falling back to a one-item list.

**Worker-side inspection (Phase 4):** inspection is heavy scraping and **runs in the Agenda
worker, never the web process** — the Next.js web bundle rewrites `require("youtube-dl-exec")`
to a virtual path (`/ROOT/…`) and dies with `MODULE_NOT_FOUND`, while the plain-Node worker
resolves it fine. `POST /api/scraper/inspect` only persists an `InspectJob` doc (TTL auto-expiry,
`INSPECT_JOB_TTL_MINUTES` default 30) and enqueues **`INSPECT_URLS`**; the worker performs the
inspection + `archived` dedup annotation and stores per-URL results; the UI polls
`GET /api/scraper/inspect/[id]` until `succeeded` and then opens the PlaylistSelectorModal.
`loadYoutubeDl()` gained a cwd-based require fallback so web-side live streaming
(`POST /api/v1/extract` `save:false`) also resolves the module.

**Fallback chain (broadened scraping, Phase 3.10):** `ScraperRegistry.extractWithFallback()`
replaces the single-strategy `extract()` in the worker and `/api/v1/extract`. It tries **every
capability-matched strategy in priority order** (direct-file → yt-dlp → stealth-browser),
recording each failure, then — for **page-like URLs only** (never obvious raw-file URLs) and
unless `STEALTH_FALLBACK=off` — any strategy declaring `isUniversalFallback`
(`StealthBrowserStrategy`: a real browser can render almost any page, so a bot-checked yt-dlp
URL gets a second chance in Puppeteer). If ALL strategies fail, one `ExtractionFailedError`
aggregating every attempt is thrown (still classified non-retryable when the underlying cause
is a bot-check/DMCA). The **winning strategy is tagged as `metadata.strategy`** and recorded on
the `ArchiveJob` for observability. `StealthBrowserStrategy` now performs **server-side byte
resolution** of the in-page media URL (via `fetch-utils.ts`, through the Proxy Vault when
injected), rejecting `blob:`/`data:` URLs with a clear error — making it a fully-functional
fallback rather than a metadata-only placeholder. Shared `fetchStream()`/`fetchViaProxy()`
helpers centralize proxy + redirect + timeout semantics across strategies.

**Extensibility contract:** adding a scraper in the future = add a class to `/scrapers` +
register it (one line). **No changes** to the routing core, UI, storage, or job pipeline.

> **Acceptance:** the registry routes a URL to the correct strategy by capability; unknown URLs
> fail gracefully with a typed error; the registry is strategy-count-agnostic; inspect() returns
> metadata for playlists/channels and never fetches media bytes; a failed strategy falls through
> to the next capable one (and to the stealth browser for page URLs) instead of failing the job.

---

### Rule B — Internal Proxy Management & Rotation (The Proxy Vault)

**Problem.** Aggressive scraping gets IPs banned. External proxy APIs add cost and coupling.

**Solution.** The system has its **own built-in proxy management** to evade IP bans.

**Data model — `Proxy` collection:**
| Field | Type | Notes |
|---|---|---|
| `protocol` | string | `http` \| `https` \| `socks4` \| `socks5` |
| `ip` | string | Host / IP |
| `port` | number | Port |
| `auth` | object (optional) | `{ username, password }` |
| `healthStatus` | enum | `unknown` \| `healthy` \| `unhealthy` \| `disabled` |
| `lastUsed` | Date | Rotation bookkeeping |
| `lastCheckedAt` | Date | Last health-check timestamp |
| `failureCount` | number | Consecutive failed checks |

**Manager logic — `ProxyManager` service:**
- **Rotation:** round-robin selection (`lastUsed` ordering), skipping unhealthy/disabled proxies.
- **Health checking:** periodic and on-demand checks (small HEAD request through the proxy, or
  via a strategy test); transient failures increment `failureCount`; persistent failures flip
  `healthStatus` to `unhealthy` (with optional auto-cooldown).
- **Per-request integration:** the `ProxyManager` injects a live proxy into the chosen strategy —
  `--proxy <proxy>` into yt-dlp, and the proxy into the Puppeteer `browserContext`/launch args.

**Extensibility:** selection policy is swappable (round-robin default; weighted/sticky later)
without touching scraper code.

> **Acceptance:** a dead proxy is never used twice in a row when healthy ones exist; health
> state is persisted; strategies receive a proxy object per request, never hardcode one.

---

### Rule C — The Storage Adapter Pattern + Orchestration (Quotas, Fallback Chain, Masked Downloads)

**Problem.** App code must **never hardcode cloud storage SDKs**, must survive provider
capacity/outage limits, and must never leak raw storage URLs to the client.

**Solution.** A three-layer stack:

```ts
interface IStorageAdapter {
  upload(input: { stream: Readable | Buffer; filename: string; contentType?: string; metadata?: object }): Promise<{ storageUrl: string; storageKey?: string; sizeBytes?: number }>;
  delete(storageUrl: string): Promise<void>;
  /** Secure internal resolution — never expose the raw URL to the client */
  getDownloadUrl(fileId: string, opts?: { expiresInSeconds?: number }): Promise<string>;
}
```

**C.1 — `StorageConfig` (quotas & fallbacks):**

```ts
{
  name: string;
  driver: 'vercel' | 's3';
  enabled: boolean;
  isDefault: boolean;
  capacityBytes: number;      // max quota; 0 = unlimited
  usedBytes: number;          // running usage, updated on upload/delete
  fallbackNodeId?: ObjectId;  // ref StorageConfig — next node in the fallback chain
  config: Mixed;              // driver-specific (token / bucket / region / keys)
}
```

**C.2 — `StorageOrchestrator` (Fallback Chain):** before an upload it walks nodes in
priority order (default node first). It skips a node when `usedBytes >= capacityBytes`
(quota full), and **catches provider errors (e.g. 5xx)** and follows `fallbackNodeId`
to retry on the next node. Only when the entire chain is exhausted does it raise
`StorageError`. `usedBytes` is credited on upload and debited on delete.

**C.3 — Masked Internal Downloads:** raw storage URLs never reach the browser. UI calls
`/api/media/[id]/download`, which verifies the session, loads the user-scoped `Media`
doc, resolves its `StorageConfig`, and returns a **short-lived URL (5-min presigned for
S3; stored URL for Vercel Blob)** via a 307 redirect. `Media` stores `storageNodeId` +
`storageKey` so the orchestrator can resolve the file without exposing provider URLs.

- **`VercelStorageAdapter`** — Vercel Blob (`@vercel/blob`).
- **`S3StorageAdapter`** — `@aws-sdk/client-s3` + `@aws-sdk/s3-request-presigner`.
- Business logic (scraper pipeline, dedup, overwrite) depends **only on the interface**.

> **Acceptance:** switching `STORAGE_DRIVER=vercel` → `s3` requires **zero** business-logic
> changes; a full/errored node transparently fails over to its fallback; no raw storage URL
> is ever serialized to the client.

---

### Rule K — Multi-Base Management & Global Storage Deduplication

**Problem.** Users archive into one undifferentiated list; the same source URL pasted into a
second collection would waste storage on a duplicate file.

**Solution.** A `Base` collection gives users named, colored organization ("Music",
"Courses", …) while storage deduplication is **global per user**.

**Data model — `Base` collection:**

```ts
{
  userId: ObjectId;         // ref User
  name: string;             // unique per (userId, name)
  description?: string;
  color: 'slate'|'red'|'orange'|'amber'|'green'|'teal'|'blue'|'indigo'|'violet'|'pink';
  createdAt: Date;
}
```

**`Media` schema upgrades:**
- `baseIds: [ObjectId]` — a single stored file can belong to one or more Bases (shared ref).
- `formatOption: 'mp4' | 'mp3' | 'original'` — the conversion/container intent chosen in the UI.
- `canonicalUrl: string` — normalized URL (tracking params + fragment stripped, host lowercased).
- **Unique index `(userId, canonicalUrl)`** replaces `(userId, sourceUrl)` as the dedup key —
  one stored file per user per source **across ALL bases**.

**Fallback base ("Others", auto-managed):** every user gets exactly one auto-created base
(`Base.isFallback: true`, name "Others", protected from deletion via a partial unique index on
`(userId, isFallback=true)`). Unassigned media — placeholder `kind:"other"` items, failed/
unscraped cards, items whose base was deleted — is linked into it automatically so nothing falls
into an unmanageable dead-end "others" bucket:
- `lib/bases-fallback.ts` — `ensureFallbackBase(userId)` (find-or-create, race-safe, also
  **backfills** media with zero `baseIds` into the fallback) + `getFallbackBaseId(userId)` (cached).
- Enqueue path (`createItem`, archive actions) auto-targets the fallback when the user selected
  no base; `retryMedia`/`deleteMedia` preserve fallback membership on retry.
- The fallback base is fully CRUD-able like any base (sidebar badge "auto", deep page,
  per-item retry/delete via the new `lib/actions/media.ts` actions).

**Global dedup flow (single upload, multi-base link):**
1. User pastes URLs and picks target Bases (default: the active Base).
2. `archiveUrls` canonicalizes each URL and checks `Media (userId, canonicalUrl)`.
3. **Exists?** → no yt-dlp/Puppeteer, no storage upload — `$addToSet: { baseIds }` on the
   existing document, instant toast "Linked existing media to selected Base(s) without extra
   storage usage!".
4. **New?** → queue `EXPAND_BATCH` (→ `INGEST_MEDIA`) with `baseIds` + `formatOption`; the
   worker extracts, converts, uploads **once**, and writes `baseIds` on the Media document.
5. The worker's `INGEST_MEDIA`/`EXPAND_BATCH` re-check the canonical dedup as a race guard
   and take the same link-only path when the file appeared meanwhile.

**Format conversion pipeline (`formatOption`):**
- `mp4` (default) — best video+audio merged into an MP4 container (`mergeOutputFormat: mp4`).
- `mp3` — audio-only extraction + conversion (`-x --audio-format mp3`).
- `original` — raw best stream, unprocessed.
- `YtDlpStrategy.formatToYtdlp` maps `formatOption` → yt-dlp flags; `StealthBrowserStrategy`
  carries the intent in metadata for the Phase 4 byte-resolution pass.

**UI:** sidebar "Bases" section (+ create dialog, `auto` badge on the fallback base), base chips
on the submission form, per-base filtering (`/workspace?base=<id>` + `/api/workspace/media?base=`),
and a "Manage Bases" action on every media card (attach/detach via `setMediaBases`). Failed
cards additionally get **Retry** (`retryMedia` → re-enqueue through the worker) and **Delete**
(`deleteMedia` → best-effort cloud file cleanup + record + related queued jobs), so the
fallback base's failed/unscraped items are never a dead-end.

> **Acceptance:** pasting the same URL into two bases stores the file once; deleting a base
> detaches (never deletes) media; MP3/MP4/Original selections reach the scraper unchanged.

---

### Rule D — MongoDB Deduplication & Overwrite Logic

**Data model — core collections:**

```ts
// User — one archive per authenticated user
{
  email: string;            // unique
  name?: string;
  image?: string;
  role: 'user' | 'admin' | 'superadmin';  // RBAC: superadmin > admin > user
  passwordHash?: string;
  createdAt: Date;
}
```

**RBAC & first-boot setup:** roles gate `/admin/*`, storage config writes, and the proxy
vault (`requireAdmin` / `requireSuperAdmin` guards). On **first boot** (no active
`superadmin` exists) the app redirects to `/setup`, which creates the root admin account;
the page and its server action refuse to run once a superadmin exists. `ADMIN_EMAILS`
remains an optional bootstrap to promote matching accounts. The `User` schema enum is
`['user', 'admin', 'superadmin']` — it must always include `superadmin` or the setup
wizard's `User.create({ role: 'superadmin' })` fails Mongoose validation.

**Admin shell (Phase 3.5):** `/admin/*` is wrapped in `app/admin/layout.tsx` which reuses
`WorkspaceShell` — admins keep the sidebar (incl. the Administration section), user menu,
and mobile drawer on every admin page. `/admin` is an overview landing (storage nodes,
proxy health, media archived, users) with quick links to the Storage Vault and Proxy Vault.

// Media — one document per archived source
{
  userId: ObjectId;         // ref User
  sourceUrl: string;        // original pasted URL
  title: string;
  kind: 'video' | 'audio' | 'playlist' | 'other';
  storageUrl?: string;      // where the media lives
  thumbnailUrl?: string;
  metadata: Mixed;          // flexible, site-specific data (Mongoose Mixed)
  formatRequested?: FormatConfig; // what the user asked for (Rule H)
  status: 'pending' | 'processing' | 'ready' | 'failed';
  sizeBytes?: number;
  durationSeconds?: number;
  createdAt: Date;
  updatedAt: Date;
}
```

**Uniqueness:** unique compound index on `(userId, sourceUrl)` — one archive per source per user.

**Overwrite UX:**
1. Default: an incoming archive request for an existing `(userId, sourceUrl)` is **rejected**
   with a `DUPLICATE` result. The user sees "Already archived" in the UI.
2. If the request includes `overwrite: true`:
   - `IStorageAdapter.delete()` the **old** `storageUrl` (if any),
   - run the **new** extraction,
   - `findOneAndUpdate` the existing `Media` document (atomic, upsert-safe).

> **Acceptance:** concurrent duplicate requests do not create duplicate rows (index enforces it);
> overwrite keeps a single document and never leaves orphaned storage files.

---

### Rule E — The Workspace UI & Mobile Responsiveness

The UI is a **focused productivity tool** ("Xtreambase"), Notion/IDE vibe:

- **Sidebar navigation** — Workspace sections (Archive, Watchers, Proxy Vault, Settings, Account);
  **collapsible on desktop, drawer on mobile**.
- **Central archive view** — grid/list toggle for the user's media, with an archive
  **command bar** (paste URL → archive, **bulk-paste multi-line**, format selector).
  Inline status for `pending / processing / ready / failed`.
- **Slide-out "Inspector" pane** — metadata details for a selected media item
  (source URL, storage URL, site-specific metadata, size, duration, timestamps, overwrite action).
- **Watchers UI** — create/track channel or playlist URLs, toggle auto-sync on/off, view last-synced
  and newly-ingested counts (Rule I).
- **Proxy Vault UI** — admins/users can: bulk-paste proxy lists, **test proxy health**,
  **enable/disable** proxies, view health status and last-used, see rotation stats.
- **Settings** — Personal API key management: generate/revoke keys, copy-once reveal, scopes (Rule J).
- **Mobile experience is critical:**
  - Action sheets become **bottom sheets** on small viewports.
  - Sidebar collapses to a drawer; Inspector pane becomes a full-screen slide-over.
  - Touch targets ≥ 44px; primary actions thumb-reachable.

> **Acceptance:** full UX parity on 360px → 4K viewports; keyboard-first on desktop; bottom
> sheets on mobile.

---

### Rule F — Async / Job Queues (Agenda.js, MongoDB-Native)

**Problem.** Scraping and downloading large media takes time and **will break Vercel
serverless limits** (10s body / 60s max duration for Hobby/Pro defaults).

**Solution (DECIDED).** **Agenda.js v6** backed by our **existing MongoDB** — no extra Redis
container on Dokploy. Server Actions/API routes **never** run long downloads inline; they only
enqueue work into the native `agendaJobs` collection.

- **Queue setup** (`lib/queue/agenda.ts`): a singleton `Agenda` instance reuses the **same
  Mongoose connection** (`mongo: <Db>`), storing every record in the native `agendaJobs`
  collection (`AGENDA_COLLECTION`, default `agendaJobs`).
- **Dual-entry point (Dokploy):**
  - **Web container** — `npm run start` / `next start` (also `next dev` for local web).
  - **Worker container** — `npm run worker` / `node dist/worker.js` (bundled by esbuild from
    `src/worker.ts`; `tsx watch src/worker.ts` for local).
  - **Unified spinup:** `npm run dev:all` and `npm run start:all` run both processes under
    `concurrently` with color-coded streams (`web` cyan / `worker` magenta).
- **Worker env parity:** the web app auto-loads `.env.local`/`.env` (Next.js), but standalone
  `tsx`/`node` workers do not. `scripts/load-env.cjs` (preloaded via `-r`) loads `.env.local`
  then `.env` with Next.js precedence (real env vars win), so `dev:worker`/`start:worker`/`worker`
  see exactly the env the web app sees.
- **Job definitions** (`lib/queue/jobs.ts`) — registered via `registerJobs(agenda)`:
  - **`INGEST_MEDIA`** `{ jobId, url, userId, format, overwrite, batchId?, watcherId? }` —
    rotates a proxy via the Proxy Vault, executes the Scraper Strategy (yt-dlp or stealth),
    streams the upload through the StorageOrchestrator **fallback chain**, and upserts the
    `Media` document with Rule D dedup/overwrite semantics.
  - **`EXPAND_BATCH`** `{ urls, userId, format, overwrite, batchId, jobId }` — fans playlists/
    channels out (Rule G), dedups each item, and enqueues one `INGEST_MEDIA` per item.
  - **`CRON_WATCHER`** — scheduled via `agenda.every()` (default `*/30 * * * *`,
    `WATCHER_CRON=off` disables); diffs each enabled `Watcher` source against existing `Media`
    and enqueues `INGEST_MEDIA` for new items (Rule I).
  - **`INSPECT_URLS`** `{ inspectJobId, userId, urls }` (Phase 4) — worker-side playlist/batch
    inspection: runs `scraperRegistry.inspect()` per URL (yt-dlp needs the worker runtime),
    annotates `archived` dedup flags with one batched Media query, and stores per-URL results
    on the `InspectJob` doc for the UI to poll. Web process only *produces* this job.
- **Retry policy:** every job definition uses Agenda's native **exponential backoff**
  (`exponential({ delay: 30s, factor: 2, maxRetries: JOB_MAX_RETRIES, jitter: 0.2 })`) so a failed
  proxy or a disconnected stream during a yt-dlp/Puppeteer run auto-retries 30s → 1m → 2m → 4m → 8m.
  **Hard retry cap:** `JOB_MAX_RETRIES` (default 5, max 20) feeds BOTH the Agenda backoff AND a
  worker-side guard — `INGEST_MEDIA` stops rethrowing once `failCount >= JOB_MAX_RETRIES` and marks
  the ArchiveJob failed permanently. A job can therefore **never retry infinitely**, regardless of
  queue config. Each retry re-runs the full extraction, so the cap doubles as a cost guard.
- **Fail-fast for deterministic failures:** `isNonRetryableError()` (lib/errors.ts) classifies
  bot-checks ("Sign in to confirm you're not a bot"), unavailable/private/DMCA videos, and
  geo-blocks as permanent — `INGEST_MEDIA` marks the job failed immediately instead of burning
  the 5 retries (which can never succeed for a permanent block).
- An `ArchiveJob` document tracks: `userId`, `sourceUrl`, `strategy`, `status`, `attempts`,
  `resultRef`, timestamps — plus **`batchId`** (Rule G) and **`watcherId`** (Rule I) tags.
- The **immediate request path** returns a `batchId` + `status: queued` synchronously while the
  heavy work happens off-path; the UI polls `/api/workspace/media` for status.

> **Acceptance:** no web request performs a full download synchronously; a dead proxy or dropped
> stream retries with exponential backoff up to `JOB_MAX_RETRIES` and then fails permanently (never
> infinite); the worker is a standalone process (Docker container) sharing only MongoDB.

---

### Rule G — Batch Operations & Playlist Expansion

**Problem.** Users want to archive many items at once: pasted link lists, playlists, or whole channels.

**Solution.**

- **Bulk input:** the command bar accepts **multiple URLs separated by newlines**
  (`archiveUrls({ urls, format, overwrite, baseIds })`).
- **Playlist/Channel detection:** `IScraperStrategy` gains an optional capability
  `canExpand(url): boolean` and `expand(url, options): Promise<ExpandedItem[]>` (or flat item URLs).
  - `YtDlpStrategy` expands playlists/channels via `--flat-playlist --dump-json`.
  - `StealthBrowserStrategy` expands by scraping channel/playlist pages.
  - The **registry** detects playlist/channel URLs at route time and routes them to the
    expanding strategy; unknown expansion requests fail with a typed error.
- **Selective Playlist / Batch Ingestion (Phase 3.5, async in Phase 4):** pasting a playlist
  URL or a multi-line batch triggers `POST /api/scraper/inspect`, which now **enqueues an
  `INSPECT_URLS` worker job and returns `{ jobId }` immediately** (the web process cannot run
  yt-dlp — see Rule A). The UI polls `GET /api/scraper/inspect/[id]` and opens the
  **PlaylistSelectorModal** with lightweight per-item metadata (title, thumbnail, duration,
  `archived` dedup flag). The modal lets the user pick items before anything is queued:
  - Select All / Deselect All + live counter ("12 of 45 selected"), search filter.
  - **Global and per-item format dropdowns** (MP4 / MP3 / Original) — per-item overrides win.
  - Base chips to assign the chosen items (defaults to the active Base).
  - "Enqueue Selected Items" → `archiveSelectedItems({ items: [{ url, format }], baseIds })`
    enqueues **only the checked items** straight into `INGEST_MEDIA` (no re-expansion); global
    dedup still links ready files instead of re-downloading.
- **Fan-out pipeline:** for each input URL — resolve strategy → if `canExpand`, expand → apply
  dedup per item (Rule D) → enqueue **one `ArchiveJob` per item** tagged with a shared `batchId`.
  **Duplicate items inside a batch are reported, not dropped:** each duplicate gets a per-item
  `duplicate` result in batch progress (the user sees "Already archived" on that row) and is
  skipped for processing; only items with `overwrite: true` re-run the overwrite path.
- **Concurrency:** the job queue processes items of the same batch with a **configurable
  concurrency limit** (provider/plan-aware), and the UI shows per-item + batch progress.

> **Acceptance:** pasting 1 URL or 50 newline-separated URLs behaves consistently; a playlist URL
> fans out to individual `Media` docs; per-item failures don't abort the batch; batch caps prevent
> abuse; users can surgically pick items + per-item formats before queuing.

---

### Rule H — Format & Quality Controls (Extraction-level Transcoding Settings)

**Problem.** Users must be able to say *what* they want before scraping (audio only, capped quality, best).

**Solution.**

- A typed **`FormatConfig`** flows UI → Server Action → `ExtractOptions` → strategy:

```ts
type FormatConfig = {
  output: 'best' | 'video' | 'audio';
  videoQuality?: '2160p' | '1440p' | '1080p' | '720p' | '480p' | 'best';
  audioFormat?: 'm4a' | 'mp3' | 'opus';
};
```

- **UI:** a format selector in the archive command bar ("Extract Audio Only", "Video: 1080p max",
  "Best Available") and per-watcher defaults (Rule I).
- **Strategy mapping:** `YtDlpStrategy` translates to yt-dlp `-f` format expressions and `-x` for
  audio-only; `StealthBrowserStrategy` resolves quality in-page and picks the matching stream.
- **Persistence:** `formatRequested` is stored on `Media` (and `ArchiveJob`) for audit and future
  re-download in a different format (note: actual transcoding is a later-phase, out of scope for MVP).

> **Acceptance:** every strategy honors the same `FormatConfig` shape; UI selections reach strategies
> unchanged; invalid combos are rejected up-front by shared validation.

---

### Rule I — Watchers (Auto-Sync Subscriptions)

**Problem.** Users want to track a creator's channel or a playlist and auto-archive *new* media.

**Solution.**

**Data model — `Watcher` collection:**

```ts
{
  userId: ObjectId;          // ref User
  sourceUrl: string;         // channel / playlist / single URL
  kind: 'channel' | 'playlist' | 'single';
  title?: string;
  enabled: boolean;
  format?: FormatConfig;     // output preference for auto-ingest (Rule H) — incl. formatOption
  baseIds: [ObjectId];       // Bases auto-ingested media is linked to (Bases sync)
  lastCheckedAt?: Date;
  lastSyncedAt?: Date;
  lastSyncCount?: number;    // items ingested in the last run
  createdAt: Date;
  updatedAt: Date;
}
// Unique compound index (userId, sourceUrl)
```

- **Sync flow:** for each enabled watcher → resolve strategy via registry → expand/scan source →
  **diff against existing `Media` (Rule D dedup handles this)** → enqueue `ArchiveJob`s for new items.
  Nothing new to ingest → no-op run, `lastSyncedAt` updated.
- **Bases + format sync (Phase 3.5):** watchers carry `baseIds` (auto-link new media to chosen
  Bases) and a `format` with `formatOption` (MP4/MP3/Original) — the same format pipeline as the
  command bar. The UI selector mirrors the workspace format dropdown.
- **Scheduler-ready architecture:** a `SchedulerProvider` interface (mirroring the queue interface)
  with adapters for **Vercel Cron**, **node-cron**, or **Inngest schedules**. Phase 1–3 implement the
  interface + a manual "Sync now" trigger; the real schedule is wired in Phase 4.
- **Respect for providers:** watcher runs share the same proxy rotation and rate-limit safeguards
  as manual archives.

> **Acceptance:** watchers only ingest genuinely new items (dedup index is the source of truth);
> disabling a watcher stops scheduled runs; a "Sync now" action works before cron wiring exists.

---

### Rule J — Personal API & Webhooks (API-First Ingestion) + Developer API Gateway ✅ COMPLETE (Phase 3.6)

**Problem.** Users want to push links into their queue from external tools (iOS Shortcuts, Zapier, scripts)
and query their archive programmatically.

**Solution.**

**Data model — `ApiKey` collection (upgraded):**

```ts
{
  userId: ObjectId;          // ref User
  name: string;              // e.g. "iOS Shortcut"
  prefix: string;            // visible id, e.g. "xa_live_3f8a…"
  keyHash: string;           // sha256 of the full key — NEVER store plaintext
  scopes: string[];          // e.g. ['archive:create']
  autoSave: boolean;         // default true — persist archives; false = live-stream only
  targetBaseIds: [ObjectId]; // default Bases new archives are linked into (Rule K)
  defaultFormat: 'mp4'|'mp3'|'original';  // default output format (Rule H)
  lastUsedAt?: Date;
  revokedAt?: Date;
  createdAt: Date;
}
```

- **API key auth guard (`lib/auth/api-key-guard.ts`):** intercepts `Authorization: Bearer xa_live_…`,
  hashes the token (SHA-256), looks it up (revoked keys rejected), attaches the owner + key profile,
  and **asynchronously bumps `lastUsedAt`**. Key verification runs in the route handler (Node runtime),
  not Edge middleware — revocation is immediate.
- **Settings UI (`/workspace/settings/api`):** generate a key (full value shown **once** in a copy
  sheet), name + auto-save toggle + target-Base chips + default format; list active keys with
  last-used; revoke immediately. Interactive curl / JS / Shortcut snippets are rendered inline.

**API surface (v1) — live now:**
- **`POST /api/v1/extract`** — Live Extraction Gateway. Body `{ url, save?, baseIds?, format?, customMeta? }`;
  omitted fields fall back to the key's defaults (`autoSave`, `targetBaseIds`, `defaultFormat`).
  - `save:false` → executes the Scraper Registry **inline** and **streams the raw media bytes** back
    (Content-Disposition attachment; zero DB writes).
  - `save:true` → runs **global dedup**: new files are queued as `INGEST_MEDIA` with `customMeta`
    injected (worker extracts/uploads); existing files are **linked + enriched** (merge
    `metadata.custom`, `$addToSet` baseIds) without re-downloading. Returns
    `{ status, saved, mediaId?, downloadUrl?, streamData }` where `downloadUrl` is the masked
    `/api/media/[id]/download` endpoint.
- **`GET /api/v1/media`** — Deep Dynamic Query API. Parses arbitrary query params into MongoDB
  dot-notation filters: `ext.<key>=value` → `metadata.custom.externalIds.<key>` (e.g. `?ext.tmdb=550`),
  `attr.<key>=value` → `metadata.custom.attributes.<key>` (e.g. `?attr.course=CS101`), `tag=value`,
  plus standard `baseId`, `type`, `search`, `page`, `limit`. Returns Media docs with masked
  `downloadUrl` + normalized dynamic metadata.
- **`GET /api/v1/bases`** — lists the key owner's Bases (IDs for targeting archives).
- **Rate limiting:** every `/api/v1/*` request is rate-limited per key (default 60 req/min;
  `API_RATE_LIMIT` / `API_RATE_WINDOW_MS`), returning 429 with `Retry-After` (Rule M).
- **Inbound webhooks** are the same routes — an iOS Shortcut/Zapier webhook simply POSTs to `/extract`.
- **Service-layer reuse:** the API route calls the same `createItem`/dedup helpers as the Server Actions
  (Rules D/G/K) — no duplicated business logic.
- **Outbound webhooks (design space, not built):** future completion notifications to user-configured
  URLs with HMAC-signed payloads.

> **Acceptance:** a key can enqueue or live-stream from curl/Shortcut with per-request custom
> metadata; revoked keys are rejected; full keys never touch the DB in plaintext; dynamic metadata
> is queryable over the API.

---

### Rule L — The Dynamic Metadata Engine (Truly Dynamic, Platform-Agnostic)

**Problem.** Hardcoding external platforms (TMDB, IMDb, …) couples the schema to specific providers.
Users need to attach **arbitrary** metadata to archived media and query it deeply.

**Solution.** `Media.metadata` is a flexible nested structure — no platform keys are hardcoded:

```ts
metadata: {
  extracted: Mixed,   // raw data straight from the scraper strategy (title, duration, urls, …)
  custom: {           // user-defined / API-injected — 100% dynamic
    externalIds: Map<String>,   // mapping to ANY external system: {"tmdb":"123","notion_id":"xyz"}
    attributes:  Map<Mixed>,    // arbitrary key-value pairs: {"course":"CS101","rating":5}
    tags:        [String]       // free-form labels, indexed
  }
}
```

- **Wildcard index:** MongoDB allows **one wildcard index per collection**, so both custom maps are
  covered by a single index using `wildcardProjection` on
  `metadata.custom.externalIds` + `metadata.custom.attributes`. Any dynamically added key becomes
  queryable (`?ext.tmdb=550` / `?attr.course=CS101`) without migrations.
- **Legacy compatibility:** `normalizeMediaMetadata` (lib/metadata.ts) treats pre-upgrade flat
  `metadata` as `extracted`, so old documents keep rendering without a migration job.
- **Injection points:** the workspace submission form has an **Advanced Metadata builder** (dynamic
  key-value rows for external IDs + attributes, plus a tag chip input); `archiveUrls` /
  `archiveSelectedItems` thread `customMeta` through `EXPAND_BATCH` → `createItem` → the Media
  placeholder; the worker writes scraper data into `metadata.extracted` only, so user custom data is
  never clobbered; re-links **merge** (`$set` per key + `$addToSet` tags) instead of replacing.
- **Inspector editing:** the slide-out Inspector renders `metadata.custom` live with add/edit/delete
  (server actions in `lib/actions/media-meta.ts`), plus the read-only extracted view.
- **Query API:** `GET /api/v1/media` exposes dot-notation queries over both maps + tags (Rule J).

> **Acceptance:** any key can be injected from the UI or API without a schema change and is
> immediately queryable; existing documents remain readable.

---### Rule M — Queue Management: Modal Feedback, CRUD, Start/Stop & Auto-Deletion

**Problem.** After clicking Archive, users saw only a toast. They needed live feedback on the queued
jobs themselves (queued → processing → succeeded/failed/cancelled, with attempts + error detail), the
ability to **control congestion** (cancel/retry/delete jobs, start/stop the queue), automatic cleanup
of old job records, and base pages that show **archived files only** — never job queues.

**Solution.**
- **`GET /api/workspace/jobs`** — polling endpoint returning the user's recent `ArchiveJob` records
  (serialized, owner-scoped) **plus `queuePaused`** for the start/stop control.
- **`QueueControl` modal** (button on `/workspace`): live job feed with status icons/badges, attempts,
  error detail (including truncated yt-dlp stderr), batch/watcher tags, and relative timestamps;
  polls every 4s while any job is active **or the modal is open** so finished states appear live.
  - **Pause/Resume (start/stop):** `setQueuePaused` toggles `User.queuePaused`. While paused, the
    archive Server Actions (`archiveUrls`, `archiveSelectedItems`) and `POST /api/v1/extract`
    (save:true) refuse with a clear message; existing jobs drain. The queue button shows a paused
    indicator.
  - **Job CRUD:** `lib/actions/jobs.ts` — `cancelJob` (removes the pending Agenda job via
    `agenda.cancel({ "data.jobId": id })`, marks the ArchiveJob `cancelled` (reason `user`), flags
    the pending Media placeholder as failed), `retryJob` (re-enqueues `INGEST_MEDIA` with the
    original format/tags, resets status to `queued`), `deleteJob` (finished records only),
    `clearFinishedJobs` (removes all succeeded/failed/cancelled).
  - **Auto-deletion:** `CLEANUP_JOBS` agenda job (registered in `lib/queue/jobs.ts`, scheduled daily
    from `src/worker.ts` via `CLEANUP_CRON`, default `0 3 * * *`) deletes succeeded jobs older than
    `JOB_RETENTION_DAYS` (default 7) and failed/cancelled older than `FAILED_JOB_RETENTION_DAYS`
    (default 30). `ArchiveJob` gains a `(userId, status, updatedAt)` index for fast cleanup.
- **Base pages show files only:** the inline queue panel is gone from `/workspace`; queue feedback
  lives exclusively in the modal. Base navigation opens the in-depth management page.
- **In-depth Base management (Rule K):** `/workspace/bases/[id]` — header with name/description/color
  + stats (file count, ready count), **Edit** (name/description/color via `updateBase`) and **Delete**
  (confirm-then-delete; detaches media, never deletes files), plus a files-only `MediaView` grid
  (status badges + secure downloads). Sidebar base links route here.
- **API rate limiting (Rule J, Phase 4 prep):** `lib/rate-limit.ts` — fixed-window per-key limiter
  (`API_RATE_LIMIT` default 60/`API_RATE_WINDOW_MS` default 60s) enforced in `requireApiKey`;
  returns 429 + `Retry-After`. In-process; swappable for a MongoDB counter if the web process scales.
- **Bugfix (baseIds conflict):** `createItem` and the worker's final upsert previously put `baseIds`
  in **both** `$setOnInsert` and `$addToSet`, which MongoDB rejects with "Updating the path 'baseIds'
  would create a conflict". `baseIds` now lives in `$addToSet` only (covers insert + update);
  `createItem` returns the created `mediaId`.

> **Acceptance:** every queued job is visible in the modal with live status and per-job controls;
> pausing refuses new archives; cancelled jobs stop processing; finished records auto-delete on a
> retention schedule; base pages show only archived files.

---

### Rule N — Admin User Management & Job Retry Cap (Phase 3.9)

**Problem.** Admins had no way to manage platform accounts (roles, disable, delete) from the UI,
and a transiently-failing job could keep retrying — burning extraction cost with no upper bound.

**Solution.**
- **`/admin/users` panel:** server page loads every account (footprint counts: media, bases,
  watchers, API keys) and renders `UsersManager` — a searchable table with role badges, status
  (Active/Disabled), joined date, and per-row actions (edit modal, enable/disable, two-step delete).
- **Edit modal:** well-styled Sheet with an identity card, a radio-style role picker
  (User / Admin / Super Admin with hints), and a "Disable account" switch; saves both in one action.
- **RBAC guards (`lib/actions/users.ts`):**
  - `listUsers` — any admin.
  - `updateUserRole` — admin may move users between `user`↔`admin`; **only superadmin** may grant
    or revoke the `superadmin` role; nobody may change their own role; the **last superadmin is
    protected** from demotion.
  - `setUserDisabled` — disabled users can't sign in (credentials `authorize` and the OAuth JWT
    callback both reject `disabled`); superadmins can't be disabled by admins; last superadmin protected.
  - `deleteUser` — cascade deletes the user's Media (with best-effort cloud-storage file cleanup
    via the StorageOrchestrator), Bases, Watchers, API keys, and ArchiveJobs, then the account.
- **Job retry cap:** `JOB_MAX_RETRIES` (default 5) drives both the Agenda `exponential` backoff and
  a worker-side guard so `INGEST_MEDIA` never rethrows past the cap — the ArchiveJob is marked
  failed permanently instead of retrying forever.
- **Bases retry sync:** `retryJob` now reads the Media document's current `baseIds` and carries them
  forward, so a re-extracted file lands back in the same Bases.

> **Acceptance:** admins can view/search every account and change roles/disable/delete with proper
> RBAC and self-protection; disabled accounts cannot authenticate; failed jobs stop retrying after
> `JOB_MAX_RETRIES`; retried jobs preserve their Base membership.

---

### Rule O — Bases CRUD Hub & Worker Observability (Phase 3.8)

**Problem.** Bases were only manageable from each base's detail page, and failing jobs were
impossible to diagnose from the worker console (no structured logging).

**Solution.**
- **Bases CRUD hub (`/workspace/bases`):** server page aggregates per-base file/ready counts
  (`Media.aggregate` unwind on `baseIds`) and renders `BasesManager` — a grid of base cards with
  inline **create sheet**, **edit sheet** (name/description/color), **confirm-then-delete**, and a
  deep link into each base's management page (`/workspace/bases/[id]`). The sidebar "Bases" header
  links to the hub; `createBase`/`updateBase`/`deleteBase` revalidate `/workspace/bases`.
- **Worker observability (`lib/log.ts`):** leveled structured logger (debug/info/warn/error,
  ISO timestamps, scoped prefixes, TTY colors, `WORKER_LOG_LEVEL`/`LOG_LEVEL`). `formatError`
  surfaces the **full stack + yt-dlp stderr/stdout/exit code** (the queue records only truncated
  error strings; the console gets the whole story).
- **Agenda lifecycle events** wired in `src/worker.ts`: `start`/`success`/`fail`/`retry`/
  `retry exhausted`/`error` — every job event logged with jobId + payload summary.
- **Step-by-step processor logging** in `lib/queue/jobs.ts`: INGEST_MEDIA logs claim, proxy
  selected (or direct), dedup hit, overwrite delete, extraction start/result, upload start/done,
  and failure with `formatError`; EXPAND_BATCH logs per-URL expansion + per-item enqueues;
  CRON_WATCHER logs per-watcher sync results; CLEANUP_JOBS logs retention counts.
- **Follow-up audit:** see `FOLLOWUP_AUDIT.md` — every follow-up suggested across the build is
  tracked with status (done / partial / recommended / out-of-scope) and a priority-ordered next-
  action list (top: unit tests, workspace media search, API scope enforcement).

> **Acceptance:** a user can fully manage Bases from one page; a failing job's root cause (incl.
> raw yt-dlp stderr) is visible in the worker console without code changes.

---

### Rule P — DB-First Platform Settings (Phase 4.6)

**Problem.** Every tunable (queue concurrency, retry caps, scrape toggles, rate limits, storage
fallback credentials, log level, cron schedules) lived in `.env` files. Changing a knob meant an
SSH edit + container restart, and operators had no UI to see what was actually in effect.

**Solution.** MongoDB is the **source of truth** for every runtime knob. Environment variables
become **bootstrap defaults** only. Resolution order, everywhere in the codebase:

```
DB value  →  env var  →  built-in default
```

**Data model — `Setting` collection:**

```ts
{
  key: string;        // stable registry key, e.g. "queue.maxRetries"
  value: Mixed;       // coerced to the registry type (number/string/boolean/csv)
  updatedBy: ObjectId; // ref User — admin who changed it
  updatedAt: Date;
}
```

**Registry (`lib/settings.ts`):** every env-based knob is declared ONCE in a typed registry
(`SettingDef`: key, env var, default, type, group, label, description, secret?, restartRequired?,
min/max). The resolver is DB-first with an in-process cache (`SETTINGS_CACHE_TTL_MS`, default
30s) so hot paths (rate limiter, queue config, scrapers) never hit MongoDB per call.

- **Hydration:** `connectToDatabase()` and the worker boot call `hydrateSettings()` (best-effort,
  never throws — if the DB is down, env/defaults apply). A periodic TTL refresh re-reads DB
  overrides so changes propagate to other processes without restart.
- **Live knobs via setters:** the log level is pushed into `lib/log.ts` via `setLogLevel()` after
  hydration (no import cycle). Restart-required settings (cron schedules, queue collection,
  process interval) are flagged in the UI.
- **`/admin/settings` UI (admin+):** every registered setting rendered grouped (Queue, Scraping,
  API, Inspection, Logging, Storage, Auth) with its effective value, source badge (DB / env /
  default), env fallback display, secret masking, edit sheet (typed input; booleans as switches;
  CSV as comma lists), and a reset action that deletes the DB override → falls back to env/default.
  Settings with `restartRequired` are labelled so operators know to redeploy.
- **Secrets:** `secret` settings (blob token, S3 keys) are masked in the UI and never serialized
  in full; leaving the edit field blank preserves the current value.
- **Migration status:** every module that previously read `process.env.X` at runtime now reads
  through the registry — queue (`jobs.ts`, `agenda.ts`), scrapers (yt-dlp player/UA/cookies,
  stealth domains/executable, page-harvest, STEALTH_FALLBACK), API (rate limit/window, key
  prefix), auth (`ADMIN_EMAILS`), inspection TTL, storage fallback credentials, log level,
  retention days, and cron schedules. Env vars remain as the bootstrap seed for first boot.

> **Acceptance:** an admin can change any platform knob from `/admin/settings` without touching
> files; changes apply after the cache TTL (instantly in the writing process); env vars still seed
> first boot; secrets are never exposed in full; the registry is the single place new knobs are added.

---

### Rule P.1 — White-Labeling (Branding, admin-configurable)

**Problem.** Operators deploy Xtreambase under their own brand (name, logo, tagline, accent color,
support email). Hardcoding "Xtreambase" strings across layouts makes white-labeling a code change
per customer.

**Solution.** Branding is **DB-first like every other knob** — a dedicated `Branding` group in the
Rule P registry (`lib/settings.ts`), read through `lib/branding.ts`:

| Key | Env fallback | Default | Purpose |
|---|---|---|---|
| `branding.name` | `PLATFORM_NAME` | `Xtreambase` | Product name in the sidebar, page titles (`generateMetadata`), login/setup/register headers |
| `branding.tagline` | `PLATFORM_TAGLINE` | "Archive share URLs — video, audio, and playlists…" | Subtitle on the workspace + auth screens |
| `branding.logoEmoji` | `PLATFORM_LOGO_EMOJI` | *(empty)* | Optional emoji mark (falls back to the default icon) |
| `branding.logoUrl` | `PLATFORM_LOGO_URL` | *(empty)* | Hosted logo image (overrides emoji/icon) |
| `branding.supportEmail` | `SUPPORT_EMAIL` | *(empty)* | Contact link in the auth footer |
| `branding.footerText` | `PLATFORM_FOOTER` | *(empty)* | Copyright/legal line in the auth footer |
| `branding.accentColor` | `PLATFORM_ACCENT` | *(empty)* | Brand hex color → `--primary`/`--ring`/foreground overrides on `:root` |

- **`BrandLogo` (client-safe):** `components/branding/brand-logo.tsx` renders image URL → emoji →
  default icon, painted with the accent when set. It receives a plain `BrandingView` object
  (`lib/branding-types.ts` — type-only, no server imports), so it is safe in the browser bundle.
- **Root layout:** `generateMetadata` awaits `hydrateSettings()` and uses `branding.name`/
  `branding.tagline`; the accent color is injected as CSS custom-property overrides on `:root`
  (`brandAccentCss`, with a luminance-computed readable foreground) so every `bg-primary` /
  `text-primary` surface follows the brand color. No DB → env/default branding, never throws.
- **Auth screens:** login, register, and first-boot setup render `BrandLogo` + name + tagline +
  optional footer (support email as `mailto:`).
- **Workspace/admin shell:** the sidebar brand row and the `/workspace` page header use the
  effective branding (layouts await hydration before passing it into the client shell).
- **`/admin/branding` UI (admin+):** dedicated page with a form (all seven fields, a color picker
  for the accent) and a **live preview** (auth header, sidebar row, footer). Save = `updateBranding`
  (writes each registered key, invalidates the cache, revalidates); Reset = `resetBranding`
  (deletes all overrides → env/default). The same settings also appear under the "Branding" group
  in `/admin/settings`. A Branding overview card links from `/admin`.
- The generic settings page re-uses the same registry, so branding keys need no special handling.

> **Acceptance:** an admin can fully re-brand the platform from `/admin/branding` without code or
> env changes; changes apply instantly (after cache TTL across processes); a missing DB falls back
> to the built-in Xtreambase identity; client components never import the server branding module.

---

## 4. End-to-End Archive Flow (Reference Sequence)

### 4.1 Manual single/batch archive
```
User pastes URL(s) in workspace (optionally with format settings)
   │
   ▼
Server Action: archiveMedia / archiveMediaBatch({ urls, format, overwrite })
   │
   ├─ Authenticate (NextAuth v5) ──► resolve User
   ├─ Normalize + validate URLs  (batch caps applied)
   ├─ Registry: resolve strategy per URL (Rule A)
   ├─ Playlist/channel? ──► expand via strategy (Rule G)
   ├─ Dedup check on (userId, sourceUrl)  ── duplicate? ──► skip or overwrite path
   │
   ▼
Enqueue ArchiveJob(s) with shared batchId (Rule F)
   │
   ▼
Worker (concurrency-limited): ProxyManager hands out live proxy (Rule B)
   ├─ Strategy.extract(url, { proxy, format, headers, cookies })  ──► ExtractResult
   │
   ▼
IStorageAdapter.uploadStream(...)    (Rule C)  ──► storageUrl
   │
   ▼
Persist/update Media document        (Rule D)  ──► emit completion event
   │
   ▼
UI Inspector pane + grid update (per-item status, batch progress)
```

### 4.2 Watcher auto-sync (scheduled)
```
SchedulerProvider fires (Vercel Cron / node-cron / Inngest) — or manual "Sync now"
   │
   ▼
For each enabled Watcher (Rule I): resolve strategy → expand/scan
   ├─ Diff against existing Media (dedup index) → only new items
   ▼
Enqueue ArchiveJob(s) with watcherId tag → same worker pipeline as 4.1
```

### 4.3 External push (Personal API)
```
POST /api/v1/archive  (Bearer xa_live_…) from iOS Shortcut / Zapier / curl
   │
   ├─ Rate limit + key scopes check (Rule J)
   ▼
Same batch pipeline as 4.1 → returns { jobIds, status: 'queued' }
```

---

## 5. Project Layout Blueprint (Anticipated)

```
xtreambase/
├─ app/                        # Next.js 15 App Router
│  ├─ (workspace)/             # authenticated workspace shell
│  │  ├─ archive/              # central grid/list + inspector
│  │  ├─ watchers/             # Watcher management UI (Rule I)
│  │  ├─ vault/                # Proxy Vault UI
│  │  ├─ settings/             # profile + API key management (Rule J)
│  │  ├─ admin/storage/        # storage config UI — quotas/fallbacks (Rule C)
│  │  └─ layout.tsx
│  ├─ api/
│  │  ├─ media/[id]/download/route.ts  # masked download resolution (Rule C.3)
│  │  └─ v1/
│  │     └─ archive/route.ts   # POST — personal API ingestion (Rule J)
│  ├─ actions/                 # Server Actions (archiveMedia[Batch], watcher CRUD, proxy CRUD, apiKey CRUD)
│  ├─ login/                   # NextAuth v5 pages
│  └─ layout.tsx / globals.css
├─ components/                 # shadcn/ui + workspace components
│  ├─ ui/                      # shadcn primitives
│  ├─ layout/                  # Sidebar, CommandBar, InspectorPane, BottomSheet
│  ├─ archive/                 # grid/list, batch input, format selector
│  └─ watchers/                # watcher list/form (Rule I)
├─ lib/
│  ├─ db/                      # Mongoose connection + models (User, Media, Proxy, ArchiveJob, Watcher, ApiKey, StorageConfig)
│  ├─ scrapers/                # IScraperStrategy, registry, YtDlpStrategy, StealthBrowserStrategy
│  ├─ storage/                 # IStorageAdapter, VercelStorageAdapter, S3StorageAdapter, StorageOrchestrator, factory
│  ├─ proxy/                   # ProxyManager (rotation, health)
│  ├─ auth/admin.ts            # requireAdmin guard
│  ├─ queue/                   # QueueProvider interface + adapters (in-process → Agenda/Inngest)
│  ├─ scheduler/               # SchedulerProvider interface + adapters (Vercel Cron / node-cron / Inngest)
│  └─ api/                     # API key hashing, verification, rate limiting (Rule J)
├─ types/                      # shared TypeScript types (incl. FormatConfig)
├─ docs/                       # in-app documentation reference (/docs)
│  └─ page.tsx                 # renders REQ.md / README.md with tabs, searchable TOC, raw download
├─ components/docs/            # DocsView (client) — react-markdown + remark-gfm, prose styling
├─ app/api/docs/[slug]/route.ts # raw markdown download (attachment)
├─ app/icon.svg                # branded Xtreambase SVG favicon (equalizer/stream mark)
├─ proxy.ts                    # NextAuth session guard — Next.js 16 renamed middleware → proxy (Edge-safe only; /api/v1 key auth lives in route handlers)
├─ .env.example
└─ package.json
```

---

## 6. Configuration & Environment (Planned)

```
MONGODB_URI=...
AUTH_SECRET=...
AUTH_URL=...
AUTH_GITHUB_ID=...          # OAuth providers per Phase 1 decision
AUTH_GITHUB_SECRET=...
STORAGE_DRIVER=vercel | s3        # swap with zero code changes (Rule C)
BLOB_READ_WRITE_TOKEN=...         # Vercel Blob
# S3 (or provided per-StorageConfig in the admin UI)
# AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, S3_BUCKET, S3_REGION
QUEUE_DRIVER=inprocess | agenda | inngest
SCHEDULER_DRIVER=vercel-cron | node-cron | inngest   # watchers (Rule I)
YTDLP_PATH=...                    # optional override for the yt-dlp binary (default: youtube-dl-exec's downloaded bin/yt-dlp)
YOUTUBE_DL_DIR=...                # optional: binary directory override (youtube-dl-exec)
YOUTUBE_DL_FILENAME=...           # optional: binary name override (youtube-dl-exec)
YTDLP_PLAYER_CLIENT=android,tv    # YouTube player-client impersonation (bot-check bypass); 'off' disables
YTDLP_USER_AGENT=...              # realistic browser UA for extraction (default: Chrome 126)
YTDLP_COOKIES=/path/cookies.txt   # optional Netscape-format cookies for logged-in/hard-blocked targets
PUPPETEER_EXECUTABLE_PATH=...     # optional, for StealthBrowserStrategy
ADMIN_EMAILS=you@example.com      # comma-separated bootstrap admin list
API_KEY_PREFIX=xa_live            # personal API key prefix (Rule J)
```

---

## 7. Execution Plan (Strict Phases — Do Not Jump Ahead)

### Phase 0 — Requirements & Architecture (in progress — awaiting owner approval)
- [x] Document product vision, architecture rules, schemas, UI strategy (this file).
- [ ] Gate: owner approves `REQ.md`.

### Phase 1 — Scaffolding
- Scaffold Next.js 15 (App Router, TypeScript strict).
- Configure MongoDB + Mongoose connection and models (**User**, **Media**, **Proxy**, **ArchiveJob**,
  **Watcher**, **ApiKey**).
- Wire NextAuth.js v5 (credentials/OAuth, session guard middleware).
- Set up Tailwind + shadcn/ui base (theme, primitives).
- `.env.example`, lint/format, CI-friendly scripts.
- **No scraping yet.** Gate: app boots, auth + DB verified.

### Phase 2 — Backend Core ✅ COMPLETE
- **Storage orchestration:** `IStorageAdapter` + `getDownloadUrl`, `VercelStorageAdapter`,
  `S3StorageAdapter`, `StorageConfig` model (quotas/fallbackNodeId), `StorageOrchestrator`
  with the fallback chain, and `/api/media/[id]/download` masked-download route (Rule C).
- **Admin foundation:** `User.role` (`user`/`admin`/`superadmin`), `requireAdmin`/`requireSuperAdmin`
  guards, first-boot `/setup` wizard, `/admin/storage` UI + server actions (add/edit nodes,
  quotas, fallbacks, cycle detection, single-default enforcement).
- `ProxyManager` (round-robin, health checks, persistence).
- Scraper Registry: `IScraperStrategy`, `YtDlpStrategy`, `StealthBrowserStrategy`, registry + URL routing.
- **Playlist/channel expansion** (`canExpand` / `expand`) on strategies (Rule G).
- **`FormatConfig`** pass-through in `ExtractOptions` (Rule H).
- Proxy injection into strategies (Rule B integration).
- Dedup/overwrite service layer (Rule D).
- **Agenda.js queue** (Rule F): `lib/queue/` (singleton, producers, job definitions with
  exponential backoff), `src/worker.ts` entry, esbuild bundle → `dist/worker.js`,
  dual-entry scripts (`dev`/`dev:all`/`start`/`start:all`/`worker`), Dokploy-ready.
- Gate: backend core built + verified (tsc/lint/build all green).

### Phase 3.6 — Developer API Gateway & Dynamic Metadata Engine ✅ COMPLETE
- [x] `ApiKey` schema upgraded: `autoSave`, `targetBaseIds`, `defaultFormat`.
- [x] `Media.metadata` → dynamic `{ extracted, custom: { externalIds, attributes, tags } }` + wildcard index (wildcardProjection).
- [x] API key guard (`lib/auth/api-key-guard.ts`), generate/revoke actions (show-once).
- [x] `POST /api/v1/extract` (save:false live-stream / save:true dedup+queue+merge), `GET /api/v1/media` (deep dynamic query), `GET /api/v1/bases`.
- [x] `/workspace/settings/api` Developer UI: key table, generate modal, show-once sheet, code snippets; sidebar nav.
- [x] Advanced Metadata builder in the submission form; customMeta threading (actions → EXPAND_BATCH → createItem → worker); re-link merge.
- [x] Inspector dynamic metadata editor (externalIds/attributes/tags add/edit/delete).
- [x] Queue Activity panel + `/api/workspace/jobs`; fixed the `baseIds` $setOnInsert/$addToSet conflict.
- Gate: tsc/lint/build green.

### Phase 3.8 — Bases CRUD Hub & Worker Observability ✅ COMPLETE
- [x] `/workspace/bases` index page with per-base file/ready counts (aggregate) + `BasesManager`
      full CRUD (create/edit/delete/confirm + deep links); sidebar Bases header → hub.
- [x] `lib/log.ts` structured logger + `formatError` (stack + stderr/stdout/exit).
- [x] Agenda lifecycle event logging in `src/worker.ts` (start/success/fail/retry/error).
- [x] Step-by-step processor logging (INGEST_MEDIA / EXPAND_BATCH / CRON_WATCHER / CLEANUP_JOBS).
- [x] `FOLLOWUP_AUDIT.md` — full audit of every suggested follow-up with status + next actions.
- Gate: tsc/lint/build green.

### Phase 3.5 — Selective Playlist Import & Direct Files ✅ COMPLETE
- [x] **`inspect()`** on `IScraperStrategy` + registry (`resolveAsync` with HEAD Content-Type probe).
- [x] **`DirectFileStrategy`** — extension + HEAD detection, direct streaming into the orchestrator,
      same global dedup + multi-Base linking. Registered first (priority 300).
- [x] **`POST /api/scraper/inspect`** — auth-guarded, lightweight per-item metadata + `archived` flag.
- [x] **`PlaylistSelectorModal`** — Select All/counter, search, thumbnails/duration, global + per-item
      formats, Base selector → `archiveSelectedItems` (direct `INGEST_MEDIA` enqueue).
- [x] **Watcher sync:** `baseIds` + `formatOption` on watchers; CRON_WATCHER threads both.
- [x] **Worker env parity** (`scripts/load-env.cjs` preload).
- Gate: tsc/lint/build green.

### Phase 3 — Workspace UI ✅ COMPLETE
- [x] App shell: Sidebar (collapsible on desktop, drawer on mobile), top bar, user menu.
- [x] Archive command bar: **bulk multi-line input**, **format selector (MP4/MP3/Original)**,
      **overwrite toggle**, **Base chip multi-select** (Rules G/H/K), wired to `archiveUrls`.
- [x] Central grid/list view with **status badges (Pending/Processing/Ready/Failed)**, live
      polling via `/api/workspace/media`, secure download via masked endpoint.
- [x] Slide-out **Inspector pane** (bottom sheet on mobile) with flexible metadata + bases.
- [x] **Bases architecture (Rule K)**: Base model, sidebar Bases section + create dialog,
      per-base filtering, global dedup link-only path, "Manage Bases" on media cards.
- [x] **Watchers UI** (`/workspace/watchers`): add/track, enable/disable, "Sync now" (Rule I).
- [x] **Proxy Vault UI** (`/admin/proxies`): bulk-add, health test, enable/disable (Rule B).
- [ ] **Settings**: personal API key generate/revoke/copy-once (Rule J) — Phase 4.
- Gate: full UX parity desktop ↔ mobile.

### Phase 4 — Final Integration, Real-Time UI & Dokploy Deployment ✅ COMPLETE
- [x] **Worker-side inspection:** `POST /api/scraper/inspect` now enqueues `INSPECT_URLS` and
      returns `{ jobId }`; `GET /api/scraper/inspect/[id]` polls the worker's `InspectJob`
      (TTL auto-expiry). Fixes the web-process `Cannot find module 'youtube-dl-exec'` failure
      by moving ALL yt-dlp work to the worker. `loadYoutubeDl()` cwd fallback for `save:false`.
- [x] **Real-time UI feedback:** queue modal + media grid poll (`/api/workspace/jobs`,
      `/api/workspace/media`); **toasts fire when polling detects a newly-failed job** (worker
      errors like "Private Video"/"Proxy dead" surface live, even with the modal closed).
- [x] **Production deployment:** multi-stage `Dockerfile` (ffmpeg + yt-dlp binary + Puppeteer
      Chromium baked in; non-root user; `CMD` overridable) + `.dockerignore`.
      `start:web` (`next start`) and `start:worker` (`node -r ./scripts/load-env.cjs dist/worker.js`).
      See **§ 10 Deployment Guide (Dokploy)**.
- [x] **API rate limiting:** per-key fixed-window limiter enforced on every `/api/v1/*` route
      via `requireApiKey` (default 60 req/min, 429 + `Retry-After`) + per-user limit on inspect.
- [x] **Masked downloads verified:** `/api/media/[id]/download` requires a session, scopes the
      Media doc to the user, and 307-redirects to a short-lived storage URL — raw URLs never
      reach the client.
- [x] Pipeline audit: URL paste → inspect (worker) → format select → queue → worker → proxy
      rotate → extract (fallback chain) → storage orchestrator (fallback chain) → MongoDB upsert
      (global dedup + dynamic metadata passthrough verified).
- Gate: paste-link-to-archive works end-to-end with async jobs; tsc/lint/build green.

---

## 10. Deployment Guide — Dokploy (Docker)

### Architecture

One **image** (`docker build -t xtreambase .`), **two Dokploy services** sharing only MongoDB:

> **Deps stage gotchas:**
> - `youtube-dl-exec`'s preinstall script requires a Python ≥3.9 binary (its
>   postinstall fetches a Python zipapp) and will fail `npm ci` in a slim Node
>   image — the Dockerfile sets `YOUTUBE_DL_SKIP_PYTHON_CHECK=1` and replaces the
>   zipapp with the true standalone `yt-dlp_linux` ELF, so no Python is needed.
> - Puppeteer's postinstall would download Chrome from Google's CDN, which is
>   frequently blocked on VPS/datacenter IPs (Dokploy) — the image installs
>   `chromium` from Debian apt instead (`PUPPETEER_SKIP_DOWNLOAD=true` +
>   `PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium`), in both the deps and runner stages.
> - `youtube-dl-exec`'s **postinstall** downloads the yt-dlp binary through the
>   GitHub **API** (`api.github.com/…/releases/latest`), which is rate-limited on
>   VPS/datacenter IPs and fails `npm ci` with "API rate limit exceeded". The
>   Dockerfile skips it (`YOUTUBE_DL_SKIP_DOWNLOAD=1`) and fetches the
>   **`yt-dlp_linux`** asset (true PyInstaller standalone ELF — the plain `yt-dlp`
>   asset is a Python zipapp) from the GitHub release CDN (`…/releases/latest/
>   download/yt-dlp_linux` — a plain asset download, NOT API-rate-limited) into
>   `node_modules/youtube-dl-exec/bin/yt-dlp`, verified with `yt-dlp --version`.
>   Use `yt-dlp_linux_aarch64` on ARM hosts.
> - If you build with your own base image, set the same env vars (or install
>   `python3`, a browser, and let the postinstall download run from an IP GitHub's
>   API accepts) or the build will fail the same way.

| Service | Command | Port | Purpose |
|---|---|---|---|
| **Web** | `npm run start:web` | 3000 | Next.js app: UI + Server Actions + `/api/*` routes (enqueues jobs only) |
| **Worker** | `npm run start:worker` | — | Standalone Agenda process: `INGEST_MEDIA`, `EXPAND_BATCH`, `CRON_WATCHER`, `INSPECT_URLS`, `CLEANUP_JOBS` |

### Steps

**Option A — Docker Compose (recommended):** a root [`docker-compose.yml`](./docker-compose.yml)
is the single deployment entry point. In Dokploy, create a **Docker Compose** service and
point it at this file:

- `web` container — `npm run start:web`, port 3000, with the **core Traefik labels** so the
  Dokploy network routes traffic: `traefik.enable=true`, `traefik.docker.network=dokploy-network`,
  `web` router (`Host(\`${DOMAIN}\`)`, `entrypoints=web`, `middlewares=redirect-to-https@file`),
  `websecure` router (`Host(\`${DOMAIN}\`)`, `entrypoints=websecure`, `tls.certresolver=letsencrypt`),
  `loadbalancer.server.port=3000` — plus a `/api/health` healthcheck (DB-free liveness probe).
- `worker` container — `npm run start:worker`, no ports, no labels (MongoDB-only egress).
- Both join the external `dokploy-network` (created by Dokploy's Traefik stack). `DOMAIN`
  (default `domain.xyz`) feeds the `Host()` rules; set it in Dokploy env to re-route.

```bash
docker compose up -d --build   # on the VPS, or via Dokploy's compose UI
```

**Option B — manual services from the same image:**

1. Build the image locally or in Dokploy: `docker build -t xtreambase .`.
2. Create the **Web** service from the image: port `3000`, env vars below.
3. Create the **Worker** service from the SAME image, overriding the command to
   `npm run start:worker` (Dokploy supports per-service commands), env vars below.
4. Add a MongoDB instance (Atlas or Dokploy-managed) and set `MONGODB_URI` on both services.
5. First boot: open the Web service → redirected to `/setup` → create the **superadmin**.

> **Traefik labels (Option B, if you manage labels manually instead of using the compose
> file):** the compose file carries the canonical, focused set — `traefik.enable=true`,
> `traefik.docker.network=dokploy-network`, the `web`/`websecure` routers above, and
> `traefik.http.services.<name>.loadbalancer.server.port=3000`. Attach the same labels to
> the Web service when deploying outside compose.

### Environment (both services)

| Variable | Required | Notes |
|---|---|---|
| `MONGODB_URI` | ✅ | Same connection string on web + worker (queue lives in `agendaJobs`) |
| `AUTH_SECRET` | ✅ | `openssl rand -base64 32` — must match across restarts |
| `AUTH_URL` | ✅ | Public URL of the web service (e.g. `https://base.example.com`) |
| `AUTH_TRUST_HOST` | ✅ | `true` when behind a proxy / non-Vercel host |
| `STORAGE_DRIVER` / `BLOB_READ_WRITE_TOKEN` | storage | Vercel Blob, or configure nodes in `/admin/storage` |
| `WORKER_CONCURRENCY` | optional | Parallel INGEST_MEDIA (default 3) |
| `JOB_MAX_RETRIES` | optional | Retry cap (default 5; jobs never retry past this) |
| `WATCHER_CRON` | optional | Watcher auto-sync schedule (default `*/30 * * * *`; `off` disables) |
| `STEALTH_FALLBACK` / `STEALTH_DOMAINS` | optional | Universal stealth fallback toggle / forced-stealth domains |
| `YTDLP_PLAYER_CLIENT` / `YTDLP_USER_AGENT` / `YTDLP_COOKIES` | optional | Anti-bot impersonation knobs |
| `API_RATE_LIMIT` / `API_RATE_WINDOW_MS` | optional | Per-key API rate limit (default 60/min) |
| `INSPECT_JOB_TTL_MINUTES` | optional | InspectJob auto-cleanup (default 30) |

### Image contents (no manual setup on the VPS)

- `ffmpeg` — required by yt-dlp for MP3/MP4 format conversion.
- yt-dlp binary — `node_modules/youtube-dl-exec/bin/yt-dlp`, the **`yt-dlp_linux`** standalone ELF fetched during the deps stage directly from the GitHub release CDN (postinstall skipped via `YOUTUBE_DL_SKIP_DOWNLOAD=1` — the GitHub API used by the postinstall is rate-limited on VPS IPs, and its zipapp artifact needs Python).
- Puppeteer Chromium — installed from Debian apt (`chromium`, wired via `PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium`; puppeteer download skipped with `PUPPETEER_SKIP_DOWNLOAD=true`), for StealthBrowserStrategy.
- Runs as the non-root `nextjs` user; the worker writes only to MongoDB (no filesystem privileges needed).

### Scaling notes

- To scale the worker, run additional worker services with the same image — Agenda's MongoDB
  locking prevents double-execution of the same job.
- The web service is stateless; the in-process rate limiter resets on restart (swap to a
  MongoDB counter if you run many web replicas).

---

## 8. Non-Functional Requirements

- **Security:** all archive/storage/proxy mutations behind NextAuth sessions; **/admin/*** and storage
  config writes gated by the `admin` role; API keys stored as **sha256 hashes only**; never expose
  proxy credentials, full keys, or **raw storage URLs** in client payloads (all downloads go through
  the masked internal route with short-lived URLs); sanitize metadata before display; HMAC-signed
  payloads for any future outbound webhooks.
- **Privacy:** media and metadata are per-user scoped on every query (`userId` filter mandatory).
- **Rate limiting & caps:** max URLs per batch request, max playlist expansion size, API-key rate
  limits, and watcher-run throttles (shared config, provider-aware).
- **Observability:** structured logs for jobs (strategy, proxy used, duration, result, batchId, watcherId).
- **Error taxonomy:** typed errors (`DuplicateArchiveError`, `StrategyNotSupportedError`,
  `ProxyUnavailableError`, `ExtractionFailedError`, `StorageError`, `RateLimitError`, `InvalidApiKeyError`)
  surfaced in UI and API responses.
- **Performance:** UI never blocks on long jobs (Rule F); optimistic updates for archive status;
  batch progress is aggregate + per-item.

---

## 9. Risks & Open Questions

| Risk / Question | Mitigation / Decision needed |
|---|---|
| Anti-bot measures evolve per site | Strategy registry isolates changes to one class; stealth strategy is upgradeable |
| Vercel serverless limits | Rule F (jobs) is mandatory before heavy scraping is exposed |
| Playlist/channel size explosion | Expansion caps + per-item concurrency limits (Rule G) |
| Watcher scheduler choice (Vercel Cron vs node-cron vs Inngest) | `SchedulerProvider` interface (Rule I); decide in Phase 4 |
| API key leakage | Hash-only storage, prefix display, instant revocation, rate limits (Rule J) |
| Provider rate limits during watcher runs | Shared throttle config + proxy rotation |
| `metadata: Mixed` schema flexibility vs. queryability | Dynamic Metadata Engine (Rule L): `metadata.extracted` + `metadata.custom` maps with a single wildcard index (MongoDB allows one per collection) |
| NextAuth v5 provider choice (credentials vs. OAuth) | Decide in Phase 1; prefer OAuth-first with optional credentials |
| Proxy legality/terms | Owner to confirm acceptable sources for proxy lists |
| Queue provider (Agenda vs. Inngest) | Decide in Phase 4; interface keeps it swappable |
| Fallback chain cycles (A→B→A) | Orchestrator tracks visited nodes and refuses loops |
| Quota drift (usedBytes vs real usage) | Reconcile via storage provider listing; unit tests on accounting |
| Admin bootstrap (first admin) | `/setup` first-boot wizard + `ADMIN_EMAILS` env fallback; guarded so it cannot run twice |

---

## Appendix A — Original Request (Verbatim)

> so i want to build a platform my goal is for user to build up their archive base of stream data with it they paste it share url and have data scrapped and uploaded to there media db for the media upload lets use vercel for now i will pivot to an s3 storage later so the adaoter for media should work as such for now just scalfold the project layout use great tech stack for the dev i like next js so for now scalfolde the auth flow and user workspace for building up file data refer to this site https://ytdlp.online/playlist-downloader this sit should be a fraction of my platform features as similar platform or features required for videos musics can be itegrated my goal is i paste in a link i get the data i need and have uploaded to the user and for a current base that is being created uniquenes should also be implemented unless a user wannt to overwite. for now write out my request to a req.d file do just that