xtreambase — Requirements & Architecture Specification (REQ.md)

This document is the single source of truth for what we build and how we architect it. It must be updated whenever a phase introduces a new architectural decision.

1. Product Vision

xtreambase is a highly advanced, future-proof platform that lets users build a personal archive base of stream data — video, audio, and playlists.

The core workflow is a focused, immersive workspace ("Xtreambase"):

1. The user pastes a share URL (or many URLs / a playlist / a channel) into the workspace.
2. The ingestion engine intercepts, scrapes, and bypasses bot-protections to extract the media and its metadata.
3. The media is uploaded to cloud storage (Vercel Blob for now; S3-compatible storage later).
4. The result is indexed into the user's personal MongoDB database, with deduplication and optional overwrite.
5. Advanced modes layer on top: batch ingestion, format/quality controls, auto-sync watchers, and programmatic access via a personal API.

The platform is a resilient, proxy-backed ingestion engine, not a one-off downloader. It must remain operable when target sites add anti-bot measures, throttle IPs, or change their HTML.

Reference for feature scope (a fraction of our platform): https://ytdlp.online/playlist-downloader.

1.1 Core value proposition

• One workspace for archiving links across many providers (video, music, playlists).
• Private, personal archive per user, backed by their own MongoDB database.
• Resilience by design: strategy registry, proxy rotation, storage abstraction, async jobs.
• Automation-ready: batch inputs, output controls, watchers, and an API for external tools.

1.2 Goals

• G1 — Paste a link → get structured media + metadata → uploaded → indexed → visible in the workspace.
• G2 — Survive anti-bot and IP-throttling via a built-in proxy vault and stealth scraping.
• G3 — Guarantee uniqueness of archived media per user unless an explicit overwrite is requested.
• G4 — Zero business-logic changes when pivoting storage (Vercel → S3) or adding scrapers/proxy providers.
• G5 — A productivity-grade UI (Notion/IDE vibe), fully mobile-responsive with bottom-sheet interactions.
• G6 — Batch: paste many URLs at once; playlists/channels auto-expand into individual items; all items processed concurrently through the job queue.
• G7 — Format control: per-request output selection (audio-only, video quality caps, best available) chosen in the UI and honored by every scraper strategy.
• G8 — Watchers: track a channel/playlist URL for automatic, scheduled ingestion of new media without manual intervention (architecture ready for cron schedulers).
• G9 — API-first: users can generate a personal API key and push links into their queue from anywhere on the web (iOS Shortcuts, Zapier, scripts).

1.3 Non-goals (for now)

• NG1 — Social features / public sharing of archives.
• NG2 — On-platform media transcoding (raw extraction only at MVP; format selection is extraction-level).
• NG3 — Support for paywalled/DRM content.
• NG4 — A fully distributed worker fleet (we anticipate it via Rule F, but do not build it in Phase 1–3).
• NG5 — Outbound webhook delivery to third parties (design space only; inbound API is in scope).

2. Tech Stack Mandate (Non-Negotiable)

3. Core Architectural Rules (Non-Negotiable)

Rule A — The Scraper Strategy Registry (Future-Proof Extensibility)

Problem. A single scraper cannot handle every target. Providers differ in anti-bot aggressiveness, site structure, and media formats.

Solution. Implement a Scraper Registry Pattern:

• A common interface IScraperStrategy:
  • canHandle(url: URL): boolean — capability routing (regex/domain/format detection).
  • extract(url: URL, options: ExtractOptions): Promise<ExtractResult> — perform the extraction.
• A registry that holds all registered strategies, ordered by priority, and routes each URL to the first strategy whose canHandle() returns true.

Baseline strategies:

Inspection capability (Selective Playlist Import): every strategy can optionally implement inspect(url, options): Promise<InspectItem[]> returning lightweight per-item metadata { id?, title?, thumbnail?, duration?, url, index } without downloading media bytes. The registry's inspect() uses an async resolver that also runs canHandleAsync probes (e.g. HEAD Content-Type for extensionless direct files) before falling back to a one-item list.

Worker-side inspection (Phase 4): inspection is heavy scraping and runs in the Agenda worker, never the web process — the Next.js web bundle rewrites require("youtube-dl-exec") to a virtual path (/ROOT/…) and dies with MODULE_NOT_FOUND, while the plain-Node worker resolves it fine. POST /api/scraper/inspect only persists an InspectJob doc (TTL auto-expiry, INSPECT_JOB_TTL_MINUTES default 30) and enqueues INSPECT_URLS; the worker performs the inspection + archived dedup annotation and stores per-URL results; the UI polls GET /api/scraper/inspect/[id] until succeeded and then opens the PlaylistSelectorModal. loadYoutubeDl() gained a cwd-based require fallback so web-side live streaming (POST /api/v1/extract save:false) also resolves the module.

Fallback chain (broadened scraping, Phase 3.10): ScraperRegistry.extractWithFallback() replaces the single-strategy extract() in the worker and /api/v1/extract. It tries every capability-matched strategy in priority order (direct-file → yt-dlp → stealth-browser), recording each failure, then — for page-like URLs only (never obvious raw-file URLs) and unless STEALTH_FALLBACK=off — any strategy declaring isUniversalFallback (StealthBrowserStrategy: a real browser can render almost any page, so a bot-checked yt-dlp URL gets a second chance in Puppeteer). If ALL strategies fail, one ExtractionFailedError aggregating every attempt is thrown (still classified non-retryable when the underlying cause is a bot-check/DMCA). The winning strategy is tagged as metadata.strategy and recorded on the ArchiveJob for observability. StealthBrowserStrategy now performs server-side byte resolution of the in-page media URL (via fetch-utils.ts, through the Proxy Vault when injected), rejecting blob:/data: URLs with a clear error — making it a fully-functional fallback rather than a metadata-only placeholder. Shared fetchStream()/fetchViaProxy() helpers centralize proxy + redirect + timeout semantics across strategies.

Extensibility contract: adding a scraper in the future = add a class to /scrapers + register it (one line). No changes to the routing core, UI, storage, or job pipeline.

Acceptance: the registry routes a URL to the correct strategy by capability; unknown URLs fail gracefully with a typed error; the registry is strategy-count-agnostic; inspect() returns metadata for playlists/channels and never fetches media bytes; a failed strategy falls through to the next capable one (and to the stealth browser for page URLs) instead of failing the job.

Rule B — Internal Proxy Management & Rotation (The Proxy Vault)

Problem. Aggressive scraping gets IPs banned. External proxy APIs add cost and coupling.

Solution. The system has its own built-in proxy management to evade IP bans.

Data model — Proxy collection:

Manager logic — ProxyManager service:

• Rotation: round-robin selection (lastUsed ordering), skipping unhealthy/disabled proxies.
• Health checking: periodic and on-demand checks (small HEAD request through the proxy, or via a strategy test); transient failures increment failureCount; persistent failures flip healthStatus to unhealthy (with optional auto-cooldown).
• Per-request integration: the ProxyManager injects a live proxy into the chosen strategy — --proxy <proxy> into yt-dlp, and the proxy into the Puppeteer browserContext/launch args.

Extensibility: selection policy is swappable (round-robin default; weighted/sticky later) without touching scraper code.

Acceptance: a dead proxy is never used twice in a row when healthy ones exist; health state is persisted; strategies receive a proxy object per request, never hardcode one.

Rule C — The Storage Adapter Pattern + Orchestration (Quotas, Fallback Chain, Masked Downloads)

Problem. App code must never hardcode cloud storage SDKs, must survive provider capacity/outage limits, and must never leak raw storage URLs to the client.

Solution. A three-layer stack:

interface IStorageAdapter {
  upload(input: { stream: Readable | Buffer; filename: string; contentType?: string; metadata?: object }): Promise<{ storageUrl: string; storageKey?: string; sizeBytes?: number }>;
  delete(storageUrl: string): Promise<void>;
  /** Secure internal resolution — never expose the raw URL to the client */
  getDownloadUrl(fileId: string, opts?: { expiresInSeconds?: number }): Promise<string>;
}

C.1 — StorageConfig (quotas & fallbacks):

{
  name: string;
  driver: 'vercel' | 's3';
  enabled: boolean;
  isDefault: boolean;
  capacityBytes: number;      // max quota; 0 = unlimited
  usedBytes: number;          // running usage, updated on upload/delete
  fallbackNodeId?: ObjectId;  // ref StorageConfig — next node in the fallback chain
  config: Mixed;              // driver-specific (token / bucket / region / keys)
}

C.2 — StorageOrchestrator (Fallback Chain): before an upload it walks nodes in priority order (default node first). It skips a node when usedBytes >= capacityBytes (quota full), and catches provider errors (e.g. 5xx) and follows fallbackNodeId to retry on the next node. Only when the entire chain is exhausted does it raise StorageError. usedBytes is credited on upload and debited on delete.

C.3 — Masked Internal Downloads: raw storage URLs never reach the browser. UI calls /api/media/[id]/download, which verifies the session, loads the user-scoped Media doc, resolves its StorageConfig, and returns a short-lived URL (5-min presigned for S3; stored URL for Vercel Blob) via a 307 redirect. Media stores storageNodeId + storageKey so the orchestrator can resolve the file without exposing provider URLs.

• VercelStorageAdapter — Vercel Blob (@vercel/blob).
• S3StorageAdapter — @aws-sdk/client-s3 + @aws-sdk/s3-request-presigner.
• Business logic (scraper pipeline, dedup, overwrite) depends only on the interface.

Acceptance: switching STORAGE_DRIVER=vercel → s3 requires zero business-logic changes; a full/errored node transparently fails over to its fallback; no raw storage URL is ever serialized to the client.

Rule K — Multi-Base Management & Global Storage Deduplication

Problem. Users archive into one undifferentiated list; the same source URL pasted into a second collection would waste storage on a duplicate file.

Solution. A Base collection gives users named, colored organization ("Music", "Courses", …) while storage deduplication is global per user.

Data model — Base collection:

{
  userId: ObjectId;         // ref User
  name: string;             // unique per (userId, name)
  description?: string;
  color: 'slate'|'red'|'orange'|'amber'|'green'|'teal'|'blue'|'indigo'|'violet'|'pink';
  createdAt: Date;
}

Media schema upgrades:

• baseIds: [ObjectId] — a single stored file can belong to one or more Bases (shared ref).
• formatOption: 'mp4' | 'mp3' | 'original' — the conversion/container intent chosen in the UI.
• canonicalUrl: string — normalized URL (tracking params + fragment stripped, host lowercased).
• Unique index (userId, canonicalUrl) replaces (userId, sourceUrl) as the dedup key — one stored file per user per source across ALL bases.

Fallback base ("Others", auto-managed): every user gets exactly one auto-created base (Base.isFallback: true, name "Others", protected from deletion via a partial unique index on (userId, isFallback=true)). Unassigned media — placeholder kind:"other" items, failed/ unscraped cards, items whose base was deleted — is linked into it automatically so nothing falls into an unmanageable dead-end "others" bucket:

• lib/bases-fallback.ts — ensureFallbackBase(userId) (find-or-create, race-safe, also backfills media with zero baseIds into the fallback) + getFallbackBaseId(userId) (cached).
• Enqueue path (createItem, archive actions) auto-targets the fallback when the user selected no base; retryMedia/deleteMedia preserve fallback membership on retry.
• The fallback base is fully CRUD-able like any base (sidebar badge "auto", deep page, per-item retry/delete via the new lib/actions/media.ts actions).

Global dedup flow (single upload, multi-base link):

1. User pastes URLs and picks target Bases (default: the active Base).
2. archiveUrls canonicalizes each URL and checks Media (userId, canonicalUrl).
3. Exists? → no yt-dlp/Puppeteer, no storage upload — $addToSet: { baseIds } on the existing document, instant toast "Linked existing media to selected Base(s) without extra storage usage!".
4. New? → queue EXPAND_BATCH (→ INGEST_MEDIA) with baseIds + formatOption; the worker extracts, converts, uploads once, and writes baseIds on the Media document.
5. The worker's INGEST_MEDIA/EXPAND_BATCH re-check the canonical dedup as a race guard and take the same link-only path when the file appeared meanwhile.

Format conversion pipeline (formatOption):

• mp4 (default) — best video+audio merged into an MP4 container (mergeOutputFormat: mp4).
• mp3 — audio-only extraction + conversion (-x --audio-format mp3).
• original — raw best stream, unprocessed.
• YtDlpStrategy.formatToYtdlp maps formatOption → yt-dlp flags; StealthBrowserStrategy carries the intent in metadata for the Phase 4 byte-resolution pass.

UI: sidebar "Bases" section (+ create dialog, auto badge on the fallback base), base chips on the submission form, per-base filtering (/workspace?base=<id> + /api/workspace/media?base=), and a "Manage Bases" action on every media card (attach/detach via setMediaBases). Failed cards additionally get Retry (retryMedia → re-enqueue through the worker) and Delete (deleteMedia → best-effort cloud file cleanup + record + related queued jobs), so the fallback base's failed/unscraped items are never a dead-end.

Acceptance: pasting the same URL into two bases stores the file once; deleting a base detaches (never deletes) media; MP3/MP4/Original selections reach the scraper unchanged.

Rule D — MongoDB Deduplication & Overwrite Logic

Data model — core collections:

// User — one archive per authenticated user
{
  email: string;            // unique
  name?: string;
  image?: string;
  role: 'user' | 'admin' | 'superadmin';  // RBAC: superadmin > admin > user
  passwordHash?: string;
  createdAt: Date;
}

RBAC & first-boot setup: roles gate /admin/*, storage config writes, and the proxy vault (requireAdmin / requireSuperAdmin guards). On first boot (no active superadmin exists) the app redirects to /setup, which creates the root admin account; the page and its server action refuse to run once a superadmin exists. ADMIN_EMAILS remains an optional bootstrap to promote matching accounts. The User schema enum is ['user', 'admin', 'superadmin'] — it must always include superadmin or the setup wizard's User.create({ role: 'superadmin' }) fails Mongoose validation.

Admin shell (Phase 3.5): /admin/* is wrapped in app/admin/layout.tsx which reuses WorkspaceShell — admins keep the sidebar (incl. the Administration section), user menu, and mobile drawer on every admin page. /admin is an overview landing (storage nodes, proxy health, media archived, users) with quick links to the Storage Vault and Proxy Vault.

// Media — one document per archived source { userId: ObjectId; // ref User sourceUrl: string; // original pasted URL title: string; kind: 'video' | 'audio' | 'playlist' | 'other'; storageUrl?: string; // where the media lives thumbnailUrl?: string; metadata: Mixed; // flexible, site-specific data (Mongoose Mixed) formatRequested?: FormatConfig; // what the user asked for (Rule H) status: 'pending' | 'processing' | 'ready' | 'failed'; sizeBytes?: number; durationSeconds?: number; createdAt: Date; updatedAt: Date; }

**Uniqueness:** unique compound index on `(userId, sourceUrl)` — one archive per source per user.

**Overwrite UX:**
1. Default: an incoming archive request for an existing `(userId, sourceUrl)` is **rejected**
   with a `DUPLICATE` result. The user sees "Already archived" in the UI.
2. If the request includes `overwrite: true`:
   - `IStorageAdapter.delete()` the **old** `storageUrl` (if any),
   - run the **new** extraction,
   - `findOneAndUpdate` the existing `Media` document (atomic, upsert-safe).

> **Acceptance:** concurrent duplicate requests do not create duplicate rows (index enforces it);
> overwrite keeps a single document and never leaves orphaned storage files.

---

### Rule E — The Workspace UI & Mobile Responsiveness

The UI is a **focused productivity tool** ("Xtreambase"), Notion/IDE vibe:

- **Sidebar navigation** — Workspace sections (Archive, Watchers, Proxy Vault, Settings, Account);
  **collapsible on desktop, drawer on mobile**.
- **Central archive view** — grid/list toggle for the user's media, with an archive
  **command bar** (paste URL → archive, **bulk-paste multi-line**, format selector).
  Inline status for `pending / processing / ready / failed`.
- **Slide-out "Inspector" pane** — metadata details for a selected media item
  (source URL, storage URL, site-specific metadata, size, duration, timestamps, overwrite action).
- **Watchers UI** — create/track channel or playlist URLs, toggle auto-sync on/off, view last-synced
  and newly-ingested counts (Rule I).
- **Proxy Vault UI** — admins/users can: bulk-paste proxy lists, **test proxy health**,
  **enable/disable** proxies, view health status and last-used, see rotation stats.
- **Settings** — Personal API key management: generate/revoke keys, copy-once reveal, scopes (Rule J).
- **Mobile experience is critical:**
  - Action sheets become **bottom sheets** on small viewports.
  - Sidebar collapses to a drawer; Inspector pane becomes a full-screen slide-over.
  - Touch targets ≥ 44px; primary actions thumb-reachable.

> **Acceptance:** full UX parity on 360px → 4K viewports; keyboard-first on desktop; bottom
> sheets on mobile.

---

### Rule F — Async / Job Queues (Agenda.js, MongoDB-Native)

**Problem.** Scraping and downloading large media takes time and **will break Vercel
serverless limits** (10s body / 60s max duration for Hobby/Pro defaults).

**Solution (DECIDED).** **Agenda.js v6** backed by our **existing MongoDB** — no extra Redis
container on Dokploy. Server Actions/API routes **never** run long downloads inline; they only
enqueue work into the native `agendaJobs` collection.

- **Queue setup** (`lib/queue/agenda.ts`): a singleton `Agenda` instance reuses the **same
  Mongoose connection** (`mongo: <Db>`), storing every record in the native `agendaJobs`
  collection (`AGENDA_COLLECTION`, default `agendaJobs`).
- **Dual-entry point (Dokploy):**
  - **Web container** — `npm run start` / `next start` (also `next dev` for local web).
  - **Worker container** — `npm run worker` / `node dist/worker.js` (bundled by esbuild from
    `src/worker.ts`; `tsx watch src/worker.ts` for local).
  - **Unified spinup:** `npm run dev:all` and `npm run start:all` run both processes under
    `concurrently` with color-coded streams (`web` cyan / `worker` magenta).
- **Worker env parity:** the web app auto-loads `.env.local`/`.env` (Next.js), but standalone
  `tsx`/`node` workers do not. `scripts/load-env.cjs` (preloaded via `-r`) loads `.env.local`
  then `.env` with Next.js precedence (real env vars win), so `dev:worker`/`start:worker`/`worker`
  see exactly the env the web app sees.
- **Job definitions** (`lib/queue/jobs.ts`) — registered via `registerJobs(agenda)`:
  - **`INGEST_MEDIA`** `{ jobId, url, userId, format, overwrite, batchId?, watcherId? }` —
    rotates a proxy via the Proxy Vault, executes the Scraper Strategy (yt-dlp or stealth),
    streams the upload through the StorageOrchestrator **fallback chain**, and upserts the
    `Media` document with Rule D dedup/overwrite semantics.
  - **`EXPAND_BATCH`** `{ urls, userId, format, overwrite, batchId, jobId }` — fans playlists/
    channels out (Rule G), dedups each item, and enqueues one `INGEST_MEDIA` per item.
  - **`CRON_WATCHER`** — scheduled via `agenda.every()` (default `*/30 * * * *`,
    `WATCHER_CRON=off` disables); diffs each enabled `Watcher` source against existing `Media`
    and enqueues `INGEST_MEDIA` for new items (Rule I).
  - **`INSPECT_URLS`** `{ inspectJobId, userId, urls }` (Phase 4) — worker-side playlist/batch
    inspection: runs `scraperRegistry.inspect()` per URL (yt-dlp needs the worker runtime),
    annotates `archived` dedup flags with one batched Media query, and stores per-URL results
    on the `InspectJob` doc for the UI to poll. Web process only *produces* this job.
- **Retry policy:** every job definition uses Agenda's native **exponential backoff**
  (`exponential({ delay: 30s, factor: 2, maxRetries: JOB_MAX_RETRIES, jitter: 0.2 })`) so a failed
  proxy or a disconnected stream during a yt-dlp/Puppeteer run auto-retries 30s → 1m → 2m → 4m → 8m.
  **Hard retry cap:** `JOB_MAX_RETRIES` (default 5, max 20) feeds BOTH the Agenda backoff AND a
  worker-side guard — `INGEST_MEDIA` stops rethrowing once `failCount >= JOB_MAX_RETRIES` and marks
  the ArchiveJob failed permanently. A job can therefore **never retry infinitely**, regardless of
  queue config. Each retry re-runs the full extraction, so the cap doubles as a cost guard.
- **Fail-fast for deterministic failures:** `isNonRetryableError()` (lib/errors.ts) classifies
  bot-checks ("Sign in to confirm you're not a bot"), unavailable/private/DMCA videos, and
  geo-blocks as permanent — `INGEST_MEDIA` marks the job failed immediately instead of burning
  the 5 retries (which can never succeed for a permanent block).
- An `ArchiveJob` document tracks: `userId`, `sourceUrl`, `strategy`, `status`, `attempts`,
  `resultRef`, timestamps — plus **`batchId`** (Rule G) and **`watcherId`** (Rule I) tags.
- The **immediate request path** returns a `batchId` + `status: queued` synchronously while the
  heavy work happens off-path; the UI polls `/api/workspace/media` for status.

> **Acceptance:** no web request performs a full download synchronously; a dead proxy or dropped
> stream retries with exponential backoff up to `JOB_MAX_RETRIES` and then fails permanently (never
> infinite); the worker is a standalone process (Docker container) sharing only MongoDB.

---

### Rule G — Batch Operations & Playlist Expansion

**Problem.** Users want to archive many items at once: pasted link lists, playlists, or whole channels.

**Solution.**

- **Bulk input:** the command bar accepts **multiple URLs separated by newlines**
  (`archiveUrls({ urls, format, overwrite, baseIds })`).
- **Playlist/Channel detection:** `IScraperStrategy` gains an optional capability
  `canExpand(url): boolean` and `expand(url, options): Promise<ExpandedItem[]>` (or flat item URLs).
  - `YtDlpStrategy` expands playlists/channels via `--flat-playlist --dump-json`.
  - `StealthBrowserStrategy` expands by scraping channel/playlist pages.
  - The **registry** detects playlist/channel URLs at route time and routes them to the
    expanding strategy; unknown expansion requests fail with a typed error.
- **Selective Playlist / Batch Ingestion (Phase 3.5, async in Phase 4):** pasting a playlist
  URL or a multi-line batch triggers `POST /api/scraper/inspect`, which now **enqueues an
  `INSPECT_URLS` worker job and returns `{ jobId }` immediately** (the web process cannot run
  yt-dlp — see Rule A). The UI polls `GET /api/scraper/inspect/[id]` and opens the
  **PlaylistSelectorModal** with lightweight per-item metadata (title, thumbnail, duration,
  `archived` dedup flag). The modal lets the user pick items before anything is queued:
  - Select All / Deselect All + live counter ("12 of 45 selected"), search filter.
  - **Global and per-item format dropdowns** (MP4 / MP3 / Original) — per-item overrides win.
  - Base chips to assign the chosen items (defaults to the active Base).
  - "Enqueue Selected Items" → `archiveSelectedItems({ items: [{ url, format }], baseIds })`
    enqueues **only the checked items** straight into `INGEST_MEDIA` (no re-expansion); global
    dedup still links ready files instead of re-downloading.
- **Fan-out pipeline:** for each input URL — resolve strategy → if `canExpand`, expand → apply
  dedup per item (Rule D) → enqueue **one `ArchiveJob` per item** tagged with a shared `batchId`.
  **Duplicate items inside a batch are reported, not dropped:** each duplicate gets a per-item
  `duplicate` result in batch progress (the user sees "Already archived" on that row) and is
  skipped for processing; only items with `overwrite: true` re-run the overwrite path.
- **Concurrency:** the job queue processes items of the same batch with a **configurable
  concurrency limit** (provider/plan-aware), and the UI shows per-item + batch progress.

> **Acceptance:** pasting 1 URL or 50 newline-separated URLs behaves consistently; a playlist URL
> fans out to individual `Media` docs; per-item failures don't abort the batch; batch caps prevent
> abuse; users can surgically pick items + per-item formats before queuing.

---

### Rule H — Format & Quality Controls (Extraction-level Transcoding Settings)

**Problem.** Users must be able to say *what* they want before scraping (audio only, capped quality, best).

**Solution.**

- A typed **`FormatConfig`** flows UI → Server Action → `ExtractOptions` → strategy:

```ts
type FormatConfig = {
  output: 'best' | 'video' | 'audio';
  videoQuality?: '2160p' | '1440p' | '1080p' | '720p' | '480p' | 'best';
  audioFormat?: 'm4a' | 'mp3' | 'opus';
};

• UI: a format selector in the archive command bar ("Extract Audio Only", "Video: 1080p max", "Best Available") and per-watcher defaults (Rule I).
• Strategy mapping: YtDlpStrategy translates to yt-dlp -f format expressions and -x for audio-only; StealthBrowserStrategy resolves quality in-page and picks the matching stream.
• Persistence: formatRequested is stored on Media (and ArchiveJob) for audit and future re-download in a different format (note: actual transcoding is a later-phase, out of scope for MVP).

Acceptance: every strategy honors the same FormatConfig shape; UI selections reach strategies unchanged; invalid combos are rejected up-front by shared validation.

Rule I — Watchers (Auto-Sync Subscriptions)

Problem. Users want to track a creator's channel or a playlist and auto-archive new media.

Solution.

Data model — Watcher collection:

{
  userId: ObjectId;          // ref User
  sourceUrl: string;         // channel / playlist / single URL
  kind: 'channel' | 'playlist' | 'single';
  title?: string;
  enabled: boolean;
  format?: FormatConfig;     // output preference for auto-ingest (Rule H) — incl. formatOption
  baseIds: [ObjectId];       // Bases auto-ingested media is linked to (Bases sync)
  lastCheckedAt?: Date;
  lastSyncedAt?: Date;
  lastSyncCount?: number;    // items ingested in the last run
  createdAt: Date;
  updatedAt: Date;
}
// Unique compound index (userId, sourceUrl)

• Sync flow: for each enabled watcher → resolve strategy via registry → expand/scan source → diff against existing Media (Rule D dedup handles this) → enqueue ArchiveJobs for new items. Nothing new to ingest → no-op run, lastSyncedAt updated.
• Bases + format sync (Phase 3.5): watchers carry baseIds (auto-link new media to chosen Bases) and a format with formatOption (MP4/MP3/Original) — the same format pipeline as the command bar. The UI selector mirrors the workspace format dropdown.
• Scheduler-ready architecture: a SchedulerProvider interface (mirroring the queue interface) with adapters for Vercel Cron, node-cron, or Inngest schedules. Phase 1–3 implement the interface + a manual "Sync now" trigger; the real schedule is wired in Phase 4.
• Respect for providers: watcher runs share the same proxy rotation and rate-limit safeguards as manual archives.

Acceptance: watchers only ingest genuinely new items (dedup index is the source of truth); disabling a watcher stops scheduled runs; a "Sync now" action works before cron wiring exists.

Rule J — Personal API & Webhooks (API-First Ingestion) + Developer API Gateway ✅ COMPLETE (Phase 3.6)

Problem. Users want to push links into their queue from external tools (iOS Shortcuts, Zapier, scripts) and query their archive programmatically.

Solution.

Data model — ApiKey collection (upgraded):

{
  userId: ObjectId;          // ref User
  name: string;              // e.g. "iOS Shortcut"
  prefix: string;            // visible id, e.g. "xa_live_3f8a…"
  keyHash: string;           // sha256 of the full key — NEVER store plaintext
  scopes: string[];          // e.g. ['archive:create']
  autoSave: boolean;         // default true — persist archives; false = live-stream only
  targetBaseIds: [ObjectId]; // default Bases new archives are linked into (Rule K)
  defaultFormat: 'mp4'|'mp3'|'original';  // default output format (Rule H)
  lastUsedAt?: Date;
  revokedAt?: Date;
  createdAt: Date;
}

• API key auth guard (lib/auth/api-key-guard.ts): intercepts Authorization: Bearer xa_live_…, hashes the token (SHA-256), looks it up (revoked keys rejected), attaches the owner + key profile, and asynchronously bumps lastUsedAt. Key verification runs in the route handler (Node runtime), not Edge middleware — revocation is immediate.
• Settings UI (/workspace/settings/api): generate a key (full value shown once in a copy sheet), name + auto-save toggle + target-Base chips + default format; list active keys with last-used; revoke immediately. Interactive curl / JS / Shortcut snippets are rendered inline.

API surface (v1) — live now:

• POST /api/v1/extract — Live Extraction Gateway. Body { url, save?, baseIds?, format?, customMeta? }; omitted fields fall back to the key's defaults (autoSave, targetBaseIds, defaultFormat).
  • save:false → executes the Scraper Registry inline and streams the raw media bytes back (Content-Disposition attachment; zero DB writes).
  • save:true → runs global dedup: new files are queued as INGEST_MEDIA with customMeta injected (worker extracts/uploads); existing files are linked + enriched (merge metadata.custom, $addToSet baseIds) without re-downloading. Returns { status, saved, mediaId?, downloadUrl?, streamData } where downloadUrl is the masked /api/media/[id]/download endpoint.
• GET /api/v1/media — Deep Dynamic Query API. Parses arbitrary query params into MongoDB dot-notation filters: ext.<key>=value → metadata.custom.externalIds.<key> (e.g. ?ext.tmdb=550), attr.<key>=value → metadata.custom.attributes.<key> (e.g. ?attr.course=CS101), tag=value, plus standard baseId, type, search, page, limit. Returns Media docs with masked downloadUrl + normalized dynamic metadata.
• GET /api/v1/bases — lists the key owner's Bases (IDs for targeting archives).
• Rate limiting: every /api/v1/* request is rate-limited per key (default 60 req/min; API_RATE_LIMIT / API_RATE_WINDOW_MS), returning 429 with Retry-After (Rule M).
• Inbound webhooks are the same routes — an iOS Shortcut/Zapier webhook simply POSTs to /extract.
• Service-layer reuse: the API route calls the same createItem/dedup helpers as the Server Actions (Rules D/G/K) — no duplicated business logic.
• Outbound webhooks (design space, not built): future completion notifications to user-configured URLs with HMAC-signed payloads.

Acceptance: a key can enqueue or live-stream from curl/Shortcut with per-request custom metadata; revoked keys are rejected; full keys never touch the DB in plaintext; dynamic metadata is queryable over the API.

Rule L — The Dynamic Metadata Engine (Truly Dynamic, Platform-Agnostic)

Problem. Hardcoding external platforms (TMDB, IMDb, …) couples the schema to specific providers. Users need to attach arbitrary metadata to archived media and query it deeply.

Solution. Media.metadata is a flexible nested structure — no platform keys are hardcoded:

metadata: {
  extracted: Mixed,   // raw data straight from the scraper strategy (title, duration, urls, …)
  custom: {           // user-defined / API-injected — 100% dynamic
    externalIds: Map<String>,   // mapping to ANY external system: {"tmdb":"123","notion_id":"xyz"}
    attributes:  Map<Mixed>,    // arbitrary key-value pairs: {"course":"CS101","rating":5}
    tags:        [String]       // free-form labels, indexed
  }
}

• Wildcard index: MongoDB allows one wildcard index per collection, so both custom maps are covered by a single index using wildcardProjection on metadata.custom.externalIds + metadata.custom.attributes. Any dynamically added key becomes queryable (?ext.tmdb=550 / ?attr.course=CS101) without migrations.
• Legacy compatibility: normalizeMediaMetadata (lib/metadata.ts) treats pre-upgrade flat metadata as extracted, so old documents keep rendering without a migration job.
• Injection points: the workspace submission form has an Advanced Metadata builder (dynamic key-value rows for external IDs + attributes, plus a tag chip input); archiveUrls / archiveSelectedItems thread customMeta through EXPAND_BATCH → createItem → the Media placeholder; the worker writes scraper data into metadata.extracted only, so user custom data is never clobbered; re-links merge ($set per key + $addToSet tags) instead of replacing.
• Inspector editing: the slide-out Inspector renders metadata.custom live with add/edit/delete (server actions in lib/actions/media-meta.ts), plus the read-only extracted view.
• Query API: GET /api/v1/media exposes dot-notation queries over both maps + tags (Rule J).

Acceptance: any key can be injected from the UI or API without a schema change and is immediately queryable; existing documents remain readable.

---### Rule M — Queue Management: Modal Feedback, CRUD, Start/Stop & Auto-Deletion

Problem. After clicking Archive, users saw only a toast. They needed live feedback on the queued jobs themselves (queued → processing → succeeded/failed/cancelled, with attempts + error detail), the ability to control congestion (cancel/retry/delete jobs, start/stop the queue), automatic cleanup of old job records, and base pages that show archived files only — never job queues.

Solution.

• GET /api/workspace/jobs — polling endpoint returning the user's recent ArchiveJob records (serialized, owner-scoped) plus queuePaused for the start/stop control.
• QueueControl modal (button on /workspace): live job feed with status icons/badges, attempts, error detail (including truncated yt-dlp stderr), batch/watcher tags, and relative timestamps; polls every 4s while any job is active or the modal is open so finished states appear live.
  • Pause/Resume (start/stop): setQueuePaused toggles User.queuePaused. While paused, the archive Server Actions (archiveUrls, archiveSelectedItems) and POST /api/v1/extract (save:true) refuse with a clear message; existing jobs drain. The queue button shows a paused indicator.
  • Job CRUD: lib/actions/jobs.ts — cancelJob (removes the pending Agenda job via agenda.cancel({ "data.jobId": id }), marks the ArchiveJob cancelled (reason user), flags the pending Media placeholder as failed), retryJob (re-enqueues INGEST_MEDIA with the original format/tags, resets status to queued), deleteJob (finished records only), clearFinishedJobs (removes all succeeded/failed/cancelled).
  • Auto-deletion: CLEANUP_JOBS agenda job (registered in lib/queue/jobs.ts, scheduled daily from src/worker.ts via CLEANUP_CRON, default 0 3 * * *) deletes succeeded jobs older than JOB_RETENTION_DAYS (default 7) and failed/cancelled older than FAILED_JOB_RETENTION_DAYS (default 30). ArchiveJob gains a (userId, status, updatedAt) index for fast cleanup.
• Base pages show files only: the inline queue panel is gone from /workspace; queue feedback lives exclusively in the modal. Base navigation opens the in-depth management page.
• In-depth Base management (Rule K): /workspace/bases/[id] — header with name/description/color
  • stats (file count, ready count), Edit (name/description/color via updateBase) and Delete (confirm-then-delete; detaches media, never deletes files), plus a files-only MediaView grid (status badges + secure downloads). Sidebar base links route here.
• API rate limiting (Rule J, Phase 4 prep): lib/rate-limit.ts — fixed-window per-key limiter (API_RATE_LIMIT default 60/API_RATE_WINDOW_MS default 60s) enforced in requireApiKey; returns 429 + Retry-After. In-process; swappable for a MongoDB counter if the web process scales.
• Bugfix (baseIds conflict): createItem and the worker's final upsert previously put baseIds in both $setOnInsert and $addToSet, which MongoDB rejects with "Updating the path 'baseIds' would create a conflict". baseIds now lives in $addToSet only (covers insert + update); createItem returns the created mediaId.

Acceptance: every queued job is visible in the modal with live status and per-job controls; pausing refuses new archives; cancelled jobs stop processing; finished records auto-delete on a retention schedule; base pages show only archived files.

Rule N — Admin User Management & Job Retry Cap (Phase 3.9)

Problem. Admins had no way to manage platform accounts (roles, disable, delete) from the UI, and a transiently-failing job could keep retrying — burning extraction cost with no upper bound.

Solution.

• /admin/users panel: server page loads every account (footprint counts: media, bases, watchers, API keys) and renders UsersManager — a searchable table with role badges, status (Active/Disabled), joined date, and per-row actions (edit modal, enable/disable, two-step delete).
• Edit modal: well-styled Sheet with an identity card, a radio-style role picker (User / Admin / Super Admin with hints), and a "Disable account" switch; saves both in one action.
• RBAC guards (lib/actions/users.ts):
  • listUsers — any admin.
  • updateUserRole — admin may move users between user↔admin; only superadmin may grant or revoke the superadmin role; nobody may change their own role; the last superadmin is protected from demotion.
  • setUserDisabled — disabled users can't sign in (credentials authorize and the OAuth JWT callback both reject disabled); superadmins can't be disabled by admins; last superadmin protected.
  • deleteUser — cascade deletes the user's Media (with best-effort cloud-storage file cleanup via the StorageOrchestrator), Bases, Watchers, API keys, and ArchiveJobs, then the account.
• Job retry cap: JOB_MAX_RETRIES (default 5) drives both the Agenda exponential backoff and a worker-side guard so INGEST_MEDIA never rethrows past the cap — the ArchiveJob is marked failed permanently instead of retrying forever.
• Bases retry sync: retryJob now reads the Media document's current baseIds and carries them forward, so a re-extracted file lands back in the same Bases.

Acceptance: admins can view/search every account and change roles/disable/delete with proper RBAC and self-protection; disabled accounts cannot authenticate; failed jobs stop retrying after JOB_MAX_RETRIES; retried jobs preserve their Base membership.

Rule O — Bases CRUD Hub & Worker Observability (Phase 3.8)

Problem. Bases were only manageable from each base's detail page, and failing jobs were impossible to diagnose from the worker console (no structured logging).

Solution.

• Bases CRUD hub (/workspace/bases): server page aggregates per-base file/ready counts (Media.aggregate unwind on baseIds) and renders BasesManager — a grid of base cards with inline create sheet, edit sheet (name/description/color), confirm-then-delete, and a deep link into each base's management page (/workspace/bases/[id]). The sidebar "Bases" header links to the hub; createBase/updateBase/deleteBase revalidate /workspace/bases.
• Worker observability (lib/log.ts): leveled structured logger (debug/info/warn/error, ISO timestamps, scoped prefixes, TTY colors, WORKER_LOG_LEVEL/LOG_LEVEL). formatError surfaces the full stack + yt-dlp stderr/stdout/exit code (the queue records only truncated error strings; the console gets the whole story).
• Agenda lifecycle events wired in src/worker.ts: start/success/fail/retry/ retry exhausted/error — every job event logged with jobId + payload summary.
• Step-by-step processor logging in lib/queue/jobs.ts: INGEST_MEDIA logs claim, proxy selected (or direct), dedup hit, overwrite delete, extraction start/result, upload start/done, and failure with formatError; EXPAND_BATCH logs per-URL expansion + per-item enqueues; CRON_WATCHER logs per-watcher sync results; CLEANUP_JOBS logs retention counts.
• Follow-up audit: see FOLLOWUP_AUDIT.md — every follow-up suggested across the build is tracked with status (done / partial / recommended / out-of-scope) and a priority-ordered next- action list (top: unit tests, workspace media search, API scope enforcement).

Acceptance: a user can fully manage Bases from one page; a failing job's root cause (incl. raw yt-dlp stderr) is visible in the worker console without code changes.

Rule P — DB-First Platform Settings (Phase 4.6)

Problem. Every tunable (queue concurrency, retry caps, scrape toggles, rate limits, storage fallback credentials, log level, cron schedules) lived in .env files. Changing a knob meant an SSH edit + container restart, and operators had no UI to see what was actually in effect.

Solution. MongoDB is the source of truth for every runtime knob. Environment variables become bootstrap defaults only. Resolution order, everywhere in the codebase:

DB value  →  env var  →  built-in default

Data model — Setting collection:

{
  key: string;        // stable registry key, e.g. "queue.maxRetries"
  value: Mixed;       // coerced to the registry type (number/string/boolean/csv)
  updatedBy: ObjectId; // ref User — admin who changed it
  updatedAt: Date;
}

Registry (lib/settings.ts): every env-based knob is declared ONCE in a typed registry (SettingDef: key, env var, default, type, group, label, description, secret?, restartRequired?, min/max). The resolver is DB-first with an in-process cache (SETTINGS_CACHE_TTL_MS, default 30s) so hot paths (rate limiter, queue config, scrapers) never hit MongoDB per call.

• Hydration: connectToDatabase() and the worker boot call hydrateSettings() (best-effort, never throws — if the DB is down, env/defaults apply). A periodic TTL refresh re-reads DB overrides so changes propagate to other processes without restart.
• Live knobs via setters: the log level is pushed into lib/log.ts via setLogLevel() after hydration (no import cycle). Restart-required settings (cron schedules, queue collection, process interval) are flagged in the UI.
• /admin/settings UI (admin+): every registered setting rendered grouped (Queue, Scraping, API, Inspection, Logging, Storage, Auth) with its effective value, source badge (DB / env / default), env fallback display, secret masking, edit sheet (typed input; booleans as switches; CSV as comma lists), and a reset action that deletes the DB override → falls back to env/default. Settings with restartRequired are labelled so operators know to redeploy.
• Secrets: secret settings (blob token, S3 keys) are masked in the UI and never serialized in full; leaving the edit field blank preserves the current value.
• Migration status: every module that previously read process.env.X at runtime now reads through the registry — queue (jobs.ts, agenda.ts), scrapers (yt-dlp player/UA/cookies, stealth domains/executable, page-harvest, STEALTH_FALLBACK), API (rate limit/window, key prefix), auth (ADMIN_EMAILS), inspection TTL, storage fallback credentials, log level, retention days, and cron schedules. Env vars remain as the bootstrap seed for first boot.

Acceptance: an admin can change any platform knob from /admin/settings without touching files; changes apply after the cache TTL (instantly in the writing process); env vars still seed first boot; secrets are never exposed in full; the registry is the single place new knobs are added.

Rule P.1 — White-Labeling (Branding, admin-configurable)

Problem. Operators deploy Xtreambase under their own brand (name, logo, tagline, accent color, support email). Hardcoding "Xtreambase" strings across layouts makes white-labeling a code change per customer.

Solution. Branding is DB-first like every other knob — a dedicated Branding group in the Rule P registry (lib/settings.ts), read through lib/branding.ts:

• BrandLogo (client-safe): components/branding/brand-logo.tsx renders image URL → emoji → default icon, painted with the accent when set. It receives a plain BrandingView object (lib/branding-types.ts — type-only, no server imports), so it is safe in the browser bundle.
• Root layout: generateMetadata awaits hydrateSettings() and uses branding.name/ branding.tagline; the accent color is injected as CSS custom-property overrides on :root (brandAccentCss, with a luminance-computed readable foreground) so every bg-primary / text-primary surface follows the brand color. No DB → env/default branding, never throws.
• Auth screens: login, register, and first-boot setup render BrandLogo + name + tagline + optional footer (support email as mailto:).
• Workspace/admin shell: the sidebar brand row and the /workspace page header use the effective branding (layouts await hydration before passing it into the client shell).
• /admin/branding UI (admin+): dedicated page with a form (all seven fields, a color picker for the accent) and a live preview (auth header, sidebar row, footer). Save = updateBranding (writes each registered key, invalidates the cache, revalidates); Reset = resetBranding (deletes all overrides → env/default). The same settings also appear under the "Branding" group in /admin/settings. A Branding overview card links from /admin.
• The generic settings page re-uses the same registry, so branding keys need no special handling.

Acceptance: an admin can fully re-brand the platform from /admin/branding without code or env changes; changes apply instantly (after cache TTL across processes); a missing DB falls back to the built-in Xtreambase identity; client components never import the server branding module.

4. End-to-End Archive Flow (Reference Sequence)

4.1 Manual single/batch archive

User pastes URL(s) in workspace (optionally with format settings)
   │
   ▼
Server Action: archiveMedia / archiveMediaBatch({ urls, format, overwrite })
   │
   ├─ Authenticate (NextAuth v5) ──► resolve User
   ├─ Normalize + validate URLs  (batch caps applied)
   ├─ Registry: resolve strategy per URL (Rule A)
   ├─ Playlist/channel? ──► expand via strategy (Rule G)
   ├─ Dedup check on (userId, sourceUrl)  ── duplicate? ──► skip or overwrite path
   │
   ▼
Enqueue ArchiveJob(s) with shared batchId (Rule F)
   │
   ▼
Worker (concurrency-limited): ProxyManager hands out live proxy (Rule B)
   ├─ Strategy.extract(url, { proxy, format, headers, cookies })  ──► ExtractResult
   │
   ▼
IStorageAdapter.uploadStream(...)    (Rule C)  ──► storageUrl
   │
   ▼
Persist/update Media document        (Rule D)  ──► emit completion event
   │
   ▼
UI Inspector pane + grid update (per-item status, batch progress)

4.2 Watcher auto-sync (scheduled)

SchedulerProvider fires (Vercel Cron / node-cron / Inngest) — or manual "Sync now"
   │
   ▼
For each enabled Watcher (Rule I): resolve strategy → expand/scan
   ├─ Diff against existing Media (dedup index) → only new items
   ▼
Enqueue ArchiveJob(s) with watcherId tag → same worker pipeline as 4.1

4.3 External push (Personal API)

POST /api/v1/archive  (Bearer xa_live_…) from iOS Shortcut / Zapier / curl
   │
   ├─ Rate limit + key scopes check (Rule J)
   ▼
Same batch pipeline as 4.1 → returns { jobIds, status: 'queued' }

5. Project Layout Blueprint (Anticipated)

xtreambase/
├─ app/                        # Next.js 15 App Router
│  ├─ (workspace)/             # authenticated workspace shell
│  │  ├─ archive/              # central grid/list + inspector
│  │  ├─ watchers/             # Watcher management UI (Rule I)
│  │  ├─ vault/                # Proxy Vault UI
│  │  ├─ settings/             # profile + API key management (Rule J)
│  │  ├─ admin/storage/        # storage config UI — quotas/fallbacks (Rule C)
│  │  └─ layout.tsx
│  ├─ api/
│  │  ├─ media/[id]/download/route.ts  # masked download resolution (Rule C.3)
│  │  └─ v1/
│  │     └─ archive/route.ts   # POST — personal API ingestion (Rule J)
│  ├─ actions/                 # Server Actions (archiveMedia[Batch], watcher CRUD, proxy CRUD, apiKey CRUD)
│  ├─ login/                   # NextAuth v5 pages
│  └─ layout.tsx / globals.css
├─ components/                 # shadcn/ui + workspace components
│  ├─ ui/                      # shadcn primitives
│  ├─ layout/                  # Sidebar, CommandBar, InspectorPane, BottomSheet
│  ├─ archive/                 # grid/list, batch input, format selector
│  └─ watchers/                # watcher list/form (Rule I)
├─ lib/
│  ├─ db/                      # Mongoose connection + models (User, Media, Proxy, ArchiveJob, Watcher, ApiKey, StorageConfig)
│  ├─ scrapers/                # IScraperStrategy, registry, YtDlpStrategy, StealthBrowserStrategy
│  ├─ storage/                 # IStorageAdapter, VercelStorageAdapter, S3StorageAdapter, StorageOrchestrator, factory
│  ├─ proxy/                   # ProxyManager (rotation, health)
│  ├─ auth/admin.ts            # requireAdmin guard
│  ├─ queue/                   # QueueProvider interface + adapters (in-process → Agenda/Inngest)
│  ├─ scheduler/               # SchedulerProvider interface + adapters (Vercel Cron / node-cron / Inngest)
│  └─ api/                     # API key hashing, verification, rate limiting (Rule J)
├─ types/                      # shared TypeScript types (incl. FormatConfig)
├─ docs/                       # in-app documentation reference (/docs)
│  └─ page.tsx                 # renders REQ.md / README.md with tabs, searchable TOC, raw download
├─ components/docs/            # DocsView (client) — react-markdown + remark-gfm, prose styling
├─ app/api/docs/[slug]/route.ts # raw markdown download (attachment)
├─ app/icon.svg                # branded Xtreambase SVG favicon (equalizer/stream mark)
├─ proxy.ts                    # NextAuth session guard — Next.js 16 renamed middleware → proxy (Edge-safe only; /api/v1 key auth lives in route handlers)
├─ .env.example
└─ package.json

6. Configuration & Environment (Planned)

MONGODB_URI=...
AUTH_SECRET=...
AUTH_URL=...
AUTH_GITHUB_ID=...          # OAuth providers per Phase 1 decision
AUTH_GITHUB_SECRET=...
STORAGE_DRIVER=vercel | s3        # swap with zero code changes (Rule C)
BLOB_READ_WRITE_TOKEN=...         # Vercel Blob
# S3 (or provided per-StorageConfig in the admin UI)
# AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, S3_BUCKET, S3_REGION
QUEUE_DRIVER=inprocess | agenda | inngest
SCHEDULER_DRIVER=vercel-cron | node-cron | inngest   # watchers (Rule I)
YTDLP_PATH=...                    # optional override for the yt-dlp binary (default: youtube-dl-exec's downloaded bin/yt-dlp)
YOUTUBE_DL_DIR=...                # optional: binary directory override (youtube-dl-exec)
YOUTUBE_DL_FILENAME=...           # optional: binary name override (youtube-dl-exec)
YTDLP_PLAYER_CLIENT=android,tv    # YouTube player-client impersonation (bot-check bypass); 'off' disables
YTDLP_USER_AGENT=...              # realistic browser UA for extraction (default: Chrome 126)
YTDLP_COOKIES=/path/cookies.txt   # optional Netscape-format cookies for logged-in/hard-blocked targets
PUPPETEER_EXECUTABLE_PATH=...     # optional, for StealthBrowserStrategy
ADMIN_EMAILS=you@example.com      # comma-separated bootstrap admin list
API_KEY_PREFIX=xa_live            # personal API key prefix (Rule J)

7. Execution Plan (Strict Phases — Do Not Jump Ahead)

Phase 0 — Requirements & Architecture (in progress — awaiting owner approval)

• Document product vision, architecture rules, schemas, UI strategy (this file).
• Gate: owner approves REQ.md.

Phase 1 — Scaffolding

• Scaffold Next.js 15 (App Router, TypeScript strict).
• Configure MongoDB + Mongoose connection and models (User, Media, Proxy, ArchiveJob, Watcher, ApiKey).
• Wire NextAuth.js v5 (credentials/OAuth, session guard middleware).
• Set up Tailwind + shadcn/ui base (theme, primitives).
• .env.example, lint/format, CI-friendly scripts.
• No scraping yet. Gate: app boots, auth + DB verified.

Phase 2 — Backend Core ✅ COMPLETE

• Storage orchestration: IStorageAdapter + getDownloadUrl, VercelStorageAdapter, S3StorageAdapter, StorageConfig model (quotas/fallbackNodeId), StorageOrchestrator with the fallback chain, and /api/media/[id]/download masked-download route (Rule C).
• Admin foundation: User.role (user/admin/superadmin), requireAdmin/requireSuperAdmin guards, first-boot /setup wizard, /admin/storage UI + server actions (add/edit nodes, quotas, fallbacks, cycle detection, single-default enforcement).
• ProxyManager (round-robin, health checks, persistence).
• Scraper Registry: IScraperStrategy, YtDlpStrategy, StealthBrowserStrategy, registry + URL routing.
• Playlist/channel expansion (canExpand / expand) on strategies (Rule G).
• FormatConfig pass-through in ExtractOptions (Rule H).
• Proxy injection into strategies (Rule B integration).
• Dedup/overwrite service layer (Rule D).
• Agenda.js queue (Rule F): lib/queue/ (singleton, producers, job definitions with exponential backoff), src/worker.ts entry, esbuild bundle → dist/worker.js, dual-entry scripts (dev/dev:all/start/start:all/worker), Dokploy-ready.
• Gate: backend core built + verified (tsc/lint/build all green).

Phase 3.6 — Developer API Gateway & Dynamic Metadata Engine ✅ COMPLETE

• ApiKey schema upgraded: autoSave, targetBaseIds, defaultFormat.
• Media.metadata → dynamic { extracted, custom: { externalIds, attributes, tags } } + wildcard index (wildcardProjection).
• API key guard (lib/auth/api-key-guard.ts), generate/revoke actions (show-once).
• POST /api/v1/extract (save:false live-stream / save:true dedup+queue+merge), GET /api/v1/media (deep dynamic query), GET /api/v1/bases.
• /workspace/settings/api Developer UI: key table, generate modal, show-once sheet, code snippets; sidebar nav.
• Advanced Metadata builder in the submission form; customMeta threading (actions → EXPAND_BATCH → createItem → worker); re-link merge.
• Inspector dynamic metadata editor (externalIds/attributes/tags add/edit/delete).
• Queue Activity panel + /api/workspace/jobs; fixed the baseIds $setOnInsert/$addToSet conflict.
• Gate: tsc/lint/build green.

Phase 3.8 — Bases CRUD Hub & Worker Observability ✅ COMPLETE

• /workspace/bases index page with per-base file/ready counts (aggregate) + BasesManager full CRUD (create/edit/delete/confirm + deep links); sidebar Bases header → hub.
• lib/log.ts structured logger + formatError (stack + stderr/stdout/exit).
• Agenda lifecycle event logging in src/worker.ts (start/success/fail/retry/error).
• Step-by-step processor logging (INGEST_MEDIA / EXPAND_BATCH / CRON_WATCHER / CLEANUP_JOBS).
• FOLLOWUP_AUDIT.md — full audit of every suggested follow-up with status + next actions.
• Gate: tsc/lint/build green.

Phase 3.5 — Selective Playlist Import & Direct Files ✅ COMPLETE

• inspect() on IScraperStrategy + registry (resolveAsync with HEAD Content-Type probe).
• DirectFileStrategy — extension + HEAD detection, direct streaming into the orchestrator, same global dedup + multi-Base linking. Registered first (priority 300).
• POST /api/scraper/inspect — auth-guarded, lightweight per-item metadata + archived flag.
• PlaylistSelectorModal — Select All/counter, search, thumbnails/duration, global + per-item formats, Base selector → archiveSelectedItems (direct INGEST_MEDIA enqueue).
• Watcher sync: baseIds + formatOption on watchers; CRON_WATCHER threads both.
• Worker env parity (scripts/load-env.cjs preload).
• Gate: tsc/lint/build green.

Phase 3 — Workspace UI ✅ COMPLETE

• App shell: Sidebar (collapsible on desktop, drawer on mobile), top bar, user menu.
• Archive command bar: bulk multi-line input, format selector (MP4/MP3/Original), overwrite toggle, Base chip multi-select (Rules G/H/K), wired to archiveUrls.
• Central grid/list view with status badges (Pending/Processing/Ready/Failed), live polling via /api/workspace/media, secure download via masked endpoint.
• Slide-out Inspector pane (bottom sheet on mobile) with flexible metadata + bases.
• Bases architecture (Rule K): Base model, sidebar Bases section + create dialog, per-base filtering, global dedup link-only path, "Manage Bases" on media cards.
• Watchers UI (/workspace/watchers): add/track, enable/disable, "Sync now" (Rule I).
• Proxy Vault UI (/admin/proxies): bulk-add, health test, enable/disable (Rule B).
• Settings: personal API key generate/revoke/copy-once (Rule J) — Phase 4.
• Gate: full UX parity desktop ↔ mobile.

Phase 4 — Final Integration, Real-Time UI & Dokploy Deployment ✅ COMPLETE

• Worker-side inspection: POST /api/scraper/inspect now enqueues INSPECT_URLS and returns { jobId }; GET /api/scraper/inspect/[id] polls the worker's InspectJob (TTL auto-expiry). Fixes the web-process Cannot find module 'youtube-dl-exec' failure by moving ALL yt-dlp work to the worker. loadYoutubeDl() cwd fallback for save:false.
• Real-time UI feedback: queue modal + media grid poll (/api/workspace/jobs, /api/workspace/media); toasts fire when polling detects a newly-failed job (worker errors like "Private Video"/"Proxy dead" surface live, even with the modal closed).
• Production deployment: multi-stage Dockerfile (ffmpeg + yt-dlp binary + Puppeteer Chromium baked in; non-root user; CMD overridable) + .dockerignore. start:web (next start) and start:worker (node -r ./scripts/load-env.cjs dist/worker.js). See § 10 Deployment Guide (Dokploy).
• API rate limiting: per-key fixed-window limiter enforced on every /api/v1/* route via requireApiKey (default 60 req/min, 429 + Retry-After) + per-user limit on inspect.
• Masked downloads verified: /api/media/[id]/download requires a session, scopes the Media doc to the user, and 307-redirects to a short-lived storage URL — raw URLs never reach the client.
• Pipeline audit: URL paste → inspect (worker) → format select → queue → worker → proxy rotate → extract (fallback chain) → storage orchestrator (fallback chain) → MongoDB upsert (global dedup + dynamic metadata passthrough verified).
• Gate: paste-link-to-archive works end-to-end with async jobs; tsc/lint/build green.

10. Deployment Guide — Dokploy (Docker)

Architecture

One image (docker build -t xtreambase .), two Dokploy services sharing only MongoDB:

Deps stage gotchas:

• youtube-dl-exec's preinstall script requires a Python ≥3.9 binary (its postinstall fetches a Python zipapp) and will fail npm ci in a slim Node image — the Dockerfile sets YOUTUBE_DL_SKIP_PYTHON_CHECK=1 and replaces the zipapp with the true standalone yt-dlp_linux ELF, so no Python is needed.
• Puppeteer's postinstall would download Chrome from Google's CDN, which is frequently blocked on VPS/datacenter IPs (Dokploy) — the image installs chromium from Debian apt instead (PUPPETEER_SKIP_DOWNLOAD=true + PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium), in both the deps and runner stages.
• youtube-dl-exec's postinstall downloads the yt-dlp binary through the GitHub API (api.github.com/…/releases/latest), which is rate-limited on VPS/datacenter IPs and fails npm ci with "API rate limit exceeded". The Dockerfile skips it (YOUTUBE_DL_SKIP_DOWNLOAD=1) and fetches the yt-dlp_linux asset (true PyInstaller standalone ELF — the plain yt-dlp asset is a Python zipapp) from the GitHub release CDN (…/releases/latest/ download/yt-dlp_linux — a plain asset download, NOT API-rate-limited) into node_modules/youtube-dl-exec/bin/yt-dlp, verified with yt-dlp --version. Use yt-dlp_linux_aarch64 on ARM hosts.
• If you build with your own base image, set the same env vars (or install python3, a browser, and let the postinstall download run from an IP GitHub's API accepts) or the build will fail the same way.

Steps

Option A — Docker Compose (recommended): a root docker-compose.yml is the single deployment entry point. In Dokploy, create a Docker Compose service and point it at this file:

• web container — npm run start:web, port 3000, with the core Traefik labels so the Dokploy network routes traffic: traefik.enable=true, traefik.docker.network=dokploy-network, web router (Host(\${DOMAIN}`), entrypoints=web, middlewares=redirect-to-https@file), websecure router (Host(`${DOMAIN}`), entrypoints=websecure, tls.certresolver=letsencrypt), loadbalancer.server.port=3000— plus a/api/health` healthcheck (DB-free liveness probe).
• worker container — npm run start:worker, no ports, no labels (MongoDB-only egress).
• Both join the external dokploy-network (created by Dokploy's Traefik stack). DOMAIN (default domain.xyz) feeds the Host() rules; set it in Dokploy env to re-route.

docker compose up -d --build   # on the VPS, or via Dokploy's compose UI

Option B — manual services from the same image:

1. Build the image locally or in Dokploy: docker build -t xtreambase ..
2. Create the Web service from the image: port 3000, env vars below.
3. Create the Worker service from the SAME image, overriding the command to npm run start:worker (Dokploy supports per-service commands), env vars below.
4. Add a MongoDB instance (Atlas or Dokploy-managed) and set MONGODB_URI on both services.
5. First boot: open the Web service → redirected to /setup → create the superadmin.

Traefik labels (Option B, if you manage labels manually instead of using the compose file): the compose file carries the canonical, focused set — traefik.enable=true, traefik.docker.network=dokploy-network, the web/websecure routers above, and traefik.http.services.<name>.loadbalancer.server.port=3000. Attach the same labels to the Web service when deploying outside compose.

Environment (both services)

Image contents (no manual setup on the VPS)

• ffmpeg — required by yt-dlp for MP3/MP4 format conversion.
• yt-dlp binary — node_modules/youtube-dl-exec/bin/yt-dlp, the yt-dlp_linux standalone ELF fetched during the deps stage directly from the GitHub release CDN (postinstall skipped via YOUTUBE_DL_SKIP_DOWNLOAD=1 — the GitHub API used by the postinstall is rate-limited on VPS IPs, and its zipapp artifact needs Python).
• Puppeteer Chromium — installed from Debian apt (chromium, wired via PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium; puppeteer download skipped with PUPPETEER_SKIP_DOWNLOAD=true), for StealthBrowserStrategy.
• Runs as the non-root nextjs user; the worker writes only to MongoDB (no filesystem privileges needed).

Scaling notes

• To scale the worker, run additional worker services with the same image — Agenda's MongoDB locking prevents double-execution of the same job.
• The web service is stateless; the in-process rate limiter resets on restart (swap to a MongoDB counter if you run many web replicas).

8. Non-Functional Requirements

• Security: all archive/storage/proxy mutations behind NextAuth sessions; /admin/* and storage config writes gated by the admin role; API keys stored as sha256 hashes only; never expose proxy credentials, full keys, or raw storage URLs in client payloads (all downloads go through the masked internal route with short-lived URLs); sanitize metadata before display; HMAC-signed payloads for any future outbound webhooks.
• Privacy: media and metadata are per-user scoped on every query (userId filter mandatory).
• Rate limiting & caps: max URLs per batch request, max playlist expansion size, API-key rate limits, and watcher-run throttles (shared config, provider-aware).
• Observability: structured logs for jobs (strategy, proxy used, duration, result, batchId, watcherId).
• Error taxonomy: typed errors (DuplicateArchiveError, StrategyNotSupportedError, ProxyUnavailableError, ExtractionFailedError, StorageError, RateLimitError, InvalidApiKeyError) surfaced in UI and API responses.
• Performance: UI never blocks on long jobs (Rule F); optimistic updates for archive status; batch progress is aggregate + per-item.

9. Risks & Open Questions

Appendix A — Original Request (Verbatim)

so i want to build a platform my goal is for user to build up their archive base of stream data with it they paste it share url and have data scrapped and uploaded to there media db for the media upload lets use vercel for now i will pivot to an s3 storage later so the adaoter for media should work as such for now just scalfold the project layout use great tech stack for the dev i like next js so for now scalfolde the auth flow and user workspace for building up file data refer to this site https://ytdlp.online/playlist-downloader this sit should be a fraction of my platform features as similar platform or features required for videos musics can be itegrated my goal is i paste in a link i get the data i need and have uploaded to the user and for a current base that is being created uniquenes should also be implemented unless a user wannt to overwite. for now write out my request to a req.d file do just that