Docs · runbook

Quick reference

# start the UI (leave it running in one terminal)
python3 -m ui.server

# provision new accounts from a vendor file
python3 register.py --vendor-file state/order.txt    --count 8
python3 register.py --vendor-file state/order-40.txt --count 40

# backfill onboarding for any row where step-7 tripped (safe no-op if none)
python3 onboard.py --concurrency 3

# start the farm — restart to pick up new accounts
python3 farm.py --loop

# audits
python3 audit.py --check proxy-lock    # verify each account still has a distinct exit IP
python3 audit.py --check token         # exercise Firebase refresh across all accounts

# one-shot play cycle on a single account (for smoke testing)
python3 farm.py --dry-run --account 3

# DB stats
python3 -m db stats
      

Pipeline overview

Four long-lived roles. Each has its own responsibility and stays out of the others' way.

Process	What it does	How to run
UI	Dashboard + operator actions. Reads DB, doesn't touch elevenmusic API except when you press a proxy-probe button.	python3 -m ui.server
register.py	Reads a vendor order*.txt, provisions accounts (Firebase magic-link + onboarding + first /v1/me warmup) through each account's locked Shifter session.	python3 register.py --vendor-file …
onboard.py	Retries POST /v1/me/onboarding for any accounts where the inline step 7 during register.py failed. No-op if all are onboarded.	python3 onboard.py --concurrency 3
farm.py --loop	The runtime. One asyncio coroutine per onboarded account; picks a track (30 % own catalog / 70 % genre playlists), plays it start → heartbeats → end through the account's locked proxy, occasionally hearts + follows.	python3 farm.py --loop

Target-track campaign

When you need to concentrate listens on one specific track by a deadline (e.g. "get 220 verified plays on SUNBURN by Saigon midnight"), toggle the campaign via .env:

TARGET_TRACK_ID=6a8d1c1eb45e8db752eb6649
TARGET_TRACK_MIN_PER_ACCOUNT=6
TARGET_TRACK_MAX_PER_ACCOUNT=8
TARGET_DEADLINE_UTC=1788195599   # Aug 31 16:59:59 UTC = Saigon midnight
PLAYS_PER_ACCOUNT_MIN=18         # bump daily quota so target fits
PLAYS_PER_ACCOUNT_MAX=25
TARGET_TRACK_GOAL=220            # UI progress bar goal (default 220)
      

Then restart farm.py. Every active account will:

• Pick a deterministic per-account "want" from [MIN, MAX]. Same account picks the same want all day, so a restart is idempotent.
• Inject the target track probabilistically with p = target_still_needed / plays_still_remaining. This smears target plays across the day rather than clustering at the start — natural superfan pattern.
• Superfan multipliers already boost heart/follow rates for own-artist tracks, so most accounts will heart the target and many will follow the artist without extra config.
• _next_sleep_seconds compresses per-account scheduling into the campaign window so plays actually fire before the deadline, not after.

The dashboard and /our-artist show a live campaign card with progress bar, rate needed vs actual, and time-to-deadline. Auto-refreshes every 10 s.

Turn the campaign off by clearing TARGET_TRACK_ID= in .env and restarting farm.py — track selection reverts to the 30/70 own/others mix.

First-time setup

Only run once per machine. This project's .env is already filled in.

cd ~/sandbox/elevenmusic-daily-gen/playback-farm2000
cp .env.example .env      # if you haven't
$EDITOR .env              # PROXY_USER/PASS/GATEWAY + UI_TOKEN
pip install -r requirements.txt
python3 -m db init        # idempotent — safe to re-run any time
      

Registering accounts

Vendor files live in state/order*.txt, one per order. Format: email:mailbox_password:ms_refresh_token:ms_client_id. Register.py dedupes by email against the DB, so re-running with the same file is safe.

# provision N rows from a specific file
python3 register.py --vendor-file state/order.txt --count 8

# no --count means "all fresh rows in the file"
python3 register.py --vendor-file state/order-40.txt

# --country overrides PROXY_DEFAULT_COUNTRY for this run
python3 register.py --vendor-file state/order.txt --count 1 --country us
      

Pacing knobs (in .env)

• REGISTER_CONCURRENCY=2 — parallel workers. 1 for slowest, 5 tops.
• REGISTER_SLEEP_MIN=60, REGISTER_SLEEP_MAX=180 — per-worker jitter between rows.

At defaults, 50 accounts take ~50 min. Interruptible with Ctrl-C — any half-registered row gets picked up next run.

Onboarding backfill

Register.py runs onboarding inline as step 7. This script is the safety net for rows where that step 7 5xx'd. It's a no-op when everything is already onboarded.

# backfill all pending onboarding
python3 onboard.py --concurrency 3

# force-run for one specific account (bypasses the "pending" filter)
python3 onboard.py --account 7
      

farm.py only runs accounts with onboarded_at > 0, so an un-onboarded row just sits idle until you fix it here.

Running the farm loop

# main loop — 1 asyncio worker per account, all through their locked proxies
python3 farm.py --loop

# single play cycle for one account (great for smoke tests)
python3 farm.py --dry-run --account 5
      

What you'll see at startup

launching 17 account workers, cap=60 concurrent plays
[marcusprimeau537362]  queued  target=10/day  first play in ~34s (worker stagger 0..85s)
[newmanmissal1358181]  queued  target=15/day  first play in ~12s (worker stagger 0..85s)
…
[newmanmissal1358181]  awake — entering play loop
[newmanmissal1358181]  play OK  track=… [own] hold=44s hbs=4 final=46102ms
[newmanmissal1358181]    engage: ♥  track=… artist=…
[newmanmissal1358181]  next play in ~342s (14/15 remaining today)
      

Herd-thinning stagger scales automatically: min(30 min, 5 × n_workers). For dev runs override with INITIAL_JITTER_MAX_SEC=3 in .env.

Audits & sanity checks

# 1) verify each account still has a distinct exit IP through its shifter sid
python3 audit.py --check proxy-lock

# 2) exercise Firebase refresh through every account's proxy — flags
#    accounts whose refresh_token has been invalidated
python3 audit.py --check token

# 3) both, in that order
python3 audit.py --check all

# 4) DB row counts (quick health check)
python3 -m db stats
      

audit.py --check proxy-lock currently uses api.ipify.org as the probe target. Some Shifter exits are blocked from ipify but still reach api.elevenmusic.io fine, so you'll see false alarms. In the UI, prefer the Proxy → Refresh probe (all) button (same code path, but wired to the collision detector).

UI actions

The UI is read-only unless you pass ?token=<UI_TOKEN> in the URL. Your token is stored in .env present . Attach it once per browsing session (the URL doesn't strip it as you navigate).

Available actions

Where	Action	What it does
Account detail (/accounts/N?token=…)	Dry-run one cycle	Spawns python3 farm.py --dry-run --account N as a background subprocess. Output is written to logs/dryrun-N.log.
Account detail	Pause 1h	Sets state='paused', paused_until = now + 3600. Farm workers skip paused accounts.
Account detail	Unpause	Flips back to state='registered', clears paused_until.
Account detail	Mark dead	Permanently retires the account. Farm workers stop picking it up. Cannot be undone from the UI.
Proxy (/proxy?token=…)	Refresh probe (all)	Concurrently probes every active account's proxy via ipify, records the exit IP into ip_history, and updates the collision detector.

Restart / hot-reload behavior

• UI (uvicorn): HTML templates and static assets (CSS/JS) reload on file save with no restart needed — the app has a cache-buster query on app.css/app.js that flips whenever those files change. Python code changes require a full restart: Ctrl-C then python3 -m ui.server.
• farm.py --loop: HAS HOT-RELOAD for new accounts. A supervisor coroutine re-queries the DB every 60 s. When a new account transitions to state='registered' AND onboarded_at > 0, the supervisor spawns a worker for it with a short 1-10 min initial delay (not the herd-thinning 30 min used at startup). Existing workers are never disturbed. You do NOT need to restart farm.py while registration runs.
• Config changes in .env: require a farm.py restart to take effect (config is loaded once at process start via dotenv).
• register.py / onboard.py / audit.py: one-shot scripts. Nothing to reload. Interruptible; restart resumes from where the DB left off.

Troubleshooting

farm.py appears stuck at "launching N workers"

Not stuck — each worker sleeps a randomized initial delay so N workers don't wake in lockstep. The formula auto-scales: min(30 min, 5 × n_workers). For 12 workers that's ~60 s; for 2000 it's the 30-min cap. Override with INITIAL_JITTER_MAX_SEC=30 in .env for dev runs. Recent farm.py builds print queued target=N/day first play in ~Xs immediately so you can see it's alive.

"WARNING: N accounts registered but not onboarded"

The farm printed this at loop startup. Those N accounts have state='registered' but onboarded_at=0. Fix with python3 onboard.py --concurrency 3, then restart the farm.

play FAIL start 403 <!DOCTYPE html>…Cloudflare

The current Shifter exit for that account's sid is on Cloudflare's watchlist. Not fatal: the sid rebinds to a different exit after the 120 s sticky TTL, and farm.py retries up to PROXY_MAX_RETRIES before pausing the account for PAUSE_MINUTES_AFTER_FAILURES.

Firebase refresh returns 400 INVALID_REFRESH_TOKEN

The account's Firebase session was revoked (rare, sometimes triggered by "sign out all sessions" or a very long idle period). farm.py auto-marks the account state='dead'. No recovery path — the mailbox can create a new identity but we'd lose the elevenmusic user id.

UI shows "actions disabled — set UI_TOKEN"

UI_TOKEN is empty in .env. Generate one:

python3 -c 'import secrets; print(secrets.token_urlsafe(18))'

Set it in .env, restart the UI, then visit any page with ?token=<value>.

Proxy dashboard shows "IP collisions"

Two accounts landed on the same exit IP inside the 15-min lookback window. That violates the 1:1 lock invariant. Most common cause: stale ip_history rows from before a sid was properly locked. Investigate account histories on /accounts/{N}. If the collision is current (both accounts probed within the last 2 min against the same IP), it means Shifter's session engine is misbehaving — pause the newer of the two accounts and open a Shifter support ticket.

Config reference

All from .env. See Config page for current values (redacted).

Variable	Default	Meaning
PROXY_GATEWAY	p.shifter.io:443	Shifter residential entry point.
PROXY_USER / PROXY_PASS	—	Base credentials. Per-account sid is appended by the template.
PROXY_USERNAME_TEMPLATE	{user}-sid-{session}	Sticky-session syntax. Change if Shifter's format shifts.
PROXY_DEFAULT_COUNTRY	(empty)	Set to us to force-pin US exits.
REGISTER_CONCURRENCY	2	Parallel workers in register.py.
REGISTER_SLEEP_MIN/MAX	60 / 180	Per-worker jitter between rows (seconds).
MAX_CONCURRENT_PLAYS	60	Semaphore cap on in-flight plays.
PLAYS_PER_ACCOUNT_MIN/MAX	10 / 20	Random daily quota per account.
MIN_INTRA_ACCOUNT_GAP	1800	Minimum seconds between two plays from the same account.
HOLD_MIN/MAX, HEARTBEAT_MIN/MAX, FIRST_HB_MIN/MAX	32/58, 8/14, 3/6	Playback timing envelope.
PROXY_MAX_RETRIES	3	Consecutive failures before account gets paused.
PAUSE_MINUTES_AFTER_FAILURES	15	How long an auto-paused account stays paused.
INITIAL_JITTER_MAX_SEC	(auto)	Override the herd-thinning first-play delay.
GENRE_PLAYLIST_IDS	(6 defaults)	Comma-separated list of playlist IDs for the "others" branch.
PLAYLIST_CACHE_TTL_SEC	300	Global cache TTL for playlist fetches.
UI_HOST / UI_PORT	127.0.0.1 / 8787	Dashboard binding.
UI_TOKEN	(random at setup)	Gates write actions in the UI.

Useful URLs

Path	What's there
/	Dashboard: KPIs, own-vs-others strip, live event stream, 24 h chart.
/our-artist ★	Deep dive on NikoTura numbers — today / 24 h / 7 d / all time, top tracks, full 1110-track catalog table.
/accounts	Full roster. Filter by state / tz / browser / free-text.
/accounts/{id}	One account: identity, biases, IP history, hearts/follows, actions.
/events	Global event log, filterable.
/playback	Charts (24h + 7d), top-played tracks, per-account distributions.
/engagement	Hearts + follows tables, split us vs others.
/proxy	Last-seen exit IP per account + collision detector.
/registration	Vendor files vs DB. Auto-refresh every 5 s during a registration run.
/config	Redacted .env + DB counts.
/api/summary	JSON: everything on the dashboard KPI strip.
/api/artist_summary	JSON: all NikoTura numbers.
/api/events?limit=…&kind=…&account_id=…	JSON event tail.
/api/plays_per_hour?hours=24&scope=all|own|others	JSON series for charts.