IHNYC Service Docs

Pub operations

Active10 min read

Table of Contents

Release model

  • Pull requests must pass two independent gates: GitHub Actions runs npm ci + npm test, while Cloudflare Workers Builds uploads a non-production preview version
  • Production auto-deploys from main via Cloudflare Workers Builds; a manual wrangler deploy is a break-glass path, not the normal release workflow
  • No build step; worker and static assets are deployed as-is
  • Custom domain: pub.ihnyc-rc.org
  • Admin UI at /admin/* behind Cloudflare Access (Zero Trust)

Prerequisites

  • Node.js v18+
  • Wrangler CLI (npm install -g wrangler && wrangler login)
  • Access to the Cloudflare account that owns the worker

Secrets and variables

Required secrets

Set once per environment via wrangler secret put <NAME>:

SecretPurpose
PUB_ADMIN_TOKENAdmin API auth token (also gates Short.io QR sync). Renamed from TENDER_QR_SYNC_TOKEN; the worker reads PUB_ADMIN_TOKEN.
NOW_PLAYING_INGEST_TOKENBearer token for Pi song-ID pipeline
CF_ACCESS_POLICY_API_TOKENUsed by the Access page to update the dedicated delegated-operator policy

Optional secrets

SecretPurpose
SHORT_IO_API_KEYShort.io API for tender QR code short links
NOTION_API_KEYRetained only while a remaining Pub-owned content entity reads from Notion. Not an events credential
ALERT_NOTIFY_TORecipient for TV-offline and warm-fridge alerts. Personally identifying, so it is a secret rather than a var. Alerting is inert until it is set
Provider and relay credentialsOptional integrations for Short.io, email, Q-SYS, lighting, and TV control

Machine-integration secrets

SecretPurpose
CALENDAR_PUB_TV_READ_CREDENTIALReads the Calendar Pub TV event projection. Calendar holds the same value as PUB_TV_CALENDAR_READ_CREDENTIAL
Closing-ingest bearerAuthenticates the WhatsApp ingest Worker posting the nightly closing report
HOUSE_SYSTEMS_SUMMARY_CREDENTIALLegacy bounded snapshot only while the final House Systems consumer is audited and retired

Each is dedicated. Reusing an admin, ingest, email, or Access-policy credential in another lane is refused rather than tolerated.

Pub deploys through Cloudflare's Git integration

Configure these in the Worker dashboard. Forcing a CLI deploy to set a secret works against the release model.

Non-secret vars in wrangler.toml

VarPurpose
TV_BUILD_IDBump on every user-facing change; triggers TV self-reload
PUBLIC_SITE_URLhttps://pub.ihnyc-rc.org
CF_ACCESS_TEAM_DOMAIN, CF_ACCESS_AUDJWT issuer and audience used for every delegated Pub operator
Access policy identifiersDedicated delegated-operator policy bridge; public identifiers, never the API token
CALENDAR_PUB_TV_BASE_URLCalendar origin serving the Pub TV event projection
CALENDAR_PUB_TV_ENVIRONMENTExactly local, staging, or production; must match Calendar
TV_STATE_PUSH_ENABLEDCanary switch for the TV-state WebSocket push. false rejects upgrades and returns clients to the legacy cadence
TAKEOVER_VIDEO_ENABLEDEmergency decoder kill switch for takeover video. Images continue when false
TV_QSYS_NOTICE_ENABLEDWhether the bar TVs show the “systems reloading” strip while the Q-SYS Core is unreachable
TV_PAIRING_STRICT, TV_PAIRING_AUTO_MIGRATETV pairing behavior. Moved from secrets to vars so they are auditable and survive deploys
RESEND_FROMVerified sender for alert email
NOTION_ANNOUNCEMENTS_DATABASE_ID, NOTION_PUB_TENDERS_DATABASE_IDVestigial; these entities read from D1 regardless
SHORT_IO_DOMAINDomain for Short.io QR links
SETLIST_LEAD_SURFACE_ENABLEDFeature flag for Top Songs surface
SETLIST_VIBE_COMPASS_ENABLEDFeature flag for Vibe Compass
SETLIST_PERSONA_ENABLEDFeature flag for Persona AI surface
SETLIST_STORIES_ENABLEDFeature flag for Stories AI surface
SPORTS_ENABLEDLive sports schedule (keyless ESPN) → /api/sports/live + cron refresh of the sports KV cache
NOW_SHOWING_ENABLED”Now Showing” fusion (/api/now-showing) — fuses the Pi content verdict with the ESPN schedule to name a live game
SCREENINGS_ENABLEDMovie-night auto “Now Showing” + /screen archive. Values: off | shadow | on
FEEDBACK_ENABLEDPatron feedback / suggestion box (/feedback + /api/feedback); admin read at /admin/feedback is always on
PUB_SCHEDULER_V2Pub-scheduler run state-machine engine. Dormant — unset until the v2 cutover; flipping it activates the new transition engine

Bindings in wrangler.toml

BindingTypePurpose
CONTENT_DBD1Pub scheduler data (tenders, shifts, assignments, submissions, audit)
SETLIST_DBD1Song-ID data, setlist analytics, theme presets, API health
IMAGE_MIRROR_BUCKETR2Image mirror for TV/announcement/tender images
EVENT_MEDIAR2Event and takeover slideshow media
DJ_VISUALSR2DJ Mode visual media
TUTORIALS_DATAKVAdmin-edited tutorial content blobs
DJ_MODE, DJ_FLASH, LYRICSKVDJ Mode state, flash queue, and cached lyrics
CONTENT_VERSIONKVPer-entity content version keys read by the version endpoints
TV_PAIRINGDurable ObjectTvPairingCoordinator for slot state and the authoritative TV state push
SCAN_ANALYTICSAnalytics EngineTender QR scan analytics. Not provided by wrangler dev; writes are null-guarded
MEDIA, IMAGESStream, ImagesVisual media processing
AIWorkers AIPersona/Stories AI inference and the parked stock interpreter

D1 migrations

Migrations are plain SQL files applied manually. Two databases:

CONTENT_DB (pub scheduler)

wrangler d1 execute CONTENT_DB --remote --file=./migrations/content/<file>.sql

Recent additions cover the closing-report subsystem and TV takeovers: closing reports (0041), stock catalog expansion (0042), stock addendum receipts (0054), stock report dismissals (0055), closing export photo dedupe (0056), stock groups (0057), closing messages and their media (0058 and 0059), Resend webhook events (0060), TV takeovers (0061), and takeover event scheduling (0062). Earlier ones add email-log attachments (0052), the fairness-report setting (0053), tender holds (0040), and unsubscribes (0051).

SETLIST_DB (song-ID, analytics, theme, API health)

wrangler d1 execute SETLIST_DB --remote --file=./migrations/<file>.sql

Migrations 0053 through 0067 add quiet-hours state, TV control, scoped Pub roles, delegated Access sync, Q-SYS control discovery and curation, Spotify Jam scheduling, DMX command results, audio snapshots, the permission split (0063), touchscreen events (0064), House shadow evidence (0065), DJ flash media retention (0066), and DJ video requests (0067). Two different 0058 files exist; use the repository’s recorded migration order rather than assuming the number is unique.

Always apply migrations before deploying code that depends on new tables/columns. Check CLAUDE.md in the repo for the full ordered migration list.

Deploy

Production deploys automatically when a change lands on main via Cloudflare Workers Builds. Before merge, the GitHub Actions test check and Cloudflare preview build must both pass. GitHub Actions is test-only and has no deploy credentials; Cloudflare owns preview uploads and the production deploy.

Because main can go live immediately after merge, required D1 migrations must either be applied before merging or remain backward-compatible with both the old and new Worker. The canonical release, verification, and rollback procedure lives in the Pub repository’s docs/DEPLOYMENT.md.

Bump TV_BUILD_ID in wrangler.toml (in the same change) if TV-facing assets changed; TVs self-reload within 60 seconds when it changes.

A manual wrangler deploy is reserved for break-glass/one-off situations and must be recorded in the PR or incident — merging a green PR to main is the normal path.

Local development

# Create .dev.vars with local secrets
echo "PUB_ADMIN_TOKEN=<token>" > .dev.vars
 
# Start local dev server
wrangler dev

Local dev serves at http://localhost:8787. Workers AI (env.AI) is not available in local mode; use wrangler dev --remote for AI surface testing (bills Neurons).

Monitoring

SignalWhere to look
Worker errorsCloudflare Dashboard > Workers & Pages > Logs
TV health/admin/tv-health — live heartbeat, build match, memory, JS errors per screen
API dependencies/admin/api-health — Notion, LRCLib, Short.io, Deezer, Formbricks, Workers AI, ESPN (keyless live-sports scoreboard), TMDB (movie posters, keyless Deezer fallback)
Scheduler state/admin/pub-schedule — current week’s wizard step and submission feed
Email delivery/admin/pub-setup > Email Log
Song-ID pipelinessh pi@<pi-tailnet-hostname> 'systemctl is-active ptp4l-eth0 aes67-daemon pi-capture'
Setlist rollupsCheck /api/setlist/<surface> for fresh computedAt timestamps

TV build version

TV screens self-reload when TV_BUILD_ID changes. Bump it on every deploy that touches public/tv/, public/bar-tv-right/, or any TV-facing asset:

TV_BUILD_ID = "YYYY-MM-DD-NN"   # e.g. 2026-06-07-01

The worker injects this into TV HTML as window.__TV_BUILD_ID__. TVs poll /api/tv-build every 60 seconds and reload when the value changes.

Cloudflare Access setup

The admin UI is gated by a Pub-specific Cloudflare Access application. Its audience remains stable when Authentik is added upstream; Authentik supplies verified identity through Access and does not replace Pub roles.

  • Full administrators remain on the standing full-admin policy and hold *.
  • Delegated operators are kept in a dedicated email-only policy managed from /admin/access.
  • The Worker verifies the Access JWT, then reads scoped permissions from pub_admin_roles. The current catalog has 88 exact keys at version 4.
  • Removing an operator revokes Pub permissions and then removes the delegated Access entry.

Add a delegated operator in /admin/access, not in Authentik groups, House Systems, Resident Council tooling, or by hand-editing the main Access application policy. The bridge touches only the configured email-only Pub delegated operators policy. If policy and D1 role synchronization diverge, the Access page shows the pending retry.

Troubleshooting

TVs show “Some TVs need attention”

The hub shows amber only for genuinely missing TVs (no heartbeat for 30+ minutes). If a TV is borderline stale (5-30 min), the full TV Health page shows details. Common causes:

  • TV went to sleep or lost network
  • Cloudflare Workers had a brief outage
  • TV browser crashed (check JS errors on the health card)

If the slot is an intentionally parked QA device, open /admin/slots and turn off Monitor for that slot. Its real state remains visible, but it no longer contributes to warning counts or offline-alert email.

Scheduler email not sending

  1. Check the This Week wizard — is Step 4 (Send) active?
  2. Confirm all shifts are Approved (Draft shifts block sending)
  3. Check /admin/pub-setup > Email Log for send failures
  4. Verify PUB_ADMIN_TOKEN is set as a Cloudflare secret

Setlist surfaces show stale data

The cron dispatcher picks ONE due surface per tick. If a surface hasn’t updated:

  1. Check its surface_config.enabled = 1 in SETLIST_DB
  2. Check its cadence (e.g., Persona/Stories run weekly, Top Songs runs hourly)
  3. Check /admin/api-health — Deezer degraded blocks Vibe Compass enrichment

Tutorial edits not saving

  1. Verify TUTORIALS_DATA KV namespace is bound in wrangler.toml
  2. Check the Access session and the operator’s tutorials.edit permission
  3. Preview a history version before restoring it
  4. Check browser console for 401/403/503 errors

Audio controls are missing or locked

  1. Confirm the Pi and Q-SYS Core are reachable on the System page.
  2. Refresh discovery on /admin/audio.
  3. Confirm the control is writable and included in an operator group.
  4. Check the group and control lock state.
  5. Capture a snapshot before changing several controls.

A sound-stage mode button is refused

The quiet curfew holds loud modes back. The refusal names the curfew window and says which setting to change: if the pub is open later than the curfew assumes, adjust tonight’s window in Quiet Mode rather than fighting the button. Blocked attempts and overrides are both recorded in the audit log with their reason.

Events are missing from the bar TV

Events come from the Calendar service now, not Notion.

  1. Confirm the event is published in the Calendar admin and carries the Pub display flag. See Event review and publication.
  2. Confirm its location contains the word Pub; that filter is unchanged.
  3. Check whether the projection response was rejected. The consumer refuses a whole response on any validation failure and serves the last known good payload, so stale-but-correct events are the expected symptom rather than an empty board.
  4. Confirm CALENDAR_PUB_TV_ENVIRONMENT matches Calendar’s value and the read credential is present.

A closing report did not appear

  1. Check whether the message was held rather than parsed. A message the parser cannot read is retained for review and an operator can overrule the classifier.
  2. Confirm the night. A report filed after midnight is credited to the previous business day, with a 06:00 venue-time cutoff.
  3. Confirm the night actually had a closing shift. Nights that owed no report are not flagged.
  4. If a warm-fridge alert did not arrive, confirm ALERT_NOTIFY_TO is set. Alerting is inert without it.

A takeover will not start or stop

  1. A start is refused while the media list has no ready and enabled item.
  2. Activation is a compare-and-set. A stale revision means someone else edited the takeover; reload and retry.
  3. An active or scheduled takeover cannot be edited or deleted. Stop it first.
  4. Stop is idempotent and unconfirmed, so clicking it twice is safe.
  5. If video is not playing but images are, check TAKEOVER_VIDEO_ENABLED.

Song-ID pipeline (Pi-side)

The pipeline runs on a Raspberry Pi reached through its approved private tailnet hostname. Three systemd services:

ServiceWhat
ptp4l-eth0.servicePTP slave syncing to Q-SYS Core
aes67-daemon.serviceRAVENNA kernel module + ALSA capture
pi-capture.serviceFingerprint + POST to worker

All enabled on boot with dependency ordering. See pi-capture-ops for the full runbook.