Re-architect: API-based acquisition, no download; ffmpeg -ss screenshots

- Comments via YouTube Data API v3 (key in UI or YOUTUBE_API_KEY secret) instead of yt-dlp
scraping; sentiment ranking unchanged
- Transcript via youtube-transcript-api (already timestamped) instead of faster-whisper
- Screenshots without downloading: yt-dlp resolves a stream URL, ffmpeg -ss grabs one
frame per weighted timestamp (sharpest of 3); falls back to text-only if blocked
- Remove the whole YouTube-access section (cookies/proxy/PO-token UI) and the browser
extension; thin fallback kept via optional YT_COOKIES / YT_PROXY secrets
- Drop faster-whisper + full video download; add youtube-transcript-api + numpy
- gitignore cookies.txt; README rewritten for the new architecture

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

.gitignore

@@ -12,3 +12,5 @@ venv/
 runs/
 .gradio/
 .DS_Store
+cookies.txt
+*.cookies.txt

README.md

@@ -14,112 +14,50 @@ short_description: Make Tutorials from YouTube Videos
 
 # YouTube → Tutorial Post Generator
 
-Give a **topic** and your **Hugging Face token**, and this Space builds a downloadable
-**`.docx` tutorial** (text + AI-captioned screenshots) from the single best YouTube video
-on that topic.
+Give a **topic**, your **Hugging Face token**, and a **YouTube Data API key**, and this
+Space builds a downloadable **`.docx` tutorial** (text + captioned screenshots) from the
+single best YouTube video on that topic. **No video download** — acquisition is API-based.
 
 ## Pipeline
 
 1. **Search** — top 5 videos via the [`adarshajay/youtube-search`](https://huggingface.co/spaces/adarshajay/youtube-search) Space.
-2. **Sentiment rank** — fetch top comments per video with `yt-dlp` and score them with the
-   BERT classifier [`OmarMedhat7/youtube-sentiment-analysis-model`](https://huggingface.co/OmarMedhat7/youtube-sentiment-analysis-model);
-   the video with the highest positive share wins.
-3. **Download** the winner with `yt-dlp` and extract audio with `ffmpeg`.
-4. **Transcribe** locally with `faster-whisper` (segment timestamps).
-5. **Candidate frames** are extracted densely from the video, then the **video is
-   auto-deleted** — only small JPEGs and the transcript remain.
-6. **Tutorial text** — `deepseek-ai/DeepSeek-V3` (HF Inference Providers, billed to your
-   token) turns the transcript into an **answer-engine-optimized** post: an answer-first
-   paragraph, H2 step headings, an FAQ, a meta description, a URL slug, and a
-   last-updated/source citation. Optional **primary/secondary keyword** placement (primary
-   ~3× + in title/slug/meta/intro/H2; each secondary once).
-7. **Screenshot selection** — a weighted indicator blends each step's LLM-suggested
-   timestamp with Whisper's actual speech timing.
-8. **Captions** — a vision model (default `Qwen/Qwen2.5-VL-72B-Instruct`, billed to your
+2. **Sentiment rank** — fetch each video's comments via the **YouTube Data API v3** and
+   score them with the BERT classifier
+   [`OmarMedhat7/youtube-sentiment-analysis-model`](https://huggingface.co/OmarMedhat7/youtube-sentiment-analysis-model);
+   the highest positive share wins.
+3. **Transcript** — fetched with **`youtube-transcript-api`** (already timestamped; no
+   download, no Whisper).
+4. **Tutorial text** — `deepseek-ai/DeepSeek-V3` (HF Inference Providers, billed to your
+   token) turns the transcript into an **answer-engine-optimized** post: answer-first
+   paragraph, H2 steps, FAQ, meta description, URL slug, last-updated/source citation.
+   Optional **primary/secondary keyword** placement.
+5. **Screenshots** — real frames at the right moments **without downloading the video**:
+   `yt-dlp` resolves a direct stream URL (metadata only), then `ffmpeg -ss T -frames:v 1`
+   grabs one frame per timestamp (sharpest of 3 candidates). Timestamps come from a
+   weighted blend of the LLM's suggestion and the transcript's actual timing.
+6. **Captions** — a vision model (default `Qwen/Qwen2.5-VL-72B-Instruct`, billed to your
    token) captions each screenshot.
-9. **Assemble** the `.docx` for download.
+7. **Assemble** the `.docx` for download.
 
-## Notes
+## Keys & setup
 
-- **Your HF token is used only for the LLM and vision-model calls** and is billed to your
-  account. Create a fine-grained token with *"Make calls to Inference Providers"* at
+- **Hugging Face token** — for the LLM + vision-model calls, billed to your account.
+  Create a fine-grained token with *"Make calls to Inference Providers"* at
   <https://huggingface.co/settings/tokens>.
-- **Free CPU tier:** Whisper runs on CPU, so transcription is slow — keep videos short
-  (default cap ~20 min).
-- **YouTube usually blocks the Space's datacenter IP.** Each visitor can supply their own
-  access in the **"YouTube access — cookies / proxy"** panel (used only for that run, then
-  deleted):
-  - **Cookies:** a `youtube.com` cookies.txt (Netscape format, raw or base64) exported
-    from a **throwaway** Google account.
-  - **Proxy:** a **residential** proxy URL. **Free *datacenter* proxies (e.g. Webshare's
-    free tier) usually do *not* bypass YouTube's block** and have tight bandwidth caps.
-  - An operator can set shared defaults via the `YT_COOKIES` / `YT_PROXY` Space secrets.
-  - A hosted Space **cannot** open or read a visitor's browser — auth must be supplied
-    explicitly.
+- **YouTube Data API key** — for fetching comments. Create one in the
+  [Google Cloud Console](https://console.cloud.google.com/) and **enable *YouTube Data API
+  v3***. Provide it in the UI, or set the **`YOUTUBE_API_KEY`** Space secret. (Without a
+  key the Space skips sentiment and just uses the top search result.)
 
-## PO token guide (free, no proxy)
+## Notes on YouTube access
 
-YouTube increasingly requires a **Proof-of-Origin (PO) token** to serve video from
-datacenter IPs. A matching **PO token + visitor data** pair can get the Space past the
-"Sign in to confirm you're not a bot" wall **without any proxy** — and it's free. It's
-fiddly to extract by hand and the tokens expire (hours), so re-do this when downloads
-start failing.
+The transcript and the stream-URL resolution hit YouTube directly. From a datacenter IP
+(like a Space) these can occasionally be **blocked**:
 
-Paste the values into the app's **"YouTube access — cookies / proxy"** panel:
-**PO token** field and **Visitor data** field.
-
-### Easiest: the PO Token Grabber extension (one click)
-
-This repo ships a tiny Chrome/Edge extension in [`extension/`](extension/) that reads the
-PO token + visitor data for you. Grab it with the **⬇️ Download the PO Token Grabber
-extension (.zip)** button in the app's *YouTube access* panel (or from
-[`extension/`](extension/)), unzip it, and load it unpacked (`chrome://extensions` →
-Developer mode → Load unpacked). Open a YouTube video and press play, then click the
-extension **while on the YouTube tab** to capture the values. Open this Space in another
-tab and hit **⤵ Fill the Space tab** to inject them straight into the fields below (or use
-the Copy buttons and paste). See
-[`extension/README.md`](extension/README.md). (It's unpublished/sideloaded and may need
-updates when YouTube changes; the manual steps below always work as a fallback.)
-
-### Get them manually (web client, ~2 min)
-
-1. Open a fresh **Incognito** window and go to <https://www.youtube.com> (a throwaway
-   logged-in account is fine; if you log in, also export that session's cookies).
-2. Open **DevTools → Network** (F12), then **play any video**.
-3. **Visitor data:** filter Network for `v1/player`, open the request to
-   `youtubei/v1/player`, and in the **Request payload** copy
-   `context.client.visitorData` (a string ending in `%3D%3D`). Paste it into the
-   **Visitor data** field.
-4. **PO token (gvs):** filter Network for `googlevideo.com`, open a `videoplayback`
-   request, and copy the value of its **`pot`** query parameter. In the **PO token**
-   field enter it prefixed with the client + context:
-
-   ```
-   web.gvs+<pot-value>
-   ```
-
-   You can supply more than one, comma-separated, e.g.
-   `web.gvs+<gvs-pot>, web.player+<player-pot>` (the player PO token comes from the
-   `v1/player` request's response, field `...poToken`).
-5. Run the Space. When a PO token is present it automatically uses the **web** player
-   client so the token applies.
-
-**Tips**
-- The PO token is bound to the **visitor data** (and cookies) it was made with — always
-  submit them together, from the same session.
-- Prefer using the **same Incognito session** for cookies + visitor data + PO token.
-- Tokens are short-lived; refresh when you next get blocked.
-
-### Automated / operator option
-
-yt-dlp can auto-generate PO tokens via the
-[`bgutil-ytdlp-pot-provider`](https://github.com/Brainicism/bgutil-ytdlp-pot-provider)
-plugin, but that needs a small provider server running alongside yt-dlp — not bundled in
-this hosted Space. A self-hosting operator can run it and set shared defaults via the
-`YT_POT` / `YT_VISITOR_DATA` Space secrets.
-
-See the authoritative, up-to-date method in the
-[yt-dlp PO Token Guide](https://github.com/yt-dlp/yt-dlp/wiki/PO-Token-Guide).
+- If transcript/stream resolution fails, set the optional **`YT_PROXY`** secret (a
+  residential proxy) and/or **`YT_COOKIES`** (Netscape cookies.txt contents, raw or
+  base64) — used only as a thin fallback.
+- If the stream URL can't be resolved, the Space still produces a **text-only** tutorial.
 
 ## Local run
 

app.py

@@ -1,8 +1,13 @@
-"""Gradio Space: YouTube topic -> captioned .docx tutorial.
+"""Gradio Space: YouTube topic -> captioned .docx tutorial (API-based acquisition).
 
-Orchestrates the pipeline stages and streams progress/status to the UI. Heavy ML
-imports (torch/transformers/faster-whisper) are lazy inside the pipeline modules, so app
-startup stays fast.
+Pipeline:
+  search top videos -> rank by YouTube Data API comment sentiment -> transcript via
+  youtube-transcript-api -> DeepSeek-V3 tutorial -> real screenshots via yt-dlp stream URL
+  + ffmpeg -ss (weighted timestamps) -> VLM captions -> .docx.
+
+No video download and no cookies/proxy/PO-token UI. A thin fallback remains via the
+optional Space secrets YT_COOKIES / YT_PROXY (used only to resolve the stream URL and the
+transcript when the Space's datacenter IP is blocked).
 """
 from __future__ import annotations
 
@@ -18,7 +23,6 @@ import gradio as gr
 from pipeline import (
     captions as captions_mod,
     docx_builder,
-    download as download_mod,
     frames as frames_mod,
     search as search_mod,
     sentiment as sentiment_mod,
@@ -26,26 +30,6 @@ from pipeline import (
     tutorial as tutorial_mod,
 )
 
-APP_DIR = os.path.dirname(os.path.abspath(__file__))
-EXTENSION_DIR = os.path.join(APP_DIR, "extension")
-
-
-def _build_extension_zip() -> str | None:
-    """Zip the bundled browser extension for one-click download; return the .zip path.
-
-    The extension's files (manifest.json, etc.) sit at the **root** of the archive, so the
-    unzipped folder directly contains manifest.json and can be selected as-is in
-    "Load unpacked". Returns None if the folder isn't present.
-    """
-    if not os.path.isdir(EXTENSION_DIR):
-        return None
-    out_base = os.path.join(tempfile.gettempdir(), "tutorialmaker-extension")
-    try:
-        return shutil.make_archive(out_base, "zip", root_dir=EXTENSION_DIR)
-    except Exception:
-        return None
-
-
 LLM_CHOICES = [
     "deepseek-ai/DeepSeek-V3",
     "meta-llama/Llama-3.3-70B-Instruct",
@@ -58,18 +42,13 @@ VLM_CHOICES = [
 ]
 
 
+# --------------------------------------------------------------- thin access fallback
 def _looks_like_netscape(text: str) -> bool:
-    """True if ``text`` is already a tab-separated Netscape cookie file."""
     head = text.lstrip()
     return head.startswith("#") or "\tTRUE\t" in text or "\tFALSE\t" in text
 
 
 def _maybe_b64_decode(text: str) -> str | None:
-    """If ``text`` is base64 that decodes to a cookie file, return the decoded text.
-
-    HF secret fields can turn the required TAB characters into spaces, which breaks the
-    Netscape format. Pasting base64 of the file avoids that; we auto-detect and decode it.
-    """
     compact = "".join(text.split())
     if len(compact) < 16 or re.search(r"[^A-Za-z0-9+/=]", compact):
         return None
@@ -80,53 +59,43 @@ def _maybe_b64_decode(text: str) -> str | None:
     return decoded if _looks_like_netscape(decoded) else None
 
 
-def _cookiefile(workdir: str, raw: str | None = None) -> str | None:
-    """Materialize cookies to a Netscape cookie file on disk; return its path or None.
-
-    ``raw`` is the per-user UI value; if empty we fall back to the operator-wide
-    ``YT_COOKIES`` secret. Accepts either raw cookies.txt contents (tabs preserved) or a
-    base64 encoding of them. A missing header line is added so yt-dlp accepts the file.
-    """
-    data = (raw or "").strip() or os.environ.get("YT_COOKIES")
+def _cookiefile(workdir: str) -> str | None:
+    """Materialize the optional YT_COOKIES secret to a Netscape file; return path or None."""
+    data = os.environ.get("YT_COOKIES")
     if not data or not data.strip():
         return None
-
     if not _looks_like_netscape(data):
         decoded = _maybe_b64_decode(data)
         if decoded:
             data = decoded
-
     if not data.lstrip().startswith(("# Netscape", "# HTTP")):
         data = "# Netscape HTTP Cookie File\n" + data.lstrip("\n")
     if not data.endswith("\n"):
         data += "\n"
-
     path = os.path.join(workdir, "cookies.txt")
     with open(path, "w", encoding="utf-8", newline="\n") as fh:
         fh.write(data)
     return path
 
 
-def _resolve_proxy(raw: str | None = None) -> str | None:
-    """Per-user proxy URL, falling back to the operator-wide YT_PROXY secret."""
-    proxy = (raw or "").strip() or os.environ.get("YT_PROXY", "").strip()
+def _resolve_proxy() -> str | None:
+    proxy = os.environ.get("YT_PROXY", "").strip()
     return proxy or None
 
 
-def _resolve_pot(po_token: str | None, visitor_data: str | None) -> tuple[str | None, str | None]:
-    """Per-user PO token + visitor data, falling back to YT_POT / YT_VISITOR_DATA secrets."""
-    pot = (po_token or "").strip() or os.environ.get("YT_POT", "").strip()
-    vis = (visitor_data or "").strip() or os.environ.get("YT_VISITOR_DATA", "").strip()
-    return (pot or None), (vis or None)
+def _resolve_api_key(ui_key: str | None) -> str | None:
+    key = (ui_key or "").strip() or os.environ.get("YOUTUBE_API_KEY", "").strip()
+    return key or None
 
 
+# ----------------------------------------------------------------------------- helpers
 def _ranking_rows(scored: list[dict]) -> list[list]:
     rows = []
     for rank, v in enumerate(scored, start=1):
         rows.append([
             rank,
             v.get("title", v["video_id"]),
-            f"{v['positive_share'] * 100:.0f}%",
+            f"{v.get('positive_share', 0) * 100:.0f}%",
             v.get("n_comments", 0),
             v.get("note", "") or "ok",
             v["url"],
@@ -139,13 +108,8 @@ def _safe_name(text: str) -> str:
 
 
 def _collect_keywords(primary_kw, secondary_kw) -> dict:
-    """Build ``{"primary": str, "secondary": [str, ...]}`` from the two keyword inputs.
-
-    Secondary keywords are comma-separated. Duplicates and the primary are removed.
-    """
     primary = (primary_kw or "").strip()
-    secondary = []
-    seen = {primary.lower()}
+    secondary, seen = [], {primary.lower()}
     for part in (secondary_kw or "").split(","):
         kw = part.strip()
         if kw and kw.lower() not in seen:
@@ -154,11 +118,10 @@ def _collect_keywords(primary_kw, secondary_kw) -> dict:
     return {"primary": primary, "secondary": secondary}
 
 
-def run_pipeline(topic, hf_token, llm_model, vlm_model, w_llm, w_whisper, lead,
-                 max_minutes, max_shots, primary_kw, secondary_kw,
-                 cookies_text, proxy_url, po_token_in, visitor_data_in,
+def run_pipeline(topic, hf_token, yt_api_key, llm_model, vlm_model,
+                 w_llm, w_whisper, lead, max_shots, primary_kw, secondary_kw,
                  progress=gr.Progress()):
-    """Generator that yields (status_md, ranking_df, transcript, docx_file)."""
+    """Generator yielding (status_md, ranking_df, transcript, docx_file)."""
     log: list[str] = []
 
     def status(msg: str):
@@ -173,62 +136,53 @@ def run_pipeline(topic, hf_token, llm_model, vlm_model, w_llm, w_whisper, lead,
 
     workdir = tempfile.mkdtemp(prefix="ytt_")
     frames_dir = os.path.join(workdir, "frames")
-    video_path = None
     try:
-        cookiefile = _cookiefile(workdir, cookies_text)
-        proxy = _resolve_proxy(proxy_url)
-        po_token, visitor_data = _resolve_pot(po_token_in, visitor_data_in)
-        auth_bits = []
-        if cookiefile:
-            auth_bits.append("cookies")
-        if proxy:
-            auth_bits.append("proxy")
-        if po_token:
-            auth_bits.append("PO token")
-        if auth_bits:
-            yield status("🔐 Using " + " + ".join(auth_bits) + " for YouTube access."), gr.update(), gr.update(), gr.update()
+        api_key = _resolve_api_key(yt_api_key)
+        cookiefile = _cookiefile(workdir)
+        proxy = _resolve_proxy()
 
         # 1. Search ------------------------------------------------------------------
-        progress(0.02, desc="Searching")
+        progress(0.03, desc="Searching")
         yield status(f"🔍 Searching top videos for **{topic}**…"), gr.update(), gr.update(), gr.update()
         videos = search_mod.search_top5(topic)
         yield status(f"Found {len(videos)} candidate videos."), gr.update(), gr.update(), gr.update()
 
-        # 2. Sentiment ranking -------------------------------------------------------
-        yield status("💬 Fetching comments and scoring sentiment…"), gr.update(), gr.update(), gr.update()
-        best, scored = sentiment_mod.rank_by_sentiment(
-            videos, cookiefile, progress, proxy, po_token, visitor_data)
+        # 2. Sentiment ranking (YouTube Data API comments) ---------------------------
+        if api_key:
+            yield status("💬 Fetching comments (YouTube Data API) and scoring sentiment…"), gr.update(), gr.update(), gr.update()
+            best, scored = sentiment_mod.rank_by_sentiment(videos, api_key, progress)
+            picked_msg = f"({best['positive_share'] * 100:.0f}% positive comments)"
+        else:
+            best = videos[0]
+            scored = [{**v, "positive_share": 0.0, "n_comments": 0,
+                       "note": "sentiment skipped (no API key)", "search_rank": i}
+                      for i, v in enumerate(videos)]
+            picked_msg = "(no YouTube Data API key → used top search result)"
         ranking = gr.update(value=_ranking_rows(scored))
-        yield (status(f"🏆 Picked **{best.get('title', best['video_id'])}** "
-                      f"({best['positive_share'] * 100:.0f}% positive)."),
+        yield (status(f"🏆 Picked **{best.get('title', best['video_id'])}** {picked_msg}."),
                ranking, gr.update(), gr.update())
 
-        # 3. Download + audio --------------------------------------------------------
-        progress(0.25, desc="Downloading")
-        yield status("⬇️ Downloading the chosen video…"), ranking, gr.update(), gr.update()
-        video_path, duration = download_mod.download_video(
-            best["url"], workdir, cookiefile, int(max_minutes), proxy, po_token, visitor_data)
-        wav = download_mod.extract_audio(video_path, workdir)
-
-        # 4. Transcribe --------------------------------------------------------------
-        progress(0.4, desc="Transcribing")
-        yield status("📝 Transcribing with Whisper (this is the slow part on CPU)…"), ranking, gr.update(), gr.update()
-        segs = transcribe_mod.transcribe(wav, progress)
+        # 3. Transcript (youtube-transcript-api) -------------------------------------
+        progress(0.3, desc="Transcript")
+        yield status("📝 Fetching the timestamped transcript…"), ranking, gr.update(), gr.update()
+        segs = transcribe_mod.get_segments(best["video_id"], proxy=proxy)
         transcript = transcribe_mod.transcript_text(segs)
         yield (status(f"Transcript ready ({len(segs)} segments)."),
                ranking, gr.update(value=transcript), gr.update())
 
-        # 5. Candidate frames, then DELETE the video --------------------------------
-        progress(0.6, desc="Extracting frames")
-        candidates = frames_mod.extract_candidates(video_path, frames_dir, duration)
-        frames_mod.delete_video(video_path)
-        video_path = None
-        yield (status(f"🎞️ Extracted {len(candidates)} candidate frames and "
-                      f"**deleted the downloaded video**."),
-               ranking, gr.update(value=transcript), gr.update())
+        # 4. Resolve a stream URL for screenshots (best-effort) ----------------------
+        progress(0.45, desc="Resolving stream")
+        stream_url = None
+        try:
+            stream_url, _ = frames_mod.get_stream_url(best["video_id"], cookiefile, proxy)
+        except Exception as exc:
+            yield (status(f"⚠️ Couldn't resolve a video stream for screenshots "
+                          f"({type(exc).__name__}) — producing a **text-only** tutorial. "
+                          "Set YT_PROXY/YT_COOKIES secrets to enable screenshots."),
+                   ranking, gr.update(value=transcript), gr.update())
 
-        # 6. Tutorial text -----------------------------------------------------------
-        progress(0.72, desc="Writing tutorial")
+        # 5. Tutorial text -----------------------------------------------------------
+        progress(0.6, desc="Writing tutorial")
         keywords = _collect_keywords(primary_kw, secondary_kw)
         kw_note = f" • primary: '{keywords['primary']}'" if keywords["primary"] else ""
         if keywords["secondary"]:
@@ -240,119 +194,77 @@ def run_pipeline(topic, hf_token, llm_model, vlm_model, w_llm, w_whisper, lead,
             yield (status(f"🔑 Primary keyword '{keywords['primary']}' appears {n}× in the post."),
                    ranking, gr.update(value=transcript), gr.update())
 
-        # 7. Weighted screenshot selection ------------------------------------------
-        selected = frames_mod.select_screenshots(
-            tut["steps"], segs, candidates,
-            w_llm=float(w_llm), w_whisper=float(w_whisper), lead=float(lead),
-            max_shots=int(max_shots),
-        )
-        yield (status(f"🖼️ Selected {len(selected)} screenshots via the weighted indicator."),
-               ranking, gr.update(value=transcript), gr.update())
-
-        # 8. Captions ----------------------------------------------------------------
-        progress(0.85, desc="Captioning")
-        yield status(f"✍️ Captioning screenshots with `{vlm_model}`…"), ranking, gr.update(value=transcript), gr.update()
-        caps = captions_mod.caption_frames(selected, tut["steps"], hf_token.strip(), vlm_model, progress)
-
-        # 9. DOCX --------------------------------------------------------------------
-        progress(0.95, desc="Building document")
+        # 6. Screenshots (weighted timestamps -> ffmpeg grab) + captions -------------
+        selected, caps = {}, {}
+        if stream_url:
+            progress(0.75, desc="Screenshots")
+            times = frames_mod.compute_shot_times(
+                tut["steps"], segs, w_llm=float(w_llm), w_whisper=float(w_whisper),
+                lead=float(lead), max_shots=int(max_shots))
+            yield status(f"🎞️ Capturing {len(times)} screenshots at weighted timestamps…"), ranking, gr.update(value=transcript), gr.update()
+            selected = frames_mod.capture_shots(times, stream_url, frames_dir, proxy, progress)
+            if selected:
+                progress(0.88, desc="Captioning")
+                yield status(f"✍️ Captioning {len(selected)} screenshots with `{vlm_model}`…"), ranking, gr.update(value=transcript), gr.update()
+                caps = captions_mod.caption_frames(selected, tut["steps"], hf_token.strip(), vlm_model, progress)
+
+        # 7. DOCX --------------------------------------------------------------------
+        progress(0.96, desc="Building document")
         out_path = os.path.join(workdir, f"{_safe_name(tut['title'])}.docx")
         docx_builder.build_docx(tut, selected, caps, out_path, source_url=best["url"])
 
         progress(1.0, desc="Done")
-        yield (status("✅ Done! Download your tutorial below."),
+        shots_msg = f"{len(selected)} screenshots" if selected else "text-only"
+        yield (status(f"✅ Done ({shots_msg}). Download your tutorial below."),
                ranking, gr.update(value=transcript), gr.update(value=out_path))
 
     except gr.Error:
         raise
-    except (download_mod.DownloadError, RuntimeError, ValueError) as exc:
+    except (transcribe_mod.TranscriptError, sentiment_mod.SentimentError,
+            RuntimeError, ValueError) as exc:
         raise gr.Error(str(exc))
-    finally:
-        # Always remove the video if it somehow survived; keep frames/docx until the
-        # response is sent (Gradio copies the returned file out).
-        if video_path:
-            frames_mod.delete_video(video_path)
 
 
 def build_ui():
     with gr.Blocks(title="YouTube → Tutorial Post") as demo:
         gr.Markdown(
             "# 📝 YouTube → Tutorial Post Generator\n"
-            "Enter a topic and your Hugging Face token. The Space picks the best video, "
-            "transcribes it, and builds a **captioned `.docx` tutorial**. Your token is "
-            "used only for the LLM + vision-model calls and **billed to your account**."
+            "Enter a topic, your **Hugging Face token** (LLM + vision model, billed to you) "
+            "and a **YouTube Data API key** (for comments). The Space picks the best video by "
+            "comment sentiment, pulls its transcript, writes an AEO-friendly tutorial, grabs "
+            "real screenshots at the right moments, and builds a **.docx**."
         )
         with gr.Row():
             with gr.Column(scale=2):
                 topic = gr.Textbox(label="Topic", placeholder="e.g. Excel pivot tables for beginners")
                 hf_token = gr.Textbox(label="Hugging Face token", type="password",
                                       placeholder="hf_…  (Inference Providers permission)")
+                yt_api_key = gr.Textbox(label="YouTube Data API key", type="password",
+                                        placeholder="for comments (or set the YOUTUBE_API_KEY secret)")
             with gr.Column(scale=1):
                 llm_model = gr.Dropdown(LLM_CHOICES, value=LLM_CHOICES[0],
                                         label="Tutorial LLM", allow_custom_value=True)
                 vlm_model = gr.Dropdown(VLM_CHOICES, value=VLM_CHOICES[0],
                                         label="Vision model (captions)", allow_custom_value=True)
 
-        with gr.Accordion("YouTube access — cookies / proxy (often required)", open=False):
-            gr.Markdown(
-                "⚠️ **YouTube usually blocks the Space's datacenter IP.** To download, give "
-                "the Space **your own** access below — it is used only for your run and "
-                "deleted afterward.\n\n"
-                "- **Use a throwaway Google account, not your main one.** yt-dlp activity "
-                "can get an account rate-limited or flagged.\n"
-                "- **Cookies:** export a `youtube.com` cookies.txt (Netscape format) from a "
-                "logged-in throwaway account and paste it (raw or base64) below.\n"
-                "- **PO token (free, no proxy):** a Proof-of-Origin token + visitor data "
-                "can pass the bot-check from a datacenter IP. **See this Space's README → "
-                "\"PO token\" guide** for how to get and paste them.\n"
-                "- **Proxy:** a **residential** proxy works; **free *datacenter* proxies "
-                "(e.g. Webshare's free tier) usually do NOT** get past YouTube's block and "
-                "have tight bandwidth caps.\n"
-                "- An operator can instead set Space secrets `YT_COOKIES` / `YT_PROXY` / "
-                "`YT_POT` / `YT_VISITOR_DATA` as shared defaults."
-            )
-            _ext_zip = _build_extension_zip()
-            if _ext_zip:
-                gr.DownloadButton(
-                    "⬇️ Download the PO Token Grabber extension (.zip)",
-                    value=_ext_zip, size="sm")
-            cookies_text = gr.Textbox(
-                label="YouTube cookies (cookies.txt contents or base64)", lines=4,
-                placeholder="# Netscape HTTP Cookie File …  (or a base64 blob)")
-            with gr.Row():
-                po_token_in = gr.Textbox(
-                    label="PO token (see README) — CLIENT.CONTEXT+TOKEN", lines=2, scale=3,
-                    elem_id="tm_pot_token",
-                    placeholder="web.gvs+AbC…,  web.player+XyZ…")
-                visitor_data_in = gr.Textbox(
-                    label="Visitor data (pairs with the PO token)", scale=2,
-                    elem_id="tm_visitor_data",
-                    placeholder="Cgt...%3D%3D")
-            proxy_url = gr.Textbox(
-                label="Proxy URL (optional)", type="password",
-                placeholder="http://user:pass@host:port")
-
         with gr.Accordion("SEO / AEO keywords (optional)", open=False):
             gr.Markdown(
                 "The **primary keyword** is used naturally ~3× in the body and placed in "
-                "the title, URL slug, meta description, the first 100 words, and one or "
-                "two H2 headings. Each **secondary keyword** is used once. The post also "
-                "follows answer-engine best practices (direct answer up top, FAQ, "
-                "last-updated date, source citation)."
+                "the title, URL slug, meta description, the first 100 words, and one or two "
+                "H2 headings. Each **secondary keyword** is used once. The post also follows "
+                "answer-engine best practices (direct answer up top, FAQ, last-updated date, "
+                "source citation)."
             )
-            primary_kw = gr.Textbox(label="Primary keyword",
-                                    placeholder="e.g. godot ai plugin")
+            primary_kw = gr.Textbox(label="Primary keyword", placeholder="e.g. excel pivot tables")
             secondary_kw = gr.Textbox(label="Secondary keywords (comma-separated)",
-                                      placeholder="e.g. gdscript assistant, ai game tools")
+                                      placeholder="e.g. pivot chart, data summary")
 
         with gr.Accordion("Advanced settings", open=False):
             with gr.Row():
                 w_llm = gr.Slider(0.0, 1.0, value=0.4, step=0.05, label="Weight: LLM timestamp")
-                w_whisper = gr.Slider(0.0, 1.0, value=0.6, step=0.05, label="Weight: Whisper timing")
+                w_whisper = gr.Slider(0.0, 1.0, value=0.6, step=0.05, label="Weight: transcript timing")
                 lead = gr.Slider(0.0, 5.0, value=1.0, step=0.5, label="Lead offset (s)")
-            with gr.Row():
-                max_minutes = gr.Slider(2, 60, value=20, step=1, label="Max video length (min)")
-                max_shots = gr.Slider(1, 15, value=8, step=1, label="Max screenshots")
+            max_shots = gr.Slider(1, 15, value=8, step=1, label="Max screenshots")
 
         run_btn = gr.Button("Generate tutorial", variant="primary")
 
@@ -366,13 +278,12 @@ def build_ui():
 
         run_btn.click(
             run_pipeline,
-            inputs=[topic, hf_token, llm_model, vlm_model, w_llm, w_whisper, lead,
-                    max_minutes, max_shots, primary_kw, secondary_kw,
-                    cookies_text, proxy_url, po_token_in, visitor_data_in],
+            inputs=[topic, hf_token, yt_api_key, llm_model, vlm_model,
+                    w_llm, w_whisper, lead, max_shots, primary_kw, secondary_kw],
             outputs=[status_md, ranking_df, transcript_box, docx_file],
         )
     return demo
 
 
 if __name__ == "__main__":
-    build_ui().queue().launch(allowed_paths=[tempfile.gettempdir()])
+    build_ui().queue().launch()

extension/README.md

@@ -1,78 +0,0 @@
-# TutorialMaker — PO Token Grabber (browser extension)
-
-A tiny **Chrome/Edge (Manifest V3)** extension that reads the two values the
-[TutorialMaker Space](https://huggingface.co/spaces/vivekchakraverty/TutorialMaker) needs
-to get past YouTube's bot-check **without a proxy**:
-
-- **`visitor_data`** — read from the YouTube page's `ytcfg`.
-- **gvs `pot`** (Proof-of-Origin token) — captured from the player's
-  `googlevideo.com/videoplayback` request.
-
-It just surfaces these for one-click copy; it sends nothing anywhere.
-
-## Install (unpacked)
-
-1. Open `chrome://extensions` (or `edge://extensions`, `comet://extensions`).
-2. Turn on **Developer mode** (top-right).
-3. Click **Load unpacked** and select the folder that **directly contains
-   `manifest.json`** — i.e. the unzipped download folder, or this `extension/` folder if
-   you cloned the repo. (If you see a "Manifest file is missing" error, you selected a
-   parent folder — go one level deeper to the one holding `manifest.json`.)
-
-> Firefox: it's MV3-compatible in spirit, but Firefox needs a `background.scripts` entry
-> and a `browser_specific_settings` block instead of `service_worker`. Minor manifest
-> tweaks required.
-
-## Use
-
-1. **Use a throwaway Google account**, not your main one.
-2. Open a YouTube **video** and press **play** (this makes the player fetch
-   `videoplayback`, which carries the `pot`).
-3. Click the extension icon **while on the YouTube tab** — this captures the PO token and
-   reads your visitor data.
-4. Open the **TutorialMaker Space** in another tab (the direct `https://<space>.hf.space`
-   app URL is the most reliable target).
-5. Click the extension icon again, then **⤵ Fill the Space tab** — it injects both values
-   straight into the Space's "PO token" and "Visitor data" fields (expand the "YouTube
-   access" panel to see them).
-   - Or use **Copy PO token** / **Copy visitor data** and paste manually.
-6. Run the Space. Tokens expire within hours — re-grab when downloads start failing.
-
-## How it works
-
-- `background.js` — a read-only `webRequest` listener on `*://*.googlevideo.com/videoplayback*`
-  pulls the `pot` (and `c`/client) query params and stores the latest in
-  `chrome.storage.session`. It prefers `c=WEB` tokens (what yt-dlp's web client uses).
-- `popup.js` — on open, runs a MAIN-world script via `chrome.scripting` on the active
-  YouTube tab to read `visitor_data` (cached in session storage), reads the captured
-  `pot`, and formats the PO token as `web.gvs+<token>`.
-- **Auto-fill** — "Fill the Space tab" injects `fillSpaceFields` into the Space tab (all
-  frames, so it works both at `*.hf.space` and when embedded in `huggingface.co/spaces`).
-  It sets `#tm_pot_token` / `#tm_visitor_data` using the native value setter plus `input`
-  events so Gradio's Svelte bindings register the change. Requires the `*://*.hf.space/*`
-  host permission.
-
-## Troubleshooting
-
-**"No PO token yet" even though the video is playing.** YouTube fetches media via
-`fetch`/XHR (SABR), so the token is captured by a page-level hook (`content_hook.js`), not
-only the service worker. If it still doesn't appear:
-
-- **Reload the video page** after installing the extension (the hook must be in place
-  *before* the player starts).
-- Press **play** and let it **buffer a few seconds**, then click **Refresh** in the popup.
-- If the popup says *"Media requests were seen but none carried a PO token,"* your current
-  session isn't attaching one — try a **logged-out Incognito** window (with the extension
-  allowed in Incognito), or just use **cookies** instead.
-- Make sure the extension is **allowed in Incognito** if you test there
-  (`chrome://extensions` → Details → Allow in Incognito).
-
-## Caveats
-
-- **Unpublished / sideloaded** — you load it yourself; trust accordingly. The code is
-  small and dependency-free on purpose.
-- **YouTube changes break it** — the `pot`/`visitor_data` locations move over time; this
-  may need updates. The authoritative manual method lives in the
-  [yt-dlp PO Token Guide](https://github.com/yt-dlp/yt-dlp/wiki/PO-Token-Guide).
-- **Per session** — tokens are short-lived; this is a one-click refresh, not a permanent fix.
-- Automating bot-check bypass is the same YouTube-ToS gray area as the rest of this tooling.

extension/background.js

@@ -1,34 +0,0 @@
-// Service worker. Two jobs:
-//  1. Raise the session-storage access level so the youtube.com content-script bridge
-//     (an "untrusted" context) can persist captured values for the popup to read.
-//  2. A secondary, read-only webRequest capture of the gvs PO token from
-//     *.googlevideo.com/videoplayback requests (the page-level fetch/XHR hook is primary).
-
-try {
-  chrome.storage.session.setAccessLevel({ accessLevel: "TRUSTED_AND_UNTRUSTED_CONTEXTS" });
-} catch (e) {}
-
-const FILTER = { urls: ["*://*.googlevideo.com/*"] };
-
-chrome.webRequest.onBeforeRequest.addListener(
-  (details) => {
-    try {
-      if (details.url.indexOf("videoplayback") === -1 &&
-          details.url.indexOf("initplayback") === -1) return;
-      chrome.storage.session.set({ gvAnySeen: true, lastSeen: Date.now() });
-      const u = new URL(details.url);
-      const pot = u.searchParams.get("pot");
-      if (!pot) return;
-      const client = u.searchParams.get("c") || "";
-      chrome.storage.session.get(["gvsClient"]).then((cur) => {
-        const haveWeb = cur.gvsClient === "WEB";
-        if (!haveWeb || client === "WEB") {
-          chrome.storage.session.set({ gvsPot: pot, gvsClient: client, potTime: Date.now() });
-        }
-      });
-    } catch (e) {
-      // ignore malformed URLs
-    }
-  },
-  FILTER
-);

extension/content_bridge.js

@@ -1,25 +0,0 @@
-// Isolated-world bridge: receives postMessages from the MAIN-world hook and persists the
-// captured values to session storage (which the popup reads). Needs the background script
-// to have raised the session-storage access level for untrusted contexts.
-window.addEventListener("message", function (e) {
-  if (e.source !== window) return;
-  const d = e.data;
-  if (!d || d.__potgrab !== true) return;
-
-  if (d.type === "seen") {
-    chrome.storage.session.set({ gvAnySeen: true, lastSeen: Date.now() });
-  } else if (d.type === "pot" && d.data && d.data.pot) {
-    chrome.storage.session.get(["gvsClient"]).then(function (cur) {
-      const haveWeb = cur.gvsClient === "WEB";
-      if (!haveWeb || d.data.c === "WEB") {
-        chrome.storage.session.set({
-          gvsPot: d.data.pot,
-          gvsClient: d.data.c || "",
-          potTime: Date.now(),
-        });
-      }
-    });
-  } else if (d.type === "visitor" && d.data && d.data.v) {
-    chrome.storage.session.set({ visitorData: d.data.v });
-  }
-});

extension/content_hook.js

@@ -1,62 +0,0 @@
-// MAIN-world hook (runs in the page's JS context at document_start).
-//
-// Modern YouTube fetches media via fetch()/XHR to *.googlevideo.com/videoplayback?...&pot=...
-// (the MV3 service-worker webRequest path can miss these). We patch fetch + XHR to observe
-// those request URLs, pull the gvs `pot`, and post it to the isolated bridge script. We
-// also report visitor_data once it's available. We only observe — requests pass through
-// untouched.
-(function () {
-  function report(type, data) {
-    try { window.postMessage({ __potgrab: true, type: type, data: data }, "*"); } catch (e) {}
-  }
-
-  function scanUrl(url) {
-    try {
-      if (!url) return;
-      const s = String(url);
-      if (
-        s.indexOf("videoplayback") === -1 &&
-        s.indexOf("initplayback") === -1 &&
-        s.indexOf("googlevideo.com") === -1
-      ) return;
-      report("seen", null);
-      const u = new URL(s, location.href);
-      const pot = u.searchParams.get("pot");
-      if (pot) report("pot", { pot: pot, c: u.searchParams.get("c") || "" });
-    } catch (e) {}
-  }
-
-  try {
-    const origFetch = window.fetch;
-    if (origFetch) {
-      window.fetch = function (input) {
-        try { scanUrl(typeof input === "string" ? input : (input && input.url)); } catch (e) {}
-        return origFetch.apply(this, arguments);
-      };
-    }
-  } catch (e) {}
-
-  try {
-    const origOpen = XMLHttpRequest.prototype.open;
-    XMLHttpRequest.prototype.open = function (method, url) {
-      try { scanUrl(url); } catch (e) {}
-      return origOpen.apply(this, arguments);
-    };
-  } catch (e) {}
-
-  function reportVisitor() {
-    try {
-      if (window.ytcfg && typeof ytcfg.get === "function") {
-        const v = ytcfg.get("VISITOR_DATA");
-        if (v) { report("visitor", { v: v }); return true; }
-      }
-    } catch (e) {}
-    return false;
-  }
-  if (!reportVisitor()) {
-    let tries = 0;
-    const iv = setInterval(function () {
-      if (reportVisitor() || ++tries > 30) clearInterval(iv);
-    }, 1000);
-  }
-})();

extension/manifest.json

@@ -1,32 +0,0 @@
-{
-  "manifest_version": 3,
-  "name": "TutorialMaker PO Token Grabber",
-  "version": "1.1.0",
-  "description": "Reads your YouTube visitor data and gvs PO token so you can paste them into the TutorialMaker Space.",
-  "permissions": ["webRequest", "storage", "scripting", "tabs"],
-  "host_permissions": [
-    "*://*.youtube.com/*",
-    "*://*.googlevideo.com/*",
-    "*://*.hf.space/*"
-  ],
-  "background": { "service_worker": "background.js" },
-  "content_scripts": [
-    {
-      "matches": ["*://*.youtube.com/*"],
-      "js": ["content_hook.js"],
-      "run_at": "document_start",
-      "world": "MAIN",
-      "all_frames": true
-    },
-    {
-      "matches": ["*://*.youtube.com/*"],
-      "js": ["content_bridge.js"],
-      "run_at": "document_start",
-      "all_frames": true
-    }
-  ],
-  "action": {
-    "default_popup": "popup.html",
-    "default_title": "PO Token Grabber"
-  }
-}

extension/popup.html

@@ -1,45 +0,0 @@
-<!DOCTYPE html>
-<html>
-<head>
-  <meta charset="utf-8" />
-  <style>
-    body { font-family: system-ui, sans-serif; width: 340px; margin: 0; padding: 12px; }
-    h1 { font-size: 14px; margin: 0 0 8px; }
-    label { display: block; font-size: 11px; color: #444; margin: 8px 0 2px; }
-    textarea { width: 100%; box-sizing: border-box; font-family: monospace; font-size: 11px;
-               resize: vertical; }
-    .row { display: flex; gap: 6px; margin-top: 6px; }
-    button { flex: 1; padding: 6px; font-size: 12px; cursor: pointer; }
-    #status { margin-top: 10px; font-size: 11px; min-height: 14px; }
-    #status.ok { color: #1a7f37; }
-    #status.warn { color: #9a6700; }
-    .hint { font-size: 10px; color: #666; margin-top: 8px; line-height: 1.4; }
-  </style>
-</head>
-<body>
-  <h1>PO Token Grabber</h1>
-
-  <label for="potField">PO token (paste into the Space's "PO token" field)</label>
-  <textarea id="potField" rows="2" readonly placeholder="web.gvs+…"></textarea>
-
-  <label for="visField">Visitor data (paste into the Space's "Visitor data" field)</label>
-  <textarea id="visField" rows="2" readonly placeholder="Cgt…%3D%3D"></textarea>
-
-  <div class="row">
-    <button id="fillSpace"><b>⤵ Fill the Space tab</b></button>
-  </div>
-  <div class="row">
-    <button id="copyPot">Copy PO token</button>
-    <button id="copyVis">Copy visitor data</button>
-    <button id="refresh">Refresh</button>
-  </div>
-
-  <div id="status"></div>
-  <div class="hint">
-    Tip: open a YouTube video, press <b>play</b> (so the token is generated), then click
-    this icon. Use a throwaway account. Tokens expire within hours.
-  </div>
-
-  <script src="popup.js"></script>
-</body>
-</html>

extension/popup.js

@@ -1,179 +0,0 @@
-// Popup: read visitor_data from the active YouTube tab (main world) + the captured gvs PO
-// token from session storage, then either copy the values or auto-fill the Space's fields.
-
-// Runs in the page's MAIN world via chrome.scripting; must be self-contained.
-function readVisitorData() {
-  try {
-    if (window.ytcfg && typeof window.ytcfg.get === "function") {
-      const v = window.ytcfg.get("VISITOR_DATA");
-      if (v) return v;
-      const ctx = window.ytcfg.get("INNERTUBE_CONTEXT");
-      if (ctx && ctx.client && ctx.client.visitorData) return ctx.client.visitorData;
-    }
-  } catch (e) {}
-  try {
-    const r = window.ytInitialPlayerResponse;
-    if (r && r.responseContext && r.responseContext.visitorData) {
-      return r.responseContext.visitorData;
-    }
-  } catch (e) {}
-  return null;
-}
-
-// Injected into the Space tab (default/isolated world) to fill the Gradio inputs by their
-// elem_id. Uses the native value setter so Gradio's Svelte bindings register the change.
-function fillSpaceFields(poToken, visitorData) {
-  function setVal(id, val) {
-    if (!val) return false;
-    const root = document.getElementById(id);
-    if (!root) return false;
-    const el = root.querySelector("textarea, input");
-    if (!el) return false;
-    const proto =
-      el.tagName === "TEXTAREA"
-        ? window.HTMLTextAreaElement.prototype
-        : window.HTMLInputElement.prototype;
-    const setter = Object.getOwnPropertyDescriptor(proto, "value").set;
-    setter.call(el, val);
-    el.dispatchEvent(new Event("input", { bubbles: true }));
-    el.dispatchEvent(new Event("change", { bubbles: true }));
-    return true;
-  }
-  let n = 0;
-  if (setVal("tm_pot_token", poToken)) n++;
-  if (setVal("tm_visitor_data", visitorData)) n++;
-  return n;
-}
-
-const SPACE_RE = /(\.hf\.space)|(huggingface\.co\/spaces\/)/i;
-
-function setStatus(msg, kind) {
-  const el = document.getElementById("status");
-  el.textContent = msg;
-  el.className = kind || "";
-}
-
-async function getActiveTab() {
-  const [tab] = await chrome.tabs.query({ active: true, currentWindow: true });
-  return tab;
-}
-
-function isYouTube(url) {
-  return /^https?:\/\/([^/]*\.)?youtube\.com\//.test(url || "");
-}
-
-async function load() {
-  const potEl = document.getElementById("potField");
-  const visEl = document.getElementById("visField");
-
-  const tab = await getActiveTab();
-
-  // Read visitor_data live if we're on YouTube, and remember it for later (e.g. when the
-  // Space tab is the one in focus during fill).
-  let liveVisitor = null;
-  if (tab && isYouTube(tab.url)) {
-    try {
-      const res = await chrome.scripting.executeScript({
-        target: { tabId: tab.id },
-        world: "MAIN",
-        func: readVisitorData,
-      });
-      liveVisitor = res && res[0] ? res[0].result : null;
-    } catch (e) {}
-    if (liveVisitor) await chrome.storage.session.set({ visitorData: liveVisitor });
-  }
-
-  const store = await chrome.storage.session.get([
-    "gvsPot",
-    "gvsClient",
-    "potTime",
-    "visitorData",
-    "gvAnySeen",
-  ]);
-  const visitor = liveVisitor || store.visitorData || "";
-
-  visEl.value = visitor;
-  potEl.value = store.gvsPot ? "web.gvs+" + store.gvsPot : "";
-
-  // Is the Space open in some tab? Enable the fill button if so.
-  const all = await chrome.tabs.query({});
-  const spaceTabs = all.filter((t) => SPACE_RE.test(t.url || ""));
-  document.getElementById("fillSpace").disabled = spaceTabs.length === 0;
-
-  const notes = [];
-  if (!visitor) notes.push("No visitor data yet — open the popup once on the YouTube tab.");
-  if (!store.gvsPot) {
-    if (store.gvAnySeen) {
-      notes.push("Media requests were seen but none carried a PO token — this session may "
-        + "not attach one. Try a logged-out Incognito window, reload the video, or use "
-        + "cookies instead.");
-    } else if (isYouTube(tab && tab.url)) {
-      notes.push("No media captured yet — press play and let the video buffer a few "
-        + "seconds, then Refresh. If a video is already playing, reload the page.");
-    } else {
-      notes.push("Open a YouTube video and press play to capture a PO token.");
-    }
-  }
-  if (store.gvsPot && store.gvsClient && store.gvsClient !== "WEB") {
-    notes.push("Captured client is '" + store.gvsClient + "', not WEB — use the normal web player.");
-  }
-  if (spaceTabs.length === 0) notes.push("Open the TutorialMaker Space in a tab to enable auto-fill.");
-  if (store.potTime) {
-    const mins = Math.round((Date.now() - store.potTime) / 60000);
-    notes.push("Token captured ~" + mins + " min ago (they expire within hours).");
-  }
-  setStatus(notes.join(" ") || "Ready.", notes.length ? "warn" : "ok");
-}
-
-async function copy(id, label) {
-  const val = document.getElementById(id).value;
-  if (!val) {
-    setStatus("Nothing to copy yet for " + label + ".", "warn");
-    return;
-  }
-  try {
-    await navigator.clipboard.writeText(val);
-    setStatus("Copied " + label + " ✓", "ok");
-  } catch (e) {
-    setStatus("Copy failed — select the text and copy manually.", "warn");
-  }
-}
-
-async function fillSpace() {
-  const poToken = document.getElementById("potField").value;
-  const visitorData = document.getElementById("visField").value;
-  if (!poToken && !visitorData) {
-    setStatus("Nothing to fill yet — capture the values first.", "warn");
-    return;
-  }
-  const all = await chrome.tabs.query({});
-  const spaceTabs = all.filter((t) => SPACE_RE.test(t.url || ""));
-  if (spaceTabs.length === 0) {
-    setStatus("No TutorialMaker Space tab found — open it first.", "warn");
-    return;
-  }
-  let filled = 0;
-  for (const t of spaceTabs) {
-    try {
-      const results = await chrome.scripting.executeScript({
-        target: { tabId: t.id, allFrames: true },
-        func: fillSpaceFields,
-        args: [poToken, visitorData],
-      });
-      for (const r of results) filled += r.result || 0;
-    } catch (e) {}
-  }
-  if (filled > 0) {
-    setStatus("Filled " + filled + " field(s) in the Space ✓ (expand the panel to see).", "ok");
-  } else {
-    setStatus("Couldn't find the fields — make sure the Space finished loading.", "warn");
-  }
-}
-
-document.addEventListener("DOMContentLoaded", () => {
-  load();
-  document.getElementById("copyPot").addEventListener("click", () => copy("potField", "PO token"));
-  document.getElementById("copyVis").addEventListener("click", () => copy("visField", "visitor data"));
-  document.getElementById("fillSpace").addEventListener("click", fillSpace);
-  document.getElementById("refresh").addEventListener("click", load);
-});

pipeline/__init__.py

@@ -1,6 +1,7 @@
-"""Pipeline package: YouTube topic -> captioned .docx tutorial.
+"""Pipeline package: YouTube topic -> captioned .docx tutorial (API-based, no download).
 
 Stages (see app.run_pipeline for orchestration):
-    search -> sentiment -> download -> transcribe -> frames (extract + delete video)
-    -> tutorial -> frames.select (weighted) -> captions -> docx_builder
+    search -> sentiment (YouTube Data API comments) -> transcribe (youtube-transcript-api)
+    -> tutorial (LLM) -> frames (yt-dlp stream URL + ffmpeg -ss, weighted timestamps)
+    -> captions (VLM) -> docx_builder
 """

pipeline/download.py

@@ -1,162 +0,0 @@
-"""Stage 3: download the chosen video and extract audio for Whisper.
-
-Downloads a <=720p mp4 with yt-dlp, then uses ffmpeg to produce a 16 kHz mono wav.
-Enforces a duration cap up front so a long video can't stall the free-CPU Space.
-
-Both YouTube-facing calls accept an optional ``cookiefile`` and ``proxy`` (per-user or
-operator-wide), and fall back across a few yt-dlp player clients to improve the odds of
-getting past datacenter-IP blocking without cookies.
-"""
-from __future__ import annotations
-
-import os
-import re
-import subprocess
-
-from yt_dlp import YoutubeDL
-
-# Player clients tried in order. "android"/"tv" sometimes succeed where "web" is blocked.
-_PLAYER_CLIENTS = ["default", "android", "tv", "web_safari"]
-# When a (web) PO token is supplied it only applies to web-family clients, so try those.
-_POT_CLIENTS = ["web_safari", "web", "default"]
-
-
-class DownloadError(RuntimeError):
-    """Raised for user-actionable download failures (blocked IP, too long, etc.)."""
-
-
-def _parse_pot(po_token: str | None) -> list[str]:
-    """Split a PO-token field into yt-dlp entries (comma/semicolon/newline separated).
-
-    Each entry should be in yt-dlp's ``CLIENT.CONTEXT+TOKEN`` form, e.g. ``web.gvs+AbC``.
-    """
-    if not po_token:
-        return []
-    return [p.strip() for p in re.split(r"[,\n;]+", po_token) if p.strip()]
-
-
-def _base_opts(cookiefile: str | None, proxy: str | None, player_client: str,
-               po_token: str | None = None, visitor_data: str | None = None) -> dict:
-    yt_args: dict = {"player_client": [player_client]}
-    pots = _parse_pot(po_token)
-    if pots:
-        yt_args["po_token"] = pots
-    if visitor_data and visitor_data.strip():
-        yt_args["visitor_data"] = [visitor_data.strip()]
-    opts = {
-        "quiet": True,
-        "no_warnings": True,
-        "noplaylist": True,
-        "extractor_args": {"youtube": yt_args},
-    }
-    if cookiefile:
-        opts["cookiefile"] = cookiefile
-    if proxy:
-        opts["proxy"] = proxy
-    return opts
-
-
-def _extract_with_fallback(url: str, base: dict, cookiefile: str | None,
-                           proxy: str | None, download: bool,
-                           po_token: str | None = None, visitor_data: str | None = None):
-    """Run extract_info, retrying across player clients on failure.
-
-    Returns ``(info, ydl)`` from the first client that works; raises DownloadError with a
-    helpful hint if all clients fail.
-    """
-    clients = _POT_CLIENTS if _parse_pot(po_token) else _PLAYER_CLIENTS
-    last_exc = None
-    for client in clients:
-        opts = {**_base_opts(cookiefile, proxy, client, po_token, visitor_data), **base}
-        try:
-            ydl = YoutubeDL(opts)
-            info = ydl.extract_info(url, download=download)
-            return info, ydl
-        except Exception as exc:  # try the next client
-            last_exc = exc
-            continue
-    raise DownloadError(
-        _blocked_hint(last_exc, bool(cookiefile), bool(proxy), bool(_parse_pot(po_token))))
-
-
-def probe_duration(url: str, cookiefile: str | None = None, proxy: str | None = None,
-                   po_token: str | None = None, visitor_data: str | None = None) -> float:
-    """Return the video duration in seconds without downloading."""
-    info, _ = _extract_with_fallback(url, {"skip_download": True}, cookiefile, proxy,
-                                     download=False, po_token=po_token,
-                                     visitor_data=visitor_data)
-    return float(info.get("duration") or 0.0)
-
-
-def download_video(url: str, out_dir: str, cookiefile: str | None = None,
-                   max_minutes: int = 20, proxy: str | None = None,
-                   po_token: str | None = None,
-                   visitor_data: str | None = None) -> tuple[str, float]:
-    """Download the video to ``out_dir``; return ``(mp4_path, duration_seconds)``."""
-    os.makedirs(out_dir, exist_ok=True)
-
-    duration = probe_duration(url, cookiefile, proxy, po_token, visitor_data)
-    if duration and duration > max_minutes * 60:
-        raise DownloadError(
-            f"Video is {duration / 60:.0f} min long, over the {max_minutes} min cap for "
-            "this free-CPU Space. Pick a shorter video or raise the cap."
-        )
-
-    base = {
-        "outtmpl": os.path.join(out_dir, "video.%(ext)s"),
-        "format": "bestvideo[height<=720][ext=mp4]+bestaudio[ext=m4a]/best[height<=720]/best",
-        "merge_output_format": "mp4",
-    }
-    info, ydl = _extract_with_fallback(url, base, cookiefile, proxy, download=True,
-                                       po_token=po_token, visitor_data=visitor_data)
-    path = ydl.prepare_filename(info)
-    duration = float(info.get("duration") or duration or 0.0)
-
-    # merge_output_format may have rewritten the extension to .mp4
-    if not os.path.exists(path):
-        base_path, _ = os.path.splitext(path)
-        path = base_path + ".mp4"
-    if not os.path.exists(path):
-        raise DownloadError("yt-dlp reported success but no output file was found.")
-    return path, duration
-
-
-def extract_audio(video_path: str, out_dir: str) -> str:
-    """Extract 16 kHz mono wav from ``video_path`` via ffmpeg; return the wav path."""
-    wav_path = os.path.join(out_dir, "audio.wav")
-    cmd = [
-        "ffmpeg", "-y", "-i", video_path,
-        "-vn", "-ac", "1", "-ar", "16000", "-f", "wav", wav_path,
-    ]
-    proc = subprocess.run(cmd, capture_output=True, text=True)
-    if proc.returncode != 0 or not os.path.exists(wav_path):
-        raise DownloadError(
-            "ffmpeg failed to extract audio. Is ffmpeg installed (packages.txt)?\n"
-            + proc.stderr[-600:]
-        )
-    return wav_path
-
-
-def _blocked_hint(exc: Exception | None, had_cookies: bool, had_proxy: bool,
-                  had_pot: bool = False) -> str:
-    msg = str(exc) if exc else "unknown error"
-    lowered = msg.lower()
-    if any(k in lowered for k in ("sign in", "bot", "403", "429", "confirm you", "cookies",
-                                  "po_token", "po token")):
-        used = []
-        if had_cookies:
-            used.append("cookies")
-        if had_proxy:
-            used.append("proxy")
-        if had_pot:
-            used.append("PO token")
-        ctx = (f" Your {' + '.join(used)} did not get past it"
-               if used else " No cookies, proxy, or PO token were provided")
-        return (
-            "YouTube blocked this request (typical from datacenter IPs)." + ctx + ". "
-            "Try throwaway-account cookies, a PO token + visitor data (see the README "
-            "'PO token' guide), and/or a residential proxy. Note: free datacenter proxies "
-            "usually do NOT bypass this block.\n"
-            f"Original error: {msg[:300]}"
-        )
-    return f"Download failed: {msg[:400]}"

pipeline/frames.py

@@ -1,122 +1,159 @@
-"""Stage 5 + 6: candidate frames and the weighted screenshot indicator.
-
-Flow:
-  * ``extract_candidates`` pulls a dense, downscaled grid of frames from the video in a
-    single ffmpeg pass, *before* the video is deleted.
-  * ``delete_video`` removes the source video (auto-delete once the transcript + frames
-    exist).
-  * ``select_screenshots`` runs *after* the LLM. For each step it computes a capture time
-    that blends the LLM's suggested timestamp with the Whisper segment timing of the text
-    the step quotes, then snaps that target to the nearest pre-extracted candidate frame.
+"""Stage 5 + 6: real screenshots via yt-dlp stream URL + ffmpeg, at weighted timestamps.
+
+No full download. We resolve a direct video stream URL with yt-dlp (metadata only), then
+``ffmpeg -ss T -frames:v 1`` grabs a single frame per timestamp over HTTP range requests
+(only a few seconds of data each). For each shot we grab 3 candidates around T and keep
+the sharpest (Laplacian variance) to avoid landing on a blurry/transition frame.
+
+The capture timestamp itself comes from the weighted indicator: a blend of the LLM's
+suggested timestamp and the transcript segment timing of the text the step quotes.
 """
 from __future__ import annotations
 
 import difflib
-import glob
 import os
 import subprocess
 
-# A 1 fps grid gives <=0.5 s snapping error. Capped so a long video can't explode disk.
-GRID_FPS = 1.0
-MAX_CANDIDATES = 1500
+from yt_dlp import YoutubeDL
 
-
-def extract_candidates(video_path: str, out_dir: str, duration: float) -> list[tuple[float, str]]:
-    """Extract a downscaled candidate-frame grid; return sorted ``[(time_s, path), ...]``."""
-    os.makedirs(out_dir, exist_ok=True)
-    eff_fps = GRID_FPS
-    if duration and duration * GRID_FPS > MAX_CANDIDATES:
-        eff_fps = MAX_CANDIDATES / duration  # thin the grid for very long videos
-
-    pattern = os.path.join(out_dir, "cand_%06d.jpg")
-    cmd = [
-        "ffmpeg", "-y", "-i", video_path,
-        "-vf", f"fps={eff_fps:.6f},scale=-2:720",
-        "-q:v", "4", pattern,
-    ]
-    proc = subprocess.run(cmd, capture_output=True, text=True)
-    files = sorted(glob.glob(os.path.join(out_dir, "cand_*.jpg")))
-    if not files:
-        raise RuntimeError(
-            "ffmpeg extracted no candidate frames.\n" + proc.stderr[-600:]
-        )
-    # fps filter emits frame i (1-based) at ~ (i-1)/eff_fps seconds.
-    return [((i) / eff_fps, f) for i, f in enumerate(files)]
-
-
-def delete_video(video_path: str) -> None:
-    """Delete the downloaded video (and any sibling temp media) if present."""
-    for p in (video_path,):
-        try:
-            if p and os.path.exists(p):
-                os.remove(p)
-        except OSError:
-            pass
+# Format preference: muxed (single A/V url) <=720p first, then video-only, then anything.
+_FORMAT = ("best[height<=720][vcodec!=none][acodec!=none]/"
+           "best[height<=720][vcodec!=none]/bestvideo[height<=720]/best[vcodec!=none]/best")
 
 
+# --------------------------------------------------------------------------- weighting
 def _clamp(x: float, lo: float, hi: float) -> float:
     return max(lo, min(hi, x))
 
 
 def best_match_segment(quote: str, t_llm: float, segs: list[dict]) -> dict:
-    """Find the Whisper segment a step refers to.
-
-    Prefer the segment whose text best matches ``quote``; if there's no decent textual
-    match, fall back to the segment nearest ``t_llm``.
-    """
+    """The transcript segment a step refers to: best text match, else nearest to t_llm."""
     quote = (quote or "").strip().lower()
     if quote:
-        scored = [
-            (difflib.SequenceMatcher(None, quote, s["text"].lower()).ratio(), s)
-            for s in segs
-        ]
-        ratio, best = max(scored, key=lambda t: t[0])
+        ratio, best = max(
+            ((difflib.SequenceMatcher(None, quote, s["text"].lower()).ratio(), s) for s in segs),
+            key=lambda t: t[0])
         if ratio >= 0.3:
             return best
     return min(segs, key=lambda s: abs((s["start"] + s["end"]) / 2 - t_llm))
 
 
-def pick_time(step: dict, segs: list[dict], cand_times: list[float], *,
-              w_llm: float = 0.4, w_whisper: float = 0.6, lead: float = 1.0,
-              max_drift: float = 20.0) -> float:
-    """Weighted capture time, snapped to the nearest candidate frame.
+def pick_time(step: dict, segs: list[dict], *, w_llm: float = 0.4, w_whisper: float = 0.6,
+              lead: float = 1.0, max_drift: float = 20.0) -> float:
+    """Weighted capture time (seconds). No snapping — we grab on demand at any T.
 
-    ``t_whisper`` (segment mid + lead) is ground truth for *when* the referenced text is
-    spoken; ``t_llm`` is the model's guess of *what* matters. We blend them, but if the
-    LLM time drifts far from the matched segment we clamp it back into that segment so a
-    hallucinated timestamp can't drag the shot off-topic.
+    Transcript timing is ground truth for *when* the text is spoken; the LLM time is its
+    guess of *what* matters. Blend them, clamping a far-off LLM time back into the matched
+    segment so a hallucinated timestamp can't drag the shot off-topic.
     """
     seg = best_match_segment(step.get("quote", ""), float(step.get("t_llm", 0.0)), segs)
     t_whisper = (seg["start"] + seg["end"]) / 2 + lead
     t_llm = float(step.get("t_llm", t_whisper))
     if abs(t_llm - t_whisper) > max_drift:
         t_llm = _clamp(t_llm, seg["start"], seg["end"])
+    total = (w_llm + w_whisper) or 1.0
+    return max(0.0, (w_llm * t_llm + w_whisper * t_whisper) / total)
 
-    total = w_llm + w_whisper or 1.0
-    t_target = (w_llm * t_llm + w_whisper * t_whisper) / total
-    return min(cand_times, key=lambda c: abs(c - t_target))
 
-
-def select_screenshots(steps: list[dict], segs: list[dict],
-                       candidates: list[tuple[float, str]], *,
-                       w_llm: float = 0.4, w_whisper: float = 0.6, lead: float = 1.0,
-                       max_shots: int = 8) -> dict[int, dict]:
-    """Return ``{step_index: {"time", "path"}}`` for the steps that get a screenshot.
-
-    When there are more steps than ``max_shots``, the highest-``importance`` steps win.
-    """
-    if not candidates:
+def compute_shot_times(steps: list[dict], segs: list[dict], *, w_llm: float = 0.4,
+                       w_whisper: float = 0.6, lead: float = 1.0,
+                       max_shots: int = 8) -> dict[int, float]:
+    """Return ``{step_index: capture_time}`` for the top-``max_shots`` steps by importance."""
+    if not segs:
         return {}
-    cand_times = [t for t, _ in candidates]
-    time_to_path = dict(candidates)
-
     order = sorted(range(len(steps)), key=lambda i: -float(steps[i].get("importance", 0.5)))
     chosen = set(order[:max_shots])
+    return {
+        i: pick_time(steps[i], segs, w_llm=w_llm, w_whisper=w_whisper, lead=lead)
+        for i in range(len(steps)) if i in chosen
+    }
+
+
+# ----------------------------------------------------------------------- stream + grab
+def get_stream_url(video_id: str, cookiefile: str | None = None,
+                   proxy: str | None = None) -> tuple[str, float]:
+    """Resolve a direct video stream URL (no download); return ``(url, duration_seconds)``."""
+    opts = {"quiet": True, "no_warnings": True, "skip_download": True, "format": _FORMAT}
+    if cookiefile:
+        opts["cookiefile"] = cookiefile
+    if proxy:
+        opts["proxy"] = proxy
+    with YoutubeDL(opts) as ydl:
+        info = ydl.extract_info(f"https://www.youtube.com/watch?v={video_id}", download=False)
+
+    stream = info.get("url")
+    if not stream:
+        for f in info.get("requested_formats") or []:
+            if f.get("vcodec") not in (None, "none") and f.get("url"):
+                stream = f["url"]
+                break
+    if not stream:
+        vids = [f for f in info.get("formats", [])
+                if f.get("vcodec") not in (None, "none") and f.get("url")]
+        if vids:
+            stream = sorted(vids, key=lambda f: f.get("height") or 0)[-1]["url"]
+    if not stream:
+        raise RuntimeError("Could not resolve a video stream URL for frame capture.")
+    return stream, float(info.get("duration") or 0.0)
+
+
+def _grab(stream_url: str, t: float, out_path: str, proxy: str | None) -> bool:
+    """Grab one frame at time ``t`` via ffmpeg input-seek; return True on success."""
+    cmd = ["ffmpeg", "-y", "-loglevel", "error", "-ss", f"{max(0.0, t):.2f}",
+           "-i", stream_url, "-frames:v", "1", "-q:v", "2", out_path]
+    env = dict(os.environ)
+    if proxy:  # stream URL is bound to the extraction IP; route ffmpeg through it too
+        env["http_proxy"] = env["https_proxy"] = proxy
+    try:
+        subprocess.run(cmd, capture_output=True, text=True, timeout=90, env=env)
+    except subprocess.TimeoutExpired:
+        return False
+    return os.path.exists(out_path) and os.path.getsize(out_path) > 0
+
+
+def _sharpness(path: str) -> float:
+    """Variance of the Laplacian (higher = sharper). Used to reject blurry frames."""
+    import numpy as np
+    from PIL import Image
+
+    im = np.asarray(Image.open(path).convert("L"), dtype=np.float64)
+    if im.shape[0] < 3 or im.shape[1] < 3:
+        return 0.0
+    lap = (-4 * im[1:-1, 1:-1] + im[:-2, 1:-1] + im[2:, 1:-1]
+           + im[1:-1, :-2] + im[1:-1, 2:])
+    return float(lap.var())
+
+
+def grab_best_frame(stream_url: str, t: float, out_dir: str, idx: int,
+                    proxy: str | None = None) -> str | None:
+    """Grab candidates around ``t`` and keep the sharpest; return its path or None."""
+    cands = []
+    for k, dt in enumerate((-0.5, 0.0, 0.5)):
+        p = os.path.join(out_dir, f"shot_{idx}_{k}.jpg")
+        if _grab(stream_url, t + dt, p, proxy):
+            cands.append(p)
+    if not cands:
+        return None
+    best = max(cands, key=_sharpness)
+    for p in cands:
+        if p != best:
+            try:
+                os.remove(p)
+            except OSError:
+                pass
+    return best
+
 
+def capture_shots(times: dict[int, float], stream_url: str, out_dir: str,
+                  proxy: str | None = None, progress=None) -> dict[int, dict]:
+    """Grab a frame for each ``{step_index: time}``; return ``{idx: {"time", "path"}}``."""
+    os.makedirs(out_dir, exist_ok=True)
     out: dict[int, dict] = {}
-    for i, step in enumerate(steps):
-        if i not in chosen:
-            continue
-        t = pick_time(step, segs, cand_times, w_llm=w_llm, w_whisper=w_whisper, lead=lead)
-        out[i] = {"time": t, "path": time_to_path[t]}
+    items = list(times.items())
+    for n, (idx, t) in enumerate(items):
+        if progress:
+            progress((n + 1) / max(1, len(items)), desc=f"Screenshot {n + 1}/{len(items)}")
+        path = grab_best_frame(stream_url, t, out_dir, idx, proxy)
+        if path:
+            out[idx] = {"time": t, "path": path}
     return out

pipeline/sentiment.py

@@ -1,29 +1,32 @@
-"""Stage 2: rank candidate videos by the sentiment of their top comments.
+"""Stage 2: rank candidate videos by the sentiment of their comments.
 
-We fetch comments with yt-dlp (no video download) and score them with the BERT text
-classifier ``OmarMedhat7/youtube-sentiment-analysis-model`` running inside the Space.
-The video with the highest positive share wins. Comment scraping is flaky, so any
-single video that fails simply scores 0 and we never abort the whole ranking.
+Comments are fetched via the **YouTube Data API v3** (commentThreads) using an API key,
+then scored with the BERT classifier ``OmarMedhat7/youtube-sentiment-analysis-model``
+running inside the Space. The video with the highest positive share wins. Any video whose
+comments are disabled or fail to fetch simply scores 0 and never aborts the ranking.
 """
 from __future__ import annotations
 
-import os
-import re
+import json
+import urllib.error
+import urllib.parse
+import urllib.request
 from functools import lru_cache
 
-from yt_dlp import YoutubeDL
-
 SENTIMENT_MODEL = "OmarMedhat7/youtube-sentiment-analysis-model"
-MAX_COMMENTS = 30
+COMMENTS_API = "https://www.googleapis.com/youtube/v3/commentThreads"
+MAX_COMMENTS = 200  # cap per video to bound time/quota (commentThreads = 1 unit/100)
 
-# Label strings vary by model; treat anything that looks positive/negative accordingly.
 _POSITIVE = {"positive", "pos", "label_2", "label_1"}
 _NEGATIVE = {"negative", "neg", "label_0"}
 
 
+class SentimentError(RuntimeError):
+    """Raised for API-key / quota problems that should surface to the user."""
+
+
 @lru_cache(maxsize=1)
 def _classifier():
-    """Lazily build and cache the text-classification pipeline (heavy import)."""
     from transformers import pipeline
 
     return pipeline(
@@ -31,16 +34,11 @@ def _classifier():
         model=SENTIMENT_MODEL,
         truncation=True,
         max_length=256,
-        top_k=None,  # return all label scores so we can read the dominant one
+        top_k=None,
     )
 
 
 def _polarity(scores) -> float:
-    """Map one classifier output to a polarity in [-1, 1].
-
-    ``scores`` is a list of ``{"label", "score"}`` (top_k=None). Positive labels add,
-    negative labels subtract; neutral/unknown contribute nothing.
-    """
     val = 0.0
     for s in scores:
         label = str(s["label"]).strip().lower()
@@ -51,43 +49,65 @@ def _polarity(scores) -> float:
     return val
 
 
-def _fetch_comments(url: str, cookiefile: str | None, proxy: str | None,
-                    po_token: str | None = None, visitor_data: str | None = None) -> list[str]:
-    yt_args: dict = {"max_comments": [str(MAX_COMMENTS)], "comment_sort": ["top"]}
-    if po_token:
-        yt_args["po_token"] = [p.strip() for p in re.split(r"[,\n;]+", po_token) if p.strip()]
-    if visitor_data and visitor_data.strip():
-        yt_args["visitor_data"] = [visitor_data.strip()]
-    opts = {
-        "skip_download": True,
-        "getcomments": True,
-        "quiet": True,
-        "no_warnings": True,
-        "extractor_args": {"youtube": yt_args},
-    }
-    if cookiefile:
-        opts["cookiefile"] = cookiefile
-    if proxy:
-        opts["proxy"] = proxy
-    with YoutubeDL(opts) as ydl:
-        info = ydl.extract_info(url, download=False)
-    comments = info.get("comments") or []
-    texts = [c.get("text", "").strip() for c in comments if c.get("text")]
-    return texts[:MAX_COMMENTS]
-
-
-def rank_by_sentiment(videos: list[dict], cookiefile: str | None = None,
-                      progress=None, proxy: str | None = None,
-                      po_token: str | None = None,
-                      visitor_data: str | None = None) -> tuple[dict, list[dict]]:
-    """Score each video and return ``(best_video, scored)``.
-
-    ``scored`` mirrors ``videos`` with added keys: ``positive_share`` (0..1),
-    ``n_comments``, ``note``. Ordering of ``scored`` is by positive_share desc, with the
-    original search order as the tie-break / fallback.
+def _fetch_comments(video_id: str, api_key: str, cap: int = MAX_COMMENTS) -> list[str]:
+    """Fetch up to ``cap`` top-relevance comments via the YouTube Data API.
+
+    Returns [] if comments are disabled. Raises SentimentError on auth/quota failures
+    (which apply to every video, so the caller should stop).
+    """
+    comments: list[str] = []
+    page = None
+    while len(comments) < cap:
+        params = {
+            "part": "snippet",
+            "videoId": video_id,
+            "maxResults": "100",
+            "order": "relevance",
+            "textFormat": "plainText",
+            "key": api_key,
+        }
+        if page:
+            params["pageToken"] = page
+        url = COMMENTS_API + "?" + urllib.parse.urlencode(params)
+        try:
+            with urllib.request.urlopen(url, timeout=30) as resp:
+                data = json.load(resp)
+        except urllib.error.HTTPError as exc:
+            body = exc.read().decode("utf-8", "ignore")
+            if exc.code == 403 and "commentsDisabled" in body:
+                return []
+            if exc.code in (400, 403) and ("keyInvalid" in body or "quota" in body.lower()
+                                           or "forbidden" in body.lower()):
+                raise SentimentError(
+                    f"YouTube Data API rejected the key (HTTP {exc.code}). Check the key "
+                    f"and that YouTube Data API v3 is enabled / has quota. {body[:160]}")
+            # video-specific error (e.g. not found): treat as no comments
+            return comments
+        except Exception:
+            return comments
+
+        for item in data.get("items", []):
+            sn = item.get("snippet", {}).get("topLevelComment", {}).get("snippet", {})
+            text = sn.get("textDisplay") or sn.get("textOriginal")
+            if text:
+                comments.append(text.strip())
+        page = data.get("nextPageToken")
+        if not page:
+            break
+    return comments[:cap]
+
+
+def rank_by_sentiment(videos: list[dict], api_key: str,
+                      progress=None) -> tuple[dict, list[dict]]:
+    """Score each video by comment sentiment; return ``(best_video, scored)``.
+
+    ``scored`` mirrors ``videos`` with ``positive_share`` (0..1), ``n_comments``, ``note``
+    and is sorted by positive_share desc (search order as tie-break / fallback).
     """
     if not videos:
         raise ValueError("No videos to rank.")
+    if not api_key:
+        raise SentimentError("A YouTube Data API key is required to fetch comments.")
 
     clf = _classifier()
     scored: list[dict] = []
@@ -96,9 +116,9 @@ def rank_by_sentiment(videos: list[dict], cookiefile: str | None = None,
             progress((i + 1) / len(videos), desc=f"Sentiment {i + 1}/{len(videos)}")
         item = dict(v)
         try:
-            comments = _fetch_comments(v["url"], cookiefile, proxy, po_token, visitor_data)
+            comments = _fetch_comments(v["video_id"], api_key)
             if comments:
-                results = clf(comments)  # list aligned with comments (each = list of labels)
+                results = clf(comments)
                 pols = [_polarity(r) for r in results]
                 positives = sum(1 for p in pols if p > 0)
                 item["positive_share"] = positives / len(pols)
@@ -107,14 +127,15 @@ def rank_by_sentiment(videos: list[dict], cookiefile: str | None = None,
             else:
                 item["positive_share"] = 0.0
                 item["n_comments"] = 0
-                item["note"] = "no comments fetched"
+                item["note"] = "no comments"
+        except SentimentError:
+            raise  # key/quota problem affects all videos
         except Exception as exc:
             item["positive_share"] = 0.0
             item["n_comments"] = 0
-            item["note"] = f"comment fetch failed: {type(exc).__name__}"
+            item["note"] = f"fetch failed: {type(exc).__name__}"
         item["search_rank"] = i
         scored.append(item)
 
-    # Best = highest positive share; ties and all-zero fall back to original search order.
     scored.sort(key=lambda d: (-d["positive_share"], d["search_rank"]))
     return scored[0], scored

pipeline/transcribe.py

@@ -1,41 +1,64 @@
-"""Stage 4: timestamped transcription with faster-whisper (CPU/int8).
+"""Stage 4: fetch a timestamped transcript via youtube-transcript-api (no download).
 
-Returns a list of segments ``{"start", "end", "text"}`` in seconds. The model is cached
-at module level so repeated runs in the same Space process don't reload weights.
+youtube-transcript-api returns caption snippets that already carry ``start`` (and
+``duration``) timestamps, so we just normalize them into ``{start, end, text}`` segments
+for the rest of the pipeline. No Whisper, no audio, no video.
 """
 from __future__ import annotations
 
-import os
-from functools import lru_cache
 
-# Small model keeps CPU transcription tractable on the free tier. Override with
-# WHISPER_MODEL (e.g. "small", "tiny") via env if needed.
-WHISPER_MODEL = os.environ.get("WHISPER_MODEL", "base")
+class TranscriptError(RuntimeError):
+    """Raised when no usable transcript can be fetched (disabled, blocked, none)."""
 
 
-@lru_cache(maxsize=1)
-def _model():
-    from faster_whisper import WhisperModel
+def get_segments(video_id: str, languages=("en", "en-US", "en-GB"),
+                 proxy: str | None = None) -> list[dict]:
+    """Return timestamped segments ``[{start, end, text}]`` for ``video_id``.
 
-    return WhisperModel(WHISPER_MODEL, device="cpu", compute_type="int8")
+    Works across youtube-transcript-api 1.x (instance ``.fetch``) and older 0.6.x
+    (classmethod ``.get_transcript``). ``proxy`` is an optional http(s) proxy URL used as
+    a thin fallback when the Space's IP is blocked.
+    """
+    from youtube_transcript_api import YouTubeTranscriptApi
 
+    langs = list(languages)
+    raw = None
+    try:
+        # --- 1.x instance API ---
+        api = None
+        if proxy:
+            try:
+                from youtube_transcript_api.proxies import GenericProxyConfig
+                api = YouTubeTranscriptApi(
+                    proxy_config=GenericProxyConfig(http_url=proxy, https_url=proxy))
+            except Exception:
+                api = YouTubeTranscriptApi()
+        else:
+            api = YouTubeTranscriptApi()
 
-def transcribe(wav_path: str, progress=None) -> list[dict]:
-    """Transcribe ``wav_path`` and return timestamped segments."""
-    model = _model()
-    segments, info = model.transcribe(wav_path, vad_filter=True)
-
-    total = float(getattr(info, "duration", 0.0)) or None
-    segs: list[dict] = []
-    for s in segments:  # generator -> streamed, so we can report progress
-        text = s.text.strip()
-        if text:
-            segs.append({"start": float(s.start), "end": float(s.end), "text": text})
-        if progress and total:
-            progress(min(s.end / total, 1.0), desc="Transcribing")
+        if hasattr(api, "fetch"):
+            fetched = api.fetch(video_id, languages=langs)
+            raw = [{"text": s.text, "start": s.start, "duration": s.duration} for s in fetched]
+        else:
+            # --- 0.6.x classmethod API ---
+            kwargs = {"languages": langs}
+            if proxy:
+                kwargs["proxies"] = {"http": proxy, "https": proxy}
+            data = YouTubeTranscriptApi.get_transcript(video_id, **kwargs)
+            raw = [{"text": d["text"], "start": d["start"], "duration": d.get("duration", 0)} for d in data]
+    except Exception as exc:
+        raise TranscriptError(_hint(exc)) from exc
 
+    segs = []
+    for r in raw or []:
+        text = (r.get("text") or "").strip()
+        if not text:
+            continue
+        start = float(r.get("start") or 0.0)
+        dur = float(r.get("duration") or 0.0)
+        segs.append({"start": start, "end": start + dur, "text": text})
     if not segs:
-        raise RuntimeError("Whisper produced an empty transcript (no speech detected).")
+        raise TranscriptError("The transcript came back empty for this video.")
     return segs
 
 
@@ -46,3 +69,20 @@ def transcript_text(segs: list[dict]) -> str:
         m, sec = divmod(int(s["start"]), 60)
         lines.append(f"[{m:02d}:{sec:02d}] {s['text']}")
     return "\n".join(lines)
+
+
+def _hint(exc: Exception) -> str:
+    name = type(exc).__name__
+    msg = str(exc)
+    low = (name + " " + msg).lower()
+    if "disabled" in low or "transcriptsdisabled" in low:
+        return "This video has transcripts/captions disabled — pick another video."
+    if "notranscript" in low or "no transcript" in low:
+        return "No transcript is available for this video in the requested languages."
+    if "unavailable" in low or "videounavailable" in low:
+        return "The video is unavailable (private/removed/region-locked)."
+    if "block" in low or "ip" in low or "too many" in low or "429" in low:
+        return ("YouTube blocked the transcript request (common from datacenter IPs). "
+                "Set a YT_PROXY Space secret (residential proxy) and retry. "
+                f"[{name}]")
+    return f"Could not fetch transcript: {name}: {msg[:200]}"

requirements.txt

@@ -2,8 +2,9 @@ gradio>=4.44
 gradio_client>=1.3
 huggingface_hub>=0.27
 yt-dlp>=2024.12.13
-faster-whisper>=1.1.0
+youtube-transcript-api>=1.0
 transformers>=4.46
 torch>=2.2
+numpy>=1.26
 python-docx>=1.1.2
 Pillow>=10.4