# Grok Imagine Favorites Scraper — Design Document

## Overview

A three-pass scraper for `grok.com/imagine/saved` that downloads all favorited
images and videos, along with metadata, into a SQLite database and local files.

---

## Data Model

### Grok Imagine's internal structure

A **favorite** is attached to a **source image**, not a video. When you generate
a video from an image, the image is favorited automatically and all derived
videos are associated with it. The favorites page shows one card per source
image, displaying the most recent video by default.

```
Favorite (source image)
└── preview_image.jpg          ← the source image (no play icon overlay)
└── Video 1 (UUID)             ← most recent, shown on favorites card
└── Video 2 (UUID)             ← older, only visible on detail page
└── Video N (UUID)
```

### Two card types on the favorites page

**Video card:**
- `<img>` + `<video>` elements present
- `img src` → `assets.grok.com/users/{user_uuid}/generated/{gen_uuid}/preview_image.jpg`
- `video src` → `assets.grok.com/users/{user_uuid}/generated/{gen_uuid}/generated_video.mp4`
- UUID extracted from path segment after `/generated/`
- Clicking navigates to `grok.com/imagine/post/{gen_uuid}`

**Image card:**
- `<img>` only, no `<video>`
- `img src` → `imagine-public.x.ai/imagine-public/images/{uuid}.jpg`
- UUID is the filename stem
- Has a "Make video" button (`aria-label="Make video"`)
- Clicking navigates to `grok.com/imagine/post/{uuid}`

### Two CDN domains

- `assets.grok.com` — user-scoped, **requires authentication** (HttpOnly cookies)
- `imagine-public.x.ai` — fully public, no auth needed

---

## URL Patterns

```
# Source image preview (clean, no play icon)
assets.grok.com/users/{user_uuid}/generated/{gen_uuid}/preview_image.jpg

# Original generated video
assets.grok.com/users/{user_uuid}/generated/{gen_uuid}/generated_video.mp4

# Public share copy of video (from og:video on detail page)
imagine-public.x.ai/imagine-public/share-videos/{uuid}.mp4

# Public image (image-type cards only)
imagine-public.x.ai/imagine-public/images/{uuid}.jpg

# Detail page for any post (image or video)
grok.com/imagine/post/{uuid}

# Composite thumbnail with play icon overlay (NOT the clean image)
grok.com/imagine/post/{uuid}/image?v=3
```

---

## Detail Page Findings

The site is a Next.js app. Server-rendered `<head>` meta tags are available
immediately at `domcontentloaded`. The React component tree requires full
hydration (`networkidle`) to render dynamic content like the sibling video strip.

### Image post detail page
- `meta[name="description"]` → Grok-generated caption (not the original prompt)
- `meta[property="og:image"]` → generic Grok OG image (useless)
- No prompt available in the static source

### Video post detail page
- `meta[name="description"]` → **raw user prompt** (the actual generation prompt)
- `meta[property="og:video"]` → share video URL on `imagine-public.x.ai`
- `meta[property="og:image"]` → composite thumbnail with play icon overlay
- Sibling video strip only available after JS hydration

### Sibling video strip (rendered DOM after hydration)
```html
<div class="snap-y snap-mandatory ... " style="width: 65px;">
  <button ...>
    <img src="assets.grok.com/users/{user}/generated/{uuid}/preview_image.jpg?cache=1">
    <!-- video camera SVG icon -->
  </button>
  <!-- one button per sibling video -->
</div>
```
- Selector: `.snap-y.snap-mandatory` container, then `button img` children
- UUID extracted from `img src` using `/generated/{uuid}/` pattern
- Currently selected video has `ring-2 ring-fg-primary` CSS classes
- All videos including current are present in the strip

---

## Three-Pass Architecture

### Pass 1 — Favorites list (`networkidle`, authenticates once)
**Status: partially working — scroll is broken due to virtualized layout**

- Navigates to `grok.com/imagine/saved`
- Scrolls to load all cards (currently broken — see Known Issues)
- For each card: extracts UUID, type, preview_url, latest_video_url
- Downloads `preview_image.jpg` immediately
- Populates `images` table

### Pass 2 — Detail pages
**Status: not yet tested**

- For **image cards**: visits detail page at `domcontentloaded`, extracts
  `meta[name="description"]` as `description`
- For **video cards**: visits detail page at `networkidle`, extracts prompt
  from meta description, waits for sibling strip, collects all video UUIDs
  into `videos` table

### Pass 3 — Per-video pages (`domcontentloaded`, fast)
**Status: not yet tested**

- For each video UUID in `videos` table
- Extracts `meta[name="description"]` → prompt
- Extracts `meta[property="og:video"]` → share_video_url
- Downloads video file
- Updates `videos` table

---

## SQLite Schema

```sql
CREATE TABLE images (
    uuid             TEXT PRIMARY KEY,
    type             TEXT NOT NULL,       -- 'image' or 'video'
    preview_url      TEXT,
    latest_video_url TEXT,                -- video type only
    description      TEXT,               -- human caption (image cards, Pass 2)
    prompt           TEXT,               -- raw generation prompt (may find later)
    pass1_at         TEXT,               -- ISO timestamp
    pass2_at         TEXT
);

CREATE TABLE videos (
    uuid             TEXT PRIMARY KEY,
    image_uuid       TEXT NOT NULL,       -- FK to images.uuid
    prompt           TEXT,               -- raw prompt from meta[name="description"]
    description      TEXT,               -- reserved for future Pass 4 LLM description
    share_video_url  TEXT,               -- og:video URL
    preview_url      TEXT,               -- assets.grok.com preview_image.jpg
    pass3_at         TEXT,
    downloaded_at    TEXT,
    FOREIGN KEY (image_uuid) REFERENCES images(uuid)
);
```

Timestamps are ISO 8601 strings (UTC). NULL = not yet completed.
`--redo` flag treats all rows as incomplete for the selected pass.

---

## Authentication

### Cookie requirements

Grok uses a mix of JS-readable and HttpOnly cookies. **The HttpOnly cookies
are mandatory** — without them, the React app will authenticate at the HTTP
level (200 OK) but then client-side navigate away from the favorites page
to `grok.com/imagine`.

Known required HttpOnly cookies (as of March 2026):
- `sso` (x2, for `.grok.com` and `.x.ai` domains)
- `sso-rw` (x2, for same domains)
- `__cf_bm` (x2, Cloudflare bot management)
- `cf_clearance` (Cloudflare clearance)

**SSO cookies expire in ~3 months. Cloudflare cookies expire in ~1 year.**

### Exporting cookies

Do NOT use `document.cookie` — it omits HttpOnly cookies. Use:
- Chrome DevTools → Application → Storage → Cookies → `https://grok.com`
  (manual copy for each HttpOnly cookie)
- Or the **Cookie-Editor** Chrome extension (exports all cookies including
  HttpOnly as JSON)

### Two-level auth checking

1. `check_auth(page, url, response)` — checks HTTP response URL (server redirects)
2. `assert_url(page, url)` — checks `page.url` after hydration (React navigation)
3. `--check-auth` CLI flag — runs both checks plus a CDN asset fetch test

---

## Known Issues

### Pass 1: Virtualized masonry layout

The favorites page uses absolute-positioned masonry with virtual DOM rendering.
Only cards near the viewport are mounted at any time. Scrolling
`document.body` doesn't grow the page height, so the scroll loop exits after
one attempt and only ~47 cards are captured (the initial viewport load).

**Proposed fix:** XHR interception. The React app fetches favorites in batches
from an internal API during scrolling. Registering `page.on('response', ...)`
before `goto()` and logging all XHR/fetch responses during the scroll loop
should reveal the API endpoint. Once identified, the endpoint can be called
directly — bypassing DOM scraping and virtualization entirely.

**To investigate:** Open Chrome DevTools → Network → XHR filter, load
`grok.com/imagine/saved`, scroll through all favorites, identify the API
calls that return favorites data.

### Pass 1: Scroll target

Even if XHR interception isn't used, the scroll target needs to be the
masonry container element, not `document.body`. The container uses
`position: absolute` children and a fixed height.

---

## config.yaml structure

```yaml
output_dir: grok_favorites
cookies_file: cookies.json
db_file: grok_favorites.db
favorites_url: https://grok.com/imagine/saved

scroll:
  max_attempts: 60
  pause_ms: 2000

concurrency: 3

browser:
  viewport_width: 1920
  viewport_height: 1080
  user_agent: "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"

check_auth:
  fallback_test_url: https://assets.grok.com/users/YOUR_USER_UUID/generated/YOUR_GEN_UUID/preview_image.jpg
```

The `fallback_test_url` should be an `assets.grok.com` URL (requires auth).
If the DB has any `images.preview_url` containing `assets.grok.com`,
`--check-auth` will use that instead.

---

## CLI Reference

```
python scrape.py --check-auth               # verify cookies, then exit
python scrape.py --pass 1                   # run Pass 1
python scrape.py --pass 2                   # run Pass 2
python scrape.py --pass 3                   # run Pass 3
python scrape.py --pass 1 --redo            # redo Pass 1 for all rows
python scrape.py --pass 2 --redo --uuid X   # redo Pass 2 for one UUID
python scrape.py --pass 1 --db foo.db       # override DB path
python scrape.py --pass 1 --output /tmp/dl  # override output dir
python scrape.py --pass 2 --concurrency 1   # single-threaded (debug)
```

---

## Future Work

- **Pass 4**: LLM-generated description for videos (stored in `videos.description`)
- **Original image prompt**: Not yet found. May be discoverable via XHR
  interception or a future API endpoint.
- **Multiple image variants**: Not investigated — Grok may generate multiple
  images per prompt; only the favorited one is currently captured.