# MediaCrawler Cloud Calling Guide

鍏ㄨ嚜鍔ㄨ皟搴︺€佺涓夋柟浼樺厛銆丆ookie 搴撳瓨鎺ュ姏鍜屾湇鍔＄鍒嗛〉璇﹁ [`SCHEDULER_AUTOMATION.md`](SCHEDULER_AUTOMATION.md)銆備笟鍔￠」鐩粯璁や娇鐢?`POST /tasks`锛屾棤闇€鑷绠＄悊 Cookie銆佷唬鐞嗐€丳rofile 鎴栫涓夋柟 Token銆?
This document is the integration contract for applications that need
Xiaohongshu or Beike collection. The caller owns its own users, business
database, AI analysis, and UI. MediaCrawler Cloud owns collection, task state,
Cookie/Proxy lifecycle, and result files.

## 1. Connection and Authentication

```text
MEDIACRAWLER_BASE_URL=https://media.154-9-238-153.sslip.io
MEDIACRAWLER_API_KEY=<the MediaCrawler Cloud service key>
```

All management and collection requests require the following header, except
`GET /health` and public result-view endpoints:

```http
X-API-Key: <MEDIACRAWLER_API_KEY>
Content-Type: application/json
```

Never send a Xiaohongshu Cookie, proxy supplier credential, Bingtop account, or
third-party provider Token from a calling application. Those are held only by
MediaCrawler Cloud.

## 1.1 Mandatory Discovery Step for Calling Projects and AI

Before creating any collection task, a new project should call:

```http
GET /integration-contract
X-API-Key: <MEDIACRAWLER_API_KEY>
```

This is the machine-readable service contract. It explicitly declares all
three channels, including the third-party entrance:

```json
{
  "id": "third_party",
  "create": {
    "method": "POST",
    "path": "/third-party-xhs/tasks"
  },
  "uses_local_cookie_proxy_profile": false
}
```

For a normal third-party collection task, the calling project must use exactly
`POST /third-party-xhs/tasks`. It must not send a Cookie,
proxy, browser Profile, or upstream provider Token. For direct low-level calls
instead of queued tasks, use the documented `/integrations/third-party/...`
gateway paths returned in this contract.

## 2. Which Interface to Use

| Need | Use | Local Cookie/IP/Profile used? | Result shape |
| --- | --- | --- | --- |
| Local Cookie collection | `POST /local-xhs/tasks` with `browser_mode=chromium` | Yes | Normalized JSONL |
| Third-party provider collection | `POST /third-party-xhs/tasks` | No | Normalized JSONL |
| Third-party provider batch | `POST /third-party-xhs/campaigns` | No | Normalized JSONL |
| Direct provider search/detail/comment call | `/integrations/third-party/...` | No | Provider raw JSON |
| General task management | `/tasks/...` | Depends on task mode | Task metadata / files |
| Batch many keywords | `/campaigns` | Depends on child task mode | Task IDs + aggregate state |

`chromium` is the only local Cookie collection mode. It is the
fingerprint-chromium isolated browser mode and requires a valid Cookie and a
proxy. API-only Cookie collection is disabled. `POST /third-party-xhs/tasks` is the dedicated independent provider
path: it does not accept or consume a local Cookie, proxy, or browser Profile.
It accepts provider-only throughput controls (`concurrency`,
`requests_per_second`, and `request_burst`), defaulting to 12 concurrent note
workflows, 20 RPS, and a burst of 24. The service fixes its execution mode to
`third_party` and uses its server-side provider configuration only.

Third-party tasks have a dedicated worker lane. They start immediately when
that lane is idle; they do not wait for the local Cookie queue's workload
threshold and are not included in Cookie demand planning. The local threshold
only counts `chromium` tasks.

## 2.1 Lifecycle and Recovery Contract

Task responses remain backward compatible and additionally expose:

```json
{
  "lifecycle_state": "waiting_third_party_devices",
  "failure_category": "third_party_capacity",
  "next_action": "One shared low-frequency probe will resume queued work",
  "retry_at": "2026-07-26T12:00:00+00:00",
  "execution_lease": {"id": "...", "worker_id": "third_party-0", "heartbeat_at": "..."}
}
```

Calling applications must treat `status=pending` together with
`lifecycle_state` as a normal waiting state, not as a failure. Important
states are `recoverable_interrupted`, `waiting_cookie`, `waiting_risk_hold`,
`waiting_third_party_balance`, and `waiting_third_party_devices`.

`waiting_third_party_balance` requires the provider account to be recharged
and is never auto-probed. `waiting_third_party_devices` uses one shared
exponential-backoff probe for the whole provider channel. Neither state falls
back to a local Cookie unless the task was explicitly created with
`allow_third_party_fallback=true`.

Local Chromium sessions are strict: one Cookie is bound to one Profile and one
dynamic IP for that session. When the IP TTL ends, MediaCrawler saves the
checkpoint and waits for a fresh Cookie session; it does not move that Cookie
to another IP automatically. CAPTCHA/461 puts the Cookie in risk hold and
keeps its Profile for review. Confirmed login invalidation is the only normal
case that deletes that Profile.

## 3. Standard Local Collection

Create a local task through the explicit local endpoint. It rejects
`third_party`, so a calling project cannot accidentally use provider quota.

```http
POST /local-xhs/tasks
```

```json
{
  "keywords": "棣欐腐澶у 绉熸埧",
  "browser_mode": "chromium",
  "max_notes": 300,
  "search_sort": "time_descending",
  "search_note_type": 2,
  "stop_when_older_than_days": 14,
  "collect_comments": true,
  "collect_sub_comments": false,
  "collect_creator_profiles": true,
  "max_comments_per_note": 20,
  "concurrency": 12,
  "requests_per_second": 20,
  "request_burst": 24
  "use_proxy": true,
  "headless": true,
  "concurrency": 1,
  "crawl_interval_seconds": 1,
  "requests_per_second": 1,
  "request_burst": 1,
  "adaptive_throttle": true
}
```

Mode rules:

- `chromium`: one Cookie has one isolated Profile and one active proxy at a
  time. It is the required local Cookie collection path.
- Real local tasks must set `use_proxy=true`. A Cookie is not prechecked when
  imported. The system binds a fresh proxy only when it actually activates the
  Cookie.
- A Cookie continues using the same proxy during its usable TTL. It is not
  rotated merely because another task begins. Hard transport failure or expiry
  can obtain a new unshared proxy; a confirmed invalid Cookie deletes its
  Profile.
- `allow_third_party_fallback` defaults to `true`: a confirmed local Cookie
  failure or CAPTCHA may hand the task to the configured third-party API. Set
  it to `false` for a Chromium/API task that must remain queued until the
  operator imports a new Cookie; it will never consume third-party quota.

## 4. Third-Party Collection Task

Use this for an independent provider-backed task. It is normalized into exactly
the same task/result model as local collection.

```http
POST /third-party-xhs/tasks
```

```json
{
  "keywords": "娴风彔鍖?绉熸埧",
  "max_notes": 100,
  "search_sort": "general",
  "search_note_type": 0,
  "third_party_note_time": "涓€鍛ㄥ唴",
  "collect_note_details": true,
  "collect_comments": true,
  "collect_sub_comments": true,
  "collect_creator_profiles": true,
  "max_comments_per_note": 20,
}
```

Third-party task rules:

- Do not send a provider Token. The service appends it internally.
- `third_party_note_time` supports `涓嶉檺`, `涓€澶╁唴`, `涓€鍛ㄥ唴`, and `鍗婂勾鍐卄.
- The complete Chinese provider-to-Cloud mapping, field list, costing formula,
  direct-gateway endpoints, and integration sequence are maintained in
  [`THIRD_PARTY_XHS_GUIDE.md`](THIRD_PARTY_XHS_GUIDE.md).
- It does not allocate a local Cookie, proxy, or browser Profile.
- `collect_note_details=false` keeps search-card fields only and skips one
  detail call per note. Images and complete text come from detail calls, so
  keep it enabled when those are required.
- `collect_creator_profiles`, `collect_comments`, and
  `collect_sub_comments` each add their corresponding provider calls. Author
  profiles are deduplicated within a task: one author is requested once even
  when several notes belong to that author.
- Every upstream provider HTTP request is counted, including retries. The
  default accounting rate is `楼0.01` per call. Task responses expose
  `third_party_api_calls` and `third_party_api_cost`; channel totals are in
  `GET /integrations/third-party/status`.
- Local `chromium` tasks may be promoted into this channel after a
  confirmed Cookie failure or 461/471/CAPTCHA risk signal, unless their
  `allow_third_party_fallback` is explicitly `false`. The task attempt records
  `execution_channel=third_party` and `fallback_reason`.
- A provider-side `402`, `insufficient_points`, or explicit quota/balance
  depletion does not fail the queued third-party work: it enters
  `waiting_third_party_recovery`, never consumes a local Cookie/proxy/Profile,
  and automatically retries one task after the configured recovery interval.

## 5. Shared Task Lifecycle

Every create call returns a task object. Store its `id` in the calling
application's own job record.

```json
{
  "id": "task-id",
  "status": "pending",
  "request": {"browser_mode": "chromium", "max_notes": 300},
  "notes_collected": 0,
  "comments_collected": 0
}
```

Poll every 5-15 seconds:

```http
GET /tasks/{task_id}
GET /tasks/{task_id}/attempts
```

Task states:

| State | Meaning | Caller action |
| --- | --- | --- |
| `pending` | Waiting for queue, Cookie, proxy, or retry delay | Continue polling |
| `running` | Collector is currently executing | Continue polling |
| `completed` | Reached the target and finalized | Download/import results |
| `partial` | Some results exist but target was not reached | Import current results; optionally resume or retry |
| `failed` | No recoverable result was produced | Inspect attempts, then retry after operator decision |
| `cancelled` | Operator or caller cancelled it | Keep any existing data; resume only if desired |

Important non-terminal stages include `waiting_workload`, `waiting_proxy`,
`retrying_account_same_proxy`, `paused_zero_yield`, `paused_proxy_risk`, and
`retrying_third_party`. These are protective states, not proof that a Cookie or
proxy is permanently invalid.

Control calls:

```http
POST /tasks/{task_id}/cancel
POST /tasks/{task_id}/resume?keep_account=true
POST /tasks/{task_id}/retry
POST /tasks/recover-incomplete
```

`resume` keeps checkpointed work where possible. `retry` creates a new task ID
from the original request. Do not create duplicate retries while a task is
`pending` or `running`.

## 6. Result Files and Import Contract

After a task is terminal, list its files:

```http
GET /tasks/{task_id}/results
```

Then download each file returned by:

```http
GET /tasks/{task_id}/files/{file_path}
```

Primary normalized files are normally under `xhs/jsonl/`:

- `*_contents.jsonl`: one JSON object per note.
- `*_comments.jsonl`: one JSON object per first- or second-level comment.
- `third_party_raw/*.json`: redacted upstream audit responses for a third-party
  task. These are diagnostic files, not the preferred business import source.

Use `note_id` as the note upsert key and `comment_id` as the comment upsert key.
Important normalized note fields include:

```json
{
  "note_id": "...",
  "title": "...",
  "desc": "...",
  "type": "normal",
  "time": 1780000000000,
  "note_url": "https://www.xiaohongshu.com/explore/...",
  "image_list": "https://...,...",
  "video_url": "https://...",
  "liked_count": "0",
  "collected_count": "0",
  "comment_count": "0",
  "share_count": "0",
  "ip_location": "...",
  "creator_id": "...",
  "creator_nickname": "...",
  "creator_red_id": "...",
  "creator_desc": "...",
  "creator_avatar": "https://...",
  "creator_fans": 0,
  "creator_follows": 0,
  "creator_liked": 0,
  "creator_note_count": 0,
  "creator_ip_location": "...",
  "source": "third_party_api"
}
```

`source` is `third_party_api` for provider-backed results. Local results do not
need to have the same source value. Missing optional author/profile fields mean
the platform or upstream response did not provide them; they do not invalidate
the note.

## 7. Direct Third-Party Gateway APIs

These APIs expose the documented provider operations through MediaCrawler Cloud
without leaking its Token. They return provider raw JSON, not the normalized
JSONL contract. Use task APIs for durable collection; use these direct APIs for
interactive lookup or a caller that needs provider-specific fields.

| Method and path | Use |
| --- | --- |
| `GET /integrations/third-party/status` | Cached configured/reachable state plus exact call count, cost, last call time and operation breakdown. Add `?refresh=true` for a real upstream probe. |
| `GET /integrations/third-party/search/notes` | `keyword`, `page`, `sort_type`, `note_type`, `note_time`. |
| `GET /integrations/third-party/notes/{note_id}` | Complete note detail. |
| `GET /integrations/third-party/notes/{note_id}/video` | Video stream/feed detail. |
| `GET /integrations/third-party/users/{user_id}` | Author profile. |
| `GET /integrations/third-party/notes/{note_id}/comments` | `start`, `sort_strategy`; first-level comments. |
| `GET /integrations/third-party/notes/{note_id}/comments/{comment_id}/replies` | `start`; second-level comments. |
| `GET /integrations/third-party/topics/{page_id}/notes` | `sort`, `cursor`; topic page notes. |
| `GET /integrations/third-party/users/{user_id}/notes` | `cursor`; author-posted notes. |

Direct search example:

```bash
curl -G "$MEDIACRAWLER_BASE_URL/integrations/third-party/search/notes" \
  -H "X-API-Key: $MEDIACRAWLER_API_KEY" \
  --data-urlencode "keyword=娴风彔鍖?绉熸埧" \
  --data-urlencode "page=1" \
  --data-urlencode "sort_type=time_descending" \
  --data-urlencode "note_type=鍥炬枃" \
  --data-urlencode "note_time=涓€鍛ㄥ唴"
```

## 8. Cookie and Proxy Operations

Operators manage inventory; calling applications normally only create tasks.

- `POST /accounts/import-text`: import multiple Cookies from pasted TXT/JSON.
- `POST /accounts/analyze-cookie`: offline structure and freshness analysis;
  not an online login-validity verdict.
- `PATCH /accounts/{account_id}/cookie`: replace a re-extracted Cookie for the
  same account, retire old proxy/Profile, and retain account history.
- `GET /planning/cookie-demand` or `POST /planning/cookie-demand`: estimate
  Cookie demand from queued work or a manual target.
- `GET/PATCH /drain`: read or set automatic high-yield queue thresholds.
- `POST /drain/start`: explicitly allow one Cookie to activate now.

## 9. Batch and Rental APIs

Use `POST /campaigns` for multiple Xiaohongshu keywords. It creates ordered
child tasks and returns campaign/task IDs. Use `GET /campaigns/{campaign_id}`
to monitor the batch and `POST /campaigns/{campaign_id}/cancel` to stop it.

Rental collection is independent from Xiaohongshu. `POST /beike/tasks` accepts
`source: "beike"` (default) or `source: "lianjia"`; each source uses its
official anonymous App rental-list endpoint while sharing the task model:

```http
POST /beike/tasks
GET  /beike/options
GET  /beike/tasks/{task_id}
GET  /beike/tasks/{task_id}/items?offset=0&limit=100
```

See [`BEIKE_LIANJIA_RENT_API.md`](BEIKE_LIANJIA_RENT_API.md) for source hosts,
conditions, output fields and verified boundaries.

## 10. Error Handling and Idempotency

- `401`: service API Key missing or invalid.
- `404`: unknown task/account/file path.
- `409`: current state does not allow the requested action, or a result is not
  ready.
- `422`: request field is invalid. For `/local-xhs/tasks`, this also means a
  non-local mode was supplied.
- `502`: an upstream provider call failed.
- `503`: the relevant server-side integration is not configured.

Persist the returned task ID before polling. Deduplicate imported notes by
`note_id` and comments by `comment_id`. Treat retry as a new task and do not
assume it is idempotent. Read `attempts` before classifying a local failure as a
Cookie or proxy failure: a risk/CAPTCHA or transport error can be temporary.