# Keep your data in sync

URL: https://submitator.com/docs/guides/keep-your-data-in-sync

> Mirror projects and listings in your own system with no gaps and no doubles, by reading the events feed from a cursor, letting webhooks say when, and retrying writes with Idempotency-Key.

## What you'll build

A sync job that keeps your system's copy of each project and listing current. It reads every change in order from the events feed, so it misses nothing, and applies each event once. Webhooks tell it when to run, and writes it repeats after a timeout run once.

## Before you start

- **Your own ids on the projects.** Set `external_id` to your CRM's id when you create a project, as [Onboard a client's product](https://submitator.com/docs/guides/onboard-a-clients-product) does.
- **A place to store 1 cursor** per mode, and the ids of the events you applied.

## 1. Link your records to ours

Store our ids, `prj_` and `lst_`, next to your records: they never change, and a listing keeps its id after a correction. To find a project from your side, ask by `external_id`.

```bash
curl "https://api.submitator.com/v1/projects?external_id=crm_8812" \
  -H "Authorization: Bearer $SUBMITATOR_API_KEY"
```

```js
const res = await fetch("https://api.submitator.com/v1/projects?external_id=crm_8812", {
  headers: { Authorization: `Bearer ${process.env.SUBMITATOR_API_KEY}` },
});
const { data } = await res.json();
console.log(data[0]?.id);
```

```python
import os
import requests

page = requests.get(
    "https://api.submitator.com/v1/projects",
    headers={"Authorization": f"Bearer {os.environ['SUBMITATOR_API_KEY']}"},
    params={"external_id": "crm_8812"},
).json()
print(page["data"][0]["id"] if page["data"] else None)
```

`external_id` matches the whole value, case-sensitive, and is unique in each mode. `metadata` holds up to 20 more keys of your own and comes back on every read.

## 2. Read the feed from your cursor

`GET /events` returns every change oldest first, and `next_cursor` is never `null`. Apply a page, store its `next_cursor`, and ask again with it until `has_more` is `false`; next time, start from the cursor you stored.

```bash
curl "https://api.submitator.com/v1/events?cursor=cur_Ev7Kq2Wx9Lt3Nb5R&limit=100" \
  -H "Authorization: Bearer $SUBMITATOR_API_KEY"
```

```js
// loadCursor, saveCursor and apply are yours: your database and your records.
let cursor = await loadCursor();
for (;;) {
  const url = new URL("https://api.submitator.com/v1/events");
  url.search = new URLSearchParams({ limit: "100", ...(cursor && { cursor }) });
  const res = await fetch(url, { headers: { Authorization: `Bearer ${process.env.SUBMITATOR_API_KEY}` } });
  const page = await res.json();
  for (const event of page.data) await apply(event);
  cursor = page.next_cursor;
  await saveCursor(cursor);
  if (!page.has_more) break;
}
```

```python
import os
import requests

# load_cursor, save_cursor and apply are yours: your database and your records.
cursor = load_cursor()
while True:
    params = {"limit": 100, **({"cursor": cursor} if cursor else {})}
    page = requests.get(
        "https://api.submitator.com/v1/events",
        headers={"Authorization": f"Bearer {os.environ['SUBMITATOR_API_KEY']}"},
        params=params,
    ).json()
    for event in page["data"]:
        apply(event)
    cursor = page["next_cursor"]
    save_cursor(cursor)
    if not page["has_more"]:
        break
```

Response 200 (one event shown):

```json
{
  "object": "list",
  "data": [
    {
      "object": "event",
      "id": "evt_5Rt8Wq2Lm7Xn3Kb9Vz1PdJ",
      "type": "listing.live",
      "created_at": "2026-10-08T14:17:01Z",
      "livemode": false,
      "project": "prj_4QzX1m9Lr2Vb7Nc8Tk3HwP",
      "data": {
        "object": { "object": "listing", "id": "lst_7Hk2Qn4Rt9Vx1Bm6Cz3LdK", "status": "live" },
        "previous_status": "submitted"
      }
    }
  ],
  "has_more": false,
  "next_cursor": "cur_Ev7Kq2Wx9Lt3Nb5R"
}
```

The feed has no gaps: a cursor returns every event recorded after it, in order. Without a cursor it starts at the oldest event it keeps, 90 days back.

## 3. Apply each event

`data.object` is the whole object right after the change, so write it over your copy by its `id`. Skip an event whose `id` you have applied: after a crash between applying a page and storing its cursor, that page comes again.

```js
async function apply(event) {
  if (await alreadyApplied(event.id)) return;
  const object = event.data.object;
  if (object.object === "listing") await saveListing(object.id, object.status, object.live_url);
  if (object.object === "project") await saveProject(object.id, object.status, object.progress);
  await markApplied(event.id);
}
```

Statuses can move back. A live listing can become `not_accepted` or `submitted` again, and a `not_accepted` one can return: [How it works](https://submitator.com/docs/how-it-works#corrections) lists the corrections, and each comes as its own event.

## 4. Let webhooks say when

A webhook tells you about a change within about a minute of it. The simplest way to use one: answer 2xx and run the feed reader from step 2. The cursor then keeps the order and fills any gap, and a delivery that comes twice changes nothing.

To act on a delivery's body directly, compare its `data.previous_status` with what you stored, since deliveries can arrive out of order. [Receive webhooks](https://submitator.com/docs/guides/receive-webhooks) builds the endpoint.

## 5. Retry a write safely

A `POST` that timed out may have run. Send each one with an `Idempotency-Key`, and send the same key when you retry: a repeat with the same key and body within 24 hours returns the first response, with `Idempotent-Replayed: true`, and changes nothing twice.

```bash
curl https://api.submitator.com/v1/projects \
  -H "Authorization: Bearer $SUBMITATOR_API_KEY" \
  -H "Idempotency-Key: crm_8812-create" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://ledgerly.test", "contact_email": "founder@ledgerly.test", "external_id": "crm_8812"}'
```

```js
async function createProject(body, key) {
  for (let attempt = 1; ; attempt++) {
    const res = await fetch("https://api.submitator.com/v1/projects", {
      method: "POST",
      headers: {
        Authorization: `Bearer ${process.env.SUBMITATOR_API_KEY}`,
        "Idempotency-Key": key,
        "Content-Type": "application/json",
      },
      body: JSON.stringify(body),
    });
    if (res.status < 500 && res.status !== 409 && res.status !== 429) return res.json();
    if (attempt === 5) throw new Error(`${res.status} after 5 attempts`);
    await new Promise((resolve) => setTimeout(resolve, 1000 * 2 ** attempt));
  }
}
```

```python
import os
import time
import requests

def create_project(body, key):
    for attempt in range(1, 6):
        res = requests.post(
            "https://api.submitator.com/v1/projects",
            headers={"Authorization": f"Bearer {os.environ['SUBMITATOR_API_KEY']}", "Idempotency-Key": key},
            json=body,
        )
        if res.status_code < 500 and res.status_code not in (409, 429):
            return res.json()
        time.sleep(2 ** attempt)
    raise RuntimeError(f"{res.status_code} after 5 attempts")
```

The examples retry a 409 too, for `idempotency_key_in_use`; check `error.code` in your own code, since `project_exists` is a 409 as well. On a 429, wait the seconds in `Retry-After` instead of the backoff.

## 6. Recover after a long gap

The feed keeps 90 days. A cursor older than that returns 410 `cursor_expired`: read the projects and listings you track again, then read the feed without a cursor and store the new one.

```bash
curl "https://api.submitator.com/v1/projects/prj_4QzX1m9Lr2Vb7Nc8Tk3HwP/listings?limit=100" \
  -H "Authorization: Bearer $SUBMITATOR_API_KEY"
```

## Check it worked

- For each project, your copy's counts by status equal `progress` on `GET /projects/prj_4QzX1m9Lr2Vb7Nc8Tk3HwP`.
- Your stored cursor moves forward after each run, and a run with nothing new reads 1 page with `has_more` `false`.
- Running the job twice in a row changes nothing the second time.

## What can go wrong

| Code | Status | Fix |
| --- | --- | --- |
| [`invalid_cursor`](https://submitator.com/docs/errors#invalid_cursor) | 400 | The cursor came from another list or was changed. Send `next_cursor` exactly as you received it. |
| [`cursor_expired`](https://submitator.com/docs/errors#cursor_expired) | 410 | The cursor is older than 90 days. Read your objects again, then read the feed without a cursor. |
| [`idempotency_key_in_use`](https://submitator.com/docs/errors#idempotency_key_in_use) | 409 | The first request with this key still runs. Retry in 1 second with the same key. |
| [`idempotency_key_reused`](https://submitator.com/docs/errors#idempotency_key_reused) | 422 | The key came with another body before. Use a new key for each new request. |

## Next steps

- [Receive webhooks](https://submitator.com/docs/guides/receive-webhooks): the endpoint that triggers the job.
- [Event types](https://submitator.com/docs/webhooks/event-types): every event and what its `data` holds.
- [Events](https://submitator.com/docs/api/events): the feed's parameters in full.