Integration Webhooks

Rule

Integration webhooks (Slack, Telegram, WhatsApp, email, Google Docs, etc.) must enqueue work to SQL and return 200 immediately, then process the work in a separate fresh function execution kicked off by a self-fired HTTP POST. SQL is always the source of truth. Long-lived hosts use an in-process retry loop as a best-effort safety net; serverless deployments must use an external durable sweep when recovery cannot depend on a process staying alive.

Do not run agent loops inside the webhook handler itself. Do not rely on fire-and-forget Promises after returning from a serverless handler — they get killed when the function freezes.

Why

Messaging platforms expect a 200 response within a tight window — Slack will retry after 3 seconds, and a retried event triggers duplicate agent runs. At the same time, an agent loop replying to the message can take 30–60+ seconds because it may make multiple LLM calls and tool calls.

Past attempts that don't work cross-host:

• Fire-and-forget Promise.then(...) after returning — Lambda/Vercel/CF freeze the execution context the moment the response goes out. The promise is silently killed, the user gets no reply, and there's no error in the logs.
• A platform background function as the queue — host-specific execution is useful as a delivery mechanism, but it cannot replace the SQL row or atomic claim shared by every host.
• Cloudflare event.waitUntil() — CF Workers only, not portable.
• Vercel Fluid / after() — Vercel-only, gated behind specific runtimes.
• A long-lived in-process queue — fine on a single Node box, but on serverless every cold start gets a fresh queue and any pending work is lost.

The only universal answer: persist the work, then trigger a brand new function execution to do it. SQL is the queue, a self-webhook is the trigger, and a recurring job is the safety net.

The Flow

┌──────────┐    1. POST /integrations/:platform/webhook
│ Platform │────────────────────────────────────────────►┌──────────────────┐
└──────────┘                                             │ Webhook handler  │
                                                         │ (function exec 1)│
                                                         └──────────────────┘
                                                                  │
                            2. INSERT INTO integration_pending_tasks
                                 (status='pending', payload=...)
                                                                  │
                            3. dispatch POST /integrations/process-task
                                 — portable self-fire by default
                                 — acknowledged background handoff when enabled
                                                                  │
                            4. return 200 to platform ◄───────────┘

                                                         ┌──────────────────┐
                          5. POST arrives at processor   │ Processor        │
                             (separate fresh function)   │ (function exec 2)│
                                                         └──────────────────┘
                                                                  │
                            6. claimPendingTask(id) → status='processing'
                            7. runAgentLoop(...) — full timeout budget here
                            8. adapter.sendResponse(...) back to platform
                            9. markTaskCompleted(id)


                          ┌──────────────────────────────────────────────┐
                          │  Recovery sweep (every 60s) — safety net    │
                          │  Re-fires processor for tasks stuck in       │
                          │  'pending' or 'processing' beyond timeout.   │
                          │  Caps retries at 3 then marks 'failed'.      │
                          └──────────────────────────────────────────────┘

The webhook handler does as little as possible. The fresh function execution that handles _process-task gets its own full timeout budget for the agent loop.

Key Files

File	Purpose
packages/core/src/integrations/plugin.ts	Mounts /_agent-native/integrations/* routes
packages/core/src/integrations/webhook-handler.ts	Verifies signature, parses, enqueues task, fires processor
packages/core/src/integrations/pending-tasks-store.ts	SQL queue: insertPendingTask, claimPendingTask, markTaskCompleted, markTaskFailed
packages/core/src/integrations/pending-tasks-retry-job.ts	Recurring retry sweep (startPendingTasksRetryJob, retryStuckPendingTasks)
packages/core/src/integrations/integration-durable-dispatch.ts	Default-off acknowledged dispatch, scoped rollout, and outcome recording
packages/core/src/integrations/types.ts	PlatformAdapter, IncomingMessage, OutgoingMessage
packages/core/src/integrations/adapters/{slack,telegram,whatsapp,email,google-docs}.ts	One adapter per platform

Routes

All under /_agent-native/integrations/:

Method	Path	Purpose
POST	/:platform/webhook	Platform pings this. Verifies, enqueues, returns 200 quickly.
POST	/process-task	Processor target. Claims a task and runs the agent loop.
POST	/retry-stuck-tasks	Signed, bounded recovery sweep for durable schedulers.
GET	/status	All integrations status (settings UI).
GET	/:platform/status	One platform's status.
POST	/:platform/enable	Enable an integration.
POST	/:platform/disable	Disable an integration.
POST	/:platform/setup	Platform-specific setup (e.g. Telegram webhook registration).

SQL Schema

The pending-task queue lives in integration_pending_tasks:

CREATE TABLE IF NOT EXISTS integration_pending_tasks (
  id                 TEXT    PRIMARY KEY,
  platform           TEXT    NOT NULL,
  external_thread_id TEXT    NOT NULL,
  payload            TEXT    NOT NULL,   -- JSON-serialized IncomingMessage
  owner_email        TEXT    NOT NULL,
  org_id             TEXT,
  status             TEXT    NOT NULL,   -- pending | processing | completed | failed
  attempts           INTEGER NOT NULL DEFAULT 0,
  dispatch_attempts  INTEGER NOT NULL DEFAULT 0,
  last_dispatch_at   INTEGER,
  last_dispatch_outcome TEXT,
  dispatch_scope     TEXT,             -- persisted channel scope for recovery
  error_message      TEXT,
  created_at         INTEGER NOT NULL,
  updated_at         INTEGER NOT NULL,
  completed_at       INTEGER
);
CREATE INDEX IF NOT EXISTS idx_pending_tasks_status_created
  ON integration_pending_tasks(status, created_at);
CREATE INDEX IF NOT EXISTS idx_pending_tasks_dispatch_scope
  ON integration_pending_tasks(platform, dispatch_scope);

The store layer creates this lazily on first use via ensureTable() and uses intType() from db/client.ts so it works on both SQLite and Postgres.

claimPendingTask is the critical concurrency primitive: it atomically flips pending → processing and increments attempts, returning null if another worker beat us to it. Both the initial dispatch and every recovery sweep funnel through the same processor endpoint, and claimPendingTask is what prevents the same task from being processed twice.

Netlify durable recovery

The portable self-dispatch remains the default. To emit and use Netlify's acknowledged background worker plus a one-minute scheduled recovery function, set AGENT_INTEGRATION_DURABLE_DISPATCH=true at build and runtime. A production rollout can be narrowed with the comma-separated AGENT_INTEGRATION_DURABLE_DISPATCH_SCOPES allowlist. Supported values are <platform>:*, <platform>:<external-thread-id>, and, for adapters that expose one, <platform>:<channel-id> (for example slack:C123). Channel-scoped handoffs persist that scope separately from the provider thread identity so a later bounded recovery sweep applies the same allowlist decision.

Workspace deploys emit this pair only for the Dispatch control-plane app, which owns workspace messaging integrations. Standalone templates emit their own pair when the same flag is enabled.

The background handoff waits only for Netlify's enqueue acknowledgement, not for the agent loop. A failed or non-2xx acknowledgement falls back to the portable processor. The scheduled function calls the HMAC-authenticated, bounded /retry-stuck-tasks route; its atomic compare-and-set update prevents overlapping sweeps from dispatching the same stale row twice. Rows acknowledged by the 15-minute background worker use a 16-minute stale cutoff, while portable synchronous dispatches retain the shorter serverless cutoff. Apply that lease predicate before LIMIT so healthy long runs cannot hide recoverable work. The background processor writes its lease marker in the same atomic update that claims the row; best-effort dispatch telemetry is not trusted for replay safety.

Adding a New Platform Adapter

1. Implement PlatformAdapter in packages/core/src/integrations/adapters/<platform>.ts:

   export function myPlatformAdapter(): PlatformAdapter {
     return {
       platform: "myplatform",
       label: "MyPlatform",
       getRequiredEnvKeys: () => [
         { name: "MYPLATFORM_TOKEN", label: "MyPlatform Bot Token", scope: "global" },
         { name: "MYPLATFORM_SIGNING_SECRET", label: "MyPlatform Signing Secret", scope: "global" },
       ],
       async handleVerification(event) {
         // Platform-specific challenge response, if any
         return { handled: false };
       },
       async verifyWebhook(event) {
         // Verify HMAC/signature with a server-side secret and constant-time comparison.
         // Never leave this as a permissive stub in production.
         return verifyMyPlatformSignature(event);
       },
       async parseIncomingMessage(event) {
         // Map raw payload → IncomingMessage, or null to ignore
         return null;
       },
       async sendResponse(message, context) {
         // POST back to the platform's API
       },
       formatAgentResponse(text) {
         return { text, platformContext: {} };
       },
       async getStatus(baseUrl) {
         return { platform: "myplatform", label: "MyPlatform", enabled: false, configured: false };
       },
     };
   }
   

2. Register it in getDefaultAdapters() inside plugin.ts. The webhook, queue, processor, and retry job are shared infrastructure — you do not write any of that per-adapter.

3. Declare required env keys so the secrets/onboarding UI surfaces them. See secrets and onboarding skills.

4. Update the platform's webhook URL to point at ${baseUrl}/_agent-native/integrations/<platform>/webhook. For platforms with a registration API (Telegram), implement POST /:platform/setup.

Never hardcode bot tokens, signing secrets, verification tokens, webhook URLs, channel/customer identifiers, or copied platform payloads with real private data inside the adapter, tests, docs, prompts, or fixtures. getRequiredEnvKeys() declares credential names only. Values come from deployment configuration, registered secrets, OAuth, or scoped credential stores, and tests should use obvious fake placeholders.

The adapter is only responsible for:

• platform-specific verification (signatures, challenges)
• payload → IncomingMessage mapping
• agent text → platform format
• delivering the response back to the platform

It does not know about the queue, the processor, retries, or the agent loop. Those are handled by the shared webhook handler.

Slack non-message events

Slack's Events API also sends events that are not agent messages, such as link_shared for app unfurls. Do not map those into IncomingMessage unless they should actually run the agent. Handle them as short, provider-specific webhook work: verify the Slack signature, return 200 OK quickly, and call the provider API needed for the event (chat.unfurl for link previews). If a single Slack app must handle both agent chat and app unfurls, put a dispatcher in front of the one Slack Events Request URL and route message events to the integration webhook flow while routing link_shared to the app-specific unfurl handler.

Long-Running Agent Work

The processor endpoint runs in a fresh function execution with its own full timeout (typically 30–60s on Netlify/Vercel, longer on background-friendly hosts). That budget is dedicated entirely to the agent loop — there is no platform-side timer racing it.

If a single agent run might exceed the function timeout (large multi-step plans, deep delegation chains), the agent should:

1. Send an interim acknowledgement back to the platform so the user knows the request landed (adapter.sendResponse({ text: "Working on it..." })).
2. Persist intermediate state in chat-thread data, application state, or a recurring job so the next invocation can pick up where this one left off.

The retry job will only re-fire tasks stuck in processing for over 5 minutes, so a normal long-running reply is safe.

Cross-Platform Considerations

• Portable correctness, optional host acceleration. Every host uses SQL, HMAC-authenticated processor routes, and atomic claims. A host-specific background function or scheduler may provide a more reliable wake-up, but disabling it must preserve the portable path.
• No assumed runtime. The processor endpoint is a normal H3 handler under /_agent-native/. It runs wherever the rest of the framework runs.
• No persistent in-memory state. The dedup map in the webhook handler is best-effort only; the SQL queue is the source of truth. Any cold start loses the dedup map but the queue stays consistent.
• Postgres + SQLite both supported. claimPendingTask uses RETURNING on Postgres and a re-read on SQLite. No platform-specific SQL.
• Self-webhook URL resolution. The processor URL is built from WEBHOOK_BASE_URL, APP_URL, or URL env vars (with localhost:3000 as the dev fallback). Templates that change their public URL must keep one of these set.

Why Fire-and-Forget on Serverless Is Unreliable

The portable webhook path gives the outbound processor request a short head start but does not wait for the response body, so that initial dispatch is not guaranteed to complete before the function freezes. On a long-lived host, the in-process retry loop is a best-effort safety net. On Netlify with durable dispatch enabled, the initial handoff is acknowledged and the external scheduled function supplies the recovery wake-up for cases where:

• The serverless platform froze the handler before the outbound fetch flushed its bytes.
• The processor function 502'd or cold-started slow enough to time out.
• The processor itself was killed mid-agent-loop (function timeout, container shutdown, deploy mid-run).

Tasks stuck in pending for >90s or processing for >5min get re-fired up to 3 times. After 3 attempts they're marked failed permanently so we stop spamming the processor.

Never assume the initial dispatch succeeded. Rely on the SQL queue, atomic claim, and a recovery mechanism whose lifetime is independent of the original request. An in-process timer alone is not that mechanism on serverless.

Debugging Checklist

1. Platform sent the webhook? Check the platform's delivery logs (Slack admin, Telegram getWebhookInfo).
2. Webhook handler returned 200? If not, the platform retries — look for duplicate task rows. Signature failures return 401.
3. Task in the queue? SELECT * FROM integration_pending_tasks WHERE external_thread_id = '...' ORDER BY created_at DESC LIMIT 5.
4. Status? pending means the processor never picked it up — check that _process-task is reachable from the box itself (the self-fetch must work over the public URL). processing for over 5 minutes means the processor died mid-run — a live recovery sweep will pick it up.
5. Failed? Check error_message and attempts. After 3 attempts the row is parked at failed and won't be retried.
6. Reply not delivered? The processor likely succeeded but adapter.sendResponse failed — check the adapter's outbound logs.

Automation connectors

Workflow systems such as n8n and Zapier are not messaging channels. Do not add them as PlatformAdapters or route them through provider-api.

For n8n-style workflow invocation and callbacks, use @agent-native/core/automation with static workflow IDs and allow-listed origins. Agents provide structured input, never a target URL or credential. Callbacks must verify a configured secret/signature, claim a durable event ID, enqueue work through the established queue, and acknowledge quickly.

Zapier is blueprint-only unless the app implements Zapier's explicit REST Hook subscribe/unsubscribe contract or configures a separate Zapier MCP connection. Do not imply a generic Zapier workflow-execution API exists.

Related Skills

• server-plugins — How /_agent-native/ routes get mounted
• recurring-jobs — Pattern the retry job follows
• actions — When to use an action vs a webhook
• secrets — Registering platform tokens
• onboarding — Surfacing setup steps for each platform
• delegate-to-agent — How the processor invokes the agent loop