Video Generation

generateVideo() starts a video job, watches it, and streams progress to the browser. A video takes minutes, so one plain request times out.

Generate your first video#

You need a React app with a server route, and an OpenAI API key with Sora access. The setup takes about 5 minutes. The video itself takes a few minutes.

1. Install the packages:

npm i @tanstack/ai @tanstack/ai-react @tanstack/ai-openai

pnpm add @tanstack/ai @tanstack/ai-react @tanstack/ai-openai

yarn add @tanstack/ai @tanstack/ai-react @tanstack/ai-openai

bun add @tanstack/ai @tanstack/ai-react @tanstack/ai-openai

2. Put your key in .env:

OPENAI_API_KEY=sk-...

3. Add a POST route at /api/generate/video:

import {
  generateVideo,
  generationParamsFromRequest,
  toServerSentEventsResponse,
} from '@tanstack/ai'
import { openaiVideo } from '@tanstack/ai-openai'

export async function POST(request: Request) {
  const { input, threadId } = await generationParamsFromRequest('video', request)
  if (typeof input.prompt !== 'string') {
    return new Response('Send a text prompt.', { status: 400 })
  }

  const stream = generateVideo({
    adapter: openaiVideo('sora-2'),
    prompt: input.prompt,
    threadId,
    stream: true,
  })
  return toServerSentEventsResponse(stream)
}

4. Call the route from a component, and show the progress:

import { fetchServerSentEvents, useGenerateVideo } from '@tanstack/ai-react'

export function VideoGenerator() {
  const { generate, result, videoStatus, isLoading, error, stop } =
    useGenerateVideo({
      connection: fetchServerSentEvents('/api/generate/video'),
      threadId: 'video-studio',
    })

  return (
    <div>
      <button
        onClick={() =>
          generate({ prompt: 'A golden retriever playing in sunflowers' })
        }
        disabled={isLoading}
      >
        {isLoading ? 'Generating...' : 'Generate video'}
      </button>

      {isLoading && (
        <div>
          <p>Status: {videoStatus?.status ?? 'starting...'}</p>
          {videoStatus?.progress != null && (
            <progress value={videoStatus.progress} max={100} />
          )}
          <button onClick={stop}>Cancel</button>
        </div>
      )}

      {error && <p role="alert">Error: {error.message}</p>}

      {result && <video src={result.url} controls width={640} />}
    </div>
  )
}

Click Generate video. The status changes from pending to processing, then the video player shows the finished clip.

How it works#

Most video providers work with jobs:

1. Create a job. The provider takes the prompt and returns a job id.
2. Poll the job. The server asks the provider for the job status until the job is done or fails.
3. Get the video. When the job is done, the provider gives a URL for the video.

With stream: true, generateVideo() does all three steps on the server. It sends an event for each step:

• video:job:created when the provider accepts the job. The hook sets jobId.
• video:status after each poll. The hook sets videoStatus.
• generation:result when the video is ready. The hook sets result.

By default, the server polls every 2 seconds and stops after 10 minutes. Set pollingInterval and maxDuration (both in milliseconds) to change this:

import { generateVideo } from '@tanstack/ai'
import { openaiVideo } from '@tanstack/ai-openai'

const stream = generateVideo({
  adapter: openaiVideo('sora-2'),
  prompt: 'A cat playing piano in a jazz bar',
  threadId: 'thread-1',
  stream: true,
  pollingInterval: 3000,
  maxDuration: 900_000,
})

stop() on the hook only closes the stream in the browser. The provider job can continue to run.

The jobId, videoStatus, onJobCreated, and onStatusUpdate hook values are listed in Generation Hooks. The other transports (a server function that returns JSON, or one that returns an SSE Response) are in Generations.

For a stream that plays while it generates, and that you can steer with a new prompt, use Live Generation or World Generation. generateVideo() always makes a finished file.

Pick a provider#

Swap the adapter in step 3 to use another provider. Each adapter reads its key from the environment.

What each provider adds:

• Veo has typed durations and a 1080p or 4k resolution. Gemini Omni Flash can edit a video it made before.
• Grok Imagine takes reference images, and grok-imagine-video can edit or extend a clip.
• Seedance takes start and end frames, plus reference images, video, and audio.

The options for each provider, with full samples, are in Video Generation: Advanced.

Options#

If you have raw seconds, for example from a slider, adapter.snapDuration(seconds) gives the closest value that the model accepts:

import { generateVideo } from '@tanstack/ai'
import { openaiVideo } from '@tanstack/ai-openai'

const adapter = openaiVideo('sora-2')
const duration = adapter.snapDuration('6s') // "4"

if (duration !== undefined) {
  await generateVideo({
    adapter,
    prompt: 'A cat walking through fog',
    threadId: 'thread-1',
    duration,
  })
}

Sora takes 4, 8, or 12 seconds. The size and duration values for every provider are in Sizes and durations.

Start from an image#

To animate a still, or to give the model a start frame, an end frame, or a reference, send the prompt as an array of parts:

import { generateVideo } from '@tanstack/ai'
import { openaiVideo } from '@tanstack/ai-openai'
import { base64Image } from './assets'

const { jobId } = await generateVideo({
  adapter: openaiVideo('sora-2'),
  prompt: [
    {
      type: 'text',
      content: 'Animate this still into a slow cinematic push-in with subtle motion',
    },
    {
      type: 'image',
      source: { type: 'data', value: base64Image, mimeType: 'image/png' },
    },
  ],
  threadId: 'thread-1',
})

The accepted part types are narrowed per model at compile time. For example, a fal endpoint only accepts the image, video, or audio parts that its input type declares.

The SDK sends your prompt text exactly as you write it. Some endpoints have their own syntax to point at an input, for example @Image1 on fal Seedance. See Referencing images from your prompt for the table per provider.

Role hints#

Set metadata.role on an image part to tell the adapter what the image is for. Parts without a role use their position: the first image is the start frame.

import { generateVideo } from '@tanstack/ai'
import { falVideo } from '@tanstack/ai-fal'
import { firstFrameUrl, lastFrameUrl } from './assets'

await generateVideo({
  adapter: falVideo('fal-ai/kling-video/v3/pro/image-to-video'),
  prompt: [
    { type: 'image', source: { type: 'url', value: firstFrameUrl } },
    { type: 'text', content: 'Slow cinematic push-in then a hard cut' },
    {
      type: 'image',
      source: { type: 'url', value: lastFrameUrl },
      metadata: { role: 'end_frame' },
    },
  ],
  threadId: 'thread-1',
})

What each provider does with image inputs, and its limits, is in Image-to-video support. An adapter whose API cannot take image inputs throws an error.

Videos with no public URL#

Some providers have no public URL for the finished video. The adapter then returns the video as a stream. The browser cannot play a stream, so your server must turn it into a URL.

This happens with these providers:

• OpenRouter: always. Its download URLs need your API key.
• Lovable: when the finished job has no URL.
• OpenAI Sora: when the finished job has no URL.

Without storage, TanStack AI keeps the full video in memory and sets url to a base64 data: URL. This works for short clips. A large video can use all the memory of a serverless runtime. A video of more than 10 MiB logs a warning.

To keep memory flat, add withGenerationPersistence with an artifactUrl. It pipes the stream into your blob store and sets url on the result. The server never holds the full video in memory:

import {
  generateVideo,
  generationParamsFromRequest,
  toServerSentEventsResponse,
} from '@tanstack/ai'
import { openRouterVideo } from '@tanstack/ai-openrouter'
import {
  memoryPersistence,
  withGenerationPersistence,
} from '@tanstack/ai-persistence'

// Swap for a database and blob store in production.
const persistence = memoryPersistence()

export async function POST(request: Request) {
  const { input, threadId } = await generationParamsFromRequest('video', request)
  if (typeof input.prompt !== 'string') {
    return new Response('Send a text prompt.', { status: 400 })
  }

  const stream = generateVideo({
    adapter: openRouterVideo('google/veo-3.1'),
    prompt: input.prompt,
    stream: true,
    threadId,
    middleware: [
      withGenerationPersistence(persistence, {
        // The URL of your route that serves the stored bytes.
        artifactUrl: (ref) => `/api/artifacts/${ref.artifactId}`,
      }),
    ],
  })

  return toServerSentEventsResponse(stream)
}

memoryPersistence() has the generationRuns, artifacts, and blobs stores this needs. Keep generated files shows how to use a real store, and how to write the route that serves the bytes.

The hook needs no change. result.url is now a URL on your own origin, so <video src={result.url} /> plays it.

Providers that return a public URL need no storage to play the video. Persistence still copies those videos into your store, so you keep them after the provider URL expires.

Keep videos after a reload#

A video run takes minutes, so a page reload in the middle is likely. Generation Persistence keeps a record of each run:

• After a reload, the hook shows the last known status and result of the run, not an empty form.
• If the run still streams from a durable stream on the server, the hook attaches to it again and finishes it.
• Otherwise the hook restores the record. The provider work does not restart.

Provider video URLs expire. To keep the finished clip, save its bytes to your own storage.

Advanced#

Run the job without streaming, the options for each provider, the result types, and error handling are in Video Generation: Advanced.