Scaling for production¶

A ready-to-run production-oriented stack lives in deploy/ in the repository. It always runs the pg_vectorize server (the API) and a separate embedding worker. Postgres and a self-hosted embedding model are optional compose profiles, because most production deployments use a managed Postgres and a hosted embedding provider instead. Only the API port is published, and only on loopback. The rest of this page explains the choices in that file and what it leaves out.

Where Postgres and embeddings run¶

The stack does not assume where either lives. Each is one of two choices, and .env selects it.

Postgres is either your own (RDS, Neon, or any Postgres with pgvector) or the bundled container:

Embeddings come from a hosted provider or from a model you run. The provider is chosen per job, by the prefix of its model (openai/text-embedding-3-small), and its credentials are read from the environment of both the server and the worker:

COMPOSE_PROFILES takes both: COMPOSE_PROFILES=postgres,tei. The full list of variables, and what each provider needs, is in Server configuration. For example, a managed Postgres with OpenAI, and nothing else to run:

cd deploy
cp .env.example .env
# in .env: delete the COMPOSE_PROFILES line, then set
#   DATABASE_URL=postgresql://user:password@your-host:5432/postgres
#   OPENAI_API_KEY=sk-...
docker compose up -d

To run everything yourself, keep COMPOSE_PROFILES=postgres,tei from .env.example, set POSTGRES_PASSWORD, pick an EMBEDDING_MODEL, and run docker compose up -d.

Choosing an embedding model¶

The best model depends on your language, domain, document length, latency and cost, and on the hardware it runs on. Plan to spend time on this: embed a sample of your own data with a few candidates and check that search returns what your users expect. Trying candidates first is cheap (vector-serve or a hosted provider makes it easy) and changing later is expensive.

If you want somewhere to start, deploy/ uses BAAI/bge-base-en-v1.5. These three work with the bundled TEI:

Larger models usually retrieve better and run slower. All three are English-language and MIT-licensed. TEI runs any embedding model it supports (see its documentation for the list), so for other languages pick a multilingual one.

Two things to know before you choose, whichever provider you use:

• The model is fixed per table. The embeddings column is typed vector(N), so changing to a model with a different dimension means re-embedding everything. Vectors from different models are not comparable even at the same dimension. Decide before loading data.
• Inputs longer than the model's max tokens are truncated (the bundled TEI runs with --auto-truncate), so the tail of a long document does not affect its embedding. Check the limit against your document or chunk size.

Self-hosting embeddings with TEI¶

This section applies when you run the embedding model yourself (the tei profile). With a hosted provider, skip to Postgres.

Use TEI instead of vector-serve¶

vector-serve is convenient: it downloads any Hugging Face model on first use and can serve several at once. That flexibility is what makes it good for demos and for comparing models. It is also a Python process running one sentence-transformers model call at a time per worker, and it is much slower than a dedicated inference server.

Text Embeddings Inference (TEI) serves one model chosen at startup, batches requests by token count, and exposes the same /v1/embeddings API. Measured on one 24-vCPU WSL2 Docker host with all-MiniLM-L6-v2 and short inputs, requests per second straight to /v1/embeddings:

This is embedding throughput only. On the end-to-end search benchmark (bench/), TEI and vector-serve came out about even: the hybrid SQL query and CPU sharing with Postgres dominate there. Expect the gain to show up when you ingest or re-embed a lot of data, not in single-query search latency.

TEI settings that matter for pg_vectorize¶

Setting the model¶

Set EMBEDDING_MODEL in .env (see Choosing an embedding model). TEI serves that one model and ignores the model field in a request. A job created with a different model string is still embedded by TEI's model, without any error, so keep the job's model and EMBEDDING_MODEL the same.

CPU sizing and batch size¶

Larger models are slower than small ones, and CPU is much slower than GPU (TEI publishes GPU images; see its documentation for the right one for your GPU). How slow depends on the model, your hardware and your document length, so time a batch of your own documents before settling on settings.

This interacts with two pg_vectorize settings:

• Rows are queued in messages of up to the job's batch_size rows, and the worker sends each message's rows to the embedding provider in one request. Set batch_size when you create the job (default 1000, maximum 10000), or change it later with PATCH /api/v1/table/{job_name}. The rows already in the table when you create a job are also capped at about 10,000 tokens per message.
• Embedding requests time out after 120 s by default. Set the EMBEDDING_REQUEST_TIMEOUT environment variable (in seconds) on the worker to change it: the worker service in the compose file, or the server if you run the worker in-process. It applies to each HTTP request to the embedding provider.

On CPU, a batch of 1000 or more long documents can take longer than 120 s. And when a client gives up, TEI does not cancel the batch it already accepted: in testing it kept all CPUs busy for minutes after the proxy timed out. If your documents are long and you are on CPU, lower the job's batch size:

curl -X PATCH http://localhost:8080/api/v1/table/my_job \
  -H "Content-Type: application/json" -d '{"batch_size": 100}'

The new size applies to rows written after the request returns; messages already queued keep their size.

Jobs created before batch_size existed keep reading the vectorize.batch_size Postgres setting until you set their batch_size with PATCH. On upgrade, their stored batch_size is set to that setting's value (or 1000), so PATCH with the same number keeps their behavior.

Postgres¶

• Prefer a managed Postgres that offers pgvector (RDS, Cloud SQL, Neon, and others): leave the postgres profile off and set DATABASE_URL. If you use the bundled one, back up the pgdata volume.
• Size memory. Postgres defaults are tiny. maintenance_work_mem bounds HNSW index builds; shared_buffers and effective_cache_size should reflect the host. The compose file exposes these as PG_SHARED_BUFFERS, PG_EFFECTIVE_CACHE_SIZE and PG_MAINTENANCE_WORK_MEM.
• Keep the embedder off the database host if you can. On the benchmark host, Postgres and the embedding service competed for the same CPUs. We did not isolate how much of the search latency that explains, so treat this as a precaution.
• Connection pool. DATABASE_POOL_MAX (default 2 * NUM_SERVER_WORKERS + 2, so 18) and NUM_SERVER_WORKERS (default 8) set how many connections the server holds. Keep the total under Postgres's max_connections.

Database role¶

The server and worker do not need a superuser. Connect them as a dedicated role, and have an administrator run this once, replacing postgres with your database and public.products with each table you create jobs on:

-- as a superuser or the database owner
CREATE EXTENSION IF NOT EXISTS vector;
CREATE ROLE vectorize LOGIN PASSWORD 'change-me';

-- lets the server create the pgmq and vectorize schemas on first start
GRANT CREATE ON DATABASE postgres TO vectorize;

-- for each table you create a job on
GRANT USAGE ON SCHEMA public TO vectorize;
GRANT SELECT, REFERENCES, TRIGGER ON public.products TO vectorize;

What each grant is for:

• vector must already be installed. The server runs CREATE EXTENSION IF NOT EXISTS vector on start, which is a no-op when it exists but fails for a non-superuser when it does not.
• SELECT, REFERENCES, TRIGGER on a source table let the server read its rows, create the embeddings and search-token tables with foreign keys to it, and add the triggers that queue changed rows.
• Deleting a job needs more. DELETE /api/v1/table/{job_name} drops those triggers, and Postgres only lets the table's owner do that. Either make the vectorize role a member of the owning role, or drop the triggers yourself as the owner.

The triggers run as whichever role writes to the source table, not as vectorize. If your application writes as a different role, that role also needs access to what the triggers write to:

GRANT USAGE ON SCHEMA vectorize, pgmq TO app_role;
GRANT INSERT ON pgmq.q_vectorize_jobs TO app_role;
GRANT USAGE ON ALL SEQUENCES IN SCHEMA pgmq TO app_role;
-- search-token tables are created per job, so grant on future tables too
ALTER DEFAULT PRIVILEGES FOR ROLE vectorize IN SCHEMA vectorize
    GRANT SELECT, INSERT, UPDATE ON TABLES TO app_role;

The background worker¶

The worker that turns queued rows into embeddings can run in two ways:

• Inside the server (the default). vectorize-server starts the worker as a task, so one process serves the API and does the embedding. The repository's demo docker-compose.yml does this.
• As its own service. Set VECTORIZE_WORKER_ENABLED=false on the server and run the vectorize-worker binary, which is in the same image, as a separate container. deploy/docker-compose.yml does this.

With the split, ingest load no longer competes with search traffic for a process or a connection pool, and you scale each side on its own:

docker compose up -d --scale worker=4    # more embedding throughput

Notes:

• Several workers can poll the same queue. Each job is claimed from the queue with a 300 s visibility timeout, so two workers do not take the same job. Embedding throughput is still capped by TEI, so more workers only help if TEI is not already saturated.
• Each vectorize-worker holds 5 Postgres connections. Add that to the server's pool when you size max_connections.
• With the in-process worker off, the server's /health and /health/ready return 200 and report the worker as Disabled. They say nothing about whether a worker is running.
• The standalone worker serves its own GET /health on WORKER_HEALTH_PORT (default 8081; the compose file does not publish it) and the compose service uses it as its healthcheck. It returns 200 with status: healthy, or 503 while the worker is failing to read the queue or process a job (for example when Postgres or TEI is down), with the last error in the body. It recovers on the next successful poll. It does not detect a worker stuck inside one slow batch, because a busy worker does not heartbeat; watch the queue depth for that (SELECT * FROM pgmq.metrics('vectorize_jobs')).
• If you turn the in-process worker off and start no worker, rows are queued and never embedded.

Security and networking¶

• The server has no authentication and allows any CORS origin. Do not expose port 8080 directly. Put a reverse proxy in front that terminates TLS and authenticates requests. The compose file binds the port to 127.0.0.1 for this reason.
• Postgres and TEI publish no ports. To get a SQL shell, use docker compose exec postgres psql -U postgres.
• If you use the bundled Postgres, set a real POSTGRES_PASSWORD. The container will not initialize without one. Keep provider API keys in .env (or your secret store), not in the compose file.

Operations¶

• Pin images. deploy/ pins TEI and pgvector to versions. Set VECTORIZE_SERVER_IMAGE to a specific tag or digest instead of latest. Changing the TEI version or model changes the embeddings, which matters for an existing index.
• Model cache. TEI stores downloaded weights in the tei-cache volume. The first start downloads the model, and the healthcheck allows two minutes for that.
• Logs are rotated by the compose logging options. Lower RUST_LOG from debug (the demo default) to info.