Migrate from Weaviate to OSYRA Memory (OME)

This guide maps Weaviate's object model onto OSYRA Memory (OME) claims. Weaviate is a vector-store ID-preserving migration, the same family as the Pinecone guide; read that guide first for the full reference, then read this one for the Weaviate-specific differences.

| Aspect | Value | |---|---| | Migration pattern | Vector-store ID-preserving (uuid → external_id) | | Source scope | One Weaviate class per run; multi-tenancy supported via --source-tenant | | Source-side write access required | No — a read-only API key is sufficient | | Fidelity guarantees | Bit-stable round-trip of uuid, property values (NFC-normalized), source timestamp | | Embedding policy | Source vectors discarded; OME re-embeds | | Rollback | Per-claim tombstone (see §8) |

§1 — Prerequisites

Source side (Weaviate)

• A Weaviate API key (if the cluster has auth enabled), or anonymous read access (if AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED=true).
• The Weaviate cluster URL (for example https://my-cluster.weaviate.network).
• The class name to migrate. Weaviate organizes by class, not namespace.
• If the cluster uses multi-tenancy (multiTenancyConfig.enabled = true), the tenant name (--source-tenant <tenant>).

OSYRA side (OME)

Identical to the Pinecone prerequisites — workspace ID, a dedicated migration signer DID (did:osyra:ws:<uuid>:signer:migration, rotation-eligible), the osyra:ome:claim:create and osyra:ome:bundle:import OPA scopes, and a tag plan.

Credential handling

Supply every credential through environment variables sourced from a secret store, never as command-line flags:

export WEAVIATE_API_KEY="$(aws secretsmanager get-secret-value --secret-id weaviate/migrate --query SecretString --output text)"

§2 — Source-system schema (Weaviate)

Weaviate organizes data as class → object, with optional tenant sub-partitioning. An object has:

| Source field | Source type | Treatment | |---|---|---| | uuid (object ID) | UUID string | Preserved verbatim as claim.external_id | | vector (embedding) | float[N] | Discarded — OME re-embeds | | properties (per-class schema) | typed (string, int, number, boolean, date, geoCoordinates, blob, cross-reference) | Field-mapped (configurable) | | _additional.id | UUID string | Same as uuid (Weaviate provides both) | | _additional.creationTimeUnix | int (epoch ms) | Becomes claim.observed_at if no source-side timestamp property is configured | | _additional.lastUpdateTimeUnix | int (epoch ms) | Discarded — tracking source updates is out of scope | | Cross-references (typed object refs) | reference (UUID) | Discarded — logged for later back-migration (see §6) |

Weaviate-specific concepts that are not preserved, and why:

| Source concept | Why not preserved | Implication | |---|---|---| | Vectorizer config (text2vec-openai, text2vec-cohere, custom) | OME has its own embedder config | Re-validate retrieval (see §6) | | Inverted-index config | OME has its own indexing | None | | Replication factor | OME provides its own durability | None | | Cross-references | OME's relation-claim type is not in scope for this migration | If your retrieval uses graph traversal, see §6 | | Blob properties | Out of scope — memory is text-first | Pre-extract blob content to a text property before migration, or skip | | Generative-search config (RAG sidecar) | OME has its own retrieval surface | None |

Schema discovery: query the class schema before running migration:

client.collections.get("MyClass").config.get()

The CLI does this automatically in --dry-run mode and prints the property inventory, the property type map, and multi-tenancy detection.

§3 — OSYRA target mapping

The mapping follows the same shape as the Pinecone mapping. Weaviate-specific deltas:

Top-level mapping

| Source field | Source type | OME claim field | OME field type | Transformation | Required/Optional | Notes | |---|---|---|---|---|---|---| | uuid | UUID string | external_id | string (max 256 bytes UTF-8) | Verbatim copy with lowercase normalization (Weaviate is case-insensitive on UUIDs, but mixed-case strings appear in some clients) | Required | | | properties.<body_field> | string | body | string (max 64 KB UTF-8, NFC-normalized) | NFC normalize; reject if > 64 KB; configurable (default text, override with --body-field) | Required | Text classes often use text or content; tune to your schema | | properties.<timestamp_field> | date (RFC3339) or int (epoch ms) | observed_at | int64 (epoch ms UTC) | Parse and convert; fall back to _additional.creationTimeUnix | Optional | Default field timestamp (configurable) | | properties.<type_field> | string | claim.type | enum {memory.fact, memory.observation} | Lookup table; default memory.fact | Optional | | | (implicit) class name + tenant | string | tag | string | Format weaviate:<class>[:tenant:<tenant>] | Required (auto-derived) | One tag per (class, tenant) | | cross-reference properties | reference | (discarded) | — | Logged in .skipped-cross-refs.jsonl for later back-migration | N/A | | | blob properties | blob | (skipped, or pre-extract required) | — | Default: skip rows with blob properties (--on-blob=skip\|fail\|extract:<field>) | N/A | extract:<field> reads a text-summary property if present |

Claim shape

Identical to the Pinecone claim shape. The Weaviate-specific metadata.source value is "weaviate", and metadata.source_class plus metadata.source_tenant replace source_index and source_namespace.

Override flags

All Pinecone flags apply, renamed for Weaviate terminology:

| Flag | Default | Effect | |---|---|---| | --body-field <name> | text | Which property becomes claim.body | | --timestamp-field <name> | timestamp | Which property becomes claim.observed_at | | --type-field <name> | type | Which property becomes claim.type | | --source-tenant <tenant> | (none) | Required if the class has multi-tenancy enabled | | --on-blob <action> | skip | Behavior for rows containing blob properties: skip, fail, extract:<text-property> |

§4 — CLI walkthrough

export WEAVIATE_URL="https://my-cluster.weaviate.network"
export WEAVIATE_API_KEY="$(aws secretsmanager get-secret-value --secret-id weaviate/migrate --query SecretString --output text)"
export OSYRA_WORKSPACE_ID="ws_abc123"
export OSYRA_SIGNER_DID="did:osyra:ws:abc123:signer:migration"
 
osyra migrate weaviate \
  --source-class Article \
  --body-field content \
  --dry-run

osyra migrate weaviate \
  --source-class Article \
  --source-tenant customer-42 \
  --body-field content \
  --tag weaviate:Article:tenant:customer-42 \
  --dry-run

osyra migrate weaviate \
  --source-class Article \
  --body-field content \
  --batch-size 1000 \
  --output-dir ./weaviate-migrate-full/

# Class 'Document' has a 'pdf' blob property plus an 'extracted_text' string property
osyra migrate weaviate \
  --source-class Document \
  --body-field extracted_text \
  --on-blob extract:extracted_text \
  --output-dir ./weaviate-doc-migrate/

Resume semantics and staging strategy are identical to the Pinecone walkthrough.

§5 — Expected output

osyra-migrate weaviate
Workspace:           ws_abc123
Signer DID:          did:osyra:ws:abc123:signer:migration
Source cluster:      https://my-cluster.weaviate.network
Source class:        Article
Source tenant:       (none — single-tenant class)
Source objects:      8,432 (from aggregate query)
Body field:          content
Batch size:          1000
Bundles planned:     9
Output dir:          ./weaviate-migrate-full/
 
[1/9] Extracting batch ... 1000/1000 objects (cursor: <cursor-token>)
[1/9] Building claims ... 1000/1000 claims (0 skipped, 0 cross-refs deferred)
[1/9] Signing bundle ... CID: bafy2bzaceabc...001
[1/9] Imported  bundle ... 1000/1000 claims accepted
 
...
 
[9/9] Extracting batch ... 432/432 objects
[9/9] Signing bundle ... CID: bafy2bzaceabc...009
[9/9] Imported  bundle ... 432/432 claims accepted
 
Summary:
  Total claims:               8,432
  Total bundles:              9
  Total skipped:              0
  Cross-refs deferred:        1,247 (logged to ./weaviate-migrate-full/.skipped-cross-refs.jsonl)

If --on-blob=skip and any objects have blob properties, a .skipped-blob.jsonl file is also written.

§6 — Caveats

1. Cross-references are not migrated

Weaviate cross-references (typed object-to-object refs) are not migrated. They are logged to .skipped-cross-refs.jsonl with the source UUIDs of both ends. OME's relation-claim type is not in scope for this migration.

If your retrieval depends on graph traversal, either wait for relation-claim support, or migrate now and back-fill relations later from the .skipped-cross-refs.jsonl file.

2. Multi-tenancy is one tenant per run

Weaviate multi-tenant classes require a separate run per tenant. Loop the CLI invocation; do not try to migrate all tenants in one run.

3. Vectorizer module nuances

If your Weaviate class uses a vectorizer (for example text2vec-openai with a specific model) that differs from your OME workspace's embedder, retrieval results will shift after migration. Where possible, configure your OME workspace embedder to match the source before migration. See the Pinecone embedding caveat for the underlying rationale.

4. Blob properties

Binary blobs cannot be migrated into OME claims (claims are text-first). The default --on-blob=skip drops rows with blob properties; --on-blob=extract:<field> falls back to a text-summary property if your class has one. A blob-heavy class with no pre-extracted text cannot be migrated by this path.

5. Sequencing, idempotency, rate limits, GDPR

These behave identically to Pinecone — see the Pinecone caveats for sequencing, idempotency on re-run, rate limits, and GDPR Art-17 tombstone semantics.

§7 — Round-trip verification

The protocol is the same as the Pinecone verification protocol, with these substitutions:

• Source id → source uuid.
• Pinecone metadata.description → Weaviate properties.<body_field>.
• osyra ome get-by-external-id uses the same flag shape; --tag is weaviate:<class>[:tenant:<tenant>].

Weaviate-specific check

If your source class had cross-references, the .skipped-cross-refs.jsonl log is your only record of which references were dropped. Verify the log is non-empty and contains the count you expect — this is your audit trail for any later back-fill.

Acceptance criterion

At least 99.9% of sampled UUIDs round-trip with byte-equal body text and an exact external_id match.

§8 — Rollback

Identical to the Pinecone rollback. Per-claim tombstone; the 24-hour observation window applies; rolling back does not touch the source Weaviate cluster (read-only access throughout).

Weaviate-specific rollback note

The .skipped-cross-refs.jsonl file is not invalidated by rollback. If you roll back and later re-migrate, the cross-ref log from the rolled-back run is still usable as back-fill input — keep it.

§9 — Troubleshooting

1. Class has multi-tenancy enabled but --source-tenant not specified

• Symptom: dry-run fails with OSY-MEM-MIG-002: class 'Article' has multiTenancyConfig.enabled=true; --source-tenant required.
• Likely cause: the source class is multi-tenant.
• Diagnostic: GET /v1/schema/Article and check multiTenancyConfig.enabled.
• Fix: re-run with --source-tenant <tenant>. Enumerate tenants with GET /v1/schema/Article/tenants.

2. Property type unsupported

• Symptom: migration fails on a row with OSY-MEM-MIG-003: property 'embedding_meta' has unsupported type 'phoneNumber'.
• Likely cause: the source schema includes a property type (geoCoordinates, phoneNumber, and similar) that the default mapping does not coerce to string.
• Diagnostic: dry-run with --property-type-report to enumerate exotic types in the source.
• Fix: add --exclude-properties phoneNumber,geoCoordinates to skip those properties, or pre-process source-side to flatten them into a string property.

3. Cross-refs blocking migration

• Symptom: you expected cross-references to migrate and find them in .skipped-cross-refs.jsonl.
• Likely cause: cross-refs are intentionally dropped (see §6 caveat 1).
• Fix: accept the drop and keep the log for back-fill, defer migration, or flatten cross-refs source-side into string-typed body content.

4. Signer DID not authorized

Same as the Pinecone signer-DID troubleshooting.

5. Round-trip verification fails on body diff

Same as the Pinecone body-diff troubleshooting. Weaviate-specific note: if your class had cross-references, verify .skipped-cross-refs.jsonl matches the expected reference count before concluding that a body-text mismatch is a mapping problem.

Related guides

• Vector-store reference: Pinecone
• Vector-store delta: ChromaDB
• Agent-framework migrations: LangGraph, CrewAI, AutoGen