Migrate from Pinecone to OSYRA Memory (OME)

This guide maps Pinecone's vector-store data model onto OSYRA Memory (OME) claims and walks through a migration end to end. It is the reference implementation for vector-store migrations: the Weaviate and ChromaDB guides describe themselves as deltas against this document.

| Aspect | Value | |---|---| | Migration pattern | Vector-store ID-preserving | | Source scope | One Pinecone namespace per run (loop to migrate all namespaces in an index) | | Source-side write access required | No — a read-only API key is sufficient | | Fidelity guarantees | Bit-stable round-trip of id, metadata field values (Unicode NFC-normalized), and source timestamp | | Embedding policy | Source vectors discarded; OME re-embeds with the workspace's configured embedder | | Rollback | Per-claim tombstone (see §8) |

§1 — Prerequisites

Source side (Pinecone)

• A Pinecone API key with read access to the source index. Generate one at Pinecone Console → API Keys. The key needs:
  • data-plane:read (vector fetch + describe)
  • control-plane:read (index describe — used to size the migration)
• The Pinecone environment string (for example us-east-1-aws).
• The index name and namespace to migrate (namespaces migrate one at a time).
• For metadata-rich indexes: the list of metadata fields you want preserved (see §3).

OSYRA side (OME)

• An OSYRA workspace ID (ws_<uuid>).
• A migration signer DID: did:osyra:ws:<uuid>:signer:migration. The migration signer DID is a special-purpose signer scoped only for bulk claim creation. Provision a dedicated migration signer DID — do not reuse a production signer key. Migration signer DIDs are rotation-eligible.
• OPA scopes the migration signer must hold:
  • osyra:ome:claim:create
  • osyra:ome:bundle:import
  • osyra:ome:bundle:simulate (optional, used by --dry-run)
• A tag plan: decide your tag convention before starting. Convention: pinecone:<index>:<namespace> (one tag per <index, namespace> pair).

Credential handling

Supply every credential through environment variables sourced from a secret store, never as command-line flags. For example:

export PINECONE_API_KEY="$(aws secretsmanager get-secret-value --secret-id pinecone/migrate --query SecretString --output text)"

Network

• Outbound HTTPS to your Pinecone control-plane URL (varies by environment).
• Outbound gRPC to ome.<region>.osyra.ai:443.

§2 — Source-system schema (Pinecone)

Pinecone organizes data hierarchically as index → namespace → vector. A vector has:

| Source field | Source type | Treatment | |---|---|---| | id | string (opaque, customer-assigned) | Preserved verbatim as claim.external_id | | values | float[N] (the embedding) | Discarded — OME re-embeds | | metadata | object (arbitrary key → string/number/bool/string[]) | Field-mapped (configurable) | | score (query-time only) | float | Not used — migration fetches, never queries |

Pinecone-specific concepts that are not preserved, and why:

| Source concept | Why not preserved | Implication | |---|---|---| | Index sharding | OME has its own sharding (configured at workspace level) | None | | Pod / serverless tier | OME has a different storage-tier model | None | | Replication factor | OME provides its own durability | None | | Sparse vectors (hybrid search) | OME does not support sparse-vector hybrid search | If you rely on hybrid search, this migration extracts the dense component and discards sparse | | Collection (snapshot) ID | OME has its own bundle identity | None | | Index metric (cosine, euclidean) | OME re-embeds; the metric is reconfigured workspace-side | Verify your OME workspace embedder and metric match your retrieval intent |

Schema discovery: before running migration, call describe_index_stats on the Pinecone index. The CLI does this automatically in --dry-run mode and prints the namespace inventory plus the metadata-key inventory.

§3 — OSYRA target mapping

This section is the contract: it defines exactly how each Pinecone field becomes an OME claim field.

Top-level mapping

| Source field | Source type | OME claim field | OME field type | Transformation | Required/Optional | Notes | |---|---|---|---|---|---|---| | id | string | external_id | string (max 256 bytes UTF-8) | Verbatim copy; reject if > 256 bytes | Required | Load-bearing for round-trip verification (§7) | | metadata.<body_field> | string | body | string (max 64 KB UTF-8, NFC-normalized) | NFC normalize; reject if > 64 KB; configurable field name (default description, override with --body-field) | Required | If the chosen body field is missing on a row, default behavior is --on-missing-body=skip | | metadata.<timestamp_field> | string (ISO-8601) or number (epoch seconds) | observed_at | int64 (epoch ms UTC) | Parse and convert; if missing, fall back to migration-time | Optional | Configurable field name (default timestamp, override with --timestamp-field) | | metadata.<type_field> | string | claim.type | enum {memory.fact, memory.observation} | Lookup table; default memory.fact | Optional | Configurable field name (default type). If present but not in the lookup, the row fails | | remaining metadata fields | object | metadata | JSON object (max 16 KB serialized) | All unmapped metadata keys pass through into claim.metadata.source_metadata.<key>; values coerced to string | Optional | Reserved keys are stripped before passthrough to avoid collision | | values (embedding) | float[N] | (discarded) | — | Source embeddings are not preserved; OME re-embeds | N/A | See §6 caveat on retrieval-result drift | | (implicit) source namespace | string | tag | string | Format pinecone:<index>:<namespace>; override with --tag | Required (auto-derived) | One tag per (index, namespace) |

Claim shape

{
  "type": "memory.fact",
  "external_id": "<source pinecone id>",
  "body": "<extracted from metadata.description by default>",
  "tag": "pinecone:<index>:<namespace>",
  "observed_at": 1716566400000,
  "metadata": {
    "source": "pinecone",
    "source_index": "<index>",
    "source_namespace": "<namespace>",
    "source_metadata": {
      "...": "any non-reserved fields from the original metadata blob"
    }
  }
}

Bundle shape

• One .ome bundle per batch (default 1000 claims).
• Bundle signer: the migration signer DID (§1).
• Source-system identification is carried in each claim's metadata.source_* fields. To verify source provenance, read claim.metadata.source, claim.metadata.source_index, and claim.metadata.source_namespace per claim.
• Bundle CIDs are surfaced in CLI stdout (§5) and persisted by OME for retrieval during §7 verification.

Override flags

| Flag | Default | Effect | |---|---|---| | --body-field <name> | description | Which metadata field becomes claim.body | | --timestamp-field <name> | timestamp | Which metadata field becomes claim.observed_at | | --type-field <name> | type | Which metadata field becomes claim.type | | --type-default <type> | memory.fact | Type to use when the type field is missing | | --tag <tag> | pinecone:<idx>:<ns> | Override the auto-derived tag | | --on-missing-body <action> | skip | skip, fail, or default:<text> | | --exclude-metadata-keys <list> | (empty) | Comma-separated metadata keys to exclude from passthrough |

§4 — CLI walkthrough

export PINECONE_API_KEY="$(aws secretsmanager get-secret-value --secret-id pinecone/migrate --query SecretString --output text)"
export PINECONE_ENVIRONMENT="us-east-1-aws"
export OSYRA_WORKSPACE_ID="ws_abc123"
export OSYRA_SIGNER_DID="did:osyra:ws:abc123:signer:migration"
 
osyra migrate pinecone \
  --source-index my-index \
  --source-namespace prod \
  --dry-run

osyra migrate pinecone \
  --source-index my-index \
  --source-namespace prod \
  --batch-size 1000 \
  --max-batches 1 \
  --output-dir ./pinecone-migrate-stage1/

osyra migrate pinecone \
  --source-index my-index \
  --source-namespace prod \
  --batch-size 1000 \
  --output-dir ./pinecone-migrate-full/ \
  --body-field description \
  --timestamp-field created_at

osyra migrate pinecone \
  --resume ./pinecone-migrate-full/.migrate-state.json

§5 — Expected output

osyra-migrate pinecone
Workspace:           ws_abc123
Signer DID:          did:osyra:ws:abc123:signer:migration
Source index:        my-index
Source namespace:    prod
Source vectors:      12,847 (from describe_index_stats)
Batch size:          1000
Bundles planned:     13
Output dir:          ./pinecone-migrate-full/
 
[ 1/13] Extracting batch ... 1000/1000 vectors
[ 1/13] Building claims ... 1000/1000 claims (0 skipped)
[ 1/13] Signing bundle ... CID: bafy2bzaceabc...001
[ 1/13] Imported  bundle ... 1000/1000 claims accepted
 
[ 2/13] Extracting batch ... 1000/1000 vectors
...
[13/13] Extracting batch ... 847/847 vectors
[13/13] Signing bundle ... CID: bafy2bzaceabc...013
[13/13] Imported  bundle ... 847/847 claims accepted
 
Summary:
  Total claims:    12,847
  Total bundles:   13
  Total skipped:   0
  Bundle CIDs file: ./pinecone-migrate-full/.cids.txt

If any rows are skipped (for example, --on-missing-body=skip with missing-body rows), a .skipped.jsonl file is written alongside the bundles, with one row per skipped vector and the skip reason.

§6 — Caveats

1. Embeddings are not preserved

Source vectors are discarded. OME re-embeds every claim's body text using the workspace's configured embedder. Consequences:

• Retrieval results will differ. Top-K nearest-neighbor results for the same query will not match Pinecone exactly, because the embedding model differs. If you rely on byte-identical retrieval, run a query-side regression suite (see §7) before cutting over.
• Re-embedding has a cost and a duration. Re-embedding contributes the largest single component of total migration time. Budget for it.

2. Untyped metadata is passthrough-stringified

Source metadata fields with non-string types (numbers, booleans, string arrays) are coerced to string when passed through to claim.metadata.source_metadata.*. If you rely on numeric metadata for downstream filtering, map it into a typed claim field with --body-field or --timestamp-field; otherwise it is string-typed in the resulting metadata.

3. Sparse vectors are not supported

If your Pinecone index uses sparse vectors (hybrid search), this migration extracts the dense component and discards the sparse component. Hybrid-search use cases should wait for OME sparse-vector support.

4. Sequencing is not preserved

Vectors arrive from Pinecone's fetch API in an order that is not guaranteed to match insert order. If your retrieval depends on insert ordering, do not rely on observed_at defaulting to migration-time — populate --timestamp-field with a source-side timestamp.

5. Idempotency on re-run

Re-running the migration with the same --source-namespace and --tag and overlapping IDs produces claims with the same external_id in OME. OME deduplicates by (workspace, tag, external_id): the second run is a no-op for already-imported IDs and imports only new IDs. To force re-import (for example, after fixing a body-extraction bug), use --force-rewrite, which tombstones the old claims and re-imports.

6. Partial bundles on failure

If a batch fails mid-signing, the partial .ome file is discarded and the batch is retried. No partially-signed bundle ever reaches OME. The resume state file tracks per-batch atomicity.

7. Rate limits

OME enforces a per-workspace bundle-import quota. The CLI throttles to stay under it. If your workspace has a higher quota, pass --max-rate <bundles/min>.

8. GDPR Art-17

A successful migration creates claims subject to OME's GDPR erasure path. Erasure of a migrated claim is tombstone-based and irreversible. If your Pinecone source holds data past your retention window, erase source-side first and migrate the remainder — do not migrate and then erase, which would leave you with overlapping retention on both sides.

§7 — Round-trip verification

Sample sizing

| Source vector count | Recommended sample size | |---|---| | ≤ 100K | 1% (minimum 100, maximum 1000) | | 100K – 1M | 0.5% (minimum 500, maximum 5000) | | > 1M | 0.1% (minimum 1000, maximum 10000) |

Protocol

osyra ome get-by-external-id \
  --workspace ws_abc123 \
  --tag pinecone:my-index:prod \
  --external-id <pinecone-id>

Acceptance criterion

At least 99.9% of sampled IDs must round-trip with byte-equal body text and an exact external_id match. The 0.1% tolerance accommodates expected variation in long-tail metadata blobs (for example, embedded control characters that get normalized).

When verification fails

Do not roll back immediately. First inspect the failing sample to classify the failure:

• Mapping issue (a metadata field shape you did not anticipate) → fix the §3 mapping and re-run with --force-rewrite.
• Encoding issue (for example, a source body containing lone surrogates) → confirm your normalization policy and decide accept-or-reject case by case.
• Source-side drift (the source schema changed mid-migration) → re-extract just the failing IDs.

Roll back (§8) only if the mapping is fundamentally wrong and you would rather start over.

§8 — Rollback

Rolling back affects only OME — the source Pinecone index is untouched at all times, because migration uses read-only access. Rollback means undoing the OME-side claim creations.

Per-claim tombstone

OME ships per-claim tombstoning (Tombstone(claimId)). To roll back a migration, iterate over the migrated external_ids (resolvable via the bundle CIDs in .cids.txt) and tombstone each claim:

for cid in $(cat ./pinecone-migrate-full/.cids.txt); do
  for claim_id in $(osyra ome list-claims-by-bundle --workspace ws_abc123 --bundle-cid "$cid"); do
    osyra ome tombstone \
      --workspace ws_abc123 \
      --claim-id "$claim_id" \
      --reason migration_rollback \
      --reason-detail "Pinecone migration failed §7 verification; rolling back."
  done
done

Tombstoning is irreversible per claim and is GDPR-Art-17-compliant. Each claim's tombstone digest replaces the claim row; cryptographic proofs of prior existence remain in the historical roots.

24-hour observation window

Wait 24 hours after migration before customer-facing cutover. Rolling back during this window is the cheapest path: tombstoning costs no application disruption, because no application is yet pointed at the migrated claims.

Post-rollback state

• OME: claims are tombstoned; bundle manifest CIDs are retained for audit; historical roots still attest prior claim existence.
• Pinecone: unchanged.
• Application: re-point at Pinecone (revert the cutover).

§9 — Troubleshooting

1. Pinecone API key unauthorized for namespace

• Symptom: dry-run fails with 401: namespace 'prod' not authorized for key '<prefix>...'.
• Likely cause: the API key is scoped to a different project or environment than the source namespace.
• Diagnostic: pinecone describe_index_stats --index my-index with the same key. If this fails, the key is wrong.
• Fix: regenerate the API key with project-level scope in the Pinecone console; reset the PINECONE_API_KEY env var.

2. Signer DID not authorized for claim creation

• Symptom: the first batch fails with OSY-AUTH-1071: signer DID '...' lacks scope 'osyra:ome:claim:create'.
• Likely cause: the migration signer DID exists but its OPA scope grant was not applied at provisioning, or it expired.
• Diagnostic: osyra iam describe-signer-did did:osyra:ws:abc123:signer:migration.
• Fix: re-grant the scope: osyra iam grant-scope --did did:osyra:ws:abc123:signer:migration --scope osyra:ome:claim:create.

3. Body field missing on N rows

• Symptom: migration completes but skip count is non-zero; .skipped.jsonl shows reason: "body field 'description' missing".
• Likely cause: some Pinecone metadata blobs use a different body field name (some rows have description, others have content).
• Diagnostic: dry-run with --metadata-field-histogram to see field-presence distribution across the source.
• Fix: standardize source data, or re-run with --body-field content --resume <state-file> to back-fill the skipped rows.

4. OME quota exceeded mid-migration

• Symptom: the run halts with OSY-RATE-2050: workspace bundle-import rate limit exceeded.
• Likely cause: the workspace's per-minute bundle-import quota is lower than the CLI's default throttle assumed.
• Diagnostic: osyra workspace describe ws_abc123 | grep rate_limit.
• Fix: resume with --max-rate <observed-quota>.

5. Round-trip verification fails on body diff

• Symptom: §7 sample diff shows a body-text mismatch on more than 0.1% of sampled IDs.
• Likely cause (in order of frequency): source description contained non-NFC-normalized Unicode (OME normalizes; your comparison did not); leading/trailing whitespace (OME preserves; some comparison tools strip); rows imported with --on-missing-body=default:<text>.
• Diagnostic: inspect the first 10 failing samples manually — one of the three causes is usually obvious.
• Fix: NFC-normalize the source side before diffing.

Related guides

• Vector-store delta guides: Weaviate, ChromaDB
• Agent-framework migrations: LangGraph, CrewAI, AutoGen