osyra_ome.cli.migrate.from_pinecone
Private pre-release / network boundary: The currently usable customer surface is offline verification of an existing signed artifact against an out-of-band trust anchor. Memory/network RPC APIs and host snippets on this page describe protocol and interface shape; the public Edge bridge, package publication, and release acceptance remain pending. Do not treat
api.osyra.ai:443examples as a released customer route.
``osyra-ome migrate from-pinecone`` — ENG-577 / Tier-4 T4-7.
Migrates vector embeddings + metadata from a Pinecone index into the OME memory store via main's gRPC ``OmeClient.memory.write_claim`` surface (T4-2).
Strategy:
- Open a Pinecone connection with the operator-supplied API key.
- Read index stats to estimate total vector count (powers progress ETA).
- Iterate vector IDs in chunks of ``--page-size`` (Pinecone returns paged listings; page boundaries align to checkpoint windows).
- For each chunk, ``index.fetch(ids=[...])`` to retrieve full vector + metadata.
- Transform each Pinecone record into an OME :class:`Claim` —
see :func:
_pinecone_to_claim. - Per-record write via :meth:`OmeClient.memory.write_claim` (main's v1.0.0 surface; batch RPC is v0.2-gated).
The Pinecone embedding dimension travels through unchanged — OME's ``embedding_hash`` is the SHA-256 of the packed float32 vector bytes; the vector itself is NOT preserved on the Claim body unless the operator passes ``--reembed=false`` (default).
Per the spec §3 acceptance criteria: ``--dry-run``, ``--target-namespace``, ``--batch-size``, ``--resume-from-checkpoint``, ``--checkpoint-out``, ``--verbose`` are all supported via the shared :mod:`_common` plumbing.
References:
- Pinecone Python SDK: https://docs.pinecone.io/reference/python-client
- OME Claim wire types: ``ome-proto/proto/ome/v1/ome_service.proto`` (T14 single source of truth merged in ome-proto PR #6)
Constants
| Name | Value |
| --- | --- |
| SOURCE_NAME | 'pinecone' |
Functions
from_pinecone
def from_pinecone(api_key: str = typer.Option('', '--api-key', envvar='PINECONE_API_KEY', help='Pinecone API key. **Prefer** `PINECONE_API_KEY` env var over the flag — flag values land in shell history.'), index: str = typer.Option(..., '--index', help="Pinecone index name (e.g. 'production-embeddings')."), pinecone_namespace: str = typer.Option('', '--pinecone-namespace', help="Pinecone-side namespace within the index. Defaults to the empty namespace (Pinecone's default). NOT the same as --target-namespace, which is the OME-side destination."), page_size: int = typer.Option(100, '--page-size', help='Pinecone list/fetch page size. Pinecone caps this at 100.'), reembed: bool = typer.Option(False, '--reembed', help="Re-embed records using OME's default embedder instead of preserving the Pinecone vector. Default: preserve (zero-cost migration)."), ome_endpoint: str = typer.Option(..., '--ome-endpoint', help="OME gRPC target as 'host:port' (e.g. 'api.osyra.ai:443'). Use --insecure to dial without TLS for local dev."), ome_token: str = typer.Option('', '--ome-token', envvar='OSYRA_OME_TOKEN', help='OME bearer token. Prefer OSYRA_OME_TOKEN env var.'), ome_workspace: str = typer.Option(..., '--ome-workspace', help="OME workspace ID (e.g. 'ws-prod')."), ome_workspace_tier: str | None = typer.Option(None, '--ome-workspace-tier', help='Deprecated: Edge derives the effective tier server-side.', hidden=True), target_namespace: str = typer.Option(..., '--target-namespace', help='OME logical namespace to write into. REQUIRED — no default, to prevent accidental writes into the workspace root.'), batch_size: int = typer.Option(100, '--batch-size', help='Records per checkpoint / progress-advance window. Writes are STILL per-record on the wire (T4-2 OmeClient surface).'), dry_run: bool = typer.Option(False, '--dry-run', help='Read + transform + validate; do NOT write to OME. Prints the summary report as if the run had completed.'), resume_from_checkpoint: Path = typer.Option(None, '--resume-from-checkpoint', help='Resume from a previously-emitted checkpoint file.'), checkpoint_out: Path = typer.Option(None, '--checkpoint-out', help='Emit progress checkpoints to this file after each batch.'), audit_file: Path = typer.Option(None, '--audit-file', help='Append per-record JSON audit events to this file. Default: stderr.'), include_pii: bool = typer.Option(False, '--include-pii', help='Log source IDs and metadata verbatim. Default: false (IDs are hashed; metadata is scanned for email/phone/SSN patterns and redacted in logs).'), strict_audit: bool = typer.Option(False, '--strict-audit', help='Treat audit-emission failures as fatal. Default: warn-and-continue.'), verbose: bool = typer.Option(False, '--verbose', '-v', help='Print per-record progress to stderr. Default: progress bar only.'), insecure: bool = typer.Option(False, '--insecure', help='Dial an insecure gRPC channel (no TLS). ONLY for local dev against a non-TLS test server.')) -> NoneMigrate vector embeddings and metadata from Pinecone into OME.
Example:
export PINECONE_API_KEY=pcsk_...
export OSYRA_OME_TOKEN=osyra_...
osyra-ome migrate from-pinecone
--index production-embeddings
--ome-endpoint api.osyra.ai:443
--ome-workspace ws-prod
--target-namespace ome-prod
--dry-run