Product Sync Workflow

Overview

A 2-phase workflow: First discover and filter metafields, then sync selected data to the database while staying within GraphQL cost limits.

Workflow Chart

┌─────────────────────────────────────────────────────────────┐
│                    PRODUCT SYNC WORKFLOW                     │
└─────────────────────────────────────────────────────────────┘

═══════════════════════════════════════════════════════════════
PHASE 1: METAFIELD DISCOVERY & FILTERING
═══════════════════════════════════════════════════════════════

DISCOVERY: Fetch All Products (Metadata Only)
┌─────────────────────────────────────────────────────────────┐
│ Query: products(first: N) { id, metafields(first: 250) {   │
│         namespace, key, type, definition { id } } }         │
│ Purpose: Discover unique metafield keys                     │
│ Output: List of all metafield namespace.key combinations   │
│         (structured + unstructured)                         │
└─────────────────────────────────────────────────────────────┘
                        ↓
USER SELECTION: Choose Metafields to Include
┌─────────────────────────────────────────────────────────────┐
│ Interface: Checklist of discovered metafields               │
│ - Group by namespace                                        │
│ - Select/deselect metafields                                │
│ - Save preferences to database/config                      │
│ Output: Selected metafield keys (namespace.key)             │
└─────────────────────────────────────────────────────────────┘

═══════════════════════════════════════════════════════════════
PHASE 2: DATA SYNC (Using Selected Metafields)
═══════════════════════════════════════════════════════════════

STEP 1: Fetch Products (Basic Data)
┌─────────────────────────────────────────────────────────────┐
│ Query: products(first: N, after: $cursor)                    │
│ Fields: id, title, productType, tags, handle, status, etc.  │
│ Output: Array of Product GIDs + basic data                  │
│ Save: → products table                                      │
└─────────────────────────────────────────────────────────────┘
                        ↓
STEP 2: Fetch Variants (Basic Data)
┌─────────────────────────────────────────────────────────────┐
│ Query: products { variants(first: N, after: $cursor) }     │
│ Fields: id, title, price, sku, availableForSale, etc.      │
│ Input: Product GIDs from Step 1                            │
│ Output: Array of Variant GIDs + basic data                 │
│ Save: → variants table                                      │
└─────────────────────────────────────────────────────────────┘
                        ↓
STEP 3: Fetch Selected Product Metafields
┌─────────────────────────────────────────────────────────────┐
│ Query: product(id: $gid) {                                  │
│         metafields(first: N, namespace: $ns, key: $key) {  │
│           namespace, key, value, type, ownerType, etc.     │
│           references { ... } (union types)                 │
│         }                                                   │
│ Fields: namespace, key, value, type, ownerType, etc.       │
│ Input: Product GIDs from Step 1                            │
│        + Selected metafield keys from Phase 1               │
│ Output: Only selected metafield data with references        │
│ Save: → product_metafields table                            │
└─────────────────────────────────────────────────────────────┘
                        ↓
STEP 4: Fetch Selected Variant Metafields
┌─────────────────────────────────────────────────────────────┐
│ Query: productVariant(id: $gid) {                          │
│         metafields(first: N, namespace: $ns, key: $key) {  │
│           namespace, key, value, type, ownerType, etc.     │
│           references { ... } (union types)                 │
│         }                                                   │
│ Fields: namespace, key, value, type, ownerType, etc.       │
│ Input: Variant GIDs from Step 2                            │
│        + Selected metafield keys from Phase 1              │
│ Output: Only selected metafield data with references       │
│ Save: → variant_metafields table                           │
└─────────────────────────────────────────────────────────────┘

Summary

Purpose: Discover metafields first, then sync only selected Shopify store data to local database.

Strategy:

• Phase 1: Discovery + user selection (happens first)
• Phase 2: Sequential sync execution (steps must complete in order)
• Cursor pagination for each step
• Stay under 1000 cost limit per query
• Use GID arrays to link related data
• Only fetch metafields user selected

Benefits:

• ✅ User controls what data is stored (filter first)
• ✅ Reduces query cost (only fetch selected metafields)
• ✅ Faster sync times
• ✅ Avoids GraphQL cost limit errors
• ✅ Retryable steps (can resume from any step)
• ✅ Clear separation of concerns
• ✅ Efficient cursor pagination

Implementation Notes

• Discovery can be cached - Only re-run when needed (store updates, new products)
• Store preferences - Save selected metafield keys to database/config
• GraphQL filters - Use metafields(namespace: $ns, key: $key) to fetch specific metafields
• Batch queries - Can query multiple products/variants with selected metafields in single query (if under cost limit)