fabric-systems Fabric Schema Reference
About Fabric
multi-cloud data fabric
Fabric continuously syncs data scattered across e-commerce, stores, customer touchpoints, and analytics, keeping it always flowing and consistent. This document describes the core schema on GCP BigQuery.
Principles
- Always flowing — batch and near-real-time pipelines that do not stall data
- Always in sync — aggregate while preserving consistency, not one-way dumps from sources
- Never down — pipelines designed for multi-cloud resilience
- Observable — monitor sync success, failures, and lag (e.g. Datadog)
Initial scope (CDP)
Phase 1 builds a Customer Data Platform (CDP) on GCP. Commerce data from Shopify, Smaregi, and similar systems is consolidated in BigQuery so customers, orders, products, and store transactions can be analyzed together.
About this reference
- Project:
fabric-493607 - Datasets:
shopify, smaregi - Tables: 6 (auto-generated from DDL)
- Sources: Shopify Admin API / Smaregi Platform API
- Load pattern: scheduled upsert via Cloud Run + Scheduler
Data flow (conceptual)
[ Shopify ] ──→ shopify_orders / shopify_customers / shopify_products / ...
[ Smaregi POS ] ──→ smaregi_transactions
│
▼
[ GCP BigQuery ]
│
▼
Analytics · CDP · future AWS / multi-cloud
ERD
// scroll to view full diagram
// drag to pan · scroll to zoom · default 100%
Schema
shopify.shopify_customers
Shopify Admin GraphQL customers (payload + parse columns at insert)
| Column | Type | NOT NULL | PK candidate |
|---|---|---|---|
customer_id | INT64 | yes | PK |
customer_gid | STRING | ||
email | STRING | ||
first_name | STRING | ||
last_name | STRING | ||
phone | STRING | ||
state | STRING | ||
verified_email | BOOL | ||
accepts_marketing | BOOL | ||
tags_json | STRING | ||
tags_count | INT64 | ||
orders_count | INT64 | ||
total_spent | NUMERIC | ||
currency | STRING | ||
note | STRING | ||
tax_exempt | BOOL | ||
tax_exemptions_json | STRING | ||
last_order_id | INT64 | ||
last_order_gid | STRING | ||
last_order_name | STRING | ||
email_marketing_consent_json | STRING | ||
sms_marketing_consent_json | STRING | ||
addresses_json | STRING | ||
addresses_count | INT64 | ||
default_address_json | STRING | ||
created_at | TIMESTAMP | ||
updated_at | TIMESTAMP | ||
synced_at | TIMESTAMP | yes | |
payload | STRING | yes |
shopify.shopify_order_line_items
Shopify Admin GraphQL order line items (payload + parse columns at insert)
| Column | Type | NOT NULL | PK candidate |
|---|---|---|---|
tenant_id | STRING | ||
shop_id | STRING | yes | PK |
order_id | STRING | yes | PK |
order_numeric_id | INT64 | ||
order_name | STRING | ||
order_created_at | TIMESTAMP | ||
line_no | INT64 | yes | PK |
line_item_gid | STRING | ||
product_gid | STRING | ||
product_id | INT64 | ||
variant_gid | STRING | ||
variant_id | INT64 | ||
sku | STRING | ||
vendor | STRING | ||
variant_title | STRING | ||
name | STRING | ||
quantity | INT64 | ||
current_quantity | INT64 | ||
unfulfilled_quantity | INT64 | ||
refundable_quantity | INT64 | ||
taxable | BOOL | ||
requires_shipping | BOOL | ||
price | NUMERIC | ||
discounted_unit_price | NUMERIC | ||
discounted_total | NUMERIC | ||
currency | STRING | ||
title | STRING | ||
line_item_json | STRING | ||
synced_at | TIMESTAMP | yes | |
payload | STRING | yes |
shopify.shopify_orders
Shopify Admin GraphQL orders (payload + parse columns at insert)
| Column | Type | NOT NULL | PK candidate |
|---|---|---|---|
tenant_id | STRING | ||
shop_id | STRING | yes | |
order_id | STRING | yes | PK |
order_numeric_id | INT64 | ||
customer_gid | STRING | ||
customer_numeric_id | INT64 | ||
customer_email | STRING | ||
source_name | STRING | ||
channel_handle | STRING | ||
order_name | STRING | ||
order_number | INT64 | ||
email | STRING | ||
phone | STRING | ||
note | STRING | ||
currency | STRING | ||
presentment_currency | STRING | ||
total_price | NUMERIC | ||
subtotal_price | NUMERIC | ||
total_tax | NUMERIC | ||
total_discount | NUMERIC | ||
total_shipping | NUMERIC | ||
financial_status | STRING | ||
fulfillment_status | STRING | ||
cancelled_at | TIMESTAMP | ||
cancel_reason | STRING | ||
processed_at | TIMESTAMP | ||
updated_at | TIMESTAMP | ||
created_at | TIMESTAMP | ||
closed | BOOL | ||
test | BOOL | ||
taxes_included | BOOL | ||
discount_codes_json | STRING | ||
billing_address_json | STRING | ||
shipping_address_json | STRING | ||
tags_json | STRING | ||
tags_count | INT64 | ||
line_items_count | INT64 | ||
total_quantity | INT64 | ||
line_items_json | STRING | ||
synced_at | TIMESTAMP | yes | |
payload | STRING | yes |
shopify.shopify_products
Shopify Admin GraphQL products (payload + parse columns at insert)
| Column | Type | NOT NULL | PK candidate |
|---|---|---|---|
product_id | INT64 | yes | PK |
product_gid | STRING | ||
title | STRING | ||
handle | STRING | ||
product_type | STRING | ||
vendor | STRING | ||
status | STRING | ||
tags_json | STRING | ||
tags_count | INT64 | ||
description_html | STRING | ||
published_at | TIMESTAMP | ||
total_inventory | INT64 | ||
first_price | STRING | ||
options_json | STRING | ||
media_json | STRING | ||
price_range_json | STRING | ||
featured_image_url | STRING | ||
variants_count | INT64 | ||
created_at | TIMESTAMP | ||
updated_at | TIMESTAMP | ||
synced_at | TIMESTAMP | yes | |
payload | STRING | yes |
shopify.shopify_variants
Shopify Admin GraphQL variants (payload + parse columns at insert)
| Column | Type | NOT NULL | PK candidate |
|---|---|---|---|
product_id | INT64 | yes | PK |
variant_id | INT64 | yes | PK |
product_gid | STRING | ||
variant_gid | STRING | ||
product_status | STRING | ||
product_title | STRING | ||
variant_title | STRING | ||
sku | STRING | ||
option1 | STRING | ||
option2 | STRING | ||
option3 | STRING | ||
selected_options_json | STRING | ||
barcode | STRING | ||
price | STRING | ||
compare_at_price | STRING | ||
inventory_quantity | INT64 | ||
inventory_item_id | INT64 | ||
inventory_item_gid | STRING | ||
inventory_policy | STRING | ||
requires_shipping | BOOL | ||
taxable | BOOL | ||
position | INT64 | ||
variant_created_at | TIMESTAMP | ||
variant_updated_at | TIMESTAMP | ||
product_created_at | TIMESTAMP | ||
handle | STRING | ||
vendor | STRING | ||
product_type | STRING | ||
variant_image_url | STRING | ||
synced_at | TIMESTAMP | yes | |
payload | STRING | yes |
smaregi.smaregi_transactions
Smaregi Platform API transactions (upsert by transaction_head_id)
| Column | Type | NOT NULL | PK candidate |
|---|---|---|---|
tenant_id | STRING | ||
contract_id | STRING | yes | PK |
transaction_head_id | STRING | yes | PK |
transaction_date_time | TIMESTAMP | ||
terminal_tran_date_time | TIMESTAMP | ||
enter_date_time | TIMESTAMP | ||
adjustment_date_time | TIMESTAMP | ||
upd_date_time | TIMESTAMP | ||
sum_date | DATE | ||
store_id | STRING | ||
store_code | STRING | ||
terminal_id | STRING | ||
terminal_tran_id | STRING | ||
staff_id | STRING | ||
staff_code | STRING | ||
staff_name | STRING | ||
customer_id | STRING | ||
customer_code | STRING | ||
transaction_head_division | STRING | ||
cancel_division | STRING | ||
sell_division | STRING | ||
sum_division | STRING | ||
sales_head_division | STRING | ||
subtotal | STRING | ||
total | STRING | ||
tax_include | STRING | ||
tax_exclude | STRING | ||
amount | STRING | ||
return_amount | STRING | ||
cost_total | STRING | ||
deposit | STRING | ||
change | STRING | ||
in_tax_sales_total | STRING | ||
guest_numbers | STRING | ||
transaction_uuid | STRING | ||
exchange_ticket_no | STRING | ||
tax_rate | STRING | ||
synced_at | TIMESTAMP | yes | |
payload | STRING | yes | |
details | STRING |
DDL Sources
data/bigquery/ddl/shopify_customers.sqldata/bigquery/ddl/shopify_order_line_items.sqldata/bigquery/ddl/shopify_orders.sqldata/bigquery/ddl/shopify_products.sqldata/bigquery/ddl/shopify_variants.sqldata/bigquery/ddl/smaregi_transactions.sql