Data Flow

Understanding how data flows through dcupl helps you make better architectural decisions. This page traces data from external sources to query results.

Overview

Data in dcupl flows through four stages:

flowchart LR
  SOURCE["1. Source
CSV, JSON, API"]
  LOAD["2. Load
Parse & Transform"]
  STORE["3. Store
In-Memory"]
  QUERY["4. Query
Filter & Retrieve"]

  SOURCE --> LOAD
  LOAD --> STORE
  STORE --> QUERY

Each stage has specific responsibilities and components.

Stage 1: Data Sources

Data originates from external sources. dcupl supports various formats:

Format	Description	Use Case
JSON	Native format, no parsing needed	APIs, exports
CSV	Comma-separated values	Spreadsheets, databases
API	REST endpoints	Live data

Data sources can be:

Static files - Hosted on CDN or bundled with your app
Dynamic endpoints - APIs that return fresh data
Workflow outputs - Transformed data from Console workflows

Stage 2: Loading

The loading stage parses raw data and transforms it according to model definitions.

flowchart TD
  subgraph loading["Loading Stage"]
    FETCH["Fetch Data"]
    PARSE["Parse Format"]
    TRANSFORM["Apply Transformers"]
    VALIDATE["Validate Against Model"]
  end

  SOURCE[Data Source] --> FETCH
  FETCH --> PARSE
  PARSE --> TRANSFORM
  TRANSFORM --> VALIDATE
  VALIDATE --> STORE[In-Memory Store]

DcuplAppLoader

The DcuplAppLoader handles loading from a LoaderConfiguration:

import { DcuplAppLoader } from '@dcupl/loader';

const loader = new DcuplAppLoader();

// 1. Fetch configuration
await loader.config.fetch({
  projectId: 'your-project',
  apiKey: 'your-api-key',
});

// 2. Process application resources
await loader.process({ applicationKey: 'default' });

The loader reads dcupl.lc.json which defines:

{
  "resources": [
    {
      "url": "${baseUrl}/models/product.dcupl.json",
      "type": "model",
      "tags": ["product"]
    },
    {
      "url": "${baseUrl}/data/products.json",
      "type": "data",
      "model": "Product",
      "tags": ["product"]
    }
  ],
  "applications": [
    {
      "key": "default",
      "resourceTags": ["product"]
    }
  ]
}

Manual Loading

You can also load data directly without a LoaderConfiguration:

const dcupl = new Dcupl();

// Add model
dcupl.models.set({
  key: 'Product',
  properties: [
    { key: 'name', type: 'string' },
    { key: 'price', type: 'int' },
  ],
});

// Add data
dcupl.data.set([
  { key: 'p1', name: 'Laptop', price: 999 },
  { key: 'p2', name: 'Mouse', price: 29 },
], { model: 'Product' });

Data Transformations

During loading, dcupl applies transformations defined in models:

Type Coercion - Converts strings to proper types

// Raw CSV data: "29.99"
// Model property: { key: 'price', type: 'float' }
// Result: 29.99 (number)

Array Conversion - Splits delimited strings into arrays

// Raw data: "red,blue,green"
// Property: { key: 'colors', type: 'Array<string>', separator: ',' }
// Result: ["red", "blue", "green"]

Value Mapping - Translates codes to readable values

// Raw data: "1"
// Mapping: { from: ["1"], to: "active" }
// Result: "active"

Stage 3: In-Memory Storage

After loading, data lives in memory organized by model.

flowchart TB
  subgraph memory["In-Memory Store"]
    subgraph product["Product Model"]
      P_DATA["Data Map"]
      P_INDEX["Indices"]
    end
    subgraph category["Category Model"]
      C_DATA["Data Map"]
      C_INDEX["Indices"]
    end
    REFS["Reference Graph"]
  end

  product --> REFS
  category --> REFS

Data Maps

Each model stores items in a Map keyed by item identifier:

// Internal structure (simplified)
modelData = new Map([
  ['p1', { key: 'p1', name: 'Laptop', price: 999 }],
  ['p2', { key: 'p2', name: 'Mouse', price: 29 }],
]);

Indices

dcupl automatically creates indices for efficient querying:

// Index for 'category' property
categoryIndex = {
  Electronics: ['p1', 'p3', 'p5'],
  Accessories: ['p2', 'p4'],
};

Indices enable fast lookups without scanning all items.

Reference Graph

The DependencyGraph tracks relationships between models:

graph LR
  Product -->|"category"| Category
  Order -->|"customer"| Customer
  Order -->|"items"| Product

This enables deep queries across related models.

Stage 4: Querying

Queries retrieve and filter data from the in-memory store.

flowchart TD
  subgraph query["Query Execution"]
    CREATE["Create List"]
    BUILD["Build Query"]
    EXECUTE["Execute Filter"]
    RESULT["Return Results"]
  end

  STORE[In-Memory Store] --> CREATE
  CREATE --> BUILD
  BUILD --> EXECUTE
  EXECUTE --> RESULT

Lists

A List is a filtered view of a model's data:

const productList = dcupl.lists.create({
  modelKey: 'Product',
});

Multiple lists can exist for the same model, each with different filters.

Query Building

Build queries using the Catalog API:

// Add filter conditions
productList.catalog.query.addCondition({
  attribute: 'price',
  operator: 'gte',
  value: 100,
});

productList.catalog.query.addCondition({
  attribute: 'category',
  operator: 'eq',
  value: 'Electronics',
});

Query Execution

The QueryManager evaluates conditions against indexed data:

flowchart LR
  QUERY["Query Conditions"]
  INDEX["Check Indices"]
  FILTER["Filter Items"]
  SORT["Apply Sorting"]
  PAGE["Apply Pagination"]
  ITEMS["Return Items"]

  QUERY --> INDEX
  INDEX --> FILTER
  FILTER --> SORT
  SORT --> PAGE
  PAGE --> ITEMS

// Execute and get results
const items = productList.catalog.query.items();

Additional Operations

Beyond filtering, the Catalog provides:

Facets - Count items by property values

const categoryFacets = productList.catalog.fn.facets({ attribute: 'category' });
const brandFacets = productList.catalog.fn.facets({ attribute: 'brand' });
// categoryFacets: [{ value: 'Electronics', count: 5 }, { value: 'Clothing', count: 3 }]
// brandFacets: [{ value: 'BrandA', count: 4 }, ...]

Aggregations - Calculate statistics

const stats = productList.catalog.fn.aggregate({
  attribute: 'price',
  types: ['min', 'max', 'avg', 'sum', 'count'],
});
// { min: 29, max: 999, avg: 450, sum: 4500, count: 10 }

Sorting and Pagination - Order and limit results

productList.catalog.query.sort({ attribute: 'price', direction: 'asc' });
productList.catalog.query.setPage(1);
productList.catalog.query.setLimit(20);

Complete Flow Example

Here is the full data flow from source to query result:

sequenceDiagram
  participant Source as Data Source
  participant Loader as DcuplAppLoader
  participant Store as In-Memory Store
  participant List as DcuplList
  participant App as Your Application

  Source->>Loader: JSON/CSV data
  Loader->>Loader: Parse & transform
  Loader->>Store: Store in memory
  Store->>Store: Build indices
  App->>List: Create list
  App->>List: Add conditions
  List->>Store: Execute query
  Store->>List: Filtered items
  List->>App: Results

import { Dcupl } from '@dcupl/core';
import { DcuplAppLoader } from '@dcupl/loader';

// 1. Initialize
const dcupl = new Dcupl();
const loader = new DcuplAppLoader();
dcupl.loaders.add(loader);

// 2. Load (Source -> Parse -> Store)
await loader.config.fetch({ projectId: 'demo' });
await loader.process({ applicationKey: 'default' });
await dcupl.init();

// 3. Query
const list = dcupl.lists.create({ modelKey: 'Product' });

list.catalog.query.addCondition({
  attribute: 'category',
  operator: 'eq',
  value: 'Electronics',
});

// 4. Results
const items = list.catalog.query.items();
const brandFacets = list.catalog.fn.facets({ attribute: 'brand' });

Performance Considerations

Loading

Load only needed data using resource tags
Use CDN for faster delivery
Consider data size (50K items is comfortable for browsers)

Storage

Indices are created automatically for filtered properties
Memory usage scales with data size
Reference resolution adds overhead for deeply nested queries

Querying

Index-based lookups are fast (O(1) or O(log n))
Complex queries with many conditions may scan more items
Facet calculations touch all matching items

Next Steps

Glossary - Understand dcupl terminology
Core Concepts - Deep dive into SDK concepts
Queries - Learn query operators