Architecture

dcupl follows a modular architecture where you can use as much or as little as you need. At its simplest, you use the SDK alone. For more complex needs, you add the Console platform.

High-Level Overview

flowchart TB
  subgraph opensource["Open Source"]
    SDK["@dcupl/core
Query Engine"]
    LOADER["@dcupl/loader
Data Loading"]
    CONNECT["@dcupl/connect
Debugging"]
  end

  subgraph saas["Console Platform (Optional)"]
    CONSOLE["Console UI
Model & Workflow Editor"]
    RUNNER["Workflow Runner
Data Transformation"]
    RAPI["RAPI Runner
Server-side Queries"]
  end

  subgraph storage["Storage"]
    CDN["CDN
JSON Files"]
    SOURCES["Data Sources
CSV, APIs, etc."]
  end

  SOURCES --> RUNNER
  RUNNER --> CDN
  CDN --> LOADER
  LOADER --> SDK
  CONSOLE --> RUNNER
  CONSOLE --> RAPI
  SDK --> APP[Your Application]
  CONNECT -.-> CONSOLE

SDK Architecture

The SDK is organized as a TypeScript monorepo with several packages:

Package Hierarchy

flowchart TD
  COMMON["@dcupl/common
Utilities & Algorithms"]
  CORE["@dcupl/core
Main Query Engine"]
  LOADER["@dcupl/loader
Data Ingestion"]
  CONNECT["@dcupl/connect
Console Integration"]

  COMMON --> CORE
  CORE --> LOADER
  CORE --> CONNECT

@dcupl/common

Foundation package containing shared utilities:

• QueryBuilder - Fluent API for building queries
• QueryManager - Query evaluation with operators
• DependencyGraph - Model relationship management
• IndicesController - Performance indexing

@dcupl/core

The main query engine and entry point:

• Dcupl - Root class for initialization
• ModelParser - Processes model definitions
• DcuplList - Filtered data views
• Catalog - Query interface with filtering, sorting, facets

@dcupl/loader

Data ingestion and configuration loading:

• DcuplAppLoader - Loads configuration from Console
• CsvParser - CSV parsing with transformations
• Variable interpolation for environment-specific URLs

@dcupl/connect

Development debugging tool:

• Real-time SDK inspection in Console UI
• Query monitoring and performance tracking
• State visualization during development

Execution Contexts

The same SDK code runs in multiple environments:

flowchart LR
  subgraph browser["Browser"]
    B_SDK["dcupl SDK"]
    B_DATA["Data in Memory"]
    B_SDK --> B_DATA
  end

  subgraph node["Node.js"]
    N_SDK["dcupl SDK"]
    N_DATA["Data in Memory"]
    N_SDK --> N_DATA
  end

  subgraph rapi["RAPI - Runtime API (Console)"]
    R_SDK["dcupl SDK"]
    R_DATA["Data in Memory"]
    R_SDK --> R_DATA
  end

  USER[User] --> browser
  SSR[SSR Framework] --> node
  API[API Request] --> rapi

Browser (Client-Side)

The most common use case. Data loads into the browser, and all queries execute locally.

Benefits:

• Zero server cost for queries
• Instant user interactions
• Works offline after initial load

Best for: Datasets under 50,000 items, interactive filtering interfaces

Node.js (Server-Side)

The same SDK runs on the server for SSR or batch processing.

Benefits:

• SEO-friendly initial renders
• Can handle larger datasets with more memory
• Server-to-server data processing

Best for: Initial page renders, large dataset processing, API endpoints

Hybrid Pattern

Combine both for optimal results:

sequenceDiagram
  participant Browser
  participant Server
  participant Data

  Server->>Data: Load data
  Server->>Server: Query for initial render
  Server->>Browser: HTML with data
  Browser->>Browser: Hydrate dcupl SDK
  Browser->>Browser: All subsequent queries (local)

The server renders the initial page for SEO. Then the browser takes over, handling all user interactions locally without additional server requests.

Console Platform Architecture

The Console is an optional SaaS platform that extends dcupl capabilities.

flowchart TB
  subgraph console["Console Platform"]
    UI["Angular Frontend"]
    API["NestJS Backend"]
    DB[(Firestore)]

    UI --> API
    API --> DB
  end

  subgraph runners["Execution"]
    WF["Workflow Runner"]
    RP["RAPI Runner"]
  end

  API --> WF
  API --> RP

  subgraph storage["Output"]
    GCS["Cloud Storage"]
    CDN["CDN"]
  end

  WF --> GCS
  GCS --> CDN

Console UI

Angular-based web application for:

• Visual model editor
• Workflow designer
• Data exploration
• Project management

Console API

NestJS backend providing:

• Authentication (Firebase Auth)
• Project and file management
• Workflow orchestration
• Runner coordination

Workflow Runner

Executes data transformation workflows:

• Fetches data from external sources
• Applies transformations
• Validates against models
• Stores processed JSON

RAPI Runner

Deploys server-side query endpoints:

• Exposes SDK queries as REST APIs
• Handles large datasets
• Manages authentication and rate limiting

Data Storage

dcupl uses a JSON-first architecture for portability:

flowchart LR
  subgraph config["Configuration"]
    LC["dcupl.lc.json
LoaderConfiguration"]
    MODELS["*.dcupl.json
Model Definitions"]
  end

  subgraph data["Data"]
    JSON["*.json
Data Files"]
  end

  LC --> MODELS
  LC --> JSON
  MODELS --> SDK
  JSON --> SDK

LoaderConfiguration

The central configuration file (dcupl.lc.json) defines:

• Resources - Model and data file URLs with tags
• Applications - Which resources to load for each app
• Environments - Environment-specific variables
• Variables - URL interpolation values

Model Files

JSON files defining data structure:

{
  "key": "Product",
  "properties": [
    { "key": "name", "type": "string" },
    { "key": "price", "type": "int" }
  ]
}

Data Files

JSON arrays of data matching model structure:

[
  { "key": "p1", "name": "Laptop", "price": 999 },
  { "key": "p2", "name": "Mouse", "price": 29 }
]

Integration Patterns

Pattern 1: SDK Only

Simplest setup for static or developer-managed data.

import { Dcupl } from '@dcupl/core';

const dcupl = new Dcupl();
dcupl.models.set(productModel);
dcupl.data.set(productData, { model: 'Product' });
await dcupl.init();

Pattern 2: SDK + Loader

Load configuration and data from URLs (Console or self-hosted).

import { Dcupl } from '@dcupl/core';
import { DcuplAppLoader } from '@dcupl/loader';

const dcupl = new Dcupl();
const loader = new DcuplAppLoader();
dcupl.loaders.add(loader);

await loader.config.fetch({
  projectId: 'your-project',
  apiKey: 'your-api-key',
});
await loader.process({ applicationKey: 'default' });
await dcupl.init();

Pattern 3: SDK + Console + RAPI

Full platform with server-side queries for large datasets.

flowchart LR
  CLIENT[Browser SDK] --> |"Small datasets"| LOCAL[Local Query]
  CLIENT --> |"Large datasets"| RAPI[RAPI Endpoint]
  RAPI --> SERVER[Server SDK]

Key Takeaways

1. Modular design - Use only what you need
2. Universal SDK - Same code runs everywhere
3. JSON-first - All configuration is portable
4. Layered architecture - SDK alone, with Loader, or full Console

Next Steps

• Data Flow - See how data moves through the system
• Glossary - Understand dcupl terminology
• SDK Quick Start - Build your first application