Performance Best Practices

This guide covers strategies to optimize dcupl for speed and efficiency. Learn when to use partial updates, how to structure queries for maximum performance, and how to manage memory effectively.

Performance Principles

High-performing dcupl applications follow these principles:

Initialize once - Call init() once, then use incremental updates
Query indexed properties - Filter on properties that are indexed
Reuse lists - Clear and reuse lists instead of creating new ones
Update incrementally - Use partial updates for changes
Limit result sets - Use pagination for large datasets

Initialization Optimization

Initialize Once, Update Incrementally

The init() method builds indexes and is expensive. Call it once at startup, then use incremental updates.

const dcupl = new Dcupl();

// Define models
dcupl.models.set(productModel);
dcupl.models.set(categoryModel);

// Load initial data
dcupl.data.set(initialProducts, { model: 'Product' });
dcupl.data.set(categories, { model: 'Category' });

// Initialize once
await dcupl.init();

// Later: Use incremental updates (not init)
dcupl.data.update({
  model: 'Product',
  data: [{ key: 'p1', price: 150 }],
});

// Re-initializing on every update
async function updateProduct(product) {
  dcupl.data.set([product], { model: 'Product' });
  await dcupl.init(); // Expensive! Rebuilds all indexes
}

Load Data Progressively

Load critical data first to reduce time-to-interactive:

progressive-loading.ts

async function initializeApp() {
  const dcupl = new Dcupl();
  dcupl.models.set(productModel);
  dcupl.models.set(categoryModel);

  // 1. Load critical data first
  const products = await fetch('/api/products?limit=100').then((r) => r.json());
  dcupl.data.set(products, { model: 'Product' });
  await dcupl.init();

  // 2. App is now usable
  renderUI();

  // 3. Load remaining data in background
  const allProducts = await fetch('/api/products').then((r) => r.json());
  dcupl.data.set(allProducts, { model: 'Product' });
}

Update Optimization

Use Partial Updates

Partial updates are 10-100x faster than full updates for large datasets.

// Update specific items only
dcupl.data.update([
  { key: 'p1', price: 150 },
  { key: 'p2', stock: 25 },
], { model: 'Product' });

// Replaces all data and rebuilds indexes
dcupl.data.set(allProducts, { model: 'Product' });
await dcupl.init();

Performance comparison (10,000 items):

Operation	Time	Relative Speed
Full update	~500ms	1x
Partial update (1 item)	~2ms	250x faster
Partial update (10 items)	~5ms	100x faster
Partial update (100 items)	~20ms	25x faster

Use Targeted Updates

For maximum performance, update only the properties that changed:

targeted-update.ts

// Update only specific properties on specific items
dcupl.data.update([
  { key: 'p1', price: 150 },  // Only include changed properties
], { model: 'Product' });

Batch Multiple Updates

Combine multiple changes into a single update call:

// Single batched update
dcupl.data.update([
  { key: 'p1', price: 150 },
  { key: 'p2', price: 250 },
  { key: 'p3', price: 350 },
], { model: 'Product' });

// Multiple individual updates
for (const product of products) {
  dcupl.data.update([product], { model: 'Product' });
}

Delete Efficiently

Remove items without full reinitialization:

delete.ts

// Remove specific items
dcupl.data.remove([
  { key: 'p1' },
  { key: 'p2' },
  { key: 'p3' },
], { model: 'Product' });

// Clear all data for a model
dcupl.data.reset({ model: 'Product' });

Query Optimization

Query Indexed Properties

All properties are automatically indexed for filtering. Queries on indexed properties are fast.

indexed-query.ts

// Fast: All standard properties are indexed
productList.catalog.query.addCondition({
  attribute: 'category',
  operator: 'eq',
  value: 'Electronics',
});

productList.catalog.query.addCondition({
  attribute: 'price',
  operator: 'gte',
  value: 100,
});

Order Filters by Selectivity

Put the most selective filters first to reduce the candidate set quickly:

// Most selective filter first
productList.catalog.query.addCondition({
  groupKey: 'filters',
  groupType: 'and',
  queries: [
    { attribute: 'sku', operator: 'eq', value: 'ABC123' }, // Very selective
    { attribute: 'inStock', operator: 'eq', value: true }, // Less selective
    { attribute: 'category', operator: 'eq', value: 'Books' }, // Least selective
  ],
});

// Least selective filter first
productList.catalog.query.addCondition({
  groupKey: 'filters',
  groupType: 'and',
  queries: [
    { attribute: 'category', operator: 'eq', value: 'Books' }, // Many matches
    { attribute: 'inStock', operator: 'eq', value: true }, // Still many
    { attribute: 'sku', operator: 'eq', value: 'ABC123' }, // Finally narrow
  ],
});

Use Derived Properties for Deep Queries

Deep queries across references are slow. Use derived properties to flatten access:

// Define derived property
const orderModel: ModelDefinition = {
  key: 'Order',
  properties: [
    { key: 'orderNumber', type: 'string' },
    {
      key: 'customerCountry',
      type: 'string',
      derive: {
        localReference: 'customer',
        remoteProperty: 'country',
      },
    },
  ],
  references: [{ key: 'customer', model: 'Customer', type: 'singleValued' }],
};

// Fast: Query derived property
orderList.catalog.query.addCondition({
  attribute: 'customerCountry',
  operator: 'eq',
  value: 'USA',
});

// Slow: Deep query across references
orderList.catalog.query.addCondition({
  attribute: 'customer.country',
  operator: 'eq',
  value: 'USA',
});

Limit Deep Query Depth

If you must use deep queries, limit depth to 2-3 levels:

Depth	Relative Speed
1 level	Baseline
2 levels	4-6x slower
3 levels	6-9x slower
4+ levels	10-15x+ slower

Use Pagination

For large result sets, always use pagination:

pagination.ts

const productList = dcupl.lists.create({ modelKey: 'Product' });

// Apply pagination via query options
productList.catalog.query.applyOptions({
  limit: 50,
  offset: 0,
});

// Get first page
const firstPage = productList.catalog.query.items();

// Navigate to next page
productList.catalog.query.applyOptions({
  limit: 50,
  offset: 50,
});
const secondPage = productList.catalog.query.items();

Avoid Full Table Scans

Always apply filters to narrow results:

// Filtered query
productList.catalog.query.addCondition({
  attribute: 'category',
  operator: 'eq',
  value: 'Electronics',
});
const electronics = productList.catalog.query.items();

// Unfiltered query (scans everything)
const allItems = productList.catalog.query.items();
const electronics = allItems.filter((p) => p.category === 'Electronics');

List Management

Reuse Lists

Create lists once and reuse them. Creating new lists has overhead.

// Create once, reuse
const productList = dcupl.lists.create({ modelKey: 'Product' });

function searchProducts(category: string) {
  productList.catalog.query.clear();
  productList.catalog.query.addCondition({
    attribute: 'category',
    operator: 'eq',
    value: category,
  });
  return productList.catalog.query.items();
}

// Creating new list for every search
function searchProducts(category: string) {
  const list = dcupl.lists.create({ modelKey: 'Product' });
  list.catalog.query.addCondition({
    attribute: 'category',
    operator: 'eq',
    value: category,
  });
  return list.catalog.query.items();
}

Destroy Unused Lists

Clean up lists when no longer needed to free memory:

cleanup.ts

const tempList = dcupl.lists.create({ modelKey: 'Product' });

// Use the list
const results = tempList.catalog.query.items();

// Clean up when done
tempList.destroy();

Use List Pooling for Dynamic Lists

If you frequently create and destroy lists, use a pool:

list-pool.ts

class ListPool {
  private pool = new Map<string, DcuplList[]>();
  private dcupl: Dcupl;

  constructor(dcupl: Dcupl) {
    this.dcupl = dcupl;
  }

  get(modelKey: string): DcuplList {
    const pool = this.pool.get(modelKey) || [];
    if (pool.length > 0) {
      return pool.pop()!;
    }
    return this.dcupl.lists.create({ modelKey });
  }

  release(list: DcuplList) {
    list.catalog.query.clear();
    const modelKey = list.model.key;
    const pool = this.pool.get(modelKey) || [];
    pool.push(list);
    this.pool.set(modelKey, pool);
  }
}

// Usage
const pool = new ListPool(dcupl);
const list = pool.get('Product');
// ... use list ...
pool.release(list);

Memory Optimization

Load Only Needed Data

Filter data at the source when possible:

// Load only active products
const activeProducts = await fetch('/api/products?active=true').then((r) => r.json());
dcupl.data.set(activeProducts, { model: 'Product' });

// Load everything, filter later
const allProducts = await fetch('/api/products').then((r) => r.json());
dcupl.data.set(allProducts, { model: 'Product' });

Remove Unused Properties

Strip properties you do not need before loading:

strip-properties.ts

const rawProducts = await fetch('/api/products').then((r) => r.json());

// Keep only needed properties
const products = rawProducts.map((p) => ({
  key: p.id,
  name: p.name,
  price: p.price,
  category: p.category,
  // Exclude: p.internalNotes, p.legacyId, p.rawData
}));

dcupl.data.set(products, { model: 'Product' });

Clear Stale Data

Remove data that is no longer needed:

clear-data.ts

// Remove specific items
dcupl.data.remove([
  { key: 'old-product-1' },
  { key: 'old-product-2' },
], { model: 'Product' });

// Clear all data for a model
dcupl.data.reset({ model: 'TempData' });

Large Dataset Handling

Load Data in Chunks

For very large datasets (100k+ items), load in chunks:

chunked-loading.ts

async function loadLargeDataset() {
  const chunkSize = 10000;
  let offset = 0;
  let isFirstChunk = true;

  while (true) {
    const chunk = await fetch(`/api/products?offset=${offset}&limit=${chunkSize}`).then((r) =>
      r.json()
    );

    if (chunk.length === 0) break;

    dcupl.data.set(chunk, {
      model: 'Product',
      replace: isFirstChunk,
    });

    offset += chunkSize;
    isFirstChunk = false;

    // Optional: Yield to UI between chunks
    await new Promise((resolve) => setTimeout(resolve, 0));
  }

  await dcupl.init();
}

Use Virtual Scrolling

For displaying large lists, use virtual scrolling to render only visible items:

virtual-scroll.tsx

import { FixedSizeList } from 'react-window';

function ProductList() {
  const productList = dcupl.lists.create({ modelKey: 'Product' });
  const products = productList.catalog.query.items();

  return (
    <FixedSizeList
      height={600}
      itemCount={products.length}
      itemSize={50}
      width="100%"
    >
      {({ index, style }) => (
        <div style={style}>
          {products[index].name} - ${products[index].price}
        </div>
      )}
    </FixedSizeList>
  );
}

Profiling and Debugging

Measure Initialization Time

profile-init.ts

console.time('init');
await dcupl.init();
console.timeEnd('init');
// init: 145ms

Measure Query Performance

profile-query.ts

const list = dcupl.lists.create({ modelKey: 'Product' });

console.time('query');
list.catalog.query.addCondition({
  attribute: 'category',
  operator: 'eq',
  value: 'Electronics',
});
const results = list.catalog.query.items();
console.timeEnd('query');
// query: 2ms

Use Quality Analytics

quality-analytics.ts

await dcupl.init();

const analytics = dcupl.quality.values;
console.log({
  loading: analytics.loading, // Data load time
  processing: analytics.processing, // Processing time
  startup: analytics.startup, // Total startup time
  counts: analytics.counts, // Item counts per model
  sizes: analytics.sizes, // Memory estimates
});

Measure Update Performance

profile-update.ts

console.time('update');
dcupl.data.update([
  { key: 'p1', price: 999 },
], { model: 'Product' });
console.timeEnd('update');
// update: 3ms

Performance Benchmarks

Query Time by Dataset Size

Items	Indexed Query	Full Scan	Improvement
1,000	1ms	15ms	15x
10,000	1ms	150ms	150x
100,000	2ms	1,500ms	750x
1,000,000	3ms	15,000ms	5,000x

Update Performance

Items Changed	Full Update	Partial Update	Improvement
1	500ms	2ms	250x
10	500ms	5ms	100x
100	500ms	20ms	25x
1,000	500ms	100ms	5x

Performance Checklist

Initialization

Call init() only once at startup
Load critical data first, supplementary data later
Use progressive loading for large datasets

Queries

Filter on indexed properties
Order filters by selectivity (most selective first)
Use derived properties for deep queries
Limit deep query depth to 2-3 levels
Use pagination for large result sets

Updates

Use partial updates (data.update()) not full updates
Specify changedAttributes when possible
Batch multiple changes into single update
Avoid re-initialization after updates

Lists

Reuse lists instead of creating new ones
Destroy lists when no longer needed
Use list pooling for dynamic list creation

Memory

Load only needed data
Remove unused properties from data
Clear stale data periodically
Use virtual scrolling for large lists

What's Next?

Data Modeling - Structure models effectively
Error Handling - Handle errors gracefully
Caching - Caching strategies
Performance Guide - Advanced performance topics