Performance Optimization

Learn how to optimize dcupl for maximum performance with large datasets, complex queries, and high-frequency updates. This guide covers indexing, partial updates, batch operations, and profiling strategies.

Performance Overview

dcupl is designed to handle large datasets efficiently. With proper optimization:

100k+ items - Fast queries with indexing
10-100x faster updates - Using partial updates
Sub-millisecond queries - With proper indexing
Efficient memory usage - With list management

Indexing Strategies

Automatic Indexing

All properties and references are automatically indexed for filtering and aggregation. dcupl handles indexing automatically, so you don't need to configure anything special for most use cases.

const productModel: ModelDefinition = {
  key: 'Product',
  properties: [
    { key: 'category', type: 'string' }, // Automatically indexed
    { key: 'brand', type: 'string' }, // Automatically indexed
    { key: 'price', type: 'float' }, // Automatically indexed + aggregatable
  ],
};

Manual Index Property

For specific lookup optimizations, you can enable manual indexing:

{
  key: 'sku',
  type: 'string',
  index: true,  // Enable additional indexing optimization
}

Partial Updates

Partial updates are 10-100x faster than full updates for large datasets.

Full Update (Slow)

// ❌ Slow: Rebuilds entire model
dcupl.data.set(updatedData, { model: 'Product' });
await dcupl.init();

Partial Update (Fast)

// ✅ Fast: Updates only changed items
dcupl.data.update(
  [
    { key: 'p1', price: 999 }, // Only update price
    { key: 'p2', stock: 50 }, // Only update stock
  ],
  { model: 'Product' }
);

Performance comparison (10,000 items):

Full update: ~500ms
Partial update: ~5ms
100x faster!

Partial Update with Specific Attributes

Update only specific attributes:

dcupl.data.update([{ key: 'p1', price: 999 }], { model: 'Product' });

Batch Partial Updates

Update multiple items efficiently:

Update 1000 items

const updates = [];
for (let i = 0; i < 1000; i++) {
  updates.push({
    key: `p${i}`,
    stock: newStock[i],
  });
}

dcupl.data.update(updates, { model: 'Product' });

Remove Items Efficiently

Remove specific items

dcupl.data.remove([{ key: 'p1' }, { key: 'p2' }, { key: 'p3' }], { model: 'Product' });

Query Optimization

Use Indexed Properties

// All properties are automatically indexed and queryable
productList.catalog.query.addCondition({
  attribute: 'category',
  operator: 'eq',
  value: 'Electronics',
});

Limit Deep Queries

// ⚠️ Slower: 3-level deep query
orderList.catalog.query.addCondition({
  attribute: 'customer.address.city',
  operator: 'eq',
  value: 'San Francisco',
});

// ✅ Faster: Use derived property
// Derive city at model level
{
  key: 'customerCity',
  type: 'string',
  derive: {
    localReference: 'customer',
    remoteProperty: 'city',
  },
}

// Then query directly
orderList.catalog.query.addCondition({
  attribute: 'customerCity',
  operator: 'eq',
  value: 'San Francisco',
});

Optimize Query Groups

// ✅ Good: Simple queries
productList.catalog.query.addCondition({
  groupKey: 'filters',
  groupType: 'and',
  queries: [
    { attribute: 'category', operator: 'eq', value: 'Electronics' },
    { attribute: 'inStock', operator: 'eq', value: true },
  ],
});

// ⚠️ Slower: Many nested groups
productList.catalog.query.addCondition({
  groupKey: 'complex',
  groupType: 'and',
  queries: [
    {
      groupKey: 'g1',
      groupType: 'or',
      queries: [
        // Many nested queries...
      ],
    },
    // ...
  ],
});

Use Pagination

For large result sets, use pagination:

const productList = dcupl.lists.create({
  modelKey: 'Product',
  pagination: {
    page: 1,
    pageSize: 50, // Limit results
  },
});

// Execute with pagination
const products = productList.catalog.query.items();
console.log(products.length); // Max 50 items

// Navigate pages
productList.pagination.next();
const nextPage = productList.catalog.query.items();

List Management

Reuse Lists

// ❌ Bad: Create new list for each query
function getProducts(category) {
  const list = dcupl.lists.create({ modelKey: 'Product' });
  list.catalog.query.addCondition({
    attribute: 'category',
    operator: 'eq',
    value: category,
  });
  return list.catalog.query.items();
}

// ✅ Good: Reuse list
const productList = dcupl.lists.create({ modelKey: 'Product' });

function getProducts(category) {
  productList.catalog.query.clear();
  productList.catalog.query.addCondition({
    attribute: 'category',
    operator: 'eq',
    value: category,
  });
  return productList.catalog.query.items();
}

Destroy Lists

Destroy lists when no longer needed:

const tempList = dcupl.lists.create({ modelKey: 'Product' });
// ... use list

// Clean up
tempList.destroy();

List Pooling

For frequently created/destroyed lists:

class ListPool {
  private pool: Map<string, DcuplList[]> = new Map();

  get(modelKey: string): DcuplList {
    const pool = this.pool.get(modelKey) || [];
    return pool.pop() || dcupl.lists.create({ modelKey });
  }

  release(list: DcuplList) {
    list.catalog.query.clear();
    const modelKey = list.model.key;
    const pool = this.pool.get(modelKey) || [];
    pool.push(list);
    this.pool.set(modelKey, pool);
  }
}

const pool = new ListPool();
const list = pool.get('Product');
// ... use list
pool.release(list);

Memory Optimization

Limit Data Loading

Load only needed data:

// ❌ Bad: Load all data
const allProducts = await fetch('/api/products').then((r) => r.json());
dcupl.data.set(allProducts, { model: 'Product' });

// ✅ Good: Load filtered data
const activeProducts = await fetch('/api/products?active=true').then((r) => r.json());
dcupl.data.set(activeProducts, { model: 'Product' });

Remove Unused Properties

Don't define properties you won't use:

// ❌ Bad: Include everything
properties: [
  { key: 'id', type: 'string' },
  { key: 'name', type: 'string' },
  { key: 'legacyField1', type: 'string' },
  { key: 'legacyField2', type: 'string' },
  { key: 'internalNote', type: 'string' },
  // ...
];

// ✅ Good: Only needed properties
properties: [
  { key: 'id', type: 'string' },
  { key: 'name', type: 'string' },
  { key: 'price', type: 'float' },
];

Clear Unused Data

Remove data that's no longer needed:

Remove specific items

dcupl.data.remove([{ key: 'old1' }, { key: 'old2' }, { key: 'old3' }], { model: 'Product' });

// Clear all data for a model
dcupl.data.reset({ model: 'Product' });

Large Dataset Handling

100k+ Items

For very large datasets:

const productModel: ModelDefinition = {
  key: 'Product',
  properties: [
    { key: 'category', type: 'string' },
    { key: 'inStock', type: 'boolean' },
    { key: 'name', type: 'string' },
    { key: 'description', type: 'string' },
  ],
};

// Load data in chunks
async function loadLargeDataset() {
  const chunkSize = 10000;
  let offset = 0;

  while (true) {
    const chunk = await fetch(`/api/products?offset=${offset}&limit=${chunkSize}`).then((r) =>
      r.json()
    );

    if (chunk.length === 0) break;

    dcupl.data.set(chunk, { model: 'Product' });

    offset += chunkSize;
  }

  await dcupl.init();
}

Virtualization

For rendering large lists:

Use virtual scrolling libraries

import { FixedSizeList } from 'react-window';

function ProductList() {
  const productList = dcupl.lists.create({ modelKey: 'Product' });
  const products = productList.catalog.query.items();

  return (
    <FixedSizeList
      height={600}
      itemCount={products.length}
      itemSize={50}
      width="100%"
    >
      {({ index, style }) => (
        <div style={style}>
          {products[index].name}
        </div>
      )}
    </FixedSizeList>
  );
}

Profiling and Debugging

Measure Init Performance

console.time('init');
await dcupl.init();
console.timeEnd('init');
// init: 145ms

Quality Analytics

await dcupl.init();

const analytics = dcupl.quality.values;
console.log({
  loading: analytics.loading, // Load time
  processing: analytics.processing, // Processing time
  startup: analytics.startup, // Total time
  counts: analytics.counts, // Data counts
  sizes: analytics.sizes, // Memory usage
});

Measure Query Performance

const list = dcupl.lists.create({ modelKey: 'Product' });

console.time('query');
list.catalog.query.addCondition({
  attribute: 'category',
  operator: 'eq',
  value: 'Electronics',
});
const results = list.catalog.query.items();
console.timeEnd('query');
// query: 2ms

Measure Update Performance

console.time('update');
dcupl.data.update([{ key: 'p1', price: 999 }], { model: 'Product' });
console.timeEnd('update');
// update: 5ms

Practical Examples

Example 1: E-commerce Product Catalog

Optimized for 100k+ products

const productModel: ModelDefinition = {
  key: 'Product',
  properties: [
    { key: 'sku', type: 'string', index: true },
    { key: 'name', type: 'string' },
    { key: 'category', type: 'string' },
    { key: 'brand', type: 'string' },
    { key: 'price', type: 'float' },
    { key: 'inStock', type: 'boolean' },

    // Derived for faster queries
    {
      key: 'categoryName',
      type: 'string',
      derive: {
        localReference: 'category',
        remoteProperty: 'name',
      },
    },

    // Display only
    { key: 'description', type: 'string' },
    { key: 'images', type: 'json' },
  ],
};

// Reuse list
const productList = dcupl.lists.create({
  modelKey: 'Product',
  pagination: { pageSize: 50 },
});

// Fast queries
function searchProducts(filters) {
  productList.catalog.query.clear();

  if (filters.category) {
    productList.catalog.query.addCondition({
      attribute: 'category',
      operator: 'eq',
      value: filters.category,
    });
  }

  if (filters.inStock) {
    productList.catalog.query.addCondition({
      attribute: 'inStock',
      operator: 'eq',
      value: true,
    });
  }

  return productList.catalog.query.items();
}

// Efficient updates
function updatePrice(sku, newPrice) {
  dcupl.data.update([{ key: sku, price: newPrice }], { model: 'Product' });
}

Example 2: Real-time Dashboard

Optimized for frequent updates

const metricsModel: ModelDefinition = {
  key: 'Metric',
  properties: [
    { key: 'timestamp', type: 'date' },
    { key: 'type', type: 'string' },
    { key: 'value', type: 'float' },
  ],
};

dcupl.models.set(metricsModel);
await dcupl.init();

// Fast aggregation
const metricsList = dcupl.lists.create({ modelKey: 'Metric' });

function getMetrics(type, startDate, endDate) {
  metricsList.catalog.query.clear();
  metricsList.catalog.query.addCondition({
    groupKey: 'filters',
    groupType: 'and',
    queries: [
      { attribute: 'type', operator: 'eq', value: type },
      { attribute: 'timestamp', operator: 'gte', value: startDate },
      { attribute: 'timestamp', operator: 'lte', value: endDate },
    ],
  });

  return metricsList.catalog.fn.aggregate({
    attribute: 'value',
    types: ['sum', 'avg', 'min', 'max'],
  });
}

// Efficient real-time updates
function addMetric(metric) {
  dcupl.data.update([metric], { model: 'Metric' });
}

Example 3: Order Management

const orderModel: ModelDefinition = {
  key: 'Order',
  properties: [
    { key: 'orderId', type: 'string', index: true },
    { key: 'status', type: 'string' },
    { key: 'total', type: 'float' },
    { key: 'createdAt', type: 'date' },

    // Derived for fast queries
    {
      key: 'customerEmail',
      type: 'string',
      derive: {
        localReference: 'customer',
        remoteProperty: 'email',
      },
    },
  ],
  references: [{ key: 'customer', type: 'singleValued', model: 'Customer' }],
};

// Efficient status updates
function updateOrderStatus(orderId, newStatus) {
  dcupl.data.update([{ key: orderId, status: newStatus }], { model: 'Order' });
}

// Fast customer order lookup
const orderList = dcupl.lists.create({ modelKey: 'Order' });

function getCustomerOrders(email) {
  orderList.catalog.query.clear();
  orderList.catalog.query.addCondition({
    attribute: 'customerEmail', // Derived, indexed
    operator: 'eq',
    value: email,
  });
  return orderList.catalog.query.items();
}

Example 4: Search Performance

Optimized search model

const articleModel: ModelDefinition = {
  key: 'Article',
  properties: [
    { key: 'title', type: 'string' },
    { key: 'slug', type: 'string', index: true },
    { key: 'tags', type: 'Array<string>' },
    { key: 'authorName', type: 'string' },
    { key: 'status', type: 'string' },
    { key: 'publishedAt', type: 'date' },

    // Content fields
    { key: 'content', type: 'string' },
    { key: 'excerpt', type: 'string' },
  ],
};

const articleList = dcupl.lists.create({
  modelKey: 'Article',
  pagination: { pageSize: 20 },
});

function searchArticles(query) {
  articleList.catalog.query.clear();

  // Fast indexed search
  articleList.catalog.query.addCondition({
    groupKey: 'search',
    groupType: 'or',
    queries: [
      { attribute: 'title', operator: 'find', value: query },
      { attribute: 'tags', operator: 'find', value: query },
      { attribute: 'authorName', operator: 'find', value: query },
    ],
  });

  return articleList.catalog.query.items();
}

Example 5: Inventory Management

const inventoryModel: ModelDefinition = {
  key: 'Inventory',
  properties: [
    { key: 'sku', type: 'string', index: true },
    { key: 'warehouse', type: 'string' },
    { key: 'location', type: 'string' },
    { key: 'quantity', type: 'int' },
    { key: 'reorderPoint', type: 'int' },
    { key: 'lastUpdated', type: 'date' },
  ],
};

// Batch stock updates
function updateStock(updates) {
  const data = updates.map((u) => ({
    key: u.sku,
    quantity: u.quantity,
    lastUpdated: new Date().toISOString(),
  }));

  dcupl.data.update(data, { model: 'Inventory' });
}

// Fast low-stock query
const inventoryList = dcupl.lists.create({ modelKey: 'Inventory' });

function getLowStock() {
  inventoryList.catalog.query.clear();
  inventoryList.catalog.query.addCondition({
    attribute: 'quantity',
    operator: 'lte',
    value: 10,
  });
  return inventoryList.catalog.query.items();
}

Performance Benchmarks

Dataset Size vs Query Time

Items	Without Index	With Index	Improvement
1,000	15ms	1ms	15x
10,000	150ms	1ms	150x
100,000	1,500ms	2ms	750x
1,000,000	15,000ms	3ms	5,000x

Update Performance

Operation	Full Update	Partial Update	Improvement
1 item	500ms	2ms	250x
10 items	500ms	5ms	100x
100 items	500ms	20ms	25x
1,000 items	500ms	100ms	5x

Memory Usage

Items	Without Optimization	With Optimization	Savings
10,000	50MB	30MB	40%
100,000	500MB	250MB	50%
1,000,000	5GB	2GB	60%

Performance Checklist

Model Setup

✅ Use derived properties for common deep queries
✅ Remove unused properties
✅ Use appropriate data types
✅ Use index: true for frequently looked-up keys

Querying

✅ Query indexed properties
✅ Avoid deep queries (use derived properties)
✅ Use pagination for large results
✅ Reuse lists instead of creating new ones
✅ Keep query groups simple

Updates

✅ Use partial updates (data.update())
✅ Batch multiple updates together
✅ Avoid full init() after updates

Memory

✅ Load only needed data
✅ Remove unused data
✅ Destroy unused lists
✅ Use virtualization for large lists

Troubleshooting

Slow Queries

Problem: Queries take too long

Solution:

Measure query time:

console.time('query');
const results = list.catalog.query.items();
console.timeEnd('query');

Simplify query:

// Avoid complex nested groups

Use derived properties for deep queries

Slow Updates

Problem: Updates are slow

Solution: Use partial updates:

// ❌ Slow
dcupl.data.set(allData, { model: 'Product' });
await dcupl.init();

// ✅ Fast
dcupl.data.update([{ key: 'p1', price: 999 }], { model: 'Product' });

High Memory Usage

Problem: Memory usage is too high

Solution:

Remove unused properties
Load filtered data
Clear unused data
Destroy unused lists

Slow Init

Problem: dcupl.init() takes too long

Solution:

Reduce data size
Remove unused models
Disable quality validation for non-critical models
Profile with quality analytics

What's Next?

Caching - Caching strategies
Data Management - Loading and updating data
Lists - Working with lists
Queries - Query optimization
Models - Property indexing