Performance

Performance characteristics and optimizations in Tracera

Performance

Tracera is designed for high-throughput, low-latency data processing. Every layer is optimized.

Batch Inserts

Price points are inserted via pgx.CopyFrom (PostgreSQL COPY protocol) in configurable chunks. This minimizes database round-trips and achieves significantly higher throughput than individual INSERT statements.

Individual INSERTs:  ~500 rows/sec
COPY batch insert:   ~50,000 rows/sec (100x improvement)

Continuous Aggregates

TimescaleDB pre-computes aggregated views for common query windows:

Window	Bucket	Metrics
1 hour	time_bucket('1 hour', time)	avg, min, max, stddev, total volume
24 hours	time_bucket('24 hours', time)	avg, min, max, stddev, total volume
7 days	time_bucket('7 days', time)	avg, min, max, stddev, total volume

These materialized views refresh automatically, so volatility queries read pre-computed data instead of scanning raw price history.

Redis Caching

Sub-millisecond reads for hot data:

• Latest prices — cached on every ingestion cycle
• Volatility rankings — sorted sets for instant “most volatile” queries
• Session data — no database hit for session lookups
• Rate limits — atomic increment counters

Connection Pooling

• pgxpool — TimescaleDB connection pool avoids connection-per-request overhead
• Pooled Redis clients — shared connections for cache and pub/sub operations

Goroutine Fan-Out

Ingestion fetches providers concurrently:

var wg sync.WaitGroup
for _, provider := range providers {
    wg.Add(1)
    go func(p Provider) {
        defer wg.Done()
        prices, err := p.Fetch(ctx)
        // handle results...
    }(provider)
}
wg.Wait()

Provider failures are isolated — one slow or failing API doesn’t block the cycle.

Targeted Queries

Portfolio valuation restricts latest-price lookups to only held item IDs, avoiding broad table scans:

SELECT DISTINCT ON (item_id)
    item_id, price
FROM price_history
WHERE item_id = ANY($1)  -- only held items
ORDER BY item_id, time DESC

Retention Policies

Raw price data is retained for a configurable period, while continuous aggregates are retained indefinitely. This keeps the database size manageable without losing historical analysis capability.