Iter8 Production Issue Report

Application Repeatedly Crashing During Heavy Load (P0)

Problem Report

The application has started crashing intermittently, often after a few minutes of normal operation. This appears correlated with a surge in holiday-season user traffic.

Impact: Users face outages and cannot access the application, leading to business disruptions. Urgent fix required.

Recent Check-Ins Analysis
No direct link found in recent commits
  • Commit #aca790 – Updated UI text for holiday campaign
    Verified: No direct relation
  • Commit #f42b9a – Fixed minor logging bug
    Verified: No direct relation
Potential Root Causes

The system identified older code diffs that may cause crashes only under heavy traffic conditions.

Commit #d2f6ab • 108 days ago • Introduced "asyncTaskPool" changes

Refactored concurrency logic to handle tasks with a custom thread pool. This may lead to race conditions or memory leaks under high load if tasks aren't properly released.

  Remediation Steps:
  • Revert concurrency patch to a stable release, then add proper load tests.

Commit #ab32f1 • 122 days ago

Introduced an in-memory session cache for speed, but sizing logic might be flawed. Could cause out-of-memory errors under heavy user sessions.

  Remediation Steps:
  • Limit session cache size or move it to a distributed in-memory store.

Commit #b9481e • 154 days ago

Changed the circuit breaker window from 30s to 5s. Under heavy load, frequent toggling of the breaker might cause repeated restarts or crashes if not throttled properly.

  Remediation Steps:
  • Tune circuit breaker thresholds for holiday traffic loads.