Azure Monitoring & Scaling Guide

Last updated: 2026-05-07 06:53:50

Azure Monitoring & Scaling Guide for Huitzo¶

Production-ready monitoring, alerting, and auto-scaling strategies for Huitzo deployments on Azure.

Part 1: Monitoring Architecture¶

Overview¶

┌─────────────────────────────────────────────────────────────┐
│              Data Collection Layer                           │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  Application Metrics    Container Metrics    VM Metrics     │
│  (Prometheus)          (Docker stats)        (Azure Monitor)│
│  • Request latency     • CPU usage           • CPU %        │
│  • Error rate          • Memory usage        • Memory %     │
│  • Queue depth         • Disk I/O            • Disk %       │
│  • Command execution   • Network I/O         • Network      │
│  • Cache hit rate      • Container restarts  • Load         │
│                                              │              │
│  └────────────────────────────────────────────┘         │
├─────────────────────────────────────────────────────────────┤
│              Aggregation & Storage Layer                     │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌──────────────────────┐     ┌──────────────────────┐    │
│  │   Azure Monitor      │     │  Log Analytics       │    │
│  │   (Metrics Storage)  │     │  Workspace           │    │
│  │  • Time-series DB    │     │  • Kusto Query       │    │
│  │  • 93 days retention │     │  • Full-text search  │    │
│  │  • 1-min granularity │     │  • Custom dashboards │    │
│  └──────────────────────┘     └──────────────────────┘    │
│                                                             │
├─────────────────────────────────────────────────────────────┤
│              Visualization & Alerting Layer                  │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌──────────────────┐  ┌──────────────┐  ┌──────────────┐ │
│  │   Dashboards     │  │   Alerts     │  │   Logging    │ │
│  │  • Real-time     │  │  • Threshold │  │  • CloudWatch│ │
│  │  • Historical    │  │  • Anomaly   │  │  • App Logs  │ │
│  │  • Custom KPIs   │  │  • Composite │  │  • Audit     │ │
│  └──────────────────┘  └──────────────┘  └──────────────┘ │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Part 2: Key Metrics¶

Application-Level Metrics¶

These are collected from the Huitzo backend using Prometheus.

Request Performance¶

# apps/backend/src/backend/observability/metrics.py
from prometheus_client import Counter, Histogram, Gauge
import time

# Latency (all requests)
http_request_duration = Histogram(
    'http_request_duration_seconds',
    'HTTP request latency in seconds',
    buckets=[0.01, 0.05, 0.1, 0.5, 1.0, 2.5, 5.0],
    labelnames=['method', 'endpoint', 'status']
)

# Error rate
http_requests_total = Counter(
    'http_requests_total',
    'Total HTTP requests',
    labelnames=['method', 'endpoint', 'status']
)

# Active requests
http_requests_in_progress = Gauge(
    'http_requests_in_progress',
    'HTTP requests in progress',
    labelnames=['method', 'endpoint']
)

Thresholds & Alerts:

Critical (Page): 
  - p95 latency > 1000ms
  - Error rate > 5% (50+ requests)
  - p99 latency > 5000ms for >5 min

Warning (Email):
  - p95 latency > 500ms
  - Error rate > 2%
  - p99 latency > 2000ms

Command Execution Metrics¶

# Command execution tracking
command_execution_duration = Histogram(
    'command_execution_duration_seconds',
    'Command execution time',
    buckets=[0.1, 0.5, 1.0, 5.0, 10.0, 30.0, 60.0, 300.0],
    labelnames=['namespace', 'command', 'status', 'executor']
)

command_execution_total = Counter(
    'command_executions_total',
    'Total command executions',
    labelnames=['namespace', 'command', 'status']
)

command_queue_depth = Gauge(
    'command_queue_depth',
    'Number of pending commands in queue',
    labelnames=['queue_name']  # fast, medium, long
)

Thresholds:

Critical:
  - Fast queue depth > 100 (backlog building)
  - Command execution failure rate > 10%
  - p95 execution time > 30s (for fast queue)

Warning:
  - Medium queue depth > 50
  - p95 execution time > 10s (for fast queue)
  - Slow commands taking >5x expected time

Cache Performance¶

# Redis cache metrics
cache_hits = Counter(
    'cache_hits_total',
    'Cache hits',
    labelnames=['key_prefix']
)

cache_misses = Counter(
    'cache_misses_total',
    'Cache misses',
    labelnames=['key_prefix']
)

cache_hit_rate = Gauge(
    'cache_hit_rate',
    'Cache hit rate (0-1)',
    labelnames=['key_prefix']
)

redis_memory_bytes = Gauge(
    'redis_memory_bytes',
    'Redis memory usage in bytes'
)

Thresholds:

Critical:
  - Hit rate < 50% (indicates poor caching strategy)
  - Redis memory > 90% (risk of eviction)

Warning:
  - Hit rate < 70%
  - Redis memory > 80%

Database Performance¶

# Database query metrics
db_query_duration = Histogram(
    'db_query_duration_seconds',
    'Database query duration',
    buckets=[0.001, 0.01, 0.05, 0.1, 0.5, 1.0],
    labelnames=['operation', 'table', 'status']
)

db_connections = Gauge(
    'db_connections',
    'Active database connections'
)

db_slow_queries = Counter(
    'db_slow_queries_total',
    'Queries exceeding 1 second',
    labelnames=['operation', 'table']
)

Thresholds:

Critical:
  - p95 query time > 500ms
  - Slow queries > 5/min
  - Connections > 80 (of 100 max)

Warning:
  - p95 query time > 200ms
  - Slow queries > 2/min
  - Connections > 60

Infrastructure-Level Metrics¶

Collected by Azure Monitor from the VM and services.

VM Metrics¶

CPU Usage:
  - Current: % of cores used
  - Alert Critical: > 80% sustained (5+ min)
  - Alert Warning: > 60% sustained (10+ min)

Memory Usage:
  - Current: % of available
  - Alert Critical: > 85% sustained
  - Alert Warning: > 75% sustained

Disk Usage:
  - Root partition: % used
  - Data partition: % used
  - Alert Critical: > 90%
  - Alert Warning: > 80%

Network:
  - Inbound bytes/sec
  - Outbound bytes/sec
  - Packet loss: should be 0%
  - Comment: Alert only if sustained packet loss > 0%

PostgreSQL Metrics¶

Connection Count:
  - Current active connections
  - Alert Critical: > 90 (of 100 max)
  - Alert Warning: > 70

Query Performance:
  - Avg query time
  - Max query time
  - Alert Warning: p95 > 200ms

Replication Lag (if HA enabled later):
  - Lag in seconds
  - Alert Critical: > 30 seconds
  - Alert Warning: > 10 seconds

Disk Usage:
  - Storage percent used
  - Alert Warning: > 80%
  - Alert Critical: > 95%

IOPS and Throughput:
  - Read IOPS
  - Write IOPS
  - Alert if sustained max (adjust based on SKU)

Redis Metrics¶

Memory Usage:
  - Current bytes used
  - % of max memory (alert if evicting)
  - Alert Warning: > 80%
  - Alert Critical: > 95% (risk of eviction)

Connection Count:
  - Current connections
  - Alert Critical: > 1000
  - Alert Warning: > 500

Key Eviction Rate:
  - Evicted keys/sec
  - Alert Critical: > 0 (any eviction is bad)
  - Alert Warning: > 1

Expiration:
  - Items with TTL
  - Items expired/sec (should be normal)
  - Alert if expiration rate very high

Blob Storage Metrics¶

Ingress/Egress:
  - Bytes written (upload)
  - Bytes read (download)
  - Alert if sustained max bandwidth

Request Count:
  - Successful requests
  - Failed requests (5xx errors)
  - Alert Critical: > 1% error rate

Availability:
  - Should be 99.9%+
  - Alert Critical: < 99%

Latency:
  - E2E latency (should be <100ms)
  - Server latency (should be <50ms)
  - Alert Warning: p95 > 100ms

Part 3: Azure Monitor Configuration¶

Setup Log Analytics Workspace¶

# Create Log Analytics workspace
az monitor log-analytics workspace create \
  --name huitzo-logs \
  --resource-group huitzo-prod-eastus-rg \
  --location eastus

# Get workspace ID
WORKSPACE_ID=$(az monitor log-analytics workspace show \
  --name huitzo-logs \
  --resource-group huitzo-prod-eastus-rg \
  --query id -o tsv)

# Get workspace key
WORKSPACE_KEY=$(az monitor log-analytics workspace get-shared-keys \
  --name huitzo-logs \
  --resource-group huitzo-prod-eastus-rg \
  --query primarySharedKey -o tsv)

Enable VM Monitoring¶

# Install Azure Monitor Agent on VM
az vm extension set \
  --resource-group huitzo-prod-eastus-rg \
  --vm-name huitzo-app-prod \
  --name AzureMonitorLinuxAgent \
  --publisher Microsoft.Azure.Monitor \
  --enable-auto-upgrade true

# Configure agent to collect metrics
az monitor data-collection rule create \
  --name huitzo-dcr \
  --resource-group huitzo-prod-eastus-rg \
  --location eastus \
  --rule-file- << 'EOF'
{
  "kind": "Linux",
  "properties": {
    "dataSources": {
      "performanceCounters": [
        {
          "name": "Unix Performance",
          "counterSpecifiers": [
            "\\Processor(_Total)\\% Processor Time",
            "\\Memory\\% Used Memory",
            "\\LogicalDisk(/)\\% Used Space"
          ],
          "samplingFrequencyInSeconds": 60
        }
      ],
      "syslog": [
        {
          "name": "syslog",
          "streams": ["Microsoft-Syslog"],
          "facilityNames": ["*"],
          "logLevels": ["Notice", "Warning", "Error", "Critical"]
        }
      ]
    },
    "destinations": {
      "logAnalytics": [
        {
          "name": "huitzo-logs",
          "workspaceResourceId": "$WORKSPACE_ID"
        }
      ]
    },
    "dataFlows": [
      {
        "streams": ["Microsoft-Syslog"],
        "destinations": ["huitzo-logs"]
      }
    ]
  }
}
EOF

Create Dashboards¶

# Create custom dashboard for v0.0.0 deployment
az portal dashboard create \
  --resource-group huitzo-prod-eastus-rg \
  --name huitzo-dashboard \
  --input-path - << 'EOF'
{
  "properties": {
    "lenses": {
      "0": {
        "order": 0,
        "parts": {
          "0": {
            "position": {"x": 0, "y": 0, "colSpan": 4, "rowSpan": 3},
            "metadata": {
              "inputs": [{
                "name": "ResourceId",
                "value": "/subscriptions/{subscriptionId}/resourceGroups/huitzo-prod-eastus-rg/providers/Microsoft.Compute/virtualMachines/huitzo-app-prod"
              }],
              "type": "Extension/Microsoft_Azure_Compute/PartType/VirtualMachinePart"
            }
          },
          "1": {
            "position": {"x": 4, "y": 0, "colSpan": 4, "rowSpan": 3},
            "metadata": {
              "inputs": [{
                "name": "ResourceId",
                "value": "/subscriptions/{subscriptionId}/resourceGroups/huitzo-prod-eastus-rg/providers/Microsoft.DBforPostgreSQL/flexibleServers/huitzo-db-prod"
              }],
              "type": "Extension/Microsoft_Azure_OSS_Databases/PartType/PostgresqlServerPart"
            }
          },
          "2": {
            "position": {"x": 8, "y": 0, "colSpan": 4, "rowSpan": 3},
            "metadata": {
              "inputs": [{
                "name": "ResourceId",
                "value": "/subscriptions/{subscriptionId}/resourceGroups/huitzo-prod-eastus-rg/providers/Microsoft.Cache/redis/huitzo-cache-prod"
              }],
              "type": "Extension/Microsoft_Azure_Cache/PartType/RedisCachePart"
            }
          }
        }
      }
    }
  }
}
EOF

Create Alert Rules¶

Critical: High Error Rate¶

az monitor metrics alert create \
  --name huitzo-high-error-rate \
  --resource-group huitzo-prod-eastus-rg \
  --scopes /subscriptions/{subscriptionId}/resourceGroups/huitzo-prod-eastus-rg/providers/Microsoft.Compute/virtualMachines/huitzo-app-prod \
  --condition "avg ErrorRate > 5" \
  --evaluation-frequency 1m \
  --window-size 5m \
  --action /subscriptions/{subscriptionId}/resourceGroups/huitzo-prod-eastus-rg/providers/microsoft.insights/actionGroups/huitzo-alerts \
  --severity 1 \
  --description "Alert when error rate exceeds 5%"

Critical: High CPU Usage¶

az monitor metrics alert create \
  --name huitzo-high-cpu \
  --resource-group huitzo-prod-eastus-rg \
  --scopes /subscriptions/{subscriptionId}/resourceGroups/huitzo-prod-eastus-rg/providers/Microsoft.Compute/virtualMachines/huitzo-app-prod \
  --condition "avg Percentage CPU > 80" \
  --evaluation-frequency 1m \
  --window-size 5m \
  --action /subscriptions/{subscriptionId}/resourceGroups/huitzo-prod-eastus-rg/providers/microsoft.insights/actionGroups/huitzo-alerts \
  --severity 2 \
  --description "Alert when CPU exceeds 80% for 5+ minutes"

Warning: High Memory Usage¶

az monitor metrics alert create \
  --name huitzo-high-memory \
  --resource-group huitzo-prod-eastus-rg \
  --scopes /subscriptions/{subscriptionId}/resourceGroups/huitzo-prod-eastus-rg/providers/Microsoft.Compute/virtualMachines/huitzo-app-prod \
  --condition "avg Available Memory < 20" \
  --evaluation-frequency 5m \
  --window-size 10m \
  --action /subscriptions/{subscriptionId}/resourceGroups/huitzo-prod-eastus-rg/providers/microsoft.insights/actionGroups/huitzo-alerts \
  --severity 2 \
  --description "Alert when available memory < 20%"

Critical: Database Connection Pool Exhausted¶

az monitor metrics alert create \
  --name huitzo-db-connections-high \
  --resource-group huitzo-prod-eastus-rg \
  --scopes /subscriptions/{subscriptionId}/resourceGroups/huitzo-prod-eastus-rg/providers/Microsoft.DBforPostgreSQL/flexibleServers/huitzo-db-prod \
  --condition "avg active_connections > 90" \
  --evaluation-frequency 1m \
  --window-size 5m \
  --action /subscriptions/{subscriptionId}/resourceGroups/huitzo-prod-eastus-rg/providers/microsoft.insights/actionGroups/huitzo-alerts \
  --severity 1 \
  --description "Alert when active database connections exceed 90"

Critical: Slow Queries¶

# Setup via Log Analytics query
az monitor log-analytics workspace query \
  --workspace-name huitzo-logs \
  --resource-group huitzo-prod-eastus-rg \
  --analytics-query "
    AzureDiagnostics
    | where ResourceProvider == \"MICROSOFT.DBFORPOSTGRESQL\"
    | where executionTime_d > 1000
    | summarize Count=count() by bin(TimeGenerated, 5m)
    | where Count > 5
  "

Part 4: Scaling Strategy¶

Phase 1: Single-Instance (v0.0.0)¶

When to Scale to Phase 2: - CPU consistently > 70% - Memory > 75% - Database connections > 80 of 100 - Command queue depth regularly > 50 - Response time p95 > 500ms

# Monitor command queue depth
docker compose exec worker celery -A backend.celery inspect active_queues

Phase 2: Multi-Instance Load Balancing (v0.1+)¶

Architecture:
┌──────────────────────────────┐
│   Azure LoadBalancer         │
│   (or Application Gateway)   │
└──────────────────────────────┘
          ↓
┌──────────────────────────────┐
│    VM Scale Set (3x)         │
├──────────────────────────────┤
│ Instance 1: Backend + Worker │
│ Instance 2: Backend + Worker │
│ Instance 3: Backend + Worker │
└──────────────────────────────┘
          ↓ (shared)
     ┌──────────────────────────┐
     │  PostgreSQL (Standard)   │
     │  Redis (Premium)         │
     │  Blob Storage            │
     └──────────────────────────┘

Scaling Rules (Auto-Scale Policy)¶

# Create scale set from single VM
# (Migrate existing data first via backup/restore)

# Configure auto-scale based on CPU
az monitor autoscale create \
  --resource-group huitzo-prod-eastus-rg \
  --resource-type Microsoft.Compute/virtualMachineScaleSets \
  --resource-name huitzo-vmss \
  --min-count 2 \
  --max-count 5 \
  --count 2

# Scale-out rule: If CPU > 70% for 5 minutes
az monitor autoscale rule create \
  --resource-group huitzo-prod-eastus-rg \
  --autoscale-name huitzo-autoscale \
  --condition "avg Percentage CPU > 70 during 5m" \
  --scale action increase percentage 50%

# Scale-in rule: If CPU < 30% for 10 minutes
az monitor autoscale rule create \
  --resource-group huitzo-prod-eastus-rg \
  --autoscale-name huitzo-autoscale \
  --condition "avg Percentage CPU < 30 during 10m" \
  --scale action decrease count 1

Considerations: - Minimum 2 instances for HA (but cost increases 2x) - Deploy in different availability zones if possible - Share PostgreSQL and Redis (single shared instance) - Session affinity recommended (sticky sessions)

Database Scaling¶

# Upgrade PostgreSQL from B4ms to D4s (scale up)
az postgres flexible-server update \
  --name huitzo-db-prod \
  --resource-group huitzo-prod-eastus-rg \
  --sku-name Standard_D4s \
  --tier GeneralPurpose

# This causes brief downtime (~2-3 minutes)
# Plan for evening or maintenance window

Upgrade Path: - v0.0.0: Standard_B4ms (burstable, ok for single customer) - v0.1+: Standard_D4s (general-purpose, multiple customers) - v2.0+: Standard_D8s with read replicas (enterprise scale)

Redis Scaling¶

# Upgrade from Standard (2GB) to Premium
az redis update \
  --name huitzo-cache-prod \
  --resource-group huitzo-prod-eastus-rg \
  --sku Premium \
  --capacity 1

# Or increase capacity within tier
az redis update \
  --name huitzo-cache-prod \
  --resource-group huitzo-prod-eastus-rg \
  --size 6gb  # Upgrade to 6GB (if Standard too slow)

Part 5: Operational Dashboards¶

Real-Time Operations Dashboard¶

// Save in Azure Monitor: Queries > "Huitzo Operations"

// Request latency trend
AzureDiagnostics
| where ResourceProvider == "Microsoft.Compute"
| where TimeGenerated > ago(1h)
| summarize AvgLatency=avg(executionTime_d), MaxLatency=max(executionTime_d), P95=percentile(executionTime_d, 95)
| by bin(TimeGenerated, 5m)

// Error rate
StdMetrics
| where MetricName == "HttpRequestDuration"
| where status_code >= 500
| summarize ErrorCount=count()
| by bin(TimeGenerated, 1m)

Daily Health Report¶

Create automated report that runs at 9 AM daily:

# Schedule: Create Logic App for daily digest
# Sends email to ops team with:
# - Previous 24h uptime %
# - Peak CPU/Memory
# - Command execution success rate
# - Top errors
# - Database size growth
# - Storage usage trend

Part 6: Incident Response Playbooks¶

High Error Rate (>5%)¶

Diagnosis:

# SSH into VM
docker compose logs --tail 100 backend | grep -i error

# Query slow requests
curl http://localhost:8000/metrics | grep http_

# Check database
docker compose exec backend psql -c "select count(*) from command_execution where status='failed' and created_at > now() - interval '5 min';"

Response: 1. Page on-call engineer immediately 2. Check application logs for common error pattern 3. If database issue: Check connection count 4. If queue issue: Increase worker instances or reset queue 5. If external API issue: Check integration logs 6. If unknown: Roll back to last known good version

High CPU (>80%)¶

Diagnosis:

# Check resource usage
docker stats

# Check top CPU-consuming process
top -b -n 1 | head -15

# Check Celery worker load
docker compose exec worker celery -A backend.celery inspect stats | grep pool

Response: 1. Check if legitimate load increase (high traffic) 2. If yes: Scale up VM size or add worker replicas 3. If no: Profile code for inefficiency 4. Identify slow queries: Enable slow query log

Database Connection Exhaustion¶

Diagnosis:

# Check active connections
docker compose exec backend psql -c "select count(*) from pg_stat_activity;"

# Check connection by client
docker compose exec backend psql -c "select client_addr, count(*) from pg_stat_activity group by client_addr;"

Response: 1. Increase connection pool size in application config 2. Identify connection leak: Are idle connections being released? 3. Scale up database if baseline connection count growing 4. Consider connection pooler (PgBouncer) if many connection clients

Part 7: Capacity Planning¶

Baseline v0.0.0 (1 Customer)¶

Peak Load: 100 concurrent users
- API Latency p95: ~200ms
- Database connections: 20-30 active
- Redis hit rate: ~85%
- CPU usage: 15-25%
- Memory usage: 40-50%
- Disk I/O: Minimal

Monthly Growth:
- Users: +50% (100 → 150 → 225)
- API latency stable (load distributed to workers)
- Database size: +500MB (archive old command logs weekly)
- Storage usage: +100MB (file uploads)

Upgrade Triggers¶

Metric	Current	Upgrade
Concurrent Users	100	200+ → Larger VM
DB Connections	30	80+ → Scale up DB
Redis Memory	512MB	1500MB+ → Upgrade Redis
Disk Usage	50GB	150GB+ → Larger disk
P95 Latency	200ms	500ms+ → Add worker
CPU Average	25%	70%+ → Scale

Part 8: Monitoring Checklist¶

Daily¶

Check Azure Monitor dashboard
Review error logs (should be minimal)
Verify daily backup completed

Weekly¶

Review scaling metrics and trends
Check disk usage and cleanup if needed
Test at least one alert rule
Review slow queries and optimize if needed

Monthly¶

Full disaster recovery test (restore from backup)
Capacity planning review
Cost analysis (Azure Cost Management)
Security audit (NSG rules, access logs)

Version History¶

Date	Version	Changes
2026-02-11	1.0.0	Initial monitoring and scaling guide

Azure Monitoring & Scaling Guide for Huitzo¶

Part 1: Monitoring Architecture¶

Overview¶

Part 2: Key Metrics¶

Application-Level Metrics¶

Request Performance¶

Command Execution Metrics¶

Cache Performance¶

Database Performance¶

Infrastructure-Level Metrics¶

VM Metrics¶

PostgreSQL Metrics¶

Redis Metrics¶

Blob Storage Metrics¶

Part 3: Azure Monitor Configuration¶

Setup Log Analytics Workspace¶

Enable VM Monitoring¶

Create Dashboards¶

Create Alert Rules¶

Critical: High Error Rate¶

Critical: High CPU Usage¶

Warning: High Memory Usage¶

Critical: Database Connection Pool Exhausted¶

Critical: Slow Queries¶

Part 4: Scaling Strategy¶

Phase 1: Single-Instance (v0.0.0)¶

Phase 2: Multi-Instance Load Balancing (v0.1+)¶

Scaling Rules (Auto-Scale Policy)¶

Database Scaling¶

Redis Scaling¶

Part 5: Operational Dashboards¶

Real-Time Operations Dashboard¶

Daily Health Report¶

Part 6: Incident Response Playbooks¶

High Error Rate (>5%)¶

High CPU (>80%)¶

Database Connection Exhaustion¶

Part 7: Capacity Planning¶

Baseline v0.0.0 (1 Customer)¶

Upgrade Triggers¶

Part 8: Monitoring Checklist¶

Daily¶

Weekly¶

Monthly¶

Related Documentation¶

Version History¶