Scalability Workflow

System Dataflows, Queues & Scale-Out Architecture

Security Mandate

This document blueprints the performance dataflow, caching matrix, asynchronous background processing architecture, and memory optimization layers for the Level 4 HMIS.

1. High-Throughput Concurrent Dataflow Architecture

The system optimizes for the highly asynchronous nature of a busy Kenyan Level 4 facility. During peak mid-morning hours, registration, triage, labs, and pharmacies generate complex read/write operations that must be handled smoothly without freezing the primary UI threads.

Rendering Chart

2. Multi-Tiered Redis Caching Framework

To reduce database roundtrips, the system implements a strict caching design. Read-heavy, static, or slow-changing domain data maps directly to multi-layered Redis cache stores.

2.1 Cache Retention & Invalidation Strategy

Cache Key Pattern	Stored Data Objects	Eviction Strategy	TTL	Invalidation Triggers
`keml:drugs:v1`	Complete Kenya Essential Medicines List reference array	volatile-lru	72 Hours	Procurement updates or KEML version updates.
`bed:occupancy:global`	Active bed maps, ward counts, availability metrics	noeviction	5 Minutes	Dynamic bed allocations, ward transfers, patient discharges.
`patient:session:auth:*`	Active encrypted RBAC access tokens, session vectors	allkeys-lru	15 Minutes	Explicit logout event or security termination block.
`tariff:nhif_sha:prices`	Approved SHA procedures compensation values matrix	volatile-lru	24 Hours	Statutory adjustment updates from the Ministry of Health.

2.2 Defensive Cache Lookaside Implementation

The system wraps query reads in a declarative lookaside pattern to prevent database traffic spikes during sudden power restorations or system reboots:

import json
from typing import Dict, Optional
from django.core.cache import cache
from apps.ancillary.models import KEMLInventory

def fetch_drug_metadata_defensive(item_code: str) -> Optional[Dict]:
    """
    Lookaside caching pattern ensuring sub-millisecond inventory reads 
    while shielding PostgreSQL from heavy concurrent query traffic.
    """
    cache_key = f"inventory:item:{item_code}"
    
    # Attempt immediate memory cache hit
    cached_data = cache.get(cache_key)
    if cached_data:
        return json.loads(cached_data)
        
    try:
        # Fallback to physical database engine read on cache miss
        db_record = KEMLInventory.objects.get(code=item_code, is_active=True)
        serialized_payload = {
            "id": str(db_record.id),
            "name": db_record.name,
            "code": db_record.code,
            "current_stock": db_record.quantity,
            "unit_price_kes": float(db_record.unit_price)
        }
        
        # Hydrate the cache memory store with a protective 1-hour TTL
        cache.set(cache_key, json.dumps(serialized_payload), timeout=3600)
        return serialized_payload
        
    except KEMLInventory.DoesNotExist:
        return None

3. Asynchronous Queue Processing via Huey

Background tasks, heavy analytics calculation engines, and downstream integrations are completely separated from the HTTP request-response thread lifecycle using Huey.

Rendering Chart

3.1 Production Worker Ingestion Configuration

The Huey queue infrastructure is tuned to run efficiently on our dual physical server setup. It uses task prioritization to ensure that patient-facing services (like printing prescriptions) are processed ahead of backend tasks (like running accounting reports).

# huey_config.py
from huey import RedisHuey
from django.conf import settings

# Initialize high-performance Huey engine backed by local Redis instance
huey_engine = RedisHuey(
    'hmis-task-scheduler', 
    url=settings.REDIS_URL,
    connection_pool_kwargs={'max_connections': 100}
)

# Task Priority Configuration Boundaries
PRIORITY_CRITICAL = 3   # e.g., Lab machine results ingestion, eMAR push
PRIORITY_STANDARD = 2   # e.g., M-Pesa STK Callback processing, SMS notifications
PRIORITY_LOW      = 1   # e.g., MoH Monthly indicator batch parsing, system data sync