Monitoring Logging

Monitoring, Observability & Structural Logging Architecture

1. Structured JSON Application Logging Standard

To ensure that machine telemetry is parseable by automated scraping tools without adding high CPU overhead, standard text logs are replaced with structured JSON lines (jsonlines) written directly to standard output streams (stdout).

1.1 JSON Trace Event Format Schema

Every application event, validation failure, or module transaction must emit a structured layout matching this schema:

{
  "timestamp": "2026-06-22T17:58:09.102Z",
  "log_level": "WARNING",
  "domain_module": "ANCILLARY_PHARMACY",
  "trace_id": "tx_99a8b7c6-2f1d-4e3a-8b5c-6d7e8f9a0b1c",
  "user_context": {
    "user_id": "usr_phm_4120",
    "system_role": "Pharmacist"
  },
  "event": {
    "action": "PHARMACY_STOCK_SHORTAGE_INTERCEPT",
    "item_code": "KEML-DRG-0412",
    "requested_units": 50,
    "available_units": 12
  },
  "context": {
    "encounter_id": "enc_7a6b5c4d-3e2f-1a0b-9c8d-7e6f5a4b3c2d",
    "execution_duration_ms": 14.2
  }
}

1.2 Python Backend Logger Implementation

The core framework implements a unified diagnostic logging wrapper to enforce structural parameters automatically:

# shared_kernel/logging_engine.py
import json
import sys
import uuid
from datetime import datetime
from typing import Dict, Any

class StructuredApplicationLogger:
    """
    Unified high-performance application logger formatting runtime events 
    into explicit JSON lines for decoupled log analytics aggregation.
    """
    def __init__(self, module_name: str):
        self.module_name = module_name

    def emit_log(self, level: str, action: str, trace_id: uuid.UUID, context: Dict[str, Any]) -> None:
        payload = {
            "timestamp": datetime.utcnow().isoformat() + "Z",
            "log_level": level,
            "domain_module": self.module_name,
            "trace_id": str(trace_id),
            "event": {
                "action": action
            },
            "context": context
        }
        # Write directly to stdout stream for processing by container log managers
        sys.stdout.write(json.dumps(payload) + "\n")
        sys.stdout.flush()

# Instantiate dedicated module diagnostic interfaces
pharma_logger = StructuredApplicationLogger("ANCILLARY_PHARMACY")

2. Real-Time Application Health Probing

The reverse proxy layer (Caddy) uses active health probes to verify container status. If a process drops or becomes unresponsive, traffic is automatically rerouted to the passive node.

Rendering Chart

2.1 Production Health Endpoint Implementation

The internal health check performs swift, non-blocking check routines against backend dependencies before returning an operational token:

# apps/reporting/health.py
from ninja import Router
from django.db import connection
from django.core.cache import cache
import time

router = Router(tags=["System Reliability Operations"])

@router.get("/health/live", response={200: dict, 503: dict})
def evaluate_infrastructure_vitality(request):
    """
    Performs dynamic low-overhead pings across storage and caching layers
    to ensure local node application health before load routing.
    """
    diagnostics = {"status": "HEALTHY", "dependencies": {}}
    unhealthy_flag = False
    
    # 1. Verify PostgreSQL operational availability
    try:
        start_time = time.time()
        with connection.cursor() as cursor:
            cursor.execute("SELECT 1;")
        diagnostics["dependencies"]["postgres_db"] = {
            "status": "UP", 
            "latency_ms": round((time.time() - start_time) * 1000, 2)
        }
    except Exception as e:
        diagnostics["dependencies"]["postgres_db"] = {"status": "DOWN", "error": str(e)}
        unhealthy_flag = True

    # 2. Verify Redis in-memory lookup availability
    try:
        start_time = time.time()
        cache.set("health_probe_token", 1, timeout=5)
        if cache.get("health_probe_token") == 1:
            diagnostics["dependencies"]["redis_cache"] = {
                "status": "UP",
                "latency_ms": round((time.time() - start_time) * 1000, 2)
            }
        else:
            raise ValueError("Cache token payload mismatch.")
    except Exception as e:
        diagnostics["dependencies"]["redis_cache"] = {"status": "DOWN", "error": str(e)}
        unhealthy_flag = True

    if unhealthy_flag:
        diagnostics["status"] = "UNHEALTHY"
        return 503, diagnostics
        
    return 200, diagnostics

3. Local Operational Alert Thresholds & Routing

Because on-premise instances run without persistent access to cloud-based notification managers, alerting routines are managed directly on the local host using background processing loops.

3.1 Core Telemetry Threshold Triggers

Telemetry VectorDiagnostic Condition MetricValidation Profile WindowAlert Escalation Path
Storage CapacityHost Volume Disk Consumption 185%Evaluation loop every 10 minutesLevel 1: In-app System Admin Notification Dashboard Alert.
Database Lock Wait StateRow access contention wait times 15000 msImmediate trigger upon event captureLevel 2: Write event to forensic audit logs and restart stuck database transaction profiles.
Memory Over-allocationActive Node RAM Consumption 190%Monitored over a sliding 5-minute windowLevel 3: Terminate low-priority background reporting tasks; shift non-essential processes to standby mode.
API Failure RateOutbound HTTP 5xx Status Returns 15%Evaluation block over a 60-second windowLevel 2: Trigger automated circuit-breaker isolation protocols across external gateway networks.

4. Host Log Rotation Protection Policies

To prevent unchecked trace logging from consuming available disk space on local storage drives, host log containers apply strict space allocation caps.

Configure the global docker container logging parameters inside /etc/docker/daemon.json across all active hardware server tower installations:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "50m",
    "max-file": "3"
  }
}

This configuration guarantees that no individual process log accumulation expands past 150 megabytes total, protecting system drive resources from overflow risks.

Document Verification Block

Author: Ian Wataka - Backend DeveloperTarget Scope: Telemetry Metrics, Health Vectors, and Trace Logging Layouts

Statutory Reporting