Policy-aware server remediation flow
MERGED
Change details HXT-AUTO-214
Repository
hoxt-platform/ops-controller
Branch
feature/policy-remediation
Commit
a7c9f6e2b41d8c3
Parent
0bc41de91f740ae
Author
Tolga Tutuncuoglu
Created
2024-03-14 09:12:47 UTC
Reviewer
Engineering Review
Approved
2024-03-14 09:18:26 UTC
Merged by
release-bot
Merged
2024-03-14 09:21:11 UTC
Summary
- Evaluate incident signals against service telemetry and baseline state.
- Apply confidence and permission gates before automated server actions.
- Persist execution trace, verify post-action health and roll back on regression.
Files changed 5 files
81
81
def evaluate_incident(signal, server, actor):
82
- return predictor.classify(signal)
82
+ inference = predictor.classify(signal, context=server.telemetry)
83
+ impact = service_criticality(server.service_id)
84
+ confidence = calibrate(inference, server.baseline)
85
+ scope = permission_scope(actor, inference.recommended_action)
86
+ return remediation_policy(inference, impact, confidence, scope)
44
44
def authorize(plan, actor):
45
+ if plan.confidence < MIN_AUTOMATION_CONFIDENCE:
46
+ return require_review(plan, reason="confidence")
47
+ if not within_policy_scope(plan.action, actor, plan.service):
48
+ return require_review(plan, reason="permission_scope")
49
+ return execute_with_guardrails(plan)
102
102
def execute_remediation(plan):
103
+ snapshot = capture_pre_action_state(plan.server_id)
104
+ result = command_gateway.execute(plan.action, timeout=45)
105
+ write_audit_event(plan.trace_id, plan.action, result)
106
+ return verify_or_rollback(plan, snapshot, result)
31
31
result = execute_remediation(plan)
32
+health = verify_service_state(plan.server_id, window="90s")
33
+if health.regression_detected:
34
+ rollback(plan.change_id)
35
+ escalate(plan.trace_id, health)
17
17
memory_pressure:
18
+ min_confidence: 0.90
19
+ action: restart_worker_pool
20
+ require_scope: service.remediate
21
+ postcheck_window: 90s