// automate

Self-healing infrastructure

Connects New Relic incidents to Ansible Automation Platform so an alert triggers automatic investigation and safe cleanup on the affected host. In testing, starting with disk alerts.

Role
Engineer
Status
In development

Ansible Automation Platform · Event-Driven Ansible · Python · New Relic · NerdGraph

Problem

Disk alerts need a person to log in, investigate and clean up the host.

Architecture

Playbook on the affected host

SERVICE

Find full filesystems

85% or more

SERVICE

Largest directories

Reported

SERVICE

Log cleanup candidates

Preview only

SERVICE

Clear package caches

dnf, yum, apt, zypper, pacman

SERVICE

Report usage again

EXTERNAL

New Relic

Incidents via NerdGraph and NRQL

SERVICE

Event source plugin

Python, asyncio, polls every 60s

SERVICE

Rulebook

Matches disk incidents

SERVICE

AAP job template

Limited to the affected host

Self-healing infrastructure (illustrative)Drag nodes to explore · Illustrative and simplified. Not a real topology.
Dependencies (text)
  • source -> nr [polls, deduplicates]
  • rulebook -> source [new incidents]
  • rulebook -> job [launches with incident context]
  • job -> find [runs]
  • find -> dirs
  • dirs -> logs
  • logs -> cache
  • cache -> report

A custom Python event source polls New Relic every 60 seconds and emits new incidents, deduplicated. A rulebook matches disk incidents and launches a job template limited to the affected host, passing the incident context: ID, severity, mount point and environment.

What the playbook does

  • Finds filesystems at 85% or more and shows the largest directories.
  • Lists log cleanup candidates as a preview only.
  • Clears package manager caches and reports usage again.

Decisions

  • Conservative remediation. Nothing destructive runs automatically.
  • Scoped to the affected host, with the incident context passed in.
  • Plain Python event source (standard library only, asyncio).

Result

  • In testing, starting with disk alerts.

← Selected work