Lead Site Reliability Engineer - Imunify Reliability Platform

The problem you'd own Imunify360 is a multi-layer Linux server security suite — WAF, IDS/IPS, malware scanning and cleanup, proactive defence, patch management, reputation — running as an agent on hundreds of thousands of customer servers, backed by a cloud estate of scanning, correlation and signature-delivery services on our own bare metal. Roughly 70 components currently ship without a defined service level indicator. Some are internal services we can scrape. Many are agent-side subsystems running on machines we do not own, reporting through a heartbeat we designed for something else. There is monitoring, and there are dashboards, and there is no coherent answer to the question "is this component doing its job right now, and how would we know if it stopped?". We know the cost of that gap precisely, because we recently paid it: a security control was silently disabled across a large fraction of the fleet for 61 days. Every dashboard was green. The telemetry reported a ruleset version but not whether the control that consumed it was switched on, so a configuration change was indistinguishable from a broken updater. Three independent safety mechanisms existed and all three were gated behind the same condition that caused the failure. Your job is to make that class of failure detectable in hours instead of months, across the whole product line, and to build the system that keeps it detectable as the product changes. This is a greenfield charter inside a brownfield estate. You are not inheriting an SRE team, an SLO framework or a paging culture. You are defining them, with the engineering leads, and then making them stick. What you'll do 1. Define what "working" means for ~70 components Run SLI definition with squad leads and senior engineers. You facilitate and hold the standard; the owning squad signs the SLI. Build the taxonomy this product actually needs, which is broader than availability and latency: Service SLIs — availability, latency, error rate for cloud-side services. Fleet SLIs — heartbeat reachability, version and configuration convergence across the installed base. Control-efficacy SLIs — the differentiator. What fraction of protected units have the control effectively enabled and current , not merely installed. Ruleset generation drift, signature age, scan coverage, enforcement-mode distribution. Delivery SLIs — artifact publish success, rule-to-fleet lead time, hotfix time-to-convergence. Pipeline SLIs — ingest lag, verdict latency, queue age, backlog burn. Enforce one non-negotiable design rule: an SLI must be measurable from outside the gate of the thing it measures. If the control being off also switches off the signal that would tell you it is off, the SLI is invalid. This is the lesson of the incident above and it is the reason this role exists. Attach an SLO, an error budget and an owning squad to each. Tiering is expected — not every component earns a 99.9% target or a pager. 2. Build the collection system Design and build the pipeline that gets these indicators off the fleet and into a queryable store: push-based, sampled, privacy-constrained, and with a cardinality budget you set and defend. Extend agent-side and service-side instrumentation where the signal does not exist yet, in Python, Go and Rust, working with the owning squads. Consolidate the current sprawl of dashboards, ad-hoc queries and reporting paths into a defensible set of instruments, and retire what does not earn its keep. 3. Build alerting and alert management Symptom-based, SLO-anchored alerting with multi-window burn-rate semantics. Not threshold soup. A three-tier taxonomy — page / ticket / dashboard — with an explicit rule for what is allowed to page a human at 03:00. Every alert ships with an owner, a runbook and a documented failure mode, or it does not ship. Alert hygiene as a standing practice: quarterly review, deletion counted as a win, actionable-rate tracked. A persistent inability to perform a security-relevant refresh should page. It currently logs a warning. 4. Build escalation Component → owning squad ownership map, kept current, machine-readable, and wired into routing so an alert reaches the right seven people rather than a shared channel. Severity matrix, acknowledgement SLAs, follow-the-sun rota design across UTC−5 … UTC+8, and clean handoff protocol. Incident command practice and blameless postmortems within 24 hours. We already do postmortems and do them honestly, including publicly retracting our own wrong findings; you raise the floor on the mechanical parts — timelines, ownership, action-item follow-through. Design the escalation system so that squads carry their own pagers. You build and operate the platform and coach on the practice; you are not the buffer that absorbs everyone else's alerts. Requirements What you'll bring Required (Must-haves): Substantial production-engineering or SRE experience, including at least one environment where you defined the SLO framework rather than inherited it. We will ask you to walk through SLIs you personally wrote and how you negotiated them with resistant teams. Strong Python. Comfortable reading and modifying Go or Rust — our agents are written in them and instrumentation lands there. Deep practical grip on time-series and event telemetry at scale: Prometheus/OpenMetrics, Grafana, an Alertmanager-class routing layer, and a columnar store for high-cardinality fleet data (ClickHouse or equivalent). Distributed systems debugging on bare metal and long-lived hosts. Most of this estate is not Kubernetes, and the reflexes that assume an orchestrator will not transfer cleanly. Configuration management and CI at production scale — Ansible, GitLab CI, Jenkins or close equivalents. The judgement to design measurement for machines you do not own and cannot scrape: push telemetry, sampling, clock skew, partial reporting, and the privacy constraints that come with running on a customer's server. Written communication that holds up async. This role is 40% telemetry engineering and 40% getting sixty engineers to agree on what "healthy" means; the remaining 20% is refusing to let the answer be a dashboard nobody reads. Valuable (Nice-to-haves): Security product background — WAF, EDR, AV, vulnerability management — and the instinct that a security control's SLI is about enforcement, not uptime. Monitoring under audit: SOC 2 CC7.x, ISO 27001 A.8.16, NIST SP 800-137 continuous monitoring. Some of this work is audit evidence and it helps if you have written for that audience. OpenTelemetry, eBPF, Sentry. Cost- and cardinality-aware telemetry design. Fluency with agentic development tooling — we run a Cursor/Claude-first SDLC with internal and third-party MCP servers, and engineers here are assessed on how well they work with it. Kubernetes, for the one workload that is on it. Not this role Not a DevOps ticket queue, not build-system ownership, not cloud cost management, not the on-call rota for other squads' services. First year, in outcomes 30 days: Component inventory with named owners. SLI taxonomy and tiering agreed. 3 pilot components fully instrumented end to end as the reference implementation. 90 days: Collection pipeline in production. Tier-1 components (the ones whose failure is a customer security exposure) carry SLO, alert, runbook, owner. Escalation routing live for tier-1. 180 days: All ~70 components have a defined SLI and an owner. Alert taxonomy enforced; page volume and actionable-rate measured and published. Squad on-call operating. 365 days: Mean time to detect a silent control-degradation is under 24 hours, measured, against a 61-day baseline. Error-budget policy influences release decisions. The function is documented well enough that hire #2 and #3 are additive, not archaeological. How we work Remote-first and async across nine time zones. Weekly PO sync and architecture sync; monthly demo and OKR review; quarterly architecture summit. Decisions land as ADRs. Every output carries an owner and a due date. Postmortems are blameless and published, and we correct ourselves on the record when we get something wrong. Benefits What's in it for you? A strong focus on professional development with opportunities for learning and growth: Interesting and challenging projects, Mentor and other knowledge-exchange programs; Fully remote work with flexible working hours, that allows you to schedule your day and work from any location worldwide; Paid 24 days of vacation per year, 10 days of national holidays, and unlimited sick leaves to ensure you maintain a healthy work-life balance; Compensation for private medical insurance; Co-working and gym/sports reimbursement; The opportunity to receive a reward for the most innovative idea that the company can patent, fostering a culture of creativity and innovation.