Staff Site Reliability Engineer - Splunk

<div class="content-intro"><p><strong>Secure Every Identity, from AI to Human<br><br></strong>Identity is the key to unlocking the potential of AI. Okta secures AI by building the trusted, neutral infrastructure that enables organizations to safely embrace this new era. This work requires a relentless drive to solve complex challenges with real-world stakes. We are looking for builders and owners who operate with speed and urgency and execute with excellence.<br><br>This is an opportunity to do career-defining work. We're all in on this mission. If you are too, let's talk.</p></div><div class="SingleJob-content"> <h3><strong>Position Overview:</strong></h3> <p>We are seeking a highly technical <strong>Staff</strong> <strong>Observability</strong> <strong>Site Reliability Engineer</strong> with a specialty in Splunk to own and evolve our Splunk ecosystem. In this role, you will move beyond simple monitoring to delivering a world class, comprehensive, scalable Observability Platform that enables our SRE teams and business partners. You will treat <strong>infrastructure as code</strong>—utilizing Terraform and strong coding proficiency in <strong>Go, Python, or Ruby</strong>—to automate the deployment of agents and collectors across complex distributed systems.</p> <p><strong>Key Responsibilities</strong></p> <ul> <li>Automated Infrastructure: Design, build, and maintain scalable observability infrastructure using tools like Terraform.</li> <li>Splunk Engineering: Optimize the collection, processing, and storage of log data to ensure high reliability and low latency of our Splunk services</li> <li>Incident Response: Participate in on-call rotations and lead post-incident reviews to drive systemic improvements and "observability-driven development."</li> <li>Automation: Eliminate "toil" by automating the deployment and scaling of observability agents and collectors.</li> </ul> <p><strong>Required Skills &amp; Experience (The Essentials)</strong></p> <p><strong>Log Management: Minimum 5+ Experience scaling and managing Splunk Cloud at scale (1000+ SVCs), including Workload Management (WLM) and HEC optimization. </strong><strong>Visualization: Expertise in creating intuitive, actionable Splunk dashboards that correlate data across multiple sources.<br></strong><strong>SRE Mindset: Minimum 5+ years of experience in an SRE, DevOps, or Systems Engineering role with a focus on high-availability systems.</strong></p> <ul> <li><strong>Programming Proficiency:</strong> Strong coding skills in <strong>SPL</strong>, <strong>Go</strong> for build