Senior Principal Site Reliability Engineer (Chaos Engineering) at Bybit | Mango CV
B
Senior Principal Site Reliability Engineer (Chaos Engineering)
Company: Bybit
Job description
About Us
Established in 2018, Bybit is one of the world’s leading cryptocurrency exchanges and digital financial platforms, serving over 80 million users across more than 200 countries and regions. Powered by world-class technology and a user-first mindset, Bybit delivers a seamless ecosystem across trading, payments, wealth management, custody, institutional services, and Web3 — connecting users to the future of digital finance.
Our core values define how we build. We listen, care and improve to create products and experiences that put users first. Backed by a global team of ambitious builders, problem-solvers, and innovators, we foster a high-performance and fast-moving environment where talent is empowered to drive real impact at the global scale. Supported by 24/7 multilingual customer service and a strong commitment to innovation, we are shaping the future of finance through technology, collaboration, and bold execution.
Today, Bybit is recognized as one of the most trusted and transparent platforms in the digital asset industry, continuing to expand its global presence while building the infrastructure for the next generation of financial services.
About This Role
As the Core Architect of Bybit’s Chaos Engineering Platform, you will be responsible for designing and building an enterprise-grade chaos engineering platform from the ground up (0 to 1), enabling fault injection and resilience validation in production environments across 2,000+ microservices.
Your mission goes beyond simply building a platform. You will drive continuous system resilience validation, with the ultimate goal of reducing the occurrence of P0/P-1 severity incidents by over 50%.
Similar jobs
Core Responsibilities
Chaos Engineering Platform Architecture & Development (50%)
Design and build an enterprise-grade chaos engineering platform supporting multi-cluster (K8s + EC2 hybrid), multi-region, and multi-environment (testnet/mainnet) deployments
Deep integration with existing monitoring, alerting, and SLO systems to achieve an automated closed loop: inject fault → observe impact → determine pass/fail
Production Resilience Validation Framework (30%)
Define safety standards and approval workflows for mainnet fault injection
Design and drive routine chaos experiments
Daily patrol-level experiments: Low-risk experiments executed automatically on a daily/weekly basis
Periodic validation experiments: Monthly/quarterly resilience verification of critical paths
Establish a resilience scoring system to quantify system health based on experiment results
Deliver improvement recommendations and drive business teams to remediate identified weaknesses
Technology Selection & Team Enablement (20%)
Evaluate and select the technology foundation (Chaos Mesh / Litmus / custom components — hybrid strategy)
Develop chaos engineering best practices and playbooks to enable SRE teams and application developers
Mentor and grow the team (2–3 engineers) in chaos engineering capabilities
Stay current with industry developments and introduce cutting-edge practices (e.g., AI-driven fault scenario discovery)
Key Requirements
Must-Have
8+ years of backend / infrastructure engineering experience, with 3+ years dedicated to chaos engineering or stability engineering
Hands-on experience with large-scale fault injection in production environments (not just test environments), with deep understanding of production safety constraints
Expert-level proficiency in Kubernetes fault injection (Chaos Mesh / Litmus / custom solutions), familiar with CRD / Operator development
Proficient in at least one backend language (Go preferred), with platform-level system architecture design capability
Deep understanding of distributed system failure modes (network partitions, split-brain, cascading failures, data inconsistency, etc.)
About Us Established in 2018, Bybit is one of the world’s leading cryptocurrency exchanges and digital financial platforms, serving over 80 million users across more than 200 countries and regions. Powered by world-class technology and a user-first mindset, Bybit delivers a seamless ecosystem…
About Us Established in 2018, Bybit is one of the world’s leading cryptocurrency exchanges and digital financial platforms, serving over 80 million users across more than 200 countries and regions. Powered by world-class technology and a user-first mindset, Bybit delivers a seamless ecosystem…
About Us Established in 2018, Bybit is one of the world’s leading cryptocurrency exchanges and digital financial platforms, serving over 80 million users across more than 200 countries and regions. Powered by world-class technology and a user-first mindset, Bybit delivers a seamless ecosystem…
About Us Established in 2018, Bybit is one of the world’s leading cryptocurrency exchanges and digital financial platforms, serving over 80 million users across more than 200 countries and regions. Powered by world-class technology and a user-first mindset, Bybit delivers a seamless ecosystem…
About Us Established in 2018, Bybit is one of the world’s leading cryptocurrency exchanges and digital financial platforms, serving over 80 million users across more than 200 countries and regions. Powered by world-class technology and a user-first mindset, Bybit delivers a seamless ecosystem…
About Us Established in 2018, Bybit is one of the world’s leading cryptocurrency exchanges and digital financial platforms, serving over 80 million users across more than 200 countries and regions. Powered by world-class technology and a user-first mindset, Bybit delivers a seamless ecosystem…