SRE, Observability Platform

Posted May 23

Site Reliability Engineer, Observability Platform

The GitLab DevSecOps platform empowers 100,000+ organizations to deliver software faster and more efficiently. We are one of the world’s largest all-remote companies with 2,000+ team members and values that foster a culture where people embrace the belief that everyone can contribute. Learn more about Life at GitLab.

Site Reliability Engineers (SREs) are responsible for keeping all user-facing services and other GitLab production systems running smoothly. SREs are a blend of pragmatic operators and software craftspeople that apply sound engineering principles, operational discipline, and mature automation to our operating environments and the GitLab codebase. SREs specialize in systems (operating systems, storage subsystems, networking), while implementing best practices for availability, reliability and scalability, with varied interests in algorithms and distributed systems.

The Observability Team's mission is to Build, Run and Own the entire lifecycle of the suite of services that enable observability of the GitLab SaaS environments. These services allow Infrastructure, Development & Product teams to observe running products and contribute to our overall reliability and scalability goals. In this role, you will maintain metrics and logs platforms for GitLab SaaS and you will be responsible for Prometheus, Thanos, Grafana, and Logging.

GitLab.com is a unique site and brings with it unique challenges: it is the largest GitLab instance in existence (and in fact, one of the largest single-tenancy open-source SaaS sites on the Internet). The team’s experience feeds back into other Engineering groups within the company, as well as to GitLab customers running self-managed installations.

As an SRE you will:

  • Be on an on-call (PagerDuty) rotation to respond to incidents that impact GitLab.com availability, and provide support for service engineers with customer incidents.
  • Use your on-call shift to prevent incidents from ever happening.
  • Run our infrastructure with Chef, Ansible, Terraform, GitLab CI/CD, and Kubernetes.
  • Build monitoring that alerts on symptoms rather than on outages.
  • Document every action so your findings turn into repeatable actions and then into automation.
  • Use the GitLab product to run GitLab.com as a first resort and improve the product as much as possible
  • Improve operational processes (such as deployments and upgrades) to make them as boring as possible.
  • Design, build and maintain core infrastructure that enables GitLab scaling to support hundreds of thousands of concurrent users.
  • Debug production issues across services and levels of the stack.
  • Plan the growth of GitLab’s infrastructure.

You may be a fit to this role if you have some of these inclinations:

  • Experience with Kubernetes deployment and management
  • Experience with Elastic Cloud, Loki, Fluentd, Promtail or other logging systems tools
  • Experience with Promethus and/or Thanos deployment management 
  • Think about systems: edge cases, failure modes, behaviors, specific implementations.
  • Experience with or exposure to Bigquery and general cloud providers such as GCP and AWS
  • Know your way around Linux and the Unix Shell.
  • Know what is the use of configuration management systems like Terraform, Chef and/or Ansible.
  • Have an urge to collaborate and communicate asynchronously.
  • Have an urge to document all the things so you don’t need to learn the same thing twice.
  • Have an enthusiastic, go-for-it attitude. When you see something broken, you can’t help but fix it.
  • Have an urge for delivering quickly and effectively, and iterating fast.
  • Share our values, and work in accordance with those values.
  • Ability to use GitLab

Projects you could work on:

  • Coding infrastructure automation with Chef, Ansible, Terraform, and GitLab CI/CD
  • Improving our Prometheus Monitoring or building new metrics
  • Helping release managers deploy and fix new versions of GitLab-EE.
  • Plan, prepare for, and execute the migration of GitLab.com from virtual machines running on Google Cloud to cloud-native container-based deployments with Kubernetes using Google Kubernetes Engine.
  • Develop a relationship with a product group, define their SLAs, share GitLab.com data on those SLAs and improve their reliability

The base salary range for this role’s listed level is currently for residents of listed locations only. Grade level and salary ranges are determined through interviews and a review of education, experience, knowledge, skills, abilities of the applicant, equity with other team members, and alignment with market data. See more information on our benefits and equity. Sales roles are also eligible for incentive pay targeted at up to 100% of the offered base salary.

Colorado/Washington pay range

$124,300—$239,700 USD

California/New York/New Jersey pay range

$124,300—$266,400 USD

Country Hiring Guidelines: GitLab hires new team members in countries around the world. All of our roles are remote, however some roles may carry specific location-based eligibility requirements. Our Talent Acquisition team can help answer any questions about location after starting the recruiting process.  

Privacy Policy: Please review our Recruitment Privacy Policy. Your privacy is important to us.

GitLab is proud to be an equal opportunity workplace and is an affirmative action employer. GitLab’s policies and practices relating to recruitment, employment, career development and advancement, promotion, and retirement are based solely on merit, regardless of race, color, religion, ancestry, sex (including pregnancy, lactation, sexual orientation, gender identity, or gender expression), national origin, age, citizenship, marital status, mental or physical disability, genetic information (including family medical history), discharge status from the military, protected veteran status (which includes disabled veterans, recently separated veterans, active duty wartime or campaign badge veterans, and Armed Forces service medal veterans), or any other basis protected by law. GitLab will not tolerate discrimination or harassment based on any of these characteristics. See also GitLab’s EEO Policy and EEO is the Law. If you have a disability or special need that requires accommodation, please let us know during the recruiting process.