On-call responsibilities

IMPORTANT

Do NOT work on your team's sprint work during your on-call shift

---

Your primary responsibility

• Keep the ResponsibilityModel service healthy: triage and resolve support requests, and alerts; keep the main branch green; address VULN upgrades; leave the repo in a better state than you found it.

• The on-call engineer is the first line of defense for all EPM Platform related asks. You may not immediately know the answer, but you should be able to find the right people to escalate/discuss further with to give our customer’s meaningful responses. We need to know our service reliability concerns are being validated and worked towards a resolution as expediently as possible.

• Improved Customer Trust: Prompt and consistent communication with customers—such as immediate acknowledgements and timely updates—helps build and maintain trust.

• Investigate, delegate, orchestrate, communicate. This should apply to HOTs as well. You are the initial point-of-contact, but the expectation is not that YOU will fix all of the issues.

Your secondary responsibility

• Contributing to our backlog of TechOps/Tech Debt work: We need to ensure we’re constantly grooming and improving the EPM's performance and reliability to enhance the experience of our product— beyond just typical feature work and deliverables. There are meaningful things we can work on to uplift the plumbing of EPM, even if those things may not be immediately visible to customers— our Developer Experience (DX) is essential as well.

• If there are no items in the TechOps backlog, you may take it upon yourself to work on uplifting the service in other meaningful ways. If absolutely necessary, you may take on sprint work, but this should be seldomly done, and only as a last resort.

• Faster Response and Resolution: By not splitting attention with other feature development tasks, the on-call engineer can investigate and resolve incidents more expediently, minimizing downtime and customer impact (reduce TTD that we experience with this HOT)

---

At the beginning of your on-call rotation

1. Sync with the previous on-call for any outstanding follow-ups. This should be done during the Weekly Handoff meeting.
2. Ensure you see the automated Slack alerts your on-call week is beginning:
   • #help-enterprise-posture-management
   • #cap-epm-platform
   • #cap-octopus-dev
   • #octo-alerts-low
3. Review open HOTs/incidents, or any inflight work related to EPM Platform which needs to be handed off by the previous support engineer.

---

During your on-call rotation

Keep the main branch GREEN

• Monitor the Bitbucket Pipelines for this repo. If the main branch is RED, investigate promptly.
• Prioritize fixes for broken builds, flaky tests, and misconfigurations.
• If a test is truly flaky and blocks deploys, quarantine or skip it with a clear TODO and a follow-up ticket. Prefer fixing root causes first.

Handle VULN tickets and dependency hygiene

• Prioritize VULN upgrades and check VULN Funnel
  • Custom query that pulls VULNs assigned to any member of our team. Only works if we keep the query up to date!
  • Apr 7, 2025— the link above doesn’t work and this is the updated query; needs a review!
• EPM's ResponsibilityModel is integrated with Renovate for dependency management. Please ensure you’re keeping up with merging the open Renovate PRs. We will discuss them briefly in the handoff meeting. Prefer Renovate-driven dependency upgrades where possible; otherwise, perform targeted updates and verify builds locally.
• Coordinate with package owners if a change affects shared code.

Low-priority alerts and hygiene

• Treat recurring low-priority alerts as early indicators. Investigate patterns and create tickets with owners and next steps.

---

During non-business hours

• We are not currently a Tier 1, 24/7 Paid On-call service, and as such, there is no expectation to respond to issues/alerts outside of business hours. You may still be paged if some extreme HOT comes in, but this not the norm.
• Please check in to any new alerts that came in while outside of business hours once you return to the office, so you can promptly address them when you are back.

---

At the end of your on-call rotation

1. Discuss these in the On-call Handoff meeting:

   • Support engineer can provide clearer documentation and smoother handovers at the end of their rotation, ensuring continuity and knowledge transfer for ongoing issues.
   • Review generated RitualUI TechOps report,
   • Review the Operations Document to discuss any action items for the in-coming on-call engineer.
   • Observe major issues handled and their status
   • Any follow-up tickets created and owners
   • Any risks or watch items for the next on-call

2. Hand off any ongoing TechOps work, with links and context, such that the next support engineer can pick up where you left off.

---

Resources/Guides

• EPM Workflows 101

Team Repos