|
Let¡¯s look first at how violations are actually detected.
We focus on four Golden Signals of system health defined by Google: Latency, Errors, Traffic, and Saturation.
Some example SLIs that we are interested in are: P95, P98, or AVG of latency, error rate of requests, which include: timeout, drop, or error requests because of rating fail or database error.
SLO violation is detected at the Gateways using collected metrics and alerting rules E.g. request success rate of a service falls below 98 percent for a minute.
Our monitoring system allows all problems in CHF/PCF, ABMF, and the database to be seen at the gateways.
|