This article will brief you on the lessons I learnt when coming up with a health endpoint for a backend service am working on.

First, What’s a health endpoint?… I would describe it as an api endpoint that returns the status of a services used by an application server, occasional it can return information about the resources available for the service.

What/Who consumes the health endpoint?

Humans hit the endpoint once in a while to check the status of services but the main consumers are robots/probes. The main reason for this is that robots/probes can effectively react to the response it gets. Additional logic can be plugged in to conditionally handle a response, e.g. when a service goes down it can automatically route alerts to various teams. Robots are not humans, they therefore can handle repetitive tasks very well and without any distractions.

Structure of a health endpoint

So health endpoint returns information in JSON format, well at least most do. Regardless of your return type one thing can remain constant, and that is the http response status code i.e. return 200 if all is okay and 503 if any service is down. It’s quicker to interpret the status code than the response body. Sample response body from a health endpoint:

{
  "problems": {},
  "services": {
    "postgres": true,
    "redis": true,
    "rabbitmq": true
  },
  "serverTime": "2021-11-01T09:44:43.584+03:00",
  "buildVersion": "0.0.1"
}

Designing for health endpoint

1. Identify the services you want to monitor.

   The services to be checked should be the ones under ones’ control or otherwise crucial to the overall working of the application. e.g. another microservice e.g. authenticating and authorization service, database, queueing service.

2. For each of those services do R&D of their connection setup i.e. libraries used and how they initiate, maintain and close connection to the external service. The main areas to check here are the connection/read/query timeout configuration, whether it closes connection after request and connection pool configuration among others.

   • Ensure the connection/read/query timeout is configurable. The timeout should be set or have a default value it should not be blank even if you expect a health check to take >10s.
   • Ensure the connection is closed after the health check request else the connection pool will be exhausted.
   • Ensure you have an accommodating connection pool. This can derive from a load test, based on the traffic expectation on your backend service.

3. Define the max response time for the health endpoint. Health endpoints should be designed to return information ASAP therefore health checks for services should not run for a long time preferable not longer that ~5s. If possible run the checks in parallel (for languages that support multi-threading).

4. Design a suitable interface/facade (for languages that support it) for the health indicators. Sample

   public interface ServiceHealthIndicator {\
       Callable<ModelMap> doHealthCheck();
   }
   

   Sample Health Endpoint Implementation (showing a postgres health endpoint setup)

     @Component
     public class PostgresServiceHealthIndicator implements ServiceHealthIndicator {
     private static final Logger logger = LogManager.getLogger(PostgresServiceHealthIndicator.class.toString());
     @Value("${['postgres.queryTimeout'] ?: 2000 }")
     private Integer queryTimeout;
     @Autowired
     private DataSource dataSource;
     private final String HEALTH_INDICATOR_KEY = "postgres";
     @Override
      public Callable<ModelMap> doHealthCheck() {
       return () -> {
        ModelMap modelMap = new ModelMap();
        boolean result = false;
        try {
         JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource);
         jdbcTemplate.setQueryTimeout(queryTimeout);
         result = jdbcTemplate.queryForObject("SELECT 1;", Integer.class) == 1;
        }
        catch (Exception e) {
         logger.error(e);
         modelMap.put(Constants.HealthIndicator.EXCEPTION, e.getMessage());
        }
        modelMap.put(Constants.HealthIndicator.STATUS, result);
        modelMap.put(Constants.HealthIndicator.INDICATOR, HEALTH_INDICATOR_KEY);
        return modelMap;
       };
      }
     }
   

And that’s it … do the same for all the other services.

Hope this helps one way or another.

• api
• backend