Request Logging

Advanced Request Logging Configuration

The advanced request logging configuration allows to learn more about how the custom logging is configured, and provide intuition on how it could be extended.

Make sure that before you go ahead with this installation, you have all the pre-requisites listed in the Installation Overview Page.


This page will show you how to set up request logging, which is used to log the audit of every input and ouput from the models, which then can be used for reproducibility, debugging or requesting explanations.

This page contains the instructions of how to set up your own ELK stack from scratch, as well as the instructions for how to set up the request logger with or without KNative (the KNative one is the default).

The Request Logging is based in the open source Seldon Core Request logging.

Further Pre-requisites

The default payload logging depends on KNative (which also has a dependency in Istio). You can configure these by running the script in seldon-core/centralised-logging/request-logging/.

KNative and istio are not hard requirements. See Running without KNative at the bottom of this document.

Running from base repository

In order to run all relevant scripts you should make sure you are in the base folder in seldon-core/centralised-logging.

Install ELK Stack


First we install elasticsearch from the elastic repo. We use version 7.5.2 but we also support version 7.6.0:

kubectl create namespace logs || echo "namespace logs exists"
helm upgrade --install elasticsearch elasticsearch --version 7.5.2 --namespace=logs --set service.type=ClusterIP --set antiAffinity="soft" --repo
kubectl rollout status statefulset/elasticsearch-master -n logs
Fluentd & Kibana

For log collection from each of the Seldon deployment models, we use Fluentd to collect the logs and Kibana to visualise them.

helm upgrade --install fluentd fluentd-elasticsearch --namespace=logs -f fluentd-values.yaml --repo
helm upgrade --install kibana kibana --version 7.5.2 --namespace=logs --set service.type=ClusterIP -f ./kubeflow/kibana-values.yaml --repo
Istio Routes

In order to make sure these are reachable we need to apply the istio virtual service through the routes.

kubectl apply -f ./kubeflow/virtualservice-kibana.yaml
kubectl apply -f ./kubeflow/virtualservice-elasticsearch.yaml

kubectl rollout status deployment/kibana-kibana -n logs

Running with KNative

These are the instructions to set up the request logger with KNative which is the default configuration. For the instructions to install KNative with default configurations you can set it up below.

Install Request Logger

Now we set up the request logger which will run on top of Knative eventing.

This is the component that will turn on only when a request is sent to a model to collect the contents and forward the processed log into elasticsearch.

# Delete logger if existing as otherwise get 'expected exactly one, got both' err if existing resource is v1alpha1
kubectl delete -f ./request-logging/seldon-request-logger.yaml || true
kubectl apply -f ./request-logging/seldon-request-logger.yaml
# remove and recreate broker if already have one to activate eventing
kubectl delete broker -n default default || true
kubectl label namespace default knative-eventing-injection- --overwrite=true
kubectl label namespace default knative-eventing-injection=enabled --overwrite=true
Setup Trigger

Once the knative request logger is set up, now we just need to configure the trigger:

kubectl -n default get broker default

kubectl apply -f ./request-logging/trigger.yaml
Check Setup

In order to check that it has been successfully set up, the only thing you need is to send a request through Seldon Deploy and check if a request logger pod has been created.

Custom Request Logger

It’s possible for you to add your own custom request logger, with any custom loggic you’d like to add. In order to do this, you need to make sure each Seldon Deployment points to the endpoint of your custom request logger.

For this, you will need to make sure you enable the Seldon Operator Environment variable for the logger. Prior to v1.2.x this was "EXECUTOR_REQUEST_LOGGER_DEFAULT_ENDPOINT_PREFIX" and then became "EXECUTOR_REQUEST_LOGGER_DEFAULT_ENDPOINT". It can be enabled through the core helm chart’s executor.requestLogger.defaultEndpoint.

Below we show an example of how you would do this for our non-knative default request logger

Running without KNative

It’s also possible to set up request logging without KNative dependency. For this, you will have to run a non-knative request logger, which you can trigger by running the configuration below.

Make sure that you edit the elasticsearch host variable below to point to the correct elasticsearch service address.

Important: you need to make sure that you are running a request logger in every namespace where you plan to deploy seldon core models

apiVersion: apps/v1
kind: Deployment
  name: seldon-request-logger
  namespace: default
    app: seldon-request-logger
  replicas: 2
      app: seldon-request-logger
        app: seldon-request-logger
        - name: user-container
          imagePullPolicy: Always
            - name: ELASTICSEARCH_HOST
              value: "elasticsearch-master.logs.svc.cluster.local"
            - name: ELASTICSEARCH_PORT
              value: "9200"
apiVersion: v1
kind: Service
  name: seldon-request-logger
  namespace: default
    app: seldon-request-logger
    - protocol: TCP
      port: 80
      targetPort: 8080
Configure Seldon Core to use that endpoint

In order to make sure the Seldon Deployments are sending the requests to that endpoint, you will need to make sure you provide the request logger prefix. In this case you will need the following extra attributes in the Seldon Core values.yaml:

          value: "http://seldon-request-logger."

It’s important that you make sure it’s on that format, which is http://<LOGGER_SERVICE>.. The Seldon Service Orchestrator will then add the namespace where the Seldon DEployment is running as a suffix.

Overriding Request Logger Endpoint for specific Seldon Deployment

Once you have created the request logger, now you have to make sure your deployments are pointing to the correct custom request logger. You can set up the custom request logger address by adding the following configuration to every Seldon Core SeldonDeployment file:

        mode: all
        url: http://seldon-request-logger.default

The mode configuration can be set to request, response or all.

The url is where this should be logged to. There’s a similar example for KFServing.

Last modified July 7, 2020: upgrading to 0.7.0 (d0e5579)