Advanced Request Logging Configuration
The advanced request logging configuration allows to learn more about how the custom logging is configured, and provide intuition on how it could be extended.
Make sure that before you go ahead with this installation, you have all the pre-requisites listed in the Installation Overview Page.
This page will show you how to set up request logging, which is used to log the audit of every input and ouput from the models, which then can be used for reproducibility, debugging or requesting explanations.
This page contains the instructions of how to set up your own ELK stack from scratch, as well as the instructions for how to set up the request logger with or without KNative (the KNative one is the default).
The Request Logging is based in the open source Seldon Core Request logging.
The default payload logging depends on KNative (which also has a dependency in Istio). You can configure these by running the
install_knative.sh script in seldon-core/centralised-logging/request-logging/.
KNative and istio are not hard requirements. See
Running without KNative at the bottom of this document.
Running from base repository
In order to run all relevant scripts you should make sure you are in the base folder in
Install ELK Stack
First we install elasticsearch from the elastic repo. We use version 7.5.2 but we also support version 7.6.0:
kubectl create namespace logs || echo "namespace logs exists" helm upgrade --install elasticsearch elasticsearch --version 7.5.2 --namespace=logs --set service.type=ClusterIP --set antiAffinity="soft" --repo https://helm.elastic.co kubectl rollout status statefulset/elasticsearch-master -n logs
Fluentd & Kibana
For log collection from each of the Seldon deployment models, we use Fluentd to collect the logs and Kibana to visualise them.
helm upgrade --install fluentd fluentd-elasticsearch --namespace=logs -f fluentd-values.yaml --repo https://kiwigrid.github.io helm upgrade --install kibana kibana --version 7.5.2 --namespace=logs --set service.type=ClusterIP -f ./kubeflow/kibana-values.yaml --repo https://helm.elastic.co
In order to make sure these are reachable we need to apply the istio virtual service through the routes.
kubectl apply -f ./kubeflow/virtualservice-kibana.yaml kubectl apply -f ./kubeflow/virtualservice-elasticsearch.yaml kubectl rollout status deployment/kibana-kibana -n logs
Running with KNative
These are the instructions to set up the request logger with KNative which is the default configuration. For the instructions to install KNative with default configurations you can set it up below.
Install Request Logger
Now we set up the request logger which will run on top of Knative eventing.
This is the component that will turn on only when a request is sent to a model to collect the contents and forward the processed log into elasticsearch.
# Delete logger if existing as otherwise get 'expected exactly one, got both' err if existing resource is v1alpha1 kubectl delete -f ./request-logging/seldon-request-logger.yaml || true kubectl apply -f ./request-logging/seldon-request-logger.yaml # remove and recreate broker if already have one to activate eventing kubectl delete broker -n default default || true kubectl label namespace default knative-eventing-injection- --overwrite=true kubectl label namespace default knative-eventing-injection=enabled --overwrite=true
Once the knative request logger is set up, now we just need to configure the trigger:
kubectl -n default get broker default kubectl apply -f ./request-logging/trigger.yaml
In order to check that it has been successfully set up, the only thing you need is to send a request through Seldon Deploy and check if a request logger pod has been created.
Custom Request Logger
It’s possible for you to add your own custom request logger, with any custom loggic you’d like to add. In order to do this, you need to make sure each Seldon Deployment points to the endpoint of your custom request logger.
For this, you will need to make sure you enable the Seldon Operator Environment variable for the logger. Prior to v1.2.x this was
"EXECUTOR_REQUEST_LOGGER_DEFAULT_ENDPOINT_PREFIX" and then became
"EXECUTOR_REQUEST_LOGGER_DEFAULT_ENDPOINT". It can be enabled through the core helm chart’s
Below we show an example of how you would do this for our non-knative default request logger
Running without KNative
It’s also possible to set up request logging without KNative dependency. For this, you will have to run a non-knative request logger, which you can trigger by running the configuration below.
Make sure that you edit the elasticsearch host variable below to point to the correct elasticsearch service address.
Important: you need to make sure that you are running a request logger in every namespace where you plan to deploy seldon core models
apiVersion: apps/v1 kind: Deployment metadata: name: seldon-request-logger namespace: default labels: app: seldon-request-logger spec: replicas: 2 selector: matchLabels: app: seldon-request-logger template: metadata: labels: app: seldon-request-logger spec: containers: - name: user-container image: docker.io/seldonio/seldon-request-logger:0.2.0 imagePullPolicy: Always env: - name: ELASTICSEARCH_HOST value: "elasticsearch-master.logs.svc.cluster.local" - name: ELASTICSEARCH_PORT value: "9200" --- apiVersion: v1 kind: Service metadata: name: seldon-request-logger namespace: default spec: selector: app: seldon-request-logger ports: - protocol: TCP port: 80 targetPort: 8080
Configure Seldon Core to use that endpoint
In order to make sure the Seldon Deployments are sending the requests to that endpoint, you will need to make sure you provide the request logger prefix. In this case you will need the following extra attributes in the Seldon Core
- name: EXECUTOR_REQUEST_LOGGER_DEFAULT_ENDPOINT value: "http://seldon-request-logger."
It’s important that you make sure it’s on that format, which is
http://<LOGGER_SERVICE>.. The Seldon Service Orchestrator will then add the namespace where the Seldon DEployment is running as a suffix.
Overriding Request Logger Endpoint for specific Seldon Deployment
Once you have created the request logger, now you have to make sure your deployments are pointing to the correct custom request logger. You can set up the custom request logger address by adding the following configuration to every Seldon Core SeldonDeployment file:
logger: mode: all url: http://seldon-request-logger.default
mode configuration can be set to
url is where this should be logged to. There’s a similar example for KFServing.