KServe.Model base class mainly defines three handlers preprocess, predict and postprocess, these handlers are executed in sequence, the output of the preprocess is passed to predict as the input, the predictor handler executes the inference for your model, the postprocess handler then turns the raw prediction result into user-friendly inference response. There is an additional load handler which is used for writing custom code to load your model into the memory from local file system or remote model storage, a general good practice is to call the load handler in the model server class __init__ function, so your model is loaded on startup and ready to serve prediction requests.

model.py

Note

return_response_headers=True can be added to return response headers for v1 and v2 endpoints

Build Custom Serving Image with BuildPacks

Buildpacks allows you to transform your inference code into images that can be deployed on KServe without needing to define the Dockerfile. Buildpacks automatically determines the python application and then install the dependencies from the requirements.txt file, it looks at the Procfile to determine how to start the model server. Here we are showing how to build the serving image manually with pack, you can also choose to use kpack to run the image build on the cloud and continuously build/deploy new versions from your source git repository.

You can use pack cli to build and push the custom model server image

Tip

If your buildpack command fails, make sure you have a .python-version file with the correct python version specified and a Procfile with correct entrypoint and arguments.

Deploy Locally and Test

Launch the docker image built from last step.

Send a test inference request locally with input.json

Expected Output

Deploy the REST Custom Serving Runtime on KServe

custom.yaml

In the custom.yaml file edit the container image and replace ${DOCKER_USER} with your Docker Hub username.

Arguments

You can supply additional command arguments on the container spec to configure the model server.

-workers: Spawn the specified number of uvicorn workers(multi-processing) of the model server, the default value is 1, this option is often used to help increase the resource utilization of the container.

-http_port: The http port model server is listening on, the default REST port is 8080.

-grpc_port: The GRPC Port listened to by the model server. Default is 8081.

-max_threads: The max number of gRPC processing threads. Default is 4.

-enable_grpc: Enable gRPC for the model server. Default is true.

-grpc_max_send_message_length: The max message length for gRPC send message. Default is 8388608 bytes (8 MB).

-grpc_max_receive_message_length: The max message length for gRPC receive message. Default is 8388608 bytes (8 MB).

-model_name: The model name deployed in the model server, the default name the same as the service name.

-max_asyncio_workers: Max number of workers to spawn for python async io loop, by default it is min(32,cpu.limit + 4).

-enable_latency_logging: Whether to log latency metrics per request, the default is True.

-configure_logging: Whether to configure KServe and Uvicorn logging, the default is True.

-log_config_file: The path of the Python config file configuration to use (can be a json, a yaml file or any other supported file format by python logging module). This file allows to override the default Uvicorn configuration shipped with KServe. The default is None.

-access_log_format: A string representing the access log format configuration to use. The functionality is provided by the asgi-logger library and it allows to override only the uvicorn.access's format configuration with a richer set of fields (output hardcoded to stdout). This limitation is currently due to the ASGI specs that don't describe how access logging should be implemented in detail (please refer to this Uvicorn github issue for more info). By default is None.

enable_latency_logging: whether to log latency metrics per request, the default is True.

-enable_docs_url: Enable docs url '/docs' to display Swagger UI.

Environment Variables

You can supply additional environment variables on the container spec.

STORAGE_URI: load a model from a storage system supported by KServe e.g. pvc:// s3://. This acts the same as storageUri when using a built-in predictor. The data will be available at /mnt/models in the container. For example, the following STORAGE_URI: "pvc://my_model/model.onnx" will be accessible at /mnt/models/model.onnx

PROTOCOL: specify the protocol version supported by the model e.g V1. This acts the same as protocolVersion when using a built-in predictor.

KSERVE_LOGLEVEL: sets the kserve and kserve_trace's logger verbosity. Default is INFO.

Apply the YAML to deploy the InferenceService on KServe

kubectl

Expected Output

Run a Prediction

The first step is to determine the ingress IP and ports and set INGRESS_HOST and INGRESS_PORT

Expected Output

Delete the InferenceService

Create and Deploy Custom gRPC ServingRuntime

KServe gRPC ServingRuntimes enables high performance inference data plane which implements the Open(v2) Inference Protocol:

gRPC is built on top of HTTP/2 for addressing the shortcomings of head-of-line-blocking and pipelining

gRPC transports binary data format with Protobuf which is efficient to send over the wire.

Compared to REST it has limited support for browser and the message is not human-readable which requires additional debugging tools.

Setup

Install pack CLI to build your custom model server image.

The code samples can be found in the KServe website repository.

Implement Custom Model using KServe API

For Open(v2) Inference Protocol, KServe provides InferRequest and InferResponse API object for predict, preprocess, postprocess handlers to abstract away the implementation details of REST/gRPC decoding and encoding over the wire.

model_grpc.py

Build Custom Serving Image with BuildPacks

Similar to building the REST custom image, you can also use pack cli to build and push the custom gRPC model server image

Tip

If your buildpack command fails, make sure you have a .python-version file with the correct python version specified and a Procfile with correct entrypoint and arguments.

Deploy Locally and Test

Launch the docker image built from last step with buildpack.

Send a test inference request locally using InferenceServerClient grpc_client.py

grpc_client.py

Expected Output

Deploy the gRPC Custom Serving Runtime on KServe

Create the InferenceService yaml and expose the gRPC port by specifying on ports section, currently only one port is allowed to expose and by default HTTP port is exposed.

custom_grpc.yaml

In the custom_grpc.yaml file edit the container image and replace ${DOCKER_USER} with your Docker Hub username.

Arguments

You can supply additional command arguments on the container spec to configure the model server.

-grpc_port: the http port model server is listening on, the default gRPC port is 8081.

-model_name: the model name deployed in the model server, the default name the same as the service name.

Apply the yaml to deploy the InferenceService on KServe

kubectl

Expected Output

Run a gRPC Prediction

The first step is to determine the ingress IP and ports and set INGRESS_HOST and INGRESS_PORT

Send an inference request to the gRPC service using InferenceServerClient grpc_client.py.

Expected Output

Parallel Model Inference

By default, the models are loaded in the same process and inference is executed in the same process as the HTTP or gRPC server, if you are hosting multiple models the inference can only be run for one model at a time which limits the concurrency when you share the container for the models. KServe integrates RayServe which provides a programmable API to deploy models as separate python workers so, the inference can be performed in parallel when serving multiple custom models.

Setup

Install pack CLI to build your custom model server image.

The code samples can be found in the KServe website repository.

model_remote.py

Fractional GPU example

The more details for ray fractional cpu and gpu can be found here.

Build Custom Serving Image with BuildPacks

You can use pack cli to build the serving image which launches each model as separate python worker and web server routes to the model workers by name.

Tip

If your buildpack command fails, make sure you have a .python-version file with the correct python version specified and a Procfile with correct entrypoint and arguments.

Deploy Locally and Test

Launch the docker image built from last step.

Send a test inference request locally with input.json

Expected Output

Configuring Logger for Custom Serving Runtime

KServe allows users to override the default logger configuration of serving runtime and uvicorn server. The logger configuration can be modified in one of the following ways:

1. Providing logger configuration as a Dict:

If you are building a custom serving runtime and want to modify the logger configuration, this method offers the easiest solution. You can supply the logging configuration as a Python Dictionary to the kserve.logging.configure_logging() method. If the logging dictionary is not provided, KServe uses the default configuration KSERVE_LOG_CONFIG.

Note

The logger should be configured before doing any actual work. A recommended best practice is to configure the logger in the main, preferably as the first line of code. If the logger is configured later on in the source code, it may lead to inconsistent logger formats.

2. Providing logger configuration as a file:

The logger configuration can be provided as a file. If the filename ends with .json, KServe will treat the file as JSON Configuration. If the filename ends with .yaml or .yml, KServe will treat the file as YAML Configuration. Otherwise, The file will be treated as a configuration file in the format specified in the Python logging module documentation. This offers a more flexible way of configuring the logger for pre-built serving runtimes.

The model server offers a command line argument which accepts a file path pointing to the configuration. For example,

For, Custom serving runtimes, they should accept the file path in their source code.

Here is an example logging config in JSON format.

Here is an example using YAML format for configuring logger.

For other file formats, Please refer Python docs.

3. Disabling logger Configuration:

If you don't want Kserve to configure the logger then, You can disable it by passing the commandline argument --configure_logging=False to the model server. The command line argument --log_config_file will be ignored, if the logger configuration is disabled. In this case, the logger will inherit the root logger's configuration.

Note

If the logger is not configured at the entrypoint in the serving runtime (i.e. logging.configure_logger() is not invoked), The model server will configure the logger using default configuration. But note that the logger is configured at model server initialization. So any logs before the initialization will use the root logger's configuration.

模型推理运行时-Triton-HuggingFace

多模型服务 - 可扩展性问题