🚀 The feature Presently, TorchServe operates within a singular Python environment, limiting its flexibility for users who rely on multiple Python environments for their machine learning projects. This proposal aims to extend TorchServe's capabilities by introducing support for Python virtual environ...

Hey all, My model's inference is quite long-running (around 50 seconds per request), so it would be great if closed client connections are handled properly by interrupting the inference that's currently in progress. I'm currently implementing initialize, preprocess, inference and postprocess methods...

🐛 Describe the bug Hello, I am trying to use the StreamPredictions2 gPRC method but the model-server produces an internal error when executing it. I successfully served a model, I tested it with API REST and some of the gRPC methods and works fine. The problem comes when using the StreamPredictions2...

📚 The doc issue No different time inferent when I change minWorkers, maxWorkers from 1 to 2 or 3. Three different worker give me a same time to handle 1000 request by using multi.thread. This is a bug or I have mistake when setting config.properties? My config.properties: inference_address=http://0....

🐛 Describe the bug Describe When i try to register a new model.mar file the system return an error and this is in log file 2023-11-23T13:13:19,271 [WARN ] W-9000-logreg_2.0-stderr MODEL_LOG - from ts.arg_parser import ArgParser 2023-11-23T13:13:19,272 [WARN ] W-9000-logreg_2.0-stderr MODEL_LOG -...

📚 The doc issue I've created a custom handler with the following initialize method class CustomHandler(VisionHandler): def initialize(self, context): print("Got here 000!") time.sleep(20) print("Got here 111!") super(VisionHandler, self).__init__() I spin up the s...

📚 The doc issue I've created a custom handler with the following initialize method class CustomHandler(VisionHandler): def initialize(self, context): print("Got here 000!") time.sleep(20) print("Got here 111!") super(VisionHandler, self).__init__() I spin up the s...

🚀 The feature Enable backwards compatibility for metrics behavior in versions <=0.8.0 by adding support for a new metrics mode: legacy Motivation, pitch In versions <=0.8.0: ts_inference_requests_total, ts_inference_latency_microseconds and ts_queue_latency_microseconds are only available via...

🚀 The feature Periodically dump the updated metrics configuration inclusive of auto detected(#2747) metrics to the metrics configuration file. Motivation, pitch As metrics not defined in the metrics configuration file are auto detected(#2747) and included in the frontend metrics cache, the metrics c...

🐛 Describe the bug On loading the docker image, I see that the workflow store is pointing to model_store instead of wf_store Error logs docker run --rm -it -p 127.0.0.1:8080:8080 -p 127.0.0.1:8081:8081 -p 127.0.0.1:8082:8082 -v $(pwd)/wf_store:/home/model-server/wf-store -v $(pwd)/model_store:/home/...