🐛 Describe the bug I am serving 3 models on my computer with GPU capacity, Torchserve is deployed as a docker container: server: image: pytorch/torchserve:test-gpu container_name: {container_name} restart: always volumes: - type: bind source: ./model_store tar...

🐛 Describe the bug About 30~40 seconds after torchserve --start ..., it prints some error messages and stops my model. The client side has not been involved yet. Error logs Start model torchserve --start --ncs \ --model-store /data/serving-models \ --ts-config config.properties...

🐛 Describe the bug My server deployed 8 models in the one gpu 24G VRAM, I tried to determine vram usage each model by command curl http://localhost:6868/models/{model_name} but each {model_name} only shows total usage of all model () curl http://localhost:6868/models/face [ { "modelName": "fac...

🚀 The feature Right now our docker build script does not support local branches, it makes a new copy of the repo and checkout to a branch. serve/docker/Dockerfile.dev Lines 65 to 67 in 130a29e && git...

🚀 The feature KServe now has rebranded the v2 inference protocol to open inference protocol specification, can we implement OIP natively in torchserve like other model servers such as Triton, MLServer, OpenVino, AMD Inference Serve ? Motivation, pitch Currently torchserve utilizes the KServe python...

We have a sentence-transformer model being packed into MAR file and saved in our storage. We are using Kserve on Kubeflow to deploy the models. we have followed these steps https://kserve.github.io/website/0.8/modelserving/v1beta1/torchserve/ and also referred to https://github.com/AshutoshDongare/H...

🐛 Describe the bug I am using python version 3.9. There is error coming that "Worker pid is not available yet." This is due to the model_service_worker.py in ts. The error is name: "Integer" is not defined. I have also attached the image for reference. Error logs 2023-05-29T13:36:37,550 [INFO ] W-9...

📚 The doc issue Recently, my team tried the TorchServe workflow, and it did the job quite well. However, I would like to know what TorchServe workflow' broker is used, or is it just a memory queue? Suggest a potential alternative/fix No response

🐛 Describe the bug Both CPU and GPU have been failing for 2 weeks https://github.com/pytorch/serve/actions/workflows/regression_tests_cpu.yml It's unsafe to merge anything until this gets fixed

ts_config file inference_address=http://0.0.0.0:8080 management_address=http://0.0.0.0:8082 metrics_address=http://0.0.0.0:8081 number_of_netty_threads=32 job_queue_size=1000 model_store=/home/model-server/model-store workflow_store=/home/model-server/wf-store enable_envvars_config=true metrics_mode...