Can't run on SageMaker with TGI
#4
by dpr882 - opened
While the README mentions that it can be used with TGI container in reality the notebook uses a custom ECR image, prob based on TGI.
But
- there's no visibility on that container image or where it's hosted
- couldn't get it to run on sagemaker with custom container: permissions errorsa and really don't see where this image is hosted so which account id to be used.
not great that an amazon model can't be ran on AWS Sagemaker simply with TGI
Hi @dpr882 Thanks for your interest!
To deploy on Sagemaker, pls refer to this notebook. https://github.com/awslabs/extending-the-context-length-of-open-source-llms/blob/main/falconlite2-tgi1.1.0/sm_deploy.ipynb Thank you!