Best practices: Cloud Run jobs with GPUs

This page provides best practices for optimizing performance when using a Cloud Run job with GPU for AI workloads such as, training large language models (LLMs) using your preferred frameworks, fine-tuning, and performing batch or offline inference on LLMs. To create a Cloud Run job that can perform compute intensive tasks or batch processing in real time, you should:
  • Use models that load fast and require minimal transformation into GPU-ready structures, and optimize how they are loaded.
  • Use configurations that allow for maximum, efficient, concurrent execution to reduce the number of GPUs needed to serve a target request per second while keeping costs down.

Recommended ways to load large ML models on Cloud Run

Google recommends downloading ML models from Cloud Storage and accessing them through the Google Cloud CLI. You might alternatively store models inside container images, but this method is best suited for smaller models less than 10 GB.

Storing and loading ML models trade-offs

Here is a comparison of the options:

Model location Deploy time Development experience Container startup time Storage cost
Cloud Storage, downloaded concurrently using the Google Cloud CLI command gcloud storage cp or the Cloud Storage API as shown in the transfer manager concurrent download code sample. Fastest. Model downloaded during container startup. Ensure the Cloud Run instance has sufficient RAM allocated to store the model files. Slightly more difficult to set up, because you'll need to either install the Google Cloud CLI on the image or update your code to use the Cloud Storage API. For more information on how to fetch credentials from the metadata server, see Introduction to service identity. Fast when you use network optimizations. The Google Cloud CLI downloads the model file in parallel. One copy in Cloud Storage.