storage

Overview ¶

Package storage provides an easy way to work with Google Cloud Storage. Google Cloud Storage stores data in named objects, which are grouped into buckets.

More information about Google Cloud Storage is available at https://cloud.google.com/storage/docs.

See https://pkg.go.dev/cloud.google.com/go for authentication, timeouts, connection pooling and similar aspects of this package.

Creating a Client ¶

To start working with this package, create a Client:

ctx := context.Background()
client, err := storage.NewClient(ctx)
if err != nil {
    // TODO: Handle error.
}

The client will use your default application credentials. Clients should be reused instead of created as needed. The methods of Client are safe for concurrent use by multiple goroutines.

You may configure the client by passing in options from the google.golang.org/api/option package. You may also use options defined in this package, such as WithJSONReads.

If you only wish to access public data, you can create an unauthenticated client with

client, err := storage.NewClient(ctx, option.WithoutAuthentication())

To use an emulator with this library, you can set the STORAGE_EMULATOR_HOST environment variable to the address at which your emulator is running. This will send requests to that address instead of to Cloud Storage. You can then create and use a client as usual:

// Set STORAGE_EMULATOR_HOST environment variable.
err := os.Setenv("STORAGE_EMULATOR_HOST", "localhost:9000")
if err != nil {
    // TODO: Handle error.
}

// Create client as usual.
client, err := storage.NewClient(ctx)
if err != nil {
    // TODO: Handle error.
}

// This request is now directed to http://localhost:9000/storage/v1/b
// instead of https://storage.googleapis.com/storage/v1/b
if err := client.Bucket("my-bucket").Create(ctx, projectID, nil); err != nil {
    // TODO: Handle error.
}

Please note that there is no official emulator for Cloud Storage.

Buckets ¶

A Google Cloud Storage bucket is a collection of objects. To work with a bucket, make a bucket handle:

bkt := client.Bucket(bucketName)

A handle is a reference to a bucket. You can have a handle even if the bucket doesn't exist yet. To create a bucket in Google Cloud Storage, call BucketHandle.Create:

if err := bkt.Create(ctx, projectID, nil); err != nil {
    // TODO: Handle error.
}

Note that although buckets are associated with projects, bucket names are global across all projects.

Each bucket has associated metadata, represented in this package by BucketAttrs. The third argument to BucketHandle.Create allows you to set the initial BucketAttrs of a bucket. To retrieve a bucket's attributes, use BucketHandle.Attrs:

attrs, err := bkt.Attrs(ctx)
if err != nil {
    // TODO: Handle error.
}
fmt.Printf("bucket %s, created at %s, is located in %s with storage class %s\n",
    attrs.Name, attrs.Created, attrs.Location, attrs.StorageClass)

Objects ¶

An object holds arbitrary data as a sequence of bytes, like a file. You refer to objects using a handle, just as with buckets, but unlike buckets you don't explicitly create an object. Instead, the first time you write to an object it will be created. You can use the standard Go io.Reader and io.Writer interfaces to read and write object data:

obj := bkt.Object("data")
// Write something to obj.
// w implements io.Writer.
w := obj.NewWriter(ctx)
// Write some text to obj. This will either create the object or overwrite whatever is there already.
if _, err := fmt.Fprintf(w, "This object contains text.\n"); err != nil {
    // TODO: Handle error.
}
// Close, just like writing a file.
if err := w.Close(); err != nil {
    // TODO: Handle error.
}

// Read it back.
r, err := obj.NewReader(ctx)
if err != nil {
    // TODO: Handle error.
}
defer r.Close()
if _, err := io.Copy(os.Stdout, r); err != nil {
    // TODO: Handle error.
}
// Prints "This object contains text."

Objects also have attributes, which you can fetch with ObjectHandle.Attrs:

objAttrs, err := obj.Attrs(ctx)
if err != nil {
    // TODO: Handle error.
}
fmt.Printf("object %s has size %d and can be read using %s\n",
    objAttrs.Name, objAttrs.Size, objAttrs.MediaLink)

Listing objects ¶

Listing objects in a bucket is done with the BucketHandle.Objects method:

query := &storage.Query{Prefix: ""}

var names []string
it := bkt.Objects(ctx, query)
for {
    attrs, err := it.Next()
    if err == iterator.Done {
        break
    }
    if err != nil {
        log.Fatal(err)
    }
    names = append(names, attrs.Name)
}

Objects are listed lexicographically by name. To filter objects lexicographically, Query.StartOffset and/or Query.EndOffset can be used:

query := &storage.Query{
    Prefix: "",
    StartOffset: "bar/",  // Only list objects lexicographically >= "bar/"
    EndOffset: "foo/",    // Only list objects lexicographically < "foo/"
}

// ... as before

If only a subset of object attributes is needed when listing, specifying this subset using Query.SetAttrSelection may speed up the listing process:

query := &storage.Query{Prefix: ""}
query.SetAttrSelection([]string{"Name"})

// ... as before

ACLs ¶

Both objects and buckets have ACLs (Access Control Lists). An ACL is a list of ACLRules, each of which specifies the role of a user, group or project. ACLs are suitable for fine-grained control, but you may prefer using IAM to control access at the project level (see Cloud Storage IAM docs).

To list the ACLs of a bucket or object, obtain an ACLHandle and call ACLHandle.List:

acls, err := obj.ACL().List(ctx)
if err != nil {
    // TODO: Handle error.
}
for _, rule := range acls {
    fmt.Printf("%s has role %s\n", rule.Entity, rule.Role)
}

You can also set and delete ACLs.

Conditions ¶

Every object has a generation and a metageneration. The generation changes whenever the content changes, and the metageneration changes whenever the metadata changes. Conditions let you check these values before an operation; the operation only executes if the conditions match. You can use conditions to prevent race conditions in read-modify-write operations.

For example, say you've read an object's metadata into objAttrs. Now you want to write to that object, but only if its contents haven't changed since you read it. Here is how to express that:

w = obj.If(storage.Conditions{GenerationMatch: objAttrs.Generation}).NewWriter(ctx)
// Proceed with writing as above.

Signed URLs ¶

You can obtain a URL that lets anyone read or write an object for a limited time. Signing a URL requires credentials authorized to sign a URL. To use the same authentication that was used when instantiating the Storage client, use BucketHandle.SignedURL.

url, err := client.Bucket(bucketName).SignedURL(objectName, opts)
if err != nil {
    // TODO: Handle error.
}
fmt.Println(url)

You can also sign a URL without creating a client. See the documentation of SignedURL for details.

url, err := storage.SignedURL(bucketName, "shared-object", opts)
if err != nil {
    // TODO: Handle error.
}
fmt.Println(url)

Post Policy V4 Signed Request ¶

A type of signed request that allows uploads through HTML forms directly to Cloud Storage with temporary permission. Conditions can be applied to restrict how the HTML form is used and exercised by a user.

For more information, please see the XML POST Object docs as well as the documentation of BucketHandle.GenerateSignedPostPolicyV4.

pv4, err := client.Bucket(bucketName).GenerateSignedPostPolicyV4(objectName, opts)
if err != nil {
    // TODO: Handle error.
}
fmt.Printf("URL: %s\nFields: %v\n", pv4.URL, pv4.Fields)

Credential requirements for signing ¶

If the GoogleAccessID and PrivateKey option fields are not provided, they will be automatically detected by BucketHandle.SignedURL and BucketHandle.GenerateSignedPostPolicyV4 if any of the following are true:

• you are authenticated to the Storage Client with a service account's downloaded private key, either directly in code or by setting the GOOGLE_APPLICATION_CREDENTIALS environment variable (see Other Environments),
• your application is running on Google Compute Engine (GCE), or
• you are logged into gcloud using application default credentials with impersonation enabled.

Detecting GoogleAccessID may not be possible if you are authenticated using a token source or using option.WithHTTPClient. In this case, you can provide a service account email for GoogleAccessID and the client will attempt to sign the URL or Post Policy using that service account.

To generate the signature, you must have:

• iam.serviceAccounts.signBlob permissions on the GoogleAccessID service account, and
• the IAM Service Account Credentials API enabled (unless authenticating with a downloaded private key).

Errors ¶

Errors returned by this client are often of the type github.com/googleapis/gax-go/v2/apierror. The [apierror.APIError] type can wrap a google.golang.org/grpc/status.Status if gRPC was used, or a google.golang.org/api/googleapi.Error if HTTP/REST was used. You might also encounter googleapi.Error directly from HTTP operations. These types of errors can be inspected for more information by using errors.As to access the specific underlying error types and retrieve detailed information, including HTTP or gRPC status codes. For example:

// APIErrors often wrap a googleapi.Error (for JSON and XML calls) or a status.Status (for gRPC calls)
var ae *apierror.APIError
if ok := errors.As(err, &ae); ok {
	// ae.HTTPCode() is the HTTP status code.
	// ae.GRPCStatus().Code() is the gRPC status code
	log.Printf("APIError: HTTPCode: %d, GRPCStatusCode: %s", ae.HTTPCode(), ae.GRPCStatus().Code())

	if ae.GRPCStatus().Code() == codes.Unavailable {
		// ... handle gRPC unavailable ...
	}
}

// This allows a user to get more information directly from googleapi.Errors (for JSON/XML calls)
var e *googleapi.Error
if ok := errors.As(err, &e); ok {
	// e.Code is the HTTP status code.
	// e.Message is the error message.
	// e.Body is the raw response body.
	// e.Header contains the HTTP response headers.
	log.Printf("HTTP Code: %d, Message: %s", e.Code, e.Message)

	if e.Code == 409 {
		// ... handle conflict ...
	}
}

This library may also return other errors that are not wrapped as [apierror.APIError]. For example, errors with authentication may return cloud.google.com/go/auth.Error.

Retrying failed requests ¶

Methods in this package may retry calls that fail with transient errors. Retrying continues indefinitely unless the controlling context is canceled, the client is closed, or a non-transient error is received. To stop retries from continuing, use context timeouts or cancellation.

The retry strategy in this library follows best practices for Cloud Storage. By default, operations are retried only if they are idempotent, and exponential backoff with jitter is employed. In addition, errors are only retried if they are defined as transient by the service. See the Cloud Storage retry docs for more information.

Users can configure non-default retry behavior for a single library call (using BucketHandle.Retryer and ObjectHandle.Retryer) or for all calls made by a client (using Client.SetRetry). For example:

o := client.Bucket(bucket).Object(object).Retryer(
	// Use WithBackoff to change the timing of the exponential backoff.
	storage.WithBackoff(gax.Backoff{
		Initial:    2 * time.Second,
	}),
	// Use WithPolicy to configure the idempotency policy. RetryAlways will
	// retry the operation even if it is non-idempotent.
	storage.WithPolicy(storage.RetryAlways),
)

// Use a context timeout to set an overall deadline on the call, including all
// potential retries.
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()

// Delete an object using the specified strategy and timeout.
if err := o.Delete(ctx); err != nil {
	// Handle err.
}

Sending Custom Headers ¶

You can add custom headers to any API call made by this package by using callctx.SetHeaders on the context which is passed to the method. For example, to add a custom audit logging header:

ctx := context.Background()
ctx = callctx.SetHeaders(ctx, "x-goog-custom-audit-<key>", "<value>")
// Use client as usual with the context and the additional headers will be sent.
client.Bucket("my-bucket").Attrs(ctx)

gRPC API ¶

This package includes support for the Cloud Storage gRPC API. This implementation uses gRPC rather than the default JSON & XML APIs to make requests to Cloud Storage. All methods on the Client support the gRPC API, with the exception of the Client.ServiceAccount, Notification, and HMACKey methods.

The Cloud Storage gRPC API is generally available.

To create a client which will use gRPC, use the alternate constructor:

ctx := context.Background()
client, err := storage.NewGRPCClient(ctx)
if err != nil {
	// TODO: Handle error.
}
// Use client as usual.

One major advantage of the gRPC API is that it can use Direct Connectivity, enabling requests to skip some proxy steps and reducing response latency. Requirements to use Direct Connectivity include:

• Your application must be running inside Google Cloud.
• Your Cloud Storage bucket location must overlap with your VM or compute environment zone. For example, if your VM is in us-east1-b, your bucket must be located in either us-east1 (single region), nam4 (dual region), or us (multi-region).
• Your client must use service account authentication.

Additional requirements for Direct Connectivity are documented in the Cloud Storage gRPC docs. If all requirements are met, the client will use Direct Connectivity by default without requiring any client options or environment variables. To disable Direct Connectivity, you can set the environment variable GOOGLE_CLOUD_DISABLE_DIRECT_PATH=true.

Dependencies for the gRPC API may slightly increase the size of binaries for applications depending on this package. If you are not using gRPC, you can use the build tag `disable_grpc_modules` to opt out of these dependencies and reduce the binary size.

The gRPC client is instrumented with Open Telemetry metrics which export to Cloud Monitoring by default. More information is available in the gRPC client-side metrics documentation, including information about roles which must be enabled in order to do the export successfully. To disable this export, you can use the WithDisabledClientMetrics client option.

To disable OpenTelemetry bucket metadata in traces, you can set the environment variable GO_OTEL_BUCKETMETADATA_DISABLED=true.

The client automatically computes and sends CRC32C checksums for uploads using Writer, providing an additional layer of data integrity validation with a slight CPU overhead.

Note: With a chunk size of 0 (no buffering) in JSON uploads, an auto-calculated checksum mismatch returns an error but may leave corrupt data on the server, requiring manual cleanup. This risk does not apply to single-shot uploads when user-provided checksum is provided.

Automatic checksumming can be disabled using Writer.DisableAutoChecksum.

Read checksumming ¶

By default, the client automatically computes and validates CRC32C checksums for reads when downloading an entire object, providing an additional layer of data integrity validation with a slight CPU overhead.

For gRPC clients, read checksumming is also performed for partial reads (range requests) by validating the checksum of each individual data chunk returned by the server.

Automatic read checksumming can be disabled using the WithDisableReaderChecksum option for a normal reader (both gRPC and JSON), or the WithDisableMRDReadChecksum option for the multi-range downloader (gRPC only).

Parallel Uploads ¶

The parallel upload feature splits a large object into multiple parts and uploads them in parallel. It is supported exclusively for gRPC clients. If used with a JSON client, the configuration is ignored and a standard upload is performed.

Parallel uploads can yield higher throughput when uploading large objects. However, there are several things which must be kept in mind when choosing to use this strategy:

• Performing parallel uploads may incur additional costs. Class A operations are performed to create each part. If a storage class other than STANDARD is used, early deletion fees apply to deletion of the parts.
• The service account/credentials used to perform the parallel upload require `storage.objects.delete` in order to clean up the temporary part objects.
• A failed upload can leave part objects behind which will count as storage usage, and you will be billed for it. Upon completion or failure of a parallel upload, the Writer makes a best-effort attempt to clean up any temporary parts created. However, if the program crashes there is no means for the client to perform the cleanup. Temporary parts have the prefix: "gcs-go-sdk-pu-tmp". It is recommended to set appropriate bucket lifecycle policies to reliably clean up any leftover objects to avoid unnecessary storage costs.
• Using parallel uploads is not a one-size-fits-all solution. They introduce overhead that is only offset when uploading sufficiently large objects. The optimal threshold depends on many factors; therefore, you should experiment with your specific workload to determine if parallel uploads provide a benefit.

Note: This feature is currently experimental and its API surface may change in future releases. It is not yet recommended for production use.

Storage Control API ¶

Certain control plane and long-running operations for Cloud Storage (including Folder and Managed Folder operations) are supported via the autogenerated Storage Control client, which is available as a subpackage in this module. See package docs at cloud.google.com/go/storage/control/apiv2 or reference the Storage Control API docs.