Get an estimate of monthly storage costs

Advanced website indexing incurs monthly data storage charges based on the size of the web data that you import into your data store. To get an estimate of the size of your web data before importing it, you can call the estimateDataSize method and specify the web pages that you want to import. The estimateDataSize method is a long-running operation that runs until the process for estimating the data size is complete. This can take from a few minutes to over an hour, depending on the number of web pages that you specify. After you have an estimate of the size of your web data, you can get an estimate of your monthly data storage costs using the Agent Search pricing page.

Before you begin

Determine the URL patterns for the websites that you intend to include (and optionally exclude) when you import web data into your data store. You specify these URL patterns when you call the estimateDataSize method.

Procedure

To get an estimate of the size of your web data, follow these steps:

1. Call the estimateDataSize method.

   curl -X POST \
   -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
   -H "Content-Type: application/json" \
   "https://discoveryengine.googleapis.com/v1alpha/projects/PROJECT_ID/locations/global:estimateDataSize" \
   -d '{
     "website_data_source": {
       "estimator_uri_patterns": [
         {
           "provided_uri_pattern": "URI_PATTERN_TO_INCLUDE",
           "exact_match": EXACT_MATCH_BOOLEAN_1
         },
         {
           "provided_uri_pattern": "URI_PATTERN_TO_EXCLUDE",
           "exact_match": EXACT_MATCH_BOOLEAN_2,
           "exclusive": EXCLUSIVE_BOOLEAN
         }
       ]
     }
   }'
   

   Replace the following:

   • PROJECT_ID: the ID of your project.

   • URI_PATTERN_TO_INCLUDE: the URL patterns for the websites that you want to include in your data size estimate.

   • URI_PATTERN_TO_EXCLUDE: (Optional) The URL patterns for the websites that you want to exclude from your data size estimate.

     For URI_PATTERN_TO_INCLUDE and URI_PATTERN_TO_EXCLUDE, you can use patterns similar to the following:

     • Entire website: www.mysite.com
     • Parts of a website: www.mysite.com/faq
     • Entire domain: mysite.com or *.mysite.com

   • EXCLUSIVE_BOOLEAN: (Optional) If true, then the provided URI pattern represents web pages that are excluded from your data size estimate. The default is false, which means that the provided URI pattern represents web pages that are included in your data size estimate.