Dec
27
2023

Choosing Google Cloud Storage and Data Solutions

gcp-cloud-storage-2
  1. Choosing Google Cloud Storage and Data Solutions
  2. Choosing a Google Cloud Deployment Platform
  3. Designing Google Cloud Networks
  4. Designing Reliable Systems
  5. GCP Compute Engine Instance Lifecycle
  6. Disaster Planning and Recovery Strategies
  7. Designing Secured Systems in GCP

Introduction

The Google Cloud storage and database portfolio covers relational, NoSQL, object, data warehouse, and in-memory stores, as shown in this table.

gcp-cloud-storage-1

  • Relational (Fixed Schema)
    • Cloud SQL is a fixed schema datastore with a storage limit of 30 terabytes. It is offered using MySQL, PostgreSQL and SQL Server. These services are good for web applications such as CMS or eCommerce.
    • Cloud Spanner is also relational and fixed schema, but it scales infinitely and can be regional or multi-regional. Example use cases include scalable relational databases greater than 30 gigabytes with high availability and also global accessibility like supply chain management and manufacturing.
  • NoSQL (Schemaless)
    • Firestore is a completely managed document datastore with a maximum document size of 1 megabyte. It is useful for hierarchal data, for example, a game state or user profiles.
    • Cloud Bigtable is also a NoSQL datastore that scales infinitely. It is good for heavy read/write events and use cases including financial services, Internet of Things, and digital output streams.
  • Object
    • Cloud Storage is schemaless and is completely managed with infinite scale. It stores binary object data and so it’s good for storing images, media serving, and backups.
  • Warehouse
    • BigQuery uses a fixed schema and supports completely managed SQL analysis of the data stored. It is excellent for performing analytics and business intelligence at dashboards.
  • In Memory
    • Memorystore provides a schemaless managed Redis database. It is excellent for caching for web and mobile applications and for providing fast access to state in microservice architectures.

Simple Decision Chart

Following is simple decision chart to decide on your cloud storage needs depending on your use cases.

gcp-cloud-storage-2

OLTP or OLAP needs

gcp-database-structured

Time Chart for Transferring Data to Google Cloud

gcp-cloud-storage-3

Data Transfer Options to Google Cloud

  • Smaller or Scheduled Data Transfer
    • Use Storage Transfer Service
      • Transfer Data from
        • Amazon S3
        • HTTP/HTTPS location
        • Transfer data between Cloud Storage buckets
      • Scheduled jobs
        • One time or recurring, import at a scheduled time of day
        • Options for delete objects not in source or after transfer
        • Filter on file name, creation date
    • Use Storage Transfer Service for On-Premises Data (size > 1 TB)
      • How
        • Install on-premises agent on your servers
        • Agent runs in a Docker container
        • Setup a connection to Google Cloud
        • Requires a minimum of 300 Mbps bandwidth
      • Benefits
        • Scales to billions of files and 100s of TBs
        • Secure
        • Automatic retires
        • Logged
        • Easy to monitor via Cloud Console
  • Large Data (or take too long to upload)
    • Use Transfer Appliance
      • Process
        gcp-cloud-storage-4
    • BigQuery Data Transfer Service (SaaS applications to BigQuery)
      • From
        • Teradata
        • Amazon Redshift
        • Amazon S3

There are other aspects you need to consider while choosing a storage solution for your needs.

Other Aspects

Google Cloud Storage Availability SLAs

The availability SLAs are typically defined per month. Monthly uptime percentage means the total number of minutes in a month minus the number of minutes of downtime suffered from all downtime periods in a month, divided by the total number of minutes in a month.

gcp-cloud-storage-5

Durability of Data in Google Cloud

The durability of your data in Google Cloud Storage is a shared responsibility. (In any cloud storage it’s shared responsibility).¬†Google Cloud’s responsibility is to ensure that data is durable in the event of a hardware failure. Your responsibility is performing backups of your data.¬†Following is quick reference for GCP storage durability.

gcp-cloud-storage-6

Scaling of Google Cloud Storage Services

gcp-cloud-storage-7

Consistency of Google Cloud Storage Services

gcp-cloud-storage-8