GCP Storage Services
Storage Types
Object Storage
Object or blob are the collection of data stored as a single entity. Objects are stored in a flat environment with no sub-directories. In GCP, objects are grouped in buckets and each object can be individually addressed by a URL like https://storage.cloud.google.com/BUCKET_NAME/OBJECT_NAME. The size of an object is not limited by the attached storage. Object has their metadata and object can have versions. Google Cloud Storage is a cloud storage service that provides object storage.
File Storage
In file storage, all the data is stored in a single file. File storage uses hierarchical directory structure. File storage is suitable for storing operating system files. File storage can also be stored on a network-attached storage like Cloud Filestore.
Block Storage
In block storage, data is divided into fixed sized blocks of data. Each block is stored in a separate file with unique identifiers. In Linux, the common size if a block is 4KB and in relational databases, the size of a block is greater than or equal to 8KB. Generally in GCP, block storage is used in ephemeral storage (short-lived) and persistent storage (long-lived) attached to a VM. Persistent Disk and Local SSD are a cloud storage services that provides block storage.
Storage Products
Cloud Storage
- Cloud Storage is a GCP's storage service. It uses object storage to store data in containers called buckets.
- The buckets are associated with project. The buckets can be accessed by a URL like
https://storage.cloud.google.com/BUCKET_NAME. - The details of a object is stored in a metadata.
- The objects are immutable, that means when the object is updated, it gets replaced by the new version.
- When object versioning is enabled, old or deleted object can be retained as a noncurrent version.
- The objects in a buckets can have different permissions for different users and services such as
update,createordelete. - The Cloud Storage has multiple storage classes. Scroll down to read about storage classes.
Use cases
- Media content storage & delivery: image, video, audio, documents, etc.
- Integrated storage for analytics and ML.
- Data backup and archival.
Cloud Filestore
- Cloud Filestore uses file storage to store data in files. It is a network-attached storage.
- Cloud Filestore offers low-latency and high-availability storage.
- Cloud Filestore offers multiple service tiers.
- Cloud FIletore supports NFSv3 protocol which supports asynchronous file operations.
- This product is mainly used with Compute Engine VMs and Google Kubernetes Engine (GKE) clusters.
Service Tiers
- Basic HDD: The storage is stored on a hard disk. This is suitable for file sharing, software development, and use with GKE (Google Kubernetes Engine) workloads.
- Basic SSD: This ties similar to Basic HDD, but the storage is stored on a solid state drive. Basic SSD is more durable and has a higher performance at a higher cost.
- High Scale SSD: High Scale SSD is suitable for high performance computing (HPC) workloads. Genome sequencing, and financial analysis are examples of HPC workloads.
- Enterprise: Enterprise is a tier that provides a higher level of availability and data replication over multiple zones. This tier is designed for enterprise-grade Network File System (NFS) storage. Batch computing, media rendering and transcoding, advanced AI are examples of enterprise workloads.
Data Encryption
- Encryption at rest: Data is encrypted at rest. This means that the data is encrypted when it is stored in the cloud. This is the default encryption. The encryption key is managed by Google.
- Encryption in transit: The data is encrypted when it is transferred from the user to the storage in the cloud.
Use cases
- Media rendering.
- Web content management.
- Enterprise application migration.
Local SSD
- Local SSD uses block storage to store data in files. It is a local storage.
- Local SSDs are ephemeral storage (short-lived) that is attached to a VM for higher performance.
- Local SSD offers very high IOPS and low latency.
- Local SSDs are used to store temporary data such as caches or scratch processing space.
- The data stored in Local SSD is automatically encrypted at rest.
- Stored data gets deleted when the VM is terminated.
Persistent Disk
- Like Local SSD Persistent Disk uses block storage to store data in files.
- Persistent Disks are dedicated HDDs and SSDs that are used for enterprise and database applications.
- Persistent Disks are durable are high performance and can be used for long-lived storage.
- Persistent Disks can be attached to VM instances or Google Compute Engine (GCE) clusters.
- Persistent Disks are located independently of the VM instance.
- Multiple VM instances can use the same Persistent Disk.
Types of Persistent Disks (Block Storage)
- Standard Persistent Disks (HDD)
- Balanced Persistent Disks (SSD)
- Performance Persistent Disks (SSD)
- Extreme Persistent Disks (SSD)
- Local SSDs
Comparing Storage Products
| Field | Cloud Storage | Cloud Filestore | Local SSD | Persistent Disk |
|---|---|---|---|---|
| Scalability | Yes | Yes | No | Yes |
| Sharing | Read/write from anywhere | Mountable on VMs, remote clients, and GKE clusters | Not shareable | Limited sharing |
| Encryption keys | Google-managed, customer-managed, and customer supplied | Google-managed and customer-managed | Google-managed | Google-managed, customer-managed, and customer supplied |
| Persistent | Lifetime of the bucket | Lifetime of the instance | Ephemeral | Lifetime of the disk |
| Management | Fully managed | Fully Managed | Manually format, strip, and mount | Manually format and mount |
Storage Classes
There are four types of storage classes in Cloud Storage. Each classes servers different purpose. The class-type is stored in the metadata of a object. Unless specified explicitly, the newly created object inherit the storage class from the bucket. The default storage class of a bucket is Standard storage.
| Storage Class | Available SLA | Min. Storage Duration | Use Cases |
|---|---|---|---|
| Standard | >99.99% (MR) 99.99% (R) | None | Content Storage |
| Nearline | 99.95% (MR) 99.9% (R) | 30 days | Backups |
| Coldline | 99.95% (MR) 99.9% (R) | 90 days | Disaster Recovery |
| Archive | 99.95% (MR) 99.9% (R) | 365 days | Archiving |
*MR -> in Multi-regional, *R -> in Regional
Additional Classes
- Multi-Regional storage
- Objects can be stored in multi-regions or dual regions.
- Regional storage
- Objects are only stored in a chosen region.
- Durable Reduced Availability (DRA) storage
- In terms of availability, DRA has low availability SLA (99%).
- DRA has higher pricing for operation.
- The durability of DRA is higher than other classes.
References
- "Cloud Storage Options | Google Cloud" Cloud, https://cloud.google.com/products/storage.
- "Google Cloud Developer Cheat Sheet" Googlecloudcheatsheet, https://googlecloudcheatsheet.withgoogle.com/.
- "Object vs. File vs. Block Storage: What’s the Difference? | IBM" Ibm, 14 Oct. 2021, https://www.ibm.com/cloud/blog/object-vs-file-vs-block-storage.