Storage Targets
Google Cloud Storage
Use Google Cloud Storage as storage backend for Parseable. Follow the setup steps and practical guidance for using Google Cloud Storage in your Parseable…
Configure Google Cloud Storage (GCS) as the storage backend for Parseable.
Overview
Using GCS with Parseable provides:
- Scalable Storage - Virtually unlimited capacity
- Cost Effective - Multiple storage classes
- Durability - 99.999999999% durability
- GCP Integration - Native Google Cloud support
Prerequisites
- Google Cloud project
- GCS bucket created
- Service account with Storage permissions
- Parseable instance
Parseable Configuration
Environment Variables
# GCS Configuration
P_GCS_URL=https://storage.googleapis.com
P_GCS_BUCKET=parseable-data
P_GCS_REGION=us-central1
# Service account key (if not using workload identity)
GOOGLE_APPLICATION_CREDENTIALS=/path/to/key.jsonDocker Compose
version: '3.8'
services:
parseable:
image: quay.io/parseablehq/parseable:latest
ports:
- "8000:8000"
environment:
- P_GCS_URL=https://storage.googleapis.com
- P_GCS_BUCKET=parseable-data
- P_GCS_REGION=us-central1
- P_USERNAME=admin
- P_PASSWORD=admin
volumes:
- ./service-account.json:/etc/gcp/key.json
command: ["parseable", "gcs-store"]Kubernetes with Workload Identity
apiVersion: v1
kind: ServiceAccount
metadata:
name: parseable
annotations:
iam.gke.io/gcp-service-account: parseable@project.iam.gserviceaccount.com
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: parseable
spec:
template:
spec:
serviceAccountName: parseable
containers:
- name: parseable
image: quay.io/parseablehq/parseable:latest
env:
- name: P_GCS_URL
value: "https://storage.googleapis.com"
- name: P_GCS_BUCKET
value: "parseable-data"
args: ["parseable", "gcs-store"]GCS Bucket Setup
Create Bucket
# Create bucket
gsutil mb -l us-central1 gs://parseable-data
# Set lifecycle policy
gsutil lifecycle set lifecycle.json gs://parseable-dataLifecycle Policy
{
"rule": [
{
"action": {"type": "SetStorageClass", "storageClass": "NEARLINE"},
"condition": {"age": 30}
},
{
"action": {"type": "SetStorageClass", "storageClass": "COLDLINE"},
"condition": {"age": 90}
},
{
"action": {"type": "SetStorageClass", "storageClass": "ARCHIVE"},
"condition": {"age": 365}
}
]
}IAM Permissions
# Create service account
gcloud iam service-accounts create parseable \
--display-name="Parseable Storage"
# Grant Storage Object Admin
gsutil iam ch \
serviceAccount:parseable@project.iam.gserviceaccount.com:objectAdmin \
gs://parseable-dataConfiguration Options
| Parameter | Description |
|---|---|
P_GCS_URL | GCS endpoint URL |
P_GCS_BUCKET | Bucket name |
P_GCS_REGION | Bucket region |
Storage Classes
| Class | Use Case | Retrieval Cost |
|---|---|---|
| Standard | Frequently accessed | None |
| Nearline | Monthly access | Low |
| Coldline | Quarterly access | Medium |
| Archive | Yearly access | High |
Best Practices
- Use Workload Identity - Avoid service account keys
- Configure Lifecycle - Optimize storage costs
- Enable Versioning - Protect against deletion
- Use Uniform Access - Simplify permissions
- Set Retention Policies - Compliance requirements
Troubleshooting
Access Denied
- Verify service account permissions
- Check bucket IAM policy
- Verify workload identity binding
- Check project permissions
Performance Issues
- Use regional buckets for low latency
- Enable parallel uploads
- Check network configuration
Next Steps
- Configure GCP Pub/Sub for streaming
- Set up alerts for storage metrics
- Create dashboards for monitoring
Was this page helpful?