Skip to main content

Cloud Infrastructure & Kubernetes Orchestration

Designing a resilient, automated deployment pipeline and IaC-driven container orchestration strategy on Google Cloud Platform (GCP).

TL;DR

Architected a production-grade cloud infrastructure on GCP using Terraform (IaC). Provisioned a GKE cluster, managed Cloud SQL databases, and established a Zero-Trust Cloudflare network. Designed a highly secure, least-privilege CI/CD pipeline via CircleCI for automated, zero-downtime microservices delivery.

Complete Deployment

The Challenge & Impact​

  • The Challenge: Manually provisioning network VPCs, IAM permissions, database instances, and K8s clusters across a distributed microservice architecture inevitably leads to configuration drift and severe security vulnerabilities.
  • The Objective: To strictly enforce Infrastructure as Code (IaC) principles. The goal was to fully automate the cloud provisioning lifecycle and establish a highly secure, automated CI/CD pipeline with strict RBAC (Role-Based Access Control) for deployments.
  • The Impact: Achieved 100% reproducible cloud environments. Eliminated manual credential handling by utilizing Terraform to inject secrets dynamically. Enforced a Zero-Trust network perimeter, completely isolating the GKE cluster from public internet exposure.

Architecture & Execution​

Tech Stack​

GCP (GKE, Cloud SQL, IAM), Terraform, Kubernetes, Cloudflare Tunnels, CircleCI, GitLab Registry, NATS JetStream, Redis

TL;DR

This deployment architecture was inspired by Rich Young's blog post, Even Cheaper Private Kubernetes.

1. Infrastructure as Code (Terraform on GCP)​

Instead of manual console clicks, the entire foundational infrastructure is codified and version-controlled using Terraform:

  • Cloud Provisioning: Automated the creation of the GCP Network Infrastructure, GKE (Google Kubernetes Engine) cluster, and Cloud SQL (PostgreSQL) instances.
  • Identity & Access Management (IAM): Provisioned dedicated GCP Service Accounts and generated Service Account Keys securely via Terraform.
  • Secret Management: Utilized Terraform's random password generation and local files execution to dynamically create and inject Kubernetes Secrets (e.g., database credentials) directly into the GKE cluster without exposing them in the source code.

2. Container Orchestration & Middleware​

  • Secure Traffic Routing (Cloudflare Tunnels): Provisioned cloudflared as a K8s Deployment using ConfigMaps and Secrets. This Zero-Trust setup proxies external traffic directly to the API Gateway, ensuring the GKE cluster requires absolutely zero inbound open ports.
  • Stateful vs. Stateless Segregation: - Stateless: Engineered the Golang microservices as scalable K8s Deployments.
    • In-Cluster Middleware: Deployed and optimized Redis and NATS JetStream within GKE for high-performance internal caching and event streaming.
    • Managed State: Leveraged Cloud SQL Proxy to securely connect the GKE microservices to the managed PostgreSQL database, offloading heavy state management to GCP.

Advanced Mechanics 1: Secure CI/CD Pipeline (CircleCI)​

⚙️ View CI/CD & RBAC security design

To achieve automated application delivery from GitLab Registry to GKE, I designed a strict, least-privilege security model for the CircleCI pipeline to interact with the Kubernetes cluster:

  1. Service Account Creation: Created a dedicated K8s ServiceAccount exclusively for CircleCI deployments.
  2. ClusterRole Permission Definition: Defined strict RBAC policies (ClusterRole), granting CircleCI only the exact permissions needed to update deployments and trigger rollouts, preventing "God-mode" access.
  3. ClusterRoleBinding & Token Generation: Bound the role to the Service Account and generated a long-lived Token Secret.
  4. Kubeconfig Automation: Dynamically generated a tightly-scoped kubeconfig file for CircleCI.
  5. Deployment & Schema Migration: The pipeline automatically applies PostgreSQL schema migrations and triggers K8s rolling updates for the application, ensuring zero downtime.

👉 Deep Dive on the LZStock Tech Blog: Read more about our CI/CD & Security Design

Advanced Mechanics 2: Observability & Proactive Monitoring​

⚙️ View health checks & profiling setup

While centralized metric scraping is on the future roadmap, I engineered a highly robust, proactive monitoring foundation tailored specifically for Kubernetes orchestration:

  • Deep Health Checks & Liveness Probes: Developed an automated health check system that actively polls critical external dependencies (e.g., Cloud SQL, Redis, NATS). This ensures K8s Liveness and Readiness probes accurately reflect the true operational state of the microservice, rather than just verifying if the HTTP server is listening.
  • Isolated Management Port (Secondary Mux): Leveraged a secondary HTTP multiplexer dedicated solely to serving health status endpoints and Go runtime profiling (pprof). This architectural pattern completely isolates operational traffic from public-facing business APIs, preventing health check spam in access logs and securing debugging endpoints from external exposure.
  • Open-Source Extraction: To enforce consistency across all 6 microservices, I packaged this robust monitoring pattern into an independent, reusable open-source library, Github: service-guardian, demonstrating a commitment to modularity and developer experience (DX).

👉 Deep Dive on the LZStock Tech Blog: Read the architectural breakdown: Health Check & Liveness

🌟 Present-Day Retrospective​

The Evolution of State Management:

  • The Architectural Decision: In the initial design phases, I considered running PostgreSQL directly inside Kubernetes. However, by leveraging Terraform, I pivoted to GCP Cloud SQL. This strategic decision offloaded the immense operational burden of database backups, disaster recovery, and IOPS tuning to the cloud provider, drastically reducing operational risk.