1|||[TOOL-INSTALL] Install gcloud CLI on Ubuntu|||phase-0,tool-install,infrastructure,gcp|||## Objective
Install the Google Cloud SDK (gcloud CLI) on Ubuntu to manage GCP resources from the command line.

## Prerequisites
- Ubuntu system with sudo privileges
- Internet connection
- Basic terminal knowledge

## Detailed Steps

### Method 1: Using apt-get (Recommended)

1. **Add the Cloud SDK distribution URI as a package source**
```bash
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
```

2. **Import the Google Cloud public key**
```bash
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
```

If you get a warning about apt-key being deprecated, use this alternative:
```bash
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg
```

3. **Update and install the Cloud SDK**
```bash
sudo apt-get update
sudo apt-get install google-cloud-cli
```

4. **Install additional components (optional but recommended)**
```bash
sudo apt-get install google-cloud-cli-gke-gcloud-auth-plugin
sudo apt-get install kubectl
```

### Method 2: Using the install script (Alternative)

```bash
curl https://sdk.cloud.google.com | bash
exec -l $SHELL
```

## Acceptance Criteria
- [ ] gcloud CLI is installed and accessible from terminal
- [ ] Version is 400.0.0 or higher
- [ ] All required components are available
- [ ] CLI responds to basic commands

## Testing/Verification

```bash
# Check gcloud is installed
which gcloud
# Expected: /usr/bin/gcloud or /home/username/google-cloud-sdk/bin/gcloud

# Check version
gcloud version
# Expected: Google Cloud SDK 400.0.0 or higher

# List available components
gcloud components list

# Check help command works
gcloud --help
```

## Common Issues and Troubleshooting

### Issue 1: Command not found
**Solution**: Add gcloud to PATH
```bash
echo 'export PATH=$PATH:/usr/bin' >> ~/.bashrc
source ~/.bashrc
```

### Issue 2: Permission denied during installation
**Solution**: Ensure you're using sudo for system-wide installation

### Issue 3: GPG key import fails
**Solution**: Try the alternative gpg --dearmor method shown above

## Documentation
- [ ] Update `docs/setup/01-tools-installation.md`
- [ ] Document the installation method used
- [ ] Add troubleshooting notes encountered

## Estimated Time
0.5-1 hours

## Dependencies
- Depends on: None (First ticket)
- Blocks: #3 (Configure gcloud CLI), #5 (Create GCP project)

## Labels
`phase-0`, `tool-install`, `infrastructure`, `gcp`

## Resources
- [Official gcloud CLI Installation Guide](https://cloud.google.com/sdk/docs/install#deb)
- [gcloud CLI Quickstart](https://cloud.google.com/sdk/docs/quickstart)
- [Component Manager](https://cloud.google.com/sdk/docs/components)
2|||[TOOL-INSTALL] Install Terraform on Ubuntu|||phase-0,tool-install,infrastructure,terraform|||## Objective
Install Terraform CLI on Ubuntu to manage infrastructure as code for GCP resources.

## Prerequisites
- Ubuntu system with sudo privileges
- Internet connection
- wget or curl installed

## Detailed Steps

### Method 1: Using Official HashiCorp Repository (Recommended)

1. **Install required packages**
```bash
sudo apt-get update
sudo apt-get install -y gnupg software-properties-common
```

2. **Add HashiCorp GPG key**
```bash
wget -O- https://apt.releases.hashicorp.com/gpg | \
gpg --dearmor | \
sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg
```

3. **Verify the key fingerprint**
```bash
gpg --no-default-keyring \
--keyring /usr/share/keyrings/hashicorp-archive-keyring.gpg \
--fingerprint
# Expected: E8A0 32E0 94D8 EB4E A189  D270 DA41 8C88 A321 9F7B
```

4. **Add the official HashiCorp repository**
```bash
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] \
https://apt.releases.hashicorp.com $(lsb_release -cs) main" | \
sudo tee /etc/apt/sources.list.d/hashicorp.list
```

5. **Update and install Terraform**
```bash
sudo apt-get update
sudo apt-get install terraform
```

### Method 2: Direct Binary Installation (Alternative)

1. **Download Terraform**
```bash
TERRAFORM_VERSION="1.7.0"
wget https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip
```

2. **Install unzip if needed**
```bash
sudo apt-get install unzip
```

3. **Extract and install**
```bash
unzip terraform_${TERRAFORM_VERSION}_linux_amd64.zip
sudo mv terraform /usr/local/bin/
rm terraform_${TERRAFORM_VERSION}_linux_amd64.zip
```

4. **Set executable permissions**
```bash
sudo chmod +x /usr/local/bin/terraform
```

## Acceptance Criteria
- [ ] Terraform is installed and accessible from terminal
- [ ] Version is 1.6.0 or higher
- [ ] Terraform can initialize an empty directory
- [ ] Auto-completion is configured (optional)

## Testing/Verification

```bash
# Check terraform is installed
which terraform
# Expected: /usr/bin/terraform or /usr/local/bin/terraform

# Check version
terraform version
# Expected: Terraform v1.6.0 or higher

# Test basic functionality
mkdir -p /tmp/terraform-test
cd /tmp/terraform-test
terraform init
# Expected: Terraform initialized successfully

# Clean up test
cd ~
rm -rf /tmp/terraform-test
```

## Optional: Enable Auto-Completion

```bash
# For bash
terraform -install-autocomplete

# Reload shell
source ~/.bashrc
```

## Common Issues and Troubleshooting

### Issue 1: Command not found
**Solution**: Verify PATH includes terraform location
```bash
echo $PATH | grep -o '/usr/bin\|/usr/local/bin'
```

### Issue 2: Permission denied
**Solution**: Check executable permissions
```bash
ls -l $(which terraform)
sudo chmod +x $(which terraform)
```

### Issue 3: Old version installed
**Solution**: Remove old version first
```bash
sudo apt-get remove terraform
# Then reinstall using Method 1
```

### Issue 4: GPG key verification fails
**Solution**: Manually download and verify the key
```bash
wget -O- https://apt.releases.hashicorp.com/gpg | gpg --import
```

## Documentation
- [ ] Update `docs/setup/01-tools-installation.md`
- [ ] Document the installation method used
- [ ] Note the Terraform version installed

## Estimated Time
0.5-1 hours

## Dependencies
- Depends on: None
- Blocks: #4 (Configure Terraform), #8 (Set up GCS bucket for Terraform state)

## Labels
`phase-0`, `tool-install`, `infrastructure`, `terraform`

## Resources
- [Official Terraform Installation Guide](https://developer.hashicorp.com/terraform/install)
- [Terraform Ubuntu/Debian Installation](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli)
- [Terraform Version Manager (tfenv)](https://github.com/tfutils/tfenv) - For managing multiple versions
3|||[TOOL-INSTALL] Configure gcloud CLI with GCP authentication|||phase-0,tool-install,infrastructure,gcp,authentication|||## Objective
Authenticate gcloud CLI with Google Cloud Platform and configure default settings for the Archie Platform V3 project.

## Prerequisites
- #1 (Install gcloud CLI) must be completed
- Google account with GCP access
- Browser for OAuth authentication

## Detailed Steps

### Step 1: Initialize gcloud CLI

```bash
# Start the initialization process
gcloud init

# Follow the prompts:
# 1. Choose to log in with a Google account
# 2. Select or create a configuration
# 3. Authenticate via browser
```

### Step 2: Authenticate with User Account

```bash
# Alternative: Direct authentication
gcloud auth login

# This will:
# 1. Open your default browser
# 2. Prompt you to select a Google account
# 3. Request permission to access GCP
# 4. Save credentials locally
```

### Step 3: Set Default Configuration

```bash
# Set default project (will be created in ticket #5)
# For now, skip this or use existing project
gcloud config set project archie-platform-v3

# Set default region for Australia
gcloud config set compute/region australia-southeast1

# Set default zone
gcloud config set compute/zone australia-southeast1-a

# Disable usage reporting (optional)
gcloud config set disable_usage_reporting true
```

### Step 4: Configure Application Default Credentials

```bash
# This is required for Terraform to authenticate
gcloud auth application-default login

# Credentials will be saved to:
# ~/.config/gcloud/application_default_credentials.json
```

### Step 5: Create Named Configuration (Recommended)

```bash
# Create a dedicated configuration for Archie Platform
gcloud config configurations create archie-platform-v3

# Activate the configuration
gcloud config configurations activate archie-platform-v3

# Set properties for this configuration
gcloud config set project archie-platform-v3
gcloud config set compute/region australia-southeast1
gcloud config set compute/zone australia-southeast1-a
```

## Acceptance Criteria
- [ ] gcloud CLI is authenticated with user credentials
- [ ] Application default credentials are configured
- [ ] Default region is set to australia-southeast1
- [ ] Configuration can be listed and verified
- [ ] User has necessary permissions

## Testing/Verification

```bash
# Verify authentication
gcloud auth list
# Expected: Shows your authenticated account with an asterisk (*)

# Check active configuration
gcloud config configurations list
# Expected: Shows archie-platform-v3 with IS_ACTIVE = True

# View all configuration properties
gcloud config list
# Expected output should include:
# [compute]
# region = australia-southeast1
# zone = australia-southeast1-a
# [core]
# account = your-email@domain.com
# project = archie-platform-v3

# Test API access (this will fail if project doesn't exist yet, that's OK)
gcloud projects describe archie-platform-v3 2>/dev/null || echo "Project not yet created (expected)"

# Verify application default credentials exist
test -f ~/.config/gcloud/application_default_credentials.json && echo "ADC configured" || echo "ADC missing"

# Check available regions
gcloud compute regions list --filter="name:australia"
# Expected: Shows australia-southeast1 and australia-southeast2
```

## Understanding Credential Types

### User Credentials (`gcloud auth login`)
- Used by gcloud CLI commands
- Stored in: `~/.config/gcloud/credentials.db`
- Required for: Running gcloud commands

### Application Default Credentials (`gcloud auth application-default login`)
- Used by Terraform and GCP client libraries
- Stored in: `~/.config/gcloud/application_default_credentials.json`
- Required for: Terraform, Python/Node SDKs

### Service Account Credentials (created in ticket #7)
- Used for automation and CI/CD
- Stored as: JSON key file
- Required for: GitHub Actions, automated deployments

## Common Issues and Troubleshooting

### Issue 1: Browser doesn't open for authentication
**Solution**: Copy the URL manually
```bash
gcloud auth login --no-launch-browser
# Copy the URL and open in browser manually
```

### Issue 2: Permission denied errors
**Solution**: Verify you have the correct GCP permissions
```bash
gcloud projects get-iam-policy PROJECT_ID --flatten="bindings[].members" --filter="bindings.members:user:YOUR_EMAIL"
```

### Issue 3: Application default credentials not working
**Solution**: Re-run the ADC login
```bash
# Remove old credentials
rm ~/.config/gcloud/application_default_credentials.json
# Re-authenticate
gcloud auth application-default login
```

### Issue 4: Wrong project or region set
**Solution**: Reset configuration
```bash
gcloud config unset project
gcloud config unset compute/region
# Then set correct values
```

## Security Best Practices

1. **Never commit credentials to git**
```bash
# Verify .gitignore includes:
grep -E '(credentials|.gcloud)' .gitignore
```

2. **Use separate configurations for different projects**
```bash
gcloud config configurations list
```

3. **Regularly rotate credentials**
```bash
# Re-authenticate every 90 days
gcloud auth login --force
```

## Documentation
- [ ] Update `docs/setup/02-gcp-configuration.md`
- [ ] Document the configuration name used
- [ ] Save default region/zone settings
- [ ] Add credential file locations to .gitignore

## Estimated Time
0.5-1 hours

## Dependencies
- Depends on: #1 (Install gcloud CLI)
- Blocks: #5 (Create GCP project), #6 (Enable required APIs)

## Labels
`phase-0`, `tool-install`, `infrastructure`, `gcp`, `authentication`

## Resources
- [gcloud CLI Configuration](https://cloud.google.com/sdk/docs/configurations)
- [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials)
- [gcloud auth Commands](https://cloud.google.com/sdk/gcloud/reference/auth)
- [Managing Configurations](https://cloud.google.com/sdk/docs/configurations#managing_configurations)
4|||[TOOL-INSTALL] Configure Terraform for GCP backend|||phase-0,tool-install,infrastructure,terraform|||## Objective
Configure Terraform to work with Google Cloud Platform, set up provider authentication, and prepare for remote state management.

## Prerequisites
- #2 (Install Terraform) must be completed
- #3 (Configure gcloud CLI) must be completed
- Application default credentials configured

## Detailed Steps

### Step 1: Create Terraform Project Structure

```bash
# Navigate to project root
cd /mnt/data-disk1/archie-platform-v3

# Create Terraform directory structure
mkdir -p terraform/{modules,environments/{dev,staging,prod}}

# Create subdirectories for modules (will be used in later tickets)
mkdir -p terraform/modules/{project,vpc,iam,secrets,cloudsql,redis,nats,cloudbuild,cloudrun,monitoring}
```

### Step 2: Create Base Provider Configuration

```bash
# Create provider configuration file
cat > terraform/provider.tf <<'EOF'
# Google Cloud Provider Configuration
# This file defines how Terraform communicates with GCP

terraform {
  required_version = ">= 1.6.0"
  
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 5.0"
    }
    google-beta = {
      source  = "hashicorp/google-beta"
      version = "~> 5.0"
    }
  }
  
  # Backend configuration will be added in ticket #8
  # For now, state is stored locally
}

# Primary provider for stable GCP resources
provider "google" {
  project = var.project_id
  region  = var.region
  zone    = var.zone
}

# Beta provider for preview features
provider "google-beta" {
  project = var.project_id
  region  = var.region
  zone    = var.zone
}
EOF
```

### Step 3: Create Variables Configuration

```bash
# Create variables file
cat > terraform/variables.tf <<'EOF'
# Global Terraform Variables
# These variables are used across all modules

variable "project_id" {
  description = "The GCP project ID"
  type        = string
  default     = "archie-platform-v3"
}

variable "region" {
  description = "The GCP region for resources"
  type        = string
  default     = "australia-southeast1"
}

variable "zone" {
  description = "The GCP zone for resources"
  type        = string
  default     = "australia-southeast1-a"
}

variable "environment" {
  description = "Environment name (dev, staging, prod)"
  type        = string
  default     = "dev"
}

variable "labels" {
  description = "Common labels to apply to all resources"
  type        = map(string)
  default = {
    project     = "archie-platform-v3"
    managed_by  = "terraform"
    environment = "dev"
  }
}
EOF
```

### Step 4: Create Development Environment Configuration

```bash
# Create dev environment configuration
cat > terraform/environments/dev/main.tf <<'EOF'
# Development Environment Configuration
# This file orchestrates all modules for the dev environment

terraform {
  # Backend will be configured in ticket #8
  required_version = ">= 1.6.0"
}

# Import root provider configuration
provider "google" {
  project = var.project_id
  region  = var.region
}

provider "google-beta" {
  project = var.project_id
  region  = var.region
}

# Local variables for environment-specific settings
locals {
  environment = "dev"
  region      = "australia-southeast1"
  zone        = "australia-southeast1-a"
  
  common_labels = {
    environment = local.environment
    managed_by  = "terraform"
    project     = "archie-platform-v3"
  }
}

# Outputs for dev environment
output "environment" {
  value = local.environment
}

output "region" {
  value = local.region
}
EOF

# Create dev environment variables
cat > terraform/environments/dev/variables.tf <<'EOF'
variable "project_id" {
  description = "GCP Project ID"
  type        = string
  default     = "archie-platform-v3-dev"
}

variable "region" {
  description = "GCP Region"
  type        = string
  default     = "australia-southeast1"
}
EOF
```

### Step 5: Create .gitignore for Terraform

```bash
# Add Terraform-specific gitignore entries
cat >> .gitignore <<'EOF'

# Terraform files
**/.terraform/*
*.tfstate
*.tfstate.*
*.tfvars
*.tfvars.json
crash.log
crash.*.log
override.tf
override.tf.json
*_override.tf
*_override.tf.json
.terraformrc
terraform.rc

# Terraform plan files
*.tfplan

# Terraform lock file (uncomment if you want to ignore it)
# .terraform.lock.hcl
EOF
```

### Step 6: Initialize Terraform

```bash
# Navigate to terraform directory
cd /mnt/data-disk1/archie-platform-v3/terraform

# Initialize Terraform
terraform init

# This will:
# 1. Download the Google provider plugins
# 2. Create .terraform directory
# 3. Create .terraform.lock.hcl file
```

### Step 7: Verify Authentication

```bash
# Test Terraform can authenticate to GCP
terraform plan

# Expected: Should run without authentication errors
# (May show no resources to create - that's fine)
```

## Acceptance Criteria
- [ ] Terraform directory structure is created
- [ ] Provider configuration files are in place
- [ ] Variables are defined with correct defaults
- [ ] Terraform initializes without errors
- [ ] Google provider plugins are downloaded
- [ ] Authentication with GCP works
- [ ] .gitignore excludes sensitive Terraform files

## Testing/Verification

```bash
# Verify directory structure
cd /mnt/data-disk1/archie-platform-v3
tree -L 3 terraform/
# Expected: Shows modules and environments directories

# Verify Terraform files exist
ls -la terraform/*.tf
# Expected: Shows provider.tf and variables.tf

# Check Terraform version
cd terraform
terraform version
# Expected: Terraform v1.6.0+ with google provider

# Validate configuration
terraform validate
# Expected: Success! The configuration is valid.

# Format check
terraform fmt -check -recursive
# Expected: No output if formatting is correct

# Show providers
terraform providers
# Expected: Shows google and google-beta providers

# Test authentication
terraform plan
# Expected: Runs without authentication errors

# Verify .gitignore
grep -E '(tfstate|tfvars|.terraform)' ../.gitignore
# Expected: Shows Terraform entries
```

## Understanding Terraform Structure

### Provider Configuration (`provider.tf`)
- Defines which cloud provider to use
- Specifies provider version constraints
- Configures authentication method

### Variables (`variables.tf`)
- Declares input variables
- Sets default values
- Provides descriptions for documentation

### Environment Directories
```
terraform/
├── provider.tf           # Shared provider config
├── variables.tf          # Shared variables
└── environments/
    ├── dev/              # Development environment
    ├── staging/          # Staging environment
    └── prod/             # Production environment
```

### Module Directory Structure (created in later tickets)
```
terraform/modules/
├── project/              # GCP project configuration
├── vpc/                  # Networking
├── iam/                  # Identity and access management
├── secrets/              # Secret Manager
├── cloudsql/             # PostgreSQL database
├── redis/                # Redis cache
├── nats/                 # NATS messaging
├── cloudbuild/           # CI/CD
├── cloudrun/             # Container hosting
└── monitoring/           # Logging and monitoring
```

## Common Issues and Troubleshooting

### Issue 1: Provider download fails
**Solution**: Check internet connection and try mirror
```bash
terraform init -upgrade
```

### Issue 2: Authentication errors
**Solution**: Verify application default credentials
```bash
gcloud auth application-default login
terraform init
```

### Issue 3: Permission denied on directories
**Solution**: Fix directory permissions
```bash
sudo chown -R $USER:$USER terraform/
chmod -R u+w terraform/
```

### Issue 4: Provider version conflicts
**Solution**: Remove lock file and reinitialize
```bash
rm .terraform.lock.hcl
terraform init -upgrade
```

## Security Best Practices

1. **Never commit state files**
   - Always use remote backend (configured in ticket #8)
   - Local state files contain sensitive data

2. **Never commit .tfvars files with secrets**
   - Use environment variables for secrets
   - Or use Secret Manager (configured in ticket #10)

3. **Use least privilege for service accounts**
   - Will be configured in ticket #7

4. **Enable state locking**
   - Will be configured with GCS backend in ticket #8

## Documentation
- [ ] Update `docs/setup/03-terraform-setup.md`
- [ ] Document directory structure
- [ ] Add provider version information
- [ ] Document variable defaults

## Estimated Time
1-2 hours

## Dependencies
- Depends on: #2 (Install Terraform), #3 (Configure gcloud CLI)
- Blocks: #8 (Set up GCS bucket for Terraform state), #19 (Create Terraform directory structure)

## Labels
`phase-0`, `tool-install`, `infrastructure`, `terraform`

## Resources
- [Terraform GCP Provider](https://registry.terraform.io/providers/hashicorp/google/latest/docs)
- [Terraform Configuration Syntax](https://developer.hashicorp.com/terraform/language/syntax/configuration)
- [GCP Provider Authentication](https://registry.terraform.io/providers/hashicorp/google/latest/docs/guides/getting_started#adding-credentials)
- [Terraform Best Practices](https://www.terraform-best-practices.com/)
5|||[GCP-SETUP] Create GCP project for Archie Platform V3|||phase-0,gcp-setup,infrastructure,gcp|||## Objective
Create a new Google Cloud Platform project named 'archie-platform-v3' with billing enabled and initial configuration for australia-southeast1 region.

## Prerequisites
- #3 (Configure gcloud CLI) must be completed
- Google account with organization/billing admin permissions
- GCP billing account available

## Detailed Steps

### Step 1: Check Billing Account

First, verify you have access to a billing account:

```bash
# List available billing accounts
gcloud billing accounts list

# Expected output:
ACCOUNT_ID            NAME                OPEN  MASTER_ACCOUNT_ID
01234A-567890-BCDEF9  My Billing Account  True

# Note: If you see no billing accounts, you need to create one first
# Visit: https://console.cloud.google.com/billing

# Save billing account ID for later
export BILLING_ACCOUNT_ID="01234A-567890-BCDEF9"  # Replace with your ID
```

### Step 2: Check Project ID Availability

```bash
# Check if project ID is available
gcloud projects describe archie-platform-v3 2>&1 | grep -q "NOT_FOUND" && echo "Project ID available" || echo "Project ID already exists"
```

### Step 3: Create the Project

#### Option 1: Without Organization (Personal/Small Team)

```bash
# Create project without organization
gcloud projects create archie-platform-v3 \
  --name="Archie Platform V3" \
  --labels=environment=production,project=archie-platform-v3,managed_by=terraform

# Set as default project
gcloud config set project archie-platform-v3
```

#### Option 2: With Organization (Enterprise)

```bash
# List organizations
gcloud organizations list

# Create project under organization
export ORG_ID="123456789012"  # Replace with your org ID

gcloud projects create archie-platform-v3 \
  --name="Archie Platform V3" \
  --organization=$ORG_ID \
  --labels=environment=production,project=archie-platform-v3,managed_by=terraform

# Set as default project
gcloud config set project archie-platform-v3
```

### Step 4: Link Billing Account

```bash
# Link billing account to project
gcloud billing projects link archie-platform-v3 \
  --billing-account=$BILLING_ACCOUNT_ID

# Verify billing is enabled
gcloud billing projects describe archie-platform-v3
# Expected: billingEnabled: true
```

### Step 5: Set Project Metadata

```bash
# Set default region metadata
gcloud compute project-info add-metadata \
  --metadata=google-compute-default-region=australia-southeast1,google-compute-default-zone=australia-southeast1-a
```

### Step 6: Create Environment-Specific Projects (Optional but Recommended)

```bash
# For development environment
gcloud projects create archie-platform-v3-dev \
  --name="Archie Platform V3 - Development" \
  --labels=environment=dev,project=archie-platform-v3,managed_by=terraform

gcloud billing projects link archie-platform-v3-dev \
  --billing-account=$BILLING_ACCOUNT_ID

# For staging environment
gcloud projects create archie-platform-v3-staging \
  --name="Archie Platform V3 - Staging" \
  --labels=environment=staging,project=archie-platform-v3,managed_by=terraform

gcloud billing projects link archie-platform-v3-staging \
  --billing-account=$BILLING_ACCOUNT_ID

# Production project is the main archie-platform-v3
```

### Step 7: Configure Project Settings

```bash
# Enable OS Login for better security
gcloud compute project-info add-metadata \
  --metadata enable-oslogin=TRUE

# Set usage export (for billing analysis)
gcloud compute project-info update \
  --default-network-tier=PREMIUM
```

## Acceptance Criteria
- [ ] Project 'archie-platform-v3' exists and is active
- [ ] Billing is enabled and linked
- [ ] Project is set as gcloud default
- [ ] Project has appropriate labels
- [ ] Default region is set to australia-southeast1
- [ ] Optional: Dev and staging projects created

## Testing/Verification

```bash
# Verify project exists
gcloud projects describe archie-platform-v3
# Expected: lifecycleState: ACTIVE

# Check billing status
gcloud billing projects describe archie-platform-v3
# Expected: billingEnabled: true

# Verify it's the default project
gcloud config get-value project
# Expected: archie-platform-v3

# List all projects
gcloud projects list --filter="name:archie-platform-v3"
# Expected: Shows all archie-platform-v3 projects

# Check project number (needed for some APIs)
gcloud projects describe archie-platform-v3 --format="value(projectNumber)"
# Save this number - you'll need it later

# Verify labels
gcloud projects describe archie-platform-v3 --format="json(labels)"
# Expected: Shows environment, project, managed_by labels

# Check project metadata
gcloud compute project-info describe --format="json(commonInstanceMetadata)"
# Expected: Shows default region and zone
```

## Understanding GCP Project Structure

### Project Hierarchy
```
Organization (optional)
  └── Folder (optional)
      └── Project (archie-platform-v3)
          ├── Resources (VMs, databases, etc.)
          └── Service Accounts
          └── APIs
          └── IAM Policies
```

### Project Identifiers

1. **Project Name**: Human-readable name ("Archie Platform V3")
2. **Project ID**: Unique identifier (archie-platform-v3)
   - Cannot be changed after creation
   - Must be globally unique across all GCP
3. **Project Number**: Auto-generated numeric ID
   - Used by some APIs and service accounts

### Labels vs Tags

- **Labels**: Key-value pairs for organization and filtering
  - Used for billing reports
  - Can be used in IAM conditions
  - Example: `environment=production`

- **Tags**: (Not covered here) Used for network security and policies

## Common Issues and Troubleshooting

### Issue 1: "Project ID already exists"

**Solution**: Choose a different project ID
```bash
gcloud projects create archie-platform-v3-prod \
  --name="Archie Platform V3 - Production"
```

### Issue 2: "Billing account not found"

**Solution**: Create a billing account first
1. Visit https://console.cloud.google.com/billing
2. Click "Create Account"
3. Enter payment details
4. Come back and run the billing link command

### Issue 3: "Permission denied"

**Solution**: Verify you have necessary permissions
```bash
# Check your permissions
gcloud projects get-iam-policy archie-platform-v3 \
  --flatten="bindings[].members" \
  --filter="bindings.members:user:$(gcloud config get-value account)"

# You need roles/resourcemanager.projectCreator
# And roles/billing.admin to link billing
```

### Issue 4: "Quota exceeded"

**Solution**: Request quota increase or delete unused projects
```bash
# List all your projects
gcloud projects list

# Delete unused projects
gcloud projects delete unused-project-id
```

### Issue 5: Cannot set default region

**Solution**: Enable Compute Engine API first
```bash
# This will be done in ticket #6, but if needed now:
gcloud services enable compute.googleapis.com
```

## Cost Management Setup

### Set up Budget Alerts

```bash
# Create a budget (via console)
# Visit: https://console.cloud.google.com/billing/budgets

# Or using gcloud (requires beta)
gcloud beta billing budgets create \
  --billing-account=$BILLING_ACCOUNT_ID \
  --display-name="Archie Platform V3 Monthly Budget" \
  --budget-amount=1000 \
  --threshold-rule=percent=50 \
  --threshold-rule=percent=90 \
  --threshold-rule=percent=100
```

### Enable Cost Breakdown

```bash
# Label resources consistently for cost tracking
# All resources should include these labels:
# - environment: dev/staging/prod
# - project: archie-platform-v3
# - component: api/worker/frontend/etc
# - managed_by: terraform
```

## Security Best Practices

1. **Use separate projects for environments**
   - Isolates resources and permissions
   - Prevents accidental production changes

2. **Enable audit logging** (will be configured in ticket #31)
   ```bash
   # Admin activity logs are enabled by default
   # Data access logs will be configured later
   ```

3. **Set up organization policies** (if using organization)
   ```bash
   # Example: Restrict public IPs
   gcloud resource-manager org-policies set-policy \
     --organization=$ORG_ID \
     policy.yaml
   ```

## Documentation
- [ ] Update `docs/setup/04-gcp-project.md`
- [ ] Document project ID, number, and billing account
- [ ] Save project numbers for all environments
- [ ] Document organization ID (if applicable)

## Estimated Time
0.5-1 hours (plus time for billing account setup if needed)

## Dependencies
- Depends on: #3 (Configure gcloud CLI)
- Blocks: #6 (Enable required APIs), #7 (Create service account)

## Labels
`phase-0`, `gcp-setup`, `infrastructure`, `gcp`

## Resources
- [Creating and Managing Projects](https://cloud.google.com/resource-manager/docs/creating-managing-projects)
- [GCP Project Best Practices](https://cloud.google.com/docs/enterprise/best-practices-for-enterprise-organizations)
- [Setting up Billing](https://cloud.google.com/billing/docs/how-to/modify-project)
- [Project Labels](https://cloud.google.com/resource-manager/docs/creating-managing-labels)
6|||[GCP-SETUP] Enable required GCP APIs for Archie Platform V3|||phase-0,gcp-setup,infrastructure,gcp,apis|||## Objective
Enable all necessary Google Cloud Platform APIs required for the Archie Platform V3 infrastructure, including compute, networking, storage, database, messaging, and monitoring services.

## Prerequisites
- #5 (Create GCP project) must be completed
- gcloud CLI configured with correct project
- Billing must be enabled on the project

## Detailed Steps

### Step 1: Verify Current Project and Billing

```bash
# Confirm you're working with the correct project
gcloud config get-value project
# Expected: archie-platform-v3

# Verify billing is enabled
gcloud billing projects describe archie-platform-v3 | grep billingEnabled
# Expected: billingEnabled: true
```

### Step 2: List Currently Enabled APIs

```bash
# Check what APIs are already enabled
gcloud services list --enabled

# Note: New projects typically have these enabled by default:
# - cloudapis.googleapis.com
# - clouddebugger.googleapis.com  
# - cloudtrace.googleapis.com
```

### Step 3: Enable Core Infrastructure APIs

```bash
# Enable essential infrastructure APIs
gcloud services enable \
  compute.googleapis.com \
  container.googleapis.com \
  servicenetworking.googleapis.com \
  cloudresourcemanager.googleapis.com \
  iam.googleapis.com \
  iamcredentials.googleapis.com

# Wait for APIs to be enabled (takes 1-2 minutes)
echo "Waiting for core APIs to be enabled..."
sleep 60
```

### Step 4: Enable Storage APIs

```bash
# Enable storage and database APIs
gcloud services enable \
  storage.googleapis.com \
  storage-component.googleapis.com \
  storage-api.googleapis.com \
  sqladmin.googleapis.com \
  redis.googleapis.com

echo "Waiting for storage APIs to be enabled..."
sleep 30
```

### Step 5: Enable Networking APIs

```bash
# Enable networking APIs
gcloud services enable \
  dns.googleapis.com \
  servicedirectory.googleapis.com \
  vpcaccess.googleapis.com \
  networkmanagement.googleapis.com

echo "Waiting for networking APIs to be enabled..."
sleep 30
```

### Step 6: Enable Cloud Run and Build APIs

```bash
# Enable serverless and CI/CD APIs
gcloud services enable \
  run.googleapis.com \
  cloudbuild.googleapis.com \
  artifactregistry.googleapis.com \
  containerregistry.googleapis.com

echo "Waiting for serverless APIs to be enabled..."
sleep 30
```

### Step 7: Enable Security APIs

```bash
# Enable security and secrets management APIs
gcloud services enable \
  secretmanager.googleapis.com \
  cloudkms.googleapis.com \
  securitycenter.googleapis.com

echo "Waiting for security APIs to be enabled..."
sleep 30
```

### Step 8: Enable Monitoring and Logging APIs

```bash
# Enable observability APIs
gcloud services enable \
  monitoring.googleapis.com \
  logging.googleapis.com \
  cloudtrace.googleapis.com \
  cloudprofiler.googleapis.com \
  clouderrorreporting.googleapis.com

echo "Waiting for monitoring APIs to be enabled..."
sleep 30
```

### Step 9: Enable Additional Required APIs

```bash
# Enable miscellaneous required APIs
gcloud services enable \
  serviceusage.googleapis.com \
  cloudscheduler.googleapis.com \
  pubsub.googleapis.com \
  eventarc.googleapis.com

echo "All APIs enabled!"
```

### Step 10: Create Convenience Script for Future Use

```bash
# Create script to enable all APIs at once
cat > /mnt/data-disk1/archie-platform-v3/scripts/enable-gcp-apis.sh <<'EOF'
#!/bin/bash
# Script to enable all required GCP APIs for Archie Platform V3

set -e

PROJECT_ID=${1:-archie-platform-v3}

echo "Enabling APIs for project: $PROJECT_ID"
gcloud config set project $PROJECT_ID

# Array of all required APIs
APIS=(
  # Core Infrastructure
  compute.googleapis.com
  container.googleapis.com
  servicenetworking.googleapis.com
  cloudresourcemanager.googleapis.com
  iam.googleapis.com
  iamcredentials.googleapis.com
  
  # Storage & Databases
  storage.googleapis.com
  storage-component.googleapis.com
  storage-api.googleapis.com
  sqladmin.googleapis.com
  redis.googleapis.com
  
  # Networking
  dns.googleapis.com
  servicedirectory.googleapis.com
  vpcaccess.googleapis.com
  networkmanagement.googleapis.com
  
  # Cloud Run & Build
  run.googleapis.com
  cloudbuild.googleapis.com
  artifactregistry.googleapis.com
  containerregistry.googleapis.com
  
  # Security
  secretmanager.googleapis.com
  cloudkms.googleapis.com
  securitycenter.googleapis.com
  
  # Monitoring & Logging
  monitoring.googleapis.com
  logging.googleapis.com
  cloudtrace.googleapis.com
  cloudprofiler.googleapis.com
  clouderrorreporting.googleapis.com
  
  # Additional Services
  serviceusage.googleapis.com
  cloudscheduler.googleapis.com
  pubsub.googleapis.com
  eventarc.googleapis.com
)

echo "Enabling ${#APIS[@]} APIs..."
gcloud services enable "${APIS[@]}"

echo "✓ All APIs enabled successfully!"
EOF

# Make script executable
chmod +x /mnt/data-disk1/archie-platform-v3/scripts/enable-gcp-apis.sh
```

## Acceptance Criteria
- [ ] All 35+ required APIs are enabled
- [ ] No errors during API enablement
- [ ] Service Usage API is accessible
- [ ] APIs can be listed with gcloud
- [ ] Convenience script is created and tested

## Testing/Verification

```bash
# List all enabled APIs
gcloud services list --enabled

# Count enabled APIs
gcloud services list --enabled | wc -l
# Expected: At least 35 APIs

# Verify specific critical APIs
for api in compute.googleapis.com run.googleapis.com sqladmin.googleapis.com secretmanager.googleapis.com; do
  if gcloud services list --enabled | grep -q $api; then
    echo "✓ $api is enabled"
  else
    echo "✗ $api is NOT enabled"
  fi
done

# Test API access - Compute API
gcloud compute regions list --filter="name:australia" --format="table(name,status)" 2>&1
# Expected: Shows australia-southeast1 and australia-southeast2 with status UP

# Test API access - Cloud Run API  
gcloud run services list --region=australia-southeast1 2>&1
# Expected: "Listed 0 items" (no services yet, but API works)

# Test API access - SQL Admin API
gcloud sql instances list 2>&1
# Expected: "Listed 0 items" (no instances yet, but API works)

# Test API access - Secret Manager API
gcloud secrets list 2>&1
# Expected: "Listed 0 items" (no secrets yet, but API works)

# Check for any APIs that failed to enable
gcloud services list --enabled --format="table(config.name,state)" | grep -i "disabled\|failed" || echo "All APIs successfully enabled"
```

## Understanding GCP APIs

### What are GCP APIs?

GCP APIs are interfaces that allow you to programmatically interact with Google Cloud services. They must be explicitly enabled before use.

### Why Enable APIs?

1. **Security**: Disabled by default to minimize attack surface
2. **Cost Control**: Only pay for services you use
3. **Quota Management**: Each API has separate quotas
4. **Audit Trail**: API enablement is logged

### API Categories in Archie Platform V3

#### Core Infrastructure (6 APIs)
- `compute.googleapis.com` - Virtual machines, networks
- `container.googleapis.com` - Kubernetes (GKE) if needed
- `servicenetworking.googleapis.com` - Private service connections
- `cloudresourcemanager.googleapis.com` - Project management
- `iam.googleapis.com` - Identity and access management
- `iamcredentials.googleapis.com` - Service account credentials

#### Storage & Databases (5 APIs)
- `storage.googleapis.com` - Cloud Storage buckets
- `sqladmin.googleapis.com` - Cloud SQL (PostgreSQL)
- `redis.googleapis.com` - Cloud Memorystore (Redis)

#### Networking (4 APIs)
- `dns.googleapis.com` - Cloud DNS
- `servicedirectory.googleapis.com` - Service discovery
- `vpcaccess.googleapis.com` - VPC connectors for Cloud Run
- `networkmanagement.googleapis.com` - Network troubleshooting

#### Cloud Run & Build (4 APIs)
- `run.googleapis.com` - Cloud Run serverless containers
- `cloudbuild.googleapis.com` - CI/CD builds
- `artifactregistry.googleapis.com` - Container image storage
- `containerregistry.googleapis.com` - Legacy container registry

#### Security (3 APIs)
- `secretmanager.googleapis.com` - Secrets storage
- `cloudkms.googleapis.com` - Encryption key management
- `securitycenter.googleapis.com` - Security monitoring

#### Monitoring & Logging (5 APIs)
- `monitoring.googleapis.com` - Cloud Monitoring (metrics)
- `logging.googleapis.com` - Cloud Logging (logs)
- `cloudtrace.googleapis.com` - Distributed tracing
- `cloudprofiler.googleapis.com` - Performance profiling
- `clouderrorreporting.googleapis.com` - Error tracking

#### Additional Services (4 APIs)
- `serviceusage.googleapis.com` - API management
- `cloudscheduler.googleapis.com` - Cron jobs
- `pubsub.googleapis.com` - Pub/Sub messaging
- `eventarc.googleapis.com` - Event-driven architecture

## Common Issues and Troubleshooting

### Issue 1: "Billing not enabled" error

**Solution**: Enable billing on the project
```bash
# Check billing status
gcloud billing projects describe archie-platform-v3

# If not enabled, link billing account
gcloud billing projects link archie-platform-v3 \
  --billing-account=YOUR-BILLING-ACCOUNT-ID
```

### Issue 2: API enablement times out

**Solution**: Enable APIs in smaller batches
```bash
# Enable one at a time if needed
gcloud services enable compute.googleapis.com
echo "Waiting..."
sleep 30
gcloud services enable run.googleapis.com
# etc.
```

### Issue 3: "Permission denied" error

**Solution**: Verify you have serviceusage.services.enable permission
```bash
# Check your permissions
gcloud projects get-iam-policy archie-platform-v3 \
  --flatten="bindings[].members" \
  --filter="bindings.members:user:$(gcloud config get-value account)"

# You need roles/serviceusage.serviceUsageAdmin or roles/owner
```

### Issue 4: API still not working after enabling

**Solution**: Wait for propagation and retry
```bash
# Some APIs take a few minutes to fully propagate
echo "Waiting for API propagation..."
sleep 120

# Retry the operation
gcloud compute regions list
```

### Issue 5: "API not found" error

**Solution**: Verify the API name is correct
```bash
# Search for the correct API name
gcloud services list --available | grep -i "storage"

# Use exact API name from output
gcloud services enable storage.googleapis.com
```

## API Quotas and Limits

### Check Current Quotas

```bash
# View quotas for a specific API
gcloud compute project-info describe --format="yaml(quotas)"

# View quota usage
gcloud logging read 'protoPayload.methodName="compute.googleapis.com.v1.quotas.get"' \
  --limit 10 \
  --format json
```

### Request Quota Increases

If you hit quota limits:
1. Visit https://console.cloud.google.com/iam-admin/quotas
2. Select the API and metric
3. Click "Edit Quotas"
4. Submit increase request with justification

## Cost Implications

### Free Tier

Many APIs have free tiers:
- Cloud Storage: 5 GB free
- Cloud Run: 2 million requests/month free
- Cloud Logging: 50 GB/month free
- Secret Manager: 6 active secrets free

### Monitoring Costs

```bash
# Set up billing alerts (via console)
# Visit: https://console.cloud.google.com/billing/budgets

# Create alert for API usage costs
# Recommended budget: Start with $100/month for dev
```

## Documentation
- [ ] Update `docs/setup/05-gcp-apis.md`
- [ ] Document all enabled APIs
- [ ] Note any quota increases requested
- [ ] Save API enablement script location

## Estimated Time
1-2 hours (including wait times for API propagation)

## Dependencies
- Depends on: #5 (Create GCP project)
- Blocks: #7 (Create service account), #9 (Configure VPC), #10 (Set up Secret Manager)

## Labels
`phase-0`, `gcp-setup`, `infrastructure`, `gcp`, `apis`

## Resources
- [Enabling and Disabling APIs](https://cloud.google.com/service-usage/docs/enable-disable)
- [Service Usage API](https://cloud.google.com/service-usage/docs/overview)
- [GCP API Reference](https://cloud.google.com/apis)
- [API Quotas](https://cloud.google.com/docs/quota)
7|||[GCP-SETUP] Create Terraform service account with required IAM roles|||phase-0,gcp-setup,infrastructure,gcp,iam,security|||## Objective
Create a dedicated service account for Terraform with least-privilege IAM permissions to manage GCP infrastructure securely.

## Prerequisites
- #5 (Create GCP project) must be completed
- #6 (Enable required APIs) must be completed
- IAM API must be enabled

## Detailed Steps

### Step 1: Verify Prerequisites

```bash
# Confirm correct project
gcloud config get-value project
# Expected: archie-platform-v3

# Verify IAM API is enabled
gcloud services list --enabled | grep iam.googleapis.com
# Expected: iam.googleapis.com
```

### Step 2: Create Service Account

```bash
# Create service account for Terraform
gcloud iam service-accounts create terraform-sa \
  --display-name="Terraform Service Account" \
  --description="Service account for Terraform infrastructure automation"

# Verify creation
gcloud iam service-accounts list | grep terraform-sa

# Get service account email
export SA_EMAIL="terraform-sa@archie-platform-v3.iam.gserviceaccount.com"
echo $SA_EMAIL
```

### Step 3: Grant Required IAM Roles

```bash
# Get project ID
export PROJECT_ID="archie-platform-v3"

# Grant roles for compute resources
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:$SA_EMAIL" \
  --role="roles/compute.admin"

# Grant roles for networking
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:$SA_EMAIL" \
  --role="roles/compute.networkAdmin"

gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:$SA_EMAIL" \
  --role="roles/dns.admin"

# Grant roles for storage
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:$SA_EMAIL" \
  --role="roles/storage.admin"

# Grant roles for Cloud SQL
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:$SA_EMAIL" \
  --role="roles/cloudsql.admin"

# Grant roles for Redis
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:$SA_EMAIL" \
  --role="roles/redis.admin"

# Grant roles for Cloud Run
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:$SA_EMAIL" \
  --role="roles/run.admin"

# Grant roles for IAM management
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:$SA_EMAIL" \
  --role="roles/iam.serviceAccountAdmin"

gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:$SA_EMAIL" \
  --role="roles/iam.serviceAccountKeyAdmin"

# Grant roles for Secret Manager
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:$SA_EMAIL" \
  --role="roles/secretmanager.admin"

# Grant roles for Cloud Build
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:$SA_EMAIL" \
  --role="roles/cloudbuild.builds.editor"

# Grant roles for Artifact Registry
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:$SA_EMAIL" \
  --role="roles/artifactregistry.admin"

# Grant roles for Service Usage (API management)
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:$SA_EMAIL" \
  --role="roles/serviceusage.serviceUsageAdmin"

# Grant roles for monitoring
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:$SA_EMAIL" \
  --role="roles/monitoring.admin"

gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:$SA_EMAIL" \
  --role="roles/logging.admin"

# Grant Service Account User role (for impersonation)
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:$SA_EMAIL" \
  --role="roles/iam.serviceAccountUser"
```

### Step 4: Create Service Account Key

```bash
# Create keys directory
mkdir -p /mnt/data-disk1/archie-platform-v3/.keys

# Set restrictive permissions
chmod 700 /mnt/data-disk1/archie-platform-v3/.keys

# Generate service account key
gcloud iam service-accounts keys create \
  /mnt/data-disk1/archie-platform-v3/.keys/terraform-sa-key.json \
  --iam-account=$SA_EMAIL

# Set restrictive permissions on key file
chmod 600 /mnt/data-disk1/archie-platform-v3/.keys/terraform-sa-key.json

# Verify key file exists
ls -la /mnt/data-disk1/archie-platform-v3/.keys/
```

### Step 5: Update .gitignore

```bash
# Ensure .keys directory is ignored
cat >> /mnt/data-disk1/archie-platform-v3/.gitignore <<'EOF'

# Service account keys - NEVER COMMIT
.keys/
*.json
**/*-key.json
**/*-sa-*.json
EOF

# Verify .gitignore
grep -E '\.keys|\*\.json' /mnt/data-disk1/archie-platform-v3/.gitignore
```

### Step 6: Configure Terraform to Use Service Account

```bash
# Create environment variable file (not committed)
cat > /mnt/data-disk1/archie-platform-v3/.env.terraform <<EOF
# Terraform Service Account Configuration
# DO NOT COMMIT THIS FILE

export GOOGLE_APPLICATION_CREDENTIALS="/mnt/data-disk1/archie-platform-v3/.keys/terraform-sa-key.json"
export GOOGLE_PROJECT="$PROJECT_ID"
export GOOGLE_REGION="australia-southeast1"
export GOOGLE_ZONE="australia-southeast1-a"
EOF

# Add to .gitignore
echo '.env.terraform' >> /mnt/data-disk1/archie-platform-v3/.gitignore

# Source the environment file
source /mnt/data-disk1/archie-platform-v3/.env.terraform
```

### Step 7: Create Service Account for Each Environment (Optional)

```bash
# If you created separate projects for dev/staging/prod
# Create separate service accounts

for env in dev staging; do
  gcloud iam service-accounts create terraform-sa \
    --project=archie-platform-v3-$env \
    --display-name="Terraform Service Account - $env" \
    --description="Terraform SA for $env environment"
  
  # Grant roles (repeat Step 3 for each environment)
  # ...
  
  # Create key
  gcloud iam service-accounts keys create \
    /mnt/data-disk1/archie-platform-v3/.keys/terraform-sa-key-$env.json \
    --iam-account=terraform-sa@archie-platform-v3-$env.iam.gserviceaccount.com
  
  chmod 600 /mnt/data-disk1/archie-platform-v3/.keys/terraform-sa-key-$env.json
done
```

## Acceptance Criteria
- [ ] Service account 'terraform-sa' is created
- [ ] All required IAM roles are granted
- [ ] Service account key is generated and secured
- [ ] Key file has correct permissions (600)
- [ ] .keys directory is in .gitignore
- [ ] Environment variables are configured
- [ ] Terraform can authenticate using service account

## Testing/Verification

```bash
# Verify service account exists
gcloud iam service-accounts describe $SA_EMAIL

# List all granted roles
gcloud projects get-iam-policy $PROJECT_ID \
  --flatten="bindings[].members" \
  --filter="bindings.members:serviceAccount:$SA_EMAIL" \
  --format="table(bindings.role)"
# Expected: Shows all 15+ roles

# Verify key file exists and has correct permissions
ls -la /mnt/data-disk1/archie-platform-v3/.keys/terraform-sa-key.json
# Expected: -rw------- (600 permissions)

# Verify key is valid JSON
python3 -c "import json; json.load(open('/mnt/data-disk1/archie-platform-v3/.keys/terraform-sa-key.json'))"
echo $?
# Expected: 0 (success)

# Test authentication with service account
gcloud auth activate-service-account --key-file=/mnt/data-disk1/archie-platform-v3/.keys/terraform-sa-key.json

# Test service account has access
gcloud compute regions list --limit=1
# Expected: Lists at least one region

# Switch back to user account
gcloud config set account $(gcloud auth list --filter=status:ACTIVE --format="value(account)" | grep -v gserviceaccount)

# Verify .gitignore
cd /mnt/data-disk1/archie-platform-v3
git status .keys/
# Expected: Not shown (ignored)

# Test Terraform authentication
cd /mnt/data-disk1/archie-platform-v3/terraform
export GOOGLE_APPLICATION_CREDENTIALS="/mnt/data-disk1/archie-platform-v3/.keys/terraform-sa-key.json"
terraform init
terraform plan
# Expected: No authentication errors
```

## Understanding Service Accounts

### What is a Service Account?

A service account is a special type of Google account that represents an application or compute workload, not a human user.

### Key Characteristics

1. **Email Format**: `[SA-NAME]@[PROJECT-ID].iam.gserviceaccount.com`
2. **No Password**: Uses key files or OAuth for authentication
3. **Programmatic Access**: Designed for applications, not interactive use
4. **Scoped Permissions**: Grant only necessary permissions (least privilege)

### Authentication Methods

#### 1. Service Account Keys (What we're using)
```
- JSON file with private key
- Stored securely, never committed
- Used by: Terraform, GitHub Actions, local development
- Rotation: Recommended every 90 days
```

#### 2. Workload Identity (Alternative for GKE)
```
- No key files needed
- GKE pods authenticate directly
- More secure, no key management
```

#### 3. Metadata Server (For GCE instances)
```
- Automatic authentication
- No key files
- Only works on GCP compute resources
```

### IAM Roles Granted

| Role | Purpose | Resources Managed |
|------|---------|-------------------|
| `compute.admin` | Full compute control | VMs, disks, images |
| `compute.networkAdmin` | Network management | VPCs, subnets, firewalls |
| `dns.admin` | DNS management | DNS zones, records |
| `storage.admin` | Storage management | GCS buckets, objects |
| `cloudsql.admin` | Database management | Cloud SQL instances |
| `redis.admin` | Cache management | Redis instances |
| `run.admin` | Serverless management | Cloud Run services |
| `iam.serviceAccountAdmin` | Service account creation | Service accounts |
| `secretmanager.admin` | Secrets management | Secrets |
| `cloudbuild.builds.editor` | CI/CD management | Build triggers |
| `artifactregistry.admin` | Container registry | Docker images |
| `serviceusage.serviceUsageAdmin` | API management | Enable/disable APIs |
| `monitoring.admin` | Observability | Metrics, dashboards |
| `logging.admin` | Log management | Log sinks, exports |

## Common Issues and Troubleshooting

### Issue 1: "Permission denied" when creating service account

**Solution**: Verify you have necessary permissions
```bash
# You need roles/iam.serviceAccountAdmin or roles/owner
gcloud projects get-iam-policy $PROJECT_ID \
  --flatten="bindings[].members" \
  --filter="bindings.members:user:$(gcloud config get-value account)"
```

### Issue 2: Key file permissions too open

**Solution**: Set correct permissions
```bash
chmod 600 /mnt/data-disk1/archie-platform-v3/.keys/terraform-sa-key.json
chmod 700 /mnt/data-disk1/archie-platform-v3/.keys
```

### Issue 3: Terraform can't authenticate

**Solution**: Verify environment variable
```bash
echo $GOOGLE_APPLICATION_CREDENTIALS
# Should point to key file

test -f "$GOOGLE_APPLICATION_CREDENTIALS" && echo "Key file exists" || echo "Key file missing"
```

### Issue 4: Service account lacks permissions

**Solution**: Add missing role
```bash
# Check what roles are granted
gcloud projects get-iam-policy $PROJECT_ID \
  --flatten="bindings[].members" \
  --filter="bindings.members:serviceAccount:$SA_EMAIL"

# Add missing role
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:$SA_EMAIL" \
  --role="roles/ROLE_NAME"
```

### Issue 5: Accidentally committed key file

**Solution**: Immediately revoke and rotate
```bash
# List keys for service account
gcloud iam service-accounts keys list --iam-account=$SA_EMAIL

# Delete compromised key
gcloud iam service-accounts keys delete KEY_ID \
  --iam-account=$SA_EMAIL

# Create new key
gcloud iam service-accounts keys create \
  /mnt/data-disk1/archie-platform-v3/.keys/terraform-sa-key.json \
  --iam-account=$SA_EMAIL

# Remove from git history (if already committed)
git filter-branch --force --index-filter \
  "git rm --cached --ignore-unmatch .keys/terraform-sa-key.json" \
  --prune-empty --tag-name-filter cat -- --all
```

## Security Best Practices

### 1. Principle of Least Privilege

```bash
# Start with minimal roles, add as needed
# Review roles quarterly
gcloud projects get-iam-policy $PROJECT_ID \
  --flatten="bindings[].members" \
  --filter="bindings.members:serviceAccount:$SA_EMAIL" \
  > sa-permissions-$(date +%Y%m%d).txt
```

### 2. Key Rotation

```bash
# Rotate keys every 90 days
# Create rotation reminder
echo "Rotate Terraform SA key" | at now + 90 days

# Script to rotate key
cat > /mnt/data-disk1/archie-platform-v3/scripts/rotate-sa-key.sh <<'EOF'
#!/bin/bash
SA_EMAIL="terraform-sa@archie-platform-v3.iam.gserviceaccount.com"
KEY_FILE="/mnt/data-disk1/archie-platform-v3/.keys/terraform-sa-key.json"

# Get old key ID
OLD_KEY_ID=$(jq -r .private_key_id < $KEY_FILE)

# Create new key
gcloud iam service-accounts keys create ${KEY_FILE}.new \
  --iam-account=$SA_EMAIL

# Test new key
export GOOGLE_APPLICATION_CREDENTIALS="${KEY_FILE}.new"
gcloud compute regions list --limit=1

# If successful, replace old key
if [ $? -eq 0 ]; then
  mv ${KEY_FILE}.new $KEY_FILE
  chmod 600 $KEY_FILE
  
  # Delete old key
  gcloud iam service-accounts keys delete $OLD_KEY_ID \
    --iam-account=$SA_EMAIL --quiet
  
  echo "Key rotated successfully"
else
  echo "Key rotation failed"
  rm ${KEY_FILE}.new
  exit 1
fi
EOF

chmod +x /mnt/data-disk1/archie-platform-v3/scripts/rotate-sa-key.sh
```

### 3. Audit Logging

```bash
# Enable audit logs for service account usage
gcloud logging read \
  'protoPayload.authenticationInfo.principalEmail="terraform-sa@archie-platform-v3.iam.gserviceaccount.com"' \
  --limit 10 \
  --format json
```

### 4. Service Account Impersonation (Alternative)

```bash
# Instead of keys, use impersonation (more secure)
# Grant your user account the ability to impersonate
gcloud iam service-accounts add-iam-policy-binding $SA_EMAIL \
  --member="user:your-email@domain.com" \
  --role="roles/iam.serviceAccountTokenCreator"

# Use impersonation in Terraform
# In provider.tf:
# provider "google" {
#   impersonate_service_account = "terraform-sa@archie-platform-v3.iam.gserviceaccount.com"
# }
```

## Documentation
- [ ] Update `docs/setup/06-service-accounts.md`
- [ ] Document service account email
- [ ] Document key file location (but not the key itself!)
- [ ] Document IAM roles granted
- [ ] Add key rotation schedule to calendar

## Estimated Time
1-2 hours

## Dependencies
- Depends on: #5 (Create GCP project), #6 (Enable required APIs)
- Blocks: #8 (Set up GCS bucket for Terraform state), All Terraform infrastructure tickets

## Labels
`phase-0`, `gcp-setup`, `infrastructure`, `gcp`, `iam`, `security`

## Resources
- [Service Accounts Overview](https://cloud.google.com/iam/docs/service-accounts)
- [Best Practices for Service Accounts](https://cloud.google.com/iam/docs/best-practices-service-accounts)
- [Understanding Service Account Keys](https://cloud.google.com/iam/docs/creating-managing-service-account-keys)
- [IAM Roles Reference](https://cloud.google.com/iam/docs/understanding-roles)
8|||[GCP-SETUP] Set up GCS bucket for Terraform remote state|||phase-0,gcp-setup,infrastructure,terraform,storage|||## Objective
Create a Google Cloud Storage bucket to store Terraform state files with versioning, encryption, and lifecycle policies for secure remote state management.

## Prerequisites
- #5 (Create GCP project) must be completed
- #6 (Enable required APIs) must be completed
- #7 (Create Terraform service account) must be completed
- Storage API must be enabled

## Detailed Steps

### Step 1: Verify Prerequisites

```bash
# Verify correct project
gcloud config get-value project
# Expected: archie-platform-v3

# Verify Storage API is enabled
gcloud services list --enabled | grep storage.googleapis.com
# Expected: storage.googleapis.com
```

### Step 2: Create GCS Bucket for Terraform State

```bash
# Set variables
export PROJECT_ID="archie-platform-v3"
export REGION="australia-southeast1"
export BUCKET_NAME="${PROJECT_ID}-terraform-state"

# Create the bucket
gcloud storage buckets create gs://${BUCKET_NAME} \
  --project=${PROJECT_ID} \
  --location=${REGION} \
  --uniform-bucket-level-access \
  --public-access-prevention

echo "Bucket created: gs://${BUCKET_NAME}"
```

### Step 3: Enable Versioning

```bash
# Enable versioning for state file history
gcloud storage buckets update gs://${BUCKET_NAME} \
  --versioning

# Verify versioning is enabled
gcloud storage buckets describe gs://${BUCKET_NAME} --format="value(versioning.enabled)"
# Expected: True
```

### Step 4: Configure Lifecycle Policy

```bash
# Create lifecycle policy file
cat > /tmp/terraform-state-lifecycle.json <<'EOF'
{
  "lifecycle": {
    "rule": [
      {
        "action": {
          "type": "Delete"
        },
        "condition": {
          "numNewerVersions": 10,
          "isLive": false
        }
      }
    ]
  }
}
EOF

# Apply lifecycle policy
gcloud storage buckets update gs://${BUCKET_NAME} \
  --lifecycle-file=/tmp/terraform-state-lifecycle.json

echo "Lifecycle policy applied (keep last 10 versions)"
```

### Step 5: Configure IAM Permissions

```bash
# Grant Terraform service account access
export SA_EMAIL="terraform-sa@${PROJECT_ID}.iam.gserviceaccount.com"

# Grant storage admin role on bucket
gcloud storage buckets add-iam-policy-binding gs://${BUCKET_NAME} \
  --member="serviceAccount:${SA_EMAIL}" \
  --role="roles/storage.objectAdmin"

# Verify permissions
gcloud storage buckets get-iam-policy gs://${BUCKET_NAME}
```

### Step 6: Enable Encryption (Optional - Default encryption is sufficient)

```bash
# Default encryption is enabled automatically
# For Customer-Managed Encryption Keys (CMEK), uncomment below:

# Create KMS keyring and key (requires cloudkms.googleapis.com enabled)
# gcloud kms keyrings create terraform-state-keyring \
#   --location=${REGION}

# gcloud kms keys create terraform-state-key \
#   --location=${REGION} \
#   --keyring=terraform-state-keyring \
#   --purpose=encryption

# Set bucket to use CMEK
# gcloud storage buckets update gs://${BUCKET_NAME} \
#   --default-encryption-key=projects/${PROJECT_ID}/locations/${REGION}/keyRings/terraform-state-keyring/cryptoKeys/terraform-state-key
```

### Step 7: Configure Terraform Backend

```bash
# Update terraform/provider.tf to use GCS backend
cd /mnt/data-disk1/archie-platform-v3/terraform

# Backup existing provider.tf
cp provider.tf provider.tf.backup

# Update provider.tf with backend configuration
cat > provider.tf <<EOF
# Google Cloud Provider Configuration
# This file defines how Terraform communicates with GCP

terraform {
  required_version = ">= 1.6.0"
  
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 5.0"
    }
    google-beta = {
      source  = "hashicorp/google-beta"
      version = "~> 5.0"
    }
  }
  
  # GCS Backend for remote state storage
  backend "gcs" {
    bucket  = "${BUCKET_NAME}"
    prefix  = "terraform/state"
  }
}

# Primary provider for stable GCP resources
provider "google" {
  project = var.project_id
  region  = var.region
  zone    = var.zone
}

# Beta provider for preview features
provider "google-beta" {
  project = var.project_id
  region  = var.region
  zone    = var.zone
}
EOF
```

### Step 8: Initialize Terraform with Remote Backend

```bash
# Remove local state if exists
rm -f terraform.tfstate terraform.tfstate.backup

# Reinitialize Terraform with backend
terraform init -migrate-state

# Confirm migration when prompted
# Type 'yes' to migrate state to GCS
```

### Step 9: Create Backend Configuration for Each Environment

```bash
# Create backend config for dev environment
cat > /mnt/data-disk1/archie-platform-v3/terraform/environments/dev/backend.tf <<EOF
terraform {
  backend "gcs" {
    bucket  = "${BUCKET_NAME}"
    prefix  = "terraform/state/dev"
  }
}
EOF

# Create backend config for staging environment
cat > /mnt/data-disk1/archie-platform-v3/terraform/environments/staging/backend.tf <<EOF
terraform {
  backend "gcs" {
    bucket  = "${BUCKET_NAME}"
    prefix  = "terraform/state/staging"
  }
}
EOF

# Create backend config for prod environment
cat > /mnt/data-disk1/archie-platform-v3/terraform/environments/prod/backend.tf <<EOF
terraform {
  backend "gcs" {
    bucket  = "${BUCKET_NAME}"
    prefix  = "terraform/state/prod"
  }
}
EOF
```

### Step 10: Test State Locking

```bash
# Test state locking works
cd /mnt/data-disk1/archie-platform-v3/terraform

# Create a test resource
cat > test-lock.tf <<'EOF'
resource "null_resource" "test" {
  provisioner "local-exec" {
    command = "echo testing state lock"
  }
}
EOF

# Apply (this will test locking)
terraform apply -auto-approve

# Clean up test
rm test-lock.tf
terraform apply -auto-approve
```

## Acceptance Criteria
- [ ] GCS bucket created for Terraform state
- [ ] Versioning is enabled on bucket
- [ ] Lifecycle policy is configured
- [ ] IAM permissions are set for service account
- [ ] Terraform backend is configured in code
- [ ] Terraform successfully initializes with remote backend
- [ ] State locking works correctly
- [ ] Backend configs created for all environments

## Testing/Verification

```bash
# Verify bucket exists
gcloud storage buckets describe gs://${BUCKET_NAME}
# Expected: Shows bucket details

# Check versioning
gcloud storage buckets describe gs://${BUCKET_NAME} --format="yaml(versioning)"
# Expected: enabled: true

# Check lifecycle policy
gcloud storage buckets describe gs://${BUCKET_NAME} --format="yaml(lifecycle)"
# Expected: Shows rule to delete old versions

# Check IAM permissions
gcloud storage buckets get-iam-policy gs://${BUCKET_NAME} \
  --flatten="bindings[].members" \
  --filter="bindings.members:serviceAccount:terraform-sa@*"
# Expected: Shows terraform-sa with storage.objectAdmin

# Verify Terraform state is in bucket
gcloud storage ls gs://${BUCKET_NAME}/terraform/state/
# Expected: Shows default.tfstate file

# Check state file details
terraform state list
# Expected: Shows resources (if any)

# Verify state locking
# In one terminal:
terraform plan
# In another terminal (while first is running):
terraform plan
# Expected: Second command waits for lock

# Check for state file locally (should not exist)
ls -la terraform.tfstate
# Expected: No such file (state is remote)
```

## Understanding Terraform Remote State

### Why Remote State?

1. **Collaboration**: Multiple team members can work together
2. **State Locking**: Prevents concurrent modifications
3. **Security**: State files contain sensitive data
4. **Backup**: Automatic versioning and backup
5. **Automation**: Required for CI/CD pipelines

### State Locking in GCS

GCS backend uses native locking:
- Lock file: `.terraform.tflock` (temporary)
- Automatic lock acquisition
- Automatic lock release after operation
- Timeout: 10 seconds default

### State File Organization

```
gs://archie-platform-v3-terraform-state/
├── terraform/state/
│   └── default.tfstate              # Root terraform state
├── terraform/state/dev/
│   └── default.tfstate              # Dev environment state
├── terraform/state/staging/
│   └── default.tfstate              # Staging environment state
└── terraform/state/prod/
    └── default.tfstate              # Prod environment state
```

### Versioning Strategy

- Keep last 10 versions (via lifecycle policy)
- Older versions automatically deleted
- Can recover from accidental changes
- View versions: `gcloud storage ls -a gs://bucket/path/`

## Common Issues and Troubleshooting

### Issue 1: "Bucket name already exists"

**Solution**: Bucket names are globally unique
```bash
# Add random suffix
export BUCKET_NAME="${PROJECT_ID}-terraform-state-$(date +%s)"
gcloud storage buckets create gs://${BUCKET_NAME} ...
```

### Issue 2: "Backend initialization failed"

**Solution**: Ensure service account has permissions
```bash
# Check service account can access bucket
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/sa-key.json"
gcloud storage ls gs://${BUCKET_NAME}
```

### Issue 3: "State locked" error persists

**Solution**: Force unlock (use carefully)
```bash
# Get lock ID from error message
terraform force-unlock LOCK_ID

# Only use if you're sure no other operation is running!
```

### Issue 4: Cannot migrate state

**Solution**: Manually copy state to bucket
```bash
# Backup local state
cp terraform.tfstate terraform.tfstate.local-backup

# Upload to bucket
gcloud storage cp terraform.tfstate gs://${BUCKET_NAME}/terraform/state/default.tfstate

# Reinitialize
terraform init -reconfigure
```

### Issue 5: Permission denied on bucket operations

**Solution**: Grant necessary IAM roles
```bash
# Grant roles at project level
gcloud projects add-iam-policy-binding ${PROJECT_ID} \
  --member="serviceAccount:${SA_EMAIL}" \
  --role="roles/storage.admin"
```

## State File Security

### Sensitive Data in State

Terraform state files contain:
- Resource configurations
- Database passwords
- API keys
- IP addresses
- Resource IDs

### Security Best Practices

1. **Never commit state files to git**
   ```bash
   # Already in .gitignore
   grep tfstate .gitignore
   ```

2. **Use encryption at rest** (automatic with GCS)
   - Google-managed keys by default
   - Or CMEK for more control

3. **Restrict bucket access**
   ```bash
   # Only service accounts should have access
   # No public access
   # Audit access logs
   ```

4. **Enable audit logging**
   ```bash
   # Bucket access logs
   gcloud storage buckets update gs://${BUCKET_NAME} \
     --log-bucket=gs://${PROJECT_ID}-logs \
     --log-object-prefix=terraform-state-access/
   ```

5. **Use sensitive variable handling**
   ```hcl
   # In Terraform code:
   variable "database_password" {
     type      = string
     sensitive = true
   }
   ```

## Backup and Recovery

### Manual Backup

```bash
# Create backup of current state
terraform state pull > terraform-state-backup-$(date +%Y%m%d-%H%M%S).json

# Store backup securely
mv terraform-state-backup-*.json /path/to/secure/backups/
```

### Restore from Version

```bash
# List versions
gcloud storage ls -a gs://${BUCKET_NAME}/terraform/state/default.tfstate

# Download specific version
gcloud storage cp gs://${BUCKET_NAME}/terraform/state/default.tfstate#VERSION \
  terraform.tfstate.restored

# Push restored state
terraform state push terraform.tfstate.restored
```

### Disaster Recovery

```bash
# Export state to JSON
terraform state pull > state-export.json

# Store in multiple locations
cp state-export.json /backup/location/
cp state-export.json gs://another-bucket/
```

## Cost Optimization

### Storage Costs

- Standard storage: ~$0.023/GB/month in australia-southeast1
- State files are typically < 1MB
- Monthly cost: < $0.10 for typical usage

### Cost Saving Tips

```bash
# Use nearline storage for old versions (cheaper)
cat > lifecycle-optimized.json <<'EOF'
{
  "lifecycle": {
    "rule": [
      {
        "action": {"type": "SetStorageClass", "storageClass": "NEARLINE"},
        "condition": {"age": 30, "isLive": false}
      },
      {
        "action": {"type": "Delete"},
        "condition": {"age": 90, "isLive": false}
      }
    ]
  }
}
EOF

gcloud storage buckets update gs://${BUCKET_NAME} \
  --lifecycle-file=lifecycle-optimized.json
```

## Documentation
- [ ] Update `docs/setup/07-terraform-backend.md`
- [ ] Document bucket name
- [ ] Document state file locations for each environment
- [ ] Add backend configuration examples
- [ ] Document recovery procedures

## Estimated Time
1-2 hours

## Dependencies
- Depends on: #5 (Create GCP project), #6 (Enable required APIs), #7 (Create service account)
- Blocks: All Terraform infrastructure tickets (#19-#30)

## Labels
`phase-0`, `gcp-setup`, `infrastructure`, `terraform`, `storage`

## Resources
- [Terraform GCS Backend](https://developer.hashicorp.com/terraform/language/settings/backends/gcs)
- [GCS Bucket Versioning](https://cloud.google.com/storage/docs/object-versioning)
- [Terraform State Management](https://developer.hashicorp.com/terraform/language/state)
- [GCS Lifecycle Management](https://cloud.google.com/storage/docs/lifecycle)
9|||[GCP-SETUP] Configure VPC network and subnets|||phase-0,gcp-setup,infrastructure,networking|||## Objective
Create VPC network with subnets for Archie Platform V3 in australia-southeast1 region.

## Prerequisites
- #6 (Enable required APIs) completed
- Compute API enabled

## Detailed Steps

### Create VPC Network
```bash
gcloud compute networks create archie-platform-vpc \
  --subnet-mode=custom \
  --bgp-routing-mode=regional

# Create subnet for Cloud Run
gcloud compute networks subnets create cloudrun-subnet \
  --network=archie-platform-vpc \
  --region=australia-southeast1 \
  --range=10.0.0.0/24

# Create subnet for Cloud SQL
gcloud compute networks subnets create cloudsql-subnet \
  --network=archie-platform-vpc \
  --region=australia-southeast1 \
  --range=10.0.1.0/24

# Create subnet for Redis
gcloud compute networks subnets create redis-subnet \
  --network=archie-platform-vpc \
  --region=australia-southeast1 \
  --range=10.0.2.0/24
```

### Configure Firewall Rules
```bash
# Allow internal traffic
gcloud compute firewall-rules create allow-internal \
  --network=archie-platform-vpc \
  --allow=tcp,udp,icmp \
  --source-ranges=10.0.0.0/16

# Allow health checks
gcloud compute firewall-rules create allow-health-checks \
  --network=archie-platform-vpc \
  --allow=tcp \
  --source-ranges=35.191.0.0/16,130.211.0.0/22
```

## Acceptance Criteria
- [ ] VPC created
- [ ] Three subnets created
- [ ] Firewall rules configured
- [ ] Private Google Access enabled

## Testing/Verification
```bash
gcloud compute networks describe archie-platform-vpc
gcloud compute networks subnets list --network=archie-platform-vpc
gcloud compute firewall-rules list --filter="network:archie-platform-vpc"
```

## Documentation
- [ ] Update `docs/setup/08-vpc-networking.md`

## Estimated Time
1-2 hours

## Dependencies
- Depends on: #6 (Enable required APIs)
- Blocks: #24 (Cloud SQL module), #25 (Redis module)

## Resources
- [VPC Networks](https://cloud.google.com/vpc/docs/vpc)
- [Subnets](https://cloud.google.com/vpc/docs/subnets)

10|||[GCP-SETUP] Set up Secret Manager with initial secrets|||phase-0,gcp-setup,infrastructure,security|||## Objective
Configure Secret Manager and create placeholders for application secrets.

## Prerequisites
- #6 (Enable required APIs) completed
- Secret Manager API enabled

## Detailed Steps

### Create Secrets
```bash
# Database password
echo -n "CHANGEME_DB_PASSWORD" | gcloud secrets create db-password \
  --data-file=- \
  --replication-policy="user-managed" \
  --locations=australia-southeast1

# Redis password
echo -n "CHANGEME_REDIS_PASSWORD" | gcloud secrets create redis-password \
  --data-file=- \
  --replication-policy="user-managed" \
  --locations=australia-southeast1

# JWT secret
openssl rand -base64 32 | gcloud secrets create jwt-secret \
  --data-file=- \
  --replication-policy="user-managed" \
  --locations=australia-southeast1

# NATS credentials
echo -n "CHANGEME_NATS_CREDS" | gcloud secrets create nats-credentials \
  --data-file=- \
  --replication-policy="user-managed" \
  --locations=australia-southeast1
```

### Grant Access to Service Accounts
```bash
# Grant Terraform SA access
export SA_EMAIL="terraform-sa@archie-platform-v3.iam.gserviceaccount.com"

for secret in db-password redis-password jwt-secret nats-credentials; do
  gcloud secrets add-iam-policy-binding $secret \
    --member="serviceAccount:$SA_EMAIL" \
    --role="roles/secretmanager.secretAccessor"
done
```

## Acceptance Criteria
- [ ] All secrets created
- [ ] IAM permissions configured
- [ ] Secrets are in australia-southeast1
- [ ] Placeholder values set (to be updated later)

## Testing/Verification
```bash
gcloud secrets list
gcloud secrets versions access latest --secret=jwt-secret
```

## Documentation
- [ ] Update `docs/setup/09-secret-manager.md`
- [ ] Document secret names (not values!)

## Estimated Time
0.5-1 hours

## Dependencies
- Depends on: #6 (Enable required APIs)
- Blocks: #24 (Cloud SQL), #25 (Redis), #26 (NATS)

## Resources
- [Secret Manager](https://cloud.google.com/secret-manager/docs)

11|||[PROJECT-STRUCT] Create test directory structure|||phase-0,project-structure,testing|||## Objective
Set up organized directory structure for tests including unit, integration, and e2e tests.

## Prerequisites
- Project root exists

## Detailed Steps

```bash
cd /mnt/data-disk1/archie-platform-v3

# Create test directories
mkdir -p tests/{unit,integration,e2e,fixtures,utils}
mkdir -p tests/unit/{api,worker,shared}
mkdir -p tests/integration/{api,worker,database}
mkdir -p tests/e2e/{api,frontend}

# Create __init__.py files
touch tests/__init__.py
touch tests/unit/__init__.py
touch tests/integration/__init__.py
touch tests/e2e/__init__.py

# Create conftest.py for pytest fixtures
cat > tests/conftest.py <<'EOF'
import pytest
import asyncio

@pytest.fixture(scope="session")
def event_loop():
    loop = asyncio.get_event_loop_policy().new_event_loop()
    yield loop
    loop.close()
EOF
```

## Acceptance Criteria
- [ ] Test directories created with proper structure
- [ ] __init__.py files in all test directories
- [ ] conftest.py created with base fixtures
- [ ] Directory structure follows pytest conventions

## Testing/Verification

```bash
# Verify structure
tree tests/ -L 3
# Expected: Shows unit, integration, e2e directories

# Verify Python can import
python3 -c "import tests; print('Tests module importable')"

```

## Documentation
- [ ] Update `docs/testing/directory-structure.md`

## Estimated Time
0.5 hours

## Dependencies
- Depends on: None
- Blocks: #15, #16 (Test dependency installation)

## Labels
`phase-0`, `project-structure`, `testing`

## Resources
- [pytest documentation](https://docs.pytest.org)
- [Test organization best practices](https://docs.pytest.org/en/stable/goodpractices.html)

12|||[PROJECT-STRUCT] Create requirements.txt for Python dependencies|||phase-0,project-structure,python,dependencies|||## Objective
Define Python dependencies for API, Worker, and testing.

## Prerequisites
- Python 3.11+ installed

## Detailed Steps

```bash
cd /mnt/data-disk1/archie-platform-v3

# Create requirements.txt
cat > requirements.txt <<'EOF'
# Core FastAPI dependencies
fastapi==0.109.0
uvicorn[standard]==0.27.0
pydantic==2.5.0
pydantic-settings==2.1.0

# Database
asyncpg==0.29.0
sqlalchemy[asyncio]==2.0.25
alembic==1.13.1

# Redis & Caching
redis[hiredis]==5.0.1
aioredis==2.0.1

# NATS messaging
nats-py==2.6.0

# Authentication & Security
python-jose[cryptography]==3.3.0
passlib[bcrypt]==1.7.4
python-multipart==0.0.6

# Google Cloud
google-cloud-storage==2.14.0
google-cloud-secret-manager==2.18.0
google-cloud-logging==3.9.0
google-cloud-error-reporting==1.10.0

# Observability
opentelemetry-api==1.22.0
opentelemetry-sdk==1.22.0
opentelemetry-instrumentation-fastapi==0.43b0
prometheus-client==0.19.0

# HTTP clients
httpx==0.26.0
aiohttp==3.9.1

# Utilities
python-dateutil==2.8.2
pytz==2023.3
structlog==24.1.0

# Development & Testing (separate requirements-dev.txt)
EOF

# Create requirements-dev.txt
cat > requirements-dev.txt <<'EOF'
# Include base requirements
-r requirements.txt

# Testing
pytest==7.4.4
pytest-asyncio==0.23.3
pytest-cov==4.1.0
pytest-mock==3.12.0
pytest-timeout==2.2.0
httpx==0.26.0  # For testing FastAPI

# Playwright for E2E
pytest-playwright==0.4.4

# Code quality
black==23.12.1
isort==5.13.2
flake8==7.0.0
mypy==1.8.0
pylint==3.0.3

# Documentation
mkdocs==1.5.3
mkdocs-material==9.5.3
EOF
```

## Acceptance Criteria
- [ ] requirements.txt created with all production dependencies
- [ ] requirements-dev.txt created with testing tools
- [ ] All versions pinned for reproducibility
- [ ] Dependencies are compatible with Python 3.11+

## Testing/Verification

```bash
# Validate requirements files
python3 -m pip install --dry-run -r requirements.txt
python3 -m pip install --dry-run -r requirements-dev.txt

# Check for conflicts
pip-compile --dry-run requirements.txt 2>&1 | grep -i conflict || echo "No conflicts"

```

## Documentation
- [ ] Update `docs/setup/10-python-dependencies.md`
- [ ] Document why each major dependency is included

## Estimated Time
1 hour

## Dependencies
- Depends on: None
- Blocks: #15 (Install Python dependencies)

## Labels
`phase-0`, `project-structure`, `python`, `dependencies`

## Resources
- [FastAPI dependencies](https://fastapi.tiangolo.com/deployment/dependencies/)
- [pip requirements files](https://pip.pypa.io/en/stable/reference/requirements-file-format/)

13|||[PROJECT-STRUCT] Create pyproject.toml configuration|||phase-0,project-structure,python,configuration|||## Objective
Set up pyproject.toml for Python project configuration, build system, and tool settings.

## Prerequisites
- #12 (requirements.txt) completed

## Detailed Steps

```bash
cd /mnt/data-disk1/archie-platform-v3

cat > pyproject.toml <<'EOF'
[build-system]
requires = ["setuptools>=65.0", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "archie-platform-v3"
version = "0.1.0"
description = "Archie Platform V3 - AI-powered platform with distributed architecture"
readme = "README.md"
requires-python = ">=3.11"
license = {text = "MIT"}
authors = [
    {name = "Archie Platform Team"}
]
keywords = ["fastapi", "nats", "microservices", "ai"]
classifiers = [
    "Development Status :: 3 - Alpha",
    "Intended Audience :: Developers",
    "Programming Language :: Python :: 3.11",
    "Framework :: FastAPI",
]

dependencies = [
    "fastapi>=0.109.0",
    "uvicorn[standard]>=0.27.0",
    # See requirements.txt for full list
]

[project.optional-dependencies]
dev = [
    "pytest>=7.4.4",
    "pytest-asyncio>=0.23.3",
    "pytest-cov>=4.1.0",
    "black>=23.12.1",
    "isort>=5.13.2",
    "mypy>=1.8.0",
]

[tool.pytest.ini_options]
minversion = "7.0"
addopts = "-ra -q --strict-markers --cov=src --cov-report=term-missing"
testpaths = ["tests"]
python_files = "test_*.py"
python_classes = "Test*"
python_functions = "test_*"
asyncio_mode = "auto"
markers = [
    "unit: Unit tests",
    "integration: Integration tests",
    "e2e: End-to-end tests",
    "slow: Slow running tests",
]

[tool.coverage.run]
source = ["src"]
omit = ["tests/*", "**/__pycache__/*"]

[tool.coverage.report]
exclude_lines = [
    "pragma: no cover",
    "def __repr__",
    "raise AssertionError",
    "raise NotImplementedError",
    "if __name__ == .__main__.:",
    "if TYPE_CHECKING:",
]

[tool.black]
line-length = 100
target-version = ["py311"]
include = '\.pyi?$'
extend-exclude = '''
/(
  | \.git
  | \.venv
  | build
  | dist
)/
'''

[tool.isort]
profile = "black"
line_length = 100
multi_line_output = 3
include_trailing_comma = true
force_grid_wrap = 0
use_parentheses = true
ensure_newline_before_comments = true

[tool.mypy]
python_version = "3.11"
warn_return_any = true
warn_unused_configs = true
disallow_untyped_defs = true
plugins = ["pydantic.mypy"]

[[tool.mypy.overrides]]
module = "tests.*"
disallow_untyped_defs = false

[tool.pylint.messages_control]
disable = [
    "C0111",  # missing-docstring
    "C0103",  # invalid-name
]

[tool.pylint.format]
max-line-length = 100
EOF
```

## Acceptance Criteria
- [ ] pyproject.toml created with build configuration
- [ ] pytest configuration included
- [ ] Black and isort configuration set
- [ ] mypy type checking configured
- [ ] Tool configurations follow best practices

## Testing/Verification

```bash
# Validate pyproject.toml
python3 -m pip install build
python3 -m build --wheel --outdir dist/ --no-isolation --dry-run

# Validate tool configs
python3 -c "import tomllib; tomllib.load(open('pyproject.toml', 'rb'))"

# Test pytest configuration
pytest --collect-only tests/ || echo "No tests yet (expected)"

```

## Documentation
- [ ] Update `docs/setup/11-python-configuration.md`

## Estimated Time
1 hour

## Dependencies
- Depends on: #12 (requirements.txt)
- Blocks: #17 (pytest.ini)

## Labels
`phase-0`, `project-structure`, `python`, `configuration`

## Resources
- [pyproject.toml spec](https://peps.python.org/pep-0621/)
- [pytest configuration](https://docs.pytest.org/en/stable/reference/customize.html)
- [Black configuration](https://black.readthedocs.io/en/stable/usage_and_configuration/the_basics.html)

14|||[PROJECT-STRUCT] Create package.json for Node dependencies|||phase-0,project-structure,nodejs,testing|||## Objective
Set up Node.js dependencies for Playwright E2E testing and frontend tooling.

## Prerequisites
- Node.js 18+ installed

## Detailed Steps

```bash
cd /mnt/data-disk1/archie-platform-v3

# Create package.json
cat > package.json <<'EOF'
{
  "name": "archie-platform-v3",
  "version": "0.1.0",
  "description": "Archie Platform V3 - Frontend and E2E testing",
  "private": true,
  "engines": {
    "node": ">=18.0.0",
    "npm": ">=9.0.0"
  },
  "scripts": {
    "test:e2e": "playwright test",
    "test:e2e:headed": "playwright test --headed",
    "test:e2e:debug": "playwright test --debug",
    "test:e2e:report": "playwright show-report",
    "playwright:install": "playwright install --with-deps chromium"
  },
  "devDependencies": {
    "@playwright/test": "^1.40.0",
    "@types/node": "^20.10.0",
    "typescript": "^5.3.0"
  },
  "dependencies": {}
}
EOF

# Create package-lock.json
npm install
```

## Acceptance Criteria
- [ ] package.json created with Playwright
- [ ] npm install runs successfully
- [ ] package-lock.json generated
- [ ] Scripts defined for E2E testing

## Testing/Verification

```bash
# Verify package.json is valid
npm pkg get name
# Expected: archie-platform-v3

# Check Playwright is listed
npm list @playwright/test

# Test script existence
npm run | grep test:e2e

```

## Documentation
- [ ] Update `docs/setup/12-node-dependencies.md`

## Estimated Time
0.5 hours

## Dependencies
- Depends on: None
- Blocks: #16 (Install Node dependencies)

## Labels
`phase-0`, `project-structure`, `nodejs`, `testing`

## Resources
- [npm package.json](https://docs.npmjs.com/cli/v10/configuring-npm/package-json)
- [Playwright](https://playwright.dev/docs/intro)

15|||[PROJECT-STRUCT] Install Python dependencies|||phase-0,project-structure,python,dependencies|||## Objective
Install all Python dependencies in a virtual environment.

## Prerequisites
- #12 (requirements.txt) completed
- Python 3.11+ installed

## Detailed Steps

```bash
cd /mnt/data-disk1/archie-platform-v3

# Create virtual environment
python3 -m venv .venv

# Activate virtual environment
source .venv/bin/activate

# Upgrade pip
pip install --upgrade pip setuptools wheel

# Install production dependencies
pip install -r requirements.txt

# Install development dependencies
pip install -r requirements-dev.txt

# Verify installations
pip list

# Create requirements-lock.txt for reproducibility
pip freeze > requirements-lock.txt
```

### Add to .gitignore
```bash
cat >> .gitignore <<'EOF'

# Virtual environment
.venv/
venv/
ENV/
env/

# Python
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
EOF
```

## Acceptance Criteria
- [ ] Virtual environment created
- [ ] All dependencies installed without errors
- [ ] requirements-lock.txt generated
- [ ] .venv added to .gitignore

## Testing/Verification

```bash
# Activate venv
source .venv/bin/activate

# Check Python version
python --version
# Expected: Python 3.11.x or higher

# Verify key packages
python -c "import fastapi; print(f'FastAPI {fastapi.__version__}')"
python -c "import pytest; print(f'pytest {pytest.__version__}')"
python -c "import sqlalchemy; print(f'SQLAlchemy {sqlalchemy.__version__}')"

# Check all imports work
pip check

```

## Documentation
- [ ] Update `docs/setup/13-python-environment.md`

## Estimated Time
0.5-1 hours

## Dependencies
- Depends on: #12 (requirements.txt)
- Blocks: Testing tickets (#31, #32)

## Labels
`phase-0`, `project-structure`, `python`, `dependencies`

## Resources
- [Python venv](https://docs.python.org/3/library/venv.html)
- [pip freeze](https://pip.pypa.io/en/stable/cli/pip_freeze/)

16|||[PROJECT-STRUCT] Install Node dependencies and Playwright|||phase-0,project-structure,nodejs,testing|||## Objective
Install Node.js dependencies and set up Playwright browsers.

## Prerequisites
- #14 (package.json) completed
- Node.js 18+ installed

## Detailed Steps

```bash
cd /mnt/data-disk1/archie-platform-v3

# Install dependencies
npm install

# Install Playwright browsers
npx playwright install chromium

# Optionally install all browsers
# npx playwright install

# Verify installation
npx playwright --version
```

### Add to .gitignore
```bash
cat >> .gitignore <<'EOF'

# Node
node_modules/
npm-debug.log*
yarn-debug.log*
yarn-error.log*
package-lock.json  # Or keep it - decide based on team preference

# Playwright
playwright-report/
test-results/
EOF
```

## Acceptance Criteria
- [ ] node_modules created
- [ ] Playwright browsers installed
- [ ] No installation errors
- [ ] Playwright executable works

## Testing/Verification

```bash
# Check Node version
node --version
# Expected: v18.x or higher

# Verify npm packages
npm list --depth=0

# Test Playwright
npx playwright --version

# Check browser installation
npx playwright install --dry-run

```

## Documentation
- [ ] Update `docs/setup/14-node-environment.md`

## Estimated Time
0.5 hours

## Dependencies
- Depends on: #14 (package.json)
- Blocks: #33 (Playwright GitHub Action)

## Labels
`phase-0`, `project-structure`, `nodejs`, `testing`

## Resources
- [npm install](https://docs.npmjs.com/cli/v10/commands/npm-install)
- [Playwright installation](https://playwright.dev/docs/intro#installing-playwright)

17|||[PROJECT-STRUCT] Create pytest.ini configuration|||phase-0,project-structure,testing,python|||## Objective
Set up pytest configuration for test discovery and execution.

## Prerequisites
- #15 (Python dependencies) completed

## Detailed Steps

```bash
cd /mnt/data-disk1/archie-platform-v3

cat > pytest.ini <<'EOF'
[pytest]
minversion = 7.0
testpaths = tests
python_files = test_*.py
python_classes = Test*
python_functions = test_*

# Pytest plugins
addopts =
    -ra
    -q
    --strict-markers
    --strict-config
    --cov=src
    --cov-report=term-missing:skip-covered
    --cov-report=html:htmlcov
    --cov-report=xml:coverage.xml
    --tb=short
    --asyncio-mode=auto

# Markers for test categorization
markers =
    unit: Unit tests (fast, isolated)
    integration: Integration tests (database, redis, etc)
    e2e: End-to-end tests (full application)
    slow: Slow running tests
    smoke: Smoke tests for quick validation
    wip: Work in progress tests

# Asyncio configuration
asyncio_mode = auto

# Coverage options
[coverage:run]
source = src
omit =
    tests/*
    **/__pycache__/*
    **/migrations/*
    **/__init__.py

[coverage:report]
precision = 2
show_missing = True
skip_covered = False
exclude_lines =
    pragma: no cover
    def __repr__
    def __str__
    raise AssertionError
    raise NotImplementedError
    if __name__ == .__main__.:
    if TYPE_CHECKING:
    @abstractmethod
    @abc.abstractmethod
EOF
```

## Acceptance Criteria
- [ ] pytest.ini created with comprehensive config
- [ ] Test markers defined
- [ ] Coverage reporting configured
- [ ] Async support enabled

## Testing/Verification

```bash
# Validate pytest configuration
source .venv/bin/activate
pytest --co tests/ || echo "No tests yet (expected)"

# Check markers
pytest --markers | grep "unit\|integration\|e2e"

# Test coverage configuration
pytest --cov-config=pytest.ini --help | grep coverage

```

## Documentation
- [ ] Update `docs/testing/pytest-configuration.md`

## Estimated Time
0.5 hours

## Dependencies
- Depends on: #15 (Python dependencies)
- Blocks: #31 (pytest GitHub Action)

## Labels
`phase-0`, `project-structure`, `testing`, `python`

## Resources
- [pytest.ini](https://docs.pytest.org/en/stable/reference/customize.html)
- [pytest markers](https://docs.pytest.org/en/stable/how-to/mark.html)

18|||[PROJECT-STRUCT] Create playwright.config.ts|||phase-0,project-structure,testing,e2e|||## Objective
Set up Playwright configuration for E2E testing.

## Prerequisites
- #16 (Node dependencies) completed

## Detailed Steps

```bash
cd /mnt/data-disk1/archie-platform-v3

cat > playwright.config.ts <<'EOF'
import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  testDir: './tests/e2e',

  // Maximum time one test can run
  timeout: 30 * 1000,

  // Run tests in parallel
  fullyParallel: true,

  // Fail build on CI if you accidentally left test.only
  forbidOnly: !!process.env.CI,

  // Retry failed tests
  retries: process.env.CI ? 2 : 0,

  // Parallel workers
  workers: process.env.CI ? 1 : undefined,

  // Reporter configuration
  reporter: [
    ['html', { outputFolder: 'playwright-report' }],
    ['json', { outputFile: 'test-results/results.json' }],
    ['junit', { outputFile: 'test-results/junit.xml' }],
  ],

  // Shared test configuration
  use: {
    // Base URL for tests
    baseURL: process.env.BASE_URL || 'http://localhost:8000',

    // Collect trace on failure
    trace: 'on-first-retry',

    // Screenshot on failure
    screenshot: 'only-on-failure',

    // Video on failure
    video: 'retain-on-failure',
  },

  // Project configuration for different browsers
  projects: [
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'] },
    },

    // Uncomment to test on other browsers
    // {
    //   name: 'firefox',
    //   use: { ...devices['Desktop Firefox'] },
    // },
    // {
    //   name: 'webkit',
    //   use: { ...devices['Desktop Safari'] },
    // },
  ],

  // Web server configuration
  webServer: {
    command: 'uvicorn src.api.main:app --host 0.0.0.0 --port 8000',
    url: 'http://localhost:8000/health',
    reuseExistingServer: !process.env.CI,
    timeout: 120 * 1000,
  },
});
EOF
```

## Acceptance Criteria
- [ ] playwright.config.ts created
- [ ] Test directory configured
- [ ] Reporters configured (HTML, JSON, JUnit)
- [ ] Retry and timeout settings configured

## Testing/Verification

```bash
# Validate TypeScript config
npx tsc --noEmit playwright.config.ts || echo "TypeScript validation"

# Test Playwright config
npx playwright test --list || echo "No tests yet (expected)"

# Show configured projects
npx playwright show-config

```

## Documentation
- [ ] Update `docs/testing/playwright-configuration.md`

## Estimated Time
0.5 hours

## Dependencies
- Depends on: #16 (Node dependencies)
- Blocks: #33 (Playwright GitHub Action)

## Labels
`phase-0`, `project-structure`, `testing`, `e2e`

## Resources
- [Playwright config](https://playwright.dev/docs/test-configuration)
- [Playwright test runners](https://playwright.dev/docs/test-runners)

19|||[TERRAFORM] Create Terraform directory structure for modules|||phase-0,terraform,infrastructure|||## Objective
Organize Terraform code into reusable modules with proper structure and documentation.

## Prerequisites
- #4 (Configure Terraform) completed
- #8 (GCS backend) completed

## Detailed Steps

```bash
cd /mnt/data-disk1/archie-platform-v3/terraform

# Create module directories
mkdir -p modules/{project,vpc,iam,secrets,cloudsql,redis,nats,cloudbuild,cloudrun,monitoring}

# Create standard files for each module
for module in modules/*; do
  cat > $module/main.tf <<'EOF'
# Module: $(basename $module)
# Purpose: [To be filled in module-specific ticket]

terraform {
  required_version = ">= 1.6.0"
}
EOF

  cat > $module/variables.tf <<'EOF'
# Input variables for $(basename $module) module
EOF

  cat > $module/outputs.tf <<'EOF'
# Output values from $(basename $module) module
EOF

  cat > $module/README.md <<'EOF'
# $(basename $module | tr '[:lower:]' '[:upper:]') Module

## Purpose
[To be documented]

## Usage
\`\`\`hcl
module "$(basename $module)" {
  source = "./modules/$(basename $module)"
  # variables here
}
\`\`\`

## Inputs
| Name | Description | Type | Default | Required |
|------|-------------|------|---------|----------|

## Outputs
| Name | Description |
|------|-------------|
EOF
done

# Create examples directory
mkdir -p examples/{dev,staging,prod}
```

## Acceptance Criteria
- [ ] All module directories created
- [ ] Standard files (main.tf, variables.tf, outputs.tf, README.md) in each module
- [ ] Examples directory structure created
- [ ] Consistent naming conventions used

## Testing/Verification
```bash
# Verify structure
tree modules/ -L 2

# Check all modules have standard files
for module in modules/*; do
  echo "Checking $module..."
  ls $module/*.tf $module/README.md || echo "Missing files in $module"
done
```

## Documentation
- [ ] Update `docs/terraform/module-structure.md`

## Estimated Time
1 hour

## Dependencies
- Depends on: #4, #8
- Blocks: #20-#30 (All module tickets)

## Resources
- [Terraform Module Structure](https://developer.hashicorp.com/terraform/language/modules/develop/structure)
- [Module Best Practices](https://www.terraform-best-practices.com/code-structure)

20|||[TERRAFORM] Create Cloud SQL (PostgreSQL) Terraform module|||phase-0,terraform,database,infrastructure|||## Objective
Create Terraform module for Cloud SQL PostgreSQL instance with high availability and backup configuration.

## Prerequisites
- #19 (Terraform structure) completed
- #9 (VPC networking) completed
- #10 (Secret Manager) completed

## Detailed Steps

```bash
cd /mnt/data-disk1/archie-platform-v3/terraform/modules/cloudsql

# Create main.tf
cat > main.tf <<'EOF'
# Cloud SQL PostgreSQL Module

resource "google_sql_database_instance" "postgres" {
  name             = var.instance_name
  database_version = var.database_version
  region           = var.region

  settings {
    tier              = var.tier
    availability_type = var.high_availability ? "REGIONAL" : "ZONAL"
    disk_size         = var.disk_size
    disk_type         = "PD_SSD"
    disk_autoresize   = true

    backup_configuration {
      enabled                        = true
      start_time                     = "02:00"
      point_in_time_recovery_enabled = true
      transaction_log_retention_days = 7
      backup_retention_settings {
        retained_backups = 30
        retention_unit   = "COUNT"
      }
    }

    ip_configuration {
      ipv4_enabled    = false
      private_network = var.vpc_id
      require_ssl     = true
    }

    database_flags {
      name  = "max_connections"
      value = "100"
    }

    insights_config {
      query_insights_enabled  = true
      query_string_length     = 1024
      record_application_tags = true
    }
  }

  deletion_protection = var.deletion_protection

  depends_on = [google_service_networking_connection.private_vpc_connection]
}

resource "google_service_networking_connection" "private_vpc_connection" {
  network                 = var.vpc_id
  service                 = "servicenetworking.googleapis.com"
  reserved_peering_ranges = [google_compute_global_address.private_ip_address.name]
}

resource "google_compute_global_address" "private_ip_address" {
  name          = "${var.instance_name}-private-ip"
  purpose       = "VPC_PEERING"
  address_type  = "INTERNAL"
  prefix_length = 16
  network       = var.vpc_id
}

resource "google_sql_database" "database" {
  for_each = toset(var.databases)

  name     = each.value
  instance = google_sql_database_instance.postgres.name
}

resource "google_sql_user" "users" {
  for_each = var.users

  name     = each.key
  instance = google_sql_database_instance.postgres.name
  password = each.value.password
}
EOF

# Create variables.tf
cat > variables.tf <<'EOF'
variable "instance_name" {
  description = "Name of the Cloud SQL instance"
  type        = string
}

variable "database_version" {
  description = "PostgreSQL version"
  type        = string
  default     = "POSTGRES_15"
}

variable "region" {
  description = "GCP region"
  type        = string
}

variable "tier" {
  description = "Machine tier"
  type        = string
  default     = "db-custom-2-7680"
}

variable "disk_size" {
  description = "Disk size in GB"
  type        = number
  default     = 20
}

variable "high_availability" {
  description = "Enable high availability"
  type        = bool
  default     = true
}

variable "vpc_id" {
  description = "VPC network ID"
  type        = string
}

variable "databases" {
  description = "List of databases to create"
  type        = list(string)
  default     = []
}

variable "users" {
  description = "Map of users to create"
  type        = map(object({
    password = string
  }))
  default     = {}
  sensitive   = true
}

variable "deletion_protection" {
  description = "Enable deletion protection"
  type        = bool
  default     = true
}
EOF

# Create outputs.tf
cat > outputs.tf <<'EOF'
output "instance_name" {
  description = "Name of the instance"
  value       = google_sql_database_instance.postgres.name
}

output "connection_name" {
  description = "Connection name for Cloud SQL Proxy"
  value       = google_sql_database_instance.postgres.connection_name
}

output "private_ip_address" {
  description = "Private IP address"
  value       = google_sql_database_instance.postgres.private_ip_address
}

output "instance_id" {
  description = "Instance ID"
  value       = google_sql_database_instance.postgres.id
}
EOF
```

## Acceptance Criteria
- [ ] Cloud SQL module creates PostgreSQL instance
- [ ] High availability configured
- [ ] Automated backups enabled
- [ ] Private IP only (no public IP)
- [ ] SSL required
- [ ] Query insights enabled

## Testing/Verification
```bash
cd terraform/modules/cloudsql
terraform init
terraform validate
terraform fmt -check
```

## Documentation
- [ ] Update `docs/terraform/cloudsql-module.md`

## Estimated Time
2-3 hours

## Dependencies
- Depends on: #19, #9, #10
- Blocks: #30 (Test modules)

## Resources
- [Cloud SQL Terraform](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/sql_database_instance)

21|||[TERRAFORM] Create Cloud Run Terraform module|||phase-0,terraform,serverless,infrastructure|||## Objective
Create Terraform module for Cloud Run services with autoscaling, traffic splitting, and secrets integration.

## Prerequisites
- #19 (Terraform structure) completed
- #9 (VPC networking) completed
- #10 (Secret Manager) completed

## Detailed Steps

```bash
cd /mnt/data-disk1/archie-platform-v3/terraform/modules/cloudrun

cat > main.tf <<'EOF'
# Cloud Run Service Module

resource "google_cloud_run_service" "service" {
  name     = var.service_name
  location = var.region

  template {
    spec {
      containers {
        image = var.image

        ports {
          name           = "http1"
          container_port = var.container_port
        }

        resources {
          limits = {
            cpu    = var.cpu_limit
            memory = var.memory_limit
          }
        }

        dynamic "env" {
          for_each = var.env_vars
          content {
            name  = env.key
            value = env.value
          }
        }

        dynamic "env" {
          for_each = var.secret_env_vars
          content {
            name = env.key
            value_from {
              secret_key_ref {
                name = env.value.secret_name
                key  = env.value.secret_key
              }
            }
          }
        }
      }

      service_account_name = var.service_account_email
      timeout_seconds      = var.timeout_seconds

      container_concurrency = var.max_concurrency
    }

    metadata {
      annotations = merge(
        {
          "autoscaling.knative.dev/minScale" = var.min_instances
          "autoscaling.knative.dev/maxScale" = var.max_instances
          "run.googleapis.com/vpc-access-connector" = var.vpc_connector_name
          "run.googleapis.com/vpc-access-egress" = "all-traffic"
        },
        var.annotations
      )
    }
  }

  traffic {
    percent         = 100
    latest_revision = true
  }

  autogenerate_revision_name = true
}

# IAM binding for public access (if needed)
resource "google_cloud_run_service_iam_member" "public_access" {
  count = var.allow_public_access ? 1 : 0

  service  = google_cloud_run_service.service.name
  location = google_cloud_run_service.service.location
  role     = "roles/run.invoker"
  member   = "allUsers"
}

# IAM binding for service account access
resource "google_cloud_run_service_iam_member" "service_account_access" {
  for_each = toset(var.invoker_service_accounts)

  service  = google_cloud_run_service.service.name
  location = google_cloud_run_service.service.location
  role     = "roles/run.invoker"
  member   = "serviceAccount:${each.value}"
}
EOF

cat > variables.tf <<'EOF'
variable "service_name" {
  description = "Name of the Cloud Run service"
  type        = string
}

variable "region" {
  description = "GCP region"
  type        = string
}

variable "image" {
  description = "Container image URL"
  type        = string
}

variable "container_port" {
  description = "Port the container listens on"
  type        = number
  default     = 8000
}

variable "cpu_limit" {
  description = "CPU limit"
  type        = string
  default     = "1000m"
}

variable "memory_limit" {
  description = "Memory limit"
  type        = string
  default     = "512Mi"
}

variable "min_instances" {
  description = "Minimum number of instances"
  type        = number
  default     = 0
}

variable "max_instances" {
  description = "Maximum number of instances"
  type        = number
  default     = 10
}

variable "max_concurrency" {
  description = "Maximum concurrent requests per instance"
  type        = number
  default     = 80
}

variable "timeout_seconds" {
  description = "Request timeout in seconds"
  type        = number
  default     = 300
}

variable "service_account_email" {
  description = "Service account email for the service"
  type        = string
}

variable "env_vars" {
  description = "Environment variables"
  type        = map(string)
  default     = {}
}

variable "secret_env_vars" {
  description = "Environment variables from secrets"
  type = map(object({
    secret_name = string
    secret_key  = string
  }))
  default = {}
}

variable "vpc_connector_name" {
  description = "VPC connector name"
  type        = string
}

variable "allow_public_access" {
  description = "Allow public access"
  type        = bool
  default     = false
}

variable "invoker_service_accounts" {
  description = "Service accounts that can invoke this service"
  type        = list(string)
  default     = []
}

variable "annotations" {
  description = "Additional annotations"
  type        = map(string)
  default     = {}
}
EOF

cat > outputs.tf <<'EOF'
output "service_name" {
  description = "Name of the service"
  value       = google_cloud_run_service.service.name
}

output "service_url" {
  description = "URL of the service"
  value       = google_cloud_run_service.service.status[0].url
}

output "service_id" {
  description = "Service ID"
  value       = google_cloud_run_service.service.id
}

output "latest_revision_name" {
  description = "Latest revision name"
  value       = google_cloud_run_service.service.status[0].latest_ready_revision_name
}
EOF
```

## Acceptance Criteria
- [ ] Cloud Run module creates service
- [ ] Autoscaling configured
- [ ] VPC connector integrated
- [ ] Secret Manager integration working
- [ ] IAM permissions configurable

## Testing/Verification
```bash
cd terraform/modules/cloudrun
terraform init
terraform validate
```

## Estimated Time
2-3 hours

## Dependencies
- Depends on: #19, #9, #10
- Blocks: #30

## Resources
- [Cloud Run Terraform](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/cloud_run_service)

22|||[CICD] Create pytest GitHub Action workflow|||phase-0,cicd,testing,github-actions|||## Objective
Create GitHub Actions workflow to automatically run pytest on pull requests and pushes.

## Prerequisites
- #17 (pytest.ini) completed
- #15 (Python dependencies) completed

## Detailed Steps

```bash
cd /mnt/data-disk1/archie-platform-v3
mkdir -p .github/workflows

cat > .github/workflows/pytest.yml <<'EOF'
name: Python Tests

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main, develop]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.11", "3.12"]

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          cache: 'pip'

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
          pip install -r requirements-dev.txt

      - name: Run unit tests
        run: |
          pytest tests/unit -v --cov=src --cov-report=xml --cov-report=html -m unit

      - name: Run integration tests
        run: |
          pytest tests/integration -v --cov=src --cov-append --cov-report=xml -m integration
        env:
          # Add environment variables for integration tests
          DATABASE_URL: postgresql://test:test@localhost:5432/test
          REDIS_URL: redis://localhost:6379

      - name: Upload coverage to Codecov
        uses: codecov/codecov-action@v3
        with:
          file: ./coverage.xml
          flags: unittests
          name: codecov-umbrella

      - name: Generate coverage report
        run: |
          coverage report
          coverage html

      - name: Upload coverage HTML
        uses: actions/upload-artifact@v3
        with:
          name: coverage-report
          path: htmlcov/
EOF
```

## Acceptance Criteria
- [ ] Workflow file created
- [ ] Tests run on push and PR
- [ ] Matrix testing for Python 3.11 and 3.12
- [ ] Coverage reporting enabled
- [ ] Codecov integration working

## Testing/Verification
```bash
# Validate workflow syntax
gh workflow view pytest.yml || echo "Workflow not yet in GitHub"

# Test locally with act
act -j test --dry-run
```

## Estimated Time
1-2 hours

## Dependencies
- Depends on: #17, #15
- Blocks: #35 (Test pipeline)

## Resources
- [GitHub Actions](https://docs.github.com/en/actions)
- [pytest-cov](https://pytest-cov.readthedocs.io/)

23|||[CICD] Create Playwright GitHub Action workflow|||phase-0,cicd,testing,e2e,github-actions|||## Objective
Create GitHub Actions workflow for E2E testing with Playwright.

## Prerequisites
- #18 (playwright.config.ts) completed
- #16 (Node dependencies) completed

## Detailed Steps

```bash
cat > .github/workflows/playwright.yml <<'EOF'
name: Playwright E2E Tests

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main, develop]

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '18'
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Install Playwright browsers
        run: npx playwright install --with-deps chromium

      - name: Set up Python (for backend)
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
          cache: 'pip'

      - name: Install Python dependencies
        run: |
          pip install -r requirements.txt

      - name: Run E2E tests
        run: npx playwright test
        env:
          BASE_URL: http://localhost:8000

      - name: Upload test results
        if: always()
        uses: actions/upload-artifact@v3
        with:
          name: playwright-report
          path: playwright-report/
          retention-days: 30

      - name: Upload test videos
        if: failure()
        uses: actions/upload-artifact@v3
        with:
          name: playwright-videos
          path: test-results/
          retention-days: 7
EOF
```

## Acceptance Criteria
- [ ] Playwright workflow created
- [ ] E2E tests run automatically
- [ ] Test reports uploaded
- [ ] Videos captured on failure

## Estimated Time
1-2 hours

## Dependencies
- Depends on: #18, #16
- Blocks: #35

## Resources
- [Playwright CI](https://playwright.dev/docs/ci)

24|||[CICD] Create Terraform validation GitHub Action|||phase-0,cicd,terraform,github-actions|||## Objective
Create GitHub Actions workflow to validate and plan Terraform changes.

## Prerequisites
- #8 (Terraform backend) completed
- #30 (Test modules) completed

## Detailed Steps

```bash
cat > .github/workflows/terraform.yml <<'EOF'
name: Terraform

on:
  push:
    branches: [main, develop]
    paths:
      - 'terraform/**'
  pull_request:
    branches: [main, develop]
    paths:
      - 'terraform/**'

jobs:
  terraform:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.7.0

      - name: Authenticate to Google Cloud
        uses: google-github-actions/auth@v2
        with:
          credentials_json: ${{ secrets.GCP_SA_KEY }}

      - name: Terraform Format Check
        run: |
          terraform fmt -check -recursive terraform/

      - name: Terraform Init
        run: |
          cd terraform
          terraform init

      - name: Terraform Validate
        run: |
          cd terraform
          terraform validate

      - name: Terraform Plan
        run: |
          cd terraform
          terraform plan -out=tfplan

      - name: Upload plan
        uses: actions/upload-artifact@v3
        with:
          name: terraform-plan
          path: terraform/tfplan
EOF
```

## Acceptance Criteria
- [ ] Terraform validation on every PR
- [ ] Format checking enforced
- [ ] Plan generated and uploaded
- [ ] GCP authentication working

## Estimated Time
1-2 hours

## Dependencies
- Depends on: #8, #30
- Blocks: #35

## Resources
- [Terraform GitHub Actions](https://developer.hashicorp.com/terraform/tutorials/automation/github-actions)

25|||[CICD] Create deployment GitHub Action workflow|||phase-0,cicd,deployment,github-actions|||## Objective
Create automated deployment workflow for Cloud Run services.

## Prerequisites
- All previous tickets completed

## Detailed Steps

```bash
cat > .github/workflows/deploy.yml <<'EOF'
name: Deploy to Cloud Run

on:
  push:
    branches: [main]
  workflow_dispatch:

jobs:
  deploy:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Authenticate to Google Cloud
        uses: google-github-actions/auth@v2
        with:
          credentials_json: ${{ secrets.GCP_SA_KEY }}

      - name: Set up Cloud SDK
        uses: google-github-actions/setup-gcloud@v2

      - name: Build and push Docker image
        run: |
          gcloud builds submit --tag gcr.io/${{ secrets.GCP_PROJECT_ID }}/api:${{ github.sha }}

      - name: Deploy to Cloud Run
        run: |
          gcloud run deploy api-service             --image gcr.io/${{ secrets.GCP_PROJECT_ID }}/api:${{ github.sha }}             --region australia-southeast1             --platform managed
EOF
```

## Acceptance Criteria
- [ ] Deployment workflow created
- [ ] Triggers on main branch push
- [ ] Docker image built and pushed
- [ ] Cloud Run service deployed

## Estimated Time
2-3 hours

## Dependencies
- Depends on: All Phase 0 tickets
- Blocks: None (last ticket)

## Resources
- [Cloud Run Deployment](https://cloud.google.com/run/docs/deploying)

26|||[CICD] Test complete CI/CD pipeline end-to-end|||phase-0,cicd,testing,validation|||## Objective
Validate the entire CI/CD pipeline from commit to deployment.

## Prerequisites
- All previous tickets (#1-#34) completed

## Detailed Steps

### 1. Create Test Commit
```bash
# Make a small change
echo "# Test" >> README.md
git add README.md
git commit -m "test: CI/CD pipeline validation"
git push origin main
```

### 2. Verify Workflows
```bash
# Check pytest workflow
gh run list --workflow=pytest.yml

# Check Playwright workflow
gh run list --workflow=playwright.yml

# Check Terraform workflow
gh run list --workflow=terraform.yml

# Check deployment workflow
gh run list --workflow=deploy.yml
```

### 3. Monitor Execution
```bash
# Watch latest run
gh run watch

# View logs
gh run view --log
```

### 4. Verify Deployment
```bash
# Check Cloud Run service
gcloud run services describe api-service   --region=australia-southeast1   --format="value(status.url)"

# Test endpoint
curl $(gcloud run services describe api-service   --region=australia-southeast1   --format="value(status.url)")/health
```

## Acceptance Criteria
- [ ] All workflows execute successfully
- [ ] Tests pass (pytest + Playwright)
- [ ] Terraform validation passes
- [ ] Deployment completes
- [ ] Service is accessible
- [ ] All artifacts uploaded
- [ ] Coverage reports generated

## Testing/Verification
```bash
# Comprehensive pipeline test
./scripts/test-pipeline.sh

# Check all workflow statuses
gh workflow list

# Verify all runs passed
gh run list --limit 10 --json conclusion | jq '.[] | select(.conclusion != "success")'
# Expected: No output (all succeeded)
```

## Documentation
- [ ] Update `docs/ci-cd/pipeline-overview.md`
- [ ] Document pipeline execution times
- [ ] Create troubleshooting guide

## Estimated Time
2-3 hours

## Dependencies
- Depends on: #1-#34 (All previous tickets)
- Blocks: None (Final Phase 0 ticket)

## Resources
- [GitHub Actions Best Practices](https://docs.github.com/en/actions/learn-github-actions/best-practices)
- [Cloud Run CI/CD](https://cloud.google.com/run/docs/continuous-deployment-with-cloud-build)