TendSocial

Appearance

This document provides detailed procedures for rolling back the Campaign Architecture deployment if critical issues arise.

When to Rollback ​

Execute this rollback plan if:

  1. Critical Functionality Failure

    • Authentication system breaks
    • Content generation completely fails
    • Publishing to social platforms fails
    • Campaign CRUD operations fail

  2. Data Integrity Issues

    • Database corruption detected
    • Data loss in existing content
    • Foreign key constraint violations

  3. Performance Degradation

    • Error rate exceeds 10%
    • Latency increases by >500%
    • AI costs spike unexpectedly (>200% of baseline)

  4. Job Processing Failure

    • Background jobs not executing
    • Analytics sync completely broken
    • Profile analysis jobs failing

Rollback Procedure ​

Phase 1: Immediate Response (0-15 minutes) ​

1. Stop Incoming Traffic (Optional - if severe) ​

bash
# Scale Cloud Run to 0 instances (emergency only)
gcloud run services update tendsocial-api \
  --region us-central1 \
  --max-instances 0

2. Revert Application Code ​

bash
# Identify previous working version
git log --oneline

# Checkout previous stable tag
git checkout v2.0-pre-campaign-arch

# Rebuild and push container
docker build -f apps/backend/Dockerfile -t gcr.io/tendsocial/tendsocial-api:rollback .
docker push gcr.io/tendsocial/tendsocial-api:rollback

# Deploy rollback version
gcloud run deploy tendsocial-api \
  --image gcr.io/tendsocial/tendsocial-api:rollback \
  --region us-central1 \
  --platform managed \
  --allow-unauthenticated

3. Revert Frontend Apps ​

bash
# Dashboard - revert to previous deployment
cd apps/frontend
vercel rollback

# Marketing site - revert to previous deployment
cd apps/marketing
vercel rollback

4. Verify Application Stability ​

bash
# Check health endpoint
curl https://tendsocial-api-xyz.run.app/health

# Check error logs
gcloud logging read "resource.type=cloud_run_revision AND severity>=ERROR" --limit 20

Phase 2: Database Rollback (15-60 minutes) ​

CAUTION

Database rollback is destructive. Only proceed if absolutely necessary and you have a verified backup.

1. Assess Database State ​

sql
-- Check if new tables exist
SELECT tablename FROM pg_tables WHERE schemaname = 'public' ORDER BY tablename;

-- Check for data in new tables
SELECT COUNT(*) FROM "Campaign";
SELECT COUNT(*) FROM "AIUsageLog";
SELECT COUNT(*) FROM "AIModelConfig";

2. Backup Current State (Before Rollback) ​

bash
# Create a backup of current state before rollback
pg_dump $DATABASE_URL > backup_before_rollback_$(date +%Y%m%d_%H%M%S).sql

3. Generate Rollback Migration ​

Prisma does not automatically generate rollback migrations. You must manually reverse the changes.

Manual Rollback SQL:

sql
-- Start transaction
BEGIN;

-- Drop new tables (in reverse dependency order)
DROP TABLE IF EXISTS "AIABAssignment" CASCADE;
DROP TABLE IF EXISTS "AIABTestResults" CASCADE;
DROP TABLE IF EXISTS "AIABTest" CASCADE;
DROP TABLE IF EXISTS "CompanyAIConfig" CASCADE;
DROP TABLE IF EXISTS "AIImageLog" CASCADE;
DROP TABLE IF EXISTS "AIUsageLog" CASCADE;
DROP TABLE IF EXISTS "AIModelConfig" CASCADE;
DROP TABLE IF EXISTS "ContentPerformanceSnapshot" CASCADE;
DROP TABLE IF EXISTS "PerformanceSyncJob" CASCADE;
DROP TABLE IF EXISTS "ProfileAnalysisJob" CASCADE;

-- Drop Campaign table
DROP TABLE IF EXISTS "Campaign" CASCADE;

-- Rollback Post model changes (remove new fields)
ALTER TABLE "Post" DROP COLUMN IF EXISTS "campaignId";
ALTER TABLE "Post" DROP COLUMN IF EXISTS "creationSource";
ALTER TABLE "Post" DROP COLUMN IF EXISTS "originalContent";
ALTER TABLE "Post" DROP COLUMN IF EXISTS "selectedFromOptions";
ALTER TABLE "Post" DROP COLUMN IF EXISTS "totalOptionsOffered";
ALTER TABLE "Post" DROP COLUMN IF EXISTS "regenerationCount";
ALTER TABLE "Post" DROP COLUMN IF EXISTS "regenerationReasons";
ALTER TABLE "Post" DROP COLUMN IF EXISTS "generationContext";
ALTER TABLE "Post" DROP COLUMN IF EXISTS "editMetrics";
ALTER TABLE "Post" DROP COLUMN IF EXISTS "userRating";
ALTER TABLE "Post" DROP COLUMN IF EXISTS "ratingComment";
ALTER TABLE "Post" DROP COLUMN IF EXISTS "markedAsExample";
ALTER TABLE "Post" DROP COLUMN IF EXISTS "performanceMetrics";
ALTER TABLE "Post" DROP COLUMN IF EXISTS "generatedAt";
ALTER TABLE "Post" DROP COLUMN IF EXISTS "editedAt";
ALTER TABLE "Post" DROP COLUMN IF EXISTS "approvedAt";
ALTER TABLE "Post" DROP COLUMN IF EXISTS "publishedAt";

-- Rollback BlogPost model changes
ALTER TABLE "BlogPost" DROP COLUMN IF EXISTS "campaignId";
ALTER TABLE "BlogPost" DROP COLUMN IF EXISTS "creationSource";
ALTER TABLE "BlogPost" DROP COLUMN IF EXISTS "originalMarkdown";
ALTER TABLE "BlogPost" DROP COLUMN IF EXISTS "regenerationCount";
ALTER TABLE "BlogPost" DROP COLUMN IF EXISTS "regenerationReasons";
ALTER TABLE "BlogPost" DROP COLUMN IF EXISTS "generationContext";
ALTER TABLE "BlogPost" DROP COLUMN IF EXISTS "editMetrics";
ALTER TABLE "BlogPost" DROP COLUMN IF EXISTS "userRating";
ALTER TABLE "BlogPost" DROP COLUMN IF EXISTS "ratingComment";
ALTER TABLE "BlogPost" DROP COLUMN IF EXISTS "markedAsExample";
ALTER TABLE "BlogPost" DROP COLUMN IF EXISTS "performanceMetrics";
ALTER TABLE "BlogPost" DROP COLUMN IF EXISTS "generatedAt";
ALTER TABLE "BlogPost" DROP COLUMN IF EXISTS "editedAt";
ALTER TABLE "BlogPost" DROP COLUMN IF EXISTS "publishedAt";

-- Rollback VideoScript model changes
ALTER TABLE "VideoScript" DROP COLUMN IF EXISTS "creationSource";
ALTER TABLE "VideoScript" DROP COLUMN IF EXISTS "originalScript";
ALTER TABLE "VideoScript" DROP COLUMN IF EXISTS "regenerationCount";
ALTER TABLE "VideoScript" DROP COLUMN IF EXISTS "regenerationReasons";
ALTER TABLE "VideoScript" DROP COLUMN IF EXISTS "generationContext";
ALTER TABLE "VideoScript" DROP COLUMN IF EXISTS "editMetrics";
ALTER TABLE "VideoScript" DROP COLUMN IF EXISTS "userRating";
ALTER TABLE "VideoScript" DROP COLUMN IF EXISTS "ratingComment";
ALTER TABLE "VideoScript" DROP COLUMN IF EXISTS "markedAsExample";
ALTER TABLE "VideoScript" DROP COLUMN IF EXISTS "generatedAt";
ALTER TABLE "VideoScript" DROP COLUMN IF EXISTS "editedAt";

-- Rollback ScheduledPost changes
ALTER TABLE "ScheduledPost" DROP COLUMN IF EXISTS "campaignId";

-- Rollback User model changes
ALTER TABLE "User" DROP COLUMN IF EXISTS "collaborationProfile";
ALTER TABLE "User" DROP COLUMN IF EXISTS "exampleContent";
ALTER TABLE "User" DROP COLUMN IF EXISTS "profileAnalyzedAt";

-- Commit transaction
COMMIT;

4. Apply Rollback Migration ​

bash
# Connect to production database
psql $DATABASE_URL

# Run the rollback SQL from file
\i rollback_migration.sql

# Verify rollback
\dt

5. Update Prisma Schema ​

bash
# Checkout previous schema version
git checkout v2.0-pre-campaign-arch -- apps/backend/prisma/schema.prisma

# Regenerate Prisma client
cd apps/backend
pnpm generate

# Push to match database state
pnpm db:push

Phase 3: Verification & Recovery (60-120 minutes) ​

1. Verify Core Functionality ​

  • [ ] Authentication works
  • [ ] User can create posts
  • [ ] User can publish to social platforms
  • [ ] Brand profiles load correctly
  • [ ] Video scripts accessible

2. Data Integrity Check ​

sql
-- Verify existing posts not corrupted
SELECT COUNT(*) FROM "Post";
SELECT COUNT(*) FROM "BlogPost";
SELECT COUNT(*) FROM "VideoScript";
SELECT COUNT(*) FROM "User";
SELECT COUNT(*) FROM "Company";

-- Check for orphaned records (should be none)
SELECT COUNT(*) FROM "Post" WHERE "userId" NOT IN (SELECT id FROM "User");

3. Monitor Error Rates ​

bash
# Check for errors after rollback
gcloud logging read "resource.type=cloud_run_revision AND severity>=ERROR" --limit 50

4. Test Critical Workflows ​

  • [ ] Login flow
  • [ ] Create new post
  • [ ] Schedule post
  • [ ] Publish post
  • [ ] View analytics

Phase 4: Post-Rollback Actions ​

1. Notify Team ​

  • [ ] Notify engineering team of rollback
  • [ ] Notify support team (expect user questions)
  • [ ] Notify stakeholders

2. Incident Analysis ​

  • [ ] Document what went wrong
  • [ ] Identify root cause
  • [ ] Create action items for fixes
  • [ ] Update deployment checklist with lessons learned

3. Plan Re-Deployment ​

  • [ ] Fix identified issues
  • [ ] Test fixes in development
  • [ ] Test on staging environment
  • [ ] Schedule new deployment

Rollback Testing (Pre-Production) ​

Before deploying to production, test the rollback procedure on a staging environment:

bash
# 1. Deploy to staging
# 2. Apply migrations
# 3. Create test data in new tables
# 4. Execute rollback procedure
# 5. Verify data integrity
# 6. Verify application functionality

Emergency Contacts ​

RoleContactAvailability
Lead Engineer___________24/7
DevOps___________Business hours
Database Admin___________On-call
Product Owner___________Business hours

Known Limitations ​

  1. Data Loss: Any data created in Campaign, AIUsageLog, or AIModelConfig tables will be lost during database rollback.
  2. User Experience: Users will lose access to campaign features during rollback period.
  3. Analytics Gap: AI usage analytics will have a gap for the deployment period.

Last Updated: 2025-11-30
Version: 1.0
Owner: Engineering Team

TendSocial Documentation

Copyright © 2024-2025 TendSocial LLC