Overview

TendSocial uses a sophisticated AI configuration system that allows for flexible model selection, A/B testing, and cost tracking. This guide explains how to use and manage these features.

Architecture

The AI system consists of three main components:

1. AI Gateway - Abstraction layer for different AI providers
2. Config Service - Manages model configuration and A/B testing
3. AI Service - High-level API for content generation

Model Configuration

Global Configuration

Each AI task (e.g., campaign_planning, social_posts, blog_writing) has a global default configuration stored in the AIModelConfig table.

Fields:

• provider - AI provider (anthropic, google, openai)
• model - Specific model identifier
• maxTokens - Maximum tokens per request
• temperature - Randomness (0.0-1.0)
• inputCostPer1M - Cost per 1M input tokens ($)
• outputCostPer1M - Cost per 1M output tokens ($)

Company-Specific Overrides

Companies can override global defaults for specific tasks using CompanyAIConfig. This allows:

• Custom model selection per company
• BYOK (Bring Your Own Key) for agencies
• Custom cost rates for billing

Configuration Priority

The system resolves configuration in this priority order:

1. Active A/B Test - If user is assigned to a variant
2. Company Override - Company-specific configuration
3. Global Default - System-wide default

A/B Testing

Creating an A/B Test

POST /api/admin/ab-tests
{
  "name": "Opus vs Sonnet for Blogs",
  "description": "Testing quality vs cost",
  "task": "blog_writing",
  "variants": [
    {
      "name": "control",
      "provider": "anthropic",
      "model": "claude-3-5-sonnet-20241022",
      "weight": 50,
      "maxTokens": 4096,
      "temperature": 0.7,
      "inputCostPer1M": 3.0,
      "outputCostPer1M": 15.0
    },
    {
      "name": "challenger",
      "provider": "anthropic",
      "model": "claude-3-opus-20240229",
      "weight": 50,
      "maxTokens": 4096,
      "temperature": 0.7,
      "inputCostPer1M": 15.0,
      "outputCostPer1M": 75.0
    }
  ],
  "targetCompanyIds": [],  // Empty = all companies
  "targetUserIds": [],      // Empty = all users
  "startsAt": "2025-12-01T00:00:00Z",
  "endsAt": "2025-12-31T23:59:59Z"
}

Variant Weights

Weights determine the distribution of users:

• Total weight across variants should sum to 100
• 50/50 = Equal distribution
• 90/10 = Most users get variant 1
• 33/33/34 = Three-way split (if using 3 variants)

Assignment Consistency

Once a user is assigned to a variant, they will always get that same variant for the duration of the test. This ensures consistent results.

Targeting

You can target specific companies or users:

{
  "targetCompanyIds": ["company-123", "company-456"],
  "targetUserIds": []  // Empty = all users in those companies
}

Or target specific users:

{
  "targetCompanyIds": [],
  "targetUserIds": ["user-789", "user-012"]
}

Activating a Test

PUT /api/admin/ab-tests/:testId
{
  "isActive": true
}

Viewing Results

GET /api/admin/ab-tests/:testId/results

Metrics Included:

• Total generations per variant
• Unique users per variant
• Average cost per generation
• Average edit ratio (how much users modify AI output)
• User ratings (if provided)
• Acceptance rate (posts published as-is)
• Engagement rates (from social platforms)

Cost Tracking

All AI usage is logged in AIUsageLog with:

• Exact token counts (input, output, total)
• Costs calculated in cents for precision
• Provider and model used
• Response time (latency)
• Success/failure status
• Reference to generated content (if applicable)
• A/B test metadata (if applicable)

Cost Calculation

Input Cost = (inputTokens / 1,000,000) × inputCostPer1M × 100 cents
Output Cost = (outputTokens / 1,000,000) × outputCostPer1M × 100 cents
Total Cost = Input Cost + Output Cost

Viewing Analytics

// Summary across all AI usage
GET /api/admin/ai-analytics/summary?startDate=2025-11-01&endDate=2025-11-30

// Breakdown by dimension
GET /api/admin/ai-analytics/by-dimension?dimension=task&startDate=2025-11-01&endDate=2025-11-30

// Company-specific
GET /api/admin/ai-analytics/company/:companyId?startDate=2025-11-01&endDate=2025-11-30

Gateway Management

The AI gateway supports multiple adapters:

1. Vercel AI SDK (default) - Unified interface
2. Direct API - Direct provider calls
3. LiteLLM (optional) - Proxy for cost tracking

Switching Adapters

POST /api/admin/gateway/switch
{
  "adapter": "direct-api"
}

Clearing Cache

POST /api/admin/gateway/clear-cache

This clears both gateway response cache and configuration cache.

Environment Variables

# AI Gateway
AI_GATEWAY_ADAPTER=vercel-ai-sdk  # or direct-api, litellm
AI_CONFIG_CACHE_TTL=60            # seconds
AI_GATEWAY_CACHE_TTL=300          # seconds

# Provider API Keys
ANTHROPIC_API_KEY=sk-ant-...
GOOGLE_AI_API_KEY=...
OPEN AI_API_KEY=sk-...

# Optional: LiteLLM
LITELLM_PROXY_URL=http://localhost:4000

Best Practices

1. Start with Quality

Use the best model (e.g., Claude Opus) initially, then A/B test against cheaper models. Don't compromise quality for cost savings.

2. Test One Variable

When A/B testing, change only one thing:

• Same model, different prompts
• Same prompt, different models
• Same everything, different temperatures

3. Statistical Significance

Run tests with at least 100 users per variant for meaningful results.

4. Monitor Edit Ratios

If edit ratio is consistently high (>50%), the AI output needs improvement. Consider:

• Better prompts
• More context (brand profile, examples)
• Different model
• Higher temperature for creativity

5. Cache Appropriately

• Config cache: 60 seconds (frequent updates to A/B tests)
• Gateway cache: 300 seconds (responses shouldn't change often for same input)

6. Budget Controls

Set budgetPerCall in AIModelConfig to prevent runaway costs:

{
  "budgetPerCall": 10  // Max 10 cents per call
}

Troubleshooting

Issue: Config changes not taking effect

Solution: Clear the cache

POST /api/admin/gateway/clear-cache

Issue: Users not being assigned to A/B test

Check:

1. Is the test isActive: true?
2. Does the user/company match targeting rules?
3. Is the date within startsAt and endsAt?
4. Has the cache been cleared after creating the test?

Issue: Costs higher than expected

Debug:

1. Check AIUsageLog for actual token counts
2. Verify inputCostPer1M and outputCostPer1M are correct
3. Look for failed requests that retry multiple times
4. Check if prompt is unnecessarily long (trim context)

Issue: Slow performance

Optimize:

1. Enable gateway caching for identical requests
2. Use faster models (Haiku vs Sonnet)
3. Reduce maxTokens if output is too long
4. Check database indexes on AIUsageLog

AI Refinement Entitlements

The AI Refinement system provides granular usage controls tied to subscription packages.

Action Types

Action	Description	Token Cost
WEBSITE_ANALYSIS	Extract brand identity from website	High (~5,000-15,000)
FIELD_ANALYSIS	Read-only insights for existing content	Medium (~500-1,000)
ONE_SHOT_REFINEMENT	Quick, single-turn field improvements	Low (~200-500)
CHAT_SESSION_START	Start a multi-turn chat session	N/A (session tracking)
CHAT_MESSAGE	Message within a chat session	Medium (~500-2,000)

Usage Limits by Tier

Feature	Free	Starter	Professional	Enterprise
Website Analysis	1/period	5/period	20/period	Unlimited
Field Analysis	10/period	50/period	200/period	Unlimited
One-Shot Refinement	25/period	100/period	500/period	Unlimited
Chat Sessions	3/period	10/period	50/period	Unlimited
Messages/Session	10	15	25	50

Pre-flight Checks

Before performing an AI action, check entitlements:

// Frontend
const { checkAction } = useAIEntitlements();
const result = await checkAction('CHAT_SESSION_START');
if (!result.allowed) {
  // Show upgrade modal
}

// Backend
const check = await entitlementsService.checkAIAction(companyId, 'WEBSITE_ANALYSIS');
if (!check.allowed) {
  reply.code(403).send({ error: 'ENTITLEMENT_EXCEEDED' });
}

Rate Limiting

In addition to entitlements, rate limits apply:

Tier	Limit
Free	10 requests/min
Starter	30 requests/min
Professional	100 requests/min
Enterprise	200 requests/min

Rate limit headers are included in responses:

X-RateLimit-Limit: 30
X-RateLimit-Remaining: 25
X-RateLimit-Reset: 1702400000

API Reference

See full API documentation at /api/docs for complete endpoint details.