TendSocial AI Configuration Guide

Overview

TendSocial uses a sophisticated AI configuration system that allows for flexible model selection, A/B testing, and cost tracking. This guide explains how to use and manage these features.

Architecture

The AI system consists of three main components:

AI Gateway - Abstraction layer for different AI providers
Config Service - Manages model configuration and A/B testing
AI Service - High-level API for content generation

Model Configuration

Global Configuration

Each AI task (e.g., campaign_planning, social_posts, blog_writing) has a global default configuration stored in the AIModelConfig table.

Fields:

provider - AI provider (anthropic, google, openai)
model - Specific model identifier
maxTokens - Maximum tokens per request
temperature - Randomness (0.0-1.0)
inputCostPer1M - Cost per 1M input tokens ($)
outputCostPer1M - Cost per 1M output tokens ($)

Company-Specific Overrides

Companies can override global defaults for specific tasks using CompanyAIConfig. This allows:

Custom model selection per company
BYOK (Bring Your Own Key) for agencies
Custom cost rates for billing

Configuration Priority

The system resolves configuration in this priority order:

Active A/B Test - If user is assigned to a variant
Company Override - Company-specific configuration
Global Default - System-wide default

A/B Testing

Creating an A/B Test

typescript

POST /api/admin/ab-tests
{
  "name": "Opus vs Sonnet for Blogs",
  "description": "Testing quality vs cost",
  "task": "blog_writing",
  "variants": [
    {
      "name": "control",
      "provider": "anthropic",
      "model": "claude-3-5-sonnet-20241022",
      "weight": 50,
      "maxTokens": 4096,
      "temperature": 0.7,
      "inputCostPer1M": 3.0,
      "outputCostPer1M": 15.0
    },
    {
      "name": "challenger",
      "provider": "anthropic",
      "model": "claude-3-opus-20240229",
      "weight": 50,
      "maxTokens": 4096,
      "temperature": 0.7,
      "inputCostPer1M": 15.0,
      "outputCostPer1M": 75.0
    }
  ],
  "targetCompanyIds": [],  // Empty = all companies
  "targetUserIds": [],      // Empty = all users
  "startsAt": "2025-12-01T00:00:00Z",
  "endsAt": "2025-12-31T23:59:59Z"
}

Variant Weights

Weights determine the distribution of users:

Total weight across variants should sum to 100
50/50 = Equal distribution
90/10 = Most users get variant 1
33/33/34 = Three-way split (if using 3 variants)

Assignment Consistency

Once a user is assigned to a variant, they will always get that same variant for the duration of the test. This ensures consistent results.

Targeting

You can target specific companies or users:

typescript

{
  "targetCompanyIds": ["company-123", "company-456"],
  "targetUserIds": []  // Empty = all users in those companies
}

Or target specific users:

typescript

{
  "targetCompanyIds": [],
  "targetUserIds": ["user-789", "user-012"]
}

Activating a Test

typescript

PUT /api/admin/ab-tests/:testId
{
  "isActive": true
}

Viewing Results

typescript

GET /api/admin/ab-tests/:testId/results

Metrics Included:

Total generations per variant
Unique users per variant
Average cost per generation
Average edit ratio (how much users modify AI output)
User ratings (if provided)
Acceptance rate (posts published as-is)
Engagement rates (from social platforms)

Cost Tracking

All AI usage is logged in AIUsageLog with:

Exact token counts (input, output, total)
Costs calculated in cents for precision
Provider and model used
Response time (latency)
Success/failure status
Reference to generated content (if applicable)
A/B test metadata (if applicable)

Cost Calculation

Input Cost = (inputTokens / 1,000,000) × inputCostPer1M × 100 cents
Output Cost = (outputTokens / 1,000,000) × outputCostPer1M × 100 cents
Total Cost = Input Cost + Output Cost

Viewing Analytics

typescript

// Summary across all AI usage
GET /api/admin/ai-analytics/summary?startDate=2025-11-01&endDate=2025-11-30

// Breakdown by dimension
GET /api/admin/ai-analytics/by-dimension?dimension=task&startDate=2025-11-01&endDate=2025-11-30

// Company-specific
GET /api/admin/ai-analytics/company/:companyId?startDate=2025-11-01&endDate=2025-11-30

Gateway Management

The AI gateway supports multiple adapters:

Vercel AI SDK (default) - Unified interface
Direct API - Direct provider calls
LiteLLM (optional) - Proxy for cost tracking

Switching Adapters

typescript

POST /api/admin/gateway/switch
{
  "adapter": "direct-api"
}

Clearing Cache

typescript

POST /api/admin/gateway/clear-cache

This clears both gateway response cache and configuration cache.

Environment Variables

bash

# AI Gateway
AI_GATEWAY_ADAPTER=vercel-ai-sdk  # or direct-api, litellm
AI_CONFIG_CACHE_TTL=60            # seconds
AI_GATEWAY_CACHE_TTL=300          # seconds

# Provider API Keys
ANTHROPIC_API_KEY=sk-ant-...
GOOGLE_AI_API_KEY=...
OPEN AI_API_KEY=sk-...

# Optional: LiteLLM
LITELLM_PROXY_URL=http://localhost:4000

Best Practices

1. Start with Quality

Use the best model (e.g., Claude Opus) initially, then A/B test against cheaper models. Don't compromise quality for cost savings.

2. Test One Variable

When A/B testing, change only one thing:

Same model, different prompts
Same prompt, different models
Same everything, different temperatures

3. Statistical Significance

Run tests with at least 100 users per variant for meaningful results.

4. Monitor Edit Ratios

If edit ratio is consistently high (>50%), the AI output needs improvement. Consider:

Better prompts
More context (brand profile, examples)
Different model
Higher temperature for creativity

5. Cache Appropriately

Config cache: 60 seconds (frequent updates to A/B tests)
Gateway cache: 300 seconds (responses shouldn't change often for same input)

6. Budget Controls

Set budgetPerCall in AIModelConfig to prevent runaway costs:

typescript

{
  "budgetPerCall": 10  // Max 10 cents per call
}

Troubleshooting

Issue: Config changes not taking effect

Solution: Clear the cache

typescript

POST /api/admin/gateway/clear-cache

Issue: Users not being assigned to A/B test

Check:

Is the test isActive: true?
Does the user/company match targeting rules?
Is the date within startsAt and endsAt?
Has the cache been cleared after creating the test?

Issue: Costs higher than expected

Debug:

Check AIUsageLog for actual token counts
Verify inputCostPer1M and outputCostPer1M are correct
Look for failed requests that retry multiple times
Check if prompt is unnecessarily long (trim context)

Issue: Slow performance

Optimize:

Enable gateway caching for identical requests
Use faster models (Haiku vs Sonnet)
Reduce maxTokens if output is too long
Check database indexes on AIUsageLog

The AI Refinement system provides granular usage controls tied to subscription packages.

Action Types

Action	Description	Token Cost
`WEBSITE_ANALYSIS`	Extract brand identity from website	High (~5,000-15,000)
`FIELD_ANALYSIS`	Read-only insights for existing content	Medium (~500-1,000)
`ONE_SHOT_REFINEMENT`	Quick, single-turn field improvements	Low (~200-500)
`CHAT_SESSION_START`	Start a multi-turn chat session	N/A (session tracking)
`CHAT_MESSAGE`	Message within a chat session	Medium (~500-2,000)

Usage Limits by Tier

Feature	Free	Starter	Professional	Enterprise
Website Analysis	1/period	5/period	20/period	Unlimited
Field Analysis	10/period	50/period	200/period	Unlimited
One-Shot Refinement	25/period	100/period	500/period	Unlimited
Chat Sessions	3/period	10/period	50/period	Unlimited
Messages/Session	10	15	25	50

Pre-flight Checks

Before performing an AI action, check entitlements:

typescript

// Frontend
const { checkAction } = useAIEntitlements();
const result = await checkAction('CHAT_SESSION_START');
if (!result.allowed) {
  // Show upgrade modal
}

// Backend
const check = await entitlementsService.checkAIAction(companyId, 'WEBSITE_ANALYSIS');
if (!check.allowed) {
  reply.code(403).send({ error: 'ENTITLEMENT_EXCEEDED' });
}

Rate Limiting

In addition to entitlements, rate limits apply:

Tier	Limit
Free	10 requests/min
Starter	30 requests/min
Professional	100 requests/min
Enterprise	200 requests/min

Rate limit headers are included in responses:

X-RateLimit-Limit: 30
X-RateLimit-Remaining: 25
X-RateLimit-Reset: 1702400000

API Reference

See full API documentation at /api/docs for complete endpoint details.

Admin Features

Overview ​

Architecture ​

Model Configuration ​

Global Configuration ​

Company-Specific Overrides ​

Configuration Priority ​

A/B Testing ​

Creating an A/B Test ​

Variant Weights ​

Assignment Consistency ​

Targeting ​

Activating a Test ​

Viewing Results ​

Cost Tracking ​

Cost Calculation ​

Viewing Analytics ​

Gateway Management ​

Switching Adapters ​

Clearing Cache ​

Environment Variables ​

Best Practices ​

1. Start with Quality ​

2. Test One Variable ​

3. Statistical Significance ​

4. Monitor Edit Ratios ​

5. Cache Appropriately ​

6. Budget Controls ​

Troubleshooting ​

Issue: Config changes not taking effect ​

Issue: Users not being assigned to A/B test ​

Issue: Costs higher than expected ​

Issue: Slow performance ​

AI Refinement Entitlements ​

Action Types ​

Usage Limits by Tier ​

Pre-flight Checks ​

Rate Limiting ​

API Reference ​

Overview

Architecture

Model Configuration

Global Configuration

Company-Specific Overrides

Configuration Priority

A/B Testing

Creating an A/B Test

Variant Weights

Assignment Consistency

Targeting

Activating a Test

Viewing Results

Cost Tracking

Cost Calculation

Viewing Analytics

Gateway Management

Switching Adapters

Clearing Cache

Environment Variables

Best Practices

1. Start with Quality

2. Test One Variable

3. Statistical Significance

4. Monitor Edit Ratios

5. Cache Appropriately

6. Budget Controls

Troubleshooting

Issue: Config changes not taking effect

Issue: Users not being assigned to A/B test

Issue: Costs higher than expected

Issue: Slow performance

AI Refinement Entitlements

Action Types

Usage Limits by Tier

Pre-flight Checks

Rate Limiting

API Reference