toon-package

Token Optimizer Example

Complete example demonstrating token optimization using @programsmagic/toon-tokenizer and @programsmagic/toon-converter.

Overview

This example shows how to:

Prerequisites

Step-by-Step Setup

Step 1: Install Dependencies

cd examples/token-optimizer
pnpm install

Step 2: Create Sample Data

Create a file data.json:

{
  "users": [
    {"id": 1, "name": "Alice", "email": "alice@example.com", "active": true},
    {"id": 2, "name": "Bob", "email": "bob@example.com", "active": true}
  ]
}

Step 3: Run Optimization

# Optimize with default model (gpt-4)
node optimize.js data.json

# Optimize with specific model
node optimize.js data.json --model gpt-3.5-turbo

# Optimize and save to file
node optimize.js data.json --output optimized.toon

Complete Code Example

optimize.js

import { selectOptimalFormat, convert } from '@programsmagic/toon-converter';
import { countTokensInData } from '@programsmagic/toon-tokenizer';
import { readFileSync, writeFileSync } from 'fs';

const [inputFile, ...options] = process.argv.slice(2);
const model = options.find(opt => opt.startsWith('--model='))?.split('=')[1] || 'gpt-4';
const outputFile = options.find(opt => opt.startsWith('--output='))?.split('=')[1];

const data = JSON.parse(readFileSync(inputFile, 'utf-8'));

// Select optimal format
const selection = selectOptimalFormat(data);
console.log(`Recommended format: ${selection.best}`);
console.log(`Savings: ${selection.savings[selection.best]}%`);

// Count tokens for each format
const formats = ['json', 'toon', 'csv'];
for (const format of formats) {
  const converted = await convert(JSON.stringify(data), 'json', format);
  const tokens = countTokensInData(converted, model);
  console.log(`${format}: ${tokens.tokens} tokens ($${tokens.estimatedCost.total})`);
}

// Optimize and save
if (outputFile) {
  const optimized = await convert(JSON.stringify(data), 'json', selection.best, {
    minimize: true,
  });
  writeFileSync(outputFile, optimized);
  console.log(`Optimized data saved to ${outputFile}`);
}

Expected Output

Recommended format: toon
Savings: 45%

json: 150 tokens ($0.0015)
toon: 82 tokens ($0.0008)
csv: 95 tokens ($0.0010)

Optimized data saved to optimized.toon

Usage Examples

Basic Optimization

node optimize.js data.json

Model-Specific Optimization

# GPT-3.5 Turbo
node optimize.js data.json --model=gpt-3.5-turbo

# Claude 3 Opus
node optimize.js data.json --model=claude-3-opus

Save Optimized Output

node optimize.js data.json --output=optimized.toon

Troubleshooting

Token Counting Errors

Problem: Error counting tokens.

Solution:

Format Selection Errors

Problem: Error selecting optimal format.

Solution:

Cost Estimation Errors

Problem: Cost estimation fails.

Solution:

Next Steps