Structured Outputs in LLMs — Getting Reliable JSON from AI

Every developer who’s integrated LLMs has faced this problem: you ask for JSON, but the model returns JSON wrapped in markdown code blocks, with missing fields, or sometimes just plain text. Parsing feels like a game of whack-a-mole.

Structured outputs solve this completely. Instead of hoping the model formats correctly, you define a schema and the model is guaranteed to match it. This guide covers how to implement reliable JSON output with OpenAI, Anthropic, and Google models.

The Problem: Unpredictable LLM Output

Classic JSON Parsing Nightmares

# The naive approach
response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Return JSON with name, age, and email"}]
)

# Model returns:
# "```json\n{\"name\": \"John\", \"age\": 30, \"email\": \"john@example.com\"}\n```"

# Or maybe:
# "Sure! Here's the JSON: {name: John, age: 30}"  # Missing quotes!

# Or even:
# "I'd be happy to provide that information..."

Every variation requires different parsing:

import json
import re

def parse_json_response(text: str) -> dict:
    # Try direct parse
    try:
        return json.loads(text)
    except json.JSONDecodeError:
        pass
    
    # Try extracting from code blocks
    code_block_match = re.search(r'```(?:json)?\s*([\s\S]*?)\s*```', text)
    if code_block_match:
        try:
            return json.loads(code_block_match.group(1))
        except json.JSONDecodeError:
            pass
    
    # Try finding JSON object in text
    json_match = re.search(r'\{[\s\S]*\}', text)
    if json_match:
        try:
            return json.loads(json_match.group(0))
        except json.JSONDecodeError:
            pass
    
    raise ValueError("Could not parse JSON from response")

This is fragile, error-prone, and wastes tokens on the model’s preamble.

The Real Impact

For production systems, unpredictable output means:

1. Retry loops — Parse failure? Try again, burn more tokens
2. Silent data loss — Optional fields silently dropped
3. Type errors — Model returns "30" instead of 30
4. Validation overhead — Complex post-processing logic

Solution 1: JSON Mode (OpenAI)

OpenAI’s JSON mode constrains output to valid JSON:

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": "You are a helpful assistant that returns JSON."},
        {"role": "user", "content": "Generate a user profile with name, age, and interests."}
    ],
    response_format={"type": "json_object"}
)

# Guaranteed valid JSON
data = json.loads(response.choices[0].message.content)

Limitation: JSON mode only guarantees valid JSON syntax, not that the JSON matches your schema.

# You asked for:
{"name": str, "age": int, "interests": list[str]}

# Model might return:
{"full_name": "John", "user_age": "30", "hobbies": ["coding"]}  # Different keys!

Solution 2: Structured Outputs with JSON Schema (OpenAI)

OpenAI’s structured outputs provide schema enforcement:

from openai import OpenAI
from pydantic import BaseModel
from typing import List

client = OpenAI()

class UserProfile(BaseModel):
    name: str
    age: int
    email: str
    interests: List[str]

response = client.chat.completions.create(
    model="gpt-4o-2024-08-06",
    messages=[
        {"role": "user", "content": "Generate a profile for a software developer."}
    ],
    response_format=UserProfile
)

# Guaranteed to match UserProfile schema
profile = UserProfile.model_validate_json(response.choices[0].message.content)
print(profile.name)  # Type-safe access

Complex Nested Schemas

from pydantic import BaseModel
from typing import List, Optional
from enum import Enum

class SkillLevel(str, Enum):
    BEGINNER = "beginner"
    INTERMEDIATE = "intermediate"
    EXPERT = "expert"

class Skill(BaseModel):
    name: str
    level: SkillLevel
    years_of_experience: int

class WorkExperience(BaseModel):
    company: str
    role: str
    duration_months: int
    description: str

class DeveloperProfile(BaseModel):
    name: str
    email: str
    skills: List[Skill]
    experience: List[WorkExperience]
    github_url: Optional[str] = None
    available_for_hire: bool

response = client.chat.completions.create(
    model="gpt-4o-2024-08-06",
    messages=[
        {"role": "user", "content": "Generate a senior developer profile."}
    ],
    response_format=DeveloperProfile
)

profile = DeveloperProfile.model_validate_json(response.choices[0].message.content)

Using Raw JSON Schema

import json

schema = {
    "type": "object",
    "properties": {
        "name": {"type": "string"},
        "age": {"type": "integer", "minimum": 0, "maximum": 150},
        "email": {"type": "string", "format": "email"},
        "interests": {
            "type": "array",
            "items": {"type": "string"},
            "minItems": 1,
            "maxItems": 10
        }
    },
    "required": ["name", "age", "email", "interests"],
    "additionalProperties": False
}

response = client.chat.completions.create(
    model="gpt-4o-2024-08-06",
    messages=[
        {"role": "user", "content": "Generate a user profile."}
    ],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "user_profile",
            "strict": True,
            "schema": schema
        }
    }
)

The strict: True flag ensures the model’s output exactly matches your schema.

Solution 3: Tool Calling for Structured Output

Tool calling (function calling) provides structured output through a different mechanism:

from openai import OpenAI

client = OpenAI()

tools = [
    {
        "type": "function",
        "function": {
            "name": "save_user_profile",
            "description": "Save a user profile to the database",
            "parameters": {
                "type": "object",
                "properties": {
                    "name": {"type": "string", "description": "User's full name"},
                    "age": {"type": "integer", "description": "User's age"},
                    "email": {"type": "string", "description": "User's email address"},
                    "interests": {
                        "type": "array",
                        "items": {"type": "string"},
                        "description": "List of user's interests"
                    }
                },
                "required": ["name", "age", "email", "interests"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "user", "content": "Create a profile for a Python developer named Sarah."}
    ],
    tools=tools,
    tool_choice={"type": "function", "function": {"name": "save_user_profile"}}
)

# Extract structured data from tool call
tool_call = response.choices[0].message.tool_calls[0]
args = json.loads(tool_call.function.arguments)
# args is guaranteed to match the schema

Tool Calling vs Structured Outputs

Feature	Tool Calling	Structured Outputs
Use case	Actions with structured params	Pure data extraction
Multiple outputs	Multiple tool calls	Single schema
Strictness	Model-inferred	Guaranteed strict
Best for	Agents, multi-step workflows	Extraction, transformation

For more on building agents with tools, see my guide on AI agent architecture patterns.

Solution 4: Anthropic’s Tool Use

Anthropic uses tool use for structured output:

import anthropic

client = anthropic.Anthropic()

tools = [
    {
        "name": "extract_user_profile",
        "description": "Extract user profile information",
        "input_schema": {
            "type": "object",
            "properties": {
                "name": {"type": "string"},
                "age": {"type": "integer"},
                "email": {"type": "string"},
                "interests": {
                    "type": "array",
                    "items": {"type": "string"}
                }
            },
            "required": ["name", "age", "email", "interests"]
        }
    }
]

response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1024,
    messages=[
        {"role": "user", "content": "Extract profile info: John Doe, 28 years old, john@example.com, likes Python and AI"}
    ],
    tools=tools
)

# Find the tool use block
for block in response.content:
    if block.type == "tool_use":
        profile = block.input  # Guaranteed to match schema
        print(profile)

Forced Tool Use

response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1024,
    messages=[
        {"role": "user", "content": "Generate a random user profile"}
    ],
    tools=tools,
    tool_choice={"type": "tool", "name": "extract_user_profile"}  # Force specific tool
)

Solution 5: Google Gemini’s Controlled Generation

Google’s Gemini offers JSON schema enforcement:

from google import genai
from pydantic import BaseModel
from typing import List

client = genai.Client()

class UserProfile(BaseModel):
    name: str
    age: int
    email: str
    interests: List[str]

response = client.models.generate_content(
    model="gemini-1.5-pro",
    contents="Generate a developer profile",
    config={
        "response_mime_type": "application/json",
        "response_schema": UserProfile
    }
)

profile = UserProfile.model_validate_json(response.text)

Schema with Enums

from enum import Enum

class Role(str, Enum):
    FRONTEND = "frontend"
    BACKEND = "backend"
    FULLSTACK = "fullstack"
    DEVOPS = "devops"

class DeveloperProfile(BaseModel):
    name: str
    role: Role
    years_experience: int
    skills: List[str]

response = client.models.generate_content(
    model="gemini-1.5-pro",
    contents="Generate a developer profile for a senior engineer",
    config={
        "response_mime_type": "application/json",
        "response_schema": DeveloperProfile
    }
)

TypeScript Implementation

For TypeScript/JavaScript projects:

import OpenAI from 'openai';
import { z } from 'zod';
import { zodResponseFormat } from 'openai/helpers/zod';

const client = new OpenAI();

const UserProfileSchema = z.object({
  name: z.string(),
  age: z.number().int().min(0).max(150),
  email: z.string().email(),
  interests: z.array(z.string()).min(1).max(10)
});

type UserProfile = z.infer<typeof UserProfileSchema>;

const response = await client.chat.completions.create({
  model: 'gpt-4o-2024-08-06',
  messages: [
    { role: 'user', content: 'Generate a user profile for a software developer' }
  ],
  response_format: zodResponseFormat(UserProfileSchema, 'user_profile')
});

const profile: UserProfile = UserProfileSchema.parse(
  JSON.parse(response.choices[0].message.content || '{}')
);

Complex TypeScript Example

import { z } from 'zod';

const AddressSchema = z.object({
  street: z.string(),
  city: z.string(),
  country: z.string(),
  postalCode: z.string()
});

const PersonSchema = z.object({
  name: z.string(),
  age: z.number(),
  email: z.string().email(),
  address: AddressSchema,
  tags: z.array(z.string()),
  metadata: z.record(z.unknown()).optional()
});

type Person = z.infer<typeof PersonSchema>;

async function generatePerson(description: string): Promise<Person> {
  const response = await client.chat.completions.create({
    model: 'gpt-4o-2024-08-06',
    messages: [
      { role: 'user', content: `Generate a person profile: ${description}` }
    ],
    response_format: zodResponseFormat(PersonSchema, 'person')
  });
  
  return PersonSchema.parse(JSON.parse(response.choices[0].message.content || '{}'));
}

Best Practices

1. Use Strict Schemas

# GOOD: Strict schema with no additional properties
schema = {
    "type": "object",
    "properties": {...},
    "required": [...],
    "additionalProperties": False  # Prevent unexpected fields
}

# BAD: Loose schema allows anything
schema = {
    "type": "object",
    "properties": {...}
}

2. Provide Clear Descriptions

class UserProfile(BaseModel):
    name: str = Field(description="User's full name (first and last)")
    age: int = Field(description="User's age in years (0-150)", ge=0, le=150)
    email: str = Field(description="Valid email address")

Descriptions help the model understand expectations.

3. Use Enums for Fixed Values

class Priority(str, Enum):
    LOW = "low"
    MEDIUM = "medium"
    HIGH = "high"
    URGENT = "urgent"

class Task(BaseModel):
    title: str
    priority: Priority  # Model must choose from these values

4. Handle Optional Fields Explicitly

from typing import Optional

class UserProfile(BaseModel):
    name: str
    age: int
    bio: Optional[str] = None  # Explicitly optional
    website: Optional[str] = Field(default=None, description="Personal website URL")

5. Validate After Extraction

Even with structured outputs, validate business logic:

from pydantic import field_validator

class UserProfile(BaseModel):
    name: str
    email: str
    age: int
    
    @field_validator('email')
    @classmethod
    def validate_email(cls, v: str) -> str:
        if '@' not in v:
            raise ValueError('Invalid email format')
        return v.lower()
    
    @field_validator('age')
    @classmethod
    def validate_age(cls, v: int) -> int:
        if v < 0 or v > 150:
            raise ValueError('Age must be between 0 and 150')
        return v

Performance Considerations

Token Overhead

Structured outputs add overhead to the prompt:

Base prompt: 100 tokens
With schema: +50-200 tokens (depending on schema complexity)

For cost-sensitive applications, use minimal schemas:

# Minimal
class User(BaseModel):
    name: str
    age: int

# Verbose
class User(BaseModel):
    name: str = Field(description="The user's full legal name...")
    age: int = Field(description="The user's age in years at the time of registration...")

Caching Structured Outputs

Combine with prompt caching for repeated schemas:

response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1024,
    messages=[{
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": large_schema_context,
                "cache_control": {"type": "ephemeral"}
            },
            {
                "type": "text",
                "text": user_query
            }
        ]
    }],
    tools=[schema_tool]
)

Learn more about cost optimization in my guide on prompt caching for LLM cost reduction.

Common Pitfalls

1. Schema Too Complex

# TOO COMPLEX: Nested unions, many optionals
class Response(BaseModel):
    data: Optional[Union[User, Order, Product, None]]
    metadata: Optional[Dict[str, Union[str, int, List[str]]]]

# BETTER: Clear, focused schema
class UserResponse(BaseModel):
    user: User
    source: str

2. Missing Descriptions

# WITHOUT descriptions, model guesses intent
class Task(BaseModel):
    title: str
    priority: int  # 1-5? 1-10? What's the scale?

# WITH descriptions, model understands
class Task(BaseModel):
    title: str
    priority: int = Field(description="Priority from 1 (lowest) to 5 (highest)")

3. Conflicting Constraints

# Conflicting: min > max
class Config(BaseModel):
    port: int = Field(ge=100, le=50)  # Impossible to satisfy

# Fixed
class Config(BaseModel):
    port: int = Field(ge=1024, le=65535)

Real-World Example: Document Extraction

from pydantic import BaseModel, Field
from typing import List, Optional
from enum import Enum
import anthropic

class DocumentType(str, Enum):
    INVOICE = "invoice"
    RECEIPT = "receipt"
    CONTRACT = "contract"
    REPORT = "report"

class Entity(BaseModel):
    name: str
    type: str = Field(description="Type of entity (person, company, organization)")
    role: Optional[str] = Field(default=None, description="Role in the document")

class MonetaryAmount(BaseModel):
    amount: float
    currency: str = Field(default="USD", description="ISO currency code")

class DocumentExtraction(BaseModel):
    document_type: DocumentType
    title: str
    date: str = Field(description="Document date in YYYY-MM-DD format")
    parties: List[Entity]
    total_amount: Optional[MonetaryAmount] = None
    summary: str = Field(description="Brief summary of document content")
    key_terms: List[str] = Field(description="Key terms or clauses identified")

client = anthropic.Anthropic()

tools = [{
    "name": "extract_document",
    "description": "Extract structured information from a document",
    "input_schema": DocumentExtraction.model_json_schema()
}]

def extract_from_text(document_text: str) -> DocumentExtraction:
    response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=2048,
        messages=[
            {"role": "user", "content": f"Extract information from this document:\n\n{document_text}"}
        ],
        tools=tools,
        tool_choice={"type": "tool", "name": "extract_document"}
    )
    
    for block in response.content:
        if block.type == "tool_use":
            return DocumentExtraction.model_validate(block.input)
    
    raise ValueError("No tool use in response")

# Usage
doc_text = """
INVOICE #12345
Date: January 15, 2026

From: TechCorp Inc.
To: StartupXYZ LLC

Software Development Services
- Backend API Development: $15,000
- Testing & QA: $3,000
- Deployment & Documentation: $2,000

Total: $20,000 USD
"""

extraction = extract_from_text(doc_text)
print(f"Type: {extraction.document_type}")
print(f"Total: {extraction.total_amount}")
print(f"Parties: {[p.name for p in extraction.parties]}")

Comparison Summary

Provider	Feature	Method	Strictness
OpenAI	JSON mode	response_format={"type": "json_object"}	Valid JSON only
OpenAI	Structured output	response_format with schema	Guaranteed match
Anthropic	Tool use	tools with schema	Constrained by schema
Google	Controlled gen	response_schema	Guaranteed match

Conclusion

Structured outputs transform LLM integration from fragile string parsing to reliable type-safe code. By defining schemas upfront, you get:

1. Guaranteed format compliance — No more parsing edge cases
2. Type safety — IDE autocomplete and compile-time checks
3. Clear contracts — Schema serves as documentation
4. Better reliability — Reduced retry loops

Key recommendations:

• Use OpenAI’s structured outputs with strict: True for maximum reliability
• Use tool calling when structured output is part of an action
• Always validate business logic beyond schema matching
• Combine with prompt caching for cost efficiency on repeated schemas

Next steps:

• Explore AI agent architecture patterns for building agents with tool use
• Learn about prompt caching to reduce costs when using complex schemas
• Check out production-ready LLM integration for scaling patterns

---

Building with structured outputs? I’d love to hear about your use cases — connect with me on Twitter!