Standardizing AI Agent Friendly Website Communication: The Case for agents.json

Jan 27, 2025

1. Introduction

Remember when OpenID Connect introduced /.well-known/openid-configuration? It solved the complex problem of OAuth discovery with a simple JSON file. Today, we're facing a similar challenge with AI agents interacting with websites.

Currently, AI agents either blindly scrape HTML or require custom API integration for each website. There's no standardized way for a website to tell an AI: "Here are my endpoints, here's what I can do, and here's how you should interact with me."

Take ChatGPT plugins or Anthropic's Claude as examples. Each requires its own specific manifest format. Every AI company is reinventing the wheel, creating their own standards for AI-website communication. This fragmentation isn't sustainable.

Why We Need a Standard

The need for standardization becomes clear when we consider the growing ecosystem of AI agents:

Efficiency: Without standards, each AI service provider must implement custom solutions for every website they interact with.
Scalability: As AI agents proliferate, websites need a systematic way to declare their capabilities and boundaries.
Control: Website owners need mechanisms to guide AI behavior, just as they do with search engines.
Innovation: A standard interface would enable faster development of AI-powered tools and services.
Trust: Clear guidelines and expectations help build trust between websites and AI systems.

Consider how chaotic email would be without SMTP, or web authentication without standardized protocols. We're at a similar inflection point with AI-website interactions.

Learning from Existing Standards

We're not starting from scratch. Several successful web standards offer valuable lessons:

robots.txt

User-agent: *
Disallow: /private/
Allow: /public/

Lesson: Simple, text-based declarations can effectively control automated access
Application: agents.json can similarly declare AI-specific permissions and restrictions

OpenID Connect Discovery

/.well-known/openid-configuration
{
  "authorization_endpoint": "https://example.com/auth",
  "token_endpoint": "https://example.com/token"
}

Lesson: The /.well-known pattern provides a predictable way to discover service capabilities
Application: agents.json can use the same pattern for AI capability discovery

security.txt

Contact: security@example.com
Expires: 2024-12-31T18:37:07z

Lesson: Standardized metadata helps establish clear communication channels
Application: agents.json can define AI-specific contact points and policies

The success of these standards shows that simple, well-defined interfaces can solve complex coordination problems. They work because they:

Are easy to implement
Solve real problems
Follow established patterns
Enable automation
Scale effectively

So, we already have proven patterns. agents.json can follow these patterns but focuses on AI interaction. Let’s say it lives at /agents.json and provides a machine-readable description of how AI agents should interact with your site.

2. Understanding agents.json

Let's look at a minimal agents.json I’m experimenting with:

{
  "version": "1.0",
  "endpoints": {
    "base_url": "https://api.example.com",
    "api_endpoints": {
      "search": {
        "path": "/search",
        "method": "GET",
        "requires_auth": false
      }
    }
  },
  "ai_capabilities": {
    "features": {
      "search": true,
      "chat": false
    },
    "supported_models": ["DeepSeek", "claude-2"],
    "context_window": 8192
  }
}

This tells AI agents:

Where to find your API
What endpoints are available
What AI features you support
What models you're compatible with

Core Components

Endpoints: Instead of scraping HTML, AI agents can use your API directly. You specify:
- Base URL for API access
- Available endpoints
- Authentication requirements
- Rate limits
AI Capabilities: Declare what AI features you support:
- Text generation
- Search
- Chat
- Image analysis
- etc.
Interaction Rules: Set boundaries for AI behavior:
- Rate limits
- Allowed/disallowed actions
- Authentication methods
- Data usage policies
Data Schemas: Define your data structures:

{
  "data_schemas": {
    "Product": {
      "type": "object",
      "properties": {
        "id": { "type": "string" },
        "name": { "type": "string" },
        "price": { "type": "number" }
      }
    }
  }
}

Real Example: E-commerce Site

{
  "version": "1.0",
  "endpoints": {
    "base_url": "https://api.shop.example",
    "api_endpoints": {
      "products": {
        "path": "/products",
        "method": "GET",
        "requires_auth": false,
        "rate_limit": {
          "requests": 60,
          "period": "minute"
        }
      },
      "search": {
        "path": "/search",
        "method": "GET",
        "requires_auth": false
      }
    }
  },
  "ai_capabilities": {
    "features": {
      "product_search": true,
      "price_check": true,
      "inventory_check": true
    },
    "supported_models": ["gpt-4", "claude-2"],
    "context_window": 8192
  },
  "interaction_rules": {
    "allowed_actions": [
      "read_products",
      "check_prices",
      "search"
    ],
    "restricted_actions": [
      "bulk_scraping",
      "automated_purchasing"
    ]
  }
}

This tells AI agents exactly how to interact with your e-commerce site programmatically, what they can do, and what they shouldn't do.

The beauty of this approach is its simplicity. Just like robots.txt and security.txt, it's a simple file that solves a complex coordination problem. No complex protocols, just structured data that both websites and AI agents can easily understand.

3. Key Components Deep Dive

Let's break down each component of agents.json and understand what they solve and how to implement them effectively.

Organization Information

This is your site's ID card for AI agents. Keep it minimal but informative:

{
  "organization": {
    "name": "Example Corp",
    "url": "https://example.com",
    "contact_email": "ai@example.com",
    "documentation_url": "https://docs.example.com"
  }
}

Don't include marketing text or unnecessary metadata. AI agents need machine-readable information, not promotional content.

Endpoints Configuration

This is where you tell AI agents how to interact with your APIs programmatically:

{
  "endpoints": {
    "base_url": "https://api.example.com",
    "api_version": "v1",
    "health_check": "/health",
    "api_endpoints": {
      "search": {
        "path": "/search",
        "method": "GET",
        "description": "Search API with OpenAPI-like parameters",
        "requires_auth": false,
        "rate_limit": {
          "requests": 60,
          "period": "minute"
        },
        "parameters": {
          "q": {
            "type": "string",
            "required": true,
            "description": "Search query"
          },
          "limit": {
            "type": "integer",
            "default": 10,
            "max": 100
          }
        }
      }
    }
  }
}

Key points:

Keep endpoint descriptions concise and technical
Always specify rate limits
Include parameter validation rules
Link to OpenAPI specs when available

AI Capabilities

This section defines what AI features your site supports:

{
  "ai_capabilities": {
    "supported_models": ["gpt-4", "claude-2"],
    "features": {
      "text_generation": {
        "enabled": true,
        "max_tokens": 4096,
        "supported_languages": ["en", "es"]
      },
      "search": {
        "enabled": true,
        "semantic": true,
        "vector_search": false
      },
      "image_analysis": false
    },
    "context_window": 8192,
    "rate_limits": {
      "tokens_per_minute": 100000,
      "requests_per_minute": 60
    }
  }
}

Don't just list features - specify their limitations and characteristics. This helps AI agents make informed decisions about how to interact with your service.

Interaction Rules

This is your bouncer - it sets clear boundaries:

{
  "interaction_rules": {
    "allowed_actions": [
      "read_content",
      "search",
      "analyze"
    ],
    "restricted_actions": [
      "bulk_scraping",
      "automated_posting",
      "user_impersonation"
    ],
    "authentication": {
      "required": true,
      "methods": ["API_KEY", "OAuth2"],
      "oauth_configuration_url": "/.well-known/oauth-configuration"
    },
    "rate_limiting": {
      "global": {
        "requests": 1000,
        "period": "hour"
      },
      "per_endpoint": true,
      "per_token": true
    }
  }
}

Be explicit about:

What actions are allowed/forbidden
Authentication requirements
Rate limiting at different levels
Any special restrictions

Data Policies

This section is crucial for privacy and data handling:

{
  "data_policies": {
    "data_retention": "30d",
    "user_consent_required": true,
    "schemas": {
      "User": {
        "type": "object",
        "properties": {
          "id": {
            "type": "string",
            "sensitive": false
          },
          "email": {
            "type": "string",
            "sensitive": true
          }
        }
      }
    },
    "privacy_policy_url": "/privacy",
    "sensitive_fields": ["email", "phone", "address"],
    "data_processing": {
      "location": "EU",
      "compliance": ["GDPR", "CCPA"]
    }
  }
}

Important aspects:

Mark sensitive fields explicitly
Specify retention periods
Include compliance information
Define data schemas clearly

Implementation Example

Here's how you might implement this in Express.js:

import express from 'express';
import { rateLimit } from 'express-rate-limit';
import { validate } from 'jsonschema';
import { agentsSchema } from './schemas';

const app = express();

// Your agents.json configuration
const agentsConfig = {
  version: "1.0",
  // ... (configuration from above examples)
};

// Validate against schema
const validation = validate(agentsConfig, agentsSchema);
if (!validation.valid) {
  throw new Error('Invalid agents.json configuration');
}

// Serve with appropriate headers
app.get('/.well-known/agents.json', (req, res) => {
  res.setHeader('Cache-Control', 'public, max-age=3600');
  res.setHeader('Content-Type', 'application/json');
  res.json(agentsConfig);
});

// Implement rate limiting as specified
const limiter = rateLimit({
  windowMs: 60 * 1000, // 1 minute
  max: agentsConfig.interaction_rules.rate_limiting.global.requests
});

app.use(limiter);

// Implement endpoints according to specification
app.get('/api/v1/search', (req, res) => {
  // Validate parameters against schema
  const { q, limit } = req.query;
  // ... implementation
});

4. Real-World Implementation Examples

Let's look at practical implementations across different types of websites. Each example demonstrates unique use cases and requirements.

E-commerce Platform (Amazon-like)

{
  "version": "1.0",
  "endpoints": {
    "base_url": "https://api.shop.example",
    "api_endpoints": {
      "products": {
        "path": "/products",
        "method": "GET",
        "parameters": {
          "category": { "type": "string" },
          "query": { "type": "string" },
          "price_range": { 
            "type": "object",
            "properties": {
              "min": { "type": "number" },
              "max": { "type": "number" }
            }
          }
        }
      },
      "inventory": {
        "path": "/products/{id}/inventory",
        "method": "GET",
        "requires_auth": true
      },
      "prices": {
        "path": "/products/{id}/price-history",
        "method": "GET",
        "rate_limit": {
          "requests": 100,
          "period": "minute"
        }
      }
    }
  },
  "ai_capabilities": {
    "features": {
      "product_search": true,
      "price_comparison": true,
      "stock_checking": true,
      "category_browsing": true
    },
    "supported_models": ["gpt-4", "claude-2"],
    "context_window": 8192
  },
  "data_schemas": {
    "Product": {
      "type": "object",
      "properties": {
        "id": { "type": "string" },
        "name": { "type": "string" },
        "price": { "type": "number" },
        "currency": { "type": "string" },
        "stock": { "type": "integer" },
        "categories": { "type": "array", "items": { "type": "string" } }
      }
    }
  }
}

SaaS Developer Platform (GitHub-like)

{
  "version": "1.0",
  "endpoints": {
    "base_url": "https://api.dev.example",
    "api_endpoints": {
      "repositories": {
        "path": "/repos/{owner}/{repo}",
        "method": "GET",
        "requires_auth": true
      },
      "code_search": {
        "path": "/search/code",
        "method": "GET",
        "parameters": {
          "q": { 
            "type": "string",
            "description": "Search query with GitHub-like syntax"
          }
        }
      },
      "issues": {
        "path": "/repos/{owner}/{repo}/issues",
        "method": "GET"
      }
    }
  },
  "ai_capabilities": {
    "features": {
      "code_analysis": true,
      "semantic_search": true,
      "dependency_analysis": true,
      "vulnerability_scanning": true
    },
    "code_understanding": {
      "languages": ["python", "javascript", "go", "java"],
      "max_file_size": 1048576
    }
  },
  "interaction_rules": {
    "rate_limiting": {
      "authenticated": {
        "requests": 5000,
        "period": "hour"
      },
      "unauthenticated": {
        "requests": 60,
        "period": "hour"
      }
    }
  }
}

Content Platform (Medium-like)

{
  "version": "1.0",
  "endpoints": {
    "base_url": "https://api.content.example",
    "api_endpoints": {
      "articles": {
        "path": "/articles",
        "method": "GET",
        "parameters": {
          "topic": { "type": "string" },
          "author": { "type": "string" },
          "published_after": { "type": "string", "format": "date-time" }
        }
      },
      "search": {
        "path": "/search",
        "method": "GET",
        "semantic_search": true
      }
    }
  },
  "ai_capabilities": {
    "features": {
      "content_summarization": true,
      "topic_classification": true,
      "semantic_search": true,
      "content_recommendation": true
    },
    "content_processing": {
      "max_length": 50000,
      "supported_formats": ["text", "markdown"],
      "supported_languages": ["en", "es", "fr"]
    }
  },
  "data_schemas": {
    "Article": {
      "type": "object",
      "properties": {
        "id": { "type": "string" },
        "title": { "type": "string" },
        "content": { "type": "string" },
        "author": { "type": "string" },
        "published_at": { "type": "string", "format": "date-time" },
        "topics": { "type": "array", "items": { "type": "string" } }
      }
    }
  }
}

Implementation Tips

Express.js Implementation:

import express from 'express';
import { readFileSync } from 'fs';

const app = express();

// Load configuration based on your platform type
const config = JSON.parse(
  readFileSync('./agents-config.json', 'utf-8')
);

// Serve agents.json
app.get('/.well-known/agents.json', (req, res) => {
  res.setHeader('Cache-Control', 'public, max-age=3600');
  res.json(config);
});

// Implement rate limiting based on config
app.use(createRateLimiter(config.interaction_rules.rate_limiting));

// Implement endpoints
Object.entries(config.endpoints.api_endpoints).forEach(([name, endpoint]) => {
  app[endpoint.method.toLowerCase()](endpoint.path, 
    endpoint.requires_auth ? authMiddleware : null,
    async (req, res) => {
      // Your endpoint implementation
    }
  );
});

Next.js Implementation:

// pages/api/.well-known/agents.json.ts
import type { NextApiRequest, NextApiResponse } from 'next';
import { agentsConfig } from '@/config/agents';

export default function handler(
  req: NextApiRequest,
  res: NextApiResponse
) {
  if (req.method !== 'GET') {
    return res.status(405).end();
  }

  res.setHeader('Cache-Control', 'public, max-age=3600');
  res.json(agentsConfig);
}

Dynamic Configuration:

function generateAgentsConfig(env: string) {
  return {
    version: "1.0",
    endpoints: {
      base_url: process.env.API_BASE_URL,
      // ... dynamic configuration based on environment
    },
    ai_capabilities: {
      features: {
        // Enable features based on environment/configuration
        semantic_search: env === 'production',
        content_recommendation: env === 'production'
      }
    }
  };
}

5. Benefits and Use Cases

Let's explore how agents.json benefits different stakeholders with practical examples and real-world scenarios.

For Website Owners

1. Resource Optimization

// Before agents.json - AI scraping causing high load
app.get('*', async (req, res) => {
  // Expensive HTML parsing and rate limiting logic
});

// After agents.json - Structured API access
const agentsConfig = {
  "endpoints": {
    "content": {
      "path": "/api/v1/content",
      "method": "GET",
      "batch_size": 100,
      "rate_limit": {
        "requests": 60,
        "period": "minute"
      }
    }
  }
};

2. Control Over AI Interactions

{
  "interaction_rules": {
    "allowed_actions": ["read", "search"],
    "restricted_actions": ["write", "modify"],
    "content_filters": {
      "exclude_paths": ["/private/*", "/draft/*"],
      "sensitive_data": ["email", "phone"]
    }
  }
}

3. Analytics and Monitoring

// Track AI agent usage
app.use((req, res, next) => {
  if (req.headers['x-ai-agent']) {
    metrics.increment('ai_agent_request', {
      agent: req.headers['x-ai-agent'],
      endpoint: req.path
    });
  }
  next();
});

For AI Developers

1. Standardized Discovery

async function discoverSiteCapabilities(url: string) {
  const response = await fetch(`${url}/.well-known/agents.json`);
  if (response.ok) {
    const config = await response.json();
    return {
      canSearch: config.ai_capabilities.features.search,
      apiEndpoints: config.endpoints.api_endpoints,
      rateLimits: config.interaction_rules.rate_limiting
    };
  }
  // Fallback to traditional scraping
  return null;
}

2. Efficient Integration

class AIAgent {
  async interact(website: string) {
    const config = await this.fetchAgentsConfig(website);
    
    // Automatic endpoint discovery
    const endpoints = new EndpointManager(config.endpoints);
    
    // Automatic rate limiting
    const rateLimiter = new RateLimiter(config.interaction_rules.rate_limiting);
    
    // Feature detection
    if (config.ai_capabilities.features.semantic_search) {
      return await this.semanticSearch(endpoints.search, query);
    }
    
    // Fallback to basic search
    return await this.basicSearch(endpoints.search, query);
  }
}

3. Error Prevention

class AIInteractionManager {
  validateRequest(action: string, data: any) {
    // Check against allowed actions
    if (!this.config.interaction_rules.allowed_actions.includes(action)) {
      throw new Error(`Action ${action} not allowed`);
    }

    // Validate data against schema
    const schema = this.config.data_schemas[data.type];
    if (schema) {
      const validation = validateAgainstSchema(data, schema);
      if (!validation.valid) {
        throw new Error(`Invalid data format: ${validation.errors}`);
      }
    }
  }
}

For End Users

1. Better AI Responses

// AI can provide more accurate information
async function getProductInfo(url: string, productId: string) {
  const config = await fetchAgentsConfig(url);
  
  if (config.ai_capabilities.features.price_tracking) {
    const priceHistory = await fetchPriceHistory(productId);
    return `Current price is $${priceHistory.current}. 
            Price has ${priceHistory.trend} by ${priceHistory.change}% 
            in the last month.`;
  }
  
  // Fallback to basic info
  return await fetchBasicProductInfo(productId);
}

2. Privacy Protection

{
  "data_policies": {
    "user_data": {
      "collection": ["public_profile"],
      "prohibited": ["private_messages", "payment_info"],
      "retention": "30d",
      "sharing": "none"
    },
    "consent_required": true
  }
}

For the AI Ecosystem

1. Standardization

// Common interface for AI interactions
interface AIWebsiteInteraction {
  readonly config: AgentsConfig;
  
  canPerform(action: string): boolean;
  getRateLimit(endpoint: string): RateLimit;
  getDataSchema(type: string): Schema;
  validateInteraction(action: string, data: any): ValidationResult;
}

2. Innovation Enablement

// Easy to build new AI features
class AIFeatureDetector {
  async suggestFeatures(website: string) {
    const config = await fetch(`${website}/.well-known/agents.json`);
    const capabilities = config.ai_capabilities;
    
    return {
      canGenerateContent: capabilities.features.text_generation,
      canAnalyzeImages: capabilities.features.image_analysis,
      maxContextWindow: capabilities.context_window,
      supportedModels: capabilities.supported_models
    };
  }
}

3. Ecosystem Growth

// Plugin system based on capabilities
class AIPlugin {
  static async isCompatible(website: string) {
    const config = await fetchAgentsConfig(website);
    return this.requiredFeatures.every(
      feature => config.ai_capabilities.features[feature]
    );
  }
  
  static async create(website: string) {
    if (await this.isCompatible(website)) {
      return new this(await fetchAgentsConfig(website));
    }
    throw new Error('Website not compatible with this plugin');
  }
}

The beauty of agents.json is that it creates a win-win situation for all stakeholders:

Website owners get better control and efficiency
AI developers get standardized integration
Users get better AI interactions
The ecosystem gets a foundation for innovation

6. Technical Implementation

Let's dive into the practical aspects of implementing agents.json, focusing on real-world scenarios and best practices.

Schema Validation

First, create a TypeScript interface for type safety:

// types/agents-config.ts
interface AgentsConfig {
  version: string;
  endpoints: {
    base_url: string;
    api_endpoints: Record<string, {
      path: string;
      method: 'GET' | 'POST' | 'PUT' | 'DELETE';
      requires_auth: boolean;
      rate_limit?: {
        requests: number;
        period: 'second' | 'minute' | 'hour' | 'day';
      };
    }>;
  };
  ai_capabilities: {
    features: Record<string, boolean>;
    supported_models?: string[];
    context_window?: number;
  };
  interaction_rules: {
    allowed_actions: string[];
    restricted_actions: string[];
    rate_limiting: {
      enabled: boolean;
      global_limit?: {
        requests: number;
        period: string;
      };
    };
  };
}

Base Implementation

Express.js Example

// src/middleware/agents-config.ts
import express from 'express';
import { rateLimit } from 'express-rate-limit';
import { validate } from 'jsonschema';
import { AgentsConfig } from '../types';

export class AgentsConfigHandler {
  private config: AgentsConfig;
  
  constructor(config: AgentsConfig) {
    this.validateConfig(config);
    this.config = config;
  }

  private validateConfig(config: AgentsConfig) {
    // Validate version
    if (!config.version.match(/^\d+\.\d+(\.\d+)?$/)) {
      throw new Error('Invalid version format');
    }

    // Validate endpoints
    if (!config.endpoints.base_url.startsWith('http')) {
      throw new Error('Invalid base URL');
    }
  }

  public middleware() {
    return (req: express.Request, res: express.Response) => {
      res.setHeader('Cache-Control', 'public, max-age=3600');
      res.json(this.config);
    };
  }

  public getRateLimiter(endpoint: string) {
    const limit = this.config.endpoints.api_endpoints[endpoint]?.rate_limit;
    if (!limit) return null;

    return rateLimit({
      windowMs: this.getPeriodInMs(limit.period),
      max: limit.requests
    });
  }

  private getPeriodInMs(period: string): number {
    const periods = {
      second: 1000,
      minute: 60 * 1000,
      hour: 60 * 60 * 1000,
      day: 24 * 60 * 60 * 1000
    };
    return periods[period] || periods.hour;
  }
}

Implementation in Express App

// src/app.ts
import express from 'express';
import { AgentsConfigHandler } from './middleware/agents-config';

const app = express();

const agentsConfig: AgentsConfig = {
  version: "1.0",
  endpoints: {
    base_url: "https://api.example.com",
    api_endpoints: {
      search: {
        path: "/search",
        method: "GET",
        requires_auth: false,
        rate_limit: {
          requests: 60,
          period: "minute"
        }
      }
    }
  },
  // ... rest of config
};

const agentsHandler = new AgentsConfigHandler(agentsConfig);

// Serve agents.json
app.get('/.well-known/agents.json', agentsHandler.middleware());

// Apply rate limiting based on config
app.get('/api/search', 
  agentsHandler.getRateLimiter('search'),
  async (req, res) => {
    // Search implementation
  }
);

Security Considerations

Authentication Middleware

// src/middleware/auth.ts
export const authMiddleware = (config: AgentsConfig) => {
  return async (req: express.Request, res: express.Response, next: express.NextFunction) => {
    const endpoint = req.path.replace('/api/', '');
    
    if (config.endpoints.api_endpoints[endpoint]?.requires_auth) {
      const apiKey = req.headers['x-api-key'];
      if (!apiKey) {
        return res.status(401).json({ error: 'Authentication required' });
      }
      
      // Validate API key
      try {
        await validateApiKey(apiKey as string);
        next();
      } catch (error) {
        res.status(401).json({ error: 'Invalid API key' });
      }
    } else {
      next();
    }
  };
};

Rate Limiting with Redis

// src/middleware/rate-limit.ts
import Redis from 'ioredis';
import { RateLimitError } from '../errors';

export class RateLimiter {
  private redis: Redis;
  
  constructor(redisUrl: string) {
    this.redis = new Redis(redisUrl);
  }

  async checkLimit(key: string, limit: number, period: number): Promise<boolean> {
    const current = await this.redis.incr(key);
    if (current === 1) {
      await this.redis.expire(key, period);
    }
    return current <= limit;
  }

  middleware(limit: number, period: string) {
    return async (req: express.Request, res: express.Response, next: express.NextFunction) => {
      const key = `rate_limit:${req.ip}:${req.path}`;
      const periodInSeconds = this.getPeriodInSeconds(period);
      
      try {
        const allowed = await this.checkLimit(key, limit, periodInSeconds);
        if (!allowed) {
          throw new RateLimitError('Rate limit exceeded');
        }
        next();
      } catch (error) {
        if (error instanceof RateLimitError) {
          res.status(429).json({ error: error.message });
        } else {
          next(error);
        }
      }
    };
  }
}

Monitoring and Analytics

// src/monitoring/ai-analytics.ts
import { Metrics } from '@your-metrics-library';

export class AIAnalytics {
  private metrics: Metrics;
  
  constructor(metricsUrl: string) {
    this.metrics = new Metrics(metricsUrl);
  }

  middleware() {
    return (req: express.Request, res: express.Response, next: express.NextFunction) => {
      const startTime = Date.now();
      
      res.on('finish', () => {
        const duration = Date.now() - startTime;
        
        this.metrics.increment('ai_request_total', {
          endpoint: req.path,
          agent: req.headers['x-ai-agent'],
          status: res.statusCode
        });
        
        this.metrics.histogram('ai_request_duration', duration, {
          endpoint: req.path
        });
      });
      
      next();
    };
  }
}

Error Handling

// src/middleware/error-handler.ts
export const errorHandler = (
  error: Error,
  req: express.Request,
  res: express.Response,
  next: express.NextFunction
) => {
  if (error instanceof RateLimitError) {
    return res.status(429).json({
      error: 'Rate limit exceeded',
      retry_after: error.retryAfter
    });
  }

  if (error instanceof ValidationError) {
    return res.status(400).json({
      error: 'Validation failed',
      details: error.details
    });
  }

  // Log unexpected errors
  console.error('Unexpected error:', error);
  res.status(500).json({ error: 'Internal server error' });
};

Testing

// src/tests/agents-config.test.ts
import request from 'supertest';
import { app } from '../app';

describe('agents.json', () => {
  it('should return valid configuration', async () => {
    const response = await request(app)
      .get('/.well-known/agents.json')
      .expect(200);
    
    expect(response.body).toHaveProperty('version');
    expect(response.body.endpoints).toBeDefined();
  });

  it('should respect rate limits', async () => {
    const endpoint = '/api/search';
    const limit = 60;
    
    // Make requests up to limit
    for (let i = 0; i < limit; i++) {
      await request(app).get(endpoint).expect(200);
    }
    
    // Next request should fail
    await request(app)
      .get(endpoint)
      .expect(429);
  });
});

7. Future Possibilities & Getting Started

Current State

Let's be transparent: agents.json isn't an official standard. It's an idea inspired by existing patterns like robots.txt and OIDC discovery, aiming to solve real problems in AI-website interactions.

While waiting for official standardization (ever), you can start implementing these ideas.

Future Possibilities

1. Dynamic Capability Discovery

class AICapabilityManager {
  async detectCapabilities() {
    return {
      semantic_search: await this.testSemanticSearch(),
      language_support: await this.getLanguageSupport(),
      max_throughput: await this.measureThroughput()
    };
  }

  async updateAgentsJson() {
    const capabilities = await this.detectCapabilities();
    this.config.ai_capabilities = capabilities;
    await this.saveConfig();
  }
}

2. AI-Specific Extensions

{
  "ai_extensions": {
    "context_preservation": {
      "enabled": true,
      "max_history": 10,
      "storage_duration": "1h"
    },
    "learning_preferences": {
      "allow_model_adaptation": true,
      "feedback_collection": true
    },
    "specialized_endpoints": {
      "embedding": "/api/embed",
      "semantic_search": "/api/semantic-search"
    }
  }
}

3. Interoperability Standards

// Future possibility: Standard AI interaction protocols
interface AIInteractionProtocol {
  readonly version: string;
  readonly capabilities: Set<string>;
  
  async negotiate(agent: string): Promise<Protocol>;
  async interact(action: string, data: any): Promise<Response>;
  async validateCompliance(): Promise<ValidationResult>;
}

Community Contribution

Here's how you can help shape this standard:

Implement and Experiment
Document your implementation and findings, like:
- Rate limiting across distributed systems
- Schema version compatibility
- Others…

Share Feedback

Implement in your projects
Document edge cases
Share success stories
Report challenges
Propose improvements

Best Practices (Based on My Implementations)

// Current recommended patterns
const bestPractices = {
  versioning: {
    pattern: "major.minor",
    example: "0.1-experimental"
  },
  caching: {
    maxAge: 3600,
    revalidation: "if-needed"
  },
  rateLimit: {
    implementation: "distributed",
    fallback: "local-memory"
  },
  security: {
    headers: [
      "Cache-Control",
      "X-Content-Type-Options",
      "X-Frame-Options"
    ],
    validation: "strict"
  }
};

Looking Forward

The future of agents.json depends on:

Community adoption and feedback
Real-world implementation experiences
Evolution of AI agent capabilities
Standardization efforts

Remember:

This is an evolving concept
Implementation details may change
Focus on solving real problems
Share your experiences
Stay flexible with your implementation

// Future-proofing your implementation
class AgentsConfig {
  constructor() {
    this.version = "0.1-experimental";
    this.extensible = true;
    this.updateCallback = async (newStandard) => {
      // Your update logic
    };
  }

  async adaptToNewStandard(standard: string) {
    // Keep your implementation flexible
    await this.updateCallback(standard);
  }
}

The goal isn't to create another rigid standard but to solve real problems in AI-website interactions. Start implementing these patterns, share your experiences, and help shape the future of this potential standard.

Want to contribute? Start by implementing these patterns in your projects and sharing your findings. The more real-world feedback folks like us get, the better this potential standard can become.

Arshad’s Lighthouse