Skip to content
AI NEWS JOURNAL
AI, IN PERSPECTIVE

Search AI news

Search headlines, summaries and articles to connect the dots.

Newest first

Agents are users, so your software needs evals

Agents interact with software through a funnel similar to humans: Discovery, Selection, Access, and Success. To measure whether agents complete user tasks correctly, end-to-end (E2E) evaluations are essential. These tests run multiple trials, score outcomes, and help identify failures across steps like searching cafes, checking budgets, and placing orders. Using frameworks like Harbor, teams can define tasks with specific instructions, environments, verifiers, and known-good solutions. Metrics such as success rate, Pass@k, and Pass^k reveal how reliably agents perform, while trace analysis uncovers root causes of failure. The process includes setting a goal based on observability, customer feedback, or roadmap needs; defining measurable metrics; designing task sets covering common and edge cases; running evaluations at scale using tools like Vercel Sandbox and AI Gateway; and analyzing traces to iterate improvements. Automated trace analysis enables scaling, and repeated evaluation supports continuous improvement, regression detection, and product optimization.

WTF are evals? How to test agents and the software they use

Agents interact with software through a funnel similar to humans: Discovery, Selection, Access, and Success. To measure whether agents complete user tasks correctly, end-to-end (E2E) evaluations are essential. These evaluations assess the full sequence of steps an agent takes, ensuring outcomes meet user requirements. Using frameworks like Harbor, developers can design tasks with defined environments, verifiers, and known-good solutions, then run trials across variants to measure success rate, pass@k, pass^k, efficiency, time, and cost. Analyzing traces reveals root causes of failures, enabling targeted improvements. Iterative evaluation helps catch regressions, test changes, and guide product development, turning evaluations into ongoing infrastructure for building better agent-driven products.

Vercel Container Registry now supports 2 GB compressed image layers

Vercel Container Registry now allows compressed image layers up to 2 GB each, while maintaining a total image size limit of 15 GB. It supports standard Docker workflows like docker push, docker pull, and docker tag, automatically creating repositories on push. Images can be made public or shared with up to 100 teams via the dashboard or CLI. The Vercel CLI v58.3.0 or later enables management of repositories, images, and tags, and GitHub Actions integration uses GitHub OIDC for secure CI logging without long-lived credentials. Failed background optimizations show error statuses in the dashboard, with retry options upon re-push.

Chat SDK agents can now stream generative UI with json-render

Vercel introduces json-render, enabling Chat SDK agents to generate and stream interactive UI components as JSON specs, which are converted into platform-native cards (e.g., Block Kit on Slack, Adaptive Cards on Teams). The system uses a typed component catalog covering tables, charts, buttons, and more, eliminating the need for separate tools. It supports streaming updates with real-time edits every 500 ms, validates component specs against schemas, and enforces platform-specific message limits to prevent content degradation.

Vercel Schedules is now in public beta

Vercel Schedules is now available in public beta across all plans, enabling users to run tasks on a recurring basis or at a specific time in production, preview, or custom environments. It supports targeting private Vercel Functions, HTTP paths, or Queue topics, with schedules definable in vercel.json or created at runtime. Static schedules use cron expressions and require an experimental trigger to make functions private, while runtime schedules dispatch to Queue topics. Schedules replace Cron Jobs, with billing starting at $5 per 1M runs and $10 per 1M creations on Pro and Enterprise plans, and Hobby plan includes 500 schedule creations and 5,000 runs monthly.

Workflow SDK adds idempotent starts

The Workflow SDK's start() function now allows initiating a workflow run and claiming a hook atomically, eliminating the race condition present in the previous getHookByToken()-then-start() pattern. This change ensures idempotency using a token derived from domain-specific identifiers like order or invoice IDs. Duplicate starts fail safely with a HookConflictError, which includes the conflicting run's ID. The claimed hook is available immediately, enabling early payload buffering and ordered delivery even before execution begins. Inside workflows, createHook with the same token binds to the existing hook without conflict or additional steps. Experimental settings like minRetention prevent late duplicates by keeping tokens reserved after run completion.

Vercel Security Review now in public beta

Vercel Security Review is now available in public beta for Pro and Enterprise teams, automatically scanning code pushes to connected GitHub repositories for vulnerabilities. Reviews trigger on every push, including direct and force pushes, and findings are reported in a dashboard with details like project, environment, branch, commit, summary, suggested fix, and AI-generated severity. Findings appear as GitHub checks or review comments, and users can fix issues via Vercel Agent or CLI. Teams can set security policies for context, but the tool does not perform runtime or penetration testing. Pricing follows Vercel Agent rates, with free access until November 10, 2026.

Diagnose Vercel CLI setup problems with vercel doctor

The Vercel CLI now includes the vercel doctor command, which diagnoses setup issues by checking CLI version and installation, login status, team affiliation, project linking, vercel.json configuration, and active incidents from the Vercel status page. It reports results as OK, Warning, Failed, or Skipped, and provides specific fix commands. The tool runs without requiring login or project linkage and supports JSON output via --json for integration with coding agents.

How Mercor ships internal apps and billions of production tokens on one platform

Mercor reduced batch job processing time from 12 hours to 2–3 hours by using Vercel AI Gateway, which provided increased inference capacity and failover routing across providers. In its first 10 days on the platform, Mercor routed 18 million requests and 63 billion tokens without hitting rate limits. The company also leveraged Vercel Passport and its internal Audiences product to provision secure access for hundreds of business teams and over 2,000 projects in seconds, enabling rapid deployment of client-specific applications while maintaining strict usage controls and automated offboarding.

Blocking malicious npm packages before and during installation

Vercel now proactively checks for and blocks malicious npm packages before and during dependency installation in builds using a dynamic blocklist, preventing harmful scripts from executing. The system scans supported lockfiles and public npm downloads, automatically failing deployments if a blocked package is detected, identifying the problematic package and version, and attempting to invalidate the build cache. Users must update to a safe dependency version and regenerate the lockfile before redeploying. These security measures apply automatically to all plans without requiring configuration.

Introducing Vercel Security Review

Vercel introduces Security Review, a new feature in Vercel Agent that automatically scans for vulnerabilities in code changes pushed to connected GitHub repositories. It provides real-time findings via pull request comments, a Security dashboard, and CLI access, with options to auto-fix issues using Vercel Agent or integrate with custom coding agents. The tool checks for logic flaws, enforces team-specific security policies, tracks vulnerabilities across deployments, and supports audit logging for Enterprise users. Available now in public beta for Pro and Enterprise teams, it is free until November 10, 2026.