Shadow AI in Engineering: Get Visibility Into Coding Agents Without Turning Into IT Surveillance

The workable answer to shadow AI is to measure where agents connect, who can expense them, and what reaches a pull request—not to record developer screens or vacuum up prompts. If you can identify an agent session, its model provider, its repository or project, its spend, and its eventual CI/PR outcome, you have enough visibility to manage the risk without making the tool unusable.

Treat “visibility” as four joins, not one dashboard

Every vendor dashboard has gaps. That is normal. The mistake is declaring one of them your source of truth, then discovering six months later that a team has been running a personal Claude Code account through an API key, an editor extension bought on a card, and a local model behind a corporate proxy.

Build a small internal dataset around four keys: person or service identity, agent/provider, repository or project, and time. Pull identity from SSO and device management; spend from corporate cards, expense reports, cloud bills, and vendor admin exports; egress from your proxy or DNS logs; and delivery evidence from GitHub, GitLab, CI, and deployment systems. You do not need to perfectly attribute every completion. You need to answer, within a day, “which tools touched this codebase, under which account, and what controls applied?”

That changes the conversation with developers. Instead of asking, “Did you use AI?”—which gets you incomplete answers—you can ask about a concrete mismatch: “This repository had traffic to an unapproved model endpoint from a managed laptop. Do you need a supported route for this workflow?” The goal is to move useful work into visible, supportable paths.

Turn on vendor telemetry, but make the collection boundary explicit

For tools your company already pays for, start with their native organization reporting. GitHub Copilot’s usage metrics can expose activity by user, repository, feature, IDE, language, model, and agent surface; the raw reports also include separate signals for CLI, cloud-agent, app, and recognized third-party-agent activity. The reporting is useful for adoption and coverage, but it is not an audit trail of every prompt or shell command.

Do not build a management dashboard around accepted-lines-of-code. It tells you something about tool interaction, not correctness, maintainability, or whether a developer spent an hour undoing the result. GitHub itself distinguishes agent adoption and usage from engineering outcomes, and its team reporting has non-obvious limitations: teams with fewer than five seated users are excluded from team reports, while people on multiple teams can be counted in each team. Use it to spot adoption and surface gaps, not to rank engineers.

If you run Copilot CLI or VS Code at enterprise scale, OpenTelemetry is the useful escape hatch. GitHub documents OTLP export for agent interactions, LLM calls, tool execution, and token usage. Enable it at the organization boundary and send it to the observability stack you already secure and retain:

export COPILOT_OTEL_ENABLED=true
export OTEL_EXPORTER_OTLP_ENDPOINT=https://otel-collector.example.internal

Keep prompt and response capture off. GitHub’s managed telemetry settings support a captureContent field and a lockCaptureContent control; set the former to false and lock it unless you have a specific, reviewed incident-response reason to collect content. Metadata such as agent, model, duration, tool name, repository, exit status, and token count is usually enough to see operational risk. Prompt bodies are where customer data, secrets, and the unpleasant parts of employee monitoring arrive.

Put a gateway in front of bring-your-own-key agents

Native reporting will not solve the biggest shadow-AI gap: direct API use. A developer can run Claude Code, Cline, a custom script, or an IDE extension with a personal or project key and never appear in your seat-management dashboard. Give people an approved API route rather than pretending you can forbid this with a policy page.

An LLM gateway is the right control point when your teams need multiple models or agent clients. Anthropic’s Claude Code documentation describes the basic value plainly: centralized authentication, usage tracking, budgets, rate limits, audit logging, and model routing. Issue short-lived credentials tied to SSO identity; require a project or cost-center tag; allowlist model providers; and send only metadata to the central log by default.

That gateway also gives security a place to set a real response rule. For example: requests to approved providers from managed machines are allowed; unknown AI endpoints are logged for review; attempts to send data from repositories tagged restricted require an approved gateway route or are blocked. This is far more defensible than blocking the word “AI” at the proxy, which catches documentation sites and misses encrypted tunnels, personal phones, and tomorrow’s endpoint name.

Make agent involvement visible where code changes hands

Network and spend data tell you an agent existed. They do not tell a reviewer what it changed. Put the disclosure at the pull request, with a lightweight convention that is easy to satisfy and easy to query.

• Add an agent-assisted PR label or a PR-template checkbox: “An AI coding agent made or materially edited changes in this PR.”
• Require a short Agent notes section only when the agent ran commands, changed more than one file, or used an external tool: model/provider, commands requiring elevated access, and tests run.
• Have CI post the real evidence—changed-file count, test results, dependency diff, secret-scan result, and deployment environment. Do not ask authors to reproduce it by hand.
• Create a saved query and a weekly report for agent-assisted PRs: merge time, rollback or revert rate, CI failure rate, and security findings. Compare trends by repository, not by individual developer.

This is deliberately boring. A label can be omitted, and a developer can misunderstand what counts as “materially edited.” That is why you pair it with gateway and provider data. The PR marker is not proof; it is the review affordance. It tells the next engineer that an agent may have traversed files, invoked tools, or made a broad edit, so they should inspect the diff accordingly.

Know what your dashboard cannot see

Your coverage will never be 100%. GitHub says some Copilot metrics depend on IDE telemetry, and activity from third-party IDEs may be inconsistent. Its Metrics API is also not real time: it processes daily data and returns through the previous day. GitHub’s own documentation notes that Copilot CLI can run in offline mode with telemetry disabled, and bring-your-own-provider use can bypass GitHub authentication entirely. Those are not defects in your rollout; they are the boundary conditions you should publish.

Publish a coverage statement next to the dashboard: which managed clients report, which gateway routes are covered, which platforms are best-effort, what data is retained, and who can access it. Then show developers the approved alternative when a tool is missing. If your answer to an unapproved agent is only “stop,” they will find a personal account. If it is “here is the gateway endpoint, supported models, budget tag, and a 10-minute onboarding path,” most use becomes visible without a fight.

Start with one organization and a 30-day baseline. Turn on Copilot usage reporting, route one API-using team through a gateway, add the PR label and CI summary, and review the four joins once a week. After that month, you will know whether the problem is genuinely shadow usage, an approved-tool usability gap, or simply that your existing telemetry has been sitting in separate dashboards nobody owns.