At this year’s AWS Summit DC, the conversation around agentic AI and multi-agent workflows had moved from experimentation to the harder question of how to run them securely against real systems and data.
My focus was narrower than the full Summit and directly connected to my current role. I attended sessions on agentic AI, AI for data engineering, intelligent document processing, multi-agent architectures, evaluation, and data governance.
This is not a summary of everything that happened at the event. It is the signal from the sessions most relevant to technical leaders responsible for real systems, sensitive data, limited budgets, and consequences that extend beyond a bad demo.
Across those sessions, the message was consistent. The model is only one part of the system.
The harder work is connecting it to organizational data and tools, controlling what it can access, evaluating whether its output is reliable, tracing what it did, and operating it safely at scale.
The demo is the easy part. Production is where the consequences begin.
AI is moving from answering questions to doing work
One of the clearest shifts was the growing role of AI agents in data engineering.
AWS is releasing open-source MCP servers for services including Amazon Redshift, AWS Glue, Amazon EMR, and Amazon Athena. MCP, or Model Context Protocol, gives AI agents a standard way to connect to tools, databases, APIs, and services.
In practical terms, an agent can discover schemas, generate SQL, run queries, inspect pipelines, and adjust its approach based on the results.
The underlying data platform does not disappear. The databases, warehouses, access controls, and pipelines are still there. What changes is the interface operating them.
Today, an engineer may explore a schema, write a query, review the output, and make corrections. An agent can potentially perform the same sequence through natural-language instructions.
That could reduce the amount of custom query tooling teams need to build. It could also improve access to data for people who understand the business questions but do not write SQL.
It also creates immediate governance questions.
What schemas can the agent discover? Which tables can it query? Which columns should it never see? How much data can it scan? Can it only read data, or can it modify something?
Giving an autonomous agent broad database access before answering those questions would make the project more exciting. Probably not in the intended way.
Document processing has become worth revisiting
Intelligent document processing was another major focus.
Many public sector and mission-driven organizations still rely heavily on PDFs, scanned forms, applications, audit reports, contracts, certification records, and other documents that do not arrive in one clean format.
Traditional extraction tools work well when documents follow a predictable template. They struggle when layouts change, scans are poor, tables move, or information appears across text, images, and attachments.
Generative AI has made more complex document processing practical, but the recommended architecture was not to send every document to the largest available model.
The better pattern is:
- Classify the document.
- Determine how complex and predictable it is.
- Route it to the cheapest tool capable of processing it accurately.
- Escalate difficult cases to a more advanced model.
- Send low-confidence results to a person for review.
For standardized forms, Amazon Textract may be enough. Amazon Bedrock Data Automation can handle more variable and multimodal documents. A full LLM workflow may be appropriate when the task requires reasoning, validation, or several processing steps.
AWS also provides an open-source GenAI Intelligent Document Processing Accelerator with classification, extraction, confidence scoring, rule validation, and human review already built in.
For organizations that evaluated OCR or document automation several years ago and found it too rigid, this area deserves another look. The capabilities have changed enough to alter the cost and feasibility of the work.
Multi-agent systems need architectural discipline
Multi-agent architecture appeared throughout the event.
Two protocols came up repeatedly:
MCP connects agents to tools and data.
A2A, or Agent2Agent, allows independent agents to communicate across systems, processes, or frameworks.
Several coordination patterns are also becoming standard. One main agent can call specialist agents as tools. Agents can operate inside a defined graph or workflow. They can also coordinate more freely as a swarm, where the next step is decided dynamically.
The most useful guidance was simple: start with one agent.
Split the system only when there is a concrete reason. That reason may be separate permissions, different models for different tasks, independent deployments, or a workflow that genuinely benefits from specialization.
A multi-agent system is not automatically more capable. Sometimes it is just a distributed system with additional opportunities for latency, debugging, and confusion.
If the process must follow the same path every time, use a graph or workflow. If one orchestrator needs help from a few specialists, call them as tools. Use a swarm only when unpredictability is acceptable and useful.
Architecture should follow the problem, not the conference diagram.
Vibe testing is not quality assurance
One session used the term “vibe testing” to describe how many AI features are still evaluated.
A few people try the tool. The answers look reasonable. Everyone feels encouraged. The feature is declared ready.
This works until a real user asks a slightly different question and receives a confident but incorrect response.
Generative AI output is non-deterministic. The same input can produce different answers, so traditional tests that expect one exact result are often not enough.
A stronger evaluation process includes:
- A gold-standard set of test cases
- A scoring rubric
- Automated comparison across prompts or models
- An LLM acting as a judge where exact comparison is not practical
- Release thresholds based on both the likelihood and severity of failure
- Continued monitoring after launch
Severity matters.
A formatting error and an answer exposing sensitive information are not equivalent failures. One accuracy percentage cannot represent every risk.
Tools such as Promptfoo and Amazon Bedrock Evaluations can help automate this work. Guardrails can also detect or block prompt attacks, personal information, harmful content, and poorly grounded responses.
Evaluation cannot be added shortly before launch. It needs to be part of the architecture alongside security, logging, tracing, and monitoring.
“It worked when I tried it” remains a weak production-readiness standard, even when everyone in the meeting tried it twice.
Governance has to cover the full execution path
Traditional retrieval-based AI systems usually apply an access filter when documents are retrieved.
Agentic systems create more points of exposure.
An agent may discover available tools, inspect database schemas, construct queries, combine information from several systems, call an external API, and use that response to decide what to do next.
Each step creates a separate authorization decision.
A production system needs controls over:
- Which tools the agent can see
- Which tools it can call
- What data it can discover
- Which rows and columns it can access
- Which actions must remain read-only
- How much data a query can scan
- What information must be removed or redacted
Security cannot exist only at the final chatbot response. By then, the agent may have already accessed restricted information or used unsafe data to trigger another action.
The controls need to cover the entire execution path.
For public sector and nonprofit organizations, this is especially important. These systems may interact with personal information, financial records, program data, partner information, or regulated datasets. Governance cannot be added later as a policy document. It has to be designed into the technical architecture.
Technical teams need protected time to keep up
My biggest leadership takeaway was not about one AWS service.
The pace of change is now too fast for organizations to treat AI learning as informal professional development that technical staff should absorb outside normal work.
New services, protocols, frameworks, and production patterns are appearing on a near-monthly cycle. The changes are also significant enough that watching conference recordings, videos, and podcasts is not a reliable substitute for hands-on learning.
Technical teams need dedicated work time to test tools, build small prototypes, compare approaches, and understand the security and operational implications.
Without that time, teams may know the vocabulary but lack the experience needed to make sound architecture decisions.
Protected experimentation does not mean chasing every new tool. It means creating enough space to identify what is durable, what is useful, and what is mostly a new name for an old problem.
Organizations that expect teams to learn this entirely in their free time will not move faster. They will make slower and less informed decisions later.
What technical leaders should do next
The immediate next step does not need to be a large AI program.
Start with one contained use case.
Connect an agent to a read-only reporting dataset. Run a small group of redacted documents through an intelligent processing pipeline. Build a gold-standard test set for one existing AI feature. Document the permissions, risks, and evaluation criteria before expanding access.
These experiments are small enough to control but practical enough to expose the real issues.
AWS Summit DC made it clear that agentic AI is becoming an infrastructure and governance discipline.
The product names will keep changing. The core leadership questions will not.
What should the system be allowed to do? How do we know its output is reliable? How do we trace what happened? How do we protect the data? Who owns it in production? What is the simplest architecture that solves the problem? That is where the serious work is now.
Sessions that stood out
These were the strongest sessions from the track I followed:
- Scaling Multi-Agent Architectures: Patterns for Mission-Critical AI provided the clearest framework for deciding between Agents as Tools, graphs, workflows, swarms, and A2A. The most useful advice was to start with one agent and add more only when there is a real architectural need.
- A Practitioner’s Guide to Data for Agentic AI showed what it actually takes to connect agents to enterprise databases, analytics platforms, and knowledge bases while maintaining control over data access.
- Beyond Vibe Testing: Productionalizing Mission-Driven GenAI Workloads focused on gold-standard test sets, LLM-as-a-judge evaluation, and release criteria based on the likelihood and severity of failure.
- Advanced Document Processing with Increased Accuracy and Compliance presented a practical architecture for classifying documents, choosing the right extraction method, and sending uncertain results to human review.
- Agentic Data Engineering with AWS Analytics MCP Servers demonstrated how agents can work with Redshift, Glue, EMR, and Athena without requiring a separate custom integration for each service.
- Build a Well-Architected Foundation for Scaling GenAI and Agentic Apps connected the individual use cases to the larger production architecture, including model access, memory, governance, observability, evaluation, security, and cost management.
Tools and technologies to explore
The following tools and technologies are worth putting on a technical team’s research and experimentation list:
- AWS DevOps Agent for exploring how agentic workflows may support incident analysis, operational troubleshooting, and software delivery.
- Amazon Bedrock AgentCore for agent runtime, tool access, memory, identity, policy enforcement, and execution tracing.
- Strands Agents SDK for prototyping single-agent and multi-agent patterns without building the orchestration layer from scratch.
- MCP and A2A as the two core protocols connecting agents to tools and connecting independent agents to each other.
- Amazon Redshift MCP Server and AWS Data Processing MCP Server for controlled natural-language access to data and analytics services.
- GenAI IDP Accelerator for document classification, extraction, confidence scoring, rule validation, and human review.
- Amazon Textract and Amazon Bedrock Data Automation for comparing the cost and accuracy of processing predictable documents versus complex or inconsistent ones.
- Promptfoo, Amazon Bedrock Evaluations, and Langfuse for building repeatable evaluations and tracing agent behavior in production.
- Amazon Bedrock Guardrails, AWS Lake Formation, and AgentCore Gateway interceptors for fine-grained data permissions, PII protection, prompt-attack detection, and authorization around individual tool calls.
- The AWS Well-Architected Generative AI Lens for reviewing AI systems across security, reliability, cost, operations, and governance.