Summary: PRINCE is an agentic AI system Bayer built to let preclinical researchers query decades of information buried in PDF study reports. Its lesson: production-ready agentic AI is not mainly about better models or prompts. Reliability comes from engineering both the context the model sees and the harness within which it acts. Context engineering shapes what each model receives and how context moves between specialized steps; harness engineering is the scaffolding — orchestration, tool boundaries, state persistence, retries, fallbacks, reflection loops, observability, and human review.

The challenge and the Search → Ask → Do evolution

Preclinical research data is fragmented across silos, much of it locked in unstructured PDF reports accumulated over decades — where the authoritative "gold standard" lives, even though the structured metadata is often incomplete or wrong after years of system migrations. Traditional keyword search fails on the nuance of preclinical questions. PRINCE evolved through three phases:

  1. Search — a unified gateway over structured study metadata.
  2. Ask — a natural-language Q&A system using Retrieval-Augmented Generation (RAG) over unstructured reports.
  3. Do — an active research assistant that orchestrates multi-agent workflows and drafts regulatory documents.

System architecture

The system is a conversational UI (React) over a backend orchestrated with LangGraph and served via FastAPI. Key design choices:

  • Orchestration — a LangGraph workflow progresses through clarifying intent, thinking/planning, research (RAG + Text-to-SQL), validating completeness, and writing — with deliberate pause points and feedback loops.
  • Data ecosystem — study-report vectors in OpenSearch; curated structured data via Athena; agent execution state persisted in PostgreSQL via a LangGraph checkpointer after each node; application-level state in DynamoDB.
  • Model platform — an internal GenAI platform exposes models from multiple providers (OpenAI, Anthropic, Google, open source) via a unified OpenAI-compatible endpoint, making model swaps and per-task selection easy while enforcing rate limits.
  • Resilience — retries at both the LLM-call and logical-node level, automatic fallback to an alternative model/provider, and errors fed back to agents so they can chart a different plan.
  • Observability — CloudWatch for health; Langfuse for detailed production traces and evaluation datasets; RAGAS for evaluation.

A principle runs through it: context discipline. Larger context windows did not remove the need to be selective. Early iterations that dumped everything into the prompt were harder to steer and evaluate. Instead, different stages receive different context — planning context for Think & Plan, retrieval context for the Researcher, evidence context for Reflection, synthesis context for the Writer — reducing context pollution.

The agentic workflow

Clarify intent

The first defense against ambiguity. As the system scaled across domains (toxicology, pharmacology), simple queries became ambiguous. Rather than expensive trial-and-error across all sources, PRINCE proactively asks clarifying questions and offers AI-assisted source recommendations (the user always retains final say). This "fail-fast" step constrains tools, domains, and sources before any retrieval begins.

Think & Plan — process reflection

A dedicated space to reason before acting (inspired by Anthropic's "think" tool). It performs process reflection: am I on the right trajectory toward the goal — not whether the data itself is good. As the tool count grew, overlapping concerns made tool selection error-prone; an explicit thinking step dramatically improved selection accuracy and lets the system sequence multi-step tool calls (query metadata → use study IDs to fetch reports → synthesize).

The Researcher Agent

The primary information gatherer, using a hybrid retriever over two patterns: RAG for unstructured PDFs and Text-to-SQL for structured data in Athena. As PRINCE expands across domains, the team is evolving the single Researcher into a hierarchy of domain-specific sub-agents, each owning its toolset and prompt instructions, with the top-level Researcher acting as a coordinator/router — preserving "one researcher" from the user's view while letting each domain evolve independently.

The query-time RAG pipeline is the engineering core: keyword extraction → metadata-filter generation (e.g. eq(study_id, T123456-2)) → query expansion (a smaller model generates ~5 semantically similar queries) → weighted hybrid search (0.7 semantic / 0.3 keyword, run in parallel per expanded query, pre-filtered by metadata to shrink the search space) → aggregate to ~20 chunks → rerank with a cross-encoder (bge-reranker-large) to the top 7 → final prompt → response with citations linked back to source chunks and study IDs. Text-to-SQL dynamically injects only the relevant schema, uses dynamic few-shot examples from a "semantic layer," blocks non-SELECT queries, limits to 50 rows, and self-corrects on errors up to 3 times. (An earlier LLM-review step for generated SQL was removed — it flagged valid queries as errors without a commensurate accuracy gain.)

Reflection Agent — data reflection

Complementary to Think & Plan, it performs data reflection: is the retrieved evidence sufficient and relevant to answer the question? If not, it generates targeted follow-up questions handed back to Think & Plan for further retrieval. Process reflection and data reflection are distinct: an agent can run a valid workflow yet retrieve insufficient data, or have sufficient data yet sequence poorly.

Writer Agent — synthesis and a third reflection loop

Turns retrieved evidence into the final answer — grounding every claim in context with accurate citations, honoring formatting and domain answer standards. For complex outputs it supports a short internal review loop (draft reflection) checking for missing sections or inconsistent tables. All regulatory drafts are intended for expert review; final submissions are authored and approved by qualified personnel.

These give PRINCE three complementary reflection loops — process (right path?), data (enough evidence?), and draft (complete output?) — and a practical context-engineering pattern: route the right context to the right capability at the right time, so each agent can be evaluated, debugged, and improved in isolation.

Building trust

In a regulated environment, trust means reliability, transparency, and verifiability:

  • Transparency — intermediate steps (queries formulated, tools used, source chunks) are shown to the user.
  • Citations — every answer carries hover-to-verify citations linking to the source document, page number, and exact quote.
  • Evaluation — dataset evaluations against SME reference answers (Faithfulness, Answer Relevancy, Context Relevancy, Accuracy, Semantic Similarity) when the workflow/prompts/models change; live-traffic evaluations run daily on real queries to catch hallucinations.

Engineering for resilience

The harness in practice: state persistence (the whole graph state in Postgres so execution resumes from the failed node), built-in retries, user-initiated retries (resume from the point of failure, skipping completed steps), framework-level support from LangGraph, and LLM fallbacks across providers. This makes the system less opaque and more recoverable than an unconstrained autonomous agent, giving clear control points for recovery, inspection, evaluation, and human intervention.

A separate utility uses Named Entity Recognition to extract annotations from study PDFs and enrich the structured metadata, with a confidence score per field: high-confidence fields auto-update Athena, low-confidence fields are quarantined for human review.

The broader lesson

PRINCE has been live since early 2024, with agentic integration later that year, developed iteratively — value early, accuracy first, cost optimization only after. As model capabilities improve, parts of today's harness may thin out or move into native model capabilities. But in enterprise research systems — where trust, traceability, and reviewability matter — explicit control over context, workflow state, recovery, reflection, and verification remains essential.