EnterpriseShift: Role-based work in a connected enterprise world

Introduction

What does it mean to benchmark enterprise work? We began by asking what work the benchmark should represent and what would count as doing it well.

Enterprise work unfolds within an organization whose history, rules, and activity extend beyond any request. Building an environment around a task risks doing some of the agent’s work: selecting relevant context, narrowing dependencies, and defining the problem’s boundaries.

For EnterpriseShift, we defined the responsibilities to evaluate and built a connected operational world grounded in real activity data. Tasks sit within that world, alongside unrelated activity. The agent must discover what matters, respect its authority, and determine when its responsibility is complete.

This post follows that process, from scoping work and constructing the world to designing rewards, including the lessons we learned when those pieces did not agree.

Review of existing enterprise benchmarks

Before building EnterpriseShift, we reviewed a broad range of available enterprise benchmarks to understand how they represent work. That review helped us identify the dimensions that shaped our design: where the world comes from, how tasks relate to it, how jobs and authority are defined, whether work can depend on other actors, and how success is measured.

Our emphasis is on connecting these layers. Job definitions establish responsibilities, authority, and the boundaries between people’s work. Operational data grounds the world in which they act. Policies guide task formation and evaluation. When completing an episode requires another person’s action, the environment supports handing off work, waiting, and continuing.

The next section explains how we scoped those jobs and responsibilities, and how they determined the world we needed to build.

Building the organizational world

A world-first benchmark needs a clear account of the organization whose work it represents. We needed to define who works there, how responsibilities are divided, and which decisions each person is authorized to make. Those definitions would guide the policies, records, and systems we built around them.

Defining the organization and jobs

We scoped the organization through explicit job definitions. Each job bundles several roles, with responsibilities, activities, and authorizations specified for each. The definitions also identify work that belongs to other jobs, when escalation is required, and which systems the position may access.

The boundaries are as important as the responsibilities. In our organization, for example, the Change Manager can authorize a change but cannot implement it. The implementing engineer owns a different part of the work. Making that distinction explicit gives task authors a basis for deciding what the agent should complete itself and what requires another actor.

The organizational specification contains thirteen positions spanning service desk, technical support, incident and change management, supplier coordination, automation, and identity administration. Each definition connects the job to its teams and documents its interfaces with neighboring positions. Where responsibilities overlap, a responsibility-assignment table identifies who performs the work and who is accountable for the outcome.

ITSM practitioners reviewed the job definitions to refine these boundaries and resolve ambiguities. The resulting documents provide a shared reference for world construction, policy development, task authoring, and evaluation. They establish whose responsibility an agent assumes and the limits within which successful work must happen.

Establishing policies and procedures

Policy generation starts with three inputs: the defined jobs and roles, established organizational facts drawn from activity logs and the organizational specification, and a policy structure prepared by domain experts.

The structure determines which practices the corpus covers, such as incident management, change enablement, and knowledge management. Within each practice, it organizes the documentation into policies, procedures, runbooks, and templates. This gives generation an explicit scope and connects organizational rules to the instructions and records needed to carry them out.

Within that structure, we use approved ITSM sources and organizational facts to develop the content. Job definitions establish responsibilities and authority. Organizational facts supply context such as teams, services, systems, and operating patterns. Shared models define concepts used across practices, including priorities, lifecycles, and approval routes, so individual documents draw on a common foundation.

Grounding remains traceable: organizational claims reference defined facts, while guidance drawn from approved sources carries supporting citations. Missing information is recorded as a gap rather than silently filled with a plausible assumption.

We then review the connections between documents, jobs, and environment capabilities. A runbook must stay within its assigned job’s authority, use operations the environment supports, and agree with the procedures it implements. Findings feed back into the relevant definitions and documents, keeping the corpus aligned with the organization it governs.

Building the operational world

With jobs and policies defined, we construct the records that make the organization operational. We ground this world in the BPI Challenge 2014 dataset, a public collection of IT service-management records from Rabobank’s ICT organization. Its linked incidents, activities, interactions, and changes provide evidence of how work moves through an organization: assignments, handoffs, delays, and state transitions.

These records supply the operational history we reconstruct at a chosen moment, including unfinished work. The dataset does not provide a complete environment, so we supplement it with organizational definitions, policies, staff identities, and explanatory text. The seed build combines four fixed inputs: the BPI records, organizational specification, policy corpus, and environment implementation.

Three principles guide the build. Preserve historical messiness, including mistakes and policy violations, while flagging it explicitly. Hold out the future, so unfinished work cannot reveal its eventual outcome. Track provenance, distinguishing replayed facts, derived records, authored additions, and reference data.

Stage 0: Establish the contracts. Before generating records, we define what the world contains, where each part comes from, and what the environment can represent. These decisions become a data dictionary, grounding table, environment contract, and list of input inconsistencies. We also choose the world’s snapshot time by measuring the work available. Near the BPI export boundary, only 18 incidents remained open; moving into the observation window exposed roughly 1,250. The choice determines whether the agent enters an organization with meaningful work in flight.

Stage 1: Construct the canonical world. We build connected record families together, such as incidents and their journals. Each source requires a different check: replayed records must preserve BPI histories, derived records must cite supporting evidence, authored additions must satisfy organizational constraints, and reference records must match their specifications.

Deterministic rules reconstruct operational facts. Model-written text banks supply missing prose, with each variant declaring the facts it asserts and appearing only where those facts hold. With inputs and text banks fixed, deterministic selection makes the build reproducible. Historical violations remain visible, and we never invent permissions to make an actor’s past behavior appear authorized.

Stage 2: Project into the services. We load the canonical records into ServiceNow, Entra ID, Teams, and SharePoint, then boot the environment and compare API read-back with the intended seed. This step exposed limits we had discovered too late: supporting the data required splitting journal records, bounding the exposed history to open work and recent activity, and changing the environment. A valid database alone does not establish that the agent can observe the intended world.

Stage 3: Validate the result. Statistical checks measure workload, repetition, and coverage. Lifecycle checks test record histories against formal state models. Sampled reviews examine operational coherence from several perspectives. We also test the reviewers: review packets contain ten planted defects, and a clean review counts only if at least eight are detected. This gate followed an early trial that reported issues while missing both planted defects.

When validation reveals a generation defect, we correct its source and rebuild. Faithful historical irregularities remain flagged. This preserves the distinction between an imperfect organization the agent must work within and an incorrectly constructed environment.

Building the executable environment

The populated world becomes an RL environment when an agent can inspect it, act within its rules, and observe the consequences. Our runtime exposes six simulated enterprise systems: ServiceNow, Microsoft Entra ID, Entra ID Governance, Teams, SharePoint, and StatusGator. The agent discovers available endpoints through a search service and interacts with them over HTTP using vendor API structures.

These services operate on a shared world state. Requests read and modify records, while service rules govern which changes are accepted. For example, resolving an incident requires resolution information, and a refused write leaves the record unchanged. Unsupported operations return an explicit error. This makes the implemented behavior of the systems part of the task.

Identity and authority shape what the agent can do. Tasks can assign the agent a named employee’s identity and token, with permissions grounded in job definitions. ServiceNow and Entra enforce their respective access models, and accepted writes are attributed to the caller. A policy can therefore require an approval that the agent lacks permission to provide itself. Recognizing that boundary and involving the appropriate colleague becomes part of successful work.

Work can depend on another actor. Some tasks attach workflows that react to changes in the world. After requesting an approval, for example, the agent must yield through a wait operation before the approving actor can respond. These workflows advance only when the agent waits, making coordination explicit and reproducible. Named colleagues act through the same APIs, with their own permissions and attribution. The sequence becomes: request or hand off work, wait, inspect the result, and continue. Waiting allows pending reactions to run; it does not simulate elapsed time.

When a task requires information held by a person, an optional simulated colleague answers through ticket comments. This runs separately from the workflows, polling the ticket journal and posting replies. Those exchanges become ordinary records in the world, allowing the agent to use the same interface for both operational work and communication.

The runtime preserves evidence of what happened. For tasks with workflows, an event queue records changes in order and identifies their sources and actors. This matters because identical final records can conceal different behavior: obtaining approval before implementation and adding approval afterward may leave the same final state.

The runtime saves the world throughout the episode. Once the agent stops, that state is passed to an isolated verifier. Together, the records and event history provide evidence for evaluating the outcome, the authority exercised, and any required coordination.

Task formulation and verification

A realistic environment does not, by itself, define a valid benchmark task. We still need to establish what work the agent owns, where its responsibility ends, and what counts as completing it correctly. The task and its verifier must agree on those answers, and those answers must reflect the organization’s policies.

A reference solution passing its verifier establishes agreement between two artifacts. It does not establish that either represents correct enterprise work. Both can share the same mistaken interpretation of an approval requirement, a handoff, or a completion condition.

Our approach makes the definition of successful work explicit and reviewable before translating it into verification checks.

Defining reviewable Work Actions

We represent policy-defined cases as Work Actions. Each specifies the responsible role, the conditions under which the work applies, and the effects it should leave in the world. The definition links these requirements to supporting policy quotations and expresses them in both readable and executable form.

For example, a Change Manager reviewing an incomplete request may need to return it for additional evidence. The corresponding Work Action identifies when that response applies, what the decision note must communicate, and what must remain unchanged. Creating an approval or completing the initiator’s missing justification would exceed the required work.

We separate cases where policy requires different responses. Approving a complete request and returning an incomplete one become distinct Work Actions, each with its own preconditions and effects. This gives a domain expert a concrete question to assess: given these conditions, does this definition accurately represent the work required by policy?

That review also constrains what we grade. Choices left to the worker’s judgment should not become arbitrary requirements. Exact wording is checked where policy mandates it; otherwise, free text is evaluated against the specific claims it must communicate.

Forming tasks within the world

A task instantiates and combines Work Actions against particular records, people, and situations. Job definitions establish the agent’s responsibility. The selected Work Actions describe the work within that responsibility, including dependencies on colleagues.

Each episode uses the shared world with explicit task-specific changes that establish its starting conditions. The agent receives an assignment and must inspect the surrounding records to determine what applies. An episode can include several work items with different outcomes: one ready for a decision, another requiring a handoff, and another that should remain untouched.

Completion follows the responsibility being evaluated. The agent’s work may end with a documented escalation, or it may require waiting for another actor and continuing afterward. Those boundaries must be represented in the task definition so that the verifier neither rewards incomplete work nor demands actions belonging to someone else.

Compiling and validating verification

The verifier is compiled from the instantiated Work Actions. Reference effects are executed through the environment’s APIs, under the appropriate identities, so service rules and side effects contribute to the expected result.

Checks evaluate the resulting records and required preservation of other state. Where the policy requires ordering, authorship, or handoffs, they also examine event history. This allows different valid tool sequences while retaining the constraints that matter to the work.

The translation itself needs testing. A reference solution must pass, and doing nothing must fail. Deliberately altered trajectories test whether the verifier rejects missing effects, incorrect values, unauthorized changes, and violations of required order. Free-text judgments are scoped to narrow claims, with their results recorded for inspection.

The design separates two sources of trust. Domain review establishes whether a Work Action represents correct work. Compiler and task validation test whether that definition is faithfully enforced. Reusing reviewed Work Actions across task instances reduces repeated review, while a correction to the compiler can improve every affected task.

Example tasks

The tasks place the agent inside an operational assignment. It must discover the relevant records, apply the organization’s policies, and determine when its responsibility is complete.

Following up on yesterday’s fixes

The agent works as an Application Support Engineer in Online & Mobile Banking. It previously applied fixes to four incidents and must follow up with the callers through Teams. Where a failure has returned, it must reopen the incident for further investigation.

This episode tests whether the agent can connect historical notes with new information from people and take the appropriate next step. Success includes keeping unrelated incidents in the team’s queue untouched. Task definition

Managing a change and access queue

The agent covers Change Manager and Identity & Access Administrator responsibilities for a morning shift. Its assignment names no individual records. It must discover which requests need attention and distinguish their requirements: a change requiring a board decision, another eligible for direct authorization, an incomplete request that must be returned, and an access item still waiting on another team.

Success requires different responses within one episode, including coordinating with colleagues and recognizing when no further action is appropriate. Task definition

Taking ownership of a supplier desk

The agent begins a morning as Supplier Coordinator for Application Vendor Liaison. It must consult the policy library, inspect the desk’s work, and carry out the required start-of-day responsibilities. Supplier channels are available for communication, and the agent can wait for responses.

The assignment also establishes a boundary: hardware supplier work belongs to a colleague. Successful completion means handling the application vendor desk’s responsibilities without taking over neighboring work. Task definition

The EnterpriseShift repository provides the evaluation harness and instructions for obtaining and running the task packages.

Conclusion and next steps

Benchmarking enterprise work requires a clear account of the organization in which that work happens. An agent must find relevant evidence among ongoing activity, understand its responsibility, respect its authority, and recognize when progress depends on someone else.

EnterpriseShift brings these requirements together through a connected operational world, explicit job definitions, and policy-grounded tasks. Reviewable Work Actions connect the organization’s expectations to the verifier, making the definition of successful work open to inspection.

We are now scaling the tools and methods for creating and validating tasks in these settings, alongside training runs to study how agents learn and perform in these environments.

References

1. Corecraft, HANDBOOK.md, EnterpriseOps-Gym, τ-Banking (τ-Knowledge), AutomationBench
2. Job definitions and review process
3. Runtime documentation, Event queue and workflows

Citation

Contributors: Vibrant Labs, Shahul, Jithin, Andrew, Ankit, and Ani.

If you use EnterpriseShift in your work, please cite:

@misc{vibrantlabs2026enterpriseshift,
  author = {{Vibrant Labs} and Shahul and Jithin and Andrew and Ankit and Ani},
  title = {{EnterpriseShift}: Role-based work in a connected enterprise world},
  year = {2026},
  month = nov,
  url = {https://vibrantlabs.com/research/enterprise-shift}
}