Client/Server Use Cases

This page describes ways to deploy DR_EVT clients and servers. The underlying transport is gRPC, but the important boundary is the client/server session: the client supplies arrivals and controls simulated time; a server owns one independent simulation per session. For installation and the streaming request sequence, see Client/Server Setup.

Recommended deployment models

Run dr_evt_server directly under your normal process supervisor on bare metal, or run it as a container and expose its gRPC port. In either case, clients connect to the server address and read their workload data themselves. The server needs a writable result directory and a readable CSV whose header defines the streamed record format. The container setup includes a server result mount and an interactive client with a host input-data mount.

MPI is not required for, or a primary deployment mechanism of, the gRPC client/server service. The optional MPI launcher below is a convenience for test and experiment orchestration.

One client, multiple servers

This example uses one controller with a separate session to each server. It works well for independent sites or queues, and for digital twins that route live arrivals to the appropriate simulation. It is only one topology: clients and servers may both be scaled independently. Scheduler state and nodes are not shared between sessions. The current server also needs a readable CSV header during session initialization; use --server-infile when its path differs from the client-side --jobs path.

python/grpc_multi_server.py demonstrates this arrangement. It reads the sample CSV on the client, partitions rows among repeated --server options, and prints one statistics row per server. The CSV is a convenient source of example arrivals. Servers validate its header but do not load its job rows.

python3 -m pip install grpcio grpcio-tools protobuf
python3 python/grpc_multi_server.py \
  --jobs /shared/jobs.csv \
  --server-infile /shared/jobs.csv \
  --session-name twin-west \
  --server server-a:50051 \
  --server server-b:50051

Round-robin is the default partitioning and preserves submit-time ordering within each server’s partition. Use --distribution contiguous to give each server a contiguous range instead.

Optional: MPI test and experiment launcher

python/grpc_mpi_launcher.py starts one client on rank 0 and one server on each remaining MPI rank. It discovers server addresses through MPI, passes them to the root client, and stops the servers when the client finishes. It is useful for test orchestration or controlled experiments; production deployments normally start independent bare-metal processes or containers instead.

python3 -m pip install mpi4py grpcio grpcio-tools protobuf
mpirun -np 4 python3 python/grpc_mpi_launcher.py \
  --server-binary ${CMAKE_INSTALL_PREFIX}/bin/dr_evt_server --base-port 50051 -- \
  --jobs /shared/jobs.csv --server-infile /shared/jobs.csv --total-nodes 1000

This launch starts three independent servers on ports 50051 through 50053. The rank count is one client plus the number of servers. MPI determines placement and endpoint discovery; it does not share scheduler state. The root client reads and streams the jobs; each server rank reads only the header from --server-infile, which may be a shared copy of the same file.

Use your MPI launcher’s host or hostfile options to distribute ranks across nodes. Ranks must resolve one another’s hostnames, selected TCP ports must be reachable, and the server binary must be available on every server node.

Test and Slurm validation commands are in the distributed client/server tests section.

Synchronized independent systems

python/grpc_sync_coordinator.py is a small experiment controller for multiple independent schedulers. It reads one ordinary-job trace per system and a long-form composite-job trace, streams ordinary jobs to their servers, advances all servers to each composite event time, then submits the composite fragments concurrently.

The coordinator is the only client that reads all three input streams. It owns the timing decision; the servers own only their independent scheduler state. Each row in systems.csv supplies a server-visible server_infile used for header validation. For one system, let tn be the last arrival in the ordinary batch before a composite event, ta the coordinator’s advance watermark, tc the composite event time, and t0 the first arrival in the following ordinary batch. The pre-composite batch must satisfy tn <= ta <= tc. This covers both an equal-time event (tn = ta = tc) and a strict-boundary event (tn <= ta < tc).

The t0, t1, …, tn notation names ordinary-job arrival timestamps in arrival order; the subscripts are indices, not simulation-time values. Thus tn denotes the final ordinary arrival before the composite boundary, t0 denotes the first one after it, and any t1 through t(n-1) are intervening ordinary arrivals in their respective batches.

For each composite event, the coordinator does the following in order:

1. Append and submit each system’s ordinary arrivals through tn.

2. Advance every server to the common watermark ta, then read pre-event statistics.

3. Append and submit one composite fragment to each server at tc.

4. Advance every server to tc, then read post-event statistics and record immediate, delayed, or partial starts.

5. Only after that evaluation, append the next ordinary batch. Its first arrival may be at t0 = tc or at a later time.

6. Advance again to t0 so the newly queued ordinary arrivals are evaluated. When t0 = tc, this is deliberately a second AdvanceTo(tc) call.

This observes independent schedules; it makes no cross-server reservation or rollback guarantee.

The coordinator’s integration-test coverage is documented in the distributed client/server tests section.

# Start one server for each address in systems.csv.
${CMAKE_INSTALL_PREFIX}/bin/dr_evt_server 127.0.0.1:50061
${CMAKE_INSTALL_PREFIX}/bin/dr_evt_server 127.0.0.1:50062

.venv-grpc/bin/python python/grpc_sync_coordinator.py \
  --systems python/examples/sync_systems.csv \
  --composites python/examples/composite_jobs.csv \
  --output composite-results.jsonl

systems.csv contains system_id,address,trace and optional server_infile and total_nodes. trace is the controller-side arrival source; server_infile is the server-visible file used only for header validation and defaults to trace. composite_jobs.csv contains one fragment per row with composite_id,submit_time,system_id,num_nodes,q_id,time_limit; a composite ID must name at least two systems.

The coordinator initially sends each system’s ordinary trace with AppendJobs; their original submit times remain attached to the jobs. For a composite event at tc, an ordinary batch may end at tn <= tc and the coordinator may first call AdvanceTo(ta), where ta <= tc. Each fragment is then appended with submit_time=tc and submitted, followed by AdvanceTo(tc), which evaluates the newly arrived same-time work. The bundled coordinator selects ta = tc; a smaller ta is useful when deliberately testing a strict boundary before the composite event. AppendJob is used for the one-fragment-per-system example; AppendJobs can submit a chronologically ordered batch when a system has multiple arrivals. In an incremental stream, the following ordinary batch begins at t0, with tc < t0; the bundled example instead pre-submits its complete ordinary trace at initialization.

The JSON Lines output records node counts before and after each fragment, and marks partial_start when only some fragments appear to start immediately. It is observational: the coordinator does not reserve, cancel, or roll back work on any server.