Trace File Format Options

DR_EVT now supports flexible trace file formats with command-line options for format, timestamp style, and timezone.

Command-Line Options

Trace Format

--trace_format {simple|lassen}

simple (default): Minimal CSV format for testing

The parser detects the mode from which columns are present - see Simulation vs Replay Modes for the full design
Simulation mode (no begin_time/end_time columns): job_submit_time, num_nodes, queue, time_limit required; actual_run_time optional (needed only for --run_time_mode actual)
Replay mode (begin_time and end_time, or begin_time and duration, present): job_submit_time, begin_time, end_time, num_nodes, exit_status, queue, time_limit required - times are historical actuals, replayed exactly, not computed by the scheduler
Column order doesn’t matter - the parser reads the header row and looks up columns by name

lassen: LLNL Lassen 33-column format

Full HPC trace format
Backward compatible with existing traces

Timestamp Format

--timestamp_format {epoch|iso}

epoch: Unix epoch seconds (integers)

Example: 0, 100, 1234567890
Fast to parse, no timezone issues
Best for synthetic test traces

iso (default): Human-readable timestamps

Example: 2024-01-15T10:30:00, 2024-01-15 10:30:00
Requires timezone specification
Used by real HPC traces

Timezone

--timezone TIMEZONE

Only used when --timestamp_format=iso

Examples:

--timezone UTC
--timezone America/New_York
--timezone America/Los_Angeles (default)
--timezone Europe/London

Usage Examples

Simple Test Trace with Epoch Times

${CMAKE_INSTALL_PREFIX}/bin/simulator test_trace.csv \
  --trace_format simple \
  --timestamp_format epoch \
  --total_nodes 100 \
  --backfill_policy easy

Simple Trace with ISO Timestamps

${CMAKE_INSTALL_PREFIX}/bin/simulator test_trace.csv \
  --trace_format simple \
  --timestamp_format iso \
  --timezone UTC \
  --total_nodes 100

Lassen Format (Default)

${CMAKE_INSTALL_PREFIX}/bin/simulator lassen_trace.csv \
  --total_nodes 795
# Uses defaults: simple format, iso timestamps, America/Los_Angeles timezone

Simple Format CSV Structure

Simulation Mode (scheduler computes start/end times)

job_submit_time,num_nodes,queue,time_limit
0,10,pbatch,100
50,10,pbatch,50
120,10,pbatch,80

Replay Mode, With Epoch Timestamps

job_submit_time,begin_time,end_time,num_nodes,exit_status,queue,time_limit
0,0,100,10,0,pbatch,100
50,100,150,10,0,pbatch,50
120,150,230,10,0,pbatch,80

Replay Mode, With ISO Timestamps

job_submit_time,begin_time,end_time,num_nodes,exit_status,queue,time_limit
2024-01-15T00:00:00,2024-01-15T00:00:00,2024-01-15T00:01:40,10,0,pbatch,100
2024-01-15T00:00:50,2024-01-15T00:01:40,2024-01-15T00:02:30,10,0,pbatch,50
2024-01-15T00:02:00,2024-01-15T00:02:30,2024-01-15T00:03:50,10,0,pbatch,80

Column Descriptions

Simple Format Columns

Columns are looked up by name in the header row, not by fixed position - any order works, and which of begin_time/end_time are present determines simulation vs replay mode (see below).

Name	Description	Required for
`job_submit_time`	When the job arrives/submits	Both modes
`num_nodes`	Number of nodes requested	Both modes
`queue`	Queue name - only `pbatch`/`pall` (and `pbatch0`-`pbatch3`) are accepted by default; see `SHOW_ALL_QUEUE` in `src/common.hpp` to change this	Both modes
`time_limit`	User-provided time limit (seconds). Accepted column-name aliases: `time_limit`, `timelimit`, `walltime`	Both modes
`begin_time`	Historical start time from trace	Replay mode only - presence of this column (together with `end_time` or `duration`) is what selects replay mode
`end_time`	Historical end time from trace	Replay mode (or use `duration` instead)
`duration`	Historical run time, as an alternative to `end_time` in replay mode	Replay mode (alternative to `end_time`)
`exit_status`	Job exit code	Replay mode
`actual_run_time`	The job’s real, historical run time (seconds); used by `--run_time_mode actual`. Accepted column-name aliases: `actual_run_time`, `duration`, `actual_duration`, `run_time`	Simulation mode, only with `--run_time_mode actual`

Column-name aliases: time_limit and actual_run_time are each detected under several accepted header names (listed above), so an existing trace can be reused as-is without editing its header - slow to do by hand on a large file. Only one alias per column is expected to actually be present in a given file; if more than one is, the first match in the order listed wins. This applies to the “simple” format only; the “lassen” format is defined by fixed column position rather than header name (see below).

Mode detection - simulation vs replay:

No begin_time/end_time columns present → simulation mode: the scheduler computes start times; how the job’s actual run time is determined is controlled separately by --run_time_mode
begin_time and (end_time or duration) present → replay mode: times are historical actuals, replayed exactly - duration = end_time - begin_time if end_time is given rather than duration directly
begin_time present without either end_time or duration (or vice versa) is rejected as an ambiguous trace format

See Simulation vs Replay Modes for the full design rationale.

Lassen Format

33-column format specific to LLNL HPC traces. Columns used:

Column 11: num_nodes
Column 23: begin_time
Column 24: end_time
Column 29: job_submit_time
Column 30: queue
Column 32: time_limit

Testing

# Create test trace
cat > test.csv << EOF
job_submit_time,begin_time,end_time,num_nodes,exit_status,queue,time_limit
0,0,100,10,0,pbatch,100
50,100,150,10,0,pbatch,50
120,150,230,10,0,pbatch,80
EOF

# Run test
${CMAKE_INSTALL_PREFIX}/bin/simulator test.csv \
  --trace_format simple \
  --timestamp_format epoch \
  --total_nodes 100 \
  --backfill_policy easy \
  --priority_policy fcfs \
  --run_time_mode actual

Expected output should show:

Job 0 starts at 0, ends at 100
Job 1 starts at 100, ends at 150
Job 2 starts at 150, ends at 230
Sequential execution (no overlap with 10 nodes each in 100-node system)