DR_EVT User Guide

Introduction

DR_EVT (Discrete Resource Event Modeling) is a high-performance HPC job scheduler simulator implementing SLURM-style backfilling algorithms. Use it to:

• Evaluate scheduling policies (EASY vs Conservative backfill)

• Test priority policies (FCFS, SJF, LJF)

• Compare run time estimation strategies

• Analyze HPC workload traces

Quick Start

Basic Usage

${CMAKE_INSTALL_PREFIX}/bin/simulator my_trace.csv \
  --total_nodes 100 \
  --backfill_policy easy \
  --priority_policy fcfs \

Example Output

Loaded 30 jobs from trace
Running simulation with 100 nodes

Job 0 submitted at 0 (50 nodes)
Job 0 started at 0 (50 nodes)
...
Job 29 ended at 2010

=== Simulation Statistics ===
Total jobs: 30
Jobs completed: 30
Average wait time: 227 sec
Makespan: 2010 sec

Trace File Formats

Simple Format (Recommended for Testing)

7-column CSV format:

job_submit_time,begin_time,end_time,num_nodes,exit_status,queue,time_limit
0,0,100,10,0,pbatch,100
50,100,150,10,0,pbatch,50

Columns:

1. job_submit_time - When job arrives (required)

2. begin_time - Historical start time (required, for duration calculation)

3. end_time - Historical end time (required, for duration calculation)

4. num_nodes - Number of nodes requested (required)

5. exit_status - Job exit code (optional)

6. queue - Queue name, must be “pbatch” or “pall” (optional)

7. time_limit - User-provided time limit in seconds (optional)

Usage:

${CMAKE_INSTALL_PREFIX}/bin/simulator trace.csv \
  --trace_format simple \
  --timestamp_format epoch \
  --total_nodes 100

Lassen Format (LLNL HPC Traces)

33-column format used by LLNL Lassen supercomputer:

• Full HPC trace format

• Includes user info, job scripts, etc.

• Backward compatible with existing traces

Usage:

${CMAKE_INSTALL_PREFIX}/bin/simulator lassen_trace.csv \
  --trace_format lassen \
  --timestamp_format iso \
  --timezone America/Los_Angeles \
  --total_nodes 795

Timestamp Formats

Epoch (Unix Time)

Integer seconds since 1970-01-01:

0,0,100,10,0,pbatch,100      # Time 0, 100 seconds
50,100,150,10,0,pbatch,50    # Time 50, 100, 150

Usage: --timestamp_format epoch

Advantages:

• Simple integer format

• No timezone issues

• Fast parsing

• Best for synthetic test traces

ISO (Human-Readable)

ISO 8601 format with timezone:

2024-01-15T00:00:00,2024-01-15T00:00:00,2024-01-15T00:01:40,10,0,pbatch,100

Usage:

--timestamp_format iso --timezone America/Los_Angeles

Advantages:

• Human-readable

• Used by real HPC traces

• Timezone-aware

Supported timezones: Any POSIX timezone (UTC, America/New_York, etc.)

Scheduler Policies

Backfill Policies

EASY Backfill (Default)

Algorithm:

1. First job in queue gets guaranteed start time (reservation)

2. Later jobs can “backfill” if they:

   • Fit in available resources NOW

   • Won’t delay first job’s reservation

Characteristics:

• Simple, fast

• Good resource utilization

• Favors first job in queue

• Used by most HPC centers

Usage: --backfill_policy easy

Example:

Job 0 (80 nodes, 1000s) - First in queue, gets reservation
Job 1 (10 nodes, 100s) - Arrives later, backfills immediately
Job 2 (15 nodes, 100s) - Can't fit, waits for Job 0

Conservative Backfill

Algorithm:

1. ALL queued jobs get guaranteed start times

2. Backfilling jobs cannot delay ANY reservation

Characteristics:

• More complex scheduling

• Guarantees to all jobs

• Lower risk of starvation

• May have lower utilization

Usage: --backfill_policy conservative

When to use: Research comparing policies, fairness studies

Priority Policies

FCFS (First-Come-First-Served) - Default

Jobs processed in submission order.

Usage: --priority_policy fcfs

Characteristics:

• Simple, fair

• No starvation

• May waste resources with large jobs

SJF (Shortest-Job-First)

Prioritize jobs with shortest run time.

Usage: --priority_policy sjf

Characteristics:

• Maximizes throughput

• Minimizes average wait time

• Can starve long jobs

• Requires run time estimates

LJF (Longest-Job-First)

Prioritize jobs with longest run time.

Usage: --priority_policy ljf

Characteristics:

• Useful for special scenarios

• Can increase average wait time

• Research/comparison purposes

Run Time Modes

Controls how the job’s actual execution length is determined in simulation.

Note: The scheduler always uses time_limit for planning decisions (realistic behavior). This setting only affects how long jobs actually run in the simulation.

actual (Default)

Read the job’s real run time from the trace’s actual_run_time column.

Usage: --run_time_mode actual (or -r actual)

Characteristics:

• Most realistic - uses historical execution times

• Jobs run for their actual observed duration

• Scheduler still plans using time_limit

• Best for realistic simulations

distribution

Sample from a statistical distribution around time_limit * scale.

Usage: --run_time_mode distribution

Characteristics:

• Adds realistic variability

• Supports normal, lognormal, uniform distributions

• Capped at time_limit (scheduler kills jobs at limit)

• Useful for synthetic workloads

limit (Debug Mode)

Jobs run for exactly their time_limit.

Usage: --run_time_mode limit (or -r limit)

Characteristics:

• Unrealistic but predictable

• Debug and test friendly

• Every job uses its full time allocation

• Useful for scheduler algorithm testing

Command-Line Options

Required Options

--total_nodes N          # Total system nodes (required)

Input/Output Options

-i, --infile FILE        # Input trace file (required, unless --infile_list is used instead)
-L, --infile_list FILE   # File listing multiple trace files, one per line - progressive loading (mutually exclusive with --infile)
-o, --outfile FILE       # Output file (default: based on input name)
-j, --max_jobs N         # Max jobs to simulate (default: all)
-t, --max_time T         # Max simulation time to run (default: unlimited)
-R, --resource_trace FILE       # Write resource usage trace to file
-H, --resource_history_capacity SIZE  # Initial resource-history buffer capacity (default: 0 = 2x loaded jobs, floored at 4096)

Trace Format Options

-f, --trace_format FORMAT     # Trace format: simple|lassen (default: simple)
-T, --timestamp_format FORMAT # Timestamp: epoch|iso (default: iso)
-z, --timezone ZONE           # Timezone for ISO timestamps (default: America/Los_Angeles)

Scheduler Options

-b, --backfill_policy POLICY   # Backfill: easy|conservative|none (default: easy)
-p, --priority_policy POLICY   # Priority: fcfs|sjf|ljf (default: fcfs)
-q, --queue_impl IMPL          # FCFS wait queue: circular|deque|multimap|block (default: circular)
-Q, --block_size SIZE          # Block size when queue_impl=block, power of 2 (default: 128)
-A, --wait_queue_capacity SIZE   # Initial capacity for queue_impl=circular (default: 0 = size of trace)
-G, --wait_queue_overflow POLICY # abort|grow if circular capacity exceeded (default: grow)

Job Store Options

-K, --job_store_capacity SIZE     # Initial job-record store capacity (default: 0 = size of trace)
-W, --job_store_overflow POLICY   # abort|grow if job_store_capacity exceeded (default: grow)
-m, --check_memory_pressure FRACTION  # Refuse to grow the job store past FRACTION of available memory (0 < FRACTION <= 1; disabled unless given)

Only --job_store_capacity actually bounds memory when used with --infile_list (single-file mode always grows to fit the whole trace); --check_memory_pressure is a separate, independent check against real available memory - see Command-Line Options and Trace as a streaming-ready state container for the full formula and rationale.

Run Time Options

-r, --run_time_mode MODE       # How jobs actually run: actual|distribution|limit (default: actual)
-D, --run_time_distribution    # Distribution type: normal|lognormal|uniform (for mode=distribution)
-S, --run_time_scale FACTOR    # Scale factor for run times (for mode=distribution)
-V, --run_time_stddev FACTOR   # Standard deviation factor for run time sampling (for mode=distribution)

Other Options

-h, --help              # Show help message
-s, --seed N            # Random seed (default: 0)
-c, --config FILE       # Load parameters from a protobuf config file (requires Protobuf support)
-M, --msec_output       # Millisecond-precision timestamps in output (default: whole seconds)
-v, --verbose           # Enable verbose output for debugging

Usage Examples

Example 1: Basic Test

Test with synthetic trace:

${CMAKE_INSTALL_PREFIX}/bin/simulator tests/test_traces/unit/timestamp_epoch_simple.csv \
  --trace_format simple \
  --timestamp_format epoch \
  --total_nodes 100 \
  --backfill_policy easy \
  --priority_policy fcfs \
  --run_time_mode limit \

Example 2: Real HPC Trace

Simulate Lassen trace:

${CMAKE_INSTALL_PREFIX}/bin/simulator lassen_trace.csv \
  --trace_format lassen \
  --timestamp_format iso \
  --timezone America/Los_Angeles \
  --total_nodes 795 \
  --backfill_policy easy \
  --priority_policy fcfs \

Example 3: Policy Comparison

Compare EASY vs Conservative:

# EASY backfill
${CMAKE_INSTALL_PREFIX}/bin/simulator trace.csv --backfill_policy easy -o results_easy.txt

# Conservative backfill  
${CMAKE_INSTALL_PREFIX}/bin/simulator trace.csv --backfill_policy conservative -o results_conservative.txt

# Compare results
diff results_easy.txt results_conservative.txt

Example 4: Priority Policy Study

Compare FCFS vs SJF:

# FCFS
${CMAKE_INSTALL_PREFIX}/bin/simulator trace.csv --priority_policy fcfs -o results_fcfs.txt

# SJF
${CMAKE_INSTALL_PREFIX}/bin/simulator trace.csv --priority_policy sjf -o results_sjf.txt

Example 5: Run Time Estimation Impact

Compare different run time modes:

# Using actual run times from trace (most realistic, default)
${CMAKE_INSTALL_PREFIX}/bin/simulator trace.csv --run_time_mode actual

# Using time limits (debug/test mode)
${CMAKE_INSTALL_PREFIX}/bin/simulator trace.csv --run_time_mode limit

Understanding Output

Job Events

Job 0 submitted at 0 (50 nodes)
  Resources allocated: 50 nodes, 50/100 remaining
Job 0 started at 0 (50 nodes)
Job 0 ended at 500
  Resources freed: 50 nodes, now 100/100 free

Interpretation:

• Job arrives at time 0

• Needs 50 nodes

• Starts immediately (resources available)

• System has 50/100 free after allocation

• Job completes at time 500

• All resources returned

Backfill Messages

Job 1 submitted at 10 (15 nodes)
  Backfill failed: job 1 needs 15 nodes, only 10 available

Interpretation: Job can’t backfill due to insufficient resources

Statistics

=== Simulation Statistics ===
Total jobs: 30
Jobs submitted: 30
Jobs completed: 30
Current time: 2010
Total nodes: 100
Average wait time: 227 sec
Average turnaround time: 500 sec
Makespan: 2010 sec

Metrics:

• Wait time: Time from submission to start

• Turnaround time: Time from submission to completion

• Makespan: Time from first submit to last completion

Common Workflows

1. Create Test Trace

cat > my_test.csv << EOF
job_submit_time,begin_time,end_time,num_nodes,exit_status,queue,time_limit
0,0,100,50,0,pbatch,100
10,10,60,10,0,pbatch,100
20,20,60,5,0,pbatch,100
EOF

2. Run Simulation

${CMAKE_INSTALL_PREFIX}/bin/simulator my_test.csv \
  --trace_format simple \
  --timestamp_format epoch \
  --total_nodes 100 \
  --backfill_policy easy

3. Analyze Results

Check output for:

• All jobs completed

• Resource utilization

• Wait times

• Backfill opportunities

4. Compare Policies

# Run with different policies
for policy in easy conservative; do
  ./simulator my_test.csv \
    --backfill_policy $policy \
    -o results_$policy.txt
done

# Compare
diff results_easy.txt results_conservative.txt

Troubleshooting

Problem: “Loaded 0 jobs from trace”

Cause: Queue filtering - only “pbatch” and “pall” queues accepted

Solution:

• Use “pbatch” in queue column

• Or set SHOW_ALL_QUEUE=1 in common.hpp and rebuild

Problem: “Job event times are incorrect”

Cause: Submit time > begin time in trace

Solution: Ensure in trace: submit_time ≤ begin_time ≤ end_time

Problem: “Resource over-subscription”

Cause: Bug in scheduler (should not happen with current code)

Solution:

• Check total_nodes matches system

• Verify job node counts

• Report bug with trace file

Problem: Very long simulation time

Cause: Large trace or infinite loop

Solution:

• Use --max_jobs to limit

• Check for scheduler deadlock

• Monitor with top command

Performance Tips

For Large Traces

1. Limit job count during testing:

   ./simulator large_trace.csv --max_jobs 100
   

2. Use simple trace format:

   • Faster parsing than Lassen format

   • Convert large traces to simple format first

3. Don’t pass -v/--verbose:

   • Off by default; only enable it when you actually need the per-event trace it prints, since it’s not free

4. Bound job-store memory with --infile_list:

   • Single-file mode (--infile) always grows its job-record store to fit the whole trace, regardless of --job_store_capacity

   • For a trace too large to comfortably hold in memory at once, split it into several submit-time-sorted files and use --infile_list (progressive loading) instead - see Command-Line Options

   • Add --check_memory_pressure FRACTION (e.g. 0.8) to refuse outright rather than risk exhausting memory, if a batch would push usage past that fraction of what’s actually available

Expected Performance

• Small traces (< 100 jobs): < 10ms

• Medium traces (100-1000 jobs): 10-100ms

• Large traces (1000-10000 jobs): 100ms-1s

• Very large (10000+ jobs): 1-10s

Advanced Topics

Creating Custom Traces

import pandas as pd

# Generate synthetic trace
jobs = []
for i in range(100):
    submit_time = i * 10
    duration = random.randint(50, 500)
    nodes = random.choice([5, 10, 20, 50])
    
    jobs.append({
        'job_submit_time': submit_time,
        'begin_time': submit_time,  # Will be rescheduled
        'end_time': submit_time + duration,
        'num_nodes': nodes,
        'exit_status': 0,
        'queue': 'pbatch',
        'time_limit': duration + 100
    })

df = pd.DataFrame(jobs)
df.to_csv('synthetic_trace.csv', index=False)

Batch Processing

#!/bin/bash
# Compare all policies on multiple traces

for trace in traces/*.csv; do
  for policy in easy conservative; do
    for priority in fcfs sjf ljf; do
      output="results/$(basename $trace .csv)_${policy}_${priority}.txt"
      ./simulator $trace \
        --backfill_policy $policy \
        --priority_policy $priority \
        -o $output
    done
  done
done

Extracting Metrics

# Extract average wait times from multiple runs
grep "Average wait time" results/*.txt | \
  awk '{print $5}' | \
  awk '{sum+=$1; n++} END {print "Mean:", sum/n}'

Best Practices

1. Start small: Test with 10-100 jobs before large traces

2. Validate traces: Check job counts, resource bounds

3. Use version control: Track traces and results

4. Document experiments: Note policy combinations tested

5. Compare baselines: Always compare against FCFS+EASY

6. Check completion: Verify jobs_submitted == jobs_completed

References

• Testing Guide - Running test suite and test philosophy

• CLI Options - Complete command-line reference

• Backfilling Algorithms - EASY and CONSERVATIVE specifications

• Streaming API - Online simulation API

Support

For issues or questions:

1. Check troubleshooting section

2. Review test cases in test_traces/

3. Check documentation files

4. File issue with trace file and command used