DR_EVT: HPC Job Scheduler Simulator

Discrete Event-Driven Simulator for High-Performance Computing Job Schedulers

DR_EVT simulates HPC job scheduling policies with EASY and CONSERVATIVE backfilling implementations. Uniquely supports online simulation via gRPC, enabling coordinated multi-cluster simulations in a distributed fashion and digital-twin scheduler interacting in real-time.

Scheduler behavior is verified against a from-scratch Python reference implementation (consistency check between implementations, not independently derived ground truth).

Getting Started

• Quick Start Guide - DR_EVT Backfilling Scheduler
  • Overview
  • Features
  • Build Instructions
  • Usage
  • Output
  • Trace File Format
  • Understanding the Algorithms
  • Troubleshooting
  • Development
  • Documentation
  • Citation
  • License
  • Support
  • What’s Next?
• Installation
  • Prerequisites
  • Building from Source
  • Installation
  • Python Environment
  • Verification
  • Troubleshooting
  • Next Steps
• Tutorial: Your First Simulation
  • Step 1: Prepare a Test Trace
  • Step 2: Run the Simulation
  • Step 3: Understand the Output
  • Step 4: Analyze Results
  • Step 5: Visualize (Optional)
  • Understanding Backfilling
  • Common Variations
  • Next Steps
  • Exercises

User Guide

• DR_EVT User Guide
  • Introduction
  • Quick Start
  • Trace File Formats
  • Timestamp Formats
  • Scheduler Policies
  • Command-Line Options
  • Usage Examples
  • Understanding Output
  • Common Workflows
  • Troubleshooting
  • Performance Tips
  • Advanced Topics
  • Best Practices
  • References
  • Support
• Command-Line Options
  • Basic Usage
  • Input/Output Options
  • System Configuration
  • Scheduling Policies
  • Trace Format Options
  • Simulation Mode Options
  • Limit Options
  • Configuration File Option
  • Debug Options
  • Common Usage Patterns
  • See Also
• Trace File Format Options
  • Command-Line Options
  • Usage Examples
  • Simple Format CSV Structure
  • Column Descriptions
  • Testing
  • See Also
• Protocol Buffer Configuration
  • Why Use Protobuf Config?
  • Configuration File Format
  • Configuration Options Reference
  • Command-Line Override
  • Common Configurations
  • Protocol Buffer Schema
  • Validation
  • See Also
• gRPC Client/Server Setup
  • Architecture
  • Building with gRPC Support
  • Running the Server
  • Running the Client
  • Network Configuration
  • Troubleshooting
  • See Also

Algorithm & Testing

• Backfilling Algorithms - Reference Implementation
  • Table of Contents
• EASY Backfilling
  • Overview
  • Visual Overview
  • Data Structures
  • Algorithm
  • Step 1: Process FCFS Head
  • Step 2: Calculate Reservation
  • Step 3: Try Backfilling
  • Early Completion Handling
  • Edge Cases & Clarifications
  • Complete Example Trace
  • Questions to Verify Understanding
  • Reference
• Conservative Backfilling
  • Overview
  • Visual Overview
  • Algorithm
  • Step 2: Calculate Reservations (Shadow Times) for ALL Waiting Jobs
  • Step 3: Conservative Backfilling
  • Conservative Backfilling Example
  • Implementation Considerations
  • Performance Comparison
  • When to Use Conservative Backfilling
  • Reference
• Comparison
  • Summary Table
  • Algorithm Choice Decision Tree
  • DR_EVT Implementation
• Testing Guide for DR_EVT Scheduler
  • Overview
  • Quick Navigation
  • How to Run Tests
  • Comprehensive Tests
  • Unit Tests
  • Feature Tests
  • Scale Tests
  • Replay Tests
  • Resource History Tests
  • Job Store Tests
  • Append-Job Tests
  • Progressive Loading Tests
  • Streaming API Tests
  • Configuration Tests
  • Queue Implementation Testing
  • Column Alias Tests
  • Run Time Mode Tests
  • Test Summary
  • Test File Formats
  • Generating Expected Outputs
  • Adding New Tests
  • Common Test Patterns
  • Troubleshooting
  • Conservative Backfilling Tests
  • References
  • Key Principles

APIs

• Streaming API Documentation
  • Overview
  • Core Concepts
  • API Methods
  • Usage Patterns
  • Complete Example
  • Implementation Details
  • Testing
  • Limitations
  • See Also
• DR_EVT Python API
  • Overview
  • Installation
  • Quick Example
  • Configuration Parameters
  • Missing Parameters in Python Bindings
  • Enumerations
  • Streaming API
  • Monitoring API
  • Batch Mode API
  • Use Cases
  • Performance
  • Testing
  • Examples
  • Troubleshooting
  • API Reference
  • Comparison: Python vs C++ API
  • Contributing
  • See Also
• gRPC Client/Server Guide
  • Overview
  • Building
  • The service definition
  • Running the server and client
  • Multi-client/multi-server: the MPI test harness
  • Testing

Development

• Developer Documentation
  • What Goes Here
• Terminology - Clarification
  • The Confusion
  • Correct Terminology
  • File Naming Convention
  • Code/Script Names
  • Examples
  • Summary Table
  • Migration Notes
  • Why This Matters
  • Verification Terminology
• Block Queue Wait Queue Implementation
  • Quick Summary
  • Performance Results (10K jobs, 500 nodes)
  • The U-Shaped Curve
  • Architecture
  • Why Deque Still Wins
  • Testing
  • Usage
  • Design Rationale
  • Implementation Details
  • Lessons Learned
  • Recommendations
  • Files
  • References
• Block Queue Testing Guide
  • Overview
  • Test Files
  • Test Traces
  • Generating Test Traces
  • Key Findings
  • Recommendation
  • Running All Tests
  • Continuous Integration
• Circular Buffer Wait Queue Implementation
  • Quick Summary
  • Performance Results (10K jobs, 500 nodes, average of 3 trials)
  • Why It’s Faster Than Deque
  • Why It’s Faster Than Block Queue
  • The Fixed-Capacity Trade-off
  • Testing
  • Usage
  • Recommendations
  • Files
  • References
• ReadTheDocs Setup Guide
  • Overview
  • Configuration Files
  • Documentation Structure
  • Building Locally
  • ReadTheDocs Project Setup
  • Version Management
  • Customization
  • Troubleshooting
  • Maintenance
  • Migration Notes
  • References
• Design Decisions
  • Documents
  • Purpose
  • Adding New Decisions
• Simulation vs Replay Modes - Design
  • Problem Statement
  • Design Goals
  • Trace Format Specifications
  • Duration Determination in Simulation Mode
  • Data Structure Changes
  • Simulation Logic Updates
  • Command Line Interface
  • Data Flow
  • Validation
  • Implementation Plan
  • Benefits
  • Migration Path
• Timezone Offset Support Design
  • Status: Implementation Complete (Untested)
  • Overview
  • Design Principles
  • Implementation Details
  • Examples
  • Benefits
  • Testing Plan
  • Remaining Work
  • Files Modified
  • Build Issues
  • Next Steps
• Trace as a self-contained, streaming-ready state container

Documentation

Getting Started

Start here if you’re new to DR_EVT:

• Quick Start - Get up and running in 5 minutes

• Installation - Build and install DR_EVT

• Basic Tutorial - Your first simulation

User Guide

Complete guide for running simulations:

• User Guide - Complete usage manual

• Command-Line Options - All available options

Algorithm and Testing

• Backfilling Algorithms - EASY and CONSERVATIVE algorithms with diagrams

• Testing Guide - Complete test catalog, how to run tests, test patterns, and verification methodology

APIs

• Streaming API - Online/incremental simulation API

• Python API - Python bindings reference

• gRPC Client/Server - Network-exposed streaming API, including the MPI-based multi-client/multi-server test harness

Development

For contributors and maintainers:

• Developer Notes - Design decision index and development resources

Quick Links

• GitHub Repository

• Report Issues

• License

Project Status

Version: 1.0

Scheduling Policies:

• Backfill: EASY and CONSERVATIVE (fully implemented)

• Priority: FCFS, an alternative FCFS implementation (for testing), FCFS with conservative/no backfilling support, SJF, LJF

APIs:

• Streaming API: Online/incremental simulation, including genuinely new jobs (append_job()/append_jobs()) not just ones already in a preloaded trace

• gRPC Service: Network-exposed streaming API for multi-cluster coordination

• Python Bindings: Batch mode and the core streaming/monitoring API; some Sim_Params fields and Simulation methods (including append_job()/append_jobs()) aren’t bound yet - see Python API

Test Suite: 150+ tests across comprehensive/unit/feature/append-job/progressive-loading/config/gRPC/scale/replay categories (see Testing Guide for the current breakdown and how to run them).

About

DR_EVT simulates discrete event-driven HPC job scheduling with:

• EASY and CONSERVATIVE backfilling implementations, verified against an independent Python reference implementation (consistency check, not mathematical ground truth - see Testing Guide)

• gRPC-based online simulation service enabling coordinated multi-cluster simulations in a distributed fashion and digital-twin scheduler interacting in real-time

• Replay and simulation modes for both offline analysis and online operation

• Streaming API for incremental job submission and online scheduling decisions

• Real HPC trace support (Lassen format) plus simpler CSV format

• Comprehensive test suite covering correctness, differential comparisons, and large-scale scenarios

Developed at Lawrence Livermore National Laboratory.