Usage
Dashboard
How to start and use the Dashboard to visualize and analyze benchmark results
Dashboard records all of your precious experiments in one place so that you don’t need spreadsheet anymore to record your experiments.
One key concept in Dashboard is
Expected output:
Once the server is running, open your browser and navigate to the local URL or using public URL.
You can share the public URL to allow others to access it remotely.
Subrun and Run. Subrun is a single benchmark run of a specific engine/benchmark/hardware/model combination. Run is a group of Subruns.
Starting the Dashboard
Starting the dashboard is simple with a single command:Dashboard Areas
The dashboard is organized into several key areas to help you analyze and compare benchmark results.Overview Panel
Overview Panel
The Overview Panel provides a high-level summary of your benchmark runs:
Key information includes:
Key information includes:- Filter of Benchmark runs and subruns
- Sub-runs Metrics Comparison
- Visualization of Sub-run Metrics
- Detailed Information of Sub-run
--max-num-seq parameter in the vLLM engine. In this run, the parameter was set to 1, 2, 4, 8, 16, 32, and 256, while the benchmark QPS was fixed at 10. From the chart, we can clearly observe the impact of increasing --max-num-seq on performance.Filter of Benchmark runs and subruns
Filter of Benchmark runs and subruns
You can select a specific run to view the performance comparison of all its sub-runs. Or a specific sub-run to view all its detailed information.
Features include:
Features include:- You can select a specific run to view the performance comparison of all its sub-runs.
- You can select a specific sub-run to view all its detailed information.
- When new result files are generated, data refresh is supported.
- Supports uploading the sub-run you wish to share to the community.
Sub-runs Metrics Comparison
Sub-runs Metrics Comparison
In this section, you can view the metrics comparison of benchmark results for each sub-run:
Information displayed includes:
Information displayed includes:- Complete configuration details (engine type, model, parameters)
- All collected metrics with exact values
- Graphs showing performance over time
Detailed Information of Sub-run
Detailed Information of Sub-run
If you select a specific sub-run, you can access all the information related to that sub-run:
Subrun Detail Information include:
Subrun Detail Information include:- Engine Information
- Benchmark Information
- Performance Metrics
- Standard Output
- Standard Error
- Full Engine Arguments
- Full Environment Variables
- Full Benchmark Config
Uploading Sub-run Results to the Community
You can upload your sub-run results to the community for others to view and compare using the button ofShare Subrun to Gloabal Leaderboard.
Once you click this button, you may need to log in with GitHub to record the uploader.
If the upload is successful, you will see the following Information, and you can find your uploaded data in the global leaderboard: