πŸ” pmu.latency_test

Individual debug report β€” smoke regression.

← Back to regression report

pmu.latency_test

Verified
Fail signature (UVMError)
UVMError: PMU_LATENCY: CMD_LATENCY_LAST=4, want 5 (see docs/verification/bug_li…

Attempt

Attempt 1 of 1 Approved
Investigation plan
  1. Searched the design spec for expected behavior (PMU_LATENCY CMD_LATENCY)
  2. Searched the design spec for expected behavior (latency_test)
  3. Searched the design graph (CMD_LATENCY_LAST)
  4. Searched the design spec for expected behavior (PMU performance monitoring latency)
  5. Searched the design spec for expected behavior (command latency cycles measurement)
  6. Searched the design graph (tpe_pmu)
  7. Inspected the module instance hierarchy (tpe_pmu)
  8. Traced the signal's driver (cmd_latency_last_q)
  9. Read the RTL source (rtl/pmu/tpe_pmu.sv)
  10. Read the RTL source (rtl/pmu/tpe_pmu.sv)
  11. Read the RTL source (verif/cocotb_tb/pmu/test_pmu.py)
  12. Computed the fanin cone (upstream dependencies) (latency_ctr_q)
  13. Traced the signal's driver (dispatch_start)
  14. Traced the signal's driver (cmd_done_valid)
  15. Read the RTL source (rtl/pmu/tpe_pmu.sv)
  16. Searched the design spec for expected behavior (dispatch latency window begins ends)
  17. Searched the design spec for expected behavior (bug #7 latency)
  18. Searched the design spec for expected behavior (CMD_LATENCY_LAST register)
  19. Searched the design spec for expected behavior (tpe_pmu register map)
  20. Searched the design spec for expected behavior (performance monitor unit)
  21. Searched the design spec for expected behavior (latency measurement cycles)
Hypothesis / root causertl/pmu/tpe_pmu.sv:107
Confidence
94%

tpe_pmu.sv's per-command latency counter (latency_ctr_q) increments via a nonblocking assignment (line 104) each cycle dispatch_active_q is set, starting at 1 on dispatch_start. When cmd_done_valid fires on the same cycle as an ongoing dispatch_active_q increment, the capture 'cmd_latency_last_q <= latency_ctr_q' at line 114 reads the pre-clock (old) value of latency_ctr_q, since both assignments are nonblocking and evaluate right-hand sides before the clock edge. This means the completion cycle itself is never counted, undercounting the true latency by exactly 1 cycle. This is explicitly documented as bug #7 in a comment directly above the buggy line. The failing test confirms this off-by-one: CMD_LATENCY_LAST=4 measured vs. want=5. The fix is to capture latency_ctr_q + 1 instead of latency_ctr_q, which correctly accounts for the completion cycle.

Retrieved evidence (21 steps)
  1. Step 1: Searched the design spec for expected behavior (PMU_LATENCY CMD_LATENCY)
    (no spec sections matched 'PMU_LATENCY CMD_LATENCY')
  2. Step 2: Searched the design spec for expected behavior (latency_test)
    (no spec sections matched 'latency_test')
  3. Step 3: Searched the design graph (CMD_LATENCY_LAST)
    - Register:cmd_latency_last_q (id=register:tpe_pmu.cmd_latency_last_q, module=tpe_pmu)
  4. Step 4: Searched the design spec for expected behavior (PMU performance monitoring latency)
    (no spec sections matched 'PMU performance monitoring latency')
  5. Step 5: Searched the design spec for expected behavior (command latency cycles measurement)
    (no spec sections matched 'command latency cycles measurement')
  6. Step 6: Searched the design graph (tpe_pmu)
    - Module:tpe_pmu (id=module:tpe_pmu, module=None)
  7. Step 7: Inspected the module instance hierarchy (tpe_pmu)
    {"module": {"name": "tpe_pmu", "orig_name": "tpe_pmu", "level": 2, "is_top": false, "node_type": "Module", "id": "module:tpe_pmu"}, "instances": []}
  8. Step 8: Traced the signal's driver (cmd_latency_last_q)
    - Assignment:assignment:tpe_pmu.assign_469 inside AlwaysBlock:always:tpe_pmu.always_437 [rtl/pmu/tpe_pmu.sv:70:3]
    - Assignment:assignment:tpe_pmu.assign_483 inside AlwaysBlock:always:tpe_pmu.always_437 [rtl/pmu/tpe_pmu.sv:70:3]
  9. Step 9: Read the RTL source (rtl/pmu/tpe_pmu.sv)
    --- rtl/pmu/tpe_pmu.sv (lines 50-90 of 218) ---
       50:     input logic dispatch_start,    // a command's dispatch-latency window begins this cycle
       51:     input logic cmd_done_valid     // a command's dispatch-latency window ends this cycle
       52: );
       53: 
       54:   // ---- Register storage ---------------------------------------------------
       55:   logic ctrl_enable_q;
       56:   logic ctrl_reset_counters_q;
       57: 
       58:   logic [31:0] cycle_count_q;
       59:   logic [31:0] mac_active_count_
  10. Step 10: Read the RTL source (rtl/pmu/tpe_pmu.sv)
    --- rtl/pmu/tpe_pmu.sv (lines 90-150 of 218) ---
       90:       // counter reset) shouldn't erase, same rationale as a "last value"
       91:       // register in any perf-counter bank.
       92:     end else if (counting) begin
       93:       cycle_count_q <= cycle_count_q + 32'd1;
       94:       if (mac_active) mac_active_count_q <= mac_active_count_q + 32'd1;
       95:       if (dma_wait) dma_wait_count_q <= dma_wait_count_q + 32'd1;
       96:       if (sched_stall) sched_stall_count_q <= sched_stall_count_q +
  11. Step 11: Read the RTL source (verif/cocotb_tb/pmu/test_pmu.py)
    (denied) 'verif/cocotb_tb/pmu/test_pmu.py' is outside rtl/ β€” only RTL source is readable
  12. Step 12: Computed the fanin cone (upstream dependencies) (latency_ctr_q)
    - Register:ctrl_reset_counters_q (distance=1)
    - Port:rst_n (distance=1)
    - Signal:counting (distance=1)
    - Port:dispatch_start (distance=1)
    - Register:dispatch_active_q (distance=1)
    - Signal:do_write (distance=2)
    - Port:s_awaddr (distance=2)
    - Port:s_wdata (distance=2)
    - Register:ctrl_enable_q (distance=2)
    - Port:cmd_done_valid (distance=2)
    - Port:s_awvalid (distance=3)
    - Port:s_wvalid (distance=3)
    ... and 1 more (truncated, reduce max_depth)
  13. Step 13: Traced the signal's driver (dispatch_start)
    - Signal:sched_dispatch_start [rtl/top/tpe_top.sv:157:17]
  14. Step 14: Traced the signal's driver (cmd_done_valid)
    - Signal:sched_done_valid [rtl/top/tpe_top.sv:153:17]
  15. Step 15: Read the RTL source (rtl/pmu/tpe_pmu.sv)
    --- rtl/pmu/tpe_pmu.sv (lines 1-120 of 218) ---
        1: // Performance Monitor Unit: free-running event counters exposed over
        2: // AXI4-Lite as the `pmu` register block (docs/register_map/tpe_regs.yaml),
        3: // per docs/architecture/tpe_architecture_spec.md section 3.6.
        4: //
        5: // Same flat-port, single-outstanding-transaction AXI4-Lite slave pattern as
        6: // tpe_cmd_proc.sv (AWVALID+WVALID presented together, one transaction at a
        7: // time) -- reused here rather than fa
  16. Step 16: Searched the design spec for expected behavior (dispatch latency window begins ends)
    (no spec sections matched 'dispatch latency window begins ends')
  17. Step 17: Searched the design spec for expected behavior (bug #7 latency)
    (no spec sections matched 'bug #7 latency')
  18. Step 18: Searched the design spec for expected behavior (CMD_LATENCY_LAST register)
    (no spec sections matched 'CMD_LATENCY_LAST register')
  19. Step 19: Searched the design spec for expected behavior (tpe_pmu register map)
    (no spec sections matched 'tpe_pmu register map')
  20. Step 20: Searched the design spec for expected behavior (performance monitor unit)
    ### [docs/architecture/tpe_architecture_spec.md] 3.6 Performance Monitor Unit (`rtl/pmu/`)
    Free-running counters exposed via the `pmu` register block: cycle count,
    MAC-active cycles, DMA-wait cycles, scheduler-stall cycles, idle cycles, and
    last-command latency. Counters reset via `PMU_CTRL.RESET_COUNTERS` (level-
    sensitive -- counters stay pinned at 0 for as long as it's held) and gated
    via `PMU_CTRL.ENABLE`. The event inputs (`mac_active`/`dma_wait`/
    `sched_stall`/`sched_idle`/`dispatch_start`
  21. Step 21: Searched the design spec for expected behavior (latency measurement cycles)
    (no spec sections matched 'latency measurement cycles')
Proposed patch
--- a/rtl/pmu/tpe_pmu.sv
+++ b/rtl/pmu/tpe_pmu.sv
@@ -104,16 +104,10 @@
         latency_ctr_q <= latency_ctr_q + 32'd1;
       end
 
-      if (cmd_done_valid) begin
-        // BUG (#7, see docs/verification/bug_list.md): this cycle's
-        // increment above (dispatch_active_q branch) hasn't landed in
-        // latency_ctr_q yet -- nonblocking assignments both read the *old*
-        // value, so capturing latency_ctr_q here misses the completion
-        // cycle itself and undercounts by exactly 1. Should be
-        // `latency_ctr_q + 32'd1`.
-        cmd_latency_last_q <= latency_ctr_q;
-        dispatch_active_q  <= 1'b0;
-      end
+      if (cmd_done_valid) begin
+        cmd_latency_last_q <= latency_ctr_q + 32'd1;
+        dispatch_active_q  <= 1'b0;
+      end
     end
   end
 
Answer-key check (final attempt)Correct expected bug #7 β€” rtl/pmu/tpe_pmu.sv:[]
Rerun PASS agent-fix/pmu-latency_test-9012cddb
Tokens (all attempts)23,878 (~$0.0908)