KEYMGR Simulation Results

Friday March 14 2025 17:31:47 UTC

GitHub Revision: `9464f06a4d`

Branch: dj-sw-nightly

Simulator: VCS

Test Results

Stage	Name	Tests	Max Job Runtime	Simulated Time	Passing	Total	Pass Rate
V1	smoke	keymgr_smoke	17.690s	3.482ms	49	50	98.00
V1	random	keymgr_random	45.450s	1.937ms	50	50	100.00
V1	csr_hw_reset	keymgr_csr_hw_reset	4.090s	236.799us	5	5	100.00
V1	csr_rw	keymgr_csr_rw	3.770s	12.845us	17	20	85.00
V1	csr_bit_bash	keymgr_csr_bit_bash	11.210s	1.007ms	4	5	80.00
V1	csr_aliasing	keymgr_csr_aliasing	8.450s	135.835us	5	5	100.00
V1	csr_mem_rw_with_rand_reset	keymgr_csr_mem_rw_with_rand_reset	4.860s	147.675us	15	20	75.00
V1	regwen_csr_and_corresponding_lockable_csr	keymgr_csr_rw	3.770s	12.845us	17	20	85.00
		keymgr_csr_aliasing	8.450s	135.835us	5	5	100.00
V1		TOTAL			145	155	93.55
V2	cfgen_during_op	keymgr_cfg_regwen	38.490s	1.031ms	47	50	94.00
V2	sideload	keymgr_sideload	27.590s	2.046ms	50	50	100.00
		keymgr_sideload_kmac	27.680s	3.618ms	50	50	100.00
		keymgr_sideload_aes	33.040s	1.696ms	49	50	98.00
		keymgr_sideload_otbn	41.320s	17.883ms	50	50	100.00
V2	direct_to_disabled_state	keymgr_direct_to_disabled	25.340s	16.037ms	49	50	98.00
V2	lc_disable	keymgr_lc_disable	20.910s	1.069ms	50	50	100.00
V2	kmac_error_response	keymgr_kmac_rsp_err	10.010s	2.443ms	37	50	74.00
V2	invalid_sw_input	keymgr_sw_invalid_input	42.700s	9.360ms	50	50	100.00
V2	invalid_hw_input	keymgr_hwsw_invalid_input	28.100s	5.866ms	49	50	98.00
V2	sync_async_fault_cross	keymgr_sync_async_fault_cross	16.720s	7.491ms	39	50	78.00
V2	stress_all	keymgr_stress_all	5.785m	37.809ms	44	50	88.00
V2	intr_test	keymgr_intr_test	3.810s	38.027us	50	50	100.00
V2	alert_test	keymgr_alert_test	2.910s	10.782us	50	50	100.00
V2	tl_d_oob_addr_access	keymgr_tl_errors	6.310s	2.011ms	20	20	100.00
V2	tl_d_illegal_access	keymgr_tl_errors	6.310s	2.011ms	20	20	100.00
V2	tl_d_outstanding_access	keymgr_csr_hw_reset	4.090s	236.799us	5	5	100.00
		keymgr_csr_rw	3.770s	12.845us	17	20	85.00
		keymgr_csr_aliasing	8.450s	135.835us	5	5	100.00
		keymgr_same_csr_outstanding	5.270s	89.219us	14	20	70.00
V2	tl_d_partial_access	keymgr_csr_hw_reset	4.090s	236.799us	5	5	100.00
		keymgr_csr_rw	3.770s	12.845us	17	20	85.00
		keymgr_csr_aliasing	8.450s	135.835us	5	5	100.00
		keymgr_same_csr_outstanding	5.270s	89.219us	14	20	70.00
V2		TOTAL			698	740	94.32
V2S	sec_cm_additional_check	keymgr_sec_cm	9.490s	2.185ms	5	5	100.00
V2S	tl_intg_err	keymgr_sec_cm	9.490s	2.185ms	5	5	100.00
		keymgr_tl_intg_err	7.990s	941.415us	12	20	60.00
V2S	shadow_reg_update_error	keymgr_shadow_reg_errors	5.580s	179.191us	2	20	10.00
V2S	shadow_reg_read_clear_staged_value	keymgr_shadow_reg_errors	5.580s	179.191us	2	20	10.00
V2S	shadow_reg_storage_error	keymgr_shadow_reg_errors	5.580s	179.191us	2	20	10.00
V2S	shadowed_reset_glitch	keymgr_shadow_reg_errors	5.580s	179.191us	2	20	10.00
V2S	shadow_reg_update_error_with_csr_rw	keymgr_shadow_reg_errors_with_csr_rw	8.590s	213.393us	2	20	10.00
V2S	prim_count_check	keymgr_sec_cm	9.490s	2.185ms	5	5	100.00
V2S	prim_fsm_check	keymgr_sec_cm	9.490s	2.185ms	5	5	100.00
V2S	sec_cm_bus_integrity	keymgr_tl_intg_err	7.990s	941.415us	12	20	60.00
V2S	sec_cm_config_shadow	keymgr_shadow_reg_errors	5.580s	179.191us	2	20	10.00
V2S	sec_cm_op_config_regwen	keymgr_cfg_regwen	38.490s	1.031ms	47	50	94.00
V2S	sec_cm_reseed_config_regwen	keymgr_random	45.450s	1.937ms	50	50	100.00
		keymgr_csr_rw	3.770s	12.845us	17	20	85.00
V2S	sec_cm_sw_binding_config_regwen	keymgr_random	45.450s	1.937ms	50	50	100.00
		keymgr_csr_rw	3.770s	12.845us	17	20	85.00
V2S	sec_cm_max_key_ver_config_regwen	keymgr_random	45.450s	1.937ms	50	50	100.00
		keymgr_csr_rw	3.770s	12.845us	17	20	85.00
V2S	sec_cm_lc_ctrl_intersig_mubi	keymgr_lc_disable	20.910s	1.069ms	50	50	100.00
V2S	sec_cm_constants_consistency	keymgr_hwsw_invalid_input	28.100s	5.866ms	49	50	98.00
V2S	sec_cm_intersig_consistency	keymgr_hwsw_invalid_input	28.100s	5.866ms	49	50	98.00
V2S	sec_cm_hw_key_sw_noaccess	keymgr_random	45.450s	1.937ms	50	50	100.00
V2S	sec_cm_output_keys_ctrl_redun	keymgr_sideload_protect	30.220s	4.406ms	49	50	98.00
V2S	sec_cm_ctrl_fsm_sparse	keymgr_sec_cm	9.490s	2.185ms	5	5	100.00
V2S	sec_cm_data_fsm_sparse	keymgr_sec_cm	9.490s	2.185ms	5	5	100.00
V2S	sec_cm_ctrl_fsm_local_esc	keymgr_sec_cm	9.490s	2.185ms	5	5	100.00
V2S	sec_cm_ctrl_fsm_consistency	keymgr_custom_cm	35.870s	1.684ms	35	50	70.00
V2S	sec_cm_ctrl_fsm_global_esc	keymgr_lc_disable	20.910s	1.069ms	50	50	100.00
V2S	sec_cm_ctrl_ctr_redun	keymgr_sec_cm	9.490s	2.185ms	5	5	100.00
V2S	sec_cm_kmac_if_fsm_sparse	keymgr_sec_cm	9.490s	2.185ms	5	5	100.00
V2S	sec_cm_kmac_if_ctr_redun	keymgr_sec_cm	9.490s	2.185ms	5	5	100.00
V2S	sec_cm_kmac_if_cmd_ctrl_consistency	keymgr_custom_cm	35.870s	1.684ms	35	50	70.00
V2S	sec_cm_kmac_if_done_ctrl_consistency	keymgr_custom_cm	35.870s	1.684ms	35	50	70.00
V2S	sec_cm_reseed_ctr_redun	keymgr_sec_cm	9.490s	2.185ms	5	5	100.00
V2S	sec_cm_side_load_sel_ctrl_consistency	keymgr_custom_cm	35.870s	1.684ms	35	50	70.00
V2S	sec_cm_sideload_ctrl_fsm_sparse	keymgr_sec_cm	9.490s	2.185ms	5	5	100.00
V2S	sec_cm_ctrl_key_integrity	keymgr_custom_cm	35.870s	1.684ms	35	50	70.00
V2S		TOTAL			105	165	63.64
V3	stress_all_with_rand_reset	keymgr_stress_all_with_rand_reset	18.870s	700.543us	30	50	60.00
V3		TOTAL			30	50	60.00
		TOTAL			978	1110	88.11

Coverage Results

Coverage Dashboard

SCORE	LINE	COND	TOGGLE	FSM	BRANCH	ASSERT	GROUP
97.67	99.10	97.51	98.26	100.00	99.01	98.61	91.18

Failure Buckets

UVM_FATAL (alert_receiver_driver.sv:145) [driver] Check failed (cfg.vif.receiver_cb.alert_tx.alert_p) alert_p not high, despite an item in r_alert_rsp_q has 29 failures:

Test keymgr_shadow_reg_errors has 18 failures.

0.keymgr_shadow_reg_errors.587982404998111834304407648403528479306795708528502120631003550579547811367
Line 95, in log /nightly/runs/scratch/dj-sw-nightly/keymgr-sim-vcs/0.keymgr_shadow_reg_errors/latest/run.log

  UVM_FATAL @ 134784625 ps: (alert_receiver_driver.sv:145) [uvm_test_top.env.m_alert_agent_fatal_fault_err.driver] Check failed (cfg.vif.receiver_cb.alert_tx.alert_p) alert_p not high, despite an item in r_alert_rsp_q
  UVM_INFO @ 134784625 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

1.keymgr_shadow_reg_errors.90233083237393111610378947022611079234487112178343490133499897578302839895243
Line 102, in log /nightly/runs/scratch/dj-sw-nightly/keymgr-sim-vcs/1.keymgr_shadow_reg_errors/latest/run.log

  UVM_FATAL @ 132620499 ps: (alert_receiver_driver.sv:145) [uvm_test_top.env.m_alert_agent_fatal_fault_err.driver] Check failed (cfg.vif.receiver_cb.alert_tx.alert_p) alert_p not high, despite an item in r_alert_rsp_q
  UVM_INFO @ 132620499 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 16 more failures.

Test keymgr_shadow_reg_errors_with_csr_rw has 11 failures.

0.keymgr_shadow_reg_errors_with_csr_rw.27587108697229467031720771053646663504849038217901667940301338860905307497755
Line 102, in log /nightly/runs/scratch/dj-sw-nightly/keymgr-sim-vcs/0.keymgr_shadow_reg_errors_with_csr_rw/latest/run.log

  UVM_FATAL @ 331094920 ps: (alert_receiver_driver.sv:145) [uvm_test_top.env.m_alert_agent_fatal_fault_err.driver] Check failed (cfg.vif.receiver_cb.alert_tx.alert_p) alert_p not high, despite an item in r_alert_rsp_q
  UVM_INFO @ 331094920 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

3.keymgr_shadow_reg_errors_with_csr_rw.113503760060132133905484153634488366857208445578322595158933239671460667108728
Line 117, in log /nightly/runs/scratch/dj-sw-nightly/keymgr-sim-vcs/3.keymgr_shadow_reg_errors_with_csr_rw/latest/run.log

  UVM_FATAL @ 213393071 ps: (alert_receiver_driver.sv:145) [uvm_test_top.env.m_alert_agent_fatal_fault_err.driver] Check failed (cfg.vif.receiver_cb.alert_tx.alert_p) alert_p not high, despite an item in r_alert_rsp_q
  UVM_INFO @ 213393071 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 9 more failures.

Offending '(d2h.d_error || ((d2h.d_data & *) == (exp_vals[*] & *)))' has 26 failures:

Test keymgr_tl_intg_err has 8 failures.

0.keymgr_tl_intg_err.108916625654650634888342630407631637555555803128411476031224537717292915030799
Line 97, in log /nightly/runs/scratch/dj-sw-nightly/keymgr-sim-vcs/0.keymgr_tl_intg_err/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[13] & 'hffffffff)))'
  UVM_ERROR @  90654102 ps: (keymgr_csr_assert_fpv.sv:431) [ASSERT FAILED] attest_sw_binding_0_rd_A
  UVM_INFO @  90654102 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

4.keymgr_tl_intg_err.22093539100551288010457739131847098096458994135177859095205913663743059938845
Line 90, in log /nightly/runs/scratch/dj-sw-nightly/keymgr-sim-vcs/4.keymgr_tl_intg_err/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[10] & 'hffffffff)))'
  UVM_ERROR @  34188244 ps: (keymgr_csr_assert_fpv.sv:416) [ASSERT FAILED] sealing_sw_binding_5_rd_A
  UVM_INFO @  34188244 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 6 more failures.

Test keymgr_shadow_reg_errors_with_csr_rw has 3 failures.

1.keymgr_shadow_reg_errors_with_csr_rw.96313988637234321137230428104251295409599095203229191267125627821483879504569
Line 80, in log /nightly/runs/scratch/dj-sw-nightly/keymgr-sim-vcs/1.keymgr_shadow_reg_errors_with_csr_rw/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[11] & 'hffffffff)))'
  UVM_ERROR @  50806407 ps: (keymgr_csr_assert_fpv.sv:421) [ASSERT FAILED] sealing_sw_binding_6_rd_A
  UVM_INFO @  50806407 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

2.keymgr_shadow_reg_errors_with_csr_rw.114498415368166302396381295732647102469642275655957940739906098548327683957341
Line 76, in log /nightly/runs/scratch/dj-sw-nightly/keymgr-sim-vcs/2.keymgr_shadow_reg_errors_with_csr_rw/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[20] & 'hffffffff)))'
  UVM_ERROR @  19055112 ps: (keymgr_csr_assert_fpv.sv:466) [ASSERT FAILED] attest_sw_binding_7_rd_A
  UVM_INFO @  19055112 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 1 more failures.

Test keymgr_csr_rw has 3 failures.

1.keymgr_csr_rw.100269095865680890813296371901141231525399542662574216864505876595571664538260
Line 76, in log /nightly/runs/scratch/dj-sw-nightly/keymgr-sim-vcs/1.keymgr_csr_rw/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[19] & 'hffffffff)))'
  UVM_ERROR @   7590231 ps: (keymgr_csr_assert_fpv.sv:461) [ASSERT FAILED] attest_sw_binding_6_rd_A
  UVM_INFO @   7590231 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

12.keymgr_csr_rw.11556757007190529882143925433244459719612448619356094812575374602289940748953
Line 77, in log /nightly/runs/scratch/dj-sw-nightly/keymgr-sim-vcs/12.keymgr_csr_rw/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[11] & 'hffffffff)))'
  UVM_ERROR @  21099361 ps: (keymgr_csr_assert_fpv.sv:421) [ASSERT FAILED] sealing_sw_binding_6_rd_A
  UVM_INFO @  21099361 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 1 more failures.

Test keymgr_same_csr_outstanding has 6 failures.

2.keymgr_same_csr_outstanding.21566882321305814403150642489394993640108482731680455130831436340987383197409
Line 76, in log /nightly/runs/scratch/dj-sw-nightly/keymgr-sim-vcs/2.keymgr_same_csr_outstanding/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[12] & 'hffffffff)))'
  UVM_ERROR @  23544712 ps: (keymgr_csr_assert_fpv.sv:426) [ASSERT FAILED] sealing_sw_binding_7_rd_A
  UVM_INFO @  23544712 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

4.keymgr_same_csr_outstanding.48577890503570130172119297755512299638075331037347881125167789620085809068437
Line 76, in log /nightly/runs/scratch/dj-sw-nightly/keymgr-sim-vcs/4.keymgr_same_csr_outstanding/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[20] & 'hffffffff)))'
  UVM_ERROR @ 360050923 ps: (keymgr_csr_assert_fpv.sv:466) [ASSERT FAILED] attest_sw_binding_7_rd_A
  UVM_INFO @ 360050923 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 4 more failures.

Test keymgr_csr_bit_bash has 1 failures.

4.keymgr_csr_bit_bash.95459670787217511299886944895496677036269922471264367304400309119202473740907
Line 76, in log /nightly/runs/scratch/dj-sw-nightly/keymgr-sim-vcs/4.keymgr_csr_bit_bash/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[13] & 'hffffffff)))'
  UVM_ERROR @ 677331356 ps: (keymgr_csr_assert_fpv.sv:431) [ASSERT FAILED] attest_sw_binding_0_rd_A
  UVM_INFO @ 677331356 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 1 more tests.

UVM_ERROR (cip_base_vseq.sv:890) [keymgr_common_vseq] Check failed (!has_outstanding_access()) Waited * cycles to issue a reset with no outstanding accesses. has 20 failures:
- Test keymgr_stress_all_with_rand_reset has 20 failures.
  - 3.keymgr_stress_all_with_rand_reset.62118495577143299729397546358792620537081482903870674860016405250200728066802
    Line 378, in log /nightly/runs/scratch/dj-sw-nightly/keymgr-sim-vcs/3.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 145360521 ps: (cip_base_vseq.sv:890) [uvm_test_top.env.virtual_sequencer.keymgr_common_vseq] Check failed (!has_outstanding_access()) Waited 10000 cycles to issue a reset with no outstanding accesses.
  UVM_INFO @ 145360521 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
  - 4.keymgr_stress_all_with_rand_reset.18726657267850722954858898898400464670394399085535316825891780558991908456958
    Line 330, in log /nightly/runs/scratch/dj-sw-nightly/keymgr-sim-vcs/4.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 1357513379 ps: (cip_base_vseq.sv:890) [uvm_test_top.env.virtual_sequencer.keymgr_common_vseq] Check failed (!has_outstanding_access()) Waited 10000 cycles to issue a reset with no outstanding accesses.
  UVM_INFO @ 1357513379 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
  - ... and 18 more failures.

UVM_ERROR (cip_base_vseq.sv:969) [keymgr_custom_cm_vseq] Check failed cfg.m_alert_agent_cfgs[alert_name].vif.get_alert() == * (* [*] vs * [*]) fatal error fatal_fault_err does not trigger! has 15 failures:

Test keymgr_custom_cm has 15 failures.

3.keymgr_custom_cm.2032457521149832683239575776572540070785687243617033240810727133539390369537
Line 168, in log /nightly/runs/scratch/dj-sw-nightly/keymgr-sim-vcs/3.keymgr_custom_cm/latest/run.log

  UVM_ERROR @ 203514447 ps: (cip_base_vseq.sv:969) [uvm_test_top.env.virtual_sequencer.keymgr_custom_cm_vseq] Check failed cfg.m_alert_agent_cfgs[alert_name].vif.get_alert() == 1 (0 [0x0] vs 1 [0x1]) fatal error fatal_fault_err does not trigger!
  UVM_INFO @ 203514447 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

5.keymgr_custom_cm.23472136698632662867922384497062387122507200517347207456495114963748869138490
Line 224, in log /nightly/runs/scratch/dj-sw-nightly/keymgr-sim-vcs/5.keymgr_custom_cm/latest/run.log

  UVM_ERROR @ 1158950784 ps: (cip_base_vseq.sv:969) [uvm_test_top.env.virtual_sequencer.keymgr_custom_cm_vseq] Check failed cfg.m_alert_agent_cfgs[alert_name].vif.get_alert() == 1 (0 [0x0] vs 1 [0x1]) fatal error fatal_fault_err does not trigger!
  UVM_INFO @ 1158950784 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 13 more failures.

UVM_ERROR (cip_base_scoreboard.sv:327) scoreboard [scoreboard] alert recov_operation_err did not trigger max_delay:* has 13 failures:

Test keymgr_kmac_rsp_err has 1 failures.

0.keymgr_kmac_rsp_err.55465406610907383143790708580817668182634068466892697296716338845391095937813
Line 211, in log /nightly/runs/scratch/dj-sw-nightly/keymgr-sim-vcs/0.keymgr_kmac_rsp_err/latest/run.log

  UVM_ERROR @  23519146 ps: (cip_base_scoreboard.sv:327) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @  23519146 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

Test keymgr_sideload_aes has 1 failures.

1.keymgr_sideload_aes.73187104538758242511739096957306518717988913942032367689475861356546263722062
Line 161, in log /nightly/runs/scratch/dj-sw-nightly/keymgr-sim-vcs/1.keymgr_sideload_aes/latest/run.log

  UVM_ERROR @  39142146 ps: (cip_base_scoreboard.sv:327) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @  39142146 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

Test keymgr_sideload_protect has 1 failures.

6.keymgr_sideload_protect.33953275249266313340343105966499170009366340722244241739964843491001819791547
Line 208, in log /nightly/runs/scratch/dj-sw-nightly/keymgr-sim-vcs/6.keymgr_sideload_protect/latest/run.log

  UVM_ERROR @  21883928 ps: (cip_base_scoreboard.sv:327) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @  21883928 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

Test keymgr_stress_all has 6 failures.

6.keymgr_stress_all.48744963929492655742145626484467416457886866338843274169342646693191975005987
Line 601, in log /nightly/runs/scratch/dj-sw-nightly/keymgr-sim-vcs/6.keymgr_stress_all/latest/run.log

  UVM_ERROR @ 613336138 ps: (cip_base_scoreboard.sv:327) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @ 613336138 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

11.keymgr_stress_all.63385884637130987946283936314895867641667860804833230589436660876379020153982
Line 1063, in log /nightly/runs/scratch/dj-sw-nightly/keymgr-sim-vcs/11.keymgr_stress_all/latest/run.log

  UVM_ERROR @ 521963097 ps: (cip_base_scoreboard.sv:327) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @ 521963097 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 4 more failures.

Test keymgr_hwsw_invalid_input has 1 failures.

10.keymgr_hwsw_invalid_input.99266527860099452628056003474806998849580912880946710976868065933760468361584
Line 303, in log /nightly/runs/scratch/dj-sw-nightly/keymgr-sim-vcs/10.keymgr_hwsw_invalid_input/latest/run.log

  UVM_ERROR @ 657024046 ps: (cip_base_scoreboard.sv:327) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @ 657024046 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 2 more tests.

UVM_ERROR (cip_base_vseq.sv:969) [keymgr_sync_async_fault_cross_vseq] Check failed cfg.m_alert_agent_cfgs[alert_name].vif.get_alert() == * (* [*] vs * [*]) fatal error fatal_fault_err does not trigger! has 11 failures:

Test keymgr_sync_async_fault_cross has 11 failures.

1.keymgr_sync_async_fault_cross.55428830682680378718930154430861702141168589456166035028677767966814929032907
Line 141, in log /nightly/runs/scratch/dj-sw-nightly/keymgr-sim-vcs/1.keymgr_sync_async_fault_cross/latest/run.log

  UVM_ERROR @ 532122391 ps: (cip_base_vseq.sv:969) [uvm_test_top.env.virtual_sequencer.keymgr_sync_async_fault_cross_vseq] Check failed cfg.m_alert_agent_cfgs[alert_name].vif.get_alert() == 1 (0 [0x0] vs 1 [0x1]) fatal error fatal_fault_err does not trigger!
  UVM_INFO @ 532122391 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

8.keymgr_sync_async_fault_cross.38659304390447196427735390236601673619035195223797886337044019427939445211195
Line 123, in log /nightly/runs/scratch/dj-sw-nightly/keymgr-sim-vcs/8.keymgr_sync_async_fault_cross/latest/run.log

  UVM_ERROR @ 100544022 ps: (cip_base_vseq.sv:969) [uvm_test_top.env.virtual_sequencer.keymgr_sync_async_fault_cross_vseq] Check failed cfg.m_alert_agent_cfgs[alert_name].vif.get_alert() == 1 (0 [0x0] vs 1 [0x1]) fatal error fatal_fault_err does not trigger!
  UVM_INFO @ 100544022 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 9 more failures.

UVM_ERROR (cip_base_scoreboard.sv:327) scoreboard [scoreboard] alert fatal_fault_err did not trigger max_delay:* has 8 failures:
- Test keymgr_kmac_rsp_err has 8 failures.
  - 3.keymgr_kmac_rsp_err.44809132337318561543705610799595216758083711594875197480194279733246640539706
    Line 275, in log /nightly/runs/scratch/dj-sw-nightly/keymgr-sim-vcs/3.keymgr_kmac_rsp_err/latest/run.log
```
  UVM_ERROR @  20028175 ps: (cip_base_scoreboard.sv:327) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert fatal_fault_err did not trigger max_delay:0
  UVM_INFO @  20028175 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
  - 7.keymgr_kmac_rsp_err.76739260723384285766290875223260865884594514290042626919007231524707293769513
    Line 524, in log /nightly/runs/scratch/dj-sw-nightly/keymgr-sim-vcs/7.keymgr_kmac_rsp_err/latest/run.log
```
  UVM_ERROR @ 171124422 ps: (cip_base_scoreboard.sv:327) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert fatal_fault_err did not trigger max_delay:0
  UVM_INFO @ 171124422 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
  - ... and 6 more failures.
UVM_ERROR (cip_base_vseq.sv:969) [keymgr_common_vseq] Check failed cfg.m_alert_agent_cfgs[alert_name].vif.get_alert() == * (* [*] vs * [*]) fatal error fatal_fault_err does not trigger! has 4 failures:
- Test keymgr_shadow_reg_errors_with_csr_rw has 4 failures.
  - 4.keymgr_shadow_reg_errors_with_csr_rw.1590526343049889534661753185970167146161835832327386217973335436491244568173
    Line 105, in log /nightly/runs/scratch/dj-sw-nightly/keymgr-sim-vcs/4.keymgr_shadow_reg_errors_with_csr_rw/latest/run.log
```
  UVM_ERROR @ 654324617 ps: (cip_base_vseq.sv:969) [uvm_test_top.env.virtual_sequencer.keymgr_common_vseq] Check failed cfg.m_alert_agent_cfgs[alert_name].vif.get_alert() == 1 (0 [0x0] vs 1 [0x1]) fatal error fatal_fault_err does not trigger!
  UVM_INFO @ 654324617 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
  - 6.keymgr_shadow_reg_errors_with_csr_rw.98700731264052804013895305725884808000998901498986446138025283643781975880290
    Line 80, in log /nightly/runs/scratch/dj-sw-nightly/keymgr-sim-vcs/6.keymgr_shadow_reg_errors_with_csr_rw/latest/run.log
```
  UVM_ERROR @  51189927 ps: (cip_base_vseq.sv:969) [uvm_test_top.env.virtual_sequencer.keymgr_common_vseq] Check failed cfg.m_alert_agent_cfgs[alert_name].vif.get_alert() == 1 (0 [0x0] vs 1 [0x1]) fatal error fatal_fault_err does not trigger!
  UVM_INFO @  51189927 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
  - ... and 2 more failures.
UVM_ERROR (cip_base_scoreboard.sv:249) scoreboard [scoreboard] alert recov_operation_err has unexpected timeout error has 3 failures:
- Test keymgr_kmac_rsp_err has 1 failures.
  - 6.keymgr_kmac_rsp_err.49900781412826786141720878903677264554158307767101684787780244255204904166669
    Line 273, in log /nightly/runs/scratch/dj-sw-nightly/keymgr-sim-vcs/6.keymgr_kmac_rsp_err/latest/run.log
```
  UVM_ERROR @  15821801 ps: (cip_base_scoreboard.sv:249) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err has unexpected timeout error
  UVM_INFO @  15821801 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
- Test keymgr_cfg_regwen has 1 failures.
  - 23.keymgr_cfg_regwen.71510594394773539804909103224959914731738175782041689463554296332917110163134
    Line 417, in log /nightly/runs/scratch/dj-sw-nightly/keymgr-sim-vcs/23.keymgr_cfg_regwen/latest/run.log
```
  UVM_ERROR @  80319959 ps: (cip_base_scoreboard.sv:249) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err has unexpected timeout error
  UVM_INFO @  80319959 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
- Test keymgr_direct_to_disabled has 1 failures.
  - 38.keymgr_direct_to_disabled.113845523717345698340088742510014929791947716724211864954414669608455927944882
    Line 213, in log /nightly/runs/scratch/dj-sw-nightly/keymgr-sim-vcs/38.keymgr_direct_to_disabled/latest/run.log
```
  UVM_ERROR @ 182672991 ps: (cip_base_scoreboard.sv:249) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err has unexpected timeout error
  UVM_INFO @ 182672991 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_ERROR (cip_base_scoreboard.sv:249) scoreboard [scoreboard] alert fatal_fault_err has unexpected timeout error has 2 failures:
- Test keymgr_kmac_rsp_err has 2 failures.
  - 5.keymgr_kmac_rsp_err.13233140779735922109747115472660312456808323454949981895868190050759638260032
    Line 476, in log /nightly/runs/scratch/dj-sw-nightly/keymgr-sim-vcs/5.keymgr_kmac_rsp_err/latest/run.log
```
  UVM_ERROR @  36517520 ps: (cip_base_scoreboard.sv:249) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert fatal_fault_err has unexpected timeout error
  UVM_INFO @  36517520 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
  - 20.keymgr_kmac_rsp_err.37356731136220162795549140877234743224754625190481037437708255844438419419663
    Line 443, in log /nightly/runs/scratch/dj-sw-nightly/keymgr-sim-vcs/20.keymgr_kmac_rsp_err/latest/run.log
```
  UVM_ERROR @  85827603 ps: (cip_base_scoreboard.sv:249) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert fatal_fault_err has unexpected timeout error
  UVM_INFO @  85827603 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_ERROR (cip_base_scoreboard.sv:261) [scoreboard] Check failed expected_alert[alert_name].expected == * (* [*] vs * [*]) alert recov_operation_err triggered unexpectedly has 1 failures:
- Test keymgr_kmac_rsp_err has 1 failures.
  - 16.keymgr_kmac_rsp_err.8725199055714922678217246236545433683436005054366143585482654122551448903299
    Line 244, in log /nightly/runs/scratch/dj-sw-nightly/keymgr-sim-vcs/16.keymgr_kmac_rsp_err/latest/run.log
```
  UVM_ERROR @ 1193254218 ps: (cip_base_scoreboard.sv:261) [uvm_test_top.env.scoreboard] Check failed expected_alert[alert_name].expected == 1 (0 [0x0] vs 1 [0x1]) alert recov_operation_err triggered unexpectedly
  UVM_INFO @ 1193254218 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```