CTRE Signal Logger causes significant loop overruns after autonInit on FMS

[Cryplo]

1076 discovered this issue at St. Joseph WK2. Although our autonomous was working at home, at competition on FMS we found that we were experiencing a significant loop overrun >800ms that caused our robot to behave unexpectedly. This overrun occurred roughly 1s into autonomous. There were high overruns after that too of about 250ms, but we believe that that is just because our auto-align is poorly written / expensive (there appeared to be a correlation between calling the auto-align command and the 250ms overruns). Between St. Joseph WK2 and Saline WK4, we were able to solve the issue. We turned off signal logging and also changed thread priority to real-time 10. Based on our limited testing in practice mode, if we keep the thread priority at real-time 10 but turn on signal logging, we experience the same large overrun, which leads us to believe that signal logging is the cause of the issue. In #programming-discussion of the FRC discord server, team 868 experienced a similar issue and was able to fix it by turning off signal logging. We talked with Daltz in the channel so they should already be aware of this issue.

Loop Overrun Image:

• https://media.discordapp.net/attachments/1337519185206444183/1355214705991614557/image.jpg?ex=67e81d8c&is=67e6cc0c&hm=a461e08ffa26b42e7e63be7adf66d3dff14565c86460841e58d91a8d6995d65f&=&format=webp&width=1745&height=1308

Example Match Video of Behavior:

• https://www.thebluealliance.com/match/2025misjo_qm43

ChiefDelphi Threads:

• https://www.chiefdelphi.com/t/auto-init-overrun/494848
• https://www.chiefdelphi.com/t/pathplanner-first-path-going-insane-help/494814
• https://www.chiefdelphi.com/t/choreo-autonomous-loop-overruns/495597

Discord Discussion:

• https://discord.com/channels/176186766946992128/368993897495527424/1355175663157383328

Our Code (Week 2 and Week 4):

• https://github.com/FRC1076/2025-Competition/releases/tag/Week2
• https://github.com/FRC1076/2025-Competition/releases/tag/Week4

[superpenguin612]

+1 from 868, we have an identical issue. Happy to provide any logs or additional data points.

[ozrien]

Got any DS logs? Specifically the .dsevents and .dslog when the issue occured.

[CoryNessCTR]

I spent some time digging into this and found some interesting results.

I wasn't able to reproduce your issue exactly, but I did find that running your week2 code I could produce some higher-than-typical timing spikes, increasing a total of 40ms with Signal Logger starting.

Then, after implementing the PathPlanner Warmup command, the timing spikes went away (and the overall performance was much better).

I was able to reproduce the symptom by just using File I/O as well, if I spawn a thread that writes to a file before scheduling the autonomous command, the timing increases by approximately 30ms compared to if I spawn it after scheduling the autonomous command.

My hypothesis is that there may be File I/O happening somewhere in the PathPlanner command that may be getting hung by the other File I/O that's occurring from the Signal Logger. By warming up the paths, it may be reducing the burden of that IO on the PathPlanner side.

I also noticed that you explicitly commented this out for Week 2
https://github.com/FRC1076/2025-Competition/blob/Week2/src/main/java/frc/robot/Robot.java#L85
And later uncommented it for Week 4
https://github.com/FRC1076/2025-Competition/blob/Week4/src/main/java/frc/robot/Robot.java#L91
Do you know if you tried leaving the warmup in and verified the problem was still present?

You should also be able to test this yourself by running Practice mode on the DS. At least on the Signal Logger side, Practice mode is the same as an FMS enable.

[pikitty314]

Here are the driver station logs (from 1076) from our week two event:
https://drive.google.com/drive/folders/1_h4lTzogXB9B3UwlBegDf_S0nYcliWPI?usp=sharing

[ldavidc]

I also noticed that you explicitly commented this out for Week 2 https://github.com/FRC1076/2025-Competition/blob/Week2/src/main/java/frc/robot/Robot.java#L85 And later uncommented it for Week 4 https://github.com/FRC1076/2025-Competition/blob/Week4/src/main/java/frc/robot/Robot.java#L91 Do you know if you tried leaving the warmup in and verified the problem was still present?

We (1076) tested with and without the warmup command with identical results, so we don't think that the lack of warmup command is the issue.

You should also be able to test this yourself by running Practice mode on the DS. At least on the Signal Logger side, Practice mode is the same as an FMS enable.

Practice mode is how we are able to recreate the issue at home. When we disable the signal logger auto start, our spike goes away.

[daltzctr]

If you can still consistently reproduce it, does commenting out path planner logic also fix it?

We deployed your code on our robot and the above is Cory's attempt at reproducing the delay. The signal logger write is completely asynchronous, so the only way it can block is indirectly through some other blocking IO operation.

[ldavidc]

We were able to reproduce the issue with an empty path planner auto command, but I don't believe we've tested with a Commands.none(). Next time we have access to the robot for testing, we can try it, but that may not be for a few days.

[Cryplo]

We are on a RIO 1 if that makes a difference

[ethanc8]

Please note this behavior:

Enabling a feature only during matches is a highly dangerous decision, as it leads to divergence of behavior between testing and production environments, which should normally be minimized. Teams would not expect signal logging to only be enabled during production, as there's no reason to expect that (neither FMS nor any other parts of the production environment especially accomodate signal logging). I think that this behavior should be changed.

[shoryamalani]

Bumping this wondering if any solution was found. We (422) had this exact issue on a rio 2 and only found out it was caused by this on a hunch. I think if there isn't a solution found, it shouldn't be a default on behavior.