OSS test failures in sequential execution

[mreso]

Hi,

Initial goal was to run the grpo example but I get an error when running the example.

To check my installation and environment I tried to run the tests but they fail at the same point they currently fail in the CI.

python/tests/builtins/test_log.py::TestLogFunctions::test_log_remote_default_level[py] PASSED                                                                                                                                                 [  9%]
python/tests/builtins/test_log.py::TestLogFunctions::test_log_remote_default_level[rs] FAILED                                                                                                                                                 [  9%]
python/tests/builtins/test_log.py::TestLogFunctions::test_log_remote_with_args[py] PASSED                                                                                                                                                     [  9%]
python/tests/builtins/test_log.py::TestLogFunctions::test_log_remote_with_args[rs] FAILED                                                                                                                                                     [  9%]
python/tests/builtins/test_log.py::TestLogFunctions::test_set_logging_level_remote[py] PASSED                                                                                                                                                 [ 10%]
python/tests/builtins/test_log.py::TestLogFunctions::test_set_logging_level_remote[rs] FAILED                                                                                                                                                 [ 10%]
python/tests/builtins/test_log.py::TestLogFunctions::test_log_remote_custom_level[py] PASSED                                                                                                                                                  [ 10%]
python/tests/builtins/test_log.py::TestLogFunctions::test_log_remote_custom_level[rs] FAILED                                                                                                                                                  [ 10%]
python/tests/builtins/test_log.py::TestLogFunctions::test_log_remote_multiple_calls[py] PASSED                                                                                                                                                [ 10%]
python/tests/builtins/test_log.py::TestLogFunctions::test_log_remote_multiple_calls[rs] FAILED

When reverting back to the last commit with sucessfull CI checks fff47ec the above test succeeds but there are tests with other errors down the road:

FAILED python/tests/test_allocator.py::TestRemoteAllocator::test_allocate_2d_mesh - FileNotFoundError: [Errno 2] No such file or directory: 'process_allocator'
FAILED python/tests/test_allocator.py::TestRemoteAllocator::test_call_allocate_twice - FileNotFoundError: [Errno 2] No such file or directory: 'process_allocator'
FAILED python/tests/test_allocator.py::TestRemoteAllocator::test_log - FileNotFoundError: [Errno 2] No such file or directory: 'process_allocator'
FAILED python/tests/test_allocator.py::TestRemoteAllocator::test_stacked_1d_meshes - FileNotFoundError: [Errno 2] No such file or directory: 'process_allocator'
FAILED python/tests/test_allocator.py::TestRemoteAllocator::test_stop_proc_mesh - FileNotFoundError: [Errno 2] No such file or directory: 'process_allocator'
FAILED python/tests/test_allocator.py::TestRemoteAllocator::test_stop_proc_mesh_context_manager - FileNotFoundError: [Errno 2] No such file or directory: 'process_allocator'
FAILED python/tests/test_allocator.py::TestRemoteAllocator::test_stop_proc_mesh_context_manager_multiple_times - FileNotFoundError: [Errno 2] No such file or directory: 'process_allocator'

My environment looks like this:

Collecting environment information...
PyTorch version: 2.7.1+cu126
Is debug build: False
CUDA used to build PyTorch: 12.6
ROCM used to build PyTorch: N/A
OS: CentOS Stream 9 (x86_64)
GCC version: (GCC) 11.5.0 20240719 (Red Hat 11.5.0-7)
Clang version: 20.1.8 (https://github.com/conda-forge/clangdev-feedstock 55af0edabf37262c7a2a1caaa38f9a9b843ba87f)
CMake version: version 3.26.5
Libc version: glibc-2.34
Python version: 3.10.18 | packaged by conda-forge | (main, Jun  4 2025, 14:45:41) [GCC 13.3.0] (64-bit runtime)
Python platform: Linux-6.4.3-0_fbk20_zion_2830_g3e5ab162667d-x86_64-with-glibc2.34
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA H100
GPU 1: NVIDIA H100
GPU 2: NVIDIA H100
GPU 3: NVIDIA H100
GPU 4: NVIDIA H100
GPU 5: NVIDIA H100
GPU 6: NVIDIA H100
GPU 7: NVIDIA H100
Nvidia driver version: 535.154.05
cuDNN version: Probably one of the following:
/usr/lib64/libcudnn.so.9.8.0
/usr/lib64/libcudnn_adv.so.9.8.0
/usr/lib64/libcudnn_cnn.so.9.8.0
/usr/lib64/libcudnn_engines_precompiled.so.9.8.0
/usr/lib64/libcudnn_engines_runtime_compiled.so.9.8.0
/usr/lib64/libcudnn_graph.so.9.8.0
/usr/lib64/libcudnn_heuristic.so.9.8.0
/usr/lib64/libcudnn_ops.so.9.8.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      52 bits physical, 57 bits virtual
Byte Order:                         Little Endian
CPU(s):                             384
On-line CPU(s) list:                0-383
Vendor ID:                          AuthenticAMD
Model name:                         AMD EPYC 9654 96-Core Processor
CPU family:                         25
Model:                              17
Thread(s) per core:                 2
Core(s) per socket:                 96
Socket(s):                          2
Stepping:                           1
Frequency boost:                    enabled
CPU(s) scaling MHz:                 74%
CPU max MHz:                        3707.8120
CPU min MHz:                        1500.0000
BogoMIPS:                           4792.94
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 invpcid_single hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid overflow_recov succor smca fsrm flush_l1d
Virtualization:                     AMD-V
L1d cache:                          6 MiB (192 instances)
L1i cache:                          6 MiB (192 instances)
L2 cache:                           192 MiB (192 instances)
L3 cache:                           768 MiB (24 instances)
NUMA node(s):                       2
NUMA node0 CPU(s):                  0-95,192-287
NUMA node1 CPU(s):                  96-191,288-383
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Not affected
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Vulnerable: eIBRS with unprivileged eBPF
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected
Versions of relevant libraries:
[pip3] mypy_extensions==1.1.0
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.6.4.1
[pip3] nvidia-cuda-cupti-cu12==12.6.80
[pip3] nvidia-cuda-nvrtc-cu12==12.6.77
[pip3] nvidia-cuda-runtime-cu12==12.6.77
[pip3] nvidia-cudnn-cu12==9.5.1.17
[pip3] nvidia-cufft-cu12==11.3.0.4
[pip3] nvidia-curand-cu12==10.3.7.77
[pip3] nvidia-cusolver-cu12==11.7.1.2
[pip3] nvidia-cusparse-cu12==12.5.4.2
[pip3] nvidia-cusparselt-cu12==0.6.3
[pip3] nvidia-nccl-cu12==2.26.2
[pip3] nvidia-nvjitlink-cu12==12.6.85
[pip3] nvidia-nvtx-cu12==12.6.77
[pip3] torch==2.7.1
[pip3] torchx-nightly==2025.7.21
[pip3] triton==3.3.1
[conda] nccl                      [2.27.6.1](https://www.internalfb.com/phabricator/paste/view/2.27.6.1)             h49b9d9a_1    conda-forge
[conda] numpy                     2.2.6                    pypi_0    pypi
[conda] nvidia-cublas-cu12        [12.6.4.1](https://www.internalfb.com/phabricator/paste/view/12.6.4.1)                 pypi_0    pypi
[conda] nvidia-cuda-cupti-cu12    12.6.80                  pypi_0    pypi
[conda] nvidia-cuda-nvrtc-cu12    12.6.77                  pypi_0    pypi
[conda] nvidia-cuda-runtime-cu12  12.6.77                  pypi_0    pypi
[conda] nvidia-cudnn-cu12         [9.5.1.17](https://www.internalfb.com/phabricator/paste/view/9.5.1.17)                 pypi_0    pypi
[conda] nvidia-cufft-cu12         [11.3.0.4](https://www.internalfb.com/phabricator/paste/view/11.3.0.4)                 pypi_0    pypi
[conda] nvidia-curand-cu12        [10.3.7.77](https://www.internalfb.com/phabricator/paste/view/10.3.7.77)                pypi_0    pypi
[conda] nvidia-cusolver-cu12      [11.7.1.2](https://www.internalfb.com/phabricator/paste/view/11.7.1.2)                 pypi_0    pypi
[conda] nvidia-cusparse-cu12      [12.5.4.2](https://www.internalfb.com/phabricator/paste/view/12.5.4.2)                 pypi_0    pypi
[conda] nvidia-cusparselt-cu12    0.6.3                    pypi_0    pypi
[conda] nvidia-nccl-cu12          2.26.2                   pypi_0    pypi
[conda] nvidia-nvjitlink-cu12     12.6.85                  pypi_0    pypi
[conda] nvidia-nvtx-cu12          12.6.77                  pypi_0    pypi
[conda] torch                     2.7.1                    pypi_0    pypi
[conda] torchx-nightly            2025.7.21                pypi_0    pypi
[conda] triton                    3.3.1                    pypi_0    pypi

Is there another way to verify my environment?

cc @colin2328

[dulinriley]

Hello @mreso, I just ran those tests on my Linux machine and they passed:

$ USE_TENSOR_ENGINE=1 pip install --no-build-isolation -e .
$ pytest python/tests/builtins/test_log.py::TestLogFunctions


================================= test session starts =================================
platform linux -- Python 3.12.3, pytest-8.4.1, pluggy-1.6.0
rootdir: /home/dulinr/code/monarch
configfile: pyproject.toml
plugins: xdist-3.8.0, asyncio-1.1.0, timeout-2.4.0
asyncio: mode=Mode.AUTO, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
timeout: 300.0s
timeout method: signal
timeout func_only: False
collected 10 items                                                                    

python/tests/builtins/test_log.py ..........                                    [100%]

================================== warnings summary ===================================
... # unimportant
=========================== 10 passed, 2 warnings in 27.26s ===========================

This was working on commit 2ff2097 (Mon Jul 21 09:26 AM Pacific).
Similarly for the GRPO actor example:

$ python examples/grpo_actor.py
[Step 00] loss=-0.071
[Step 01] loss=-0.074
[Step 02] loss=-0.075
[Step 03] loss=0.003
[Step 04] loss=-0.000
✅ done

I am not using conda BTW, just installed the various packages with rustup and apt on Ubuntu directly.
I use a virtualenv to manage the dependencies (python -m venv .venv, then install).

When you did pip install, did you use the -e flag (for "editable wheel")? That might explain the "FileNotFoundError: [Errno 2] No such file or directory: 'process_allocator'" you see.

$ pip list
Package                  Version   Editable project location
------------------------ --------- -------------------------
certifi                  2025.7.14
charset-normalizer       3.4.2
cloudpickle              3.1.1
docker                   7.1.0
docstring_parser         0.17.0
execnet                  2.1.1
filelock                 3.18.0
fsspec                   2025.5.1
idna                     3.10
iniconfig                2.1.0
Jinja2                   3.1.6
lark                     1.2.2
MarkupSafe               3.0.2
monarch                  0.0.1     /home/dulinr/code/monarch
mpmath                   1.3.0
mypy_extensions          1.1.0
networkx                 3.5
nodeenv                  1.9.1
numpy                    2.3.1
nvidia-cublas-cu12       12.6.4.1
nvidia-cuda-cupti-cu12   12.6.80
nvidia-cuda-nvrtc-cu12   12.6.77
nvidia-cuda-runtime-cu12 12.6.77
nvidia-cudnn-cu12        9.5.1.17
nvidia-cufft-cu12        11.3.0.4
nvidia-cufile-cu12       1.11.1.6
nvidia-curand-cu12       10.3.7.77
nvidia-cusolver-cu12     11.7.1.2
nvidia-cusparse-cu12     12.5.4.2
nvidia-cusparselt-cu12   0.6.3
nvidia-nccl-cu12         2.26.2
nvidia-nvjitlink-cu12    12.6.85
nvidia-nvtx-cu12         12.6.77
packaging                25.0
pip                      25.1.1
pluggy                   1.6.0
Pygments                 2.19.2
pyre-extensions          0.0.32
pyright                  1.1.403
pytest                   8.4.1
pytest-asyncio           1.1.0
pytest-timeout           2.4.0
pytest-xdist             3.8.0
PyYAML                   6.0.2
pyzmq                    27.0.0
requests                 2.32.4
semantic-version         2.10.0
setuptools               80.9.0
setuptools-rust          1.11.1
sympy                    1.14.0
tabulate                 0.9.0
torch                    2.7.1
torchx-nightly           2025.7.21
triton                   3.3.1
typing_extensions        4.14.0
typing-inspect           0.9.0
urllib3                  1.26.20
wheel                    0.45.1

[mreso]

Thanks for checking on the issue and sorry for forgetting to mention this in the original issue. I also tried running the tests separately and they pass:

$ pytest python/tests/builtins/test_log.py::TestLogFunctions
================================================================================================================ test session starts ================================================================================================================
platform linux -- Python 3.10.18, pytest-8.4.1, pluggy-1.6.0
rootdir: /home/mreso/grpo/monarch
configfile: pyproject.toml
plugins: xdist-3.8.0, timeout-2.4.0, asyncio-1.1.0
timeout: 300.0s
timeout method: signal
timeout func_only: False
asyncio: mode=auto, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 10 items

python/tests/builtins/test_log.py ..........                                                                                                                                                                                                  [100%]

================================================================================================================ 10 passed in 27.14s ================================================================================================================

But running all tests:

pytest python/tests/ -v -m "not oss_skip"
================================================================================================================ test session starts ================================================================================================================
platform linux -- Python 3.12.9, pytest-8.4.1, pluggy-1.6.0 -- /home/mreso/grpo/.venv/bin/python3
cachedir: .pytest_cache
rootdir: /home/mreso/grpo/monarch
configfile: pyproject.toml
plugins: xdist-3.8.0, timeout-2.4.0, asyncio-1.1.0
timeout: 300.0s
timeout method: signal
timeout func_only: False
asyncio: mode=Mode.AUTO, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 595 items / 10 deselected / 585 selected

python/tests/_monarch/test_actor.py::test_python_message PASSED                                                                                                                                                                               [  0%]
python/tests/_monarch/test_actor_mesh.py::test_bind_and_pickling PASSED                                                                                                                                                                       [  0%]
python/tests/_monarch/test_actor_mesh.py::test_cast_handle PASSED                                                                                                                                                                             [  0%]
python/tests/_monarch/test_actor_mesh.py::test_cast_ref PASSED                                                                                                                                                                                [  0%]
python/tests/_monarch/test_actor_mesh.py::test_slice_actor_mesh_handle PASSED                                                                                                                                                                 [  0%]
python/tests/_monarch/test_actor_mesh.py::test_slice_actor_mesh_ref PASSED                                                                                                                                                                    [  1%]
python/tests/_monarch/test_client.py::TestClient::test_simple_with_error_response PASSED                                                                                                                                                      [  1%]
python/tests/_monarch/test_client.py::TestClient::test_simple_with_result_response PASSED                                                                                                                                                     [  1%]
python/tests/_monarch/test_client.py::TestClient::test_tensor PASSED                                                                                                                                                                          [  1%]
python/tests/_monarch/test_controller.py::TestController::test_node PASSED                                                                                                                                                                    [  1%]
python/tests/_monarch/test_controller.py::TestController::test_node_serde PASSED                                                                                                                                                              [  1%]
python/tests/_monarch/test_controller.py::TestController::test_send PASSED                                                                                                                                                                    [  2%]
python/tests/_monarch/test_controller.py::TestController::test_send_no_ranks PASSED                                                                                                                                                           [  2%]
python/tests/_monarch/test_controller.py::TestController::test_send_serde PASSED                                                                                                                                                              [  2%]
python/tests/_monarch/test_hyperactor.py::test_import PASSED                                                                                                                                                                                  [  2%]
python/tests/_monarch/test_hyperactor.py::test_actor_id PASSED                                                                                                                                                                                [  2%]
python/tests/_monarch/test_hyperactor.py::test_no_hang_on_shutdown PASSED                                                                                                                                                                     [  2%]
python/tests/_monarch/test_hyperactor.py::test_allocator PASSED                                                                                                                                                                               [  3%]
python/tests/_monarch/test_hyperactor.py::test_proc_mesh PASSED                                                                                                                                                                               [  3%]
python/tests/_monarch/test_hyperactor.py::test_actor_mesh PASSED                                                                                                                                                                              [  3%]
python/tests/_monarch/test_mailbox.py::test_accumulator PASSED                                                                                                                                                                                [  3%]
python/tests/_monarch/test_mailbox.py::test_reducer PASSED                                                                                                                                                                                    [  3%]
python/tests/_monarch/test_ndslice.py::TestNdslice::test_ndslice PASSED                                                                                                                                                                       [  3%]
python/tests/_monarch/test_ndslice.py::TestNdslice::test_pickle PASSED                                                                                                                                                                        [  4%]
python/tests/_monarch/test_ndslice.py::TestNdslice::test_slice PASSED                                                                                                                                                                         [  4%]
python/tests/_monarch/test_ndslice.py::TestNdslice::test_slice_coordinates PASSED                                                                                                                                                             [  4%]
python/tests/_monarch/test_ndslice.py::TestNdslice::test_slice_from_list PASSED                                                                                                                                                               [  4%]
python/tests/_monarch/test_ndslice.py::TestNdslice::test_slice_index PASSED                                                                                                                                                                   [  4%]
python/tests/_monarch/test_ndslice.py::TestNdslice::test_slice_iter PASSED                                                                                                                                                                    [  4%]
python/tests/_monarch/test_ndslice.py::TestNdslice::test_slice_repr PASSED                                                                                                                                                                    [  5%]
python/tests/_monarch/test_ndslice.py::TestShape::test_shape_repr PASSED                                                                                                                                                                      [  5%]
python/tests/_monarch/test_ndslice.py::TestSelection::test_parse_invalid PASSED                                                                                                                                                               [  5%]
python/tests/_monarch/test_ndslice.py::TestSelection::test_parse_repr PASSED                                                                                                                                                                  [  5%]
python/tests/_monarch/test_worker.py::TestWorker::test_backend_network_init PASSED                                                                                                                                                            [  5%]
python/tests/_monarch/test_worker.py::TestWorker::test_backend_network_init_point_to_point PASSED                                                                                                                                             [  5%]
python/tests/_monarch/test_worker.py::TestWorker::test_borrow_create PASSED                                                                                                                                                                   [  6%]
python/tests/_monarch/test_worker.py::TestWorker::test_borrow_drop PASSED
python/tests/_monarch/test_worker.py::TestWorker::test_borrow_create PASSED                                                                                                                                                       13:13:46 [55/19962]
python/tests/_monarch/test_worker.py::TestWorker::test_borrow_drop PASSED                                                                                                                                                                     [  6%]
python/tests/_monarch/test_worker.py::TestWorker::test_borrow_first_use PASSED                                                                                                                                                                [  6%]
python/tests/_monarch/test_worker.py::TestWorker::test_borrow_last_use PASSED                                                                                                                                                                 [  6%]
python/tests/_monarch/test_worker.py::TestWorker::test_call_function PASSED                                                                                                                                                                   [  6%]
python/tests/_monarch/test_worker.py::TestWorker::test_call_function_live_function PASSED                                                                                                                                                     [  7%]
python/tests/_monarch/test_worker.py::TestWorker::test_call_function_referencable_args PASSED                                                                                                                                                 [  7%]
python/tests/_monarch/test_worker.py::TestWorker::test_command_group PASSED                                                                                                                                                                   [  7%]
python/tests/_monarch/test_worker.py::TestWorker::test_create_device_mesh PASSED                                                                                                                                                              [  7%]
python/tests/_monarch/test_worker.py::TestWorker::test_create_pipe PASSED                                                                                                                                                                     [  7%]
python/tests/_monarch/test_worker.py::TestWorker::test_create_remote_process_group PASSED                                                                                                                                                     [  7%]
python/tests/_monarch/test_worker.py::TestWorker::test_create_stream PASSED                                                                                                                                                                   [  8%]
python/tests/_monarch/test_worker.py::TestWorker::test_delete_refs PASSED                                                                                                                                                                     [  8%]
python/tests/_monarch/test_worker.py::TestWorker::test_exit PASSED                                                                                                                                                                            [  8%]
python/tests/_monarch/test_worker.py::TestWorker::test_pipe_recv PASSED                                                                                                                                                                       [  8%]
python/tests/_monarch/test_worker.py::TestWorker::test_reduce PASSED                                                                                                                                                                          [  8%]
python/tests/_monarch/test_worker.py::TestWorker::test_request_status PASSED                                                                                                                                                                  [  8%]
python/tests/_monarch/test_worker.py::TestWorker::test_send_tensor PASSED                                                                                                                                                                     [  9%]
python/tests/_monarch/test_worker.py::TestWorker::test_send_value PASSED                                                                                                                                                                      [  9%]
python/tests/builtins/test_log.py::TestLogFunctions::test_log_remote_default_level[py] PASSED                                                                                                                                                 [  9%]
python/tests/builtins/test_log.py::TestLogFunctions::test_log_remote_default_level[rs] FAILED
...

My first attempt was in a uv venv but I reinstalled in conda to eliminate possible variables with the same result.

not sure if this is related but I also noticed that this test sometimes hangs on my end: python/tests/_monarch/test_mailbox.py::test_reducer but again if I run it separately is runs fine.

[mreso]

This was working on commit 2ff2097 (Mon Jul 21 09:26 AM Pacific). Similarly for the GRPO actor example:

That commit failed the test in CI with the same error as I see btw: https://github.com/pytorch-labs/monarch/actions/runs/16422747756/job/46406304139#step:15:2183

[dulinriley]

Thank you for pointing that out, when I run python/tests/ -v -m "not oss_skip" I also notice the hang in test_reducer and the failure in test_log_remote_default_level[rs]. The test_reducer seems to be non-deterministic as I was not able to repro it a second time.

However, I don't think those are indicative of a problem with the GRPO actor example, but rather a problem with our pytest setup. I'll make some subtasks for those testing issues, but since I was able to reproduce them and GRPO still worked for me, they don't seem related to the original issue.

[mreso]

Thanks @dulinriley, that a good signal. As the unit tests seem to not work in your case as well I can strike them from the list of potential causes but I am still not sure about a good method to check my environment then as the GRPO example fails for me. Is there a more basic example I can try to check if the installation was successful than the grpo example?

[amirafzali]

Thanks for checking on the issue and sorry for forgetting to mention this in the original issue. I also tried running the tests separately and they pass:
...
But running all tests:
...

My first attempt was in a uv venv but I reinstalled in conda to eliminate possible variables with the same result.

not sure if this is related but I also noticed that this test sometimes hangs on my end: python/tests/_monarch/test_mailbox.py::test_reducer but again if I run it separately is runs fine.

Since #395, tests are no longer subprocessed but instead run in the same process sequentially. The fact that those pass in isolation but fail when the entire suite is run implies that a prior test is ending in a bad state and causing future failure. Eyeballing signals think something broke between 0588650 and d280486 but a bisect would probably be needed to boil it down further. The first logged error in the sequence is:

______________ TestLogFunctions.test_log_remote_default_level[rs] ______________

self = <test_log.TestLogFunctions object at 0x7fdbd2c34310>
mock_log = <MagicMock name='logger' id='140573207152864'>, backend_type = 'rs'

    @patch("monarch.builtins.log.logger")
    def test_log_remote_default_level(self, mock_log, backend_type):
>       with self.local_device_mesh(1, 1, backend_type):

self = <monarch.rust_local_mesh.WorkerServer object at 0x7fdbe46de4a0>
timeout = 9.999992609024048

    def _recv(
        self, timeout: float | None = None
    ) -> WorkerServerResponse | ControllerServerResponse:
        assert not self._aborted
        assert self._lock is not None
        assert self._lock.locked()
        assert self._pipe is not None
>       ready, _, _ = select.select([self._pipe[0]], [], [], timeout)
E       ValueError: filedescriptor out of range in select()

/opt/conda/envs/venv/lib/python3.10/site-packages/monarch/rust_local_mesh.py:327: ValueError

cc @dulinriley

[mreso]

@dulinriley I am going to make this issue exclusively about the test failures so you can track the progress. As my grpo example issue seems to be unrelated I am going to create a new issue for that one.

[mreso]

To follow up on the grpo issue, I was able to make some progress by trying it on a different machine. Turns out that for some reason the monarch.rdma binding got compiled on the first machine but where not functioning due to lack of hw support. On the new machine (of the same hw type) the binding where not created and I already got an error during module imports.

ModuleNotFoundError: No module named 'monarch.rdma'

The question not becomes: why was monarch.rdma created on the first machine which does not support it. We will track that in a separate issue.

[allenwang28]

Re-assigning this to me as the current oncall. I am working on this in #616.

I've synced with Matthias regarding monarch.rdma - this is fixable by monarch.tensor_engine.rdma. I will add in this fix, although we may want to create a version of this that doesn't actually use RDMA (since this requires machines with backend network)

[allenwang28]

Bisect pinpointed to this commit: #551

Following up with @pzhan9 on this - I agree with @amirafzali in #604 (comment) that it's likely that something isn't cleaning up properly