[L0] Backend context will contain all devices even if frontend context contains single device

Migrate from https://github.com/intel/llvm/issues/10295

The reason that I bring this up again because with PyTorch usage, we can't strictly assign device to specific process and defend wrongfully submission of kernels, for example, to device that different from allocation residency.