Can we stop lowering FC to Matmul+BatchedAdd?

[bertmaher]

GEMM is almost always implemented as Y = AX+b, so we actually end up doing something less efficient by separating these into two nodes. It's pretty universal on HW backends, too. We could write new optimizations to re-fuse them, but why?

[jackm321]

@bertmaher should we disable this per-backend with Backend::shouldLower() or do you think we should just get rid of it altogether?

[bertmaher]

@jackm321 the unfortunate thing about disabling it with shouldLower is that IRGen doesn't know what to do with FullyConnected, so it aborts. So instead you have to create a new backend-specific node and transform to that. I'd be OK with selectively disabling via shouldLower, as long as we had a generic FullyConnectedInst to make IRGen happy.

(Btw, this issue is more for discussion -- I want to know if people think this is a good idea before we go off and do it.)

[nadavrot]

@bertmaher +1
I think that this suggestion makes sense. We'll need to update our optimizations that deal with matrix multiplications (merge horizontally).

Our convolutions could also be split to [Conv + BatchedAdd] but we've decided to keep them fused. We can apply the same logic here and decide that FCs should also stay fused.