I would say that latency doesn't matter here.
You have to differentiate between "local" accesses (core-local 32 KB memory) and "remote" accesses (anything else).
For local reads and writes, you won't experience any latency.
For remote writes, you won't experience any latency (assuming an idle mesh); writes are fire-and-forget. If the mesh is busy, there is round-robin arbitration.
For remote reads, you always have a huge latency, so you should avoid them in any case.
The available bandwidth is defined by the external eLink interface, not the memory you attach to the other side. So I wouldn't expect much difference. Also, FPGA RAM is expensive.