  We analyze the performance of a quantum computer architecture combining a
small processor and a storage unit. By focusing on integer factorization, we
show a reduction by several orders of magnitude of the number of processing
qubits compared to a standard architecture using a planar grid of qubits with
nearest-neighbor connectivity. This is achieved by taking benefit of a
temporally and spatially multiplexed memory to store the qubit states between
processing steps. Concretely, for a characteristic physical gate error rate of
$10^{-3}$, a processor cycle time of 1 microsecond, factoring a 2048 bits RSA
integer is shown possible in 177 days with a processor made with 13436 physical
qubits and a multimode memory with 2 hours storage time. By inserting
additional error-correction steps, storage times of 1 second are shown to be
sufficient at the cost of increasing the runtime by about 23 %. Shorter
runtimes (and storage times) are achievable by increasing the number of qubits
in the processing unit. We suggest realizing such an architecture using a
microwave interface between a processor made with superconducting qubits and a
multiplexed memory using the principle of photon echo in solids doped with
rare-earth ions.
