MemoryConstrained[expr, b] evaluates expr, stopping if more than b bytes of memory are requested.
MemoryConstrained[expr, b, failexpr] returns failexpr if the memory constraint is not met.
MemoryConstrained generates an interrupt to stop the evaluation of expr if the amount of additional memory requested during the evaluation of expr exceeds b bytes.
MemoryConstrained evaluates failexpr only if the evaluation is aborted.
MemoryConstrained returns $Aborted if the evaluation is aborted and no failexpr is specified.
Aborts generated by MemoryConstrained are treated just like those generated by Abort, and can thus be overruled by AbortProtect.
See The Mathematica Book: Section 2.13.4.
See also: TimeConstrained, MaxMemoryUsed, $RecursionLimit, Abort.