That is an issue with R, not R.NET. I could reproduce the issue with the following code in RGui.exe, without R.NET involved. I suggest you try to reproduce it in R, look for similar issues reported on StackOverflow, and post a question if you do not find answers already posted.
R version 3.0.2 (2013-09-25)
Platform: i386-w64-mingw32/i386 (32-bit)
R version 3.0.2 (2013-09-25)
Platform: i386-w64-mingw32/i386 (32-bit)
> blah <- rep(1:3, 1e7)
> blah <- rep(1:3, 1e8)
> blah <- rep(1:3, 2e7)
Error: cannot allocate vector of size 228.9 Mb
> blah <- rep(1:3, 2e7)
Error: cannot allocate vector of size 228.9 Mb
> rm(blah)
> blah <- rep(1:3, 2e7)