It isn't just memory use, the transfer time to the device starts to introduce significant latency.
Large displays usually (always?) have their own framebuffer and provide support for graphics primitives. In my opinion the way to write drivers for these is to use these rather than subclassing from framebuffer. Examples of such drivers are the official LCD160CR display driver and Robert's SSD1963 driver.
Subclassing framebuf is a quick way of writing a driver for small displays for applications where some latency is tolerable, or for displays which lack their own framebuf. My SSD1331 driver shows how simple such a driver can be. Consequently I see the nano-gui as being quite a minimal interface for small displays only. The biggest I plan to support is the 128*128 pixel 1.5 inch Adafruit colour OLED.
To minimise the size of the framebuf I intend to support 8-bit colour only (rrrgggbb) with colours being mapped to the device's colour space on the fly. Nobody is going to be displaying pictures, but 8-bit colour is fine for applications such as displaying sensor data.