Text size is handled differently in the Layout window from what you see in a Map window. The Layout window contains a live map that you can edit, so that you do not need to toggle between the layout and editing in a separate Map window. The Layout window shows the map using the printer resolution, so that you know beforehand what your printout will look like (you will know what labels will be on the printed map). This is based on the current layout page setup including the printer resolution. If you are running out of memory, you may want to use Page Setup to lower the printer resolution.
See also, Viewing and Printing Text in a Map.