Hello Franz,
from your parameters a user can allocate 2GB in extended memory:
ztta/roll_extension 2000000000
and from your dump is seen 2GB in extended memory were really allocated:
erweiterter Speicherbereich(EM) 2002720576
The other amount of memory in your dump (heap):
fest allokierter Speicher(HEAP) 1706998000
is limited by these parameters:
abap/heap_area_total 6000000000
abap/heap_area_dia 6000000000
abap/heap_area_nondia 0
A user in a dialog process can allocate 6GB and in a non-dialog process can allocate unlimited amount od memory. But the users altogether cannot allocate more than 6GB (parameter abap/heap_area_total). Since you see less than 6GB in the heap in dump, then you should check, how many user were on the system at that time and if they altogether have allocated 6GB in heap. You need to increase the value for abap/heap_area_total if you have enough RAM and space in SWAP on the server.
To get a better overview about memory consumption please take a look at the history in ST02 part "Extended and heap memory (MEM)". You will have then an idea, if the application server requires more memory and in which area.
Regards, Alwina