Using dbms_system.set_int_param_in_session, one can set max_dump_file_size of a running session. In the example it sets trace file to 100MB
exec sys.dbms_system.set_int_param_in_session(sid => <sid>, serial# => <serial#>, parnam => 'MAX_DUMP_FILE_SIZE', intval => 100*1024*1024);