During installation you may encounter the error
ORA-03206: maximum file size of (13107200) blocks in AUTOEXTEND clause is out of range
In my case, the RCU attempted to create a temporary tablespace with autoextend of 100G. This is too much for a smallfile tablespace, and the error is thrown.
I checked the database creation scripts, and it is indeed created with smallfile as the default tablespace type.
To solve the problem, alter your database default setting:
ALTER DATABASE SET DEFAULT BIGFILE TABLESPACE;
and retry the operation.
Documentation for Oracle 12.2 can be found here