I faced with that error during running DBCA in silent mode. Complete error message is:
ORA-27104: system-defined limits for shared memory was misconfigured

I have been checked all related parameter such as /etc/sysctl.conf & /etc/security/limits.conf

After investigate issue I found MOS  1922934.1 which is refer to bug.

To resolve this issue , please apply or ask for one-off patch for Bug 14202016.


As a workaround, increase the value for kernel parameter shmall to 9272480