I face with that issue during 10.2.0.5 to 11.2.0.3 database upgrade. My os is : SUN-Sparc x86-64

Here is the steps:

1. Set 11R2 Oracle_home

2. Open 10.2.0.5 database

sqlplus “/as sysdba”

SQL*Plus: Release 10.2.0.5.0 – Production

Copyright (c) 1982, 2010, Oracle. All Rights Reserved.

Connected to an idle instance.

SQL> startup UPGRADE

ORACLE instance started.

Total System Global Area 1253031936 bytes
Fixed Size 2158896 bytes
Variable Size 822087376 bytes
Database Buffers 419430400 bytes
Redo Buffers 9355264 bytes
Database mounted.
Database opened.

SQL> @$ORACLE_HOME/rdbms/admin/catupgrd.sql

The below error is received while running catupgrd.sql script

SQL> set errorlogging on table sys.registry$error;
SP2-0735: unknown SET option beginning “errorloggi…”
SQL> set errorlogging on identifier ‘RDBMS’;

SQL> insert into props$ (name, value$, comment$)
2 (select ‘DEFAULT_PERMANENT_TABLESPACE’, ‘SYSTEM’,
3 ‘Default Permanent Tablespace ID’ from dual
4 where ‘DEFAULT_PERMANENT_TABLESPACE’ NOT IN
5 (select name from props$ where name = ‘DEFAULT_PERMANENT_TABLESPACE’));

Segmentation fault (core dumped)

There was no any other error on trc file or alertlog.

After investigate issue I found MOS Segmentation fault (core dumped) – Upgrade to 11.2.0.2 (Doc ID 1401297.1)

Here is the steps:
Set environment variable for LD_LIBRARY_PATH to 11.2.0.2 or TARGET DATABASE
export LD_LIBRARY_PATH=/<path to the 11.2 Oracle_Home>/lib

Connect to 11gR2 database as sysdba and Start upgrade

SQL> sqlplus “/ as sysdba”
SQL> startup upgrade
SQL> @$ORACLE_HOME/rdbms/admin/catupgrd.sql