Oracle 10g EM Dbconsole will not start up!

Question : Oracle 10g EM Dbconsole will not start up!

Hello…
I’m running Oracle 10g (10.2.0.1) on RHEL4. My listener, database and Oracle HTTP Server are up and running. The sole purpose of this database is to run Oracle Application Express applications. The oracle apex applications are all running fine.

However, I cannot for the life of me get Oracle EM (dbconsole) running. I’ve tried everything but obviously not the right thing. I can’t console to my running database? It keeps failing everytime I start it. I’ve looked at some logs but I don’t know what I’m looking for. Oracle EM was running just fine on Friday, but wouldn’t come up Monday morning.

My url for EM is http://172.23.183.169:1158/em
At the terminal session prompt I type >emctl start dbconsole
It comes back after a 5 minute timeout saying it failed. It appears that the agent (3938) does not want to start.
Here is some log info further below…
Please let me know what direction I should go on to resolve this. Thx!
Regards,
Ken

ENV VARIABLES:
[oracle@clthcmmb03 ~]$ env|sort
COLORTERM=gnome-terminal
DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-uEKexk6EGd
DESKTOP_SESSION=default
DESKTOP_STARTUP_ID=
DISPLAY=:0.0
G_BROKEN_FILENAMES=1
GDMSESSION=default
GNOME_DESKTOP_SESSION_ID=Default
GNOME_KEYRING_SOCKET=/tmp/keyring-uFau6k/socket
GTK_RC_FILES=/etc/gtk/gtkrc:/home/oracle/.gtkrc-1.2-gnome2
HISTSIZE=1000
HOME=/home/oracle
HOSTNAME=clthcmmb03
INPUTRC=/etc/inputrc
KDEDIR=/usr
LANG=en_US.UTF-8
LD_LIBRARY_PATH=/home/oracle/oracle/product/10.2.0/db_1/lib:/home/oracle/oracle/product/10.2.0/db_2/opmn/lib:
LESSOPEN=|/usr/bin/lesspipe.sh %s
LOGNAME=oracle
LS_COLORS=no=00:fi=00:di=00;34:ln=00;36:pi=40;33:so=00;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=00;32:*.cmd=00;32:*.exe=00;32:*.com=00;32:*.btm=00;32:*.bat=00;32:*.sh=00;32:*.csh=00;32:*.tar=00;31:*.tgz=00;31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.zip=00;31:*.z=00;31:*.Z=00;31:*.gz=00;31:*.bz2=00;31:*.bz=00;31:*.tz=00;31:*.rpm=00;31:*.cpio=00;31:*.jpg=00;35:*.gif=00;35:*.bmp=00;35:*.xbm=00;35:*.xpm=00;35:*.png=00;35:*.tif=00;35:
MAIL=/var/spool/mail/oracle
ORACLE_HOME=/home/oracle/oracle/product/10.2.0/db_1
ORACLE_SID=apex
PATH=/home/oracle/oracle/product/10.2.0/db_1/bin:/home/oracle/oracle/product/10.2.0/db_2/opmn/bin:/home/oracle/oracle/product/10.2.0/db_1/bin:/home/oracle/oracle/product/10.2.0/db_2/opmn/bin:/usr/kerberos/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin
PWD=/home/oracle
SESSION_MANAGER=local/clthcmmb03:/tmp/.ICE-unix/6580
SHELL=/bin/bash
SHLVL=2
SSH_AGENT_PID=6632
SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
SSH_AUTH_SOCK=/tmp/ssh-dybiGw6580/agent.6580
TERM=xterm
USER=oracle
_=/usr/bin/env
WINDOWID=39845962
XAUTHORITY=/home/oracle/.Xauthority

TERMINAL SESSION:
[oracle@clthcmmb03 ~]$ emctl start dbconsole
TZ set to US/Eastern
Oracle Enterprise Manager 10g Database Control Release 10.2.0.1.0
Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
http://clthcmmb03:1158/em/console/aboutApplication
Starting Oracle Enterprise Manager 10g Database Control ………………………………………………………………………………… failed.
——————————————————————
Logs are generated in directory /home/oracle/oracle/product/10.2.0/db_1/clthcmmb03_apex/sysman/log

EMDCTL.TRC:
2007-08-13 15:08:39 Thread-3086911712 WARN http: snmehl_connect: connect failed to (clthcmmb03:3938): Connection refused (error = 111)
2007-08-13 15:08:45 Thread-3086911712 WARN http: snmehl_connect: connect failed to (clthcmmb03:1158): Connection refused (error = 111)

EMAGENT.TRC:
2007-08-13 14:40:34 Thread-3086911712 ERROR http: snmehl_allocateListenerPort: failed to bind socket: Address already in use (errno=98)
2007-08-13 14:40:34 Thread-3086911712 ERROR http: nmehl_startHttpListener: failed to listen to HTTP port for http://clthcmmb03:3938/emd/main
2007-08-13 14:40:34 Thread-3086911712 ERROR main: nmehl_startHttpListener failed starting http listener
2007-08-13 15:10:12 Thread-3086911712 ERROR http: snmehl_allocateListenerPort: failed to bind socket: Address already in use (errno=98)
2007-08-13 15:10:12 Thread-3086911712 ERROR http: nmehl_startHttpListener: failed to listen to HTTP port for http://clthcmmb03:3938/emd/main
2007-08-13 15:10:12 Thread-3086911712 ERROR main: nmehl_startHttpListener failed starting http listener

EMOMS.TRC:
2007-08-12 02:00:36,607 [NotificationMgrThread] ERROR jdbc.ConnectionWrapper _applySessionParams.2980 – SQLException message: ORA-01089: immediate shutdown in progress – no operations are permitted

java.sql.SQLException: ORA-01089: immediate shutdown in progress – no operations are permitted

at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:111)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:330)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:287)
at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:742)
at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:214)
at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:953)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1260)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3290)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3334)
at oracle.sysman.util.jdbc.ConnectionWrapper._applySessionParams(ConnectionWrapper.java:2971)
at oracle.sysman.util.jdbc.ConnectionWrapper.t>(ConnectionWrapper.java:181)
at oracle.sysman.emSDK.svc.conn.FGAConnection.it>(FGAConnection.java:115)
at oracle.sysman.emSDK.svc.conn.FGAConnectionCache.getFGAConnection(FGAConnectionCache.java:192)
at oracle.sysman.emSDK.svc.conn.ConnectionService.getSystemConnection(ConnectionService.java:683)
at oracle.sysman.emdrep.notification.NotificationMgr.getConnection(NotificationMgr.java:252)
at oracle.sysman.emdrep.notification.NotificationMgr.resetConnection(NotificationMgr.java:298)
at oracle.sysman.emdrep.notification.NotificationMgr.run(NotificationMgr.java:1190)

Solution: Oracle 10g EM Dbconsole will not start up!

Because I will be on leave next week I will advise you
after setting the environment variables correctly also to
recreate the OEM repository:

To recreate the configuration files and repository for Database Console, run:
$ emca -config dbcontrol db -repos recreate

Enter the following information:
Database SID: TESTDB
Database Control is already configured for the database TESTDB
You have chosen to configure Database Control for managing the database TESTDB
This will remove the existing configuration and the default settings and perform a fresh configuration
Do you wish to continue? [yes(Y)/no(N)]: Y
Listener port number: 1521
Password for SYS user: change_on_install
Password for DBSNMP user: manager
Password for SYSMAN user: manager
Email address for notifications (optional):
Outgoing Mail (SMTP) server for notifications (optional):
—————————————————————–

You have specified the following settings

Database ORACLE_HOME ……………. /u01/app/oracle/product/10.2.0/db_1

Database hostname ……………. linux3
Listener port number ……………. 1521
Database SID ……………. TESTDB
Email address for notifications ……………
Outgoing Mail (SMTP) server for notifications ……………

—————————————————————–
Do you wish to continue? [yes(Y)/no(N)]: Y
Jul 24, 2006 10:08:50 PM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at /u01/app/oracle/product/10.2.0/db_1/cfgtoollogs/emca/TESTDB/emca_2006-07-24_10-08-20-PM.log.
Jul 24, 2006 10:08:53 PM oracle.sysman.emcp.util.DBControlUtil stopOMS
INFO: Stopping Database Control (this may take a while) …
Jul 24, 2006 10:09:29 PM oracle.sysman.emcp.EMReposConfig dropRepository
INFO: Dropping the EM repository (this may take a while) …
Jul 24, 2006 10:11:23 PM oracle.sysman.emcp.EMReposConfig invoke
INFO: Repository successfully dropped
Jul 24, 2006 10:11:23 PM oracle.sysman.emcp.EMReposConfig createRepository
INFO: Creating the EM repository (this may take a while) …
Jul 24, 2006 10:15:09 PM oracle.sysman.emcp.EMReposConfig invoke
INFO: Repository successfully created
Jul 24, 2006 10:15:18 PM oracle.sysman.emcp.util.DBControlUtil startOMS
INFO: Starting Database Control (this may take a while) …
Jul 24, 2006 10:17:21 PM oracle.sysman.emcp.EMDBPostConfig performConfiguration
INFO: Database Control started successfully
Jul 24, 2006 10:17:22 PM oracle.sysman.emcp.EMDBPostConfig performConfiguration
INFO: >>>>>>>>>>> The Database Control URL is http://linux3:1158/em <<<<<<<<<<<
Enterprise Manager configuration completed successfully
FINISHED EMCA at Jul 24, 2006 10:17:22 PM