Administrator's Guide
CHAPTER 14
This chapter describes some techniques and procedures that you can use for troubleshooting the Novell exteNd Application Server. This chapter contains the following sections:
Consider turning on error logging at all times when running the server. Error logging is a lightweight process that prints detailed information about error messages either to the AgErrorLog table in the SilverMaster or to a file you designate. You can activate logging using the SMC. For more information, see Using server logging.
When logging to the SilverMaster, you can view the log in the SMC by selecting the Monitor icon from the toolbar and then selecting Logs.
For more information, see Displaying logs.
The Debug options in the SMC enable the printing of server debug messages to the server console. Options include debugging client requests, Web applications, and SQL statements. You should activate debugging options only for application debugging purposes, as this activity can significantly inhibit server performance.
NOTE: If you are running the server as a service in Windows, the output is printed to the error log instead of the console window.
Change the value for the type of activity you are debugging.
The number you enter indicates the level of detail you want displayed. The value 0 means that messages are not printed. You have the following debugging options:
If you have persistent problems with a database connection, use JDBC tracing.
Shut down the server (see Shutting down the application server).
Open the httpd.props file (located in server's \Resources directory).
Add an http-server.Jdbc.DriverManager.LogFile entry to the props file and point it to the log file. For example, if the log file is d:\test\jdbc.log, create this line in your httpd.props file:
http-server.Jdbc.DriverManager.LogFile
=d:\\test\\jdbc.log
NOTE: Use JDBC tracing for troubleshooting only, as it will slow down the server and use considerable disk space.
You can type commands into the server's console window (the window in which the server was started) to obtain diagnostic information about the state of the server. For example, you can get information on memory usage, threads, sessions, and server system properties. You can also enable tracing for different subsystems.
To learn more after the server has started:
help
All available commands are listed, and the command line is marked by a ! character (you can change the character using the prompt command).
The Watcher tool helps you understand the state of the server in cases when the server becomes unresponsive. When it is activated, the Watcher logs the state of the server once a minute.
TIP: You might find the Watcher valuable when faced with problems that are hard to debug.
Add the following property to the httpd.props file:
http-server.com.sssw.srv.httpdwatcher
Set the value of the property to the pathname of a watcher configuration file. For example:
http-server.com.sssw.srv.httpdwatcher=c:\\temp\\watchconfig.txt
TIP: Remember to escape backslashes in the httpd.props file.
If this property is set when the server is started, the server will create a watcher thread. The watcher thread sleeps, waking up once a minute to check for the existence of the watcher configuration file supplied as the value of the
httpdwatcher property.
If the watcher configuration file does not exist As long as the configuration file doesn't exist, the Watcher does nothing and just goes back to sleep. Under these circumstances, the Watcher has minimal impact on server performance. And even if the server hangs, in most cases the Watcher will not hang.
If the watcher configuration file exists When the Watcher discovers that the watcher configuration file exists, it reads the configuration file and uses it to control its further actions.
About the watcher configuration file The watcher configuration file is an ASCII text file that must include:
A flags value that tells the Watcher what information to print
The flags value is a bit-coded integer in which the bits are defined as follows:
An (optional) output file name for the Watcher output (if not supplied, the output goes to the server console)
A typical watcher configuration file looks like this:
7 c:\temp\watchout.txt
This file tells the Watcher to dump information once every minute about threads, sessions, and database connections to the specified output file.
This section describes some reasons the application server might fail to start, and how you can address the problem. For more information on troubleshooting server problems, see Using SilverMasterInit to recreate or refresh SilverMaster.
NOTE: Server failure is often related to the specific database you are using. For database-specific information, see the Database Configuration Guide.
There are two causes of server failure that are due to inadequate system resources:
If you see errors related to database consistency, go to the SMC and execute the Synchronize database schema option, then restart the server.
For more information, see Synchronizing the database schema.
SilverMonitor is a background process running on the server that monitors server status and attempts to restart the server if it terminates abnormally. By default, this process is activated when the server is started. SilverMonitor starts with default parameters, which you can modify. You can also run the server without SilverMonitor.
There are two ways you can modify the parameters:
Specify parameter(s) as server startup options on the DOS command line. See Using startup options.
Specify machine default values by editing the registry (Windows only).
The order of precedence is as follows:
The following is a summary of SilverMonitor parameters:
To modify SilverMonitor parameters in the NT Registry:
HKEY_LOCAL_MACHINE>SOFTWARE>Novell>exteNd>AppServer>version number
When the following dialog displays, type one or more of the options described above. Separate each option with a space.
SilverMonitor writes to the NT EventLog when it restarts the server. It also writes to a SilverMonitor.log file in the directory where it is run (usually the server's \bin directory). This log file gets an entry every time the monitor starts.
When you restart the SilverMonitor, the log file is emptied and restarted.
The application server relies on the SilverMaster database for overall system management. SilverMasterInit is a command-line program that performs several types of processes on the SilverMaster database. The SilverMasterInit executable is located in the server's \bin directory. SilverMasterInit can:
This section contains the following topics:
The table below describes how and when to run each of the SilverMasterInit command-line options. To see a list of options, type the following at the command prompt:
SilverMasterInit -?
Administration accounts There are two administration accounts (database and server). Both accounts are defined during installation. The server administration account restricts who can log in and administer the application server. You define the server administration account using SilverMasterInit. After a default installation, the server administrator user account is part of the predefined Administrators group and has the Locksmith privilege.
The application server uses the database administration account when connecting to the SilverMaster database. The only time you need to specify the SilverMaster database account is when you are running SilverMasterInit.
Entering options You must enter the database user account name and password for all command-line options. You also need to specify Full or Refresh mode for all SilverMasterInit optionsexcept those noted in the following table. When you specify a Full mode database initialization, three options (-A, -n, and -W) require that you also define the server administration account name and password on the command line.
You specify parameter(s) as SilverMasterInit startup options on the command line.
SilverMasterInit startup option |
Description |
Use |
---|---|---|
-? |
Displays usage for SilverMasterInit. |
Use to check option usage. |
+cp:a path |
Appends specified path to the class path. |
This option makes additional Java classes available to applications by appending the specified path to the class path. NOTE: Use the AGCLASSPATH environment variable to extend Java classes. Example: SilverMasterInit |
+cp:p path |
Prepends specified path to the class path. |
Don't use this debugging option without first contacting Novell exteNd Technical Support. Instead, use AGCLASSPATH to make additional Java classes available to applications. See Setting the AGCLASSPATH variable. |
-A adminname |
Specifies the server administrator user name used to log in to and administer the application server. |
This option lets you define a server administrative account name (and password) when you are creating a new SilverMaster database catalog. The server user account you specify will be part of the Administrators group and have full Locksmith privilege. Use this account to administer the server. See About your administrator account. When running a Full mode database initialization you must specify the server administration account name and password. Example: SilverMasterInit -f -U |
-a |
Causes the application server to require users to authenticate themselves. |
Set this parameter if you accidentally restrict Read access to your SilverMaster database, which includes the login resource. This option also provides a quick way to set authentication without running the SMC. See Using server authentication to access the login resource. You don't need to specify Refresh or Full mode when running the -a option. Example: SilverMasterInit -a -U |
-b |
Displays boot environment settings. |
Run to see the initial SilverMaster environment properties used by Full mode or Refresh mode. Example: SilverMasterInit [-f |
-c |
Checks that BLOBs are inserted into the database correctly. |
Run to verify that these objects are properly stored. Example: SilverMasterInit [-f |
-C properties file |
Reads the nameServicePort or IIOP SSL ports properties from the specified properties file. |
The properties file has the same syntax as the httpd.props file, except that you do not specify http-server in front of the property. Instead you specify <hostname>.<portnumber> The hostname and portnumber are separated by a dot (not a colon). The portnumber is the administration port and is only required when it is some value other than 80. To set the nameServicePort, set the following property: <hostname>.<portnumber>.com.sssw.srv.nameServicePort=<nameServicePort> For example: tundra.8080.com.sssw.srv.nameServicePort=55597 To set the IIOP SSL port range, use these properties: <hostname>.<portnumber>.com.sssw.srv.port_iiop_ssl_min <hostname>.<portnumber>.com.sssw.srv.port_iiop_ssl_max The port_iiop_ssl_min property specifies the lower bounds for IIOP SSL. If you do not specify a range, the ORB picks the first available port. Use -1 if you don't need to specify a range. The port_iiop_ssl_max property specifies the upper bounds for IIOP SSL communications. Use -1 for no upper bound. Run SilverMasterInit with the -r option to update an existing SilverMaster. Example: SilverMasterInit -C port.props -U smbb -P password -A administrator -W admin -r |
-D database |
Deletes all Ag tables from the specified SilverMaster database. |
Deletes all existing application server system tables (including users, groups, and licensing data) from the specified SilverMaster database. Use to remove a deployment database from the server. Unlike Full mode, this option deletes application server system data but does not replace it with initial properties. Example:
|
-e error log file |
Writes errors to the specified file, which is created as necessary. |
If no errors are found, a log file is not created. If you don't specify a path, the error log file is stored in the directory from which you ran SilverMasterInit. The default file name is sminit.log. Example:
|
-f |
Runs in Full mode to create a new SilverMaster database. |
Creates new SilverMaster system data and resources. This option deletes existing users, groups, and licensing data. NOTE: By default, the server will be restricted when you run Full mode. To install the server unrestricted (for a development environment), run SilverMasterInit in Full mode with -n. When running a Full mode database initialization, you must specify the server administration account name and password. Example: SilverMasterInit -f -U |
-L jdbc log file |
Writes JDBC debugging information to the specified log file. |
If you don't specify a log file name, this option is ignored. If you don't specify a path, the JDBC log file is stored in the server's \bin directory. Example: SilverMasterInit [-f
|
-l locksmith account |
Specifies a user or group account to grant Locksmith privilege to. |
Use this option if you accidentally delete all accounts that have Locksmith privilege. See Regaining access to SilverMaster. You don't need to specify Refresh or Full mode when running the -l option. Example:
|
-n |
Unrestricts access to the application server. |
Use when you do not want to lock down access to system data nor require user authentication. This option means any user can perform administrative operations and browse directory listings until you lock down access by setting permissions. See Default server and object security. You need to specify Full mode and the server administration user name and password when running the -n option. Example: SilverMasterInit -n -f -U |
-O table space |
Creates all Ag tables in the specified Oracle table space for SilverMaster. |
Use this option when creating a SilverMaster database to use with Oracle. More space (than the default) must be allocated for SilverMaster table objects because of the way an Oracle database stores data. Example: SilverMasterInit [-f |
-P dbpassword |
Specifies the database password used by the application server to access SilverMaster. |
The database administration password and associated user account are stored encrypted in the registry during server installation. The server will use the specified account name and password at startup to access the SilverMaster database. Example: SilverMasterInit [-f |
-p properties file |
Reads startup properties from the specified file. Defaults to httpd.props in the server's \Resources directory. |
Use to specify a SilverMaster startup property file name and location other than the default. After you set the property file option with SilverMasterInit, you need to start the application server from the command line with the -p option to use the new property file. Example: SilverMasterInit [-f |
-r |
Runs in Refresh mode to update SilverMaster resources. |
This process skips some of the database installation steps used by Full mode. Use this option to refresh SilverMaster system data and resources when you don't want to delete existing users, groups, and licensing data. Example:
|
-W adminpassword |
Specifies the server administrator account password used to log in to and administer the application server. |
Use the server administrator user and account password to administer the server. When running a Full mode database initialization, you must specify the server administration account name and password. Example: SilverMasterInit -f -U |
-U dbusername |
Specifies the application server's SilverMaster database user account. |
The database administration user account and associated password are stored encrypted in the Registry. The server will use the specified account name and password at startup to access the SilverMaster database. Example:
|
-v |
Generates verbose output as SilverMasterInit runs. |
If the process fails, run this option to identify where the failure occurred. Example:
|
-x |
Displays SilverMaster initialization properties and then exits without starting SilverMasterInit. |
Run to view local server startup properties. This option does not change or refresh properties. Use this debugging option to check for misdirected initialization settings. Example:
|
The SilverMaster database, which is created during installation, can also be recreated or updated using SilverMasterInit. The SilverMaster database keeps track of all deployment databases used by the application server and also holds the application server's system tables, including those containing group, user, and licensing information. There is one SilverMaster catalog for each server or cluster.
For more information about the SilverMaster database, see Configuring deployment databases.
If your SilverMaster database is damaged, you can run SilverMasterInit. If you cannot start the application server, try one of the following procedures if nothing else has worked:
CAUTION: Connection problems may be due to a corrupt driver connection, damaged deployment database, or network problems. If you have questions about what is causing your server problem, call Novell exteNd Technical Support before running SilverMasterInit. Running SilverMasterInit in Full mode will delete the contents of all your existing system tables and replace them with initialized data. Do not run in Full mode if you want to preserve existing system tables, including those that contain deployed J2EE archives, group, and user data.
You can run SilverMasterInit in Refresh mode to upgrade or access SilverMaster properties. The refresh process skips some of the database installation steps used by Full mode. Run SilverMasterInit in Refresh mode to refresh SilverMaster system data and resources without deleting existing users, groups, and deployed J2EE archives.
NOTE: As part of the application server installation process, SilverMasterInit upgrades resources. You typically upgrade the application server by running the installation program.
To run SilverMasterInit in Refresh mode:
From server's \bin directory, enter:
SilverMasterInit -r options
You see the message Creating Resources will take a few minutes; please wait.
When SilverMasterInit completes without errors, restart the application server.
SilverMasterInit can often fix problems caused by someone removing or renaming a file or table that SilverMaster relies on. If you cannot start the application server or connect to the SilverMaster database, you may need to run SilverMasterInit.
While SilverMasterInit can reset corrupted SilverMaster properties, this program cannot repair a corrupted Registry key, configuration files, sample databases, or files associated with databases. To address these types of problems, run the installation program.
To avoid deleting all your database tables, try running SilverMasterInit in Refresh mode (before running it in Full mode) to see if that resolves the server problem.
If you run SilverMasterInit in Full mode to regenerate new SilverMaster properties, you will have to recreate any Silver Security users and groups, re-add any deployment databases, and redeploy any J2EE archives deployed to SilverMaster.
To run SilverMasterInit in Full mode:
From the server's \bin directory, enter:
SilverMasterInit -f options
You see the message Creating Resources will take a few minutes; please wait.
Redeploy any J2EE archives that were deployed to the SilverMaster.
NOTE: If you do not want to lock down access to data and require user authentication, you can run the -n option in Full mode.
You can use SilverMasterInit to regain access to locked resources. The SilverMaster database is where the application server stores all system resources and links to other databases. By default, any user with the Locksmith privilege has Read access permission to the SilverMaster. If all users are accidentally denied access to the SilverMaster database, no one will be able to access the application server through the SMC.
See the following sections if you suspect that Read access to SilverMaster has been restricted:
By default, the administrator and any other user with Locksmith privilege can get and set data access permissions for any resource in any database, read all SilverMaster resources, and grant Locksmith privilege to users and groups. The only user able to grant the Locksmith privilege is someone who is already a Locksmith. If all accounts with the Locksmith privilege get deleted, use the SilverMasterInit Locksmith option to grant this privilege to a user to regain access to resources.
NOTE: By default, after a new installation or after you run SilverMasterInit in Full mode, an administrator account is automatically created that has Locksmith privilege.
Once a user with the Locksmith privilege can access SilverMaster, that user can unlock resources and reset access privileges.
For more information, see Using the Locksmith privilege.
You can set server authentication from either the SMC or the SilverMasterInit command line. You should set server authentication if you accidentally restrict Read access to your SilverMaster database. If users cannot access SilverMaster, run the SilverMasterInit server authentication option to allow users to authenticate themselves when they initially connect to the server. When a user logs in to the application server from the SMC, a request for the login resource is issued. Users cannot access the login dialog if their access to SilverMaster is restricted, because they have no Read access to the database. This is because the SilverMaster has the login resource.
You do not need to specify Full or Refresh mode when you run the server authentication option. When you restart the server after setting server authentication, your first attempt to access the server will bring up the credentials dialog and you can log in.
From the server's \bin directory, enter:
SilverMasterInit -a-U
dbusername
-P
dbpassword
In some obscure situations it is possible to exceed the limits of the stack, in which case the Java Virtual Machine (JVM) throws a java.lang.StackOverflowError.
On a Windows system in the Java environment, there are at least two program stacks (and possibly more, depending on the JVM implementation), any one of which can overflow and cause a StackOverflowError to be thrown:
There is always a hardware stack, which is used by native code in the JVM itself and by native code that is compiled from Java bytecodes by the Just In Time (JIT) compiler.
There is always a Java bytecode stack, which is used for temporary storage of method call arguments and local variables for Java methods. This is a soft stack that is created and managed by the JVM.
Each thread created in the JVM has its own hardware and Java stacks.
It is possible to alter the size of each of the stacks if it is determined that the default stack size is too small. However, the most common cause of stack overflow errors is a programming error where a method is called recursively a number of times. If this is the case, increasing the size of the stack will not fix the stack overflow problem. Before attempting to increase the stack size, verify that the code does not contain any errors of this nature. Assuming the stack overflow is not caused by an infinite recursion error, it should be possible to fix the stack overflow by increasing the stack size. To determine which stack overflowed is largely a matter of trial and error.
The size of the hardware stack is determined by the operating system using a value stored in the header of the executable. The executable (SilverServer.exe) specifies a default stack size of 256K.
In order to change the stack size, you must modify the executable header using Microsoft's EDITBIN utility. For example, to change the default stack size for SilverServer.exe to 512K, use the following command line:
EDITBIN /STACK:0x80000 SilverServer.exe
TIP: Of course, you should make a backup before modifying any executable.
If increasing the size of the hardware stack doesn't work, it is possible that the Java stack is the problem. In the JDK documentation, there are two command-line options affecting the stack size:
The defaults were 128K and 400K respectively. Although these options are no longer documented in JDK 1.2 (Java 2), they appear to have been carried forward as nonstandard (-X) switches. To set these options for an application server executable, use +X instead of -X (the application server executables interpret + options as options to be passed to the JVM).
For more information about application server startup options, see Using startup options.
For example, to set both the native and Java stacks for the application server to a maximum of 512K, use the following command line:
SilverServer +Xss512k +Xoss512k
NOTE: Increasing any of the default stack size values will increase the amount of virtual memory allocated per thread. Virtual memory is a finite resource, albeit a large one (in a 32-bit operating system such as Windows NT, processes can address up to 2G of virtual memory). Increasing the per-thread virtual memory requirement will reduce the number of threads that can be created. It is important to realize that this could reduce the number of simultaneously connected users that the server is able to support (since the server uses one thread per connected client).
This section describes some uncategorized issues you may need to address.
If the application server seems to be hung or in a loop, you can generate a listing for each thread with a stack trace. Doing this does not stop the server.
In NetWare Make sure the Logger buffer size is set large enough to handle the thread stack information. (You'll want to set the Recall and Layout to large numbers.)
Determine the JVM ID of the application server process:
java -show
To generate the stack trace:
java -showstacksXXXX
where XXXX is the JVM ID of the application server process.
To see the stack trace, you can:
Use the GUI version of the Console Log (where you can save to a file)
Go to http://hostname:8008 on a browser and use the Remote Manager to view the logs
In UNIX Determine the process that the application server is running under:
ps -all | grep Silver
Issue the following command:
kill -3 SilverServer_process_ID
The application server lists each thread with a stack trace in the window where the server was started from.
In Windows In the window where you started the server, press Ctrl+Break. The application server lists each thread with a stack trace.
You may receive a Socket Exception message in your NT application log. Typically, this is not a problem: it usually indicates that a client has unilaterally closed a socket. Browsers such as Internet Explorer frequently do this when the connection has been idle for a while, and it will show up as a Socket Exception in the server's console when running with debugging.
You can usually ignore such warnings; they simply reflect a normal situation.
Copyright © 2004 Novell, Inc. All rights reserved. Copyright © 1997, 1998, 1999, 2000, 2001, 2002, 2003 SilverStream Software, LLC. All rights reserved. more ...