Copyright 2008 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.

You can always find the latest version of Sun-GDD-N1SPS at:
http://kenai.com/projects/sun-gdd-n1sps

Please see the file LICENSE for more information about the licensing terms.

About Sun-GDD-N1SPS.sh
======================

Sun-GDD-N1SPS tool helps you to automatically gather debugging data needed
to troubleshoot a problem that you can reliably reproduce. For a quick start
you typically need to run:

./Sun-GDD-N1SPS.sh auto-collect

and then follow the oncreen instructions.
The full usage instructions are available by running:

./Sun-GDD-N1SPS.sh -h

If you understand what Sun-GDD-N1SPS is doing you can as well use only:

./Sun-GDD-N1SPS.sh collect


Reporting Bugs a/o Feature Enhancements
=======================================

If you find a bug, or you want to request some nice feature, please don't
hesistate to go to http://kenai.com/projects/sun-gdd-n1sps project site and get
in contact with us.

Please consult the Known Issues secion below to see whether your issue has
already been reported.


Example usage
=============

For a quick start, you just unpack Sun-GDD-N1SPS on your system and start it as
a user who is owning the SPS installation. Please make sure that your
startup/shutdown of SPS MS works without any user interaction. More information
below in Known Issues paragraph.

    $ ./Sun-GDD-N1SPS.sh auto-collect
    Sun-GDD-N1SPS Version 0.6-$Revision$
    Copyright 2008 Sun Microsystems, Inc. All rights reserved.
    Use is subject to license terms.
    
    2008-11-05-14-53 INFO: Sun-GDD-N1SPS Version 0.6-$Revision$
    2008-11-05-14-53 INFO: Logging to /var/tmp/11111111-gdd-n1sps-vn1-240a-gmp03-2008-11-05-14-53/sun-gdd-n1sps.log
    2008-11-05-14-53 INFO: Working directory: /home/mm201025/repos/sun-gdd-n1sps~mercurial
    2008-11-05-14-53 INFO: User: uid=202025(mm201025) gid=10(staff)
    2008-11-05-14-53 INFO: Started as: ./Sun-GDD-N1SPS.sh auto-collect
    
    2008-11-05-14-53 INFO: Found SPS MS: /opt/SUNWn1sps/N1_Service_Provisioning_System_5.2
    2008-11-05-14-53 INFO: Found SPS MS: /opt/SUNWn1sps/N1_Service_Provisioning_System_6.0
    
    2008-11-05-14-53 ERROR: Multiple SPS Master Servers found on this system.
    2008-11-05-14-53 ERROR: Use the -d <sps_installation_directory> option to pickup the right one.
    2008-11-05-14-53 ERROR: Please see ./Sun-GDD-N1SPS.sh -h for more information.


From the command output, we can see that Sun-GDD-N1SPS.sh detected that I have
multiple SPS installations available on my system. I have to explicitely choose
which one we are going to debug. I will use the -d switch to specify the SPS
home directory. I will also use the -f switch to enable automatic restart of
SPS MS when needed. Moreover, I will use the -c 12345678 switch to specify the
case number for which we are capturing the data.

    $ ./Sun-GDD-N1SPS.sh -c 12345678 -f -d /opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/ auto-collect
    Sun-GDD-N1SPS Version 0.6-$Revision$
    Copyright 2008 Sun Microsystems, Inc. All rights reserved.
    Use is subject to license terms.
    
    2008-11-05-14-56 INFO: Sun-GDD-N1SPS Version 0.6-$Revision$
    2008-11-05-14-56 INFO: Logging to /var/tmp/12345678-gdd-n1sps-vn1-240a-gmp03-2008-11-05-14-56/sun-gdd-n1sps.log
    2008-11-05-14-56 INFO: Working directory: /home/mm201025/repos/sun-gdd-n1sps~mercurial
    2008-11-05-14-56 INFO: User: uid=202025(mm201025) gid=10(staff)
    2008-11-05-14-56 INFO: Started as: ./Sun-GDD-N1SPS.sh -c 12345678 -f -d /opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/ auto-collect
    
    2008-11-05-14-56 INFO: Collecting debugging data for SPS MS:
    2008-11-05-14-56 INFO: basedir: /opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/
    2008-11-05-14-56 INFO: start: /opt/SUNWn1sps/N1_Service_Provisioning_System_5.2//server/bin/cr_server start
    2008-11-05-14-56 INFO: stop: /opt/SUNWn1sps/N1_Service_Provisioning_System_5.2//server/bin/cr_server stop
    2008-11-05-14-56 INFO: ms port: 1130
    2008-11-05-14-56 INFO: db port: 5432

As you can see, the Sun-GDD-N1SPS.sh detected where is N1 SPS installed, what
is the method to stop/start the server and at which ports the main Java process
and PostgreSQL database are listening.
    
    2008-11-05-14-56 INFO: Verifying permissions needed to restart SPS MS...
    2008-11-05-14-56 INFO: ...done

At this point we have determined whether we will be able to restart the SPS MS.
The restart is needed to fully enable debugging information that is being
collected.
    
    2008-11-05-14-56 INFO: Stopping SPS MS...
    2008-11-05-14-56 INFO: ...done
    2008-11-05-14-56 INFO: Starting SPS MS (waiting up to 30s)...
    2008-11-05-14-56 INFO: ...done (pid 11267/11266, listening at port 1130/5432)

SPS debugging has been enabled, SPS was restarted and Java and PostgreSQL were
found listening at the displayed ports.
    
    2008-11-05-14-56 INFO: Your SPS MS is now ready for problem reproduction.
    2008-11-05-14-56 INFO: Please reproduce the problem and press <Enter>.

Now you should login to the SPS Web UI or use the SPS CLI to reproduce your
problem. Once you think the problem is reproduced, just press <Enter> and
Sun-GDD-N1SPS.sh will begin collecting necessary data.
    
    2008-11-05-14-57 INFO: Thank You.
    
    2008-11-05-14-57 INFO: Collecting SPS MS debugging data...
    2008-11-05-14-57 INFO: ...getting the system info...
    2008-11-05-14-57 INFO: ...getting 11267 java thread dump...
    2008-11-05-14-57 INFO: ...getting 11267 core dump...
    2008-11-05-14-57 INFO: ...getting 11266 core dump...
    2008-11-05-14-58 INFO: ...collecting server log files and configuration...
    2008-11-05-14-58 INFO: ...collecting server plugins...
    2008-11-05-14-58 INFO: ...done
    
Once we have collected all necessary debugging data, we disable debugging and
restart SPS MS again to revert it back to production usage.

    2008-11-05-14-58 INFO: Compressing the output directory /var/tmp/12345678-gdd-n1sps-vn1-240a-gmp03-2008-11-05-14-56...
    2008-11-05-14-59 INFO: ...done
    2008-11-05-14-59 INFO: Please send the following file to Sun Support
    2008-11-05-14-59 INFO: case number: 12345678, file size: 16M
    2008-11-05-14-59 INFO: /var/tmp/12345678-gdd-n1sps-vn1-240a-gmp03-2008-11-05-14-56.tar.gz

Note that we are compressing the output file here, before we restart the MS
back, because it might happen that the problem you are reporting is actually
hanging SPS, in which case you will need to kill -9 the MS anyway. All the
necessary data have been collected at this point.

    2008-11-05-14-58 INFO: Stopping SPS MS...
    2008-11-05-14-58 INFO: ...done
    2008-11-05-14-58 INFO: Starting SPS MS (waiting up to 30s)...
    2008-11-05-14-58 INFO: ...done (pid 12208/12291, listening at port 1130/5432)
    
After the successfull run, you should be having an output file containing all
necessary debugging information, and you should just attach it to your case
opened with Sun Services.


Known Issues
============

SSL connectivity with LDs/RAs but no password in config.properties?
-------------------------------------------------------------------

Sun-GDD-N1SPS uses 'cr_server start noprompt' when starting the MS. If you are
using SSL but if you haven't put your encoded passowrd into
server/config/config.properties, the startup will fail and you will have to
kill -9 the MS process. Please put the encoded password into config.properties
to enable smooth startup of SPS MS. Please see CR6771570 for more information.


What gets collected
===================

If you take a look at the contents of the output file, you can see what
information about your SPS we are collecting.


Getting started to troubleshoot the problem
===========================================

Once you have the Sun-GDD-N1SPS output file, you can start troubleshooting the
problem. Good problem description from the customer, in <object> <defect> SGR
terminology is always needed. The following steps might help you getting
started quickly. The following files/directories are part of the captured archive:

  -rw-r--r--   1 mm201025 staff     143485 Nov  7 14:23 ls-lR
  drwxr-xr-x   8 mm201025 staff        512 Nov  7 14:22 server
  -rw-r--r--   1 mm201025 staff     232610 Nov  7 14:25 sun-gdd-n1sps.log
  drwxr-xr-x   2 mm201025 staff        512 Nov  7 14:23 system


1) First of all, take a look at the sun-gdd-n1sps.log which captures all
commands invoked during the execution of Sun-GDD-N1SPS.sh. You can verify how
exactly customer ran the command, what were the results of each capture step.
Use this to spot general problems and to verify that script captured what it
was asked to.

2) Take a look at the ls-lR file, which contains detailed directory listing of
the SPS base directory. You can get a quick overview of what files are
installed where, what are their modification times, how many resources the SPS
has, how many plugins and which versions are installed.

3) Examine environmental files in system/ subdirectory, such as uname-a, ps-ef,
showrev-a, that will give you better understanding of where is SPS running.

4) Finally dive into server/ subdirectory which contains configuration files,
data files, log files, and process cores of SPS master server.