ZOS System log – SYSLOG

C O N T E N T S:

Bottom

 1. SYSLOG information 

     1.1. What is SYSLOG?

     1.2. How SYSLOG file is created ?

     1.3. How to process SYSLOG?

     1.4. Checking syslog  and console buffers   status

2. SYSLOG prossessing of production system 

     2.1. Job SYSLGST    (daily) –  start daily SYSLOG processing

     2.2. Job  SYSWRT – External writer (daily – invoked by SYSLGST) – copy       SYSLOG to daily SLGSAV.LWTR

     2.3. Job SYSLGSVD (daily – invoked by SYSLGST) – copy SLGSAV.LWTR

to  weekly SLGSAV.SYSLOGW

     2.4. Job SYSLGPR (daily – invoked by SYSLGSVD) –  print & empty

SLGSAV.LWTR

     2.5. Job SYSLGSVW (weekly) – copies & empties weekly SLGSAV.SYSLOGW (+1)

to backup tape

3.  Extra SYSLOG Utilities

     3.1. GDG data sets creation on  Syslog weekly tapes

     3.2.  Restore weekly generation file  BACKUP.SYSLOG.G0185V00 with job  RESAVW1 from backup  tape to SLGSAV.TEST  file for analysis

     3.3.  Search syslog data  in weekly SLGSAV.TEST  file restored  for analysis

     3.4. Converting current z/OS binary logs for problem determination consumption

on PC

     3.5. Display all non-processed SYSLOG output groups on spool

     3.6. CICS&VTAM  SYSOUT data sets saved on SYSLOG

     3.7. Display maximum number of buffers,  threshold percent for warning messages        and  free buffers

 

1.  SYSLOG information

          1.1.  What is SYSLOG?

 The system log (SYSLOG) is a data set  residing in the primary job entry subsystem’s spool space. This is SPOOL data, not a data set that can be accessed in any normal way and  there is no data set name assigned to a spool file.  It can be used by application and system programmers (through the WTL macro)  to record communications about problem programs and system functions.

When we say “syslog” it  means the following SPOOL data:
– console output
– output of a particular job (and if so, all output,  one particular output)
– system logger output
The SYSLOG panel allows authorized users to display the system log, which is a collection of JES2 data sets that contain console messages, operator commands  and  operator responses for a z/OS JES2 system.  It is accessed   with the LOG command (TSO/ISPF- sd.log).

The operator can use the LOG command  to add (if necessary) an entry to the system log –  using  command  / LOG ‘1111 22′ (as result the line ‘1111 22′  is included in SYSLOG).

 The SYSLOG is part of SYSOUT (SYSOUT data sets are output spool data sets on direct access storage devices – DASD in the MVS system)  and  actually consists of two separate logs: the system log and the hardcopy log. The hardcopy (consoles)  log is a record of all system message traffic, written on each saved current SYSLOGXX file,  specifically:

–        Console Messages – to and from all consoles

–        Commands and replies entered by the operator  on sd.log

The hardcopy log is always written to the SYSLOG and  indicates that the system log is to be the hardcopy medium.

The SYSLOG consists of the following:

–        All messages issued through Write To Log (WTL) macros.

–        All messages entered by the LOG operator commands.

–        Usually, the hardcopy log (console messages).

–       Any messages routed to the SYSLOG from any system component or program. In each job stream  the steps  JESMSGLG  and JESYSMSG  write messages about the job starting,  ending  and communications  with the system   in SYSLOG as follows:

JESMSGLG JES2 output data set contains system messages related to the job starting  and   ending. JESYSMSG JES2 output data set   saves all system communications /information  messages  on SYSLOG.

          1.2. How SYSLOG file is created ?

During  z/os IPL /07.04.2011/  the SYSLOG data set is  created   and some important IPL messages follows:

——————————————– IPL started   ——————————————————–

14:52:30.18 SYSLOG 00000000  IEE042I SYSTEM LOG DATA SET INITIALIZED

(The system is being initialized and the system has just created the SYSLOG data set.)

……………….

14:49:48.06                       00000290 IEA371I SYS1.IPLPARM on device 0644 selected for IPL parameters

14:49:48.06                        00000290 IEA246I LOAD ID 39 selected

14:49:48:06                        00000290 IEA245I NUCLST ID 00 selected

14:49:48:06                         00000290 IEA519I IODF  dsn= SYS1.IODF39

14:49:49.81                          00000290  IEA101A SPECIFY SYSTEM PARAMETERS FOR z/OS 01.04.00 HBB7707

14:49:49.81                          00000290  IEE600I REPLY TO 00 IS;CLPA

14:50:03.24                         00000290  IEE252I MEMBER IEASYS00 FOUND IN SYS1.PARMLIB

14:51:03.28 INTERNAL 00000290  S JES2 – /start JES2 from  bc2a.*.stcproc /

14:51:03.54                        00000098  DFH0100 CICS subsystem is now initialized

14:51:10.31                         00000290  IEF196I  1 //JES2     JOB MSGLEVEL=1

14:51:10.32                        00000290  IEF196I   2 //STARTING EXEC JES2     (from BC2A.ZOS140.STCPROC)

14:51:10.38                       00000290   IEF196I   3 XXJES2 proc member=jes2iprm, almem=jes2parm, cmds=jes2cmds

14:51:11.67                       00000090  IEF403I JES2 – STARTED – TIME=14.51.11

14:52:30.17          00000290  IEF196I IEF237I JES2 ALLOCATED TO SYSLOG00

14:52:30.18          00000090  IEE041I THE SYSTEM LOG IS NOW ACTIVE 

………………..

———————————————- End of IPL —————————————————–

The first SYSLOG00 is created and allocated to JES2  at 14:52:30.17 and SYSTEM LOG IS   ACTIVE  at 14:52:30.18.

In sd.st at present  the system  task SYSLOG is running  with  initialized SYSLOG09 by BPXOINIT as Data set number 110:

NP   JOBNAME  JobID         Owner      Prty   Queue      C  Pos  SAff    ASys Status

SYSLOG     STC01072 +MASTER +   15 EXECUTION      BC2A  BC2A

with the same contents as of the sd.log panel of TSO/ISPF:

After   – ? SYSLOG we see the file SYSLOG09:

SDSF JOB DATA SET DISPLAY – JOB SYSLOG   (STC01072)    LINE 1-1 (1)

NP   DDNAME   StepName ProcStep DSID Owner        C Dest               Rec-Cnt Page

        SYSLOG09 BPXOINIT                  110 +MASTER+ L                      9,855      

created by job SYSLOGD.

Steps to take: To view SYSLOG data sets that contain activity for  z/OS ,  with sd.st  of  ISPF  locate the SYSLOG  and type S in the NP column next to this SYSLOG to select and browse it. The same may be done with sd.log  of ISPF.

          1.3.  How to process SYSLOG?

SYSLOG periodically should  be printed and checked for problems.

You might  want to review the SYSLOG at a later date when you are doing problem diagnosis. Instead of having the SYSLOG job data written to the JES2 offload output queue after a certain number of WTLs, you should set up a procedure to organize the SYSLOG data set depending on your installation’s requirement.

You can perform your housekeeping as follows:

–        Set up a procedure to have the system operator issue the WRITELOG command to close

and allocate a new SYSLOGxx at the end of each processing day. For example, WRITELOG L.

–        At the end of the week, you will have seven SYSLOGxx data sets in the spool. You can now decide to offload the SYSLOG to an MVS data set or route it to a printer for a hardcopy printout.

–        It’s really not necessary to keep SYSLOG data sets for more than a week, so if you keep a  week’s worth of SYSLOG data sets, you should not have a problem.

The maximum number of WTLs (messages) allowed for the SYSLOG are defined at IPL time in SYS1.PARMLIB(IEASYS00):

LOGCLS=L,                     WILL NOT BE PRINTED BY DEFAULT

LOGLMT=999999,                MAX WTL MESSAGES QUEUED, MUST BE 6 DIGITS

Usually LOGLMT is not reached and WRITELOG command  is issued (by operator or special job)   to close and free the current SYSLOG. The system  releases the closed SYSLOG to the printer queue (scheduled for SYSOUT processing on output class L) as specified by the LOGCLS (L)  parameter of the SYS1.PARMLIB(IEASYS00). Then the system  allocates and opens a new – SYSLOGXX.

Writelog (W) – initialize  SYSLOG00, 01, 02 .. begin new syslog from 1,2 .. lines/recs:

IEF196I IEF237I JES2 ALLOCATED TO SYSLOG01

IEE042I SYSTEM LOG DATA SET INITIALIZED

IEF196I IEF285I   +MASTER+.SYSLOG.STC03395.D0000101.?          SYSOUT

IEE043I A SYSTEM LOG DATA SET HAS BEEN QUEUED TO SYSOUT CLASS L

Explanation:

The system closed the system log (SYSLOG) data set and queued it  to a SYSOUT class L. The SYSLOG data set might not be immediately available  until JES processes it. The system queued the data set to a SYSOUT class L for one of these reasons:

–       (1) The SYSLOG data set is full. The maximum number of SYSLOG records, as specified in the limit parameter(LOGLMT),  has been reached. The system issues an internal WRITELOG command to close the current SYSLOG data set.

–       (2) The operator entered a WRITELOG command to close the SYSLOG data set.

–       (3) The operator entered a WRITELOG CLOSE or HALT command after the SYSLOG data set was opened in response to message  IEE037D LOG NOT ACTIVE  preceded by IEE533E SYSTEM LOG INITIALIZATION HAS FAILED.

The system does the following:

–       For case (1) or (2), the system closes the currently recording SYSLOG data set and obtains a new data set (SYSLOGXX).

–       For case (3), the system closes the currently recording SYSLOG data set and makes the SYSLOG unavailable.

You may organize automatically issuing of WRITELOG command as follows:

$ta wlog,t=23.58,i=86400,’$vs,''W L''’

Automatically spin SYSLOG to sysout L once per day:

–       $ta  ALL – JES2 displays all automatic command entries you are authorized

–       $ta – Set command to specify automatic command named wlog

–        t=23.58 –  Start time of the automatic command  (just before midnight today)

          1.4. Checking syslog  and console buffers status

               1.4.1. SYSLOG buffers and messages status

You can display the LOGLIM value and the number of buffers in use, by issuing the D C,HC command in TSO sd.log:

/D C,HC – and the result follows:  no log buffers in use (limit 1000), WTO=1 (limit 3000), WTOR=1 (limit 100):

IEE889I 16.37.38 CONSOLE DISPLAY 277

MSG: CURR=1    LIM=3000 RPLY:CURR=1    LIM=100  SYS=BC1B      PFK=NONE

CONSOLE/ALT        ID  ————— SPECIFICATIONS —————

SYSLOG                 COND=H      AUTH=CMDS         NBUF=0    UD=N

ROUTCDE=ALL

LOG BUFFERS IN USE:        0  LOG BUFFER LIMIT:     1000     

You can  verify the LOGLIM value (1000) if enter from sd.log:

/K M                                                                     

IEE144I K M,AMRF=Y,MLIM=3000,RLIM=0100,UEXIT=N,LOGLIM=001000,ROUTTIME=00

0,RMAX=9999,APPLID=SMCS2A

If this limit is not desired it can be increased temporary  for  this IPL to 2000  with the CONTROL M,LOGLIM command (/k m,loglim=2000).

If by any reason SYSLOG buffers are full the following messages may appear:

 IEE012A NO LONGER SAVING MESSAGES FOR HARDCOPY, LOGLIM REACHED

 Operator Response: Either activate a hardcopy medium, or increase the value of LOGLIM with the CONTROL M,LOGLIM command.

IEE767A SEVERE BUFFER SHORTAGE FOR SYSTEM LOG – 100% FULL

Explanation: The system found that the system log task (SYSLOG) may be active and that 100% of its buffers are full.

System Action: Write-to-log (WTL) processing ceases to allocate WTL queue elements. Subsequent WTL requests result in a return code of 4 to the user.

Operator Response: Increase the number of WTL buffers available to the system by entering the CONTROL M,LOGLIM=xxxxxx command.

System Programmer Response: Before the next system initialization, increase the value of LOGLIM in the CONSOLxx member of parmlib. This will allow for a higher number of outstanding WTL buffers.

If a VARY SYSLOG,HARDCPY,OFF,UNCOND command has previously been issued because   the limit of LOGLIM has been reached  or the buffer storage has been exhausted, then  z/os   has saved SYSLOG  messages  in SYSLOG buffer to be hardcopied later.

Example:

/VARY SYSLOG,HARDCPY,OFF,UNCOND – this command temporary stops writing on syslog as limits are reached

System action: Messages issued after this point will not be sent to SYSLOG, unless SYSLOG is     reactivated. After some time of work the number of syslog buffers are increasing (less than 1000) and the contents of buffers  with the saved SYSLOG messages after the last command in SYSLOG  is as follows:

/D C,HC                – command issued     when SYSLOG is stoped and  24 records/messages are issued from the system:

IEE889I 17.04.50 CONSOLE DISPLAY 287

MSG: CURR=1    LIM=3000 RPLY:CURR=1    LIM=100  SYS=BC1B      PFK=NONE

HARDCOPY SUSPENDED ON THIS SYSTEM

 LOG BUFFERS IN USE:       24  LOG BUFFER LIMIT:     1000

/D C,HC                 - command issued     when SYSLOG is stoped and  34 records/messages are issued from the system:

IEE889I 17.19.48 CONSOLE DISPLAY 291

MSG: CURR=1    LIM=3000 RPLY:CURR=1    LIM=100  SYS=BC1B      PFK=NONE

HARDCOPY SUSPENDED ON THIS SYSTEM

LOG BUFFERS IN USE:       34  LOG BUFFER LIMIT:     1000

/D C,HC        - command issued     when SYSLOG is stoped and  59 records/messages are issued from the system:

RESPONSE=BC1B

IEE889I 18.07.23 CONSOLE DISPLAY 310

MSG: CURR=1    LIM=3000 RPLY:CURR=1    LIM=100  SYS=BC1B      PFK=NONE

HARDCOPY SUSPENDED ON THIS SYSTEM

LOG BUFFERS IN USE:       59  LOG BUFFER LIMIT:     1000

To reactivate SYSLOG (the system log) and SYSLOG  to become  again the hardcopy medium,  the following command is  entered on TSO sd.log screen:

/V SYSLOG,HARDCPY – SYSLOG data set  is assigned again as Hardcopy medium for system messages. After this command all 59 log buffers/messages  following  the last log command VARY SYSLOG,HARDCPY,OFF,UNCOND are restored/added  in SYSLOG (before that should be changed LOGLIM if needed).  If SYSLOG is reactivated the LOGLIM will be defaulted to the value of 1000.

The following messages are displayed proving that the saved buffers are restored and no log buffers are longer  in use:

 /D C,HC,L=IBMUSER          – this command is issued from TSO  user= IBMUSER

IEE889I 18.29.18 CONSOLE DISPLAY 313

MSG: CURR=1    LIM=3000 RPLY:CURR=1    LIM=100  SYS=BC1B      PFK=NONE

CONSOLE/ALT        ID  ————— SPECIFICATIONS —————

SYSLOG                 COND=H      AUTH=CMDS         NBUF=0    UD=N

ROUTCDE=ALL                                    

LOG BUFFERS IN USE:        0  LOG BUFFER LIMIT:     1000

After restoring the saved 59 buffers in SYSLOG the following  last  messages  are proving the normal processing of SYSLOG:

/D SMF                                 – this command is  issued   after activating SYSLOG and the following information can be found in sd.log:

IEE974I 18.29.33 SMF DATA SETS 315

NAME                VOLSER SIZE(BLKS) %FULL  STATUS

P-BC1B.MAN1           BC1MC3      9000    52  ACTIVE

S-BC1B.MAN2           BC1MC3      4500     0  ALTERNATE

S-BC1B.MAN3           BC1MC3      4500     0  ALTERNATE

/D C,HC           – this command is  issued   after activating SYSLOG and the following information can be found also  in sd.log:

IEE889I 16.13.01 CONSOLE DISPLAY 337

MSG: CURR=1    LIM=3000 RPLY:CURR=1    LIM=100  SYS=BC1B      PFK=NONE

CONSOLE/ALT        ID  ————— SPECIFICATIONS —————

SYSLOG                 COND=H      AUTH=CMDS         NBUF=0    UD=N

ROUTCDE=ALL

LOG BUFFERS IN USE:        0  LOG BUFFER LIMIT:     1000     

To display all SYSLOG output groups/files  with not processed by the external writer (old or present)  on spool enter:

/$DOS’SYSLOG’,CRTIME,AGE,REC,ODISP,Q

 

If some old SYSLOGXX files with are waiting 6 days (from creating date till now)  to be processed by the external writer, the following information is received for date=2011.125

$HASP686 OUTPUT(SYSLOG) 799

$HASP686 OUTPUT(SYSLOG)    CRTIME=(2011.125,12:57:12),DAYS=6,    – at 05.05.2011 created

$HASP686                   RECORDS=(164 OF 164),ODISP=WRITE,

$HASP686                   QUEUE=L

$HASP686 OUTPUT(SYSLOG) 800

$HASP686 OUTPUT(SYSLOG)    CRTIME=(2011.125,12:34:23),DAYS=6, – at 05.05.2011 created

$HASP686                   RECORDS=(23358 OF 23358),ODISP=WRITE,

$HASP686                   QUEUE=L

If no SYSLOGXX files with  are   waiting to be processed by the external writer, the following information is received:

$HASP003 RC=(52), 841                                     

$HASP003 RC=(52),S’SYSLOG’  – NO SELECTABLE ENTRIES FOUND 

$HASP003         MATCHING SPECIFICATION                          

 

               1.4.2. WTO & WTOR console buffers and messages status

To verify console buffers status enter:

/D CONSOLES   – curr=4 buffers used  for console messages – WTO(limit 3000) and curr=1 buffer for system messages which are awaiting a response from the system operator WTOR (limit 100):

IEE889I 17.31.14 CONSOLE DISPLAY 021

MSG: CURR=4    LIM=3000 RPLY:CURR=1    LIM=100  SYS=BC2A      PFK=NONE

or

/D C,HC – no log buffers in use (limit 1000), Messages – WTO=1 (limit 3000), Replies –  WTOR=1 (limit 100):

IEE889I 16.37.38 CONSOLE DISPLAY 277

MSG: CURR=1    LIM=3000 RPLY:CURR=1    LIM=100  SYS=BC1B      PFK=NONE

CONSOLE/ALT        ID  ————— SPECIFICATIONS —————

SYSLOG                 COND=H      AUTH=CMDS         NBUF=0    UD=N

ROUTCDE=ALL

LOG BUFFERS IN USE:        0  LOG BUFFER LIMIT:     1000

To verify the present WTO/WTOR limits –  MLIM/WTO(3000)  and RLIM/WTOR(100) values you can enter also:

/K M                                                                     

IEE144I K M,AMRF=Y,MLIM=3000,RLIM=0100,UEXIT=N,LOGLIM=001000,ROUTTIME=00

0,RMAX=9999,APPLID=SMCS2A

If these  limits are not desired they can be increased temporary  for  this IPL to 5000 and 200  with the CONTROL M,LOGLIM command (/k m,mlim=5000  and  /k m,rlim=200).

If the console is OK displaying messages and the screen   is not full up, the following message appears left below:

IEE163I MODE= RD – This must be always verified and maintained/keeped  !!!

If the console screen is full up or the limit for WTO or WTOR is reached  and no messages may be written on the console,  the following messages  appear bellow:

On left: IEE163I MODE= HELD  On right: IEE159E MESSAGE WAITING

The same me be done  if by careless mistake is pressed Enter key !!! To clear the screen you have to press ENTER key and  HELD is changed to RD  and message IEE159E MESSAGE WAITING  disappears – the restore action is OK.

The same screen clear may be done by pressing F1 for each line to deleted/moved  it up.

Dealing with System Messages Awaiting Replies (System Requests):

 Enter /d r,l  to display the current list of system messages which are awaiting a response from the system operator (outstanding System Requests), including action messages and WTORs. Action messages can be deleted, and replies to WTORs can be entered as line commands directly on the displayed panel.

Syntax

DR [n1,n2,…] [ALL/MVS/USER] [EVENTUAL]

n1,n2,… List of route codes for which messages are to be displayed.

ALL.Show messages with route codes 1 through 28.

MVS. Show messages with route codes 1 through 12.

USER. Show messages with route codes 13 through 28.

EVENTUAL Also show eventual action messages.

Example:

/D R,L      –    only one message (0002 R *0002)   is waiting reply                                                    

IEE112I 10.16.56 PENDING REQUESTS 210

RM=1    IM=0     CEM=4     EM=13    RU=0    IR=0    AMRF

ID:R/K     T MESSAGE TEXT

0002 R *0002 EMS0990A BC2ANVAS READY FOR COMMANDS.

32 E  IEF234E K 0301,HV0311,PVT,RESTSIK1,DFDSS – RACK=HV0311

31 E  IEF234E K 0300,HV0262,PVT,RESTSIK0,DFDSS – RACK=HV0262

In critical/disaster  cases the following command should be entered on master or   alternate console (if  overflow messages  inform that WTO buffers are 100% or less  full):

K q,r=hc – this command from master or alternate console delete  all WTO console messages in the WTO buffers  and free them.

 

2. SYSLOG prossessing of production system

The next time when SYSLOG00 is full  a  sequence of steps should be  executed    with replies on sd.log. The following  sequence of daily and weekly  jobs are described that write syslog messages on DAILY ( SLGSAV.LWTR)  and    WEEKLY (SLGSAV.SYSLOGW)  ACCUMULATION DATA SETS and when this data set SLGSAV.SYSLOGW is  full,  to  copy it on tape and then empties SLGSAV.SYSLOGW.

          2.1.  Job SYSLGST    (daily) –  start daily SYSLOG processing

This job starts the external writer  ‘SYSWRT  to write class L syslog  to a disk data set  SLGSAV.LWTR.  When SYSWRT  has processed all syslog data, then it  is stopped. After that a job ‘SYSLGSVD’ is started from  job SYSLGST  to write system log data  set  SLGSAV.LWTR to weekly accumulation system log  data set  SLGSAV.SYSLOGW and to  invoke job SYSLGPR  to print syslog (system log data  set  SLGSAV.LWTR) and  to empty   SLGSAV.LWTR.  By job SYSLGSVW  (invoked weekly)  the operator  is asked to mount labeled scratched or work tape where weekly syslog data set  SLGSAV.SYSLOGW is copied  as a generated data set  and then empted SLGSAV.SYSLOGW.

//SYSLGST   JOB   (),

//         ‘OPERATOR’,

//        ,

//         MSGCLASS=X,

//         MSGLEVEL=(1,1),

//         NOTIFY=IBMUSER,

//         TIME=1440

//*

//SYSLOG1 EXEC IPOWTO,REGION=512K  – * ONE OR MORE MESSAGES TO BE SENT TO THE  OPERATOR

//SYSIN    DD  *

—-     THIS JOB STARTS SYSWRT TO WRITE CLASS L SYSLOG TO

—-     SLGSAV.LWTR. WHEN SYSWRT HAS PROCESSED ALL SYSLOG

—-     DATA, THEN REPLY TO EXECUTE ‘STOP L’.

/*

//SYSLGSVD  EXEC JOB,N= SYSLGSVD,REGION=512K- * START A JOBSTREAM FROM A  PDS

/*

//*

//DISPLAY T                DATE AND TIME ENTERED IN SYSLOG

//WRITELOG L               WRITES LOG TO SYSOUT = L

//START SYSWRT.L,,,L       STARTS SYSWRT TO DISK(SLGSAV.LWTR)

//STOP L                   STOPS SYSWRT. DO NOT REPLY UNTIL

//*                         MESSAGE ‘WTR L WAITING FOR WORK

          2.2. Job  SYSWRT – External writer (daily – invoked by SYSLGST) – copy 

SYSLOG to daily SLGSAV.LWTR

This job write class L syslog  to a disk data set  SLGSAV.LWTR and when  SYSWRT  has processed all syslog data, then it  is stopped.

// SYSWRT   JOB MSGLEVEL=1

//L        EXEC SYSWRT

// SYSWRT  PROC

//* DOC: THIS PROCEDURE EXECUTES THE EXTERNAL WRITER WITH AN OUTPUT

//*      CLASS EQUAL TO ‘L’ .

//*

//IEFPROC EXEC PGM=IASXWR00,

//             PARM=’PL’

//IEFRDER  DD  DSN= SLGSAV.LWTR,

//             DISP=(MOD,KEEP),

//             DCB=(BLKSIZE=4088,LRECL=137,BUFL=4088,BUFNO=5,RECFM=VBA)

The following events and replies occur during execution of the previous jobs (2.1, 2.2):

IEFC165I // WRITELOG L 644

*0018 IEFC166D REPLY Y/N TO EXECUTE/SUPPRESS COMMAND

R 18,Y                                                               

IEE600I REPLY TO 0018 IS;Y

WRITELOG L        – SYSLOG00 is to be written on SYSOUT

and External writer  SYSWRT     is started:

IEFC165I // START SYSWRT.L,,,L 436

*0009 IEFC166D REPLY Y/N TO EXECUTE/SUPPRESS COMMAND

IEF196I IEF237I JES2 ALLOCATED TO SYSLOG01

IEE042I SYSTEM LOG DATA SET INITIALIZED      (new SYSLOG01 is initialized in sd.log)

IEF196I IEF285I   +MASTER+.SYSLOG.STC01072.D0000101.?          SYSOUT

IEE043I A SYSTEM LOG DATA SET HAS BEEN QUEUED TO SYSOUT CLASS L

R 9,Y                                                                   

IEE600I REPLY TO 0009 IS;Y

START SYSWRT.L,,,L

IEFC165I // STOP L 444

0010 IEFC166D REPLY Y/N TO EXECUTE/SUPPRESS COMMAND

$HASP100 SYSWRT   ON STCINRDR

IEF695I START SYSWRT   WITH JOBNAME SYSWRT   IS ASSIGNED TO USER STCOPER

, GROUP STCGRP

$HASP373 SYSWRT   STARTED

IEF403I SYSWRT – STARTED – TIME=17.14.54

$HASP250 SYSLOG PURGED — (JOB KEY WAS C0FA25BC)

$HASP250 SYSLOG PURGED — (JOB KEY WAS C6E3EB9F)

IEF176I WTR L        WAITING FOR WORK,, DEST=LOCAL

R 10,Y                                                     

IEE600I REPLY TO 0010 IS;Y

STOP L

IEF301I L        WTR CLOSED

IEF404I SYSWRT – ENDED – TIME=17.16.09

          2.3. Job SYSLGSVD (daily – invoked by SYSLGST) - copy SLGSAV.LWTR

to  weekly SLGSAV.SYSLOGW

This job uses IEBGENER to copy the system log data set SLGSAV.LWTR to weekly syslog data set SLGSAV.SYSLOGW and invokes  SYSPRT to print and clear SLGSAV.LWTR.

//SYSLGSVD  JOB   (9999,CSOPER,10,10),

//         ‘OPERATOR’,

//        ,

//         MSGCLASS=X,

//         MSGLEVEL=(1,1),

//         NOTIFY=,

//         TIME=1440,

//         USER=

//*

//SLGSAV1 EXEC PGM=IEBGENER,REGION=512K

//SYSPRINT DD  SYSOUT=*

//SYSIN    DD  DUMMY

//SYSUT1   DD  DSN=SLGSAV.LWTR,

//             DISP=SHR

//SYSUT2   DD  DSN=SLGSAV.SYSLOGW,

//             DISP=MOD

//*

//SLGSAV3 EXEC JOB,N=SYSLGPR,REGION=512K

/*

          2.4. Job SYSLGPR (daily – invoked by SYSLGSVD) –  print & empty SLGSAV.LWTR

This job is invoked by SYSLGSVD to print  and  empty SLGSAV.LWTR data set.

//SYSLGPR   JOB   (9999,CSOPER,10,10),

//         ‘OPERATOR’,

//        ,

//         MSGCLASS=X,

//         MSGLEVEL=(1,1),

//         NOTIFY=,

//         TIME=1440,

//         USER=

//*                             

//*      STEP1. PRINTS SLGSAV.LWTR

//*      STEP2. EMPTIES SLGSAV.LWTR

//*

//SLGPRT1 EXEC PGM=IEBGENER,REGION=512K

//SYSPRINT DD  SYSOUT=*

//SYSIN    DD  DUMMY

//SYSUT1   DD  DSN= SLGSAV.LWTR,

//             DISP=SHR

//SYSUT2   DD  SYSOUT=*

//*

//SLGPRT2 EXEC PGM=IEBGENER,COND=EVEN,REGION=512K

//SYSPRINT DD  SYSOUT=*

//SYSIN    DD  DUMMY

//SYSUT1   DD  DUMMY,DCB= SLGSAV.LWTR

//SYSUT2   DD  DSN= SLGSAV.LWTR,

//             DISP=OLD

//*

          2.5. Job SYSLGSVW (weekly) – copies & empties weekly SLGSAV.SYSLOGW(+1)  to backup tape 

This job asks the operator to mount a standard labeled scratch tape for using by GDG (Generation Data Group) and copies  weekly data set    SLGSAV.SYSLOGW  as a generation data set.  Then  empties  SLGSAV.SYSLOGW.

//SYSLGSVW  JOB   (9999,CSOPER,10,10),

//         ‘OPERATOR’,

//        ,

//         MSGCLASS=X,

//         MSGLEVEL=(1,1),

//         NOTIFY=,

//         TIME=1440,

//         USER=

//*

//SYSLGW1 EXEC IPOWTO,MSG=’SIGN OUT 1 STD LBL TAPE FOR NEXT STEP’

//SYSIN    DD  *            – following text written to the operator

DSN=BACKUP.SYSLOG(+1)  —   DUMP OF ONE WEEKS SYSLOG

RETPD = 365                                                      

THE INTERNAL AND EXTERNAL TAPE LABELS SHOULD MATCH.

USE THE STANDARD LABEL TAPE NUMBER IN REPLY TO THE TAPE MOUNT REQUEST.

/*

//SYSLGW2 EXEC PGM=IEBGENER,REGION=512K,COND=(0,LT)

//SYSPRINT DD  SYSOUT=*

//SYSIN    DD  DUMMY

//SYSUT1   DD  DSN=SLGSAV.SYSLOGW,

//             DISP=SHR

//SYSUT2   DD  DSN=BACKUP.SYSLOG(+1),

//             DISP=(,CATLG,DELETE),VOL=(,,,25),

//             LABEL=RETPD=365,UNIT=(300,,DEFER),

//             DCB=(SLGSAV.SYSLOGW)

//*

//SYSLGW3 EXEC PGM=IEBGENER,REGION=512K,COND=(0,LT,SYSLGW2)

//SYSPRINT DD  SYSOUT=*

//SYSIN    DD  DUMMY

//SYSUT1   DD  DSN=NULLFILE,DCB=SLGSAV.SYSLOGW

//SYSUT2   DD  DSN=SLGSAV.SYSLOGW,

//             DISP=OLD

//*

RETPD=365 – specify a retention period of 365 days  for the data set (may be used 52 generations for one year).

VOL=(,,,25), – using scratch tape

If GDG named BACKUP.SYSLOG  consists  of 10 generations,  the first 10 generations will require 10  new  scratch tapes and the respective  DSN names of the generation files  BACKUP.SYSLOG.G0001V00 – BACKUP.SYSLOG.G0010V00 will  be written on  each generation tape  in menu ‘Volume Display’  of REMOVABLE MEDIA MANAGER (DFSMSrmm)   when a scratch tape is required and processed. When the  11st generation file BACKUP.SYSLOG.G0011V00 requires a tape we should mount a new scratch  tape  and the first generation tape with  dsn = BACKUP.SYSLOG.G0001V00 will be freed to scratch spool (this is done when the job/program for verification and expiration tapes  is executed – //HSKP     EXEC PGM=EDGHSKP, PARM= ‘VRSEL,EXPROC’). Then the first generation tape with  dsn = BACKUP.SYSLOG.G0001V00 can be released (TSO/ISPF – r – 5 – 1- 4 – release a volume) and after execution the   job for expiration tapes it  becomes ‘scratch’  and is returned to the scratch spool. On the scratch tape  the generation file will be written as “master” nevertheless of parameters:  “Retention date . . : 2011/195”  which was  expired  and nothing in     “Expiration date  . . . . . “. After writing the  generation file on this scratch tape, nothing will be on “ Retention date . . :” but “ Expiration date  . . . . . : 2012/196” will show one year period for expiration time starting from the current day. When you whish to use “VRS       – Vital record specifications”, then  retention date will be changed  to “Retention date . . : PERMANENT”  without changes on “Expiration date  . . . . . : 2012/196”. If you want the last generation file to be on a separate tape then you can use  in the previous job VOL=SER=CB0961  instead of VOL=(,,,25).  When  the separated tape is ‘master’ or ‘user’ –  the generation file will   be written on it  without problem.

3.  Extra SYSLOG Utilities

          3.1. GDG data sets creation on Syslog weekly tapes

Generation Data Groups or GDGs are used to automatically manage datasets that are created periodically, for example log files. GDG name is not necessarily intuitive. The GDG naming standard appends a version segment to the end of the file name, as GxxxxVyy, where xxxx is the generation number from 0000 to 9999 and yy is the cycle number, from 00 to 99. The biggest advantage of using a GDG is that the system keeps track of the generations for you so you don’t have to change your JCL each time you create a new version. So while GDGs cannot be considered a panacea for cyclic datasets, they are very widely used and very useful.

//DEFGDG   JOB   (9999,IPO,10,10),

//         ‘OPERATOR’,

//        ,

//         MSGCLASS=X,

//         MSGLEVEL=(1,1),

//         TIME=1440

//*

//* DOC: THIS IS JOB CREATES A GDG FOR ARCHIVING

//*     10  SYSLOG FILES – FOR 10 generations

//*

//BR14    EXEC PGM=IDCAMS

//SYSPRINT DD  SYSOUT=*

//SYSIN    DD  *

DEF GDG (NAME(BACKUP.SYSLOG) –

LIM(10) NEMP SCR)

/*

LIM(10) – means that this GDG will consist of 10 generations

NEMP -  means that when the 10 generation limit is reached, the oldest will be deleted, but all the others will be retained.

SCR – means uncatalog and delete files when they exceed the generation limit.

Before you can create the files that attach to a GDG base, you need to create a model dataset that contains the DCB information for the files. This model data just needs to be a VTOC entry, it does not need to actually use up disk space.

Many sites use a single model dataset, and then override the DCB information to build the correct type of file. To create a model DCB, use a dummy IEFBR14 job like this

//MODLDSCB JOB   (9999,IPO,10,10),

//         ‘OPERATOR’,

//        ,

//         MSGCLASS=X,

//         MSGLEVEL=(1,1),

//         TIME=1440

//* DOC: THIS IS JOB CREATES A MODEL DSCB FOR USE

//*        TO ARCHIVE THE SYSLOG FILE TO A GDG.

//BR14    EXEC PGM=IEFBR14

//MODEL    DD  DSN= BACKUP.SYSLOG.MODEL,

//             DISP=(NEW,CATLG,DELETE),

//             DCB= (LRECL=137,RECFM=VBA,BLKSIZE=4088),   or  DCB= SLGSAV.SYSLOGW with the same parameters

//             SPACE=(0,(0,0)),

//             VOL=SER=XXXXXX,

//             UNIT=3390

//*

For our case it is DCB= SLGSAV.SYSLOG (DCB of existing file) .

The SPACE=(0,(0,0)) parameter means that this file will just be created as a VTOC entry, it will not use up disk space.

OK,   now   we have a GDG base and a GDG model, we can allocate a GDG file. All you do is specify the new GDG file with a +1 generation number in your JCL like this in job SYSLGSVW:

// SYSLGW1 EXEC PGM=IEBGENER,REGION=512K,COND=(0,LT)

//SYSUT2   DD  DSN=BACKUP.SYSLOG(+1),

//             DISP=(,CATLG,DELETE),VOL=(,,,25),

//             LABEL=RETPD=365,UNIT=(300,,DEFER),

//             DCB=(SLGSAV.SYSLOGW)

 

This source code will create a file called BACKUP.SYSLOG.G0001V00. When you add the 11st file it will be created as BACKUP.SYSLOG.G0011V00 and the BACKUP.SYSLOG.G0001V00 will be deleted, assuming you have SCRATCH coded in the base definition –  VOL=(,,,25).

If you want to refer to a particular GDG after it has been created, you can either use the full GDG name, or a relative number. For example if the current generation of BACKUP.SYSLOG is 04, then you can refer to the previous version either as BACKUP.SYSLOG.G0003V00 or as BACKUP.SYSLOG (-1).

          3.2.  Restore weekly generation file  BACKUP.SYSLOG.G0185V00 with job  RESAVW1 from backup  tape to SLGSAV.TEST  file for analysis

 When restoring a disk file with IEBGENER, you must know the amount of space required for the file and code the SPACE parameter on the SYSUT2 dd statement. The received file SLGSAV.TEST is with DCB=SLGSAV.SYSLOGW of the backed up weekly syslog file.

The TSO/ISPF p.3.4 function for GDG=BACKUP.SYSLOG  display  the name of the  following  GDG cataloged file: BACKUP.SYSLOG.G0185V00 on   tape CB0131:

p.3.4 – BACKUP.SYSLOG  

DSLIST – Data Sets Matching BACKUP.SYSLOG:

——————————————————————–         Volume

BACKUP.SYSLOG.G0184V00                                         CB0125

BACKUP.SYSLOG.G0185V00                                         CB0131

Using job  RESAVW1 we can receive the generation file BACKUP.SYSLOG.G0185V00 on disk  as  file  SLGSAV.TEST  for analysis of the respective weekly/daily  SYSLOG we look for:

//RESAVW1  JOB   (9999,CSOPER,10,10),

//         ‘OPERATOR’,

//        ,

//         MSGCLASS=X,

//         MSGLEVEL=(1,1),

//         NOTIFY=,

//         TIME=1440,

//         USER=

//*

//SYSLGW2 EXEC PGM=IEBGENER,REGION=512K

//SYSPRINT DD  SYSOUT=*

//SYSIN    DD  DUMMY

//SYSUT1   DD  DSN=BACKUP.SYSLOG.G0185V00,

//             DISP=OLD

//SYSUT2   DD  DSN=SLGSAV.TEST,

//             DISP=(,CATLG,DELETE),

//             SPACE=(TRK,1500),UNIT=DISK,

//             DCB=(SLGSAV.SYSLOGW)

          3.3.  Search syslog data  in weekly SLGSAV.TEST  file restored  for analysis

  If you need to find syslog data in SLGSAV.TEST for analysis for exact day of  year and exact  hour, do the following with  TSO/ISPF p.3.4:

–       First: F  ‘ALLOCATED TO SYSLOG’ – to find the respective SYSLOGXX file where the log data   are saved (SYSLOG00 is the first file after IPL –here  SYSLOG06 is allocated at 11:32:17.04 AM):

11:32:17.04          00000290  IEF196I IEF237I JES2 ALLOCATED TO SYSLOG06

–       Second: F   ICH70001I – to find  the respective job Owner  LAST ACCESS – on date: JUNE 1, 2011

11:33:04.36 JOB04920 00000090  ICH70001I BCA1MGP  LAST ACCESS AT 11:20:36 ON   WEDNESDAY, JUNE 1, 2011

–       Second: F   ICH70001I (PF 5 – next job Owner)

11:35:22.61 JOB04925 00000090  ICH70001I BCO1DSL  LAST ACCESS AT 11:16:47 ON WEDNESDAY, JUNE 1, 2011

–       Second: F   ICH70001I (PF 5 – next job Owner)

13:26:35.38 JOB04930 00000090  ICH70001I BC2AC10A LAST ACCESS AT 09:49:54 ON WEDNESDAY, JUNE 1, 2011

      - Second: F   ICH70001I (PF 5 – next job Owner)

13:26:40.44 JOB04938 00000090  ICH70001I BC2AC10A LAST ACCESS AT 13:26:35 ON  WEDNESDAY, JUNE 1, 2011

          3.4. Converting current z/OS binary logs for problem determination consumption on PC

To receive the current syslog data in character format easy on sequential file (Organization=PS)  and  transfer it  to PC,  you need to  perform the following  step-by-step actions:

–       TSO/ISPF  – sd.st

–       xdc SYSLOG  – Fill up the following fields: Data set name  ===> ‘bca1mgp.syslog.sdst’,  Disposition    ===> NEW  – enter  – the next file is ready for investigation in character format or to transfer by  FTP to PC:

–        TSO/ISPF – p.3.4 – BCA1MGP.SYSLOG.SDST/ ready for FTP – 35701 recs in character format and :

  • Organization  . . . : PS
  • Record format . . . : VBA
  • Record length . . . : 240
  • Block size  . . . . : 3120
  • 1st extent cylinders: 6

          3.5. Display all non-processed SYSLOG output groups on spool

TSO/ISPF – sd.logenter:  /$DOS’SYSLOG’,CRTIME,AGE,REC,ODISP,Q

 19.05.2011/$DOS’SYSLOG’,CRTIME,AGE,REC,ODISP,Q – Display all SYSLOG output groups on spool  – BC2A /90 –created on 05.05.2011 + 13 days passed = 18.05.2011 –current 2011.139=19.05.2011

 $HASP686 OUTPUT(SYSLOG) 988                                     

 $HASP686 OUTPUT(SYSLOG)    CRTIME=(2011.125,12:57:12),DAYS=13,  

 $HASP686                   RECORDS=(164 OF 164),ODISP=WRITE,    

 $HASP686                   QUEUE=L                              

 $HASP686 OUTPUT(SYSLOG) 989                                     

 $HASP686 OUTPUT(SYSLOG)    CRTIME=(2011.125,12:34:23),DAYS=13,  

 $HASP686                   RECORDS=(23358 OF 23358),ODISP=WRITE,

 $HASP686                   QUEUE=L                 

            If the syslog is processed and no output groups  exist on spool the following messages are received:

$HASP003 RC=(52),S’SYSLOG’  – NO SELECTABLE ENTRIES FOUND

$HASP003         MATCHING SPECIFICATION

          3.6. CICS&VTAM  SYSOUT data sets saved on SYSLOG

For JES2 data of long running jobs (as CICS job BC2AC10A) there is a danger of losing log information if CICS failed.  One way of saving JES2 CICS data is to stop or not CICS and offload all of the following JES2 CICS data  on tape:

This example shows the SYSOUT data sets that might be associated with a CICS address space:

NP     DDNAME        StepName ProcStep DSID Owner

  •    JESJCLIN                                          1        CICSTS
  •    JESMSGLG    JES2                          2        CICSTS
  •    JESJCL           JES2                          3        CICSTS
  •    JESYSMSG    JES2                          4        CICSTS
  •    $INTTEXT       JES2                          5        CICSTS
  •    CAFF              SCSCPAA1               101     CICSTS
  •    CINT               SCSCPAA1               103     CICSTS
  •    DFHCXRF      SCSCPAA1               104     CICSTS
  •    COUT             SCSCPAA1               105     CICSTS
  •    CEEMSG       SCSCPAA1                106     CICSTS
  •    CEEOUT        SCSCPAA1               107     CICSTS
  •    PLIMSG         SCSCPAA1               108    CICSTS
  •    CRPO             SCSCPAA1              109    CICSTS
  •    MSGUSR       SCSCPAA1               110    CICSTS

The key SYSOUT data sets to review for problem determination data are the JESMSGLG and MSGUSR data sets. The CEEMSG and CEEOUT data sets will contain Language Environment® (LE) problem data typically associated with application problems.

The CICS JESMSGLG SYSOUT data set includes information related to CICS startup and errors related to system problems, not specifically transaction related.

Another way is saving  CICS intermediate log records in SYSLOGXX saved files by WRITELOG command. In JESMSSLG and JESYSMSG  data sets of BC2AC10A (STC01184) CICS job stream  reside  all messages issued from  CICS job starting  (07.04.2011)   up to   current day. The previous SYSLOGXX-1 files   in the weekly saved IPOSAV.SYSLOGW file contain all CICS important  messages issued after  CICS job starting  (07.04.2011).

JES2 CICS production data:

NP   DDNAME       StepName ProcStep DSID Owner      C Dest              Rec-Cnt

JESMSGLG  JES2                          2        BC2AC10 A X                      2,003

JESJCL         JES2                          3        BC2AC10 A X                         138

JESYSMSG  JES2                          4        BC2AC10 A X                      2,464

DFHCXRF    BC2AC10A            101        BC2AC10 A X                          0

MSGUSR     BC2AC10A             103        BC2AC10 A X                     31,929

PLIMSG       BC2AC10A             104        BC2AC10 A X                           0

COUT          BC2AC10A             105        BC2AC10 A X                            0

CEEMSG     BC2AC10A             106        BC2AC10 A X                           0

CEEOUT     BC2AC10A             107         BC2AC10 A X                           0

PRINTER    BC2AC10A             114         BC2AC10 A X                           0

CAFF           BC2AC10A             115        BC2AC10 A X                            0

CRPO          BC2AC10A             116        BC2AC10 A X                            0

The same outputs are changed/increased later  (on 18.07.2011) but the old JESMSGLG and   JESYSMSG data are already saved in  SYSLOG10 file and if CICS fall down the old JESMSGLG and   JESYSMSG data will be available. CICS is still running started earlier (07.04.2011) on IPL time and records are increased as shown bellow:

NP   DDNAME       StepName ProcStep DSID Owner      C Dest              Rec-Cnt

JESMSGLG  JES2                          2        BC2AC10 A X                      2,304

JESJCL         JES2                          3        BC2AC10 A X                         138

JESYSMSG  JES2                          4        BC2AC10 A X                      2,993

DFHCXRF    BC2AC10A            101        BC2AC10 A X                          0

MSGUSR     BC2AC10A             103        BC2AC10 A X                     45,122

PLIMSG       BC2AC10A             104        BC2AC10 A X                           0

COUT          BC2AC10A             105        BC2AC10 A X                            0

CEEMSG     BC2AC10A             106        BC2AC10 A X                           0

CEEOUT     BC2AC10A             107         BC2AC10 A X                           0

PRINTER    BC2AC10A             114         BC2AC10 A X                           0

CAFF           BC2AC10A             115        BC2AC10 A X                            0

CRPO          BC2AC10A             116        BC2AC10 A X                            0

Conclusion: All CICS log messages are possibly saved  daily in SYSLOGxx files as they are produced and then are saving in the weekly IPOSAV.SYSLOGW file beginning from the start of CICS online (07.04.2011) – if every day is allocating a new SYSLOGXX and the previous is saved – we will have all CICS system  messages saved  in the weekly IPOSAV.SYSLOGW file. Each week a GDG  tape may be created on 8 times cycle to  be sure that if the system is damaged we   shall have always saved all CICS messages for the last 8 weeks.  If we chose 52 times cycle of GDG tapes we will have CICS messages for all days of the year.

The same old JESMSGLG and JESYSMSG data for another long running job – VTAM (job NET      STC01056) are already saved in  SYSLOG10 file  – VTAM   is still running as started on IPL time (07.04.2011):

NP   DDNAME       StepName ProcStep DSID Owner      C  Dest              Rec-Cnt

JESMSGLG  JES2                                 2 STCNET   X                       1,327

JESJCL         JES2                                 3 STCNET   X                           39

JESYSMSG  JES2                                 4 STCNET   X                             2

          3.7. Display maximum number of buffers,  threshold percent for warning messages and  free buffers

TSO/ISPF  – sd.log  – buffer commands:

–       /$D BUFDEF  – command to  display current  buffers definitions

$HASP840 BUFDEF

$HASP840 BUFDEF  BELOWBUF=(LIMIT=805,WARN=80,FREE=805),

$HASP840         EXTBUF=(LIMIT=839,WARN=80,FREE=839)

JES2 displays the buffer definitions and current free counts:

BELOWBUF= displays the maximum number of buffers(LIMIT=805), the threshold percent for warning messages (WARN=80) and the free(FREE=805) buffers below 16-megabytes in virtual storage.

EXTBUF= displays the maximum number of buffers(LIMIT=839), the threshold percent for warning messages (WARN=80) and the free (FREE=839) buffers above 16 megabytes in virtual storage.

In JES2 member SYS1.PARMLIB(JES2PARM) all this keywords  are defined with their current values.

–       /$T BUFDEF – Modify local buffers  definitions if needed.

Incoming search terms:

  • mvs SYSLOG
  • mainframe syslog
  • z/os syslog
  • syslog in mainframe
  • mvs syslog format
  • IEE767A
  • mainframe system log
  • iefc166d
  • IASXWR00
  • syslog mainframe

Leave A Reply

Comments

Dinn (Sep 03, 2014)

dear,you are a great one.
But I think there’s a little miss.
?1.4.1. SYSLOG buffers and messages status?this chapter is not the right one should appear here,the command listed here is not used to determine values in IEASYSxx,but ‘The LOGLIM parameter on the INIT statement in the CONSOLxx parmlib member controls the number of WTL buffers. ‘
Again,this really a good log,thank you.

Stefan (Sep 27, 2014)

I’m not sure that there is a mistake.. I can not see it right now.
Thanks for the input.
Will check soon ad get back to you with a better answer.
Thanks

7ads6x98yWordPress Login Protected by Clef