ZOS System log – SYSLOG
C O N T E N T S:
1.1. What is SYSLOG?
1.2. How SYSLOG file is created ?
1.3. How to process SYSLOG?
1.4. Checking syslog and console buffers status
2. SYSLOG prossessing of production system
2.1. Job SYSLGST (daily) - start daily SYSLOG processing
2.2. Job SYSWRT – External writer (daily – invoked by SYSLGST) – copy SYSLOG to daily SLGSAV.LWTR
2.3. Job SYSLGSVD (daily – invoked by SYSLGST) – copy SLGSAV.LWTR
to weekly SLGSAV.SYSLOGW
2.4. Job SYSLGPR (daily – invoked by SYSLGSVD) - print & empty
SLGSAV.LWTR
2.5. Job SYSLGSVW (weekly) – copies & empties weekly SLGSAV.SYSLOGW (+1)
to backup tape
3.1. GDG data sets creation on Syslog weekly tapes
3.2. Restore weekly generation file BACKUP.SYSLOG.G0185V00 with job RESAVW1 from backup tape to SLGSAV.TEST file for analysis
3.3. Search syslog data in weekly SLGSAV.TEST file restored for analysis
3.4. Converting current z/OS binary logs for problem determination consumption
on PC
3.5. Display all non-processed SYSLOG output groups on spool
3.6. CICS&VTAM SYSOUT data sets saved on SYSLOG
3.7. Display maximum number of buffers, threshold percent for warning messages and free buffers
1. SYSLOG information
1.1. What is SYSLOG?
The system log (SYSLOG) is a data set residing in the primary job entry subsystem’s spool space. This is SPOOL data, not a data set that can be accessed in any normal way and there is no data set name assigned to a spool file. It can be used by application and system programmers (through the WTL macro) to record communications about problem programs and system functions.
When we say “syslog” it means the following SPOOL data:
- console output
- output of a particular job (and if so, all output, one particular output)
- system logger output
The SYSLOG panel allows authorized users to display the system log, which is a collection of JES2 data sets that contain console messages, operator commands and operator responses for a z/OS JES2 system. It is accessed with the LOG command (TSO/ISPF- sd.log).
The operator can use the LOG command to add (if necessary) an entry to the system log - using command / LOG ’1111 22′ (as result the line ’1111 22′ is included in SYSLOG).
The SYSLOG is part of SYSOUT (SYSOUT data sets are output spool data sets on direct access storage devices – DASD in the MVS system) and actually consists of two separate logs: the system log and the hardcopy log. The hardcopy (consoles) log is a record of all system message traffic, written on each saved current SYSLOGXX file, specifically:
- Console Messages – to and from all consoles
- Commands and replies entered by the operator on sd.log
The hardcopy log is always written to the SYSLOG and indicates that the system log is to be the hardcopy medium.
The SYSLOG consists of the following:
- All messages issued through Write To Log (WTL) macros.
- All messages entered by the LOG operator commands.
- Usually, the hardcopy log (console messages).
- Any messages routed to the SYSLOG from any system component or program. In each job stream the steps JESMSGLG and JESYSMSG write messages about the job starting, ending and communications with the system in SYSLOG as follows:
JESMSGLG JES2 output data set contains system messages related to the job starting and ending. JESYSMSG JES2 output data set saves all system communications /information messages on SYSLOG.
1.2. How SYSLOG file is created ?
During z/os IPL /07.04.2011/ the SYSLOG data set is created and some important IPL messages follows:
——————————————– IPL started ——————————————————–
14:52:30.18 SYSLOG 00000000 IEE042I SYSTEM LOG DATA SET INITIALIZED
(The system is being initialized and the system has just created the SYSLOG data set.)
……………….
14:49:48.06 00000290 IEA371I SYS1.IPLPARM on device 0644 selected for IPL parameters
14:49:48.06 00000290 IEA246I LOAD ID 39 selected
14:49:48:06 00000290 IEA245I NUCLST ID 00 selected
14:49:48:06 00000290 IEA519I IODF dsn= SYS1.IODF39
14:49:49.81 00000290 IEA101A SPECIFY SYSTEM PARAMETERS FOR z/OS 01.04.00 HBB7707
14:49:49.81 00000290 IEE600I REPLY TO 00 IS;CLPA
14:50:03.24 00000290 IEE252I MEMBER IEASYS00 FOUND IN SYS1.PARMLIB
14:51:03.28 INTERNAL 00000290 S JES2 – /start JES2 from bc2a.*.stcproc /
14:51:03.54 00000098 DFH0100 CICS subsystem is now initialized
14:51:10.31 00000290 IEF196I 1 //JES2 JOB MSGLEVEL=1
14:51:10.32 00000290 IEF196I 2 //STARTING EXEC JES2 (from BC2A.ZOS140.STCPROC)
14:51:10.38 00000290 IEF196I 3 XXJES2 proc member=jes2iprm, almem=jes2parm, cmds=jes2cmds
14:51:11.67 00000090 IEF403I JES2 – STARTED – TIME=14.51.11
14:52:30.17 00000290 IEF196I IEF237I JES2 ALLOCATED TO SYSLOG00
14:52:30.18 00000090 IEE041I THE SYSTEM LOG IS NOW ACTIVE
………………..
———————————————- End of IPL —————————————————–
The first SYSLOG00 is created and allocated to JES2 at 14:52:30.17 and SYSTEM LOG IS ACTIVE at 14:52:30.18.
In sd.st at present the system task SYSLOG is running with initialized SYSLOG09 by BPXOINIT as Data set number 110:
NP JOBNAME JobID Owner Prty Queue C Pos SAff ASys Status
SYSLOG STC01072 +MASTER + 15 EXECUTION BC2A BC2A
with the same contents as of the sd.log panel of TSO/ISPF:
After - ? SYSLOG we see the file SYSLOG09:
SDSF JOB DATA SET DISPLAY – JOB SYSLOG (STC01072) LINE 1-1 (1)
NP DDNAME StepName ProcStep DSID Owner C Dest Rec-Cnt Page
SYSLOG09 BPXOINIT 110 +MASTER+ L 9,855
created by job SYSLOGD.
Steps to take: To view SYSLOG data sets that contain activity for z/OS , with sd.st of ISPF locate the SYSLOG and type S in the NP column next to this SYSLOG to select and browse it. The same may be done with sd.log of ISPF.
1.3. How to process SYSLOG?
SYSLOG periodically should be printed and checked for problems.
You might want to review the SYSLOG at a later date when you are doing problem diagnosis. Instead of having the SYSLOG job data written to the JES2 offload output queue after a certain number of WTLs, you should set up a procedure to organize the SYSLOG data set depending on your installation’s requirement.
You can perform your housekeeping as follows:
- Set up a procedure to have the system operator issue the WRITELOG command to close
and allocate a new SYSLOGxx at the end of each processing day. For example, WRITELOG L.
- At the end of the week, you will have seven SYSLOGxx data sets in the spool. You can now decide to offload the SYSLOG to an MVS data set or route it to a printer for a hardcopy printout.
- It’s really not necessary to keep SYSLOG data sets for more than a week, so if you keep a week’s worth of SYSLOG data sets, you should not have a problem.
The maximum number of WTLs (messages) allowed for the SYSLOG are defined at IPL time in SYS1.PARMLIB(IEASYS00):
LOGCLS=L, WILL NOT BE PRINTED BY DEFAULT
LOGLMT=999999, MAX WTL MESSAGES QUEUED, MUST BE 6 DIGITS
Usually LOGLMT is not reached and WRITELOG command is issued (by operator or special job) to close and free the current SYSLOG. The system releases the closed SYSLOG to the printer queue (scheduled for SYSOUT processing on output class L) as specified by the LOGCLS (L) parameter of the SYS1.PARMLIB(IEASYS00). Then the system allocates and opens a new – SYSLOGXX.
Writelog (W) – initialize SYSLOG00, 01, 02 .. begin new syslog from 1,2 .. lines/recs:
IEF196I IEF237I JES2 ALLOCATED TO SYSLOG01
IEE042I SYSTEM LOG DATA SET INITIALIZED
IEF196I IEF285I +MASTER+.SYSLOG.STC03395.D0000101.? SYSOUT
IEE043I A SYSTEM LOG DATA SET HAS BEEN QUEUED TO SYSOUT CLASS L
Explanation:
The system closed the system log (SYSLOG) data set and queued it to a SYSOUT class L. The SYSLOG data set might not be immediately available until JES processes it. The system queued the data set to a SYSOUT class L for one of these reasons:
- (1) The SYSLOG data set is full. The maximum number of SYSLOG records, as specified in the limit parameter(LOGLMT), has been reached. The system issues an internal WRITELOG command to close the current SYSLOG data set.
- (2) The operator entered a WRITELOG command to close the SYSLOG data set.
- (3) The operator entered a WRITELOG CLOSE or HALT command after the SYSLOG data set was opened in response to message IEE037D LOG NOT ACTIVE preceded by IEE533E SYSTEM LOG INITIALIZATION HAS FAILED.
The system does the following:
- For case (1) or (2), the system closes the currently recording SYSLOG data set and obtains a new data set (SYSLOGXX).
- For case (3), the system closes the currently recording SYSLOG data set and makes the SYSLOG unavailable.
You may organize automatically issuing of WRITELOG command as follows:
$ta wlog,t=23.58,i=86400,’$vs,''W L''’
Automatically spin SYSLOG to sysout L once per day:
- $ta ALL – JES2 displays all automatic command entries you are authorized
- $ta – Set command to specify automatic command named wlog
- t=23.58 – Start time of the automatic command (just before midnight today)
1.4. Checking syslog and console buffers status
1.4.1. SYSLOG buffers and messages status
You can display the LOGLIM value and the number of buffers in use, by issuing the D C,HC command in TSO sd.log:
/D C,HC – and the result follows: no log buffers in use (limit 1000), WTO=1 (limit 3000), WTOR=1 (limit 100):
IEE889I 16.37.38 CONSOLE DISPLAY 277
MSG: CURR=1 LIM=3000 RPLY:CURR=1 LIM=100 SYS=BC1B PFK=NONE
CONSOLE/ALT ID ————— SPECIFICATIONS —————
SYSLOG COND=H AUTH=CMDS NBUF=0 UD=N
ROUTCDE=ALL
LOG BUFFERS IN USE: 0 LOG BUFFER LIMIT: 1000
You can verify the LOGLIM value (1000) if enter from sd.log:
/K M
IEE144I K M,AMRF=Y,MLIM=3000,RLIM=0100,UEXIT=N,LOGLIM=001000,ROUTTIME=00
0,RMAX=9999,APPLID=SMCS2A
If this limit is not desired it can be increased temporary for this IPL to 2000 with the CONTROL M,LOGLIM command (/k m,loglim=2000).
If by any reason SYSLOG buffers are full the following messages may appear:
IEE012A NO LONGER SAVING MESSAGES FOR HARDCOPY, LOGLIM REACHED
Operator Response: Either activate a hardcopy medium, or increase the value of LOGLIM with the CONTROL M,LOGLIM command.
IEE767A SEVERE BUFFER SHORTAGE FOR SYSTEM LOG – 100% FULL
Explanation: The system found that the system log task (SYSLOG) may be active and that 100% of its buffers are full.
System Action: Write-to-log (WTL) processing ceases to allocate WTL queue elements. Subsequent WTL requests result in a return code of 4 to the user.
Operator Response: Increase the number of WTL buffers available to the system by entering the CONTROL M,LOGLIM=xxxxxx command.
System Programmer Response: Before the next system initialization, increase the value of LOGLIM in the CONSOLxx member of parmlib. This will allow for a higher number of outstanding WTL buffers.
If a VARY SYSLOG,HARDCPY,OFF,UNCOND command has previously been issued because the limit of LOGLIM has been reached or the buffer storage has been exhausted, then z/os has saved SYSLOG messages in SYSLOG buffer to be hardcopied later.
Example:
/VARY SYSLOG,HARDCPY,OFF,UNCOND – this command temporary stops writing on syslog as limits are reached
System action: Messages issued after this point will not be sent to SYSLOG, unless SYSLOG is reactivated. After some time of work the number of syslog buffers are increasing (less than 1000) and the contents of buffers with the saved SYSLOG messages after the last command in SYSLOG is as follows:
/D C,HC – command issued when SYSLOG is stoped and 24 records/messages are issued from the system:
IEE889I 17.04.50 CONSOLE DISPLAY 287
MSG: CURR=1 LIM=3000 RPLY:CURR=1 LIM=100 SYS=BC1B PFK=NONE
HARDCOPY SUSPENDED ON THIS SYSTEM
LOG BUFFERS IN USE: 24 LOG BUFFER LIMIT: 1000
/D C,HC - command issued when SYSLOG is stoped and 34 records/messages are issued from the system:
IEE889I 17.19.48 CONSOLE DISPLAY 291
MSG: CURR=1 LIM=3000 RPLY:CURR=1 LIM=100 SYS=BC1B PFK=NONE
HARDCOPY SUSPENDED ON THIS SYSTEM
LOG BUFFERS IN USE: 34 LOG BUFFER LIMIT: 1000
/D C,HC - command issued when SYSLOG is stoped and 59 records/messages are issued from the system:
RESPONSE=BC1B
IEE889I 18.07.23 CONSOLE DISPLAY 310
MSG: CURR=1 LIM=3000 RPLY:CURR=1 LIM=100 SYS=BC1B PFK=NONE
HARDCOPY SUSPENDED ON THIS SYSTEM
LOG BUFFERS IN USE: 59 LOG BUFFER LIMIT: 1000
To reactivate SYSLOG (the system log) and SYSLOG to become again the hardcopy medium, the following command is entered on TSO sd.log screen:
/V SYSLOG,HARDCPY – SYSLOG data set is assigned again as Hardcopy medium for system messages. After this command all 59 log buffers/messages following the last log command VARY SYSLOG,HARDCPY,OFF,UNCOND are restored/added in SYSLOG (before that should be changed LOGLIM if needed). If SYSLOG is reactivated the LOGLIM will be defaulted to the value of 1000.
The following messages are displayed proving that the saved buffers are restored and no log buffers are longer in use:
/D C,HC,L=IBMUSER – this command is issued from TSO user= IBMUSER
IEE889I 18.29.18 CONSOLE DISPLAY 313
MSG: CURR=1 LIM=3000 RPLY:CURR=1 LIM=100 SYS=BC1B PFK=NONE
CONSOLE/ALT ID ————— SPECIFICATIONS —————
SYSLOG COND=H AUTH=CMDS NBUF=0 UD=N
ROUTCDE=ALL
LOG BUFFERS IN USE: 0 LOG BUFFER LIMIT: 1000
After restoring the saved 59 buffers in SYSLOG the following last messages are proving the normal processing of SYSLOG:
/D SMF – this command is issued after activating SYSLOG and the following information can be found in sd.log:
IEE974I 18.29.33 SMF DATA SETS 315
NAME VOLSER SIZE(BLKS) %FULL STATUS
P-BC1B.MAN1 BC1MC3 9000 52 ACTIVE
S-BC1B.MAN2 BC1MC3 4500 0 ALTERNATE
S-BC1B.MAN3 BC1MC3 4500 0 ALTERNATE
/D C,HC – this command is issued after activating SYSLOG and the following information can be found also in sd.log:
IEE889I 16.13.01 CONSOLE DISPLAY 337
MSG: CURR=1 LIM=3000 RPLY:CURR=1 LIM=100 SYS=BC1B PFK=NONE
CONSOLE/ALT ID ————— SPECIFICATIONS —————
SYSLOG COND=H AUTH=CMDS NBUF=0 UD=N
ROUTCDE=ALL
LOG BUFFERS IN USE: 0 LOG BUFFER LIMIT: 1000
To display all SYSLOG output groups/files with not processed by the external writer (old or present) on spool enter:
/$DOS’SYSLOG’,CRTIME,AGE,REC,ODISP,Q
If some old SYSLOGXX files with are waiting 6 days (from creating date till now) to be processed by the external writer, the following information is received for date=2011.125
$HASP686 OUTPUT(SYSLOG) 799
$HASP686 OUTPUT(SYSLOG) CRTIME=(2011.125,12:57:12),DAYS=6, – at 05.05.2011 created
$HASP686 RECORDS=(164 OF 164),ODISP=WRITE,
$HASP686 QUEUE=L
$HASP686 OUTPUT(SYSLOG) 800
$HASP686 OUTPUT(SYSLOG) CRTIME=(2011.125,12:34:23),DAYS=6, – at 05.05.2011 created
$HASP686 RECORDS=(23358 OF 23358),ODISP=WRITE,
$HASP686 QUEUE=L
If no SYSLOGXX files with are waiting to be processed by the external writer, the following information is received:
$HASP003 RC=(52), 841
$HASP003 RC=(52),S’SYSLOG’ – NO SELECTABLE ENTRIES FOUND
$HASP003 MATCHING SPECIFICATION
1.4.2. WTO & WTOR console buffers and messages status
To verify console buffers status enter:
/D CONSOLES – curr=4 buffers used for console messages – WTO(limit 3000) and curr=1 buffer for system messages which are awaiting a response from the system operator WTOR (limit 100):
IEE889I 17.31.14 CONSOLE DISPLAY 021
MSG: CURR=4 LIM=3000 RPLY:CURR=1 LIM=100 SYS=BC2A PFK=NONE
or
/D C,HC – no log buffers in use (limit 1000), Messages – WTO=1 (limit 3000), Replies – WTOR=1 (limit 100):
IEE889I 16.37.38 CONSOLE DISPLAY 277
MSG: CURR=1 LIM=3000 RPLY:CURR=1 LIM=100 SYS=BC1B PFK=NONE
CONSOLE/ALT ID ————— SPECIFICATIONS —————
SYSLOG COND=H AUTH=CMDS NBUF=0 UD=N
ROUTCDE=ALL
LOG BUFFERS IN USE: 0 LOG BUFFER LIMIT: 1000
To verify the present WTO/WTOR limits – MLIM/WTO(3000) and RLIM/WTOR(100) values you can enter also:
/K M
IEE144I K M,AMRF=Y,MLIM=3000,RLIM=0100,UEXIT=N,LOGLIM=001000,ROUTTIME=00
0,RMAX=9999,APPLID=SMCS2A
If these limits are not desired they can be increased temporary for this IPL to 5000 and 200 with the CONTROL M,LOGLIM command (/k m,mlim=5000 and /k m,rlim=200).
If the console is OK displaying messages and the screen is not full up, the following message appears left below:
IEE163I MODE= RD – This must be always verified and maintained/keeped !!!
If the console screen is full up or the limit for WTO or WTOR is reached and no messages may be written on the console, the following messages appear bellow:
On left: IEE163I MODE= HELD On right: IEE159E MESSAGE WAITING
The same me be done if by careless mistake is pressed Enter key !!! To clear the screen you have to press ENTER key and HELD is changed to RD and message IEE159E MESSAGE WAITING disappears – the restore action is OK.
The same screen clear may be done by pressing F1 for each line to deleted/moved it up.
Dealing with System Messages Awaiting Replies (System Requests):
Enter /d r,l to display the current list of system messages which are awaiting a response from the system operator (outstanding System Requests), including action messages and WTORs. Action messages can be deleted, and replies to WTORs can be entered as line commands directly on the displayed panel.
Syntax
DR [n1,n2,...] [ALL/MVS/USER] [EVENTUAL]
n1,n2,… List of route codes for which messages are to be displayed.
ALL.Show messages with route codes 1 through 28.
MVS. Show messages with route codes 1 through 12.
USER. Show messages with route codes 13 through 28.
EVENTUAL Also show eventual action messages.
Example:
/D R,L – only one message (0002 R *0002) is waiting reply
IEE112I 10.16.56 PENDING REQUESTS 210
RM=1 IM=0 CEM=4 EM=13 RU=0 IR=0 AMRF
ID:R/K T MESSAGE TEXT
0002 R *0002 EMS0990A BC2ANVAS READY FOR COMMANDS.
32 E IEF234E K 0301,HV0311,PVT,RESTSIK1,DFDSS – RACK=HV0311
31 E IEF234E K 0300,HV0262,PVT,RESTSIK0,DFDSS – RACK=HV0262
In critical/disaster cases the following command should be entered on master or alternate console (if overflow messages inform that WTO buffers are 100% or less full):
K q,r=hc – this command from master or alternate console delete all WTO console messages in the WTO buffers and free them.
2. SYSLOG prossessing of production system
The next time when SYSLOG00 is full a sequence of steps should be executed with replies on sd.log. The following sequence of daily and weekly jobs are described that write syslog messages on DAILY ( SLGSAV.LWTR) and WEEKLY (SLGSAV.SYSLOGW) ACCUMULATION DATA SETS and when this data set SLGSAV.SYSLOGW is full, to copy it on tape and then empties SLGSAV.SYSLOGW.
2.1. Job SYSLGST (daily) - start daily SYSLOG processing
This job starts the external writer ‘SYSWRT‘ to write class L syslog to a disk data set SLGSAV.LWTR. When SYSWRT has processed all syslog data, then it is stopped. After that a job ‘SYSLGSVD’ is started from job SYSLGST to write system log data set SLGSAV.LWTR to weekly accumulation system log data set SLGSAV.SYSLOGW and to invoke job SYSLGPR to print syslog (system log data set SLGSAV.LWTR) and to empty SLGSAV.LWTR. By job SYSLGSVW (invoked weekly) the operator is asked to mount labeled scratched or work tape where weekly syslog data set SLGSAV.SYSLOGW is copied as a generated data set and then empted SLGSAV.SYSLOGW.
//SYSLGST JOB (),
// ‘OPERATOR’,
// ,
// MSGCLASS=X,
// MSGLEVEL=(1,1),
// NOTIFY=IBMUSER,
// TIME=1440
//*
//SYSLOG1 EXEC IPOWTO,REGION=512K – * ONE OR MORE MESSAGES TO BE SENT TO THE OPERATOR
//SYSIN DD *
—- THIS JOB STARTS SYSWRT TO WRITE CLASS L SYSLOG TO
—- SLGSAV.LWTR. WHEN SYSWRT HAS PROCESSED ALL SYSLOG
—- DATA, THEN REPLY TO EXECUTE ‘STOP L’.
/*
//SYSLGSVD EXEC JOB,N= SYSLGSVD,REGION=512K- * START A JOBSTREAM FROM A PDS
/*
//*
//DISPLAY T DATE AND TIME ENTERED IN SYSLOG
//WRITELOG L WRITES LOG TO SYSOUT = L
//START SYSWRT.L,,,L STARTS SYSWRT TO DISK(SLGSAV.LWTR)
//STOP L STOPS SYSWRT. DO NOT REPLY UNTIL
//* MESSAGE ‘WTR L WAITING FOR WORK‘
2.2. Job SYSWRT – External writer (daily – invoked by SYSLGST) – copy
SYSLOG to daily SLGSAV.LWTR
This job write class L syslog to a disk data set SLGSAV.LWTR and when SYSWRT has processed all syslog data, then it is stopped.
// SYSWRT JOB MSGLEVEL=1
//L EXEC SYSWRT
// SYSWRT PROC
//* DOC: THIS PROCEDURE EXECUTES THE EXTERNAL WRITER WITH AN OUTPUT
//* CLASS EQUAL TO ‘L’ .
//*
//IEFPROC EXEC PGM=IASXWR00,
// PARM=’PL’
//IEFRDER DD DSN= SLGSAV.LWTR,
// DISP=(MOD,KEEP),
// DCB=(BLKSIZE=4088,LRECL=137,BUFL=4088,BUFNO=5,RECFM=VBA)
The following events and replies occur during execution of the previous jobs (2.1, 2.2):
IEFC165I // WRITELOG L 644
*0018 IEFC166D REPLY Y/N TO EXECUTE/SUPPRESS COMMAND
R 18,Y
IEE600I REPLY TO 0018 IS;Y
WRITELOG L – SYSLOG00 is to be written on SYSOUT
and External writer SYSWRT is started:
IEFC165I // START SYSWRT.L,,,L 436
*0009 IEFC166D REPLY Y/N TO EXECUTE/SUPPRESS COMMAND
IEF196I IEF237I JES2 ALLOCATED TO SYSLOG01
IEE042I SYSTEM LOG DATA SET INITIALIZED (new SYSLOG01 is initialized in sd.log)
IEF196I IEF285I +MASTER+.SYSLOG.STC01072.D0000101.? SYSOUT
IEE043I A SYSTEM LOG DATA SET HAS BEEN QUEUED TO SYSOUT CLASS L
R 9,Y
IEE600I REPLY TO 0009 IS;Y
START SYSWRT.L,,,L
IEFC165I // STOP L 444
0010 IEFC166D REPLY Y/N TO EXECUTE/SUPPRESS COMMAND
$HASP100 SYSWRT ON STCINRDR
IEF695I START SYSWRT WITH JOBNAME SYSWRT IS ASSIGNED TO USER STCOPER
, GROUP STCGRP
$HASP373 SYSWRT STARTED
IEF403I SYSWRT – STARTED – TIME=17.14.54
$HASP250 SYSLOG PURGED — (JOB KEY WAS C0FA25BC)
$HASP250 SYSLOG PURGED — (JOB KEY WAS C6E3EB9F)
IEF176I WTR L WAITING FOR WORK,, DEST=LOCAL
R 10,Y
IEE600I REPLY TO 0010 IS;Y
STOP L
IEF301I L WTR CLOSED
IEF404I SYSWRT – ENDED – TIME=17.16.09
2.3. Job SYSLGSVD (daily – invoked by SYSLGST) - copy SLGSAV.LWTR
to weekly SLGSAV.SYSLOGW
This job uses IEBGENER to copy the system log data set SLGSAV.LWTR to weekly syslog data set SLGSAV.SYSLOGW and invokes SYSPRT to print and clear SLGSAV.LWTR.
//SYSLGSVD JOB (9999,CSOPER,10,10),
// ‘OPERATOR’,
// ,
// MSGCLASS=X,
// MSGLEVEL=(1,1),
// NOTIFY=,
// TIME=1440,
// USER=
//*
//SLGSAV1 EXEC PGM=IEBGENER,REGION=512K
//SYSPRINT DD SYSOUT=*
//SYSIN DD DUMMY
//SYSUT1 DD DSN=SLGSAV.LWTR,
// DISP=SHR
//SYSUT2 DD DSN=SLGSAV.SYSLOGW,
// DISP=MOD
//*
//SLGSAV3 EXEC JOB,N=SYSLGPR,REGION=512K
/*
2.4. Job SYSLGPR (daily – invoked by SYSLGSVD) - print & empty SLGSAV.LWTR
This job is invoked by SYSLGSVD to print and empty SLGSAV.LWTR data set.
//SYSLGPR JOB (9999,CSOPER,10,10),
// ‘OPERATOR’,
// ,
// MSGCLASS=X,
// MSGLEVEL=(1,1),
// NOTIFY=,
// TIME=1440,
// USER=
//*
//* STEP1. PRINTS SLGSAV.LWTR
//* STEP2. EMPTIES SLGSAV.LWTR
//*
//SLGPRT1 EXEC PGM=IEBGENER,REGION=512K
//SYSPRINT DD SYSOUT=*
//SYSIN DD DUMMY
//SYSUT1 DD DSN= SLGSAV.LWTR,
// DISP=SHR
//SYSUT2 DD SYSOUT=*
//*
//SLGPRT2 EXEC PGM=IEBGENER,COND=EVEN,REGION=512K
//SYSPRINT DD SYSOUT=*
//SYSIN DD DUMMY
//SYSUT1 DD DUMMY,DCB= SLGSAV.LWTR
//SYSUT2 DD DSN= SLGSAV.LWTR,
// DISP=OLD
//*
2.5. Job SYSLGSVW (weekly) – copies & empties weekly SLGSAV.SYSLOGW(+1) to backup tape
This job asks the operator to mount a standard labeled scratch tape for using by GDG (Generation Data Group) and copies weekly data set SLGSAV.SYSLOGW as a generation data set. Then empties SLGSAV.SYSLOGW.
//SYSLGSVW JOB (9999,CSOPER,10,10),
// ‘OPERATOR’,
// ,
// MSGCLASS=X,
// MSGLEVEL=(1,1),
// NOTIFY=,
// TIME=1440,
// USER=
//*
//SYSLGW1 EXEC IPOWTO,MSG=’SIGN OUT 1 STD LBL TAPE FOR NEXT STEP’
//SYSIN DD * – following text written to the operator
DSN=BACKUP.SYSLOG(+1) — DUMP OF ONE WEEKS SYSLOG
RETPD = 365
THE INTERNAL AND EXTERNAL TAPE LABELS SHOULD MATCH.
USE THE STANDARD LABEL TAPE NUMBER IN REPLY TO THE TAPE MOUNT REQUEST.
/*
//SYSLGW2 EXEC PGM=IEBGENER,REGION=512K,COND=(0,LT)
//SYSPRINT DD SYSOUT=*
//SYSIN DD DUMMY
//SYSUT1 DD DSN=SLGSAV.SYSLOGW,
// DISP=SHR
//SYSUT2 DD DSN=BACKUP.SYSLOG(+1),
// DISP=(,CATLG,DELETE),VOL=(,,,25),
// LABEL=RETPD=365,UNIT=(300,,DEFER),
// DCB=(SLGSAV.SYSLOGW)
//*
//SYSLGW3 EXEC PGM=IEBGENER,REGION=512K,COND=(0,LT,SYSLGW2)
//SYSPRINT DD SYSOUT=*
//SYSIN DD DUMMY
//SYSUT1 DD DSN=NULLFILE,DCB=SLGSAV.SYSLOGW
//SYSUT2 DD DSN=SLGSAV.SYSLOGW,
// DISP=OLD
//*
RETPD=365 – specify a retention period of 365 days for the data set (may be used 52 generations for one year).
VOL=(,,,25), – using scratch tape
If GDG named BACKUP.SYSLOG consists of 10 generations, the first 10 generations will require 10 new scratch tapes and the respective DSN names of the generation files BACKUP.SYSLOG.G0001V00 – BACKUP.SYSLOG.G0010V00 will be written on each generation tape in menu ‘Volume Display’ of REMOVABLE MEDIA MANAGER (DFSMSrmm) when a scratch tape is required and processed. When the 11st generation file BACKUP.SYSLOG.G0011V00 requires a tape we should mount a new scratch tape and the first generation tape with dsn = BACKUP.SYSLOG.G0001V00 will be freed to scratch spool (this is done when the job/program for verification and expiration tapes is executed – //HSKP EXEC PGM=EDGHSKP, PARM= ‘VRSEL,EXPROC’). Then the first generation tape with dsn = BACKUP.SYSLOG.G0001V00 can be released (TSO/ISPF – r – 5 – 1- 4 – release a volume) and after execution the job for expiration tapes it becomes ‘scratch’ and is returned to the scratch spool. On the scratch tape the generation file will be written as “master” nevertheless of parameters: “Retention date . . : 2011/195” which was expired and nothing in “Expiration date . . . . . “. After writing the generation file on this scratch tape, nothing will be on “ Retention date . . :” but “ Expiration date . . . . . : 2012/196” will show one year period for expiration time starting from the current day. When you whish to use “VRS – Vital record specifications”, then retention date will be changed to “Retention date . . : PERMANENT” without changes on “Expiration date . . . . . : 2012/196”. If you want the last generation file to be on a separate tape then you can use in the previous job VOL=SER=CB0961 instead of VOL=(,,,25). When the separated tape is ‘master’ or ‘user’ – the generation file will be written on it without problem.
3. Extra SYSLOG Utilities
3.1. GDG data sets creation on Syslog weekly tapes
Generation Data Groups or GDGs are used to automatically manage datasets that are created periodically, for example log files. GDG name is not necessarily intuitive. The GDG naming standard appends a version segment to the end of the file name, as GxxxxVyy, where xxxx is the generation number from 0000 to 9999 and yy is the cycle number, from 00 to 99. The biggest advantage of using a GDG is that the system keeps track of the generations for you so you don’t have to change your JCL each time you create a new version. So while GDGs cannot be considered a panacea for cyclic datasets, they are very widely used and very useful.
//DEFGDG JOB (9999,IPO,10,10),
// ‘OPERATOR’,
// ,
// MSGCLASS=X,
// MSGLEVEL=(1,1),
// TIME=1440
//*
//* DOC: THIS IS JOB CREATES A GDG FOR ARCHIVING
//* 10 SYSLOG FILES – FOR 10 generations
//*
//BR14 EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
DEF GDG (NAME(BACKUP.SYSLOG) -
LIM(10) NEMP SCR)
/*
LIM(10) – means that this GDG will consist of 10 generations
NEMP - means that when the 10 generation limit is reached, the oldest will be deleted, but all the others will be retained.
SCR – means uncatalog and delete files when they exceed the generation limit.
Before you can create the files that attach to a GDG base, you need to create a model dataset that contains the DCB information for the files. This model data just needs to be a VTOC entry, it does not need to actually use up disk space.
Many sites use a single model dataset, and then override the DCB information to build the correct type of file. To create a model DCB, use a dummy IEFBR14 job like this
//MODLDSCB JOB (9999,IPO,10,10),
// ‘OPERATOR’,
// ,
// MSGCLASS=X,
// MSGLEVEL=(1,1),
// TIME=1440
//* DOC: THIS IS JOB CREATES A MODEL DSCB FOR USE
//* TO ARCHIVE THE SYSLOG FILE TO A GDG.
//BR14 EXEC PGM=IEFBR14
//MODEL DD DSN= BACKUP.SYSLOG.MODEL,
// DISP=(NEW,CATLG,DELETE),
// DCB= (LRECL=137,RECFM=VBA,BLKSIZE=4088), or DCB= SLGSAV.SYSLOGW with the same parameters
// SPACE=(0,(0,0)),
// VOL=SER=XXXXXX,
// UNIT=3390
//*
For our case it is DCB= SLGSAV.SYSLOG (DCB of existing file) .
The SPACE=(0,(0,0)) parameter means that this file will just be created as a VTOC entry, it will not use up disk space.
OK, now we have a GDG base and a GDG model, we can allocate a GDG file. All you do is specify the new GDG file with a +1 generation number in your JCL like this in job SYSLGSVW:
// SYSLGW1 EXEC PGM=IEBGENER,REGION=512K,COND=(0,LT)
//SYSUT2 DD DSN=BACKUP.SYSLOG(+1),
// DISP=(,CATLG,DELETE),VOL=(,,,25),
// LABEL=RETPD=365,UNIT=(300,,DEFER),
// DCB=(SLGSAV.SYSLOGW)
This source code will create a file called BACKUP.SYSLOG.G0001V00. When you add the 11st file it will be created as BACKUP.SYSLOG.G0011V00 and the BACKUP.SYSLOG.G0001V00 will be deleted, assuming you have SCRATCH coded in the base definition - VOL=(,,,25).
If you want to refer to a particular GDG after it has been created, you can either use the full GDG name, or a relative number. For example if the current generation of BACKUP.SYSLOG is 04, then you can refer to the previous version either as BACKUP.SYSLOG.G0003V00 or as BACKUP.SYSLOG (-1).
3.2. Restore weekly generation file BACKUP.SYSLOG.G0185V00 with job RESAVW1 from backup tape to SLGSAV.TEST file for analysis
When restoring a disk file with IEBGENER, you must know the amount of space required for the file and code the SPACE parameter on the SYSUT2 dd statement. The received file SLGSAV.TEST is with DCB=SLGSAV.SYSLOGW of the backed up weekly syslog file.
The TSO/ISPF p.3.4 function for GDG=BACKUP.SYSLOG display the name of the following GDG cataloged file: BACKUP.SYSLOG.G0185V00 on tape CB0131:
p.3.4 – BACKUP.SYSLOG
DSLIST – Data Sets Matching BACKUP.SYSLOG:
——————————————————————– Volume
BACKUP.SYSLOG.G0184V00 CB0125
BACKUP.SYSLOG.G0185V00 CB0131
Using job RESAVW1 we can receive the generation file BACKUP.SYSLOG.G0185V00 on disk as file SLGSAV.TEST for analysis of the respective weekly/daily SYSLOG we look for:
//RESAVW1 JOB (9999,CSOPER,10,10),
// ‘OPERATOR’,
// ,
// MSGCLASS=X,
// MSGLEVEL=(1,1),
// NOTIFY=,
// TIME=1440,
// USER=
//*
//SYSLGW2 EXEC PGM=IEBGENER,REGION=512K
//SYSPRINT DD SYSOUT=*
//SYSIN DD DUMMY
//SYSUT1 DD DSN=BACKUP.SYSLOG.G0185V00,
// DISP=OLD
//SYSUT2 DD DSN=SLGSAV.TEST,
// DISP=(,CATLG,DELETE),
// SPACE=(TRK,1500),UNIT=DISK,
// DCB=(SLGSAV.SYSLOGW)
3.3. Search syslog data in weekly SLGSAV.TEST file restored for analysis
If you need to find syslog data in SLGSAV.TEST for analysis for exact day of year and exact hour, do the following with TSO/ISPF p.3.4:
- First: F ‘ALLOCATED TO SYSLOG’ – to find the respective SYSLOGXX file where the log data are saved (SYSLOG00 is the first file after IPL –here SYSLOG06 is allocated at 11:32:17.04 AM):
11:32:17.04 00000290 IEF196I IEF237I JES2 ALLOCATED TO SYSLOG06
- Second: F ICH70001I – to find the respective job Owner LAST ACCESS – on date: JUNE 1, 2011
11:33:04.36 JOB04920 00000090 ICH70001I BCA1MGP LAST ACCESS AT 11:20:36 ON WEDNESDAY, JUNE 1, 2011
- Second: F ICH70001I (PF 5 – next job Owner)
11:35:22.61 JOB04925 00000090 ICH70001I BCO1DSL LAST ACCESS AT 11:16:47 ON WEDNESDAY, JUNE 1, 2011
- Second: F ICH70001I (PF 5 – next job Owner)
13:26:35.38 JOB04930 00000090 ICH70001I BC2AC10A LAST ACCESS AT 09:49:54 ON WEDNESDAY, JUNE 1, 2011
- Second: F ICH70001I (PF 5 – next job Owner)
13:26:40.44 JOB04938 00000090 ICH70001I BC2AC10A LAST ACCESS AT 13:26:35 ON WEDNESDAY, JUNE 1, 2011
3.4. Converting current z/OS binary logs for problem determination consumption on PC
To receive the current syslog data in character format easy on sequential file (Organization=PS) and transfer it to PC, you need to perform the following step-by-step actions:
- TSO/ISPF – sd.st
- xdc SYSLOG – Fill up the following fields: Data set name ===> ‘bca1mgp.syslog.sdst’, Disposition ===> NEW – enter - the next file is ready for investigation in character format or to transfer by FTP to PC:
- TSO/ISPF – p.3.4 – BCA1MGP.SYSLOG.SDST/ ready for FTP – 35701 recs in character format and :
- Organization . . . : PS
- Record format . . . : VBA
- Record length . . . : 240
- Block size . . . . : 3120
- 1st extent cylinders: 6
3.5. Display all non-processed SYSLOG output groups on spool
TSO/ISPF – sd.log – enter: /$DOS’SYSLOG’,CRTIME,AGE,REC,ODISP,Q
19.05.2011 – /$DOS’SYSLOG’,CRTIME,AGE,REC,ODISP,Q – Display all SYSLOG output groups on spool - BC2A /90 –created on 05.05.2011 + 13 days passed = 18.05.2011 –current 2011.139=19.05.2011
$HASP686 OUTPUT(SYSLOG) 988
$HASP686 OUTPUT(SYSLOG) CRTIME=(2011.125,12:57:12),DAYS=13,
$HASP686 RECORDS=(164 OF 164),ODISP=WRITE,
$HASP686 QUEUE=L
$HASP686 OUTPUT(SYSLOG) 989
$HASP686 OUTPUT(SYSLOG) CRTIME=(2011.125,12:34:23),DAYS=13,
$HASP686 RECORDS=(23358 OF 23358),ODISP=WRITE,
$HASP686 QUEUE=L
If the syslog is processed and no output groups exist on spool the following messages are received:
$HASP003 RC=(52),S’SYSLOG’ – NO SELECTABLE ENTRIES FOUND
$HASP003 MATCHING SPECIFICATION
3.6. CICS&VTAM SYSOUT data sets saved on SYSLOG
For JES2 data of long running jobs (as CICS job BC2AC10A) there is a danger of losing log information if CICS failed. One way of saving JES2 CICS data is to stop or not CICS and offload all of the following JES2 CICS data on tape:
This example shows the SYSOUT data sets that might be associated with a CICS address space:
NP DDNAME StepName ProcStep DSID Owner
- JESJCLIN 1 CICSTS
- JESMSGLG JES2 2 CICSTS
- JESJCL JES2 3 CICSTS
- JESYSMSG JES2 4 CICSTS
- $INTTEXT JES2 5 CICSTS
- CAFF SCSCPAA1 101 CICSTS
- CINT SCSCPAA1 103 CICSTS
- DFHCXRF SCSCPAA1 104 CICSTS
- COUT SCSCPAA1 105 CICSTS
- CEEMSG SCSCPAA1 106 CICSTS
- CEEOUT SCSCPAA1 107 CICSTS
- PLIMSG SCSCPAA1 108 CICSTS
- CRPO SCSCPAA1 109 CICSTS
- MSGUSR SCSCPAA1 110 CICSTS
The key SYSOUT data sets to review for problem determination data are the JESMSGLG and MSGUSR data sets. The CEEMSG and CEEOUT data sets will contain Language Environment® (LE) problem data typically associated with application problems.
The CICS JESMSGLG SYSOUT data set includes information related to CICS startup and errors related to system problems, not specifically transaction related.
Another way is saving CICS intermediate log records in SYSLOGXX saved files by WRITELOG command. In JESMSSLG and JESYSMSG data sets of BC2AC10A (STC01184) CICS job stream reside all messages issued from CICS job starting (07.04.2011) up to current day. The previous SYSLOGXX-1 files in the weekly saved IPOSAV.SYSLOGW file contain all CICS important messages issued after CICS job starting (07.04.2011).
JES2 CICS production data:
NP DDNAME StepName ProcStep DSID Owner C Dest Rec-Cnt
JESMSGLG JES2 2 BC2AC10 A X 2,003
JESJCL JES2 3 BC2AC10 A X 138
JESYSMSG JES2 4 BC2AC10 A X 2,464
DFHCXRF BC2AC10A 101 BC2AC10 A X 0
MSGUSR BC2AC10A 103 BC2AC10 A X 31,929
PLIMSG BC2AC10A 104 BC2AC10 A X 0
COUT BC2AC10A 105 BC2AC10 A X 0
CEEMSG BC2AC10A 106 BC2AC10 A X 0
CEEOUT BC2AC10A 107 BC2AC10 A X 0
PRINTER BC2AC10A 114 BC2AC10 A X 0
CAFF BC2AC10A 115 BC2AC10 A X 0
CRPO BC2AC10A 116 BC2AC10 A X 0
The same outputs are changed/increased later (on 18.07.2011) but the old JESMSGLG and JESYSMSG data are already saved in SYSLOG10 file and if CICS fall down the old JESMSGLG and JESYSMSG data will be available. CICS is still running started earlier (07.04.2011) on IPL time and records are increased as shown bellow:
NP DDNAME StepName ProcStep DSID Owner C Dest Rec-Cnt
JESMSGLG JES2 2 BC2AC10 A X 2,304
JESJCL JES2 3 BC2AC10 A X 138
JESYSMSG JES2 4 BC2AC10 A X 2,993
DFHCXRF BC2AC10A 101 BC2AC10 A X 0
MSGUSR BC2AC10A 103 BC2AC10 A X 45,122
PLIMSG BC2AC10A 104 BC2AC10 A X 0
COUT BC2AC10A 105 BC2AC10 A X 0
CEEMSG BC2AC10A 106 BC2AC10 A X 0
CEEOUT BC2AC10A 107 BC2AC10 A X 0
PRINTER BC2AC10A 114 BC2AC10 A X 0
CAFF BC2AC10A 115 BC2AC10 A X 0
CRPO BC2AC10A 116 BC2AC10 A X 0
Conclusion: All CICS log messages are possibly saved daily in SYSLOGxx files as they are produced and then are saving in the weekly IPOSAV.SYSLOGW file beginning from the start of CICS online (07.04.2011) – if every day is allocating a new SYSLOGXX and the previous is saved – we will have all CICS system messages saved in the weekly IPOSAV.SYSLOGW file. Each week a GDG tape may be created on 8 times cycle to be sure that if the system is damaged we shall have always saved all CICS messages for the last 8 weeks. If we chose 52 times cycle of GDG tapes we will have CICS messages for all days of the year.
The same old JESMSGLG and JESYSMSG data for another long running job – VTAM (job NET STC01056) are already saved in SYSLOG10 file – VTAM is still running as started on IPL time (07.04.2011):
NP DDNAME StepName ProcStep DSID Owner C Dest Rec-Cnt
JESMSGLG JES2 2 STCNET X 1,327
JESJCL JES2 3 STCNET X 39
JESYSMSG JES2 4 STCNET X 2
3.7. Display maximum number of buffers, threshold percent for warning messages and free buffers
TSO/ISPF – sd.log – buffer commands:
- /$D BUFDEF – command to display current buffers definitions
$HASP840 BUFDEF
$HASP840 BUFDEF BELOWBUF=(LIMIT=805,WARN=80,FREE=805),
$HASP840 EXTBUF=(LIMIT=839,WARN=80,FREE=839)
JES2 displays the buffer definitions and current free counts:
BELOWBUF= displays the maximum number of buffers(LIMIT=805), the threshold percent for warning messages (WARN=80) and the free(FREE=805) buffers below 16-megabytes in virtual storage.
EXTBUF= displays the maximum number of buffers(LIMIT=839), the threshold percent for warning messages (WARN=80) and the free (FREE=839) buffers above 16 megabytes in virtual storage.
In JES2 member SYS1.PARMLIB(JES2PARM) all this keywords are defined with their current values.
- /$T BUFDEF – Modify local buffers definitions if needed.
Incoming search terms:
- mvs SYSLOG
- mainframe syslog
- IEE767A
- mainframe system log
- z os syslog
- IEE042I
- z/os syslog
- syslog in mainframe
- set up syslogd z/os
- zOS syslog
Comments
rack card, rack card printing, rack cards printing, rack cards (Dec 06, 2011)
Excellent submit, very informative. I ponder why the other experts of this sector don’t notice this. You must continue your writing. I’m confident, you have a huge readers’ base already!|What’s Going down i’m new to this, I stumbled upon this I have discovered It positively useful and it has aided me out loads. I’m hoping to give a contribution & aid other users like its aided me. Great job.
pa puppy (Dec 07, 2011)
Hi my friend! I wish to say that this article is awesome, nice written and include approximately all important infos. I’d like to see more posts like this .
benefits of being self employed (Dec 07, 2011)
Interesting Post , Many thanks. Staying in North of Scotland it’s great to keep returning to a helpful bloglike yours, especially with goodposts like the one above. Many thanks and keep up the good work.
Freelance (Dec 07, 2011)
I feel that is one of the most significant information for me. And i’m happy reading your article. However should remark on few normal issues, The website taste is great, the articles is really great : D. Excellent activity, cheers
Lindsey Josephpauline (Dec 09, 2011)
Great web site. Plenty of useful info here. I’m sending it to a few friends ans also sharing in delicious. And naturally, thanks for your sweat!