Troubles with TSO Logon time delay and system wait

Contents

Bottom

1. Logon time delay

2. TSO system wait

If you or other customers are waiting too much when logging on TSO of z/OS in only one particular LPAR of the CPU and no TSO logon time delay exists on other LPAR’s, go to item 1 of the Contents. If TSO is in system wait troubles during normal operations, go to item 2 of the Contents.

          1. Logon time delay

During TSO logon of any TSO user after signing with his ID/PW and after initial messages:

ICH70001I BCA1MGP LAST ACCESS AT 13:01:42 ON FRIDAY, OCTOBER 26, 2012

IKJ56455I BCA1MGP LOGON IN PROGRESS AT 14:10:41 ON OCTOBER 26, 2012

IKJ56951I NO BROADCAST MESSAGES

more than 30 seconds time delay are passing before the asterisks (***) to appear. After pressing Enter following asterisks (***) – more than 30 seconds are passing before the first ISPF logon menu panel to appear. This TSO time delay arises unexpectedly when Cobol program execution is processing enormous data, the power of CPU is restricted and shortage of the virtual memory occurs.

In ISPF/SDSF menu some functions are normal (as usual) and others – very slow. These TSO logon and ISPF time delays are happening to all users in the same LPAR BC2A where Cobol program is running. All other activities started by any TSO user – batch jobs for restore tapes, online ditto functions, FTP for download/upload to/from PC are delayed more than 3 to 6 times. Once happened – these delays continue after successfull end of the Cobol program execution and not disappear until the new IPL of this LPAR BC2A.

In the two other LPARs (BC3A and BC1B) where the Cobol program is not running – no such problems exist with their TSO Logon’s .

The first IBM support reaction of this case was as follows:

Probably the Problem is cobol program that allocates large number of pages. Those pages are stored in central memory and when there is no more free frames pages are stored in page data sets on disks under ASM (Auxiliary storage manager) control. When cobol batch job is finished all those pages in central memory and in page data sets must be released. Real problem appears if this process do not take place and shortage of the virtual memory occurs.

The customer was advised by IBM support to send the following documentation for investigation:

(1). Job list of the Cobol job.

(2). SVC dump of the current situation when Cobol job is finished, but problem with slow responses remain.

(3). RMF III VSAM data sets since last day from the time of IPL until present time.

(4). Syslog and LOGREC data set for further analysis to check if software errors are cause of the memory usage problem.

The Explanation of the events from IBM support after the analysis of documentation data was as follows:

                    (1). Job list of the Cobol job execution analysis

Cobol job JGL11N in the JES file execution JGL11N.txt is job executed after IPL and caused immediate high usage of the virtual memory, that is VSM and RSM resources. At the end of the preparation job step STEP4D used resources are:

VIRT 4K

SYS 324K

EXT 4K

SYS 11196K

At the end of the job step STEP4 running application program GLTRIBAN used resources are:

VIRT 336K

SYS 372K

EXT 2348K

SYS 11936K

Private region usage below 16MB and extended private region above 16MB is expected to be high because 7.535.031 records were processed.

                    (2). SVC dump analysis of the current situation when Cobol job is finished, but problem with slow responses remain

SVC dump analysis shows the following data from the dump:

Dump Title: LONG TSO RESPONSE -02.04.2013

Original dump data set: SYS1.BC2A.DMP00004

Sysplex name: BC2APLEX

System name: BC2A

ASIDs in captured SVC dump:

001 004D TSO

002 0025 JTRTDIBA

003 0027 JTRLOAN

004 0029 JTRCKIBA

005 002B RESDBCK0

006 002A RESDBCK1

SVC dump shows no resource contention, GRS queue could not cause delays.

The following steps bellow were executed to prepare the required SVC dump above.

The dump parameters entered from master console of LPAR BC2A are the following:

DUMP COMM=(Long TSO response – 02.04.2013)

R 9, JOBNAME=(TSO,JTRTDIBA,JTRLOAN,JTRCKIBA,RESDBCK0,RESDBCK1), SDATA= (ALLNUC,PSA,RGN,SQA,TRT,CSA,SUM),END

SVC dump (Long TSO response) is captured as SYS1.BC2A.DMP00004.

After data set is tersed with AMATERSE (TRSMAIN) (following the program instruction link http://techsupport.services.ibm.com/390/trsmain.html ) and saved as ‘PRINT01.SYS1.BC2A.DMP00004.CONDNS’ (condensed from 2974 trk to 547.5 = 750/73% trk changing recfm from VB to FB and lrecl from 32756 to 1024) this binary file is downloaded from MF to PC as binary file d:\ PRINT01.SYS1.BC2A.DMP00004.CONDNS (30049 KB).

This file ‘PRINT01.SYS1.BC2A.DMP00004.CONDNS’ was sent from PC to ECuRep using link http://www.ecurep.ibm.com/app/upload

and the confirmation received from the ECuRep team for the successful upload from ECuRep team was for File: 13097.644.644.PRINT01.SYS1.BC2A.DMP00004.CONDNS 30770176 bytes.

                    (3). RMF III VSAM data sets analysis since last day from the time of IPL until present time

The IBM support analyzes of last logs (RMF III data sets) showed that performances were good, as expected. Only critical time when performances were lower then expected is at 03/38 PM on 11/11/12 , and causer was job DUMPXY. Before and after this point in time performances were normal for all type of workload. System BC2A is highly utilized regarding the usage of the virtual memory. IBM could not relate problem with TSO logon to bad performances on system BC2A.To produce the required RMF III VSAM data sets the following was done:

Starting RMFGAT of rmf Monitor III since last day from the time of IPL to gather TSO delay monitoring information with

TSO/ISPF – Sd.log – /F RMF,START III

/Command MODIFY RMF,START III or /F RMF,START III or

MODIFY RMF,START III,CYCLE(2000),MINTIME(300),STOP(12H)

(2000 millisecond cycle, combine samples after a 300 second interval, and run for 12 hours)/

IEE252I MEMBER ERBRMF04 FOUND IN SYS1.PARMLIB

ERB115I START RMFGAT MONITOR III SESSION III

$HASP100 RMFGAT ON STCINRDR

IEF695I START RMFGAT WITH JOBNAME RMFGAT IS ASSIGNED TO USER RMFGAT

, GROUP OMVSGRP

$HASP373 RMFGAT STARTED

IEF403I RMFGAT – STARTED – TIME=15.18.10

ERB105I III: DATA GATHERER ACTIVE

ERB322I III: SMSVSAM SERVER IS CURRENTLY NOT AVAILABLE.

ERB100I III: ACTIVE

ERB425I III: UNABLE TO GATHER RESOURCE HSM

ERB821I III: 003 OUT OF 003 MONITOR III DATA SET(S) ARE USABLE

ERB813I III: ACTIVE MONITOR III DATA SET IS NOW ‘RMF.BC2A.MONIII.DS01’

ERB813I III: ‘

Stopping RMFGAT at present time to gather TSO delay monitoring information with command:

/F RMF,STOP III

ERB803I III: MONITOR III DATA SET SUPPORT TERMINATED

ERB102I III: TERMINATED

IEF404I RMFGAT – ENDED – TIME=15.26.04

IEF352I ADDRESS SPACE UNAVAILABLE

$HASP395 RMFGAT ENDED

To investigate the problem from IBM Support, the z/OS utility ERBV2S (converting RMF III VSAM data set ‘RMF.BC2A.MONIII.DS01’ to sequential file as was required) was used in TSO/ISPF – P.6 – with:

ERBV2S ‘RMF.BC2A.MONIII.DS01’ ‘ RMF.BC2A.MONIII.DS01.UNLOAD’

The new generated sequential file RMF.BC2A.MONIII.DS01.UNLOAD was with the following parameters:

Data Set Name . . . : RMF.BC2A.MONIII.DS01.UNLOAD

General Data Current Allocation

Volume serial . … .. : BC2MCC Allocated blocks . : 2,200

Device type . . ….. . : 3390 Allocated extents . : 9

Organization . …. . : PS

Record format .. . . : VB

Record length . . .. : 32756

Block size . . ……… : 32760 Current Utilization

1st extent blocks . : 250 Used blocks . . . . : 2,200

Secondary blocks . : 250 Used extents . . . : 9

Creation date . . . : 2012/11/12

Referenced date . . : 2012/11/12

Expiration date . . : ***None***

Before sending this enormous sequential data set (2,200 tracks) to the IBM service (ECuRep) the data set size was reduced by using program AMATERSE (TRSMAIN) following the program instruction link http://techsupport.services.ibm.com/390/trsmain.html .

The source JCL library member TRSMAINJ in library BCA1MGP.STEF.JCL2 was submitted to condense RMF.BC2A.MONIII.DS01.UNLOAD into RMF.BC2A.MONIII.DS01.CONDNS as follows:

//TRSMAINJ JOB MSGLEVEL=(1,1),MSGCLASS=X,TIME=1440

//***

//* PTFLCG.TERSE409.LOADLIB

//STEPNAME EXEC PGM=TRSMAIN,PARM=’SPACK’

//STEPLIB DD DISP=SHR,DSN=PTFLCG.TERSE409.LOADLIB

//SYSPRINT DD SYSOUT=*,DCB=(LRECL=133,BLKSIZE=12901,RECFM=FBA)

//INFILE DD DISP=OLD,DSN=RMF.BC2A.MONIII.DS01.UNLOAD

//OUTFILE DD DCB=RMF.BC2A.MONIII.DS01.UNLOAD,

DSN=RMF.BC2A.MONIII.DS01.CONDNS,

// SPACE=(BLK,(250,250)),DISP=(NEW,CATLG,DELETE)

/*

After data set was tersed as rmf.bc2a.moniii.ds01.condns (condensed from 2200/100% trk to 500/85% = 425 trk changing recfm from VB to FB and lrecl from 32756 to 1024) this binary file was downloaded from MF to PC file as d:\ rmf.bc2a.moniii.ds01.condns (20471 KB) with the following steps:

From TSO/ISPF – p.6 – Actions – Receive File From Host …

Host file: rmf.bc2a.moniii.ds01.condns

PC File: d:\ rmf.bc2a.moniii.ds01.condns

Transfer Type: binary

Options: MVS/TSO – Transfer Type: binary – Record Format: Fixed – Logical Record Length: 1024

The binary condensed file rmf.bc2a.moniii.ds01.condns was received from MF on PC as d:\ rmf.bc2a.moniii.ds01.condns (20471 KB) and was sent to ECuRep using link

http://www.ecurep.ibm.com/app/upload

with the following parameters:

PMR number:* 13030,644,644 (1330 – PMR /Problem Management Record number – given by IBM/, 644 – Branch office number, 644 – IBM/BG country code)

Upload is for:* MVS

Email address: xxxxxxxxxx

Continue – press it – Browse to find d:\rmf.bc2a.moniii.ds01.condns – Submit

The confirmation is received from ECuRep team for the successful upload of file 13030.644.644.rmf.bc2a.moniii.ds01.condns 20471 KB.

                    (4). Syslog and LOGREC data sets analysis

LOGDATA shows several errors one day before POR (IPL) was executed, and software errors are not cause for the time delay of TSO users.

TSO logon time delay Conclusion:

As the problem with TSO logon time delay could not relate to bad tuned performance of z/OS production system (which is balanced as online and short batch jobs priority), the only possible action was to do IPL for LPAR BC2A every time to resume the system. After that all z/OS sub systems have started in a normal way and no TSO logon time delay existed until the next Cobol program execution delayed the system again.

The next step to find solution of the problem was an attempt (successful) to minimize the number of LPARs from 3 to 1 and to assign the all machine restricted power (3 MSU) to only the LPAR BC2A with Cobol program running. The result was that the TSO logon delay continued in the same slow way during Cobol program execution, FTP file transfer and Restore of DB VSAM files from archive tapes, but after finishing the above activities the normal state of TSO logon was returned back as usual without necessity to do a new IPL.

The final conclusion was that the minimal power of LPAR BC2A (1 MSU) was not enough when the Cobol program execution was running in it with enormous data to be processed in parallel with the two other LPARs (each with 1 MSU).

The practical solution was to plan and perform the above activities (Cobol program execution, FTP, Restore) in appropriate sequence to avoid the overload of the Mainframe with increased but yet restricted power (3 MSU).

          2. TSO system wait

The following TSO system wait troubles may occur::

– If during TSO Logon (after signing with ID and PW) the system is pending (in system) and no one can logon to TSO, all TSO terminal sessions remain in logon status and if repeated Logon the status stay pending the same. To avoid from this system wait it is necessary to cancel any task that has higher priority of TSO and which is the cause of TSO system wait (usually this is wait in the running CICS application system bc2ac10a and it is necessary to cancel bc2ac10a online system from CPU console with: c bc2ac10a).

– Another system wait (no TSO logon is possible) may arise when Jes2 queue is full . Then it is necessary to purge (with P command using CPU main console) some job members of JES2 queue to release the system wait as follows:

$D J(RESDB704,JOB05932) or $D J(RESDB70*) – display jobs eligible to delete/purge

$P J(RESDB704,JOB05932) or $P J(RESDB70*) – delete/purge job,s

TSO/ISPF – sd.st – //p ….// – job group purged in TSO/ISPF when TSO running

– Similar system wait (no TSO logon is possible) may arise if  HC file is full due to many WTO/R messages (issued by some wrong task and not deleted). The solution is to issue the command k q,r=hc from CPU console to clear the HC file and to release the system wait.

– Sometimes during FTP file transfer from CPU to PC (PC Get function initiated from PC by TSO logon), the transfer is delayed and not successful but aborted with following messages:

> .\Tmp4D1.tmp:No CSI structure available

451 Transfer aborted: send error.

This is due to slow response time of CPU with not enough power. You need to repeat the FTP file transfer and the next time usually it completes successful.

If the FTP/TSO file transfer is delayed to much and you wish to stop it, from PC side enter Ctrl + C and you’ll receive:

Aborting any active data connections…

Connection closed by remote host.

To start again the same FTP file transfer get function from PC, do the following:

ftp> bye

L:\05.2004>ftp 10.212.4.90

Connected to 10.212.4.90.

220-FTPD1 IBM FTP CS V1R4 at BC2AJES, 11:12:57 on 2013-07-22.

220 Connection will close if idle for more than 5 minutes.

User (10.212.4.90:(none)): bca1mgp

331 Send password please.

Password:

230 BCA1MGP is logged on. Working directory is “BCA1MGP.”.

ftp> get ‘PRINT01.KCI01.CK05045.TXT’ CK05045.TXT

End

Leave A Reply

Comments

No comments yet, be the first to add one!

Places

 

Clef two-factor authentication

woolrich outlet woolrich outlet online tiffany milano tiffany outlet tiffany outlet tiffany outlet tiffany outlet tiffany milano tiffany outlet peuterey outlet peuterey uomo hogan outlet hogan rebel hogan outlet hogan rebel hogan outlet hogan rebel hogan outlet hogan rebel hogan outlet hogan rebel hogan outlet hogan rebel roshe run pas cher roshe run parajumpers pas cher parajumpers soldes parajumpers pas cher parajumpers soldes parajumpers pas cher parajumpers soldes louboutin pas cher louboutin pas cher louboutin pas cher nike tn pas cher nike tn pas cher