Reserving TCP/IP Ports on z/OS for use by ITM
If you are using the TCP/IP protocol (defined in ITM as IP.PIPE, IP6.PIPE or IP6.SPIPE) as a transport protocol between your ITM components on z/OS then you may need to consider configuring TCP/IP to reserve specific ports for use by the ITM components in order to prevent other non ITM address spaces from acquiring them.
ITM TCP/IP port allocation algorithm
ITM and ITM agents use the following rules to allocate TCP/IP ports
- The Hub or Remote TEMS always uses the well known port, typically 1918.
- The agents then attempt to acquire ports in sequence using the algorithm well known port + (n*4096) until they either obtain a port or they run out of ports to try. The actual starting port and number of attempts can be controlled by the SKIP and COUNT parameters as described later.
For example, if the well known port assigned to the hub or remote TEMS is 1918, the first agent to start will attempt to obtain port 1918+(1*4096) or 6014. If that fails, it will attempt to obtain port 1918+(2*4096) or 10110 and so on.
The actual port assigned to any given agent for any given execution will vary based on the startup order of the agents and various other timing related factors but will always follow the above pattern.
Thus you can predict the port numbers that will be required by the ITM infrastructure on a given LPAR.
Using 1918 as a starting point, the following port numbers are potential candidates for use by ITM:
1918 – Always assigned to the hub or remote TEMS
Notice that this means you can have a maximum of one hub or remote TEMS and 15 agents on an LPAR.
What are the potential problems?
Unless you take specific action by configuring TCP/IP, there is nothing to prevent any other non ITM application on a z/OS LPAR from using any of these ports. Thus it is entirely possible that enough of these ports are in use by other applications that there are not enough for all the ITM agents, resulting in connectivity issues.
This type of problem can be difficult to diagnose since it may only occur randomly as it depends on the the unavailability of specific ports, and that will depend on which other application are running and the ports they have acquired.
What ports do I need to reserve?
If you are running for example, a remote TEMS and four agents on a z/OS LPAR then the environment will need as a minimum, the well known port plus four additional ports. The four additional ports do NOT have to be consecutive (using the list above). In the absence of the SKIP and COUNT parameters, each agent will try specific ports from the list (assuming 1918 as the base well known port) until it obtains a port. So you could quite validly reserve for example, ports 34686, 38782, 42878, 46974 for use by the four agent address spaces. In that case you might want to use the SKIP parameter to prevent agents from attempting to bind to the the first 7 ports in the list.
Remember, these examples are based on 1918 being the well known port. The actual values in use will change if you use a different port number for the well known port.
Reserving port in TCP/IP
The PROFILE DD of the TCP/IP started task JCL points to a dataset, or more typically a member of a dataset, that configures the TCP/IP environment. Within this configuration dataset or member you can use entries within the PORT statement to restrict specific ports to specific address spaces.
The port section might look like this:
7 TCP MISCSERV
7 UDP MISCSERV
9 TCP MISCSERV
9 UDP MISCSERV
19 TCP MISCSERV
19 UDP MISCSERV
20 TCP OMVS
Where the first value of each line is the port number, the second value is the protocol and the third value is the name of the address space that the port is limited to.
The critical port to reserve for ITM is the well known port which is typically 1918. So if your TEMS address space is named CANSDSST and the well known port is 1918 then you could add the following entry to the TCP/IP configuration deck PORTS section
1918 TCP CANSSDDT
This would prevent any other address space from obtaining port 1918.
Do I need to reserve ports for the agents?
The way that the allocation algorithm works allows you to have up to 15 agents running on an LPAR. If you only have a few agents on an LPAR then even if there are some ports from the potential list in use by non ITM applications, there are probably enough free ports from the list of potential ports to satisfy the needs of the agents.
However, if you have a lot of agents on an LPAR or if you want to guarantee that specific ports are available to the ITM agent address spaces then you may need to reserve specific ports to specific agents address spaces in order to ensure that they are available for the agent address spaces.
There are a couple of ways to do this:
- Configure TCP/IP to allow any agent to connect to any of the potential ports
- Configure each ITM agent to use a specific port and limit each port to a specific address space in the TCP/IP profile.
Configure TCP/IP to allow any agent to connect to any of the potential ports
The advantage of this method it that it only requires TCP/IP configuration and allows ITM to continue to dynamically assign ports to each agent based on availability and startup sequence. For example, if you had three agent started tasks CANSCICS, CANSMQ and CANSMFN, you would need to configure TCPIP to reserve all the potential ports for each address space as follows:
6014 TCP CANSCICS
6014 TCP CANSMQ
6014 TCP CANSMFN
10110 TCP CANSCICS
10110 TCP CANSMQ
10110 TCP CANSMFN
14206 TCP CANSCICS
14206 TCP CANSMQ
14206 TCP CANSMFN
etc all the way up to port 63358
Obviously as you add more agent address spaces you have to add more entries for each port.
Configure each ITM agent to use a specific port and limit each port to a specific address space in the TCP/IP profile.
This method uses a combination of ITM and TCP/IP configuration options to achieve the desired result. The advantage of this method is that the port assigned to each agent address space is predictable.
First, lets configure TCP/IP to reserve a single specific port for each agent address space:
6014 TCP CANSCICS
10110 TCP CANSMQ
14206 TCP CANSMFN
Now you need to configure each agent to only use a specific port. To do this you need to edit the KDE_TRANSPORT IP.PIPE (and/or IP6.PIPE or IP6.SPIPE if in use) entry of each agent’s KppENV member in RKANPARU as follows:
For the CICS agent address space CANSCICS:
KDE_TRANSPORT=IP.PIPE PORT:1918 USE:Y COUNT:1
For the MQ agent address space CANSMQ:
KDE_TRANSPORT=IP.PIPE PORT:1918 USE:Y COUNT:1 SKIP:1
For the MFN gent address space CANSMFN:
KDE_TRANSPORT=IP.PIPE PORT:1918 USE:Y COUNT:1 SKIP:2
The COUNT:1 parameter tells the agent address space to only try 1 port number from the potential list or ports.
The SKIP parameter tells the agent address space to skip that number of ports in the available list before trying to bind to the port. It is not required for the CICS agent (in this example) because the CICS agent address space will bind to the first port in the list.
Remember, these examples are based on the well know port number being 1918.
High Availability Hubs
A high availability hub is a hub TEMS address space that uses a DVIPA IP address to enable it to run on any candidate LPAR (one with the DVIPA address configured) within the sysplex. The location of the IP address moves with the hub address space so that as it moves from LPAR to LPAR, the agents and remote TEMS can connect to it, no matter where it is.
A hub TEMS uses the same well known port number as the rest of the ITM infrastructure (remote TEMS), however because a high availability hub connects to a different IP address (the DVIPA address) from that defined for the host LPAR, the high availability hub can execute on the same LPAR as a remote TEMS, even though both are using the same well known port.
However, if you have reserved the well known port on each LPAR for a remote TEMS address space, you must also reserve the same address for the high availability hub address space name.
So, in the PORT section of the TCPIP profile dataset or member you might have the following entry to reserve the well known port 1918 for the remote TEMS:
1918 TCP CANSSDDT
To this you would need to add (assuming the high availability hub address space name is CANSHAHB):
1918 TCP CANSHAHB
Both address spaces can bind to the same port because the high availability hub is using the DVIPA IP address.
Refreshing the TCPIP configuration
After making changes to the TCPIP profile dataset or member you can use the OBEYFILE command to cause the TCPIP address space on an LPAR to reload its configuration file.
Listing reserved ports
You can list the currently configured TCP/IP reserved ports using the TSO NETSTART PORTLIST command.
In TSO, issue the command:
NETSTAT PORTLIST [REP DSN dsn]
The REP DSN dsn option causes the command to write the output the to the dataset specified by the dsn operand.
Remember. Just because a port or address space name is defined in the TCPIP profile dataset or member does not mean it is actually defined to TCP/IP. You must restart TCPIP or issue the OBEYFILE command to refresh the TCPIP configuration.
Diagnosing Port Permissions Problems
In the agent or RTEMS RKLVLOG output you may see this sort of message:
(0017-D69321E3:kdebbbi.c,128,”KDEB_BaseBind”) Status 1DE00000=KDE1_STC_CANTBIND=2: EACCES
(0018-D69321E3:kdebbbi.c,132,”KDEB_BaseBind”) <0x29615378,0x10> bind: ASD 289FD7A0, status 1DE00000, errno 2
+0018 00000000 00022774 092A2E16 00000000 00000000 …………….
- KDE1_STC_CANTBIND indicates a bind failure.
- EACCES indicates the bind failed because of a permissions issue. The address space is not authorized to bind to the port.
- The port number (in hex) is the lower four digits of the word indicated in the message above. In this case 2774 (hex) is port 10100 decimal.
Use the NETSTAT PORTLIST command to determine if the address space is authorized to access the port, for example:
NETSTAT PORTLIST (PORT 10100
The (PORT portnumber operand limits the output to just the requested port.
A note about EACCES issues.
My research indicates that an EACCES return code is returned when the application first attempts to use a port, NOT when it is allocated. What this means for ITM is that the port allocation algorithm may select a free port that is actually restricted by TCP/IP. However ITM will not find out about this until it tries to bind to the port. At that point the port allocation process is complete so you end up with an assigned port that you cannot use.
In this instance I believe the only solution is to use the COUNT and SKIP parameters to force agents to only use ports in specific ranges and to avoid any reserved port ranges. You don’t have to assign a specific port to each agent, you can still let them pick from a range based on first available but if you have for example, a range of reserved ports in the middle of the normally free range of ports, you may need to configure some agents to only use ports below that reserved range and others to use the ports above the range.
The actual configuration you need to use to make the agents avoid the reserved range or ranges will depend on the number of agents you are running on the system and where in the list of normally free ports, the reserved range or ranges are.