How can I configure our SMTP server so that it will not be used as an "open mail relay"?

You should set INBOUNDOPENLIMIT to 0 in your SMTP configuration file, and then recycle the SMTP server started task. This causes SMTP to not open a listening socket on port 25. Nobody can connect into the SMTP server and use it as a relay.

INBOUNDOPENLIMIT specifies the maximum number of simultaneous TCP connections over which the SMTP server receives mail. When 0 is coded for INBOUNDOPENLIMIT, the SMTP server does not open a listening connection.

Note that if you set INBOUNDOPENLIMIT to 0, you cannot use AUTOLOG to monitor that SMTP is up. If you use AUTOLOG to start SMTP, you need to specify NOAUTOLOG on the TCP/IP PROFILE PORT statement for SMTP.

In z/OS Communications Server: IP Configuration Reference, the NOAUTOLOG parameter on the PORT statement is described as follows:

NOAUTOLOG

Tells the TCP/IP address space not to restart the server if it was stopped previously. Otherwise, the default is to restart the server if it was stopped previously. If the application associated with the job name is an AUTOLOG started procedure, and the port is inactive (for TCP connections, the procedure must have a socket open to that port in the LISTEN state; for UDP connections, the procedure must have a socket open to that port), then AUTOLOG assumes that the procedure is hung; it cancels and restarts it every five minutes. Use NOAUTOLOG to prevent this from occurring.