Overview

Skill Level: Intermediate

Hands on expertise with Ansible Tower/AWX and ssh ProxyCommand

Part 2 shows the ssh commands, python code, Ansible playbooks and finally the configuration of Ansible Tower to run jobs on Windows/Linux hosts over multiple hops of jumphosts by creating ssh tunnel SOCKS5 proxy.

Ingredients

Ansible Tower/AWX, python

Step-by-step

  1. Introduction

    Unlike Linux/Unix hosts, which use ssh by default, Windows hosts are configured with WinRM. WinRM is a management protocol used by Windows to remotely communicate with another server. It is a SOAP-based protocol that communicates over HTTP/HTTPS, and is included in all recent Windows operating systems. We will route this traffic over SOCKS that stands for SOCKet Secure. It is assumed that the Windows host endpoints are already setup with WinRM configured. It is easiest to run the powershell script that should be executed on the host endpoint so that Ansible Tower can connect to it. Although we can connect to Windows hosts with ssh too with the ssh server installed on the Windows host, we will connect using WinRM for the rest of this article.

    A SOCKS5 proxy at layer 5 of the Open Systems Interconnection (OSI) model uses a tunneling method allowing public cloud users to access resources behind the firewall using SOCKS5 over a secured tunnel such as SSH. It supports traffic generated by protocols, such as HTTP, SMTP and FTP. SOCKS5 doesn’t care about anything below layer 5 in the OSI model. You can’t use it to tunnel things such as ping, Address Resolution Protocol (ARP), etc.

    The connection from Ansible to host consists of the following parts:

    1. Ansible to the SOCKS listener (the port is configured with ssh parameter -D “dynamic” application-level port forwarding)
    2. SOCKS listener to the SSH client (currently the SOCKS4 and SOCKS5 protocols are supported, and ssh will act as a SOCKS server)
    3. SSH multi jumphost channel (with nested ProxyCommand)
    4. SSH Server to the WinRM listener (Windows hosts listen on port 5985 http or 5986 https). The data contains the WinRM payload encapsulated in a SOCKS packet. It can also encapsulate data to connect to Linux hosts (default port 22) or http request.

     

    Ansible uses the pywinrm package to communicate with Windows servers over WinRM. It is not installed by default with the Ansible package. Run the “pip install pywinrm” to install it. When connecting to a Windows host, there are several different options that can be used when authenticating with an account. The authentication type may be set on inventory hosts or groups with the ansible_winrm_transport variable. The requests-credssp wrapper can be installed using “pip install pywinrm[credssp]”. Also install the additional python dependencies “pip install pypsrp” and “pip install pysocks” and “pip install requests-credssp”. Your Ansible Tower needs these python module dependencies installed.

     

  2. Creating the tunnel

    The parameters used in the creation of the tunnel are: -C is to compress the data that goes over this channel, -f is to run SSH in the background after the connection is setup, -N means do not execute a remote command, -q is quiet mode, -D is for the bind address and port for the SOCKS proxy.

    For illustration purpose, the bind port on localhost when using Ansible (and Ansible Tower celery/task container) used in this article is 1234 (or 1235). The six VMs used as jumphosts are ec2-52-201-237-93.compute-1.amazonaws.com, aakrhel001, aakrhel002, aakrhel003, aakrhel005:2222 and aakrhel006. Only the aakrhel005 ssh server listens on port 2222. The rest of the VMs listen on port 22. It is assumed that the private keys have been copied to the respective jumphosts and each jumphost has connectivity to the next jumphost with the public key of the prior host added to the next host in the user’s ~/.ssh/authorized_keys.

    The following shows the full ssh commands to create the tunnels from 1 to 6 jumphost hops. There is lot of escaping (number of backslashes double for every hop for the innermost ProxyCommand) for the multiple nested ProxyCommand parameters to ssh. These are essentially the commands that will be created and run using a role when using Ansible Tower. Only the command with number of jumphosts relevant to your environment should be used.

     

    Tunnel over 1 hop

    Laptop port 1234-> ec2-52-201-237-93.compute-1.amazonaws.com ->endpoint

    ssh -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -o ExitOnForwardFailure=yes -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -CfNq -D 127.0.0.1:1234 -p 22 -i ~/amazontestkey.pem ec2-user@ec2-52-201-237-93.compute-1.amazonaws.com -v

     

    Tunnel over 2 hops

    Laptop port 1234-> ec2-52-201-237-93.compute-1.amazonaws.com ->aakrhel001 -> endpoint

    ssh -CfNq -D 127.0.0.1:1234 -v -i ~/amazontestkey.pem -o ProxyCommand='ssh -v -i ~/amazontestkey.pem -W aakrhel001.yellowykt.com:22 ec2-user@ec2-52-201-237-93.compute-1.amazonaws.com' ec2-user@aakrhel001.yellowykt.com

     

    Tunnel over 3 hops

    Laptop port 1234-> ec2-52-201-237-93.compute-1.amazonaws.com ->aakrhel001 ->aakrhel002->endpoint

    ssh -CfNq -D 127.0.0.1:1234 -v -i ~/amazontestkey.pem -o ProxyCommand="ssh -v -i ~/amazontestkey.pem -W aakrhel003.yellowykt.com:22 ec2-user@aakrhel001.yellowykt.com -o ProxyCommand='ssh -v -i ~/amazontestkey.pem -W aakrhel001.yellowykt.com:22 ec2-user@ec2-52-201-237-93.compute-1.amazonaws.com'" ec2-user@aakrhel002.yellowykt.com

     

    Tunnel over 4 hops

    Laptop port 1234-> ec2-52-201-237-93.compute-1.amazonaws.com ->aakrhel001->aakrhel002->aakrhel003->endpoint

    ssh -CfNq -D 127.0.0.1:1234 -v -i ~/amazontestkey.pem -o ProxyCommand="ssh -v -i ~/amazontestkey.pem -W %h:%p ec2-user@aakrhel002.yellowykt.com -o ProxyCommand=\"ssh -v -i ~/amazontestkey.pem -W aakrhel002.yellowykt.com:22 ec2-user@aakrhel001.yellowykt.com -o ProxyCommand=\\\"ssh -v -i ~/amazontestkey.pem -W aakrhel001.yellowykt.com:22 ec2-user@ec2-52-201-237-93.compute-1.amazonaws.com\\\"\"" ec2-user@aakrhel003.yellowykt.com

     

    Tunnel over 5 hops

    Laptop port 1234-> ec2-52-201-237-93.compute-1.amazonaws.com ->aakrhel001->aakrhel002->aakrhel003->aakrhel005:2222->endpoint

    ssh -CfNq -D 127.0.0.1:1234 -v -i ~/amazontestkey.pem -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oProxyCommand="ssh -v -i ~/amazontestkey.pem -W aakrhel005.yellowykt.com:2222 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oProxyCommand=\"ssh -v -i ~/amazontestkey.pem -W aakrhel003.yellowykt.com:22 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oProxyCommand=\\\"ssh -v -i ~/amazontestkey.pem -W aakrhel002.yellowykt.com:22 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oProxyCommand=\\\\\\\"ssh -v -i ~/amazontestkey.pem -W aakrhel001.yellowykt.com:22 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null ec2-user@ec2-52-201-237-93.compute-1.amazonaws.com\\\\\\\" ec2-user@aakrhel001.yellowykt.com\\\" ec2-user@aakrhel002.yellowykt.com\" ec2-user@aakrhel003.yellowykt.com" ec2-user@aakrhel005.yellowykt.com -p 2222

     

    Tunnel over 6 hops

    Laptop port 1234-> ec2-52-201-237-93.compute-1.amazonaws.com ->aakrhel001->aakrhel002->aakrhel003->aakrhel005:2222->aakrhel006->endpoint

    ssh -CfNq -D 127.0.0.1:1234 -v -i ~/amazontestkey.pem -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oProxyCommand="ssh -v -i ~/amazontestkey.pem -W %h:%p -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oProxyCommand=\"ssh -v -i ~/amazontestkey.pem -W aakrhel005.yellowykt.com:2222 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oProxyCommand=\\\"ssh -v -i ~/amazontestkey.pem -W aakrhel003.yellowykt.com:22 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oProxyCommand=\\\\\\\"ssh -v -i ~/amazontestkey.pem -W aakrhel002.yellowykt.com:22 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oProxyCommand=\\\\\\\\\\\\\\\"ssh -v -i ~/amazontestkey.pem -W aakrhel001.yellowykt.com:22 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null ec2-user@ec2-52-201-237-93.compute-1.amazonaws.com\\\\\\\\\\\\\\\" ec2-user@aakrhel001.yellowykt.com\\\\\\\" ec2-user@aakrhel002.yellowykt.com\\\" ec2-user@aakrhel003.yellowykt.com\" ec2-user@aakrhel005.yellowykt.com -p 2222" ec2-user@aakrhel006.yellowykt.com

     

    Autossh

    We can use the autossh to start the tunnel. Autossh is a program to start a copy of ssh and monitor it, restarting it as necessary should it die or stop passing traffic. The command for single jumphost (ec2-52-201-237-93.compute-1.amazonaws.com) is shown below:

    autossh -M 0 -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3" -o ExitOnForwardFailure=yes -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o "ControlMaster=auto" -o "ControlPersist=no" -o "ControlPath=/tmp/ssh-%r@%h:%p" -CfNq -D 127.0.0.1:1234 -p 22 -i ~/amazontestkey.pem ec2-user@ec2-52-201-237-93.compute-1.amazonaws.com -v

    Make sure that ControlPath folder is already present, otherwise the tunnel will not get created. The -M 0 is to disable the built-in AutoSSH monitoring port by giving it a value of 0. If ssh connection times out (server-side timeout), the tunnel should be re-established automatically.

     

    The following sections show commands to use these tunnels to connect to Windows/Linux hosts and as a http proxy. Since the tunnels created above all bind to localhost on port 1234, the commands below use socks5h://127.0.0.1:1234. In a proxy string, socks5h:// means that the hostname is resolved by the SOCKS server. Without the “h” socks5:// means that the hostname is resolved locally.

  3. Connecting to Windows target host from Linux/Mac

    After the tunnel is established, we can verify the connectivity to Windows using multiple mechanisms as follows:

    1a. Using curl request to verify response from wsman service

    password="<mysecretpassword>" # The password for the Windows local Administrator

    curl -vvv --proxy socks5h://127.0.0.1:1234 --header "Content-Type: application/soap+xml;charset=UTF-8" http://aakwin2012-1.yellowykt.com:5985/wsman --basic -u Administrator:$password --data '<s:Envelope xmlns:s="http://www.w3.org/2003/05/soap-envelope" xmlns:wsmid="http://schemas.dmtf.org/wbem/wsman/identity/1/wsmanidentity.xsd"><s:Header/><s:Body><wsmid:Identify/></s:Body></s:Envelope>'

    Proxy protocol prefix “socks5h://” makes it the equivalent of –socks5-hostname. It uses the specified SOCKS5 proxy and lets the proxy resolve the host name. The output of the command shows an HTTP response code of 200 for successful connection.

    *   Trying 127.0.0.1...
    * TCP_NODELAY set
    debug1: Connection to port 1234 forwarding to socks port 0 requested.
    * SOCKS5 communication to aakwin2012-1.yellowykt.com:5985
    debug1: channel 1: new [dynamic-tcpip]* SOCKS5 request granted.
    * Connected to 127.0.0.1 (127.0.0.1) port 1234 (#0)
    * Server auth using Basic with user 'Administrator'
    > POST /wsman HTTP/1.1
    > Host: aakwin2012-1.yellowykt.com:5985
    > Authorization: Basic xxxxxxxxxxxxxxxxx
    > User-Agent: curl/7.64.1
    > Accept: */*
    > Content-Type: application/soap+xml;charset=UTF-8
    > Content-Length: 198

    * upload completely sent off: 198 out of 198 bytes
    < HTTP/1.1 200
    < Content-Type: application/soap+xml;charset=UTF-8
    < Server: Microsoft-HTTPAPI/2.0
    < Date: Mon, 29 Jun 2020 14:24:07 GMT
    < Content-Length: 787

    * Connection #0 to host 127.0.0.1 left intact
    <s:Envelope xml:lang="en-US" xmlns:s="http://www.w3.org/2003/05/soap-envelope"><s:Header></s:Header><s:Body><wsmid:IdentifyResponse xmlns:wsmid="http://schemas.dmtf.org/wbem/wsman/identity/1/wsmanidentity.xsd"><wsmid:ProtocolVersion>http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd</wsmid:ProtocolVersion><wsmid:ProductVendor>Microsoft Corporation</wsmid:ProductVendor><wsmid:ProductVersion>OS: 6.3.9600 SP: 0.0 Stack: 3.0</wsmid:ProductVersion><wsmid:SecurityProfiles><wsmid:SecurityProfileName>http://schemas.dmtf.org/wbem/wsman/1/wsman/secprofile/http/basic</wsmid:SecurityProfileName><wsmid:SecurityProfileName>http://schemas.dmtf.org/wbem/wsman/1/wsman/secprofile/http/spnego-kerberos</wsmid:SecurityProfileName></wsmid:SecurityProfiles></wsmid:IdentifyResponse></s:Body></s:Envelope>* Closing connection 0
    debug1: channel 1: free: direct-tcpip: listening port 1234 for aakwin2012-1.yellowykt.com port 5985, connect from 127.0.0.1 port 58982 to 127.0.0.1 port 1234, nchannels 2

     

    1b. Testing Windows connectivity using python to run a Windows cmd

    The following output shows how to run a command on the Windows host using the tunnel using the windowstest_with_tunnel.py

    python windowstest_with_tunnel.py –host aakwin2012-1.yellowykt.com –port 5985 –username=Administrator –password $password

    0
     
    Windows IP Configuration
     
       Host Name . . . . . . . . . . . . : aakwin2012-1
       Primary Dns Suffix  . . . . . . . : yellowykt.com
       Node Type . . . . . . . . . . . . : Hybrid
       IP Routing Enabled. . . . . . . . : No
       WINS Proxy Enabled. . . . . . . . : No
       DNS Suffix Search List. . . . . . : yellowykt.com

    Ethernet adapter Ethernet0:

    Source code for windowstest_with_tunnel.py

    from winrm.protocol import Protocol
    import argparse
     
    parser = argparse.ArgumentParser(description='Run command on Windows host')
    parser.add_argument("--host", required=True)
    parser.add_argument("--port", default=5985)
    parser.add_argument("--socksport", default=1234)
    parser.add_argument("--username", required=True)
    parser.add_argument("--password", required=True)
    parser.add_argument("--protocol", default="http")
    parser.add_argument("--transport", default="basic")
     
    args = parser.parse_args()
     
    p = Protocol(
        endpoint=args.protocol+'://'+args.host+':'+str(args.port)+'/wsman',
        transport=args.transport,
        username=args.username,
        password=args.password,
        server_cert_validation='ignore',
        proxy='socks5h://localhost:'+str(args.socksport))
    shell_id = p.open_shell()
    command_id = p.run_command(shell_id, 'ipconfig', ['/all'])
    std_out, std_err, status_code = p.get_command_output(shell_id, command_id)
    p.cleanup_command(shell_id, command_id)
    p.close_shell(shell_id)
    print(status_code)
    print(std_out)
    print(std_err)

     

    1c. Testing Windows connectivity using python to run a powershell on Windows host

    The following output shows how to run a command on the Windows host using the tunnel using the windowstest_with_tunnel2.py. Powershell scripts will be base64 UTF16 little-endian encoded prior to sending to the Windows host and run with “powershell -EncodedCommand”. The results are decoded to ascii.

    Invoking the powershell from python:

    python windowstest_with_tunnel2.py --host aakwin2012-1.yellowykt.com --port 5985 --username=Administrator --password $password

    Output from python code python windowstest_with_tunnel2.py:

    ('status_code:', 0)
    std_out:
    Installed Memory: 4095 MB

    std_err:
    #< CLIXML
    <Objs Version="1.1.0.1" xmlns="http://schemas.microsoft.com/powershell/2004/04"><Obj S="progress" RefId="0"><TN RefId="0"><T>System.Management.Automation.PSCustomObject</T><T>System.Object</T></TN><MS><I64 N="SourceId">1</I64><PR N="Record"><AV>Preparing modules for first use.</AV><AI>0</AI><Nil /><PI>-1</PI><PC>-1</PC><T>Completed</T><SR>-1</SR><SD> </SD></PR></MS></Obj></Objs>

     

    Source code for windowstest_with_tunnel2.py

    from winrm.protocol import Protocol
    import winrm
    import argparse
    from base64 import b64encode
     
    script = """$strComputer = $Host
    Clear
    $RAM = WmiObject Win32_ComputerSystem
    $MB = 1048576
     
    "Installed Memory: " + [int]($RAM.TotalPhysicalMemory /$MB) + " MB" """
     cmd = """
     
    # Load script from env-vars
    . ([ScriptBlock]::Create($Env:WINRM_SCRIPT))
    """
    encoded_cmd = b64encode(cmd.encode('utf_16_le')).decode('ascii')
     
    parser = argparse.ArgumentParser(description='Run command on Windows host')
    parser.add_argument("--host", required=True)
    parser.add_argument("--port", default=5985)
    parser.add_argument("--socksport", default=1234)
    parser.add_argument("--username", required=True)
    parser.add_argument("--password", required=True)
    parser.add_argument("--protocol", default="http")
    parser.add_argument("--transport", default="basic")

    args = parser.parse_args()
     
    p = Protocol(
        endpoint=args.protocol+'://'+args.host+':'+str(args.port)+'/wsman',
        transport=args.transport,
        username=args.username,
        password=args.password,
        server_cert_validation='ignore',
        proxy='socks5h://localhost:'+str(args.socksport))
    # Load script to env vars.
    shell_id = p.open_shell(env_vars=dict(WINRM_SCRIPT=script))
    command_id = p.run_command(shell_id, "powershell -EncodedCommand {}".format(encoded_cmd))
     
    #rs = winrm.Response(p.get_command_output(shell, command)
    #print(str(rs.std_out, 'ascii'))
    std_out, std_err, status_code = p.get_command_output(shell_id, command_id)
    p.cleanup_command(shell_id, command_id)
    p.close_shell(shell_id)
    print("status_code:",status_code)
    print("std_out:")
    #print(str(std_out, 'ascii')) # python3 only
    print(str(std_out).encode('ascii'))
    print("std_err:")
    #print(str(std_err, 'ascii')) # python3 only
    print(str(std_err).encode('ascii'))

  4. Connecting to Linux target host over the tunnel

    Part 1 showed how to connect via multiple jumphosts to Linux host endpoints using the nested ProxyCommand with ssh. This section now shows how to use the pre-established ssh tunnel that was created above with required number of jumphosts to invoke commands on the Linux host endpoints.

    2a. Connecting to Linux target host from Mac using nc

    The /usr/local/bin/nc will not work with -X. Instead, you can install the nc with “brew install netcat”. Then you can use the /usr/bin/nc with -X. The command to connect to remote linux host aakrhel005 on port 2222 using the localhost socks bind port 1234 is as follows:

    ssh ec2-user@aakrhel005.yellowykt.com -p 2222 -i ~/amazontestkey.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -vvv -o ProxyCommand="/usr/bin/nc -X 5 -x 127.0.0.1:1234 %h %p" echo Hello \`hostname\`

    Hello aakrhel005

    2b. Connect to Linux target host from Linux using connect-proxy

    On CentOS7/RHEL7 Linux, nc does not work. Instead you need to use the “connect-proxy -S” or “ncat –proxy-type socks5” or “socat – socks4a”. You can install connect-proxy with “yum -y install connect-proxy”.

    ssh ec2-user@aakrhel005.yellowykt.com -p 2222 -i ~/amazontestkey.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ProxyCommand="connect-proxy -S 127.0.0.1:1234 %h %p" echo Hello \`hostname\`

    Hello aakrhel005

    2c. Connect to Linux target host from Linux using ncat

    ssh ec2-user@aakrhel005.yellowykt.com -p 2222 -i ~/amazontestkey.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ProxyCommand="ncat --proxy-type socks5 --proxy 127.0.0.1:1234 %h %p" echo Hello \`hostname\`

    Hello aakrhel005

    2d. Connect to Linux target host from Linux using socat

    ssh ec2-user@aakrhel005.yellowykt.com -p 2222 -i ~/amazontestkey.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ProxyCommand="socat - socks4a:127.0.0.1:%h:%p,socksport=1234" echo Hello \`hostname\`

    Hello aakrhel005

  5. Web request

    You can use the curl command with the –socks5-hostname pointing to the socks port of the tunnel we opened. The ifconfig.me will show the outgoing/external ip address of the last jumphost (not of the Laptop/machine where you issue the command). You can use this to access external ip addresses that may be blocked on your Laptop, but accessible from the Jumphost.

    curl -sSf --socks5-hostname localhost:1234 ifconfig.me

    <Outputs the proxy ip address of the final jumphost>

  6. Killing the tunnel

    Find the command that created the tunnel with “ps -ef | grep ssh”

      501 12166     1   0 10Jun20 ??         0:00.53 /usr/bin/ssh-agent -l
      501 95404     1   0 10:22AM ??         0:00.01 ssh -CfNq -D 127.0.0.1:1234 -v -i /Users/karve/amazontestkey.pem -o ProxyCommand=ssh -v -i ~/amazontestkey.pem -W %h:%p ec2-user@aakrhel002.yellowykt.com -o ProxyCommand="ssh -v -i ~/amazontestkey.pem -W aakrhel002.yellowykt.com:22 ec2-user@aakrhel001.yellowykt.com -o ProxyCommand=\"ssh -v -i ~/amazontestkey.pem -W aakrhel001.yellowykt.com:22 ec2-user@ec2-52-201-237-93.compute-1.amazonaws.com\"" ec2-user@aakrhel003.yellowykt.com
      501 95401     1   0 10:22AM ttys004    0:00.03 ssh -v -i /Users/karve/amazontestkey.pem -W aakrhel003.yellowykt.com:22 ec2-user@aakrhel002.yellowykt.com -o ProxyCommand=ssh -v -i ~/amazontestkey.pem -W aakrhel002.yellowykt.com:22 ec2-user@aakrhel001.yellowykt.com -o ProxyCommand="ssh -v -i ~/amazontestkey.pem -W aakrhel001.yellowykt.com:22 ec2-user@ec2-52-201-237-93.compute-1.amazonaws.com"
      501 95402 95401   0 10:22AM ttys004    0:00.03 ssh -v -i /Users/karve/amazontestkey.pem -W aakrhel002.yellowykt.com:22 ec2-user@aakrhel001.yellowykt.com -o ProxyCommand=ssh -v -i ~/amazontestkey.pem -W aakrhel001.yellowykt.com:22 ec2-user@ec2-52-201-237-93.compute-1.amazonaws.com
      501 95403 95402   0 10:22AM ttys004    0:00.06 ssh -v -i /Users/karve/amazontestkey.pem -W aakrhel001.yellowykt.com:22 ec2-user@ec2-52-201-237-93.compute-1.amazonaws.com
      501 95890  3001   0 10:38AM ttys004    0:00.00 grep ssh

     

    The following command kills the tunnel (along with the intermediate hops that are also Killed by signal 1).

    kill 95404 # Kill the tunnel

    debug1: channel 0: free: port listener, nchannels 1
    Transferred: sent 7144, received 5968 bytes, in 951.0 seconds
    Bytes per second: sent 7.5, received 6.3
    debug1: Exit status 0
    debug1: compress outgoing: raw data 7301, compressed 4426, factor 0.61
    debug1: compress incoming: raw data 6566, compressed 3717, factor 0.57
    debug1: channel 0: free: direct-tcpip: listening port 0 for aakrhel003.yellowykt.com port 22, connect from 127.0.0.1 port 65535 to UNKNOWN port 65536, nchannels 1
    debug1: fd 0 clearing O_NONBLOCK
    Killed by signal 1.
    debug1: channel 0: free: direct-tcpip: listening port 0 for aakrhel002.yellowykt.com port 22, connect from 127.0.0.1 port 65535 to UNKNOWN port 65536, nchannels 1
    debug1: fd 0 clearing O_NONBLOCK
    Killed by signal 1.
    debug1: channel 0: free: direct-tcpip: listening port 0 for aakrhel001.yellowykt.com port 22, connect from 127.0.0.1 port 65535 to UNKNOWN port 65536, nchannels 1
    debug1: fd 0 clearing O_NONBLOCK
    Killed by signal 1.

  7. Ansible to connect to Windows host from Mac (from where the tunnel is pre-established)

    The common Ansible parameters used for Windows are shown below:

    #ansible_port=5985
    ansible_port=5986
    ansible_connection=winrm
    #ansible_winrm_scheme=http
    ansible_winrm_scheme=https
    ansible_winrm_server_cert_validation=ignore

    On Mac you need to use the env no_proxy=’*’ to avoid the “ERROR! A worker was found in a dead state”. It is a known problem on macOS and is documented for the WinRM connection plugins. It also mentions to avoid using Kerberos authentication.

    Test with win_ping module

    env no_proxy='*' ansible -vvv -i "aakwin2012-1.yellowykt.com," aakwin2012-1.yellowykt.com -m win_ping -e "ansible_user=Administrator" -e "ansible_password=$password" -e "ansible_connection=psrp" -e "ansible_psrp_protocol=http" -e "ansible_psrp_proxy=socks5h://localhost:1234" -e "ansible_port=5985"

    Output of win_ping module above:

    aakwin2012-1.yellowykt.com | SUCCESS => {
        "changed": false,
        "invocation": {
            "module_args": {
                "data": "pong"
            }
        },
        "ping": "pong"
    }

     

    Show the directory “c:\\” output with win_shell module

    env no_proxy='*' ansible -vvv -i "aakwin2012-1.yellowykt.com," aakwin2012-1.yellowykt.com -m win_shell -a "dir c:\\" -e "ansible_user=Administrator" -e "ansible_password=$password" -e "ansible_connection=psrp" -e "ansible_psrp_protocol=http" -e "ansible_psrp_proxy=socks5h://localhost:1234" -e "ansible_port=5985"

     

    You can use the ansible-connection=winrm with ansible_winrm_transport=basic (or ansible_winrm_transport=ntlm or ansible_winrm_transport=credssp). For ansible_winrm_transport=credssp, you need to run the following in PowerShell on the target host: Enable-WSManCredSSP -Role “Server”. To disable credssp, you can run: Disable-WSManCredSSP -Role “Server”.

    env no_proxy='*' ansible -vvv -i "aakwin2012-1.yellowykt.com," aakwin2012-1.yellowykt.com -m win_ping -e "ansible_user=Administrator" -e "ansible_password=$password" -e "ansible_connection=winrm" -e "ansible_winrm_proxy=socks5h://localhost:1234" -e "ansible_port=5985" -e "ansible_winrm_transport=basic”

  8. Ansible to connect to Linux host from Mac (from where the tunnel is established)

    With Ansible command, we can use the /usr/bin/nc with -X in the ProxyCommand from Mac to connect to Linux host same as we did directly with ssh. The SOCKS5 ssh tunnel is already established with socks port 127.0.01:1234 previously. If you killed the tunnel, you will need to recreate it.

    ansible -vvv -i "aakrhel006.yellowykt.com," aakrhel006.yellowykt.com -m ping -e "ansible_user=ec2-user" -e "ansible_ssh_private_key_file=~/amazontestkey.pem" -e "ansible_ssh_common_args='-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ProxyCommand=\"/usr/bin/nc -X 5 -x 127.0.0.1:1234 %h %p\"'"

    Output is as follows:

    aakrhel006.yellowykt.com | SUCCESS => {
        "ansible_facts": {
            "discovered_interpreter_python": "/usr/bin/python"
        },
        "changed": false,
        "invocation": {
            "module_args": {
                "data": "pong"
            }
        },
        "ping": "pong"
    }

     

     

  9. Ansible Tower

    Finally, we have all the ingredients to make this work on Ansible Tower. We start by adding the hosts to the inventory.

    In the inventory, there is a aakwin2012-1.yellowykt.com host (Windows 2012 VM) and aakwin2016-1.yellowykt.com host (Windows 2016 VM). The following variables are added to the host variables of both:

    ansible_psrp_proxy: socks5h://localhost:{{ jh1_socks_port if jh1_socks_port is defined else jh_socks_port }}
    ansible_connection: psrp
    ansible_psrp_protocol: http
    ansible_port: 5985

    The reason to use “{{ jh1_socks_port if jh1_socks_port is defined else jh_socks_port }}” is to allow defining the socks port in the jumphost credential with the “jh_” or the “jh1_” prefix. This also demonstrates variables using the Jinja2 templating. This is also shown in the screenshot below:

    Screen-Shot-2020-06-29-at-5.06.13-PM

     

    Here is the job template windows_test_job_template for playbook windowstest_with_tunnel.yaml. Both the Windows host machine credential (yellowzone_windows_endpoint_credential) and the required jumphost credential (same jumphost_credential created in the Part 1) are passed. The jumphost credential is used by the first play that invokes the role in the playbook to establish the tunnel and runs on localhost (on Tower celery/task container). The role correctly selects the number of jumps based on the number of hops specified by the credential type. When the next play runs, it uses the tunnel to run the tasks on the Windows endpoint via the tunnel because of the ansible_psrp_proxy that connect to the socks5h://localhost:{{ jh_socks_port }}. The tunnel runs in an Isolated Job and automatically gets destroyed when the Ansible Tower job finishes.

    Older versions of Tower (3.5.3) were able to persist the tunnel across multiple jobs. But now with Tower Version 3.6.x, with Isolated Jobs (bubblewrap) enabled, any processes started in background are killed at end of the job. So we cannot persist the Socks tunnel using roles. The tunnel must be established for every job.

    Do not pass multiple jumphost credentials of different types. Only one machine credential and one jumphost credential should be passed to the job template. Additional credentials (for example Tower credential, etc. can be passed, but only one of each type).
     Screen-Shot-2020-07-01-at-7.38.51-AM

     

    We ran the job against the LIMIT: “win2012-1* localhost”. The localhost is required to establish the tunnel. The output of the job run is as follows. It will use the correct task from the role to establish the tunnel. For example, in this case with single jumphost:

    TASK [ansible-role-socks5-tunnel-nopassphrase : Creating socks tunnel without passphrase for single jumphost] ***
    task path: /tmp/awx_12514_77i2qfqs/project/Jumphosts/roles/ansible-role-socks5-tunnel-nopassphrase/tasks/main.yml:4

     

    Screen-Shot-2020-07-01-at-7.42.01-AM

    The screenshot above shows the commands from the playbook being executed successfully.

    We can also run with the full Inventory and localhost (if you keep the LIMIT empty) or with additional Windows hosts in the LIMIT such as “aakwin2012-1* aakwin2016-1* localhost”. Just remember to have the localhost in addition to any other hosts that you want to run the play on if you specify the LIMIT. Multiple patterns can be separated by colons (”:”).

     

    The source code for the windowstest_with_tunnel.yaml is shown below with the win_ping, win_shell and win_copy that all run successfully over the tunnel.

    windowstest_with_tunnel.yaml

    - name: Role ensures that the socks tunnel is setup
      hosts: localhost
      connection: local
      gather_facts: false
      roles:
        - ansible-role-socks5-tunnel-nopassphrase
     
    - hosts: all
      gather_facts: no
      tasks:
        - name: Do Ping
          win_ping:
          register: ping_output
          ignore_unreachable: true
     
        - debug:
            var: ping_output
     
        - win_shell: set
          args:
            executable: cmd
          register: homedir_out
     
        # Works on Windows 2016 and 2008 only if someone is logged in using that Userid
        #- win_shell: echo '%HOMEDRIVE%%HOMEPATH%'
        - win_shell: echo %SystemRoot% %USERPROFILE%
          args:
            executable: cmd
          register: homedir_out
        - debug:
            var: homedir_out
     
        - name: get powershell version
          raw: $PSVersionTable

    - name: Copying files to all Windows Endpoints
    win_copy:
    content: abc123
    #src : "windowstest_with_tunnel.yaml"
    dest: C:\Temp\foo.txt

     

    The source code for the role ansible-role-socks5-tunnel-nopassphrase is shown below:

    tasks/main.yml

    ---
    - name: Creating socks tunnel without passphrase for single jumphost
      block:
        - name: Creating socks tunnel without passphrase for single jumphost
          shell: ssh -i {{ jh_ssh_private_key }} -oPubkeyAuthentication=yes -oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no -fN -D localhost:{{ jh_socks_port }} -p {{ jh_ssh_port }} {{ jh_ssh_user }}@{{ jh_ip }};sleep 2
      when:
        - jh_socks_port is defined
        - (jh_ssh_private_key|length > 0)
     
    - name: Setting of required socks_port variables for single jumphost
      set_fact:
        jh_socks_port: "{{ jh_socks_port if jh_socks_port is defined else jh1_socks_port }}"
     
    - name: Creating socks tunnel without passphrase for single jumphost
      shell: ssh -i {{ jh1_ssh_private_key }} -oPubkeyAuthentication=yes -oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no -fN -D localhost:{{ jh_socks_port }} -p {{ jh1_ssh_port }} {{ jh1_ssh_user }}@{{ jh1_ip }};sleep 2
      when:
        - (jh2_ssh_private_key is undefined) or (jh2_ssh_private_key|length == 0)
        - (jh1_ssh_private_key|length > 0)
     
    - name: Creating socks tunnel without passphrase for two jumphosts
      shell: ssh -i {{ jh2_ssh_private_key }} -oPubkeyAuthentication=yes -oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no -oProxyCommand="ssh -i {{ jh1_ssh_private_key }} -W {{ jh2_ip }}:{{ jh2_ssh_port }} -oPubkeyAuthentication=yes -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -p {{ jh1_ssh_port }} {{ jh1_ssh_user }}@{{ jh1_ip }}" -fN -D localhost:{{ jh_socks_port }} -p {{ jh2_ssh_port }} {{ jh2_ssh_user }}@{{ jh2_ip }};sleep 2
      when:
        - (jh3_ssh_private_key is undefined) or (jh3_ssh_private_key|length == 0)
        - (jh2_ssh_private_key|length > 0)
     
    - name: Creating socks tunnel without passphrase for 3 jumphosts
      shell: ssh -i {{ jh3_ssh_private_key }} -oPubkeyAuthentication=yes -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oProxyCommand="ssh -i {{ jh2_ssh_private_key }} -W {{ jh3_ip }}:{{ jh3_ssh_port }} -oPubkeyAuthentication=yes -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oProxyCommand=\"ssh -i {{ jh1_ssh_private_key }} -W {{ jh2_ip }}:{{ jh2_ssh_port }} -oPubkeyAuthentication=yes -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -p {{ jh1_ssh_port }} {{ jh1_ssh_user }}@{{ jh1_ip }}\" -p {{ jh2_ssh_port }} {{ jh2_ssh_user }}@{{ jh2_ip }}" -fN -D localhost:{{ jh_socks_port }} -p {{ jh3_ssh_port }} {{ jh3_ssh_user }}@{{ jh3_ip }};sleep 2
      when:
        - (jh4_ssh_private_key is undefined) or (jh4_ssh_private_key|length == 0)
        - (jh3_ssh_private_key|length > 0)
     
    - name: Creating socks tunnel without passphrase for 4 jumphosts
      shell: ssh -i {{ jh4_ssh_private_key }} -oPubkeyAuthentication=yes -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oProxyCommand="ssh -i {{ jh3_ssh_private_key }} -W {{ jh4_ip }}:{{ jh4_ssh_port }} -oPubkeyAuthentication=yes -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oProxyCommand=\"ssh -i {{ jh2_ssh_private_key }} -W {{ jh3_ip }}:{{ jh3_ssh_port }} -oPubkeyAuthentication=yes -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oProxyCommand=\\\"ssh -i {{ jh1_ssh_private_key }} -W {{ jh2_ip }}:{{ jh2_ssh_port }} -oPubkeyAuthentication=yes -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -p {{ jh1_ssh_port }} {{ jh1_ssh_user }}@{{ jh1_ip }}\\\" -p {{ jh2_ssh_port }} {{ jh2_ssh_user }}@{{ jh2_ip }}\" -p {{ jh3_ssh_port }} {{ jh3_ssh_user }}@{{ jh3_ip }}" -fN -D localhost:{{ jh_socks_port }} -p {{ jh4_ssh_port }} {{ jh4_ssh_user }}@{{ jh4_ip }};sleep 2
      when:
        - (jh5_ssh_private_key is undefined) or (jh5_ssh_private_key|length == 0)
        - (jh4_ssh_private_key|length > 0)
     
    - name: Creating socks tunnel without passphrase for 5 jumphosts
      shell: ssh -i {{ jh5_ssh_private_key }} -oPubkeyAuthentication=yes -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oProxyCommand="ssh -i {{ jh4_ssh_private_key }} -W {{ jh5_ip }}:{{ jh5_ssh_port }} -oPubkeyAuthentication=yes -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oProxyCommand=\"ssh -i {{ jh3_ssh_private_key }} -W {{ jh4_ip }}:{{ jh4_ssh_port }} -oPubkeyAuthentication=yes -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oProxyCommand=\\\"ssh -i {{ jh2_ssh_private_key }} -W {{ jh3_ip }}:{{ jh3_ssh_port }} -oPubkeyAuthentication=yes -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oProxyCommand=\\\\\\\"ssh -i {{ jh1_ssh_private_key }} -W {{ jh2_ip }}:{{ jh2_ssh_port }} -oPubkeyAuthentication=yes -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -p {{ jh1_ssh_port }} {{ jh1_ssh_user }}@{{ jh1_ip }}\\\\\\\" -p {{ jh2_ssh_port }} {{ jh2_ssh_user }}@{{ jh2_ip }}\\\" -p {{ jh3_ssh_port }} {{ jh3_ssh_user }}@{{ jh3_ip }}\" -p {{ jh4_ssh_port }} {{ jh4_ssh_user }}@{{ jh4_ip }}" -fN -D localhost:{{ jh_socks_port }} -p {{ jh5_ssh_port }} {{ jh5_ssh_user }}@{{ jh5_ip }};sleep 2
      when:
        - (jh6_ssh_private_key is undefined) or (jh6_ssh_private_key|length == 0)
        - (jh5_ssh_private_key|length > 0)

     

    The vars/main.yml is shown below. This looks up the environment variables for the JH*_SSH_PRIVATE_KEY and JH*_SSH_PRIVATE_KEY_PASSPHRASE and converts them to playbook variables jh*_ssh_private_key and jh*_ssh_private_key_passphrase respectively for each of the jumphosts.

    ---
    jh_ssh_private_key: "{{ lookup('env','JH_SSH_PRIVATE_KEY') }}"
    jh_ssh_private_key_passphrase: "{{ lookup('env', 'JH_SSH_PRIVATE_KEY_PASSPHRASE') or '' }}"
    jh1_ssh_private_key: "{{ lookup('env','JH1_SSH_PRIVATE_KEY') }}"
    jh1_ssh_private_key_passphrase: "{{ lookup('env', 'JH1_SSH_PRIVATE_KEY_PASSPHRASE') or '' }}"
    jh2_ssh_private_key: "{{ lookup('env','JH2_SSH_PRIVATE_KEY') }}"
    jh2_ssh_private_key_passphrase: "{{ lookup('env', 'JH2_SSH_PRIVATE_KEY_PASSPHRASE') or '' }}"
    jh3_ssh_private_key: "{{ lookup('env','JH3_SSH_PRIVATE_KEY') }}"
    jh3_ssh_private_key_passphrase: "{{ lookup('env', 'JH3_SSH_PRIVATE_KEY_PASSPHRASE') or '' }}"
    jh4_ssh_private_key: "{{ lookup('env','JH4_SSH_PRIVATE_KEY') }}"
    jh4_ssh_private_key_passphrase: "{{ lookup('env', 'JH4_SSH_PRIVATE_KEY_PASSPHRASE') or '' }}"
    jh5_ssh_private_key: "{{ lookup('env','JH5_SSH_PRIVATE_KEY') }}"
    jh5_ssh_private_key_passphrase: "{{ lookup('env', 'JH5_SSH_PRIVATE_KEY_PASSPHRASE') or '' }}"
    jh6_ssh_private_key: "{{ lookup('env','JH6_SSH_PRIVATE_KEY') }}"
    jh6_ssh_private_key_passphrase: "{{ lookup('env', 'JH6_SSH_PRIVATE_KEY_PASSPHRASE') or '' }}"

  10. Connecting to Linux host using SOCKS5 tunnel

    Shown below is the hostname aakrhel005.yellowykt.com in the yellowzone inventory with the ansible_port: 2222

    Screen-Shot-2020-06-29-at-6.18.58-PM

    In the host variables, this host has been configured to use the connect-proxy for ProxyCommand in the ansible_ssh_common_args (instead of the ssh that was used in Part 1 – The rest of the ansible_ssh_common_args are commented out). Either the “connect-proxy” or the “ncat –proxy-type socks5” or the “socat” shown below will work on Linux with the tunnel. Your Tower image needs to have the connect-proxy and ncat installed. Instructions to create a custom Tower Image for a podified install will be shown in Part 5.

    ansible_ssh_common_args: '-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ProxyCommand="connect-proxy -S 127.0.0.1:{{ jh_socks_port if jh_socks_port is defined else jh1_socks_port }} %h %p"'

    ansible_ssh_common_args: '-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ProxyCommand="ncat --proxy-type socks5 --proxy 127.0.0.1:{{ jh_socks_port  if jh_socks_port is defined else jh1_socks_port }} %h %p"'

    ansible_ssh_common_args: '-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ProxyCommand="socat - socks4a:127.0.0.1:%h:%p,socksport={{ jh_socks_port if jh_socks_port is defined else jh1_socks_port }}"'

    The job template that invokes the same tunnel role as before (ansible-role-socks5-tunnel-nopassphrase that was used for the windows job template) is shown below. Two credentials: the jumphost credential and the Linux endpoint credential are passed as shown in Part 1. However the difference this time is that it uses the hello_with_tunnel.yaml that invokes the role in its first play to establish the tunnel and in the second play runs the hostname using the shell module on the linux host. This second play uses the tunnel that was already established in the first play. The tunnel automatically gets terminated when the job completes because the ENABLE ISOLATED JOBS is enabled.

    Screen-Shot-2020-06-29-at-6.18.06-PM

    Source code for the hello_with_tunnel.yaml

    ---
    - name: Role ensures that the socks tunnel is setup
      hosts: localhost
      connection: local
      gather_facts: false
      roles:
        - ansible-role-socks5-tunnel-nopassphrase
     
    - hosts: all
      gather_facts: no
      tasks:
        - shell: echo Hello `hostname`
          register: result

     

    The job run shows the output where “ProxyCommand=connect-proxy -S 127.0.0.1:1235 %h %p” (that we selected in the host variables) is used to establish the tunnel. It outputs the hostname aakrhel005 by connecting to the host on port 2222.

    Screen-Shot-2020-06-29-at-6.17.32-PM

     

    A final thing to note before ending this section is about using ad-hoc commands. In Ansible Tower, the “RUN COMMAND” only allow passing a Machine credential. Therefore, you cannot test running a module against a host directly when it needs to connect through one or more jumphosts because there is no direct connectivity to the endpoint host. Specifically, you cannot pass the Jumphost Credentials and the ssh key cannot be passed with extra vars.

  11. Conclusion

    Part 2 of this article showed how to create a tunnel with a role and run commands on target host Windows and Linux endpoints via multiple jumphost hops with playbooks in Ansible Tower with the use of new credential types for multiple jumphosts.

    Passphrase for jumphosts will be enabled in Part 3 and Part 4 of this series where a role will be created to accept the same credentials types with an optional passphrase.

  12. References

    Python client for the Windows Remote Management (WinRM) service https://github.com/diyan/pywinrm

    Python library for WinRM https://pypi.org/project/pywinrm/0.2.2/

    Run Powershell script results in ‘The command line is too long.’  https://github.com/diyan/pywinrm/issues/184

    Multiple Jumphosts in Ansible Tower – Part 1: Connecting to Linux hosts using ssh with nested ProxyCommand https://developer.ibm.com/recipes/tutorials/multiple-jumphosts-in-ansible-tower-part-1

    Multiple Jumphosts in Ansible Tower – Part 3: Ssh tunnel SOCKS5 proxy with passphrase enabled for ssh keys https://developer.ibm.com/recipes/tutorials/multiple-jumphosts-in-ansible-tower-part-3

    Multiple Jumphosts in Ansible Tower – Part 4: Multi jumphost connections to Linux hosts using ssh-add to add keys to ssh-agent https://developer.ibm.com/recipes/tutorials/multiple-jumphosts-in-ansible-tower-part-4/

    Multiple Jumphosts in Ansible Tower – Part 5: Unix domain socket file instead of socks port https://developer.ibm.com/recipes/tutorials/multiple-jumphosts-in-ansible-tower-part-5/

    Multiple Jumphosts in Ansible Tower – Part 6: Primary and Secondary/Backup Jumphosts and Reverse ssh Tunnel https://developer.ibm.com/recipes/tutorials/multiple-jumphosts-in-ansible-tower-part-6/

     

     

Join The Discussion