RTCM3 messages are not being sent via TCP?

[harshil21]

Hello,

This is my setup:

1. LC29H (DA) Pi hat
2. That is sitting on top of a raspberry pi zero 2w
3. Data from the GPS module is being sent to the serial on the raspberry pi
4. Pi zero 2w and my PC are on the same wifi network
5. I use a python script to ssh into the pi and read the serial port.
6. I start a TCP server so that PyGPSClient can read this data (and also write back).

Here's my reading/writing script:

paramiko and TCP server script

import paramiko
import socket
import threading
import time
import sys
import select

# Configuration - customize these
PI_IP = '10.140.80.202'  # Raspberry Pi's IP address
PI_USERNAME = 'redacted'       # Default username
PI_PASSWORD = 'redacted'  # Replace with your password (or use key-based auth)
SERIAL_DEVICE = '/dev/serial0'  # Serial port on Pi
BAUD_RATE = 115200         # Baud rate for your sensors
LOCAL_HOST = '127.0.0.1'  # Localhost for security
LOCAL_PORT = 12345       # Port for local TCP server (change if needed)


def configure_serial(ssh):
    config_cmd = f'stty -F {SERIAL_DEVICE} {BAUD_RATE} raw -echo'
    stdin, stdout, stderr = ssh.exec_command(config_cmd)
    error = stderr.read().decode()
    if error:
        print(f"Error configuring serial: {error}")
        sys.exit(1)
    print("Serial port configured.")

def start_pi_channels(ssh):
    # Reader channel: cat from serial
    read_cmd = f'cat {SERIAL_DEVICE}'
    read_stdin, read_stdout, read_stderr = ssh.exec_command(read_cmd)
    
    # Writer channel: cat to serial
    write_cmd = f'cat > {SERIAL_DEVICE}'
    write_stdin, write_stdout, write_stderr = ssh.exec_command(write_cmd)
    
    return read_stdout, write_stdin

def forward_to_client(read_stdout, client_sock, stop_event):
    while not stop_event.is_set():
        if read_stdout.channel.recv_ready():
            data = read_stdout.read(1024)
            if data:
                try:
                    client_sock.sendall(data)
                except:
                    stop_event.set()
                    break
        time.sleep(0.01)

def forward_from_client(write_stdin, client_sock, stop_event):
    while not stop_event.is_set():
        r, _, _ = select.select([client_sock], [], [], 0.1)
        if r:
            data = client_sock.recv(1024)
            print(f"Received {data!r} from pygpsclient, forwarding to Pi.")
            if not data:
                stop_event.set()
                break
            write_stdin.write(data)
            write_stdin.flush()

def proxy_serial_to_tcp():
    global stop_event
    stop_event = threading.Event()
    
    # SSH connection to Pi
    ssh = paramiko.SSHClient()
    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    ssh.connect(PI_IP, username=PI_USERNAME, password=PI_PASSWORD)
    print("Connected to Raspberry Pi via SSH.")
    
    configure_serial(ssh)
    read_stdout, write_stdin = start_pi_channels(ssh)
    
    # Start local TCP server
    server_sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    server_sock.bind((LOCAL_HOST, LOCAL_PORT))
    server_sock.listen(1)  # Accept one client
    print(f"Local TCP server listening on {LOCAL_HOST}:{LOCAL_PORT}. Connect with PyGPSClient...")
    
    try:
        client_sock, addr = server_sock.accept()
        print(f"Client connected from {addr}. Proxying bidirectional data...")
        
        # Start forwarding threads
        to_client_thread = threading.Thread(target=forward_to_client, args=(read_stdout, client_sock, stop_event))
        from_client_thread = threading.Thread(target=forward_from_client, args=(write_stdin, client_sock, stop_event))
        
        to_client_thread.start()
        from_client_thread.start()
        
        # Wait for stop
        to_client_thread.join()
        from_client_thread.join()
        
    except KeyboardInterrupt:
        stop_event.set()
        print("\nStopping...")
    except Exception as e:
        print(f"Error: {e}")
    finally:
        stop_event.set()
        client_sock.close() if 'client_sock' in locals() else None
        server_sock.close()
        read_stdout.channel.close()
        write_stdin.channel.close()
        ssh.close()
        print("Connections closed.")

if __name__ == "__main__":
    proxy_serial_to_tcp()

Problem:

Whenever I setup the NTRIP client, and start getting RTCM3 messages, I don't see that data being sent to my TCP socket. I know that $PQTMVERNO messags are being forwarded properly, because I manually sent them from PyGPSClient (see the output below):

Connected to Raspberry Pi via SSH.
Serial port configured.
Local TCP server listening on 127.0.0.1:12345. Connect with PyGPSClient...
Client connected from ('127.0.0.1', 44304). Proxying bidirectional data...
Received b'$PQTMVERNO*58\r\n' from pygpsclient, forwarding to Pi.
Received b'$PQTMVERNO*58\r\n' from pygpsclient, forwarding to Pi.
Received b'$PQTMVERNO*58\r\n' from pygpsclient, forwarding to Pi.

The RTCM3 messages are being output to the console, so I don't know why it doesn't reach my module?

[harshil21]

Additionally, I noticed that the module version and such things are not displayed in the UI for some reason, but when directly connecting and opening serial, this is shown.

[semuadmin]

Hi @harshil21

This, for the moment, is expected behaviour. PyGPSClient currently only sends UBX commands/polls via a TCP socket in support of UBX devices which can be configured this way. It was primarily intended to work with directly-connected serial devices.

Bear in mind most serial GNSS devices don’t support TCP input natively - there would need to be some kind of custom TCP client/parser at the receiving end.

I’ll look into the implications of supporting such functionality but it won’t be a quick fix.

[harshil21]

I understand. My use case however is quite special since I can't directly run pygpsclient on my raspberry pi zero 2w, python is too slow to run on it, and there is a massive latency in everything in the UI.

Can you point me to where in the code this happens? I can fork the repo and use this feature for the time being.

[semuadmin]

Can you point me to where in the code this happens?

Where what happens, exactly?

It's not a question of forking PyGPSClient code.

You would need to construct an entirely new Python TCP client to run on your raspberry pi zero 2w. It wouldn't necessarily need to parse or decode any incoming or outgoing binary UBX or RTCM data, but simply forward it between the TCP socket(s) and the relevant UART port(s).

Have you considered using the pygnssutils CLI utilities instead?

[harshil21]

Where what happens, exactly?

I imagine there is a specific place in the codebase where sending RTCM3 messages to the TCP socket is prohibited.

You would need to construct an entirely new Python TCP client to run on your raspberry pi zero 2w. It wouldn't necessarily need to parse or decode any incoming or outgoing binary UBX or RTCM data, but simply forward it between the TCP socket(s) and the relevant UART port(s).

The script I pasted above already does this. When I run that script on my laptop, and open pygpsclient on my laptop, connect via TCP, and send the PQTMVERNO command, my module receives this. I know this because the RX LED on my GPS module lights up, indicating it actually got data. PyGPSClient also shows that the command was acknowledged by the module.

The LED does not light up when I connect to a NTRIP Client however, the RTCM messages are shown in the console, but they are not transferred to my localhost TCP server.

Have you considered using the pygnssutils CLI utilities instead?

I would eventually switch to using CLI utilities, but for now I need a fast way to test functionality (specifically the scatter plot when I have a RTK fix so I can see how the module performs in various conditions while testing)

[semuadmin]

https://github.com/semuconsulting/PyGPSClient/releases/tag/v1.5.15