Skip to content

SSH tunnel #93

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 10 commits into
base: master
Choose a base branch
from
Open

SSH tunnel #93

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
1f61ddf
Update setup.py
aroder Jun 17, 2020
46a5773
added support for SSH tunneling and for using an array for the filter…
aroder Jun 18, 2020
6a19c27
cast string 'true' in config to boolean
aroder Jun 18, 2020
f919ac7
handle SSL status if there are multiple connections for the same user
aroder Jun 18, 2020
9fc707e
logging
aroder Jun 18, 2020
e6d030d
support catalog as well as properties
aroder Jun 18, 2020
ef59742
support default_replication_key
aroder Jun 18, 2020
0546163
timing
aroder Jun 19, 2020
724dbc0
merged from singer-io/tap-postgres
aroder Jul 13, 2020
4623e21
updated unit tests
aroder Jul 14, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,4 +106,5 @@ ENV/
env-vars.txt
tap_oracle/__pycache__/
*~
config.json
*config.json
.vscode/
37 changes: 37 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,43 @@ using the wal2json decoder plugin.
SELECT * FROM pg_create_logical_replication_slot('stitch', 'wal2json');
```

## Configuration
Check out `config.json.sample` for an example configuration file.

| Field | Required? | Default | Details |
|----------------------------|-----------|---------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| | | | |
| dbname | Yes | | The name of the database to connect to |
| default_replication_method | No | | Allows setting a default replication method. If the replication method for a stream is not set in the catalog.json, then this replication method is used. Can be one of ` |
| default_replication_key | No | | Allows setting a default replication key. Should be a string value containing the field name, e.g. `updated_at` |
| host | Yes | | The host to be used to connect |
| password | Yes | | The password to be used to connect |
| port | Yes | | The port to be used to connect |
| user | Yes | | The user to be used to connect |
| use_ssh_tunnel | No | False | Set to true to open an SSH tunnel and connect to the database through the tunnel |
| ssh_jump_server | No | | Only used if `use_ssh_tunnel` is set to true. This is the URL or IP address of the jump server that the connection should tunnel through |
| ssh_jump_server_port | No | 22 | Only used if `use_ssh_tunnel` is set to true. This is the port of the jump server the SSH tunnel will attempt to connect to |
| ssh_private_key_path | No | | Only used if `use_ssh_tunnel` is set to true. This is the path on the local machine to the private SSH key |
| ssh_username | No | | Only used if `use_ssh_tunnel` is set to true. This is the username to be used to connect to the jump server |
| ssl | No | | If true, the sslmode value is set to "require" otherwise sslmode value is not set |

## Development
### Running tests
Install requirements for development: `pip install -e .[test]`

Install postgres to run tests:
1. sudo apt update
1. sudo apt install postgresql postgresql-contrib
1. sudo service postgresql start
1. sudo passwd postgres (set to postgres)
1. sudo -u postgres psql
1. alter user postgres password 'postgres'
1. \q

Run unit tests: `nosetests --where tests/unittests`



---

Copyright © 2018 Stitch
13 changes: 10 additions & 3 deletions setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,20 @@
classifiers=['Programming Language :: Python :: 3 :: Only'],
install_requires=[
'singer-python==5.3.1',
'psycopg2==2.7.4',
'psycopg2==2.8.4',
'strict-rfc3339==0.7',
'nose==1.3.7'
'sshtunnel==0.1.5'
],
extras_require={
'dev': [
'autopep8>=1.5.3',
'python-dotenv>=0.14.0',
'nose>=1.3.7',
]
},
entry_points='''
[console_scripts]
tap-postgres=tap_postgres:main
''',
packages=['tap_postgres', 'tap_postgres.sync_strategies']
)
)
Loading