Data Pipeline
01README
ETL data pipeline with source connectors, S3 sinks, and integrated secret management.
Note: Requires
@swampadmin/vault-secretsfor database credential rotation.
Supports incremental extraction, partitioned writes, and automatic compaction for cost-efficient storage.
02Release Notes
First stable release.
- Postgres source with CDC and batch modes
- S3 sink with Parquet and JSON output
- Daily sync workflow with extract → load → compact stages
- Env-based vault for local development
03Models
@swampadmin/data-pipeline/postgres-sourcev1.0.0postgres-source.ts
Global Arguments
| Argument | Type | Description |
|---|---|---|
| connection_string | string | Database connection URL |
| schema? | string | Database schema name |
fn connect(connection_string: string, pool_size?: number)
connect operation
| Argument | Type | Description |
|---|---|---|
| connection_string | string | Database connection URL |
| pool_size? | number | Connection pool size |
fn query(sql: string, params?: array)
query operation
| Argument | Type | Description |
|---|---|---|
| sql | string | SQL query to execute |
| params? | array | Query parameters |
fn stream(table: string, batch_size?: number)
stream operation
| Argument | Type | Description |
|---|---|---|
| table | string | Source table name |
| batch_size? | number | Rows per batch |
Resources
postgres-source.state(persistent)— Managed resource state
postgres-source.lock(ephemeral)— Concurrency lock
@swampadmin/data-pipeline/s3-sinkv1.0.0s3-sink.ts
Global Arguments
| Argument | Type | Description |
|---|---|---|
| connection_string | string | Database connection URL |
| schema? | string | Database schema name |
fn write(path: string, format?: enum)
write operation
| Argument | Type | Description |
|---|---|---|
| path | string | Destination path |
| format? | enum | Output format (parquet, json, csv) |
fn partition(name: string)
partition operation
| Argument | Type | Description |
|---|---|---|
| name | string | Resource name |
fn compact(name: string)
compact operation
| Argument | Type | Description |
|---|---|---|
| name | string | Resource name |
Files
s3-sink.log(text/plain)— Operation audit log
s3-sink.json(application/json)— Structured output
04Workflows
Daily Data Syncdaily-sync
Daily Data Sync workflow
daily-sync-jobExecute Daily Data Sync
1.Extract@swampadmin/data-pipeline/postgres-source.stream— Extract step
2.Load@swampadmin/data-pipeline/s3-sink.write— Load step
3.Compact@swampadmin/data-pipeline/s3-sink.compact— Compact step
05Vaults
env-credentials
vault-providerenv-vault.ts
env-credentials secrets provider
06Stats
Downloads
0
Archive size
87.0 KB
Not yet scored.
A score will be generated the next time this extension is published. The owner can also trigger scoring manually.
07Platforms
08Labels