Overview
Unit provides its Banks and Clients with direct access to their raw data, enabling them to build custom reports, analyze, monitor, and make informed business decisions. This integration supports a CDC-based (Change Data Capture) pipeline that incrementally writes your data from Unit’s internal databases to your cloud storage, whether on AWS S3, GCP, or Azure, in Parquet format.
The exported data can be easily loaded into any modern data warehouse using standard connectors, eliminating the need to build a sequence of API calls to obtain data. Additionally, for clients who can securely store personally identifiable information (PII), the PII data will be included in the exported data too.
For guidelines for migrating from ETL V.1 to incremental ETL, please visit the ETL Migration Guide .
By accessing the raw data and creating your own analytics stack on top of it, decision makers in your organization can make data driven decisions on various areas. Some questions that can be looked at include:
- Which end-customer cohorts and/or marketing campaigns created the highest revenue?
- How does pricing (interest, payment fees and rewards programs) affect engagement?
- Which end-customers are most engaged and deserve to be promoted to your "VIP" terms? (example: higher cash-back)
- Which merchants do your end-customers spend most on?
- How do different end-customers segments use ACH, wire and check deposits differently?
The data streamed into your bucket is raw output from Unit’s internal database. As such, its structure and availability may change over time.
As mentioned above, the data transferred includes the RAW data from our database. Example for transferred tables:
- Authorization Requests
- Payments
- Applications
- Cards
- Disputes
- Customers
- Check deposits
- Transactions (V2)
- Accounts
Most Data pipelines frameworks allow consuming data from the common cloud storage very easily. We currently support AWS S3, Google GCS, and Azure Blob (please reach out to your Unit contact for more information).
Data Structure
Files Location
Each table will contain files in a specific path with the format:
<table_name>/year=yyyy/month=mm/day=dd/hour=hh/<file_name>.parquet
For example transactions/year=2024/month=01/day=01/hour=01/part-00000-2bfb9dac-fdd5-4025-b8fa-174eedde9d45-c000.snappy.parquet.
Each of the hourly locations will contain one or more Parquet files, each containing a portion of the relevant hour's incremental data. Do not count on a specific number of files; the number can vary.
Incremental Data Content
Due to the nature of a Change Data Capture (CDC) pipeline, the data we transmit is incremental. This means that every change to a row—whether an INSERT, UPDATE, or DELETE—is treated as a separate incremental event. Each change results in the full row being written to the corresponding file. For DELETE operations, the row is transmitted with NULL values for all columns except the identifier (id) column.
Because the application is dynamic and constantly updating, you may occasionally receive the same row more than once, even if the visible data hasn't changed. This can happen, for example, when internal columns—those not configured to be included—are updated.
In such cases, you'll receive "duplicate rows".
An INSERT event is not guaranteed to be the first operation you receive for a given row. In certain tables, the association with a specific Platform is established only after the initial write to the database.
Until that association is in place, the row is not eligible for transmission.
As a result, the first event you receive for a row may be an UPDATE rather than the initial INSERT.
This behavior is expected and depends on the table's design and data flow.
In some tables, the same row can be updated multiple times in the same timestamp (transaction).
In order to get the latest state of the row, you should use a special column (cdc_change_seq). More info in the section below - special columns.
Special Columns
To track changes in the continuous data stream, there is a need for special columns that help identify the operation type, the change sequence and the commit timestamp. The three columns below are added to each table:
cdc_operation_type- Operation type executed for the row (UPDATE, INSERT or DELETE).cdc_change_seq- A unique incrementing number. Use it for ordering the results.cdc_commit_timestamp- The commit timestamp.
Data Freshness
Data transfers occur hourly with changes from the previous hour. For each table, if there is a change, you will get it in the relevant Files Location.
- Data updates may experience delays of up to 4 hours.
- Column deletions occur automatically without notice. The column will disappear from future files.
- New columns and tables will be introduced over time as part of ongoing product development and enhancements.
Reloading Data
Our internal export process uses a checkpointing mechanism to track which data has already been delivered, ensuring efficient and consistent incremental updates. If you ever need to re-export data from a specific point in time, please contact the Unit team.
The schema of each table can be inferred from the Parquet files. If you wish to get a table-to-column mapping, please reach out to your Unit contact.