Know what your data promises, and whether it keeps it.
Sahifa reads a whole data store, from Postgres to Parquet on S3. It profiles every table, writes the checks for you, scores each quality dimension with an honest error bar, and tells you where the bad rows come from.
§1orders.customer_id refers to customers.id312 orphans since 29 Sep, 94 % of them from source = pos_berlin.Broken
§2events loaded by 06:00
Last load 3 h 40 min ago. Alert threshold set for 1 false alarm in 100 days.Late
§3customers.iban is a valid IBAN (ISO 13616)
99.97 % of 2.1 M rows pass the check digits.Kept
§4invoices.vat_id matches the EU VAT format
Suggested from the profile. Waiting for your approval.Proposed
The problem
The checks exist. The assessment doesn't.
Data teams already run dbt tests, a Great Expectations suite here, a Soda scan there. Each covers one engine, writes its own report, and gives a score, if any, with no sense of how sure it is.
Coverage stops at hand-written tests. Every check someone has to write is a check that usually isn't written. The 1,200 tables nobody wrote tests for are where the problems live.
One engine at a time. Deequ needs Spark, Elementary needs dbt models, Pandera needs DataFrames, TestGen needs a SQL warehouse. Nothing reads Postgres and the Parquet files on S3 in one pass.
Scores without error bars. A score from a 1 % sample looks as certain as one from every row. Nobody can tell a real drop from sampling noise.
The open tools closed up. Soda Core moved to the Elastic License, GX Cloud shut down, and anomaly detection sits behind cloud paywalls. Self-hosting the good parts got harder in 2026.
How it works
Profile it. Check it. Score it. Explain it.
One loop for every table in the store. Each run adds to a history, so the checks, the thresholds and the scores get sharper the longer Sahifa watches.
1
Profile every column
Types, null shares, distinct counts, patterns, ranges and keys, computed inside your database or on the files where they lie. Roles and semantic types (IBAN, email, VAT ID) are inferred.
2
Checks written for you
Rules start active, baselines learned from your data start as proposals. You lock what you trust; Sahifa never re-generates a locked check.
3
Score with an error bar
A score per ISO/IEC 25012 dimension for every column, table and the store, with a 95 % interval from the rows actually read.
4
Explain and propose
Every finding names the count, examples (masked when personal), the SQL that found them and a next step. Sahifa never writes to your data.
$sahifa scan "postgresql://reader@db/shop?schemas=public" --pretty
sampling 100,000 rows per table · read-only session✓ 5 tables · 38 columns · 71 checks active · 23 proposed
Store score 92.795 % interval 92.3–93.1
completeness 98.1 97.8–98.4 uniqueness 99.4 99.3–99.5
validity 90.3 89.9–90.8 consistency 94.2 93.7–94.6
accuracy 89.6 88.1–91.0 currentness 85.0 80.2–88.9● critical orders.customer_id: 312 orphan rows (sah.foreign_key)● high customers.iban: 630 invalid IBANs (sah.semantic_format)● high events: newest row 3 h 40 min old (sah.freshness)● low customers.city: 41 casing variants (sah.casing_variants)
What is different
Six gaps we close.
We studied the open-source and commercial tools before writing a line of code. These are the gaps Sahifa closes.
Scores you can trust
No other tool says how certain its quality score is. Sahifa follows the ISO/IEC 25024 measures and attaches an interval from the rows it read.
ISO/IEC 25012 · 25024
Every store in one pass
SQL databases, warehouses, lakehouse tables and plain files: checks run where the data lives, through one connector layer.
Postgres · SAP HANA · Snowflake · BigQuery · DuckDB · Parquet · Iceberg · CSV
Alerts with a promise
Thresholds for volume, freshness and drift come with a false-alarm rate you choose, instead of a sensitivity slider.
Seasonal baselines · conformal thresholds
Rules for your country
Rule packs for IBANs, VAT IDs and postcodes across Europe and beyond, not only US ZIP codes and states.
ISO 13616 · EU VAT · UPU postcodes
Contracts you can take with you
A contract is inferred from each table's profile. Edit it, approve it and export it in the open standard.
Open Data Contract Standard v3
A health check for the store
Unused and duplicate tables, small-file problems on lakehouse tables and missing indexes, next to the quality findings.
Query logs · MinHash · compaction advice
Coverage
Thirty checks in six dimensions.
The first catalogue, written down before the code. R1 ships twenty of them; the rest need a scan history, mining or a custom-rule editor, and follow in R2 and R3.
empty tables · no primary key · numbers stored as text · unused and duplicate tables · small files · missing indexes
R1R3
Open, with nothing held back
Sahifa runs on your own server under the Apache 2.0 licence. Many users, many connections and many projects come at no cost. Your data never leaves your network, and no telemetry is sent anywhere. It is built only on engines with Apache or MIT licences, so you can embed it and host it as you like.
Apache 2.0
Postgres
DuckDB
Apache Arrow
Ibis
ODCS
OpenLineage
Splink
Family
A sister of Tabayyun
Tabayyun verifies industrial time series. Sahifa covers the rest of the store and shares its ideas: findings that live through a lifecycle, scores per dimension, corrections as separate versions with full lineage. When Sahifa finds a timestamp column with a measured value, it hands the series to Tabayyun.
Your storeDatabases, warehouses, lakehouse tables, files
SahifaTables and columns: profile, check, score, explain
→ TabayyunTime series inside those tables: gaps, spikes, drift, flatlines
One reportA score with its interval, findings with evidence, an exportable contract
Architecture
Checks run where the data lives.
A Python core turns the catalogue into SQL and pushes it down: one profile query and one check query per table, on a reproducible sample. DuckDB reads files and object storage. Results, history and findings live in Postgres. Runs on one server with docker compose, or on CapRover.
Boundaries
What Sahifa deliberately is not.
Not a cleaning tool. It never writes to a source. Fixes are proposals you apply in your own pipeline.
Not a copy of your data. It keeps profiles, counts and at most five masked examples per finding, never the tables.
Not an AI that judges your data. Models may propose rules; a person approves them and the engine runs them deterministically.
Not a SaaS or an open-core teaser. Every feature is in the Apache-2.0 build, for any number of users and connections.
صحيفة
The name
Why the name
The Ṣaḥīfa of Medina was the written charter the Prophet ﷺ agreed with the people of the city. It set down in writing what each party could expect of the others. A data contract does the same for a table.
The Sīra also tells of another ṣaḥīfa: the boycott document hung in the Kaaba, found eaten by termites except for the name of Allah. Records decay unless someone looks after them. Sahifa does the looking.
Plan
When
NowSpecs and first build. Research, decision records, the check catalogue and the first working core are written in the open on GitHub.
October 2026First preview at sahifa-stg.siralabs.org: profiling, generated checks and ISO scores with intervals for Postgres, DuckDB, Parquet and CSV.
Nov – Dec 2026Run history, anomaly thresholds with a false-alarm rate, drift with effect sizes, sign-in and teams.
Q1 2027Explanations of where bad rows cluster, rules proposed by an AI model and approved by you, contract export, Snowflake, BigQuery and SAP HANA with an SAP rule pack.
Q2 2027The store health report and root cause along lineage.
Dates are the current plan and may move. Sahifa is part of Sira Labs.