Tutorials

Step-by-step usage examples.

Simplified Database Functions

Use an origin namespace to fetch data — the data source is explicit:

import pysus

# Download SINAN Dengue data (DATASUS FTP, via the S3 catalog mirror)
df = pysus.ftp.sinan(disease="deng", year=2000)

# Multiple years
df = pysus.ftp.sinan(disease="deng", year=[2023, 2024])

# SINASC births for São Paulo (dados.gov.br)
df = pysus.dadosgov.sinasc(state="SP", year=[2020, 2021, 2022, 2023])

# SIM mortality data
df = pysus.ftp.sim(state="SP", year=2024)

# SIH hospitalizations with month filter
df = pysus.ftp.sih(state="SP", year=2024, month=[1, 2, 3])

# CNES health facilities
df = pysus.ftp.cnes(state="SP", year=2024, month=1)

The as_dataframe=True fetchers above return a single concatenated pandas.DataFrame. See the FileBag Workflow section below for how to inspect files before downloading or work with the downloaded files individually.

FileBag Workflow

A namespaced fetcher returns either a high-level FileBag or a DataFrame:

download=False → a remote FileBag listing the files that would be fetched, without downloading anything (as_dataframe is ignored here).
download=True (default) + as_dataframe=False → a local FileBag of downloaded files.
as_dataframe=True → a single concatenated pandas.DataFrame.

A FileBag is synchronous — the underlying async client is started and awaited internally:

import pysus

# 1. List what would be downloaded (nothing is fetched yet)
bag = pysus.ftp.sinan(disease="deng", year=2020, download=False)
print(bag)
# Files[DENGBR20.parquet (remote)]

# 2. Inspect individual files (path, type, dataset, ...)
f = bag[0]
print(f.path)                 # public/data/ftp/sinan/DENG/2020/_/BR/DENGBR20.parquet

# 3. Download all (or a subset) -> a local FileBag
local = bag.download()        # Files[DENGBR20.parquet]
# local = bag.download(indexes=[0])   or   bag.download_one(0)

# 4. Concatenate the downloaded tabular files into one DataFrame
df = local.to_dataframe()     # same as local.df
print(df.shape)               # (975842, 121)

len/iter/[] (including slices) and paths let you introspect a bag, and kind tells you whether it holds "remote" or "local" files. Remote bags from URL-only origins (e.g. pysus.saude.*) also support this workflow.

OpenDataSUS (Saude) Functions

import pysus

# Dengue/Chik/Zika notifications from OpenDataSUS
df = pysus.saude.arboviroses(disease="dengue", year=2024)

# Vaccination coverage
df = pysus.saude.vacinacao(state="SP", year=2024)

# Hospital and health facility data
df = pysus.saude.assistencia_saude(state="SP", year=2024)

# Primary care (Previne Brasil)
df = pysus.saude.atencao_primaria(state="SP", year=2024)

# Nutrition surveillance
df = pysus.saude.sisvan(state="SP", year=2024)

Discovery

from pysus import info_table, search, list_files

# Browse available datasets
info_table()

# Search for datasets by keyword
results = search("dengue")

# List files in a dataset
df = list_files("SINAN", group="DENG", year=2024)

Discovery is also scoped per origin:

import pysus

pysus.ftp.info()                                   # FTP origin datasets
pysus.ftp.list_files("SINAN", year=2024, state="RJ")
pysus.dadosgov.get_origin_meta()                   # origin metadata

Using the PySUS Client

from pysus import PySUS

async def main():
    async with PySUS() as pysus:
        # Query DuckLake catalog
        files = await pysus.query(
            dataset="sinan",
            group="DENG",
            state="SP",
            year=2024,
        )

        # Download files
        for f in files:
            local = await pysus.download(f)
            print(local.path)

        # Read multiple parquet files
        import glob
        paths = glob.glob("/cache/sinan/**/*.parquet")
        df = pysus.read_parquet(paths, mode="union")

Parallel Downloads

# Download many files in parallel
downloaded = await pysus.download_many(files, max_concurrent=5)

read_parquet Modes

# Union (default) - all columns from any file
df = pysus.read_parquet(paths, mode="union")

# Intersection - only common columns across all files
df = pysus.read_parquet(paths, mode="intersection")

# Strict - raises error if schemas don't match
df = pysus.read_parquet(paths, mode="strict")

# With custom SQL filter
df = pysus.read_parquet(paths, sql="SELECT * WHERE column > 100")

Data Quality

from pysus import column_stats, missing_values, quality_score, validate_data

# Per-column statistics (types, nulls, uniques, sample values)
stats = column_stats(df)

# Missing value summary
missing = missing_values(df)

# Overall quality score (0-100)
score = quality_score(df)

# Validate data against expected schema
issues = validate_data(df, dataset="SINAN")

Data Transformation

from pysus import (
    aggregate_by_age_group,
    aggregate_by_period,
    aggregate_by_state,
    detect_units,
    optimize_memory,
    rename_columns,
    to_english,
)

# Aggregate cases by age group
grouped = aggregate_by_age_group(df, age_col="IDADE", period_col="DT_NOTIFIC")

# Detect measurement units in columns
units = detect_units(df)

# Rename columns with a mapping
df = rename_columns(df, mapping={"DT_NOTIFIC": "notification_date"})

# Convert Portuguese column/value names to English
df_en = to_english(df)

# Optimize memory usage
df = optimize_memory(df)

Column Metadata

from pysus import search_columns, load_column_metadata, available_databases

# List curated schema databases
print(available_databases())  # ["sim", "sinan"]

# Load columns for a specific SINAN disease
columns = load_column_metadata("sinan", "Dengue")

# Search for a column across all datasets
results = search_columns("CON_CLASSI")
for r in results:
    print(r.dataset, r.name, r.categories)

Export

from pysus import export, to_csv, to_excel, to_geojson, to_sql

# Export to various formats
to_csv(df, "output.csv")
to_excel(df, "output.xlsx")
to_geojson(df, "output.geojson", lat_col="LATITUDE", lon_col="LONGITUDE")

# Full export with options
export(df, "output.csv", compression="gzip")

Data Diff

from pysus import diff_dfs, diff_summary, diff_rows

# Compare two DataFrames
diff = diff_dfs(df_old, df_new)

# Summary of differences
summary = diff_summary(df_old, df_new)

# Row-level differences
rows = diff_rows(df_old, df_new)

Cache Management

from pysus import cache_status, clear_cache

# Show cache statistics
cache_status()

# Clear all cached files
clear_cache()