Return latest active dataset per category, each with embedded file entries.

Combines the metadata from get_latest_per_category with only root-level BackfillVersionFile rows. Deeper partition rows are fetched lazily through the tree endpoints so large datasets do not need to be fully serialized here.

Pass include_full=true to populate the schema and statistics fields (used by the datasets page detail panel). When omitted, those fields are null / empty.

Pass published_before to restrict results to datasets published strictly before that datetime — useful for point-in-time queries.