Observations and estimation#

bonsai.io.estimation separates reported evidence, requested target quantities, accounting constraints and estimates. It does not read provider files or mutate the SUT graph. Domain adapters decide compatibility and own workflow artifact IO.

Data model#

Observation describes an indicator, an existing Property, a Location or canonical location code, a Period, classification scope, population, source key and qualifications. A ratio has a separate fixed denominator Property.
Scope explicitly lists covered classification members. A missing dimension or None is unspecified, not an instruction to aggregate over that dimension. An aggregate lists its members. Scope membership does not imply allocation.
TargetSpec states the target indicator, unit, population, location, period and dimensions. A table supplies the individual target cells.
EstimationProblem contains labelled unknowns, bounds, linear relations and assumptions. compile_observations checks compatibility before constructing additive relations. Ratios require an explicit domain relationship.

Objects do not generate a context ID for every classification combination. Observation records are stored as tables; classification membership and source periods remain explicit. Existing node locations and times are not migrated.

Numerical contracts#

An exact relation is an accounting identity or an explicitly declared treatment of evidence. An uncertain observation needs supplied bounds or a standard error; source priority never determines statistical precision. Missing and suppressed observations are retained without substituting zero.

The initial method solves feasibility, fits uncertain observations by constrained weighted least squares, and calculates identification ranges over the fitted solution set. Underidentified cells remain unknown. Exact identities and target bounds are checked independently in original units before results are accepted. Infeasible or failed problems return no target values.

An identification range describes ambiguity under the supplied problem. It is not a confidence interval. A unique fitted value does not establish that its source data are certain. Published rounding bounds likewise do not measure survey sampling error. Holdout scoring supports evaluation against evidence excluded from estimation.

The current implementation operates on small connected problems. Labour uses country/activity or country/sector/sex blocks; it does not construct one dense matrix for all observations. New statistical or sparse numerical methods can consume the same labelled problem representation.

First application: labour#

The workflow preserves native evidence before allocating source totals:

clean_labour_sources writes cleaned_source_observations.pkl/csv, retaining native LFS occupation counts, totals, suppression flags, rounding bounds and explicit armed-forces/unspecified categories. ILO derived native profiles and SES source employee/earnings profiles retain their lineage. Native ILO/LFS profile pickles and existing BONSAI provider profile artifacts are also retained.
The labour adapter can infer one missing skill from a published total when excluded occupation counts are explicitly zero and the resulting range is within published rounding precision. Other unknowns remain unresolved.
merge_labour_sources prefers an existing complete source. Otherwise the adapter combines individually usable employment/hour cells, retaining source priority for overlaps. Same-country, within-two-year transfer remains the collection policy; source-year evidence stays in provenance. Cross-provider employment population alignment and retained point estimates are explicitly recorded assumptions. Payroll weights with different currencies or arbitrary normalisations cannot be combined as partial additive quantities.
labour_estimates.pkl/csv retains known and unresolved target cells, per-cell source, method, provenance and available identification ranges. Only complete six-cell targets enter account exports. selected_source_conservation.csv reconciles actual selected cells against each provider’s native totals.

In labour_estimates, exportable describes a complete profile. Compensation rows at this stage are explicitly labelled allocation weights, not monetary compensation totals; account construction additionally requires usable fixed D1. Uncertainty marked not_quantified must not be interpreted as zero uncertainty.

The approved SES equal-paid-hours proxy remains labelled. Fixed activity D1 is split once and conserved. Estimation does not change balanced value added, monetary use/supply, or HIOT technology. Remaining missing labour stays unknown; coverage and estimation provenance must accompany footprint results.

Hierarchical gap filling#

With Config.labour_hierarchical=True, source merging fits empirical-Bayes hierarchical regressions using country, broad sector family, section and industry effects. Gaussian prior variances are learned from native profiles by EM, with restricted-likelihood refinement for slowly converging variance components. Employment totals use monetary output as a predictor. Employment shares use observed-subset log contrasts; payroll shares use complete centred-log-ratio responses. Both distributions use softmax prediction; annual hours per employed person use log responses. Observed zeros receive only explicit numerical regularisation in the compositional training transform, not fabricated survey sample sizes. Existing observed or allocated cells remain fixed.

Total-employment training can also use selected ILO A employment totals when the six sex/skill cells are incomplete. Male and female totals must come from the same survey, year and occupational basis. Explicit armed/unclassified counts are subtracted to retain the classified population; missing excluded counts are not assumed zero. Existing complete-profile totals remain unchanged. The task exports employment_total_training_candidates.csv and employment_total_training_audit.csv to document eligibility. The hours and payroll components keep their own completeness rules. Compare predictive changes on identical held-out profiles and folds; scores over an expanded test population are a separate result.

Employment shares always use the model in estimation.composition; the old complete-profile model and its configuration switch have been removed. For each native profile it fits only contrasts between observed sex/skill counts. With observed subset S, its response is log(n_g) - mean_S(log(n)); the observation projection has rank len(S)-1. It does not assert that S is the full population. The same country, family, section and industry effects predict six scores, converted to shares by softmax. Gaussian effect and residual variances are learned from these observed contrasts; missing cells are never imputed as training responses. Predictions use a sum-zero orthonormal basis and conditional isotropic log-contrast covariance. Their prediction ranges remain uncalibrated and differ from the complete-CLR model’s empirical residual covariance.

The adapter retains all existing complete donors. Additional ILO skill counts must be direct observations or sums with every constituent major occupation present. Additional Eurostat cells must be observed and usable, not estimates or suppression bounds. Surveys, periods and occupational bases must be coherent; ROW aggregates are excluded. At least two positive-total observed cells are needed. One count without a denominator carries no share information. This version does not fit aggregate sex margins or count/total ratios as extra likelihood terms; national additive margins still enter downstream calibration. Observed zero shares receive numerical regularisation only during fitting. Existing account cells, including zeros, and D1 stay fixed during gap filling.

employment_share_training.csv contains complete and incomplete count vectors, with NaN for unavailable cells; employment_share_training_audit.csv records eligibility. Both training selection and the model identifier enter the model hash. Complete evaluation profiles stay in hierarchical_training.csv. Validation hides the same complete profiles and purges overlapping partial, national and ALL evidence before fitting. The pooled donor baseline also remains the same, allowing a paired comparison.

Use scripts/compare_labour_employment_shares.py with --labour, --raw and --output directories to reproduce that comparison without changing production artifacts. It compares the current contrast model using complete donors only against the same model using all eligible donors. Saved historical comparison results retain the removed complete-CLR model’s predictions. The 2026-11-12 test expanded 1,004 to 4,475 profiles, including 2,975 incomplete vectors. On the same 1,004 complete profiles, whole-country WAPE rose from 39.99% to 44.07%; country/sector WAPE rose from 39.47% to 42.64%. Complete-only contrast predictions reproduced the current point predictions. Therefore the new model has higher error on those complete-profile holdouts. Following the user’s 2026-11-12 decision, employment-share fitting always uses the broader native evidence. Model eligibility depends on available native donors, not the number of complete evaluation profiles. This is an explicit coverage preference, not evidence that the previous model overfits or that the new model is more accurate on incomplete profiles.

Employment totals are predicted at the available classification grain, then allocated to child activities by output. Splitting an activity therefore does not change its group’s employment total. Rates and payroll distributions are fitted separately; payroll prediction uses native derived wage bills and, if enabled, the approved SES equal-paid-hours profiles. It does not turn SES employee counts into total employment, nor working hours into employee paid hours.

collect_hierarchical_labour pins EMP_TEMP_SEX_OCU_NB_A in its own manifest. Cleaning retains a coherent same-country survey/year within two years. Classified sex/skill counts become national benchmark constraints. An incomplete total that may contain unclassified occupations supplies an upper bound, unless known unclassified counts can be subtracted. Joint KL calibration respects overlapping sex/skill margins and known activity cells. Inconsistent benchmarks are reported; missing employment is withheld for those countries instead of changing knowns. National employment is assigned to domestic activity output as a modelling assumption; no cross-border commuting correction is available in this workflow.

Validation masks complete native country/sector-family blocks and entire countries before fitting. Overlapping national and broader ALL records are purged, and ROW aggregates never replicate member-country training observations. All model fitting and feature encoding occur inside each fold. Compare with the pooled industry/family donor baseline in hierarchical_validation.csv; inspect every reference prediction in hierarchical_holdout_predictions.csv. Component scores and combined employment/annual-hours cell scores are separate. Payroll scores against SES proxy or derived ILO profiles are not validation against independent observed sex/skill compensation accounts.

Prediction ranges are conditional on fitted hyperparameters and modelling assumptions. Their empirical held-out coverage is reported; they are not claimed to be calibrated confidence intervals. Employment ranges rescaled by national benchmarking are labelled benchmark_scaled_model_range. Footprints report total employment and hours across employees and self-employed workers, plus employee compensation, each by sex and skill. Employee counts and employee-hours remain internal inputs to D1 allocation; no vulnerable-hours account is calculated. Inspect unseen-category flags, model convergence and benchmark conflicts alongside coverage gains.

The final labour_observations.pkl/csv combines cleaned source observations and national observations. Supporting files include hierarchical_training.csv, hierarchical_targets.csv, hierarchical_predictions.csv, hierarchical_benchmarks.csv, hierarchical_models.pkl and hierarchical_model_diagnostics.json. The model ID hashes training data and settings and fitted model state; per-cell provenance references it. Model pickles require the project’s pinned environment. labour_validation_folds, labour_estimation_seed and labour_hierarchical_iterations control reproducible fitting and evaluation.

The training table stores one response vector per native profile. y0 through y5 follow male low/medium/high, then female low/medium/high. For employment_total, only y0 is used and it is persons. Employment-share rows contain raw person counts, and payroll-share rows contain source payroll weights; both are normalised before the log-ratio transform. Annual-hours responses are hours per employed person per year. The prediction table likewise contains employment in persons, annual-hours rates, and dimensionless payroll shares. Account construction multiplies rates by persons and splits fixed D1 using payroll weights. A model interval is not an uncertainty assessment of the fixed D1 input or of these accounting/population assumptions.