What each section answers, how to interpret the charts, and what the scores can and cannot tell you.
Start with provenance and governance at the top of a card. Provenance links to the original resources and publications. Governance records the license, redistribution conditions and content identity.
Open source access means an open original-source route is recorded. Restricted redistribution is the display label for the catalog's private tier: rights are restricted or not cleared for redistribution. It does not mean that the original source is necessarily private.
Raw reproducible identifies a retrieval route. Assembly incomplete means raw resources can be available while dataset-specific conversion is missing. The loading example uses retrieve(id, prepare=False) for these raw downloads; it returns resource metadata rather than X/Y arrays. A complete original-source recipe uses get(id). Two Harvard datasets require explicit guestbook answers.
Token required means no open automatic route is recorded. A Dataverse token works only once the canonical data have a configured hosted DOI. Manual access requires the original provider's documented steps. Availability labels describe the catalog routes, not a fresh live check of every URL.
Spectra have many correlated wavelengths. Principal component analysis (PCA) compresses their variation into a few orthogonal directions, called principal components (PCs). It is an exploratory summary of X, calculated without using the target Y.
The score plot places each plotted observation on PC1 and PC2. Nearby dots have similar projections; groups can reveal batches, instruments or natural subpopulations. Axis percentages show explained spectral variance. Large sources are sampled to at most 800 observations. Similar positions in two dimensions do not guarantee similar full spectra; an isolated dot does not prove an acquisition fault.
The scree plot reports spectral variance per component, helping you see whether variation is concentrated in a few dimensions. It is not explained target variance or predictive accuracy. The scree projection and score-plot projection may use different row samples; their percentages need not match.
Q / SPE measures residual variation outside the retained PCA subspace. Hotelling T² measures an observation's scaled distance within that subspace; Mahalanobis H summarizes PCA-space distance. The displayed p95/median ratios compare the distribution's upper tail with its center, without an absolute rejection threshold.
Historical profiles replace missing cells with column medians before PCA, center channels, and use bounded row/component counts. Loadings and distances depend on signal units and preprocessing. Compare compatible sources and inspect the original spectra before acting.
Read measured values first. An alert score maps selected metrics to a heuristic 0–1 scale. The display bands are weak below 0.42, medium from 0.42 to below 0.72, and strong from 0.72. They are not universally validated pass/fail thresholds.
The radar's center is 0 and its outer ring is 1: outward means a stronger anomaly or heterogeneity signal for that axis. Its table gives values rounded to two decimals and explains each axis. Missing axes are left blank; partial profiles have no filled polygon. The mean profile score averages only available axes and is not an overall quality grade.
Diagnostic bars rank possible explanations from heuristic signatures. A score of 0.78 does not mean a 78% causal probability. Evidence coverage is the proportion of required signals available, not confidence in the cause. A dash or “unavailable” means the quantity could not be computed; zero is an actual recorded value.
RMS (root mean square distance) and SAM (Spectral Angle Mapper, in radians) use this dataset's mean spectrum as their reference. They do not compare with an independently acquired white or dark reference. Repeated IDs require a documented protocol before within-ID spread can be called technical repeatability.
X–Y plots show Pearson correlation within this dataset. They identify linear associations; they are not cross-validated predictions and do not establish causal relationships. Targets, metadata, observation counts and physical sample counts have different meanings.
The same explanations appear beside values through the question-mark controls. Hover for a short hint, or expand a control with touch, Enter or Space. p95 means the 95th percentile; SD means standard deviation; CV means coefficient of variation.
| Metric | Meaning and limitations |
|---|---|
| Wavelengths | Number of spectral channels. A channel can represent wavelength, wavenumber or another feature axis. |
| Axis range | Lowest and highest recorded spectral-axis coordinates, in the stated unit. |
| Mean spacing | Average difference between adjacent spectral-axis coordinates; compare only within the same unit. |
| Grid | Uniform means adjacent axis spacings are approximately equal. An irregular grid may need care when applying derivatives. |
| Observations | Exported spectral rows in this source, including repeated measurements when present. |
| Value range | Minimum and maximum recorded spectral values, in the original signal representation. |
| Mean range | Minimum and maximum of the mean spectrum across spectral channels. |
| Mean level | Average signal over observations and channels. The historical key mean_reflectance may also store other signal representations. |
| Area | Trapezoidal integral of the mean spectrum over the sorted recorded axis; its unit is signal multiplied by axis unit. |
| PTP | Peak-to-peak: maximum minus minimum of the mean spectrum. Low values indicate little variation along the mean spectrum. |
| Noise RMS | Historical roughness sigma proxy from robust second differences. Narrow genuine bands can also raise it; this is not repeat-measured instrument noise. |
| Roughness S/N proxy | Average absolute signal divided by the roughness proxy. Higher means smoother signal relative to its amplitude, not certified instrument SNR. |
| Roughness S/N proxy dB | 20 log10 of the roughness signal-to-noise proxy. Units are decibels; compare only compatible signal representations. |
| Dynamic range | Maximum minus minimum of the mean spectrum. Its magnitude depends on the signal representation. |
| Smoothness | Mean per-spectrum RMS of second differences, in signal units. Smaller values indicate smoother spectra under this protocol. |
| Saturated | Fraction at the observed global maximum; a heuristic indicator, not a comparison with known detector limits. |
| X-outliers | Number flagged by the qualification outlier method. An unusual spectrum can be valid and should not be discarded automatically. |
| NaN ratio | Missing spectral cells divided by all spectral cells, displayed as a percentage. |
| Inf count | Number of positive or negative infinite spectral values; investigate invalid calculations or export. |
| Zero ratio | Exactly zero spectral cells divided by all cells. Zero can be a valid measurement and is not automatically missing. |
| Spike count | Local narrow peaks flagged by the second-difference heuristic; real sharp spectral bands may also be flagged. |
| Spike rate | Fraction of tested second-difference positions flagged as spikes, displayed as a percentage. |
| Jump count | Adjacent-channel changes flagged by the first-difference heuristic, potentially including detector joins. |
| Jump rate | Fraction of tested adjacent-channel differences flagged as jumps, displayed as a percentage. |
| Clip fraction | Fraction of finite cells near the observed global minimum or maximum; inspect detector limits before concluding saturation. |
| Baseline slope | Linear slope fitted to the mean spectrum over an axis normalized to 0–1. It describes total baseline change, not change per physical axis unit. |
| Curvature RMS | Median per-spectrum RMS of second differences, summarizing curvature and high-frequency variation. |
| D1 RMS | Median per-spectrum RMS of adjacent-channel first differences; reflects both real spectral structure and artefacts. |
| RMS to mean | Root mean square distance to this dataset's mean spectrum, in signal units. |
| RMS p95 | 95th percentile of distances to the mean spectrum: 95% of observations have a distance at or below this value. |
| SAM to mean | Spectral Angle Mapper: angle to the mean spectrum, in radians. Smaller angles indicate more similar shapes. |
| SAM p95 | 95th percentile of spectral angles to the dataset mean, in radians. It is less sensitive to overall amplitude than RMS. |
| Affine offset p95 | 95th percentile of absolute fitted offsets when matching each spectrum to the dataset mean. |
| Affine gain p95 Δ | 95th percentile of the absolute gain deviation from 1 in the fit to the mean spectrum. |
| Affine residual p95 | 95th percentile of residual RMS after gain and offset correction to the mean spectrum. |
| Xcorr lag p95 | 95th percentile of the absolute cross-correlation shift relative to the mean spectrum, in spectral channels. |
| PCA Q p95/median | Q (squared prediction error) is variation left outside the retained PCA subspace. The displayed ratio compares its 95th percentile with its median. |
| Hotelling T2 p95/median | Hotelling T² measures distance within the retained PCA subspace, scaled by component variance. Ratio of the 95th percentile to the median. |
| Mahalanobis H p95/median | Distance from the PCA-score center after variance scaling. Ratio of the 95th percentile to the median; large values identify relatively extreme spectra. |
| Repeat groups | Number of IDs with several observations. Source documentation is needed to decide whether these are technical repeats, positions or time points. |
| RMS intra-ID | Within-ID root mean square deviation from the group's mean spectrum, in signal units. |
| SAM intra-ID | Within-ID spectral angle to the group's mean spectrum, in radians. |
| CV intra-ID | Within-ID coefficient of variation: standard deviation relative to absolute mean; unstable near zero signal. |
| Effective rank | Entropy-based summary of the variance distribution across components. Larger values indicate variation spread across more dimensions. |
| PCs → 95% var | Number of principal components needed to explain 95% of spectral variance, when reached by the computed projection. |
| PCs → 99% var | Number of principal components needed to explain 99% of spectral variance, when reached by the computed projection. |
| Top-10 cum. var | Fraction of spectral variance explained by up to the first ten computed principal components, displayed as a percentage. |
| n / missing | Recorded observation count and number of missing values for this variable. |
| Mean ± SD | Arithmetic mean and standard deviation in the variable's stated unit. |
| Median | 50th percentile: half the non-missing values lie at or below it. |
| Range | Smallest and largest recorded non-missing values. |
| CV | Coefficient of variation: standard deviation divided by absolute mean; unstable when the mean is near zero. |
| Skew / kurtosis | Distribution asymmetry and excess kurtosis. A Gaussian distribution has skewness 0 and excess kurtosis 0. |
| Normal? | Descriptive normality-test result. A non-normal target is not automatically a defective variable. |
| Classes | Number of distinct recorded category values. |
| Entropy (norm.) | Normalized Shannon entropy, from 0 for one dominant class to 1 for equal class frequencies. |
| Balance (entropy) | Normalized Shannon entropy, from 0 for one dominant class to 1 for equal class frequencies. |
| Top class | Most frequent recorded category and its observation count. |
| Imbalance ratio | Largest class count divided by smallest class count; 1 indicates equal counts. |
| Minority | Fraction of observations in the least frequent class. |
Definitions, formulas and possible explanations for the reported values. A possible cause requires verification against acquisition records. Reading guide
Look up each measured quantity, its calculation and the heuristic alert mapping before interpreting a high or low value.
| Family | Metric | What it measures | Higher values suggest | Lower values suggest | Possible causes | Calculation / score |
|---|---|---|---|---|---|---|
| Data integrity | Invalid/masked cell count (owner report) | Null or masked cells in the evaluated population | To investigate | Complete according to the declared mask | Missing data or validity mask; not evidence of an acquisition cause | Native Preflight count of cells unavailable under x/null and valid_mask; denominator stays in native_metric.denominators.n_cells. Not legacy NaN ratio. |
| Data integrity | NaN ratio | Missing data | Corrupted spectrum | Full spectrum | Acquisition/export error | count(isnan(X)) / X.sizealert = min(1, nan_ratio / 0.05) |
| Data integrity | Inf count | Infinite values | Corruption | Normal | Invalid calculations | count(isinf(X))alert = min(1, inf_count / 1) |
| Data integrity | Zero ratio | Null columns or cells | Truncated spectrum | Normal | Export, saturation | count(X == 0) / count(finite X)alert = min(1, zero_ratio / 0.05) |
| Overall amplitude | Mean reflectance | Average level | Too light / background visible | Too dark | Background, geometry | mean(X finite)alert reuses baseline/shape drift because absolute reflectance ranges are technology-dependent |
| Overall amplitude | Area under curve | Overall intensity | Illumination difference | Normal | Probe distance | trapezoid(mean_spectrum, spectral_axis)alert reuses baseline/shape drift because area scale depends on axis and units |
| Overall amplitude | Peak-to-peak (PTP) | Dynamic range | Strong variability | Flat spectrum | Saturation | max(mean_spectrum) - min(mean_spectrum)alert increases when dynamic range is abnormally flat |
| Overall amplitude | Variance | Spectral variability | Normal or heterogeneous | Flat spectrum | Bad contact | var(X finite)alert increases when variance/dynamic range is abnormally flat |
| Noise | Roughness sigma proxy | High frequency noise | Noisy | Stable | Lamp, detector | median MAD(second derivative) * 1.4826 / sqrt(6)alert = noise_rms / signal_scale, saturated at 5% |
| Noise | Roughness S/N proxy | Signal quality | Good signal | Bad signal | Acquisition | mean(abs(X)) / noise_rmsalert decreases with SNR dB; >=40 dB is treated as low alert |
| Noise | Bandwise roughness S/N proxy | Localized noise | Reliable zone | Problem area | Detector | min(abs(mean_spectrum) / local second-derivative noise)alert decreases with worst-band SNR dB; >=35 dB is treated as low alert |
| Local artifacts | Spike count | Narrow peaks | Artifacts | Clean spectrum | Cosmic rays, splice | count robust outliers in second derivativealert follows spike_rate, saturated at 1% |
| Local artifacts | Spike rate | Peak density | Suspicious spectrum | Normal | Interpolation | spike_count / (n_samples * (n_features - 2))alert = min(1, spike_rate / 0.01) |
| Local artifacts | Jump count | Discontinuities | Detector connection | Continuous | Splice | count robust outliers in first derivativealert follows jump_rate, saturated at 1% |
| Local artifacts | Jump rate | Jump frequency | Spectral problem | Normal | Calibration | jump_count / (n_samples * (n_features - 1))alert = min(1, jump_rate / 0.01) |
| Local artifacts | Clip fraction | Saturation | Clipping | Normal | Detector saturated | fraction of finite cells equal to repeated min/max extremaalert = min(1, clip_fraction / 0.01) |
| Spectral form | Baseline slope | Overall slope | Drift | Stable | Illumination | linear slope of mean_spectrum over normalized axisalert = abs(slope / signal_scale), saturated at 0.5 |
| Spectral form | Curvature RMS | Curvature | Unusual shape | Smooth | Background, splice | median RMS(second derivative per spectrum)alert = curvature_rms / signal_scale, saturated at 1% |
| Spectral form | D1 RMS | Local variability | Structured spectrum | Flat | Biology or artifact | median RMS(first derivative per spectrum)alert = d1_rms / signal_scale, saturated at 5% |
| Multivariate outliers | PCA Q (SPE) | Not explained by PCA | Atypical spectrum | Compliant | Artifact, mixture | p95(Q/SPE residual) / median(Q/SPE residual)alert = min(1, pca_q_ratio / 8) |
| Multivariate outliers | Hotelling T² | Extreme in PCA | Extreme but consistent | Central | Natural variability | p95(Hotelling T2) / median(Hotelling T2)alert = min(1, hotelling_t2_ratio / 8) |
| Multivariate outliers | Mahalanobis H | Distance to cloud | Global outlier | Normal population | Different domain | p95(sqrt(T2)) / median(sqrt(T2))alert = min(1, mahalanobis_h_ratio / 4) |
| Comparison to average spectrum | RMS to mean spectrum | Average distance | Different spectrum | Typical | Domain shift | p95 RMS distance to dataset mean spectrumalert = RMS_p95 / signal_scale, saturated at 25% |
| Comparison to average spectrum | Spectral Angle Mapper (SAM) | Difference in shape | Different shape | Similar | Background, geometry | p95 spectral angle to dataset mean spectrumalert = min(1, SAM_p95 / 0.35 rad) |
| Repeatability | RMS intra-ID | Reproducibility | Poor repeatability | Stable | Positioning | median RMS distance to repeated-sample centroidalert = RMS_intra_ID / signal_scale, saturated at 10% |
| Repeatability | SAM intra-ID | Shape variation | Unstable | Stable | Acquisition | median SAM to repeated-sample centroidalert = min(1, SAM_intra_ID / 0.15 rad) |
| Repeatability | CV intra-ID | Internal variability | Bad control | Stable | Operator | median within-ID band CValert = min(1, CV_intra_ID / 0.25) |
| Structure of the dataset | PCA score density | Clusters | Subpopulations | Homogeneous | Different batches | 1 / median kNN distance in PCA score spacealert follows density_cv/profile structure complexity, not raw density alone |
| Structure of the dataset | kNN distance ratio (not LOF) | Local anomaly | Isolated spectrum | Normal population | Rare cases | p95(mean kNN distance / median population mean kNN distance) in PCA-score space; not LOFalert = min(1, max(0, kNN_distance_ratio_p95 - 1) / 2) |
| Structure of the dataset | Isolation Forest score | Global anomaly | Atypical spectrum | Normal | Various causes | p95 IsolationForest anomaly score on PCA scoresalert follows structure complexity; raw score is implementation-dependent |
Find measurement-specific checks and acquisition context that can help interpret a source's spectral patterns.
| Technology | Adaptations / metrics | Target anomalies | Practical guidance |
|---|---|---|---|
| UV-Vis 300-1000 nm | Baseline, overall slope, drift at edges 300-350 and 900-1000; metrics by zones | Stray light, bad white, saturation, weak signal at the ends | The edges are often unstable; also calculate edge/middle scores. |
| UV-Vis 300-1000 nm | Saturation / clipping near max absorbance or max reflectance | Signal clipped | Very important if strong absorption. |
| UV-Vis 300-1000 nm | Red-edge, peak position, band ratios for vegetation | Biological shift or optical artifact | Helps distinguish real change from acquisition problem. |
| UV-Vis 300-1000 nm | Smoothness / roughness index | High frequency noise | Often more informative than the SNR alone. |
| MIR / ATR-FTIR | ATR contact quality index: overall intensity, total area, depth of key bands | Poor crystal-sample contact | Crucial: many anomalies come from the ATR contact. |
| MIR / ATR-FTIR | CO2 / H2O atmospheric bands | Bad atmospheric correction | Frequent spurious peaks. |
| MIR / ATR-FTIR | Baseline curvature / rubber-band residual | Diffusion, contact, baseline drift | Very useful before PCA. |
| MIR / ATR-FTIR | Peak position shift | Bad spectral alignment/calibration | Important in FTIR because small shifts count. |
| MIR / ATR-FTIR | Band area ratios on known bands | Chemically incoherent spectrum | To be adapted by matrix: polysaccharides, proteins, lipids, etc. |
| HS-MS | Total Ion Current (TIC), Base Peak Intensity (BPI) | Weak injection, unstable ionization | MS equivalent of the global spectral level. |
| HS-MS | Number of peaks detected | Poor or too noisy spectrum | Too little = bad signal; too much = noise/contamination. |
| HS-MS | Mass accuracy / m/z drift | Mass calibration problem | Fundamental in HRMS. |
| HS-MS | Retention time drift if LC/GC-MS | Chromatographic drift | Monitor on standards/QC pools. |
| HS-MS | Blank contamination score | Contaminants / carry-over | Compare samples vs blanks. |
| HS-MS | Internal standard CV | Instrumental variability | Very robust if standards available. |
| HS-MS | Missingness by feature | Detection instability | Crucial for filtering variables. |
| With repetitions | Intra-sample RMS | Overall repeatability | Applicable to all technologies. |
| With repetitions | SAM / intra-sample correlation | Shape repeatability | Very useful for spectra. |
| With repetitions | Intra-sample CV per band / feature | Local repeatability | Detects unstable areas. |
| With repetitions | ICC or variance components | Share of sample vs. technical variance | Very useful if several repetitions per sample. |
| With repetitions | Distance to intra-ID centroid | Aberrant repetition | Allows you to flag the bad repetition rather than the entire sample. |
Review suggested follow-up methods for suspected data issues and whether their results are available in this explorer.
| Potential issue family | Suggested methods | What it detects | Explorer availability |
|---|---|---|---|
| Global spectral shift | Inter-dataset average spectrum correlation, DTW, cross-correlation, comparison of peak positions | Wavelength shift, misalignment, different interpolation | Partially calculated: cross-correlation lag and dispersion of peak positions vs. average spectrum. |
| Baseline / offset / gain | Regression each spectrum vs average spectrum: x = a + b ref + residual; followed by a, b, residual RMS | Additive offset, multiplicative effect, baseline drift | Calculated in reference.affine_*. |
| Row mixing / X-M-Y mismatch | Index check, row hash, ID duplication, intra-ID spectral distance, inconsistent labels | Mixed up rows, misaligned metadata, Y assigned to wrong spectrum | Partially covered by intra-ID repeatability; checks index/hash to add to the canonical pipeline. |
| Information leak / poorly split repetitions | GroupKFold by sample_id vs StratifiedKFold random; auditing partitions by sample_id | Artificially good performance due to repetitions | Requires splits and benchmark model; not calculated by the descriptive card. |
| Label bugs | Close samples in X but different Y, confident learning, systematic FP/FN errors | Inverted Y, input errors, ambiguous classes | Requires Y and/or model; recommended for supervised explorer. |
| Hidden subdomains | PCA/UMAP/t-SNE + unsupervised clustering + association with dataset/Y/date/operator | Batches, campaigns, probes, backgrounds not provided | Partially calculated by PCA and distance ratio kNN (not LOF); UMAP/t-SNE outside the static card. |
| Unknown localized artifacts | Wavelength x dataset map: mean difference, variance difference, KS per wavelength | Unanticipated anomalous spectral regions | To be calculated at the bank level when several datasets share a spectral axis. |
| Instrumental breaks | Discontinuities in derivatives, changepoint detection | Splice, detector connection, local jump not provided | Calculated by jump/spike rates; more advanced changepoint to add. |
| Spectral mixing/contamination | NMF / unmixing / reconstruction by convex hull | External component: background, plastic, soil | Not calculated automatically; requires component assumptions or large library. |
| Unstable but predictive features | Model importance vs. QC instability by variable | Model that learns an artifact rather than a biological signal | Requires supervised model; recommended for benchmark reporting. |