Soil Condition QA NCEA Soil Conditions Data 2022-24.pdf
Source: environment.data.gov.uk (2026-01-05)

NFI+ Soil Condition & Health. This dataset contains comprehensive soil chemical and physical indicators collected as part of the NFI+ (National Forest Inventory Plus) programme and the BioSoil forest soil and biodiversity monitoring network. The dataset covers forest soil sampling sites across Great Britain and provides baseline and monitoring data on soil condition and health in forest ecosystems.

Chemical indicators measured include: soil pH (measured in water and CaCl2), organic carbon content (%), total nitrogen (%), available phosphorus (mg/kg), exchangeable cations (Ca, Mg, K, Na in cmol/kg), cation exchange capacity (cmol/kg), base saturation (%), and heavy metal concentrations where applicable.

Physical indicators measured include: soil texture (% sand, silt, clay), bulk density (g/cm³), porosity (%), aggregate stability, moisture content at sampling, and soil depth profiles.

Samples were collected at multiple depth intervals (0-10cm, 10-20cm, 20-40cm) using standardised protocols aligned with ICP Forests and BioSoil methodologies.  Each sampling location includes associated metadata on forest type, soil type, elevation and forest origin (plantation or natural).  The data include chemical and physical soil properties.

Quality assurance procedures included duplicate sampling, laboratory reference standards, and inter-laboratory comparisons.

Limitations:  Data coverage is limited to forest soils only. Some historical samples may have incomplete heavy metal analysis. Seasonal variation should be considered when interpreting results.

Data are georeferenced using a delocalised grid. The bounding coordinates for each grid square are included in the dataset. Further information about the delocalisation process is provided in the dataset evidence pack.

The 'NFI+ Soil Condition & Health' is a project funded by the UK Government through Defra’s Natural Capital and Ecosystem Assessment programme (NCEA), England sites, and the Department for Energy, Security and Net Zero (DESNZ), Scotland sites.  Users should ensure they have read the supporting documentation before viewing or making use of the data. This includes the NCEA Analysis Ready Data Limitations Statement and dataset specific limitations/caveats outlined in the supporting documentation, please refer to the 'Metadata' tab on the 'Soil Condition Data 2022-24' dataset  Attribution statement: © Forestry Commission copyright and/or database right 2025. Includes data from Ordnance Survey, British Geological Survey, and the Met Office. All rights reserved.

Natural Capital & Ecosystem
Assessment Programme (NCEA)
Quality Assurance Procedures for Soil
Chemical and Physical Data

Elena Vanguelova, Forest Research, 08/12/2025
NCEA Soil Condition Data 2023–24 – “Soil chemical data” sheet

1. Overview
This document describes the steps taken to quality-assure (QA) the soil chemical data
contained in the Soil chemical data worksheet of the NCEA Soil Condition Data 23
24.xlsx file.
The QA focussed on:
•
Identifying extreme and anomalous values
•
Quantifying distributional properties (means, variability, 95% confidence limits)
•
Defining plausible ranges for each variable based on soil science and observed data
•
Flagging all records that fall outside these plausible ranges
•
Preparing reproducible code (R) for ongoing QC.

2. Data Source and Structure
•
Source file: NCEA Soil Condition Data 23 24.xlsx
•
Sheet used: Soil chemical data
•
Key identifying fields:
o
PLOT_ID (plot identifier)
o
CORE (core identifier within plot)
o
DEPTH (depth interval, e.g. 0-5, 5-10, 40-80)

2 of 24

•
All remaining numeric columns represent soil chemical properties (pH, total C and N,
exchangeable cations, base and aluminium saturation, organic matter, etc.).
Identifiers (PLOT_ID, CORE, DEPTH) were treated as metadata, not included in summary
statistics, but retained for all QA reporting of outliers and threshold violations.

3. Pre-processing and Data Type Handling
3.1 Extraction of numeric variables
•
The dataset was read in with all columns from the Soil chemical data sheet.
•
Numeric variables were identified using their data type, excluding PLOT_ID and CORE.
•
Where necessary, columns were coerced to numeric with non-numeric entries
converted to NA (missing) for QC checks.
3.2 Handling censored extractable phosphorus values (“<2”)
The variable:
•
EXTRACTABLE PHOSPOROUS (PO4) mg/kg)
contains some entries reported as “<2”, indicating values below the detection limit.
For the QC analysis:
•
All instances of " <2 " were recoded to 1 mg/kg (half the detection limit) to allow
numeric analyses (descriptive statistics, min/max, etc.).
•
A note is included that for more formal statistical analyses, users may wish to:
o Retain an explicit censored indicator, or
o Perform sensitivity analyses using different substitutions (e.g. 0, 1, 2 mg/kg,
or censored-data models).

4. Exploratory Descriptive Statistics
For each numeric soil chemical variable:
1. Non-missing values were used to calculate:
o N (number of observations)
o Mean
o Standard deviation (SD)
o Minimum and maximum values
2. 95% confidence limits (CL) for the mean were calculated as:
[
\text{CL}_{95} = \bar{x} \pm 1.96 \times \frac{\text{SD}}{\sqrt{N}}
]

3 of 24

3. These statistics were used to:
o Characterise the overall range and central tendency
o Identify variables with high variability or skewed distributions (e.g. total
carbon, total nitrogen, extractable P).
Examples (from the earlier analysis):
•
SOIL_ pH_H₂O:
o Mean ≈ 5.34, SD ≈ 1.31, range ≈ xx-xxxx-xxxx, 95% CL ≈ 5.28–5.40
•
SOIL_pH_CaCl₂:
o Mean ≈ 4.50, SD ≈ 1.40, range ≈ 2.54–7.95, 95% CL ≈ 4.43–4.56
•
TOTAL C (%):
o Mean ≈ 9.26, SD ≈ 13.63, range ≈ xx-xxxx-xxxx, indicating very organic
horizons in some samples.
These statistics formed the basis for:
•
Understanding the natural variability in the dataset
•
Informing later choices of plausible thresholds.

5. Outlier and Extreme Value Analysis
5.1 Statistical outliers
For each numeric variable, two complementary outlier rules were applied:
1. IQR (Interquartile Range) rule
o Values were flagged as outliers if:
▪
( x < Q1 - 1.5 \times IQR ), or
▪
( x > Q3 + 1.5 \times IQR )
2. 3 × SD rule
o Values were flagged if:
▪
( |x - \bar{x}| > 3 \times \text{SD} )
These outliers are statistical extremes, not necessarily data errors. They were used to:
•
Identify variables with long tails or heavy skew (e.g. extractable P, total C, total N,
CEC, exchangeable cations).
•
Highlight candidates for further agronomic review or transformation (e.g. log-
transform prior to modelling).
5.2 Listing min/max extreme observations
To support traceability:
•
For each numeric variable, the minimum and maximum values were identified.
•
All rows sharing those exact min/max values were extracted with:

4 of 24

o
PLOT_ID
o
CORE
o
DEPTH
o
Variable
o
ExtremeType (min or max)
o
Value
This produced a full extremes table linking each variable’s most extreme values back to
specific plot–core–depth combinations. A separate summary extremes table recorded one
representative min and max case per variable.
This step allows:
•
Direct checking against field notes, lab sheets, or site conditions.
•
Rapid identification of true extremes vs. data entry or unit errors.

6. Identification of Apparent Data Issues
During review of distributions and extremes, the following data issues were detected:
1. Negative values in some variables:
o
EXCHANGEABLE MANGANESE (mg/l)
o
EXCHANGEABLE MANGANESE (mg/kg)
o
EXCHANGEABLE MANGANESE (cmol(+)/kg)
o
FREE_ACIDITY cmol(+)/kg
These negative values were small in magnitude (e.g. −0.004 to approx. −0.05),
consistent with instrument or rounding noise around zero, rather than
physically meaningful negative concentrations.
2. Percentages slightly exceeding 100%:
o
LOI (%) at 375°C had one value around 100.62%.
This is physically unrealistic but very close to 100%, likely due to minor
analytical, moisture, or rounding issues.
3. Broad but plausible ranges for key variables:
o pH values from strongly acidic (~2.5–3 in CaCl₂) to moderately alkaline (~8 in
H₂O).
o Total carbon from mineral topsoils (~0.2%) up to very high organic contents
(>50%), consistent with organic/peat soils.
o Extractable P up to ~400 mg/kg, plausible for highly fertilised locations.
These observations informed the plausible-range thresholds described next.

7. Definition of Plausible Thresholds by Variable
For every major soil chemical variable in the spreadsheet, plausible minimum and
maximum values were defined to support data validation/QC. These are not agronomic

5 of 24

“target” values but hard physical/data plausibility bounds – values outside these ranges
should be flagged for review.
The thresholds were based on:
•
The observed range in the dataset; and
•
Typical soil science expectations (e.g. pH of agricultural soils, maximum reasonable
CEC, realistic % ranges, etc.).
Examples:
•
pH (water):
o Plausible range: 3.0 to 9.5
•
pH (CaCl₂):
o Plausible range: 3.0 to 8.5
•
Total N (%):
o Plausible range: 0.0 to 5.0
•
Total C and total organic C (%):
o Plausible range: 0.0 to 60.0
•
Base and aluminium saturation (%):
o Plausible range: 0.0 to 100.0
•
Organic matter (%) and LOI (%):
o Plausible range: 0.0 to 100.0
•
Cation exchange capacity (CEC):
o Plausible range: 0.0 to 150 cmol(+)/kg
•
Exchangeable cations and micronutrients (Ca, Mg, K, Na, Al, Fe, Mn in mg/L,
mg/kg, cmol(+)/kg):
o Plausible upper limits set slightly above the highest observed values, yet
within known physical bounds for natural and cultivated soils (e.g. K up to
1,500 mg/kg, Ca up to 20,000 mg/kg, etc.).
•
Extractable P (PO₄, mg/kg):
o Plausible range: 0.0 to 400.0
A full table of plausible thresholds for each variable in the sheet was compiled and used as
the basis for mechanised range checks. See Table 1 below.

8. Automated Range Checks and Flagging of Violations
8.1 Implementation logic
For each variable with defined thresholds:
1. The column was coerced to numeric (with non-numeric values set to NA).
2. Two logical tests were applied per record:
o Below minimum: value < plausible_min
o Above maximum: value > plausible_max
3. Any record where either condition was true was flagged as a violation.

6 of 24

The output was a long-format table with one row per violation, containing:
•
Row index (zero-based index from the original sheet)
•
PLOT_ID
•
CORE
•
DEPTH
•
Variable
•
Observed Value
•
Issue (below_min or above_max)
•
Plausible_min and Plausible_max
A summary table of violation counts by variable and issue type was also generated.
8.2 Summary of flagged violations
The range-check process highlighted:
•
Negative or near-zero anomalies:
o Small negative values for FREE_ACIDITY and EXCHANGEABLE MANGANESE
(mg/L, mg/kg, cmol(+)/kg), all flagged as below_min with a lower bound of
0.0.
•
Values slightly above 100%:
o One LOI (%) at 375°C value slightly over 100%, flagged as above_max.
•
High exchangeable calcium values:
o Several records where EXCHANGEABLE CALCIUM (cmol(+)/kg) exceeded the
plausible maximum (80 cmol(+)/kg), flagged as above_max.
•
High potassium values:
o Isolated exceedances in EXCHANGEABLE POTASSIUM (mg/kg) and
EXCHANGEABLE POTASSIUM (mg/l), flagged as above_max.
•
Very low pH in CaCl₂:
o A subset of observations with SOIL_pH_CaCl2 < 3.0, flagged as below_min.
These are possible in extreme acid soils but were flagged in line with
conservative bounds chosen for data validation.
These flags are intended for review and documentation, not automatic deletion. For
example, extreme Ca or K levels may be real but warrant checking against site history and
lab notes.

9. Recommended Handling of Flagged Values
The QA did not automatically alter the original data but proposes the following handling
guidelines, and steps implemented shown:
1. Negative concentrations and negative acidity measures
o Treat as measurement noise around zero.
o Recommended actions:

7 of 24

▪
Either set to 0.0 (truncation) or to missing (NA), depending on
analysis requirements. Set to 0.0 implemented.
▪
Document the choice in metadata/analysis logs - implemented.
2. Percentages >100% (LOI, organic matter, saturation)
o Values slightly above 100% likely reflect minor analytical/rounding errors.
o Options:
▪
Cap at 100%; or
▪
Retain the original numbers but keep them flagged, and consider their
small magnitude unlikely to affect conclusions materially.
▪
Cap at 100% implemented.
3. Extremely high cation or nutrient values
o Should be reviewed case-by-case with reference to:
▪
Site management (e.g. heavy fertiliser or lime application) -
implemented,
▪
Codings/units (ensure mg/kg vs mg/L vs cmol(+)/kg are correct),
▪
Lab QA/QC reports – implemented.
o If confirmed as valid, they can be retained but remain flagged as extremes. -
implemented.
4. Very low pH (<3 in CaCl₂)
o Although rare, such values are physically possible. This was the case with all
low soil pH values for the forest soils. - Implemented.
o Recommended to cross-check a few most extreme cases against lab records or
replicate measurements. Implemented.

10. Reproducibility and Code
To ensure that QA can be re-run and extended, a standalone R script was prepared that:
1. Reads the Soil chemical data sheet from the Excel file.
2. Recodes "<2" values in extractable P to 1 mg/kg.
3. Defines the plausible threshold table for all key soil chemical variables.
4. Applies the range checks described above, generating:
o
soil_chem_threshold_violations.csv (all flagged records),
o
soil_chem_threshold_violation_counts.csv (summary counts).
5. Prints a summary of violation counts to the console.
This script can be:
•
Incorporated into a wider data ingestion/QC pipeline,
•
Reused for future years’ data,
•
Adapted to refine thresholds as new information (e.g. lab QC, field audits) becomes
available.

11. Summary

8 of 24

In summary, QA of the soil chemical data involved:
•
Checking distributions, extremes, and 95% confidence limits for all numeric
variables.
•
Explicitly listing min/max values by plot/core/depth to support traceability.
•
Identifying and characterising negative values and percentages >100%.
•
Defining physically plausible thresholds for all soil chemistry variables in the dataset.
•
Automatically flagging all records that fall outside these thresholds and exporting
them for review.
•
Providing reproducible R code so that the entire process is transparent, repeatable, and
modifiable.
This combination of statistical, physical, and traceability-oriented checks provides a robust
foundation for subsequent analysis and reporting based on the soil chemical data.

Table 1. A list of plausible thresholds for each variable on the basis for mechanised range
checks.

Variable
Observed
min
Observed
max
Plausible
min
Plausible
max
ALUMINIUM SATURATION (%)
0.007
91.085
0.0
100.0
BASE SATURATION (%)
6.001
99.972
0.0
100.0
CATION EXCHANGE
CAPACITYcmol(+)/kg
0.838
101.18
0.0
151.77
EXCHANGEABLE AlUMINIUM
(cmol(+)/kg)
0.004
16.263
0.0
25.0
EXCHANGEABLE AlUMINIUM
(mg/kg)
0.371
1462.567
0.0
3000.0
EXCHANGEABLE AlUMINIUM (mg/l) 0.031
118.012
0.0
200.0
EXCHANGEABLE CALCIUM
(cmol(+)/kg)
0.047
95.684
0.0
80.0
EXCHANGEABLE CALCIUM (mg/kg) 9.359
xx-xxxx-xxxx
20000.0
EXCHANGEABLE CALCIUM (mg/l)
0.775
1494.467
0.0
1500.0
EXCHANGEABLE IRON (cmol(+)/kg)
0.0
5.11
0.0
10.0
EXCHANGEABLE IRON (mg/kg)
0.0
951.257
0.0
5000.0
EXCHANGEABLE IRON (mg/l)
0.0
68.537
0.0
200.0
EXCHANGEABLE MAGNESIUM
(cmol(+)/kg)
0.027
16.056
0.0
20.0
EXCHANGEABLE MAGNESIUM
(mg/kg)
3.316
1951.244
0.0
3000.0
EXCHANGEABLE MAGNESIUM
(mg/l)
0.275
142.245
0.0
300.0

9 of 24

Variable
Observed
min
Observed
max
Plausible
min
Plausible
max
EXCHANGEABLE MANGANESE
(cmol(+)/kg)
-0.0
1.558
0.0
5.0
EXCHANGEABLE MANGANESE
(mg/kg)
-0.054
427.835
0.0
3000.0
EXCHANGEABLE MANGANESE
(mg/l)
-0.004
35.038
0.0
200.0
EXCHANGEABLE POTASSIUM
(cmol(+)/kg)
0.021
4.513
0.0
5.0
EXCHANGEABLE POTASSIUM
(mg/kg)
8.164
1764.514
0.0
1500.0
EXCHANGEABLE POTASSIUM (mg/l) 0.679
137.759
0.0
100.0
EXCHANGEABLE SODIUM
(cmol(+)/kg)
0.009
4.608
0.0
10.0
EXCHANGEABLE SODIUM (mg/kg)
1.97
1059.473
0.0
2000.0
EXCHANGEABLE SODIUM (mg/l)
0.163
83.483
0.0
200.0
EXTRACTABLE PHOSPOROUS (PO4)
mg/kg) (with '<2' treated as 1)
0.0
303.11
0.0
400.0
FREE_ACIDITY cmol(+)/kg
-0.003
5.988
0.0
8.981
LOI (%) at 375°C
0.0
100.62
0.0
100.0
SOIL ORGANIC MATTER (%)
0.487
96.873
0.0
100.0
SOIL_ pH_H20
3.59
8.71
3.0
9.5
SOIL_pH_CaCl2
2.54
7.95
3.0
8.5
TOTAL ACIDITY cmol(+)/kg
0.005
18.909
0.0
28.364
TOTAL CARBON (%)
0.243
58.364
0.0
60.0
TOTAL INORGANIC CARBON (%)
0.0
11.322
0.0
20.0
TOTAL NITROGEN (%)
0.0
3.123
0.0
5.0
TOTAL ORGANIC CARBON (%)
0.243
58.364
0.0
60.0

10 of 24

Quality Assurance Report – Hot/Cold
Water Extractable Carbon
(HWEC/CWEC)
Elena Vanguelova, Forest Research, 08/12/2025
NCEA Soil Condition Data 2023–24 – “Soil chemical data” sheet

1. Overview
This report documents the QA checks applied to the new extractable carbon variables
added to the Soil chemical data sheet in:
•
File: NCEA Soil Condition Data 23 24.xlsx
•
Sheet: Soil chemical data
The focus is on the following five variables:
1. HOT WATER EXTRACTABLE CARBON (HWEC) (mg/l)
2. COLD WATER EXTRACTABLE CARBON (HWEC) (mg/l)
3. HOT WATER EXTRACTABLE CARBON (HWEC) (mg/kg)
4. COLD WATER EXTRACTABLE CARBON (HWEC) (mg/kg)
5. RATIO HWEC/CWEC
The QA objectives were to:
•
Check basic data integrity (types, negatives, missing values).
•
Characterise distributional properties (mean, SD, min, max, 95% CL).
•
Identify statistical outliers (IQR and 3×SD rules).
•
Define and apply plausible physical ranges (min/max thresholds).
•
Produce traceable lists of flagged values (by PLOT_ID / CORE / DEPTH).

2. Data source and structure
2.1 Location and identifiers
The HWEC/CWEC variables are stored in the Soil chemical data sheet alongside
standard chemistry variables (pH, C, N, CEC, etc.).
Key identifier columns used for QA reporting:

11 of 24

•
PLOT_ID – plot identifier
•
CORE – core identifier within plot
•
DEPTH – depth interval (e.g. 0–5, 5–10, 20–40)
These act purely as metadata, not used in numeric calculations, but retained in all QA
outputs to locate flagged records.
2.2 Variables
New variables:
1. Solution concentrations (mg/L):
o
HOT WATER EXTRACTABLE CARBON (HWEC) (mg/l)
o
COLD WATER EXTRACTABLE CARBON (HWEC) (mg/l)
2. Soil-basis concentrations (mg/kg dry soil):
o
HOT WATER EXTRACTABLE CARBON (HWEC) (mg/kg)
o
COLD WATER EXTRACTABLE CARBON (HWEC) (mg/kg)
3. Ratio of HWEC to CWEC:
o
RATIO HWEC/CWEC = HWEC / CWEC (dimensionless)
All five are treated as continuous numeric variables.

3. Pre-processing and basic checks
1. Type coercion
All five HWEC/CWEC variables were coerced to numeric. Any non-numeric entries
(if present) would become NA.
2. Missing values
o Each variable has a number of NA values where measurements were not made.
o No special censored codes (e.g. "<2") are used for HWEC/CWEC, so no
special recoding was required.
3. Negatives and implausible codes
During initial inspection:
o One negative value was found in COLD WATER EXTRACTABLE CARBON
(HWEC) (mg/kg) (~−16.3 mg/kg).
o One negative value in RATIO HWEC/CWEC (~−0.55).
These are non-physical and were later flagged via threshold checks.
No other obvious coding artefacts (e.g. text values, swapped units) were observed.

4. Descriptive statistics and distributions
For each HWEC/CWEC variable, the following were computed (N ≈ 1949 observations):
•
N (non-missing count)

12 of 24

•
Mean and standard deviation (SD)
•
Minimum and maximum
•
95% confidence limits (CL) of the mean
•
IQR (Q1, Q3), used later for IQR-based outliers.
4.1 Summary patterns
•
HOT WATER EC (mg/L)
o Mean ≈ 16 mg/L, SD ≈ 17 mg/L.
o Range ≈ 0.4–342 mg/L.
o Mild to strong right-skew; a number of high values but generally moderate.
•
COLD WATER EC (mg/L)
o Mean ≈ 211 mg/L, SD ≈ 197 mg/L.
o Range ≈ 6–1477 mg/L.
o Strongly right-skewed with many high values, reflecting larger dissolved C in
cold extracts.
•
HOT WATER EC (mg/kg)
o Mean ≈ 177 mg/kg, SD ≈ 259 mg/kg.
o Range ≈ 2.7–3607 mg/kg.
o Long right tail; a number of high values reflecting highly C-rich soils.
•
COLD WATER EC (mg/kg)
o Mean ≈ 3140 mg/kg, SD ≈ 5090 mg/kg.
o Range ≈ −16 to ~48,686 mg/kg.
o Highly right-skewed with a long tail of very large values; one non-physical
negative value.
•
RATIO HWEC/CWEC
o Mean ≈ 16.5, SD ≈ 11.0.
o Range ≈ −0.55–94.6.
o Most values are positive and moderate, but there is a long tail of high ratios
(up to ~95) and one small negative.
These distributions inform both the statistical outlier detection and the setting of plausible
thresholds.

5. Plausible-range (threshold) checks
5.1 Threshold definitions
Plausible ranges were defined to capture physically reasonable values for typical soils, while
still allowing for highly organic or fertilised conditions. These ranges are used only for
flagging (not automatic deletion):
Variable
Plausible
min
Plausible
max
Notes
HOT WATER EXTRACTABLE
CARBON (HWEC) (mg/l)
0
2,000
Must not be negative; 2,000 mg/L
is a generous upper bound for
solution C

13 of 24

Variable
Plausible
min
Plausible
max
Notes
COLD WATER
EXTRACTABLE CARBON
(HWEC) (mg/l)
0
2,000
Same reasoning as above
HOT WATER EXTRACTABLE
CARBON (HWEC) (mg/kg)
0
20,000
Allows up to ~2% of soil mass as
hot-water-extractable C
COLD WATER
EXTRACTABLE CARBON
(HWEC) (mg/kg)
0
20,000
Same, but for cold-water
extractable C
RATIO HWEC/CWEC
0
100
Ratios should not be negative; 0–
100 covers observed range plus
some headroom
5.2 Threshold-check procedure
For each HWEC/CWEC variable:
1. Convert to numeric.
2. For each record:
o If value < plausible_min → flag as below_min.
o If value > plausible_max → flag as above_max.
3. For each flagged record, record:
o
PLOT_ID, CORE, DEPTH
o
Variable, Value
o
Issue (below_min or above_max)
o
Plausible_min, Plausible_max
All such rows are included in soil_QC_threshold_violations.csv and summarised by
variable/issue in soil_QC_violation_counts.csv.
5.3 Threshold violations – summary
Threshold violations among the HWEC/CWEC variables are:
Variable
Issue
Count
Interpretation
HOT WATER EC (mg/L)
none
0
All within 0–2000 mg/L
COLD WATER EC (mg/L) none
0
All within 0–2000 mg/L
HOT WATER EC (mg/kg) none
0
All within 0–20,000 mg/kg
COLD WATER EC
(mg/kg)
below_min 1
One negative value (non-physical)
COLD WATER EC
(mg/kg)
above_max 35
35 values >20,000 mg/kg (very high
soluble C)
RATIO HWEC/CWEC
below_min 1
One negative ratio, non-physical
RATIO HWEC/CWEC
above_max 0
None above 100

14 of 24

Key points:
•
The single negative cold-water EC (mg/kg) and negative ratio are almost certainly
measurement or calculation errors and was reviewed and corected.
•
The 35 very high cold-water EC (mg/kg) values (>20,000 mg/kg, max ~48,686
mg/kg) are extreme but could reflect:
o Very high organic carbon content, which was the case in the forest soil dataset
- implemented
o Very low dry soil mass used in calculation, or
o A formula/unit error.
These should be cross-checked carefully against lab sheets and calculation
steps.

6. Statistical outlier analysis (HWEC/CWEC)
In addition to threshold checks, statistical outliers were identified using:
1. IQR rule – values outside Q1 ± 1.5·IQR.
2. 3×SD rule – values outside mean ± 3·SD.
These were computed for all numeric variables in Soil chemical data, and HWEC/CWEC
variables are a subset of those results. For HWEC/CWEC:
•
All variables show strong right-skew, especially the mg/kg variables and the ratio.
•
Large numbers of observations were flagged as outliers by the IQR rule (especially in
the upper tail), and a smaller subset by the 3×SD rule.
This does not mean these values are wrong—it simply flags them as statistically extreme
relative to the majority of the dataset. Because HWEC/CWEC measures naturally vary with
organic matter and management, these extremes are real due to the high carbon content in the
soils samples as well as the natural high variability in carbon.
All outlier records (including identifiers and bounds) are stored in:
•
soil_QC_statistical_outliers.csv
•
Summary counts by variable and method are in
soil_QC_statistical_outlier_counts.csv.

7. Recommended handling of HWEC/CWEC QA flags
7.1 Non-physical values
•
Negative concentrations (COLD WATER EC mg/kg) and negative ratio:
o Inspect original lab and calculation records.
o If they result from small rounding or subtraction errors near zero, then:

15 of 24

▪
Set to 0 (implemented) or NA, and
▪
Document the decision in QA notes (implemented).
o They should not be used as-is in analysis.
7.2 Very high COLD WATER EC (mg/kg) values
•
The 35 values >20,000 mg/kg (with maximum ~48,686 mg/kg) should be:
1. Checked for:
▪
Correct units (mg/kg vs mg/L or %) (implemented)
▪
Correct soil mass and extract volume used in calculations,
(implemented)
▪
Data entry errors (e.g. misplaced decimal points, duplicated factors).
(implemented)
2. If confirmed as valid:
▪
Retained, but always treated as extreme values in
analysis.(implemented)
▪
Consider log-transformation or robust methods to avoid
overweighting them.
7.3 Ratio HWEC/CWEC
•
Ratios close to zero are plausible when CWEC ≫ HWEC.
•
Very large ratios (up to ~95) and the negative value should be scrutinised for:
o Small denominators (very low CWEC → inflates ratio),
o Calculation errors (e.g. reversed ratio, misaligned units).
•
For modelling, consider:
o Working directly with HWEC and CWEC instead of the ratio, or
o Censoring or winsorising extreme ratios.
7.4 General modelling guidance
•
Because HWEC/CWEC variables are strongly skewed with many large outliers, for
downstream modelling/analysis it is recommended to:
o Apply log transforms (e.g. log(HWEC mg/kg + 1), log(CWEC mg/kg + 1)).
o Use robust regression or quantile-based summaries rather than methods
assuming normality.
o Run sensitivity analyses with and without extreme HWEC/CWEC values.

8. Reproducibility and script implementation
All QA steps for HWEC/CWEC are implemented in the R script:
•
qa_soils_extended.R
Within this script:
1. The Soil chemical data sheet is read.

16 of 24

2. HWEC/CWEC variables are coerced to numeric.
3. Plausible thresholds (Section 5.1) are defined and applied, producing:
o
soil_QC_threshold_violations.csv
o
soil_QC_violation_counts.csv
4. Statistical outliers for all numeric variables (including HWEC/CWEC) are computed
via IQR and 3×SD rules, producing:
o
soil_QC_statistical_outliers.csv
o
soil_QC_statistical_outlier_counts.csv
This ensures that QA for HWEC/CWEC can be:
•
Re-run on updated data,
•
Audited and extended (e.g. adjusted thresholds), and
•
Integrated into a larger data ingestion/QA pipeline.

9. Summary
For the HWEC/CWEC variables, the QA process:
•
Confirmed that most values lie in agronomically plausible ranges, but
•
Identified:
o One negative cold-water EC (mg/kg) and one negative ratio, both non-
physical, which were corrected in the database.
o 35 extremely high cold-water EC (mg/kg) values above 20,000 mg/kg,
which were checked and explained by the high soil carbon content in the
samples.
o Many statistically extreme high-end values (especially in mg/kg and the ratio)
due to strong skew.
The recommended approach is to:
•
Correct or discard non-physical values after verification.
•
Investigate and document the very high cold-water EC values.
•
Use transformations and robust methods in analysis to accommodate genuine but
extreme variability in HWEC/CWEC.
This provides a transparent, reproducible QA framework for the new extractable carbon
variables in the NCEA soil condition dataset.

17 of 24

Quality Assurance Report – Soil Physical
Data
Elena Vanguelova, Forest Research, 08/12/2025
NCEA Soil Condition Data 2023–24 – “Soil physical data” sheet

1. Overview
This report documents the quality assurance (QA) checks applied to the soil physical data in
the workbook:
•
File: NCEA Soil Condition Data 23 24.xlsx
•
Sheet: Soil physical data
The QA objectives were to:
1. Verify that all physical measurements fall within physically plausible ranges.
2. Identify extreme and anomalous values using statistical outlier methods.
3. Provide a traceable list of flagged records (with PLOT_ID / CORE / DEPTH).
4. Implement a reproducible workflow via the qa_soils_extended.R script.

2. Data source and structure
2.1 Sheet and identifiers
Sheet: Soil physical data
Key identifier fields:
•
PLOT_ID – plot identifier
•
CORE – core identifier within plot
•
DEPTH – depth interval (e.g. 0-15, 15-30)
These three fields are treated as metadata and are never transformed or used in numeric QC
calculations; they are only used to identify and report problematic records.
2.2 Physical variables
The physical variables QA’d are:
1. STONE WEIGHT >50mm (g)

18 of 24

2. VOLUME OF STONES >2mm (cm3)
3. FIELD MOISTURE CONTENT (%) FRESH
4. FIELD MOISTURE CONTENT (%) DRY
5. BULK DENSITY (g/cm3)
6. SOIL DENSITY (g/cm3)
All of these were treated as continuous numeric variables.

3. Pre-processing
•
The Soil physical data sheet was read directly from the Excel file with no recoding of
variable names.
•
All six physical variables were coerced to numeric types (where necessary) using
standard conversion; non-numeric strings (if any) would become NA.
•
No special text codes (e.g. “<2”) exist in this sheet, so no censoring logic was
required.

4. Descriptive statistics and distribution checks
For each of the six physical variables, the following were computed:
•
N (number of non-missing observations)
•
Mean and standard deviation (SD)
•
Minimum and maximum values
•
25th percentile (Q1) and 75th percentile (Q3)
•
95% confidence limits (CL) for the mean, using:
CL95 = 𝑥ˉ ± 1.96 × SD
√𝑁

These statistics were used to:
•
Understand the central tendency and spread of each variable.
•
Identify variables with strong skewness or long tails (e.g. Field moisture content
(dry), bulk and soil density).
•
Inform the definition of plausible minimum and maximum ranges (Section 5).
Qualitative observations from the distributions:
•
Stone weight and volume: Many cores with zero stones and some with substantial
stone contents. No values obviously beyond what could be produced in a standard
core volume.
•
Field moisture content (fresh): Mostly moderate values, with a few very wet
samples approaching saturation.

19 of 24

•
Bulk density: Most values cluster within typical mineral-soil ranges (~0.7–1.8
g/cm³), with a few unusually low and unusually high values.
•
Soil density: Typical particle density is expected around ~2.6 g/cm³; the data include
values both far below and above this, flagging potential calculation issues or atypical
material composition.

5. Plausible-range (threshold) checks
5.1 Threshold definitions
For each physical variable, a plausible minimum and maximum was defined based on
typical soil physics and the observed data. These are data-validation bounds (for flagging),
not agronomic targets.
The thresholds used:
Variable
Plausible
min
Plausible
max
Rationale (brief)
STONE WEIGHT
>50mm (g)
0.0
1000.0
Cannot be negative; 1 kg stone mass per
core treated as upper bound for flagging
VOLUME OF STONES
>2mm (cm3)
0.0
500.0
Cannot be negative; upper bound ~half
litre stone volume per core
FIELD MOISTURE
CONTENT (%) FRESH 0.0
100.0
Negative impossible; 100% covers from
oven-dry to fully water-saturated
BULK DENSITY
(g/cm3)
0.1
2.2
Realistic range for intact cores from
organic to compact mineral soils
SOIL DENSITY (g/cm3) 0.5
3.0
Reasonable particle densities for
soils/minerals; outside suggests issues

Values outside these ranges are not automatically wrong, but they are flagged for review
because they are unusual or non-physical.
5.2 Threshold-check procedure
For each variable and each record:
•
Convert the variable to numeric.
•
If the value is less than the plausible minimum, flag as below_min.
•
If the value is greater than the plausible maximum, flag as above_max.
For every flagged record, we stored:
•
sheet = "Soil physical data"
•
row_index = zero-based row index from the sheet (to line up with R script outputs)

20 of 24

•
PLOT_ID, CORE, DEPTH
•
Variable, Value
•
Issue (below_min or above_max)
•
Plausible_min, Plausible_max
All such records were combined into soil_QC_threshold_violations.csv, and
summarised (by variable and issue) in soil_QC_violation_counts.csv.
5.3 Threshold violations – summary
From the physical variables, the pattern of threshold violations was:
•
Stone variables
o
STONE WEIGHT >50mm (g): no threshold violations (all between 0 and 1000
g).
o
VOLUME OF STONES >2mm (cm3): no threshold violations (all between 0 and
500 cm³).
•
Field moisture content
o
FIELD MOISTURE CONTENT (%) FRESH: all within 0–100%; no violations.
•
Bulk density
o
BULK DENSITY (g/cm3):
▪
1 record below 0.1 g/cm³ (implausibly low for bulk density, similar to
extremely fluffy organic material or a calculation error).
▪
104 records above 2.2 g/cm³ (unusually dense; may reflect
compaction, stone influence, or calculation issues).
•
Soil density
o
SOIL DENSITY (g/cm3):
▪
93 records below 0.5 g/cm³ (physically unrealistic as a true particle
density; strongly suggests errors or misinterpretation of this variable).
▪
34 records above 3.0 g/cm³ (very high; values much above typical
mineral density).
These threshold results highlight where data review and possible correction should be
prioritised, especially for bulk and soil densities.

6. Statistical outlier analysis
In addition to hard thresholds, statistical outliers were identified for all numeric variables
(chemistry and physical). Here we summarise the physical-side procedure.
6.1 Variables included
For the Soil physical data sheet, numeric columns included:
•
STONE WEIGHT >50mm (g)
•
VOLUME OF STONES >2mm (cm3)
•
FIELD MOISTURE CONTENT (%) FRESH

21 of 24

•
BULK DENSITY (g/cm3)
•
SOIL DENSITY (g/cm3)
PLOT_ID and CORE (numeric IDs) were explicitly excluded from the outlier computation, as
they are identifiers rather than measurements.
6.2 Methods
For each numeric variable:
1. IQR rule (boxplot rule)
o Compute Q1 (25th percentile), Q3 (75th percentile), IQR = Q3 − Q1.
o Flag as IQR outliers any values where:
▪
𝑥< 𝑄1 −1.5 × 𝐼𝑄𝑅, or
▪
𝑥> 𝑄3 + 1.5 × 𝐼𝑄𝑅.
o Recorded with Method = "IQR" and Rule = "outside_1.5*IQR".
2. 3×SD rule
o Compute mean (𝑥ˉ) and SD.
o Flag as SD outliers any values where:
▪
𝑥< 𝑥ˉ −3 × 𝑆𝐷, or
▪
𝑥> 𝑥ˉ + 3 × 𝑆𝐷.
o Recorded with Method = "SD3" and Rule = "outside_3*SD".
For every outlying record, the following were stored:
•
sheet = "Soil physical data"
•
row_index, PLOT_ID, CORE, DEPTH
•
Variable, Value
•
Method ("IQR" or "SD3")
•
Rule (IQR or 3×SD description)
•
Lower_bound, Upper_bound (the numeric thresholds that were exceeded)
•
Mean, SD, Q1, Q3 (for context)
All statistical outliers (chem + physical) are in soil_QC_statistical_outliers.csv. A
summary count by sheet / variable / method is in
soil_QC_statistical_outlier_counts.csv.
6.3 Interpretation for physical variables
•
Stone weight & volume
o A subset of samples with very high stone contents were flagged as outliers
(IQR and/or 3×SD), but these remain physically possible and may simply
represent stony horizons.
•
Field moisture content (fresh)
o This variable had some numbers of upper outliers
o These are consistent with soils that were extremely wet at sampling and/or
have very high organic matter (low dry weight), so they are not necessarily
errors but are extreme cases.
•
Bulk and soil density

22 of 24

o Many of the same records flagged by threshold checks were also flagged as
statistical outliers.
o Values far below 0.5 g/cm³ for soil density or below 0.1 g/cm³ for bulk
density, and values well above usual soil limits, are prime candidates for
detailed review (e.g. checking core volume, stone correction, or data entry).

7. Recommended handling of flagged physical values
Based on the QA results, the following handling strategy is recommended:
1. Hard non-physical values (bulk/soil density)
o Very low densities (e.g. <0.5 g/cm³ for particle density, <0.1 g/cm³ bulk
density) should be scrutinised against field and lab records. (implemented)
o If confirmed as errors, they should be corrected where possible or set to
missing (NA) and documented (implemented).
o There was no need for correction as all low bulk density were related to highly
organic soils (implemented).
2. Very high densities (>2.2 bulk, >3.0 soil)
o Review for:
▪
Incorrect core volume or mass used in calculations (implemented)
▪
Stones not correctly accounted for (implemented).
▪
Data entry or unit errors (implemented).
o Retain but keep as flagged extremes until verified (implemented).
o After all the checks, all the samples with very high-density values had to be
recalculated with updated volume of the sampling core device (implemented).
o Bulk density values greater than 2.2 g/cm³ were initially flagged during QA as
exceeding a conservative plausibility threshold. However, on review, these
high values were retained in the database because they are consistent with
extremely dense, stone-rich or compacted horizons that can occur under real
field conditions. The 2.2 g/cm³ limit is therefore treated as a soft QA flag
rather than a censoring boundary. Removing or truncating these values would
bias estimates of soil mass and carbon stocks and underrepresent the true
variability of soil physical conditions. Instead, the high values are kept,
documented as flaggable extremes, and handled using robust or sensitivity-
based approaches in downstream analysis
3. Field moisture extremes
o High “fresh” moisture may be real in organic soils; consider them as extreme
but plausible.
o For modelling, consider:
▪
Transformations (e.g. log scale), and/or
▪
Robust statistical methods resistant to heavy tails.
4. Stone variables
o Outliers are likely real differences in stoniness; retain but recognise they can
be influential in models involving bulk density or water storage.

23 of 24

8. Reproducibility and code
All steps described above are implemented in the R script:
•
qa_soils_extended.R
This script:
1. Reads the Soil physical data sheet from the Excel workbook.
2. Defines plausible thresholds for each physical variable.
3. Performs the threshold checks and exports:
o
soil_QC_threshold_violations.csv
o
soil_QC_violation_counts.csv
4. Runs statistical outlier detection (IQR and 3×SD) on all numeric physical variables
and exports:
o
soil_QC_statistical_outliers.csv
o
soil_QC_statistical_outlier_counts.csv
This ensures the QA can be:
•
Re-run on updated datasets (e.g. new years, new plots).
•
Audited and extended (e.g. adjusting thresholds or adding new variables).

9. Summary
For the Soil physical data sheet, QA included:
•
Computing descriptive statistics and 95% confidence limits to understand each
variable’s distribution.
•
Defining and applying physically plausible ranges to all physical variables.
•
Flagging implausible or extreme measurements, particularly:
o Very low and very high bulk densities
o Very low and very high soil densities
•
Applying IQR and 3×SD outlier rules to identify statistically extreme observations
in stone content, moisture, and densities.
•
Exporting fully traceable lists of all flagged records for manual review and
documenting the process in a reusable R script.
This provides a robust and transparent QA framework for the soil physical component of the
NCEA soil condition dataset.

When you run R:
source("qa_soils_full_QA.R")
(in the same folder as NCEA Soil Condition Data 23 24.xlsx), it will:

24 of 24

1. Read Soil chemical data and Soil physical data.
2. Recode "<2" in extractable P to 1 mg/kg.
3. Compute descriptive statistics (including 95% confidence limits) for all numeric
variables →
soil_QC_descriptives.csv
4. Identify min and max extremes for each numeric variable with PLOT_ID / CORE /
DEPTH →
soil_QC_extremes.csv
5. Apply plausible min/max thresholds to:
o All chemical variables (including HWEC/CWEC)
o All physical variables
and write:
o
soil_QC_threshold_violations.csv
o
soil_QC_violation_counts.csv
6. Flag statistical outliers (IQR rule and 3×SD rule) for all numeric variables and write:
o
soil_QC_statistical_outliers.csv
o
soil_QC_statistical_outlier_counts.csv
That’s the full QA pipeline into one reusable R file.

Data Blanks and Zeros
1. Blanks in the dataset indicate missing values – either because the measurement was
not made, the sample was unusable, or the value was not available at the time of data
collation.
2. Zeros in stone variables (stone weight and volume) reflect true zero observations
(no stones observed).
3. Zeros in chemical variables generally indicate very low or below-detection
concentrations that have been reported as zero, not necessarily an absolute absence
of that element.
Practical recommendations:
•
Treat blanks as NA for all analyses.
•
Treat stone zeros as true zeros.
•
For chemical zeros:
o If you have lab metadata confirming zeros = “below DL”, you can:
▪
leave as 0 for descriptive stats, or
▪
replace with half the detection limit if you’re doing log-transforms or
modelling. This was done for soil phosphorous content in the NCEA
Soil condition database (implemented).