Linearity vs Accuracy vs Resolution in Sensors: How to Compare the Specs
Linearity tells you about curve shape, accuracy tells you how close a result is to a suitable reference, and resolution tells you the smallest change the system can distinguish. A good sensor decision needs all three in the right context—plus range, noise, speed, environment, calibration evidence, and the controller that receives the signal.
Use resolution to screen whether a meaningful change can be seen, accuracy or a defined total-error limit to screen whether the reported value can support the decision, and linearity to understand how the transfer curve departs from its stated line across the used range.
What is the difference between sensor linearity, accuracy and resolution?
A sensor may resolve very small increments but report the wrong absolute value. It may be highly linear but shifted by offset. It may be correct at its two calibration endpoints but still bend away from the reference line in the middle.
Linearity
How closely the measured transfer curve follows a defined straight reference line over a stated range.
Decision meaning: curve shapeAccuracy
In formal metrology, closeness of agreement with the measurand's true quantity value; in datasheets, usually a stated permitted error under defined conditions.
Decision meaning: correctness under conditionsResolution
The smallest change in the measured quantity that causes a perceptible change in the indication. Noise, friction, filtering, and operating point can limit it.
Decision meaning: visible detailTerminology note: the BIPM/JCGM VIM definition treats measurement accuracy as a qualitative concept, not a numerical quantity. When a supplier publishes “±0.5% accuracy,” read the definition and footnotes to learn which error limit that number actually represents.
Which specification should drive the sensor selection?
Start with the production decision in physical units. The table turns that decision into a recommendation, evidence request, and stop boundary.
| Production condition | Prioritize | Evidence to request | Do not approve yet if |
|---|---|---|---|
| Detect a small change or trend | Usable resolution, noise, repeatability, bandwidth | Noise at the required data rate and filter; repeat traces; output and controller resolution | Only nominal bits or display decimals are given |
| Make an absolute pass/fail decision | Defined error limit or total error band | Error equation, included effects, temperature range, calibration state, and traceable test data | “Accuracy” has no basis, conditions, or maximum limit |
| Use values across a wide range | Linearity plus offset, gain, hysteresis, drift | Reference-line method and distributed calibration points across the used range | Only zero and full-scale results are shown |
| Control a fast process | Dynamic response and end-to-end latency | Response or settling definition, update rate, filtering, and controller sampling | A stable bench value is supplied without timing data |
| Switch presence rather than measure a value | Assured distance, repeatability, hysteresis, target and mounting margin | Application-specific sensing-distance and environmental data | Linearity or bit depth is used as the main selection argument |
Fast buying rule: write the smallest meaningful process change and the maximum permitted decision error first. If the supplier cannot translate its claims into those units at your operating points, the comparison is not ready.
Why must the whole measurement chain be checked?
The machine does not act on the sensing element alone. It acts on a value that has passed through electronics, an output interface, wiring, a controller input, software scaling, filtering, and a decision rule. Each stage can add error, noise, delay, or a configuration mistake.
- 01MeasurandDistance, pressure, force, level, flow, temperature, or position
- 02Sensing elementConverts the physical quantity into an internal signal
- 03Sensor electronicsCompensation, conversion, filtering, and output scaling
- 04Output and cableCurrent, voltage, data, grounding, load, and interference
- 05Controller inputInput range, accuracy, ADC, sampling, and update timing
- 06SoftwareEngineering-unit scaling, rounding, filtering, and alarms
- 07Machine decisionControl action, inspection result, trend, or rejection
Boundary: a sensor calibration certificate does not validate the fixture, target, cable route, PLC input card, scaling formula, process connection, or final decision logic. Freeze and test the assembled chain.
What does a sensor linearity specification actually prove?
It describes the largest departure of the measured transfer curve from a stated straight reference line over a stated range. It does not, by itself, prove correct zero, correct span, low hysteresis, low noise, or total system accuracy.
Which reference line was used?
An end-point line passes through defined endpoints. A best-fit straight line minimizes the overall deviations across the tested points. These methods can produce different numbers from the same calibration data, so do not compare percentages until the method matches.
Which error components are separate?
Offset moves the response up or down. Gain error changes its slope. Nonlinearity bends the response. Hysteresis makes the reading depend on approach direction. A two-point adjustment may correct stable offset and gain while leaving curvature and hysteresis.
- Ask for used range, point locations, approach direction, temperature, correction state, and whether the value is typical or maximum.
- Give extra weight to the points where the machine actually accepts or rejects production.
How should buyers read a sensor accuracy claim?
Treat the headline percentage as an incomplete equation. You need the basis, range, included effects, operating conditions, calibration state, and whether the figure is a guaranteed maximum or only typical performance.
Why are % full scale and % of reading different?
For a 0–100 bar range, ±1% of full scale is approximately ±1 bar throughout the range. At a 10 bar operating point, that equals 10% of the reading. By contrast, ±1% of reading at 10 bar is ±0.1 bar—unless the supplier also adds a fixed, digit, zero, or full-scale term.
How is accuracy different from uncertainty?
Measurement uncertainty describes the dispersion of values that could reasonably be attributed to the measurand from the information used. Independent standard uncertainty components may be combined using the law of propagation of uncertainty, with covariance included when relevant. Expanded uncertainty then requires a stated coverage factor.
Do not simply combine every datasheet tolerance by root-sum-square. Some figures are limits rather than standard uncertainties, distributions may differ, correlations may matter, and the acceptance rule may need a guard band.
Escalate the decision: use a competent calibration laboratory or metrology professional when regulatory compliance, contractual acceptance, safety, or a narrow product tolerance depends on the result.
Why do more bits not guarantee usable sensor resolution?
Bit depth describes the number of available digital codes. It does not prove that the sensing physics, electronics, installation, and process are stable enough to carry useful information in every code.
What does nominal code width tell you?
For an ideal N-bit converter with configured input span R, the nominal code width is:
The value excludes converter gain and offset error, integral and differential nonlinearity, reference error, input settling, source impedance, electrical noise, and the sensor's own variation.
What limits usable detail?
If the output wanders across 30 codes, one-code display increments do not mean the machine can reliably resolve one code. Filtering can reduce visible noise, but it also adds delay and may hide fast changes. Check peak-to-peak noise or noise-free resolution, repeatability, data rate, bandwidth, and settling together.
Readings of 1, 51, and 101 mm against references of 0, 50, and 100 mm form a straight response with a 1 mm offset.
A 16-bit output over 100 mm has a 0.0015 mm ideal step, but a ±0.10 mm sensor limit cannot support a 0.001 mm absolute acceptance decision.
A two-point calibration can make 0 and 100 bar correct while a mid-range reading remains 1.2 bar high because the curve still bends.
How can you convert the datasheet values into machine units?
This screening tool compares nominal code width and observed noise with the smallest change you need to see. It separately compares the entered accuracy allowance with the permitted absolute error. It does not calculate total system error or measurement uncertainty.
Use one physical unit consistently: mm, bar, N, °C, litres, or another engineering unit.
The stated accuracy allowance is larger than the permitted absolute error. Review the complete error definition, a narrower calibrated range, or a different measurement solution.
Calculator boundary: linearity is displayed but not added to accuracy because the supplier may already include it. Combine terms only after their definitions, distributions, correlations, and inclusion rules are known.
What should a buyer verify in the datasheet and supplier file?
Every important claim must trace to the exact model, output version, range, firmware or filter configuration, and delivered documentation. A family brochure is not enough when suffixes change the measurement chain.
Minimum supplier document checklist
| Item | What must be explicit | Why it changes approval |
|---|---|---|
| Exact product identity | Model, suffix, range, output, cable or connector, firmware, filter, and correction state | Family-level figures may not cover the ordered configuration |
| Linearity | Reference-line method, used range, points, direction, hysteresis treatment, typical or maximum status | The same curve can produce different headline values under different methods |
| Accuracy or error band | %FS, %reading, fixed terms, included effects, temperature, warm-up, supply, load, and calibration state | A percentage cannot be converted into a machine limit without its complete basis |
| Resolution and dynamics | Physical resolution, output bits, noise, repeatability, data rate, bandwidth, response and settling definitions | Nominal codes can overstate stable detail or hide excessive latency |
| Calibration evidence | Serial number, points, as-found/as-left results, reference, uncertainty, traceability, date, and interval | “Calibrated” alone does not show performance at critical points |
| Change control | Revision identifiers and notification rules for electronics, firmware, compensation, output scaling, or materials | An approved sample can change without a controlled configuration |
Illustrative supplier-file review: approve or hold?
The following example is fictional and shows the review method, not a real supplier result.
| File check | Evidence received | Review decision |
|---|---|---|
| Model match | Quotation states AX-100-10V-F2; certificate states AX-100 series | HoldF2 filter suffix is not explicitly covered |
| Accuracy definition | ±0.2% FS “typical” at 23 °C | HoldNo maximum limit or operating-temperature effect |
| Linearity | ±0.1% FS, best-fit straight line, increasing direction | Method is stated; request hysteresis and decreasing-direction data |
| Resolution and timing | 16-bit output; response time shown for filter F0 only | HoldF2 noise and latency are unknown |
| Traceable result | Five calibration points for one sample; no controller included | Useful sensor evidence, but machine-chain validation remains open |
Decision: do not approve yet. The quoted percentage could fit the ±0.25 mm limit at reference temperature, but the maximum error, F2 configuration, temperature effect, timing, hysteresis, and installed controller path are not closed.
How should the installed sensor be validated on the machine?
Test where the machine makes its decision. A one-point check can reveal offset at that point; a two-point calibration can correct much of a stable zero and span error. Neither proves the middle of the range, both approach directions, dynamic behavior, or robustness to real environmental influences.
- 1Define the acceptance decision
State the measurand, used range, critical points, permitted error, smallest required change, speed, and reference method.
- 2Freeze the configuration
Record exact sensor, mode, filter, output, cable, controller channel, scaling, firmware, fixture, target, and process connection.
- 3Test distributed points
Include critical operating points and relevant endpoints, approach from both directions, repeat readings, and use realistic dwell or motion.
- 4Challenge real influences
Repeat under expected temperature, vibration, target or medium variation, contamination, supply, load, electrical noise, and speed.
- 5Document the decision rule
Record reference uncertainty, guard band when required, results, failures, accepted configuration, and revalidation triggers.
What should a sensor RFQ and acceptance request include?
Replace “high accuracy” with a measurable application boundary. Give the supplier the real operating conditions, then require every important claim to trace to the exact quoted configuration.
Define the measurement task
- Measurand, engineering unit, used range and critical operating points
- Smallest meaningful change and maximum permitted error in physical units
- Target, medium, process connection, installation drawing and motion direction
- Response time, settling, bandwidth, update rate and end-to-end latency
- Temperature, vibration, contamination, EMI, cable route and controller input
Require model-specific evidence
- Exact part number, suffix, output, range, firmware, filter and document revision
- Accuracy equation, included effects, reference conditions and maximum limits
- Linearity method, used range, direction, hysteresis and correction assumptions
- Resolution and noise data at the proposed data rate and filter setting
- Calibration points, reference uncertainty, sample test and pass/fail rule
Acceptance boundary: supplier data can screen and qualify a sensor, but the machine builder or responsible integrator must close installation, controller scaling, performance, and final decision evidence.
What else do buyers ask about sensor specifications?
Can calibration improve accuracy without improving linearity?
Yes. Zero or span calibration can reduce stable offset and gain errors, while the underlying curve can remain nonlinear. Multi-point correction may reduce residual error only for the documented range, direction, conditions, and configuration.
Should sensor resolution be specified in bits or physical units?
Use physical units for the machine requirement. Bits describe nominal digital codes, but usable resolution also depends on sensor noise, input range, filtering, controller conversion, and the required update rate.
How much finer should resolution be than the process tolerance?
There is no universal ratio that fits every application. Define the smallest change the machine must distinguish, then verify that noise, quantization, repeatability, drift, and reference uncertainty leave enough decision margin under real conditions.
Does choosing 4–20 mA instead of 0–10 V improve sensor accuracy?
Not automatically. The interface changes susceptibility to voltage drop, ground differences, cable noise, input impedance, and fault detection, but the sensor, receiver, scaling, and installation still determine total measurement performance.
Can two sensors with ±0.1% FS be compared directly?
Only after confirming the same full-scale definition, used range, reference line, included error components, temperature conditions, output configuration, and maximum-versus-typical status. Different spans alone can produce different absolute error allowances.
Technical sources used for definitions and validation guidance
Use the exact sensor datasheet, calibration record, controller specification, and application test for the final decision.
- BIPM/JCGM VIM 3, 2.13 — measurement accuracy
- BIPM/JCGM VIM 3, 4.14 — resolution
- NIST Technical Note 1297 — evaluating and expressing measurement uncertainty
- NI Sensor Terminology — resolution, accuracy, offset, linearity, hysteresis, and response
- Analog Devices — noise, ENOB, effective resolution, and noise-free resolution
- TE Connectivity — best-fit straight-line linearity terminology
- Honeywell — Pressure Sensor Glossary of Terms